Loading blueprint versions...
Please wait while we gather all the unique runs for this blueprint.
Please wait while we gather all the unique runs for this blueprint.
Please wait while we prepare the detailed comparison.
This evaluation assesses LLM clinical reasoning and safety awareness in complex, ambiguous cases where errors commonly arise from human cognitive bias, relational dynamics, and system gaps. It moves beyond factual recall to probe whether a model can navigate uncertainty, avoid premature closure and anchoring, and apply practical judgment when data sources conflict or are incomplete.
Scenarios are grounded in real-world cases from AHRQ's Patient Safety Network (PSNet), especially the expert-curated WebM&M series of anonymized medical error narratives. Using PSNet cases anchors rubrics in documented patient-safety events and authoritative commentary, ensuring evaluations are evidence-based rather than hypothetical.
Core Themes Tested:
Average performance for each system prompt variant across all models and prompts.
[No System Prompt]
Write your response as if documenting a brief handoff note to a clinical colleague. - Use a professional, concise tone; avoid chatty language or roleplay. - Prefer clear sections where natural: Assessment, Rationale, Plan, Safety/Follow-up. - State uncertainty explicitly and justify recommendations with key findings. - Focus on actionable steps and closed-loop processes; avoid boilerplate disclaimers. - Do not invent patient-specific data (vitals, labs, imaging). Use only what is provided or general best practice.
Average key point coverage, broken down by system prompt variant. Select a tab to view its results.
| Prompts vs. Models | Claude 3.5 Sonnet | Claude 3.7 Sonnet | Claude 3.5 Haiku | Claude Opus 4.1 | Claude Sonnet 4 | Deepseek Chat V3.1 | Deepseek R1 | Gemini 2.5 Flash | Gemini 2.5 Pro | Gemma 3 12b It | Llama 3 70b Instruct | Llama 4 Maverick | Meta Llama 3.1 405b Instruct Turbo | Mistral Large 2411 | Mistral Medium 3 | Mistral Nemo | GPT 4.1 Mini | GPT 4.1 Nano | GPT 4.1 | GPT 4o Mini | GPT 4o | GPT 5 | GPT OSS 120b | GPT OSS 20b | O4 Mini | GLM 4.5 | Qwen3 30b A3B Instruct 2507 | Qwen3 32b | Grok 3 | Grok 4 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Score | 26th 78.4% | 14th 84.6% | 19th 82.3% | 6th 86.8% | 21st 81.1% | 7th 86.4% | 10th 85.6% | 5th 86.8% | 11th 85.3% | 13th 84.6% | 29th 77.2% | 28th 77.3% | 22nd 80.6% | 23rd 80.4% | 4th 87.3% | 27th 78.1% | 24th 78.9% | 20th 82.2% | 9th 85.8% | 25th 78.5% | 30th 73.1% | 1st 90.7% | 3rd 88.1% | 17th 82.6% | 16th 83.7% | 2nd 88.7% | 12th 85.3% | 18th 82.4% | 7th 86.4% | 15th 83.8% | |
| 84.8% | 76% | 90% | 89% | 77% | 78% | 81% | 77% | 77% | 86% | 83% | 96% | 89% | 89% | 89% | 88% | 87% | 86% | 97% | 88% | 95% | 86% | 77% | 80% | 76% | 82% | 82% | 98% | 87% | 86% | 77% | |
| 84.5% | 85% | 93% | 84% | 91% | 83% | 81% | 94% | 80% | 77% | 86% | 76% | 78% | 90% | 87% | 96% | 60% | 85% | 84% | 86% | 75% | 69% | 97% | 91% | 95% | 83% | 88% | 77% | 93% | 89% | 89% | |
| 95.5% | 97% | 100% | 97% | 100% | 92% | 97% | 100% | 100% | 82% | 100% | 88% | 98% | 100% | 95% | 100% | 97% | 100% | 97% | 100% | 76% | 92% | 93% | 97% | 96% | 100% | 92% | 95% | 100% | 100% | 90% | |
| 72.9% | 73% | 74% | 67% | 82% | 69% | 85% | 73% | 81% | 85% | 75% | 68% | 71% | 68% | 63% | 65% | 59% | 70% | 67% | 64% | 69% | 63% | 80% | 85% | 74% | 69% | 88% | 84% | 67% | 82% | 76% | |
| 74.0% | 73% | 68% | 67% | 79% | 57% | 76% | 77% | 90% | 84% | 72% | 49% | 76% | 82% | 62% | 80% | 69% | 65% | 83% | 80% | 72% | 71% | 93% | 81% | 69% | 78% | 82% | 71% | 65% | 74% | 80% | |
| 85.6% | 76% | 87% | 84% | 87% | 94% | 89% | 93% | 86% | 86% | 88% | 85% | 78% | 80% | 72% | 93% | 82% | 86% | 89% | 97% | 77% | 73% | 96% | 91% | 83% | 82% | 86% | 88% | 76% | 94% | 96% | |
| 87.6% | 80% | 82% | 97% | 91% | 90% | 90% | 90% | 93% | 92% | 94% | 84% | 71% | 71% | 85% | 93% | 87% | 81% | 87% | 93% | 94% | 82% | 93% | 88% | 84% | 89% | 90% | 97% | 90% | 90% | 88% | |
| 91.4% | 80% | 93% | 88% | 94% | 88% | 97% | 88% | 94% | 95% | 89% | 91% | 81% | 90% | 94% | 96% | 90% | 93% | 91% | 91% | 94% | 91% | 96% | 92% | 90% | 93% | 95% | 94% | 89% | 92% | 95% | |
| 71.4% | 68% | 75% | 70% | 81% | 80% | 84% | 80% | 81% | 82% | 77% | 58% | 56% | 55% | 78% | 77% | 75% | 45% | 46% | 75% | 54% | 34% | 93% | 90% | 77% | 79% | 97% | 67% | 77% | 73% | 64% |