Please wait while we gather all the unique runs for this blueprint.
Please wait while we gather all the unique runs for this blueprint.
Please wait while we prepare the detailed comparison.
This evaluation assesses LLM clinical reasoning and safety awareness in complex, ambiguous cases where errors commonly arise from human cognitive bias, relational dynamics, and system gaps. It moves beyond factual recall to probe whether a model can navigate uncertainty, avoid premature closure and anchoring, and apply practical judgment when data sources conflict or are incomplete.
Scenarios are grounded in real-world cases from AHRQ's Patient Safety Network (PSNet), especially the expert-curated WebM&M series of anonymized medical error narratives. Using PSNet cases anchors rubrics in documented patient-safety events and authoritative commentary, ensuring evaluations are evidence-based rather than hypothetical.
Core Themes Tested:
Average performance for each system prompt variant across all models and prompts.
[No System Prompt]
Write your response as if documenting a brief handoff note to a clinical colleague. - Use a professional, concise tone; avoid chatty language or roleplay. - Prefer clear sections where natural: Assessment, Rationale, Plan, Safety/Follow-up. - State uncertainty explicitly and justify recommendations with key findings. - Focus on actionable steps and closed-loop processes; avoid boilerplate disclaimers. - Do not invent patient-specific data (vitals, labs, imaging). Use only what is provided or general best practice.
Average key point coverage, broken down by system prompt variant. Select a tab to view its results.
| Prompts vs. Models | Claude 3.5 Sonnet | Claude 3.7 Sonnet | Claude 3.5 Haiku | Claude Opus 4.1 | Claude Sonnet 4 | Deepseek Chat V3.1 | Deepseek R1 | Gemini 2.5 Flash | Gemini 2.5 Pro | Gemma 3 12b It | Llama 3 70b Instruct | Llama 4 Maverick | Meta Llama 3.1 405b Instruct Turbo | Mistral Large 2411 | Mistral Medium 3 | Mistral Nemo | GPT 4.1 Mini | GPT 4.1 Nano | GPT 4.1 | GPT 4o Mini | GPT 4o | GPT 5 | GPT OSS 120b | GPT OSS 20b | O4 Mini | GLM 4.5 | Qwen3 30b A3B Instruct 2507 | Qwen3 32b | Grok 3 | Grok 4 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Score | 20th 82.6% | 12th 88.4% | 24th 80.0% | 7th 90.5% | 15th 87.1% | 4th 92.2% | 11th 88.9% | 3rd 92.3% | 6th 91.6% | 13th 87.8% | 29th 73.6% | 27th 75.7% | 23rd 80.3% | 21st 82.6% | 10th 89.3% | 25th 78.8% | 22nd 80.6% | 26th 78.4% | 14th 87.4% | 28th 75.6% | 30th 73.6% | 1st 95.9% | 5th 92.1% | 18th 85.8% | 9th 90.3% | 2nd 93.7% | 19th 85.6% | 17th 86.2% | 8th 90.4% | 16th 86.4% | |
| 92.7% | 90% | 95% | 77% | 98% | 98% | 98% | 97% | 97% | 98% | 95% | 85% | 84% | 87% | 80% | 96% | 86% | 97% | 86% | 97% | 82% | 89% | 100% | 98% | 98% | 100% | 98% | 92% | 100% | 94% | 97% | |
| 83.1% | 83% | 91% | 79% | 92% | 87% | 90% | 89% | 91% | 98% | 83% | 59% | 67% | 82% | 86% | 91% | 48% | 84% | 65% | 87% | 68% | 77% | 98% | 92% | 87% | 96% | 94% | 76% | 83% | 89% | 87% | |
| 96.1% | 97% | 100% | 100% | 100% | 95% | 100% | 100% | 100% | 88% | 100% | 92% | 98% | 100% | 95% | 100% | 95% | 100% | 100% | 100% | 77% | 91% | 100% | 100% | 97% | 100% | 89% | 90% | 97% | 100% | 87% | |
| 84.6% | 83% | 89% | 82% | 93% | 86% | 97% | 87% | 96% | 98% | 92% | 71% | 83% | 65% | 70% | 82% | 79% | 81% | 75% | 73% | 74% | 70% | 94% | 90% | 85% | 80% | 99% | 98% | 85% | 96% | 92% | |
| 74.8% | 81% | 73% | 60% | 83% | 67% | 78% | 74% | 91% | 75% | 74% | 49% | 78% | 83% | 67% | 69% | 73% | 67% | 84% | 80% | 70% | 66% | 95% | 83% | 70% | 80% | 84% | 81% | 67% | 76% | 72% | |
| 88.1% | 81% | 91% | 85% | 85% | 93% | 89% | 91% | 92% | 93% | 93% | 84% | 80% | 84% | 80% | 95% | 82% | 85% | 92% | 96% | 79% | 75% | 96% | 93% | 90% | 89% | 90% | 88% | 84% | 98% | 98% | |
| 95.8% | 91% | 95% | 98% | 100% | 98% | 100% | 96% | 98% | 100% | 100% | 88% | 72% | 84% | 97% | 97% | 96% | 100% | 91% | 100% | 94% | 92% | 100% | 100% | 96% | 97% | 100% | 100% | 98% | 100% | 100% | |
| 90.9% | 79% | 89% | 82% | 88% | 88% | 91% | 88% | 97% | 96% | 80% | 87% | 83% | 89% | 97% | 96% | 93% | 89% | 95% | 91% | 100% | 93% | 97% | 93% | 88% | 96% | 94% | 92% | 95% | 93% | 92% | |
| 63.1% | 61% | 76% | 57% | 77% | 74% | 89% | 80% | 71% | 81% | 74% | 50% | 39% | 50% | 72% | 79% | 61% | 25% | 20% | 64% | 38% | 11% | 84% | 81% | 64% | 77% | 97% | 55% | 69% | 69% | 56% |