Loading blueprint versions...
Please wait while we gather all the unique runs for this blueprint.
Please wait while we gather all the unique runs for this blueprint.
Please wait while we prepare the detailed comparison.
Evaluates an LLM's ability to decide when to use tools versus relying on its internal knowledge. Tests for overconfidence (not searching when it should) and underconfidence (searching unnecessarily).
Average key point coverage extent for each model across all prompts.
| Prompts vs. Models | Claude 3.5 Sonnet | Claude 3.7 Sonnet | Claude 3.5 Haiku | Claude Haiku 4.5 | Claude Opus 4.1 | Claude Sonnet 4 | Claude Sonnet 4.5 | Deepseek Chat V3.1 | Deepseek R1 | Gemini 2.5 Flash | Gemini 2.5 Pro | Gemma 3 12b It | Llama 3 70b Instruct | Llama 4 Maverick | Meta Llama 3.1 405b Instruct Turbo | Mistral Large 2411 | Mistral Medium 3 | Mistral Nemo | GPT 4.1 | GPT 4.1 Mini | GPT 4.1 Nano | GPT 4o | GPT 4o Mini | GPT 5 | GPT OSS 120b | GPT OSS 20b | O4 Mini | GLM 4.5 | Qwen3 30b A3B Instruct 2507 | Qwen3 32b | Grok 3 | Grok 4 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Score | 8th 82.7% | 4th 86.4% | 5th 86.0% | 28th 65.1% | 20th 73.6% | 3rd 87.3% | 5th 86.0% | 32nd 53.4% | 9th 82.6% | 13th 80.4% | 7th 84.6% | 29th 64.4% | 26th 70.6% | 2nd 88.3% | 24th 72.0% | 25th 71.9% | 16th 79.0% | 27th 67.9% | 12th 81.0% | 31st 58.6% | 22nd 73.3% | 15th 79.4% | 10th 81.9% | 19th 74.9% | 23rd 72.2% | 30th 62.8% | 13th 80.4% | 21st 73.4% | 11th 81.1% | 18th 75.3% | 1st 88.3% | 17th 76.4% | |
| 87.9% | 100% | 97% | 100% | 100% | 100% | 100% | 100% | 50% | 100% | 100% | 100% | 50% | 100% | 100% | 50% | 50% | 50% | 100% | 100% | 100% | 100% | 100% | 100% | 90% | 100% | 50% | 100% | 100% | 100% | 50% | |||
| 94.4% | 100% | 100% | 100% | 50% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 94% | 100% | 100% | 100% | 100% | 100% | 100% | 50% | 94% | 94% | 100% | 100% | 50% | 94% | 100% | 100% | 100% | 100% | 100% | 100% | ||
| 96.6% | 90% | 100% | 100% | 100% | 87% | 99% | 100% | 90% | 100% | 100% | 100% | 87% | 100% | 100% | 87% | 91% | 87% | 100% | 93% | 93% | 100% | 93% | 99% | 99% | 99% | 99% | 100% | 99% | 100% | 99% | 100% | 100% | |
| 87.8% | 89% | 100% | 100% | 89% | 100% | 89% | 100% | 67% | 100% | 90% | 90% | 89% | 63% | 89% | 39% | 90% | 100% | 100% | 90% | 50% | 90% | 90% | 100% | 90% | 90% | 90% | 92% | 90% | 93% | 89% | 83% | ||
| 75.1% | 100% | 100% | 100% | 11% | 22% | 100% | 100% | 67% | 78% | 73% | 100% | 79% | 79% | 79% | 78% | 72% | 73% | 82% | 67% | 79% | 79% | 74% | 33% | 44% | 73% | 71% | 78% | 83% | 100% | 79% | |||
| 46.9% | 50% | 50% | 50% | 50% | 50% | 50% | 50% | 0% | 50% | 50% | 50% | 0% | 0% | 100% | 100% | 50% | 50% | 0% | 50% | 50% | 50% | 50% | 50% | 100% | 50% | 50% | 50% | 50% | 50% | 0% | 50% | 50% | |
| 45.5% | 50% | 58% | 52% | 56% | 56% | 73% | 52% | 0% | 50% | 50% | 52% | 52% | 52% | 50% | 50% | 50% | 52% | 52% | 0% | 0% | 50% | 50% | 2% | 0% | 50% | 52% | 50% | 52% | 79% | 73% |