Loading blueprint versions...
Please wait while we gather all the unique runs for this blueprint.
Please wait while we gather all the unique runs for this blueprint.
Please wait while we prepare the detailed comparison.
Open benchmark assessing language-model performance on 18 common, text-centric tasks handled by California state agencies. Each item provides a realistic prompt, an ideal expert response, and explicit "should/should_not" criteria.
Average key point coverage extent for each model across all prompts.
| Prompts vs. Models | Claude 3.5 Haiku | Claude Opus 4 | Claude Sonnet 4 | Command A | Deepseek Chat V3 | Deepseek R1 | Gemini 2.5 Flash | Gemini 2.5 Pro | Llama 3 70b Instruct | Llama 4 Maverick | Meta Llama 3.1 405b Instruct Turbo | Mistral Large 2411 | Mistral Medium 3 | GPT 4.1 | GPT 4.1 Mini | GPT 4.1 Nano | GPT 4o | GPT 4o Mini | O4 Mini | Kimi K2 Instruct | Grok 3 | Grok 3 Mini | Grok 4 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Score | 20th 62.1% | 9th 73.8% | 13th 69.6% | 14th 69.5% | 8th 73.8% | 7th 73.9% | 12th 71.2% | 4th 78.8% | 17th 64.6% | 21st 62.1% | 15th 69.2% | 18th 63.7% | 6th 74.0% | 11th 71.8% | 16th 67.1% | 22nd 59.9% | 19th 63.5% | 23rd 59.3% | 3rd 79.2% | 9th 73.8% | 2nd 79.9% | 5th 74.1% | 1st 82.0% | |
| 75.5% | 80% | 80% | 73% | 80% | 80% | 70% | 70% | 80% | 73% | 70% | 80% | 83% | 80% | 80% | 65% | 50% | 80% | 65% | 80% | 60% | 90% | 88% | 80% | |
| 93.2% | 100% | 100% | 80% | 100% | 100% | 100% | 80% | 80% | 80% | 60% | 100% | 80% | 100% | 100% | 100% | 83% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | |
| 74.2% | 60% | 78% | 80% | 60% | 93% | 60% | 77% | 90% | 53% | 70% | 65% | 70% | 80% | 80% | 93% | 70% | 65% | 82% | 90% | 80% | 60% | 70% | 80% | |
| 47.8% | 40% | 30% | 50% | 50% | 40% | 50% | 50% | 95% | 40% | 50% | 40% | 30% | 40% | 27% | 50% | 40% | 40% | 20% | 85% | 50% | 50% | 50% | 83% | |
| 48.1% | 40% | 50% | 40% | 43% | 43% | 60% | 33% | 55% | 43% | 63% | 60% | 40% | 65% | 45% | 40% | 40% | 50% | 40% | 40% | 40% | 65% | 43% | 68% | |
| 79.0% | 60% | 80% | 60% | 85% | 83% | 83% | 83% | 80% | 90% | 50% | 80% | 80% | 85% | 70% | 83% | 83% | 70% | 80% | 70% | 80% | 83% | 98% | 100% | |
| 60.5% | 60% | 73% | 65% | 57% | 53% | 78% | 60% | 68% | 55% | 40% | 60% | 60% | 68% | 78% | 63% | 60% | 60% | 60% | 45% | 68% | 60% | 55% | 45% | |
| 53.1% | 52% | 53% | 55% | 50% | 43% | 73% | 40% | 60% | 45% | 57% | 40% | 73% | 45% | 43% | 60% | 48% | 43% | 25% | 80% | 80% | 63% | 33% | 60% | |
| 71.6% | 65% | 93% | 85% | 88% | 75% | 83% | 78% | 68% | 60% | 73% | 73% | 60% | 58% | 65% | 60% | 50% | 60% | 60% | 95% | 60% | 75% | 70% | 93% | |
| 82.9% | 59% | 100% | 89% | 88% | 84% | 79% | 93% | 88% | 79% | 71% | 71% | 71% | 86% | 75% | 80% | 89% | 71% | 86% | 77% | 84% | 98% | 93% | 96% | |
| 71.8% | 70% | 73% | 70% | 78% | 73% | 73% | 75% | 83% | 68% | 65% | 65% | 70% | 75% | 73% | 85% | 50% | 70% | 48% | 75% | 80% | 90% | 62% | 80% | |
| 81.8% | 75% | 100% | 72% | 91% | 84% | 75% | 100% | 100% | 91% | 78% | 63% | 66% | 88% | 100% | 53% | 56% | 75% | 75% | 91% | 66% | 91% | 91% | 100% | |
| 63.1% | 60% | 60% | 80% | 60% | 80% | 63% | 60% | 43% | 50% | 60% | 60% | 63% | 67% | 63% | 60% | 60% | 60% | 48% | 85% | 60% | 83% | 63% | 63% | |
| 64.3% | 53% | 40% | 80% | 80% | 68% | 60% | 80% | 70% | 73% | 60% | 63% | 63% | 60% | 68% | 50% | 53% | 48% | 55% | 60% | 68% | 80% | 68% | 80% | |
| 62.4% | 33% | 68% | 60% | 40% | 100% | 73% | 33% | 88% | 53% | 20% | 60% | 50% | 85% | 75% | 43% | 33% | 68% | 20% | 80% | 80% | 73% | 100% | 100% | |
| 66.0% | 60% | 78% | 75% | 43% | 55% | 60% | 80% | 80% | 63% | 75% | 80% | 43% | 80% | 80% | 63% | 43% | 45% | 40% | 73% | 73% | 80% | 80% | 68% | |
| 76.3% | 68% | 73% | 58% | 60% | 75% | 90% | 90% | 90% | 77% | 77% | 85% | 45% | 70% | 70% | 60% | 70% | 68% | 78% | 100% | 100% | 100% | 70% | 80% | |
| 94.0% | 83% | 100% | 80% | 98% | 100% | 100% | 100% | 100% | 70% | 78% | 100% | 100% | 100% | 100% | 100% | 100% | 70% | 85% | 100% | 100% | 97% | 100% | 100% |