GPQA diamond
GPQA Diamond, run by Epoch AI.
No filters. Showing 338 of 338 rows.
1[1–19] GoogleDossier | high | No harness | Gemini 4 Carbon, Accuracy (best score across scorers): 96.5% (scale 0.0 to 100.0), 95% confidence interval 94.2 to 98.996.5%(94.2–98.9) | 96.5% | Runs unstated | Cost kind unknown |
2[1–17] GoogleDossierTied #1 | high | No harness | Gemini 4 Argon, Accuracy (best score across scorers): 96.4% (scale 0.0 to 100.0), 95% confidence interval 94.1 to 98.796.4%(94.1–98.7) | 96.4% | Runs unstated | Cost kind unknown |
3[1–26] AnthropicDossierTied #1 | max · fallback | No harness | Fable 5.2, Accuracy (best score across scorers): 96% (scale 0.0 to 100.0), 95% confidence interval 93.4 to 98.696%(93.4–98.6) | 96% | Runs unstated | Cost kind unknown |
4[1–29] OpenAIDossierTied #1 | max | No harness | GPT-6 Astra, Accuracy (best score across scorers): 95.8% (scale 0.0 to 100.0), 95% confidence interval 93.1 to 98.595.8%(93.1–98.5) | 95.8% | Runs unstated | Cost kind unknown |
5[1–32] AnthropicDossierTied #1 | max | No harness | Sonnet 5.5, Accuracy (best score across scorers): 95.6% (scale 0.0 to 100.0), 95% confidence interval 92.9 to 98.395.6%(92.9–98.3) | 95.6% | Runs unstated | Cost kind unknown |
6[1–30] GoogleDossierTied #1 | high | No harness | Gemini 3.8 Flash, Accuracy (best score across scorers): 95.4% (scale 0.0 to 100.0), 95% confidence interval 92.6 to 98.195.4%(92.6–98.1) | 95.4% | Runs unstated | Cost kind unknown |
6[1–32] OpenAIDossierTied #1 | max | No harness | GPT-6.1 Sol, Accuracy (best score across scorers): 95.4% (scale 0.0 to 100.0), 95% confidence interval 92.7 to 98.195.4%(92.7–98.1) | 95.4% | Runs unstated | Cost kind unknown |
8[1–35] GoogleDossierTied #1 | high | No harness | Gemini 3.7 Flash, Accuracy (best score across scorers): 94.8% (scale 0.0 to 100.0), 95% confidence interval 92.2 to 97.594.8%(92.2–97.5) | 94.8% | Runs unstated | Cost kind unknown |
9[1–49] OpenAIDossierTied #1 | xhigh | No harness | GPT-5.4 Pro, Accuracy (best score across scorers): 94.6% (scale 0.0 to 100.0), 95% confidence interval 91.5 to 97.794.6%(91.5–97.7) | 94.6% | Runs unstated | Cost kind unknown |
10[2–48] GoogleDossierTied #1 | high | No harness | Gemini 3.1 Pro Preview, Accuracy (best score across scorers): 94.4% (scale 0.0 to 100.0), 95% confidence interval 91.2 to 97.694.4%(91.2–97.6) | 94.4% | Runs unstated | Cost kind unknown |
11[3–46] OpenAIDossierTied #1 | max | No harness | GPT-6 Sol, Accuracy (best score across scorers): 94.3% (scale 0.0 to 100.0), 95% confidence interval 91.4 to 97.294.3%(91.4–97.2) | 94.3% | Runs unstated | Cost kind unknown |
11[2–47] OpenAIDossierTied #1 | max | No harness | GPT-6.1 Terra, Accuracy (best score across scorers): 94.3% (scale 0.0 to 100.0), 95% confidence interval 91.4 to 97.294.3%(91.4–97.2) | 94.3% | Runs unstated | Cost kind unknown |
13[2–44] Moonshot AIDossierTied #1 | max | No harness | Kimi K3.1 Pro, Accuracy (best score across scorers): 94.2% (scale 0.0 to 100.0), 95% confidence interval 91.4 to 97.094.2%(91.4–97.0) | 94.2% | Runs unstated | Cost kind unknown |
14[3–44] GoogleDossierTied #1 | high | No harness | Gemini 3.6 Flash, Accuracy (best score across scorers): 94.1% (scale 0.0 to 100.0), 95% confidence interval 91.4 to 96.994.1%(91.4–96.9) | 94.1% | Runs unstated | Cost kind unknown |
15[2–51] GoogleDossierTied #1 | No effort set | No harness | Gemini 3.1 Pro Preview, Accuracy (best score across scorers): 94.1% (scale 0.0 to 100.0), 95% confidence interval 90.8 to 97.494.1%(90.8–97.4) | 94.1% | Runs unstated | Cost kind unknown |
16[3–47] AnthropicDossierTied #1 | max · fallback | No harness | Opus 5.5, Accuracy (best score across scorers): 94.1% (scale 0.0 to 100.0), 95% confidence interval 91.1 to 97.094.1%(91.1–97.0) | 94.1% | Runs unstated | Cost kind unknown |
17[3–53] OpenAIDossierTied #1 | xhigh | No harness | GPT-5.5 Pre-release, Accuracy (best score across scorers): 94% (scale 0.0 to 100.0), 95% confidence interval 91.0 to 97.094%(91.0–97.0) | 94% | Runs unstated | Cost kind unknown |
17[4–44] xAIDossierTied #1 | high | No harness | Grok 4.6, Accuracy (best score across scorers): 94% (scale 0.0 to 100.0), 95% confidence interval 91.2 to 96.894%(91.2–96.8) | 94% | Runs unstated | Cost kind unknown |
19[2–52] OpenAIDossierTied #1 | xhigh | No harness | GPT-5.5 Pro Pre-release, Accuracy (best score across scorers): 93.9% (scale 0.0 to 100.0), 95% confidence interval 90.8 to 97.093.9%(90.8–97.0) | 93.9% | Runs unstated | Cost kind unknown |
20[2–51] AnthropicDossierTied #1 | max · fallback | No harness | Fable 5, Accuracy (best score across scorers): 93.9% (scale 0.0 to 100.0), 95% confidence interval 90.6 to 97.293.9%(90.6–97.2) | 93.9% | Runs unstated | Cost kind unknown |
20[3–49] AnthropicDossierTied #1 | max | No harness | Fable 5.2, Accuracy (best score across scorers): 93.9% (scale 0.0 to 100.0), 95% confidence interval 90.7 to 97.193.9%(90.7–97.1) | 93.9% | Runs unstated | Cost kind unknown |
20[3–51] AnthropicDossierTied #1 | max | No harness | Opus 5, Accuracy (best score across scorers): 93.9% (scale 0.0 to 100.0), 95% confidence interval 91.0 to 96.893.9%(91.0–96.8) | 93.9% | Runs unstated | Cost kind unknown |
- 1GoogleDossier
high
Accuracy (best score across scorers)
Gemini 4 Carbon, Accuracy (best score across scorers): 96.5% (scale 0.0 to 100.0), 95% confidence interval 94.2 to 98.996.5%(94.2–98.9)- Mean score
- 96.5%
Accuracy (best score across scorers)
Gemini 4 Argon, Accuracy (best score across scorers): 96.4% (scale 0.0 to 100.0), 95% confidence interval 94.1 to 98.796.4%(94.1–98.7)- Mean score
- 96.4%
Accuracy (best score across scorers)
Fable 5.2, Accuracy (best score across scorers): 96% (scale 0.0 to 100.0), 95% confidence interval 93.4 to 98.696%(93.4–98.6)- Mean score
- 96%
Accuracy (best score across scorers)
GPT-6 Astra, Accuracy (best score across scorers): 95.8% (scale 0.0 to 100.0), 95% confidence interval 93.1 to 98.595.8%(93.1–98.5)- Mean score
- 95.8%
Accuracy (best score across scorers)
Sonnet 5.5, Accuracy (best score across scorers): 95.6% (scale 0.0 to 100.0), 95% confidence interval 92.9 to 98.395.6%(92.9–98.3)- Mean score
- 95.6%
Accuracy (best score across scorers)
Gemini 3.8 Flash, Accuracy (best score across scorers): 95.4% (scale 0.0 to 100.0), 95% confidence interval 92.6 to 98.195.4%(92.6–98.1)- Mean score
- 95.4%
Accuracy (best score across scorers)
GPT-6.1 Sol, Accuracy (best score across scorers): 95.4% (scale 0.0 to 100.0), 95% confidence interval 92.7 to 98.195.4%(92.7–98.1)- Mean score
- 95.4%
Accuracy (best score across scorers)
Gemini 3.7 Flash, Accuracy (best score across scorers): 94.8% (scale 0.0 to 100.0), 95% confidence interval 92.2 to 97.594.8%(92.2–97.5)- Mean score
- 94.8%
Accuracy (best score across scorers)
GPT-5.4 Pro, Accuracy (best score across scorers): 94.6% (scale 0.0 to 100.0), 95% confidence interval 91.5 to 97.794.6%(91.5–97.7)- Mean score
- 94.6%
Accuracy (best score across scorers)
Gemini 3.1 Pro Preview, Accuracy (best score across scorers): 94.4% (scale 0.0 to 100.0), 95% confidence interval 91.2 to 97.694.4%(91.2–97.6)- Mean score
- 94.4%
Accuracy (best score across scorers)
GPT-6 Sol, Accuracy (best score across scorers): 94.3% (scale 0.0 to 100.0), 95% confidence interval 91.4 to 97.294.3%(91.4–97.2)- Mean score
- 94.3%
Accuracy (best score across scorers)
GPT-6.1 Terra, Accuracy (best score across scorers): 94.3% (scale 0.0 to 100.0), 95% confidence interval 91.4 to 97.294.3%(91.4–97.2)- Mean score
- 94.3%
Accuracy (best score across scorers)
Kimi K3.1 Pro, Accuracy (best score across scorers): 94.2% (scale 0.0 to 100.0), 95% confidence interval 91.4 to 97.094.2%(91.4–97.0)- Mean score
- 94.2%
Accuracy (best score across scorers)
Gemini 3.6 Flash, Accuracy (best score across scorers): 94.1% (scale 0.0 to 100.0), 95% confidence interval 91.4 to 96.994.1%(91.4–96.9)- Mean score
- 94.1%
Accuracy (best score across scorers)
Gemini 3.1 Pro Preview, Accuracy (best score across scorers): 94.1% (scale 0.0 to 100.0), 95% confidence interval 90.8 to 97.494.1%(90.8–97.4)- Mean score
- 94.1%
Accuracy (best score across scorers)
Opus 5.5, Accuracy (best score across scorers): 94.1% (scale 0.0 to 100.0), 95% confidence interval 91.1 to 97.094.1%(91.1–97.0)- Mean score
- 94.1%
Accuracy (best score across scorers)
GPT-5.5 Pre-release, Accuracy (best score across scorers): 94% (scale 0.0 to 100.0), 95% confidence interval 91.0 to 97.094%(91.0–97.0)- Mean score
- 94%
Accuracy (best score across scorers)
Grok 4.6, Accuracy (best score across scorers): 94% (scale 0.0 to 100.0), 95% confidence interval 91.2 to 96.894%(91.2–96.8)- Mean score
- 94%
Accuracy (best score across scorers)
GPT-5.5 Pro Pre-release, Accuracy (best score across scorers): 93.9% (scale 0.0 to 100.0), 95% confidence interval 90.8 to 97.093.9%(90.8–97.0)- Mean score
- 93.9%
Accuracy (best score across scorers)
Fable 5, Accuracy (best score across scorers): 93.9% (scale 0.0 to 100.0), 95% confidence interval 90.6 to 97.293.9%(90.6–97.2)- Mean score
- 93.9%
Accuracy (best score across scorers)
Fable 5.2, Accuracy (best score across scorers): 93.9% (scale 0.0 to 100.0), 95% confidence interval 90.7 to 97.193.9%(90.7–97.1)- Mean score
- 93.9%
Accuracy (best score across scorers)
Opus 5, Accuracy (best score across scorers): 93.9% (scale 0.0 to 100.0), 95% confidence interval 91.0 to 96.893.9%(91.0–96.8)- Mean score
- 93.9%
Accuracy (best score across scorers)
GPT-5.6 Sol, Accuracy (best score across scorers): 93.5% (scale 0.0 to 100.0), 95% confidence interval 90.4 to 96.693.5%(90.4–96.6)- Mean score
- 93.5%
Accuracy (best score across scorers)
Grok 4.5, Accuracy (best score across scorers): 93.4% (scale 0.0 to 100.0), 95% confidence interval 90.6 to 96.293.4%(90.6–96.2)- Mean score
- 93.4%
Accuracy (best score across scorers)
GPT-5.6 Terra, Accuracy (best score across scorers): 93.3% (scale 0.0 to 100.0), 95% confidence interval 90.3 to 96.393.3%(90.3–96.3)- Mean score
- 93.3%
Accuracy (best score across scorers)
GPT-5.4, Accuracy (best score across scorers): 93.3% (scale 0.0 to 100.0), 95% confidence interval 89.8 to 96.893.3%(89.8–96.8)- Mean score
- 93.3%
Accuracy (best score across scorers)
DeepSeek V4.1 Flash 1017, Accuracy (best score across scorers): 93.2% (scale 0.0 to 100.0), 95% confidence interval 90.1 to 96.393.2%(90.1–96.3)- Mean score
- 93.2%
Accuracy (best score across scorers)
Grok 4.6, Accuracy (best score across scorers): 93.2% (scale 0.0 to 100.0), 95% confidence interval 90.2 to 96.293.2%(90.2–96.2)- Mean score
- 93.2%
Accuracy (best score across scorers)
Doubao Seed Evolving, Accuracy (best score across scorers): 93.1% (scale 0.0 to 100.0), 95% confidence interval 90.0 to 96.293.1%(90.0–96.2)- Mean score
- 93.1%
Accuracy (best score across scorers)
Kimi K3, Accuracy (best score across scorers): 93.1% (scale 0.0 to 100.0), 95% confidence interval 90.2 to 96.093.1%(90.2–96.0)- Mean score
- 93.1%
Accuracy (best score across scorers)
Opus 5, Accuracy (best score across scorers): 92.9% (scale 0.0 to 100.0), 95% confidence interval 89.3 to 96.592.9%(89.3–96.5)- Mean score
- 92.9%
Accuracy (best score across scorers)
Gemini 3.5 Flash, Accuracy (best score across scorers): 92.8% (scale 0.0 to 100.0), 95% confidence interval 89.6 to 96.092.8%(89.6–96.0)- Mean score
- 92.8%
Accuracy (best score across scorers)
Grok 4.7, Accuracy (best score across scorers): 92.7% (scale 0.0 to 100.0), 95% confidence interval 89.7 to 95.892.7%(89.7–95.8)- Mean score
- 92.7%
Accuracy (best score across scorers)
Qwen3.8-Max, Accuracy (best score across scorers): 92.7% (scale 0.0 to 100.0), 95% confidence interval 89.4 to 96.092.7%(89.4–96.0)- Mean score
- 92.7%
Accuracy (best score across scorers)
Gemini 3 Pro Preview, Accuracy (best score across scorers): 92.6% (scale 0.0 to 100.0), 95% confidence interval 89.4 to 95.992.6%(89.4–95.9)- Mean score
- 92.6%
- 36AlibabaDossier
xhigh
Accuracy (best score across scorers)
Qwen3.8 Max (0902), Accuracy (best score across scorers): 92.3% (scale 0.0 to 100.0), 95% confidence interval 88.9 to 95.792.3%(88.9–95.7)- Mean score
- 92.3%
- 37Moonshot AIDossier
high
Accuracy (best score across scorers)
Kimi K3, Accuracy (best score across scorers): 91.9% (scale 0.0 to 100.0), 95% confidence interval 88.1 to 95.791.9%(88.1–95.7)- Mean score
- 91.9%
- 38Z.aiDossier
max
Accuracy (best score across scorers)
GLM-5.2, Accuracy (best score across scorers): 91.9% (scale 0.0 to 100.0), 95% confidence interval 88.7 to 95.091.9%(88.7–95.0)- Mean score
- 91.9%
- 39DeepSeekDossier
max
Accuracy (best score across scorers)
DeepSeek V4 Pro 0813, Accuracy (best score across scorers): 91.7% (scale 0.0 to 100.0), 95% confidence interval 88.7 to 94.791.7%(88.7–94.7)- Mean score
- 91.7%
- 40OpenAIDossier
max
Accuracy (best score across scorers)
GPT-5.6 Luna, Accuracy (best score across scorers): 91.6% (scale 0.0 to 100.0), 95% confidence interval 88.2 to 95.091.6%(88.2–95.0)- Mean score
- 91.6%
- 41OpenAIDossier
xhigh
Accuracy (best score across scorers)
GPT-5.2, Accuracy (best score across scorers): 91.4% (scale 0.0 to 100.0), 95% confidence interval 87.9 to 94.991.4%(87.9–94.9)- Mean score
- 91.4%
- 42DeepSeekDossier
max
Accuracy (best score across scorers)
DeepSeek V4-Flash 0731, Accuracy (best score across scorers): 91% (scale 0.0 to 100.0), 95% confidence interval 87.6 to 94.591%(87.6–94.5)- Mean score
- 91%
- 42AnthropicDossier
max
Accuracy (best score across scorers)
Opus 4.8, Accuracy (best score across scorers): 91% (scale 0.0 to 100.0), 95% confidence interval 87.3 to 94.891%(87.3–94.8)- Mean score
- 91%
- 44DeepSeekDossier
high
Accuracy (best score across scorers)
DeepSeek V4-Pro, Accuracy (best score across scorers): 90.9% (scale 0.0 to 100.0), 95% confidence interval 86.9 to 94.990.9%(86.9–94.9)- Mean score
- 90.9%
- 44Z.aiDossier
max
Accuracy (best score across scorers)
GLM-5.3, Accuracy (best score across scorers): 90.9% (scale 0.0 to 100.0), 95% confidence interval 87.8 to 94.090.9%(87.8–94.0)- Mean score
- 90.9%
- 44MiniMaxDossier
Accuracy (best score across scorers)
MiniMax M3, Accuracy (best score across scorers): 90.9% (scale 0.0 to 100.0), 95% confidence interval 86.9 to 94.990.9%(86.9–94.9)- Mean score
- 90.9%
- 44AlibabaDossier
Accuracy (best score across scorers)
Qwen3.7 Max, Accuracy (best score across scorers): 90.9% (scale 0.0 to 100.0), 95% confidence interval 86.9 to 94.990.9%(86.9–94.9)- Mean score
- 90.9%
- 48Moonshot AIDossier
Accuracy (best score across scorers)
Kimi K2.6, Accuracy (best score across scorers): 90.8% (scale 0.0 to 100.0), 95% confidence interval 87.4 to 94.190.8%(87.4–94.1)- Mean score
- 90.8%
- 49OpenAIDossier
low
Accuracy (best score across scorers)
GPT-5.5, Accuracy (best score across scorers): 90.7% (scale 0.0 to 100.0), 95% confidence interval 87.2 to 94.190.7%(87.2–94.1)- Mean score
- 90.7%
- 50AnthropicDossier
max
Accuracy (best score across scorers)
Opus 5.5, Accuracy (best score across scorers): 90.6% (scale 0.0 to 100.0), 95% confidence interval 86.8 to 94.490.6%(86.8–94.4)- Mean score
- 90.6%
- 51AnthropicDossier
32,000 budget
Accuracy (best score across scorers)
Opus 4.6, Accuracy (best score across scorers): 90.5% (scale 0.0 to 100.0), 95% confidence interval 87.2 to 93.890.5%(87.2–93.8)- Mean score
- 90.5%
- 51AnthropicDossier
xhigh
Accuracy (best score across scorers)
Sonnet 5, Accuracy (best score across scorers): 90.5% (scale 0.0 to 100.0), 95% confidence interval 87.0 to 94.090.5%(87.0–94.0)- Mean score
- 90.5%
- 53OpenAIDossier
max
Accuracy (best score across scorers)
GPT-6 Luna, Accuracy (best score across scorers): 90.5% (scale 0.0 to 100.0), 95% confidence interval 86.8 to 94.190.5%(86.8–94.1)- Mean score
- 90.5%
- 54Z.aiDossier
max
Accuracy (best score across scorers)
GLM-5.3 Flash, Accuracy (best score across scorers): 90.2% (scale 0.0 to 100.0), 95% confidence interval 86.8 to 93.590.2%(86.8–93.5)- Mean score
- 90.2%
- 54MistralDossier
high
Accuracy (best score across scorers)
Mistral Large 4 OSS, Accuracy (best score across scorers): 90.2% (scale 0.0 to 100.0), 95% confidence interval 86.6 to 93.790.2%(86.6–93.7)- Mean score
- 90.2%
- 54AnthropicDossier
xhigh
Accuracy (best score across scorers)
Opus 4.7, Accuracy (best score across scorers): 90.2% (scale 0.0 to 100.0), 95% confidence interval 86.5 to 93.890.2%(86.5–93.8)- Mean score
- 90.2%
- 57Z.aiDossier
Accuracy (best score across scorers)
GLM-5.1, Accuracy (best score across scorers): 89.9% (scale 0.0 to 100.0), 95% confidence interval 85.7 to 94.189.9%(85.7–94.1)- Mean score
- 89.9%
- 57OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5.4, Accuracy (best score across scorers): 89.9% (scale 0.0 to 100.0), 95% confidence interval 85.7 to 94.189.9%(85.7–94.1)- Mean score
- 89.9%
- 57OpenAIDossier
low
Accuracy (best score across scorers)
GPT-5.6 Sol, Accuracy (best score across scorers): 89.9% (scale 0.0 to 100.0), 95% confidence interval 85.7 to 94.189.9%(85.7–94.1)- Mean score
- 89.9%
- 60DeepSeekDossier
max
Accuracy (best score across scorers)
DeepSeek V4.1 Flash, Accuracy (best score across scorers): 89.8% (scale 0.0 to 100.0), 95% confidence interval 86.4 to 93.289.8%(86.4–93.2)- Mean score
- 89.8%
- 61MetaDossier
Accuracy (best score across scorers)
Muse Spark, Accuracy (best score across scorers): 89.8% (scale 0.0 to 100.0), 95% confidence interval 85.5 to 94.189.8%(85.5–94.1)- Mean score
- 89.8%
- 62DeepSeekDossier
max
Accuracy (best score across scorers)
DeepSeek V4-Pro, Accuracy (best score across scorers): 89.6% (scale 0.0 to 100.0), 95% confidence interval 86.2 to 93.189.6%(86.2–93.1)- Mean score
- 89.6%
- 62AnthropicDossier
max
Accuracy (best score across scorers)
Haiku 5.5, Accuracy (best score across scorers): 89.6% (scale 0.0 to 100.0), 95% confidence interval 85.8 to 93.589.6%(85.8–93.5)- Mean score
- 89.6%
- 64ByteDanceDossier
high
Accuracy (best score across scorers)
Doubao Seed 2.1 Turbo (260628), Accuracy (best score across scorers): 89.6% (scale 0.0 to 100.0), 95% confidence interval 86.1 to 93.189.6%(86.1–93.1)- Mean score
- 89.6%
- 65GoogleDossier
high
Accuracy (best score across scorers)
Gemini 3 Flash Preview, Accuracy (best score across scorers): 89.4% (scale 0.0 to 100.0), 95% confidence interval 85.1 to 93.789.4%(85.1–93.7)- Mean score
- 89.4%
- 66ByteDanceDossier
high
Accuracy (best score across scorers)
Doubao Seed 2.1 Pro (260628), Accuracy (best score across scorers): 89.3% (scale 0.0 to 100.0), 95% confidence interval 85.7 to 92.989.3%(85.7–92.9)- Mean score
- 89.3%
- 66xAIDossier
Accuracy (best score across scorers)
Grok 4.20 0309, Accuracy (best score across scorers): 89.3% (scale 0.0 to 100.0), 95% confidence interval 85.7 to 93.089.3%(85.7–93.0)- Mean score
- 89.3%
- 68GoogleDossier
low
Accuracy (best score across scorers)
Gemini 3.5 Flash, Accuracy (best score across scorers): 88.9% (scale 0.0 to 100.0), 95% confidence interval 84.5 to 93.388.9%(84.5–93.3)- Mean score
- 88.9%
- 68OpenAIDossier
medium
Accuracy (best score across scorers)
GPT-5.4, Accuracy (best score across scorers): 88.9% (scale 0.0 to 100.0), 95% confidence interval 84.5 to 93.388.9%(84.5–93.3)- Mean score
- 88.9%
- 70xAIDossier
high
Accuracy (best score across scorers)
Grok 4.3, Accuracy (best score across scorers): 88.8% (scale 0.0 to 100.0), 95% confidence interval 85.0 to 92.788.8%(85.0–92.7)- Mean score
- 88.8%
- 71AnthropicDossier
64,000 budget
Accuracy (best score across scorers)
Opus 4.6, Accuracy (best score across scorers): 88.8% (scale 0.0 to 100.0), 95% confidence interval 85.0 to 92.688.8%(85.0–92.6)- Mean score
- 88.8%
- 72MetaDossier
xhigh
Accuracy (best score across scorers)
Muse Spark Open, Accuracy (best score across scorers): 88.6% (scale 0.0 to 100.0), 95% confidence interval 84.9 to 92.288.6%(84.9–92.2)- Mean score
- 88.6%
- 73Thinking MachinesDossier
xhigh
Accuracy (best score across scorers)
Inkling Small, Accuracy (best score across scorers): 88.5% (scale 0.0 to 100.0), 95% confidence interval 84.8 to 92.388.5%(84.8–92.3)- Mean score
- 88.5%
- 74AnthropicDossier
max
Accuracy (best score across scorers)
Opus 4.6, Accuracy (best score across scorers): 88.4% (scale 0.0 to 100.0), 95% confidence interval 83.9 to 92.988.4%(83.9–92.9)- Mean score
- 88.4%
- 74AnthropicDossier
low
Accuracy (best score across scorers)
Opus 4.8, Accuracy (best score across scorers): 88.4% (scale 0.0 to 100.0), 95% confidence interval 83.9 to 92.988.4%(83.9–92.9)- Mean score
- 88.4%
- 74AlibabaDossier
Accuracy (best score across scorers)
Qwen3.6 Plus, Accuracy (best score across scorers): 88.4% (scale 0.0 to 100.0), 95% confidence interval 83.9 to 92.988.4%(83.9–92.9)- Mean score
- 88.4%
- 77Thinking MachinesDossier
xhigh
Accuracy (best score across scorers)
Inkling, Accuracy (best score across scorers): 88.3% (scale 0.0 to 100.0), 95% confidence interval 84.5 to 92.188.3%(84.5–92.1)- Mean score
- 88.3%
- 78ByteDanceDossier
high
Accuracy (best score across scorers)
Doubao Seed 2.0 Pro (260215), Accuracy (best score across scorers): 88.2% (scale 0.0 to 100.0), 95% confidence interval 84.4 to 92.088.2%(84.4–92.0)- Mean score
- 88.2%
- 78OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5.2, Accuracy (best score across scorers): 88.2% (scale 0.0 to 100.0), 95% confidence interval 84.4 to 91.988.2%(84.4–91.9)- Mean score
- 88.2%
- 80ByteDanceDossier
high
Accuracy (best score across scorers)
Doubao Seed 2.0 Lite (260428), Accuracy (best score across scorers): 88.1% (scale 0.0 to 100.0), 95% confidence interval 84.3 to 91.988.1%(84.3–91.9)- Mean score
- 88.1%
- 81MistralDossier
high
Accuracy (best score across scorers)
Mistral Large 4, Accuracy (best score across scorers): 88.1% (scale 0.0 to 100.0), 95% confidence interval 84.3 to 91.988.1%(84.3–91.9)- Mean score
- 88.1%
- 82Z.aiDossier
low
Accuracy (best score across scorers)
GLM-5.2, Accuracy (best score across scorers): 87.9% (scale 0.0 to 100.0), 95% confidence interval 83.3 to 92.487.9%(83.3–92.4)- Mean score
- 87.9%
- 82OpenAIDossier
medium
Accuracy (best score across scorers)
GPT-5.2, Accuracy (best score across scorers): 87.9% (scale 0.0 to 100.0), 95% confidence interval 84.1 to 91.687.9%(84.1–91.6)- Mean score
- 87.9%
- 82Moonshot AIDossier
Accuracy (best score across scorers)
Kimi K2.7 Code, Accuracy (best score across scorers): 87.9% (scale 0.0 to 100.0), 95% confidence interval 83.3 to 92.487.9%(83.3–92.4)- Mean score
- 87.9%
- 82AnthropicDossier
low
Accuracy (best score across scorers)
Opus 5, Accuracy (best score across scorers): 87.9% (scale 0.0 to 100.0), 95% confidence interval 83.3 to 92.487.9%(83.3–92.4)- Mean score
- 87.9%
- 82AlibabaDossier
Accuracy (best score across scorers)
Qwen3.7 Plus, Accuracy (best score across scorers): 87.9% (scale 0.0 to 100.0), 95% confidence interval 83.3 to 92.487.9%(83.3–92.4)- Mean score
- 87.9%
- 87Z.aiDossier
Accuracy (best score across scorers)
GLM-5, Accuracy (best score across scorers): 87.8% (scale 0.0 to 100.0), 95% confidence interval 83.2 to 92.487.8%(83.2–92.4)- Mean score
- 87.8%
- 88OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5.1, Accuracy (best score across scorers): 87.6% (scale 0.0 to 100.0), 95% confidence interval 84.0 to 91.387.6%(84.0–91.3)- Mean score
- 87.6%
- 89Moonshot AIDossier
Accuracy (best score across scorers)
Kimi K2.5, Accuracy (best score across scorers): 87.6% (scale 0.0 to 100.0), 95% confidence interval 83.9 to 91.387.6%(83.9–91.3)- Mean score
- 87.6%
- 90OpenAIDossier
low
Accuracy (best score across scorers)
GPT-5.6 Terra, Accuracy (best score across scorers): 87.4% (scale 0.0 to 100.0), 95% confidence interval 82.7 to 92.087.4%(82.7–92.0)- Mean score
- 87.4%
- 90AlibabaDossier
Accuracy (best score across scorers)
Qwen3.6 Max Preview, Accuracy (best score across scorers): 87.4% (scale 0.0 to 100.0), 95% confidence interval 82.7 to 92.087.4%(82.7–92.0)- Mean score
- 87.4%
- 90AnthropicDossier
32,000 budget
Accuracy (best score across scorers)
Sonnet 4.6, Accuracy (best score across scorers): 87.4% (scale 0.0 to 100.0), 95% confidence interval 83.4 to 91.387.4%(83.4–91.3)- Mean score
- 87.4%
- 93xAIDossier
Accuracy (best score across scorers)
Grok 4 (0709), Accuracy (best score across scorers): 87% (scale 0.0 to 100.0), 95% confidence interval 83.1 to 90.987%(83.1–90.9)- Mean score
- 87%
- 94OpenAIDossier
xhigh
Accuracy (best score across scorers)
GPT-5.4 mini, Accuracy (best score across scorers): 86.9% (scale 0.0 to 100.0), 95% confidence interval 82.2 to 91.686.9%(82.2–91.6)- Mean score
- 86.9%
- 95GoogleDossier
minimal
Accuracy (best score across scorers)
Gemini 3.5 Flash, Accuracy (best score across scorers): 86.4% (scale 0.0 to 100.0), 95% confidence interval 81.6 to 91.286.4%(81.6–91.2)- Mean score
- 86.4%
- 95GoogleDossier
low
Accuracy (best score across scorers)
Gemini 3.6 Flash, Accuracy (best score across scorers): 86.4% (scale 0.0 to 100.0), 95% confidence interval 81.6 to 91.286.4%(81.6–91.2)- Mean score
- 86.4%
- 95AnthropicDossier
max
Accuracy (best score across scorers)
Opus 4.7, Accuracy (best score across scorers): 86.4% (scale 0.0 to 100.0), 95% confidence interval 81.6 to 91.286.4%(81.6–91.2)- Mean score
- 86.4%
- 95AlibabaDossier
none
Accuracy (best score across scorers)
Qwen3.5 397B A17B, Accuracy (best score across scorers): 86.4% (scale 0.0 to 100.0), 95% confidence interval 81.6 to 91.286.4%(81.6–91.2)- Mean score
- 86.4%
- 99OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5, Accuracy (best score across scorers): 86.2% (scale 0.0 to 100.0), 95% confidence interval 82.1 to 90.286.2%(82.1–90.2)- Mean score
- 86.2%
- 100AnthropicDossier
32,000 budget
Accuracy (best score across scorers)
Opus 4.5, Accuracy (best score across scorers): 86% (scale 0.0 to 100.0), 95% confidence interval 82.0 to 90.186%(82.0–90.1)- Mean score
- 86%
- 101AnthropicDossier
max
Accuracy (best score across scorers)
Fable 5, Accuracy (best score across scorers): 85.9% (scale 0.0 to 100.0), 95% confidence interval 81.0 to 90.785.9%(81.0–90.7)- Mean score
- 85.9%
- 101GoogleDossier
minimal
Accuracy (best score across scorers)
Gemini 3.6 Flash, Accuracy (best score across scorers): 85.9% (scale 0.0 to 100.0), 95% confidence interval 81.0 to 90.785.9%(81.0–90.7)- Mean score
- 85.9%
- 101AlibabaDossier
Accuracy (best score across scorers)
Qwen3.5 397B A17B, Accuracy (best score across scorers): 85.9% (scale 0.0 to 100.0), 95% confidence interval 81.0 to 90.785.9%(81.0–90.7)- Mean score
- 85.9%
- 101AlibabaDossier
Accuracy (best score across scorers)
Qwen3.6 27B, Accuracy (best score across scorers): 85.9% (scale 0.0 to 100.0), 95% confidence interval 81.0 to 90.785.9%(81.0–90.7)- Mean score
- 85.9%
- 105AnthropicDossier
16,000 budget
Accuracy (best score across scorers)
Opus 4.5, Accuracy (best score across scorers): 85.5% (scale 0.0 to 100.0), 95% confidence interval 81.3 to 89.785.5%(81.3–89.7)- Mean score
- 85.5%
- 106OpenAIDossier
medium
Accuracy (best score across scorers)
GPT-5, Accuracy (best score across scorers): 85.4% (scale 0.0 to 100.0), 95% confidence interval 81.2 to 89.585.4%(81.2–89.5)- Mean score
- 85.4%
- 106NVIDIADossier
Accuracy (best score across scorers)
Nemotron 3 Ultra 550B A55B, Accuracy (best score across scorers): 85.4% (scale 0.0 to 100.0), 95% confidence interval 80.4 to 90.385.4%(80.4–90.3)- Mean score
- 85.4%
- 106AnthropicDossier
none
Accuracy (best score across scorers)
Opus 4.8, Accuracy (best score across scorers): 85.4% (scale 0.0 to 100.0), 95% confidence interval 80.4 to 90.385.4%(80.4–90.3)- Mean score
- 85.4%
- 109GoogleDossier
Accuracy (best score across scorers)
Gemini 2.5 Pro, Accuracy (best score across scorers): 85.3% (scale 0.0 to 100.0), 95% confidence interval 81.1 to 89.485.3%(81.1–89.4)- Mean score
- 85.3%
- 110OpenAIDossier
medium
Accuracy (best score across scorers)
GPT-5.1, Accuracy (best score across scorers): 85% (scale 0.0 to 100.0), 95% confidence interval 80.9 to 89.285%(80.9–89.2)- Mean score
- 85%
- 111GoogleDossier
Accuracy (best score across scorers)
Gemini 2.5 Pro Preview (06-05), Accuracy (best score across scorers): 84.8% (scale 0.0 to 100.0), 95% confidence interval 79.8 to 89.984.8%(79.8–89.9)- Mean score
- 84.8%
- 111OpenAIDossier
low
Accuracy (best score across scorers)
GPT-5.4, Accuracy (best score across scorers): 84.8% (scale 0.0 to 100.0), 95% confidence interval 79.8 to 89.984.8%(79.8–89.9)- Mean score
- 84.8%
- 111Moonshot AIDossier
low
Accuracy (best score across scorers)
Kimi K3, Accuracy (best score across scorers): 84.8% (scale 0.0 to 100.0), 95% confidence interval 79.8 to 89.984.8%(79.8–89.9)- Mean score
- 84.8%
- 111AlibabaDossier
Accuracy (best score across scorers)
Qwen3.5 Plus, Accuracy (best score across scorers): 84.8% (scale 0.0 to 100.0), 95% confidence interval 79.8 to 89.984.8%(79.8–89.9)- Mean score
- 84.8%
- 111AlibabaDossier
none
Accuracy (best score across scorers)
Qwen3.6 27B, Accuracy (best score across scorers): 84.8% (scale 0.0 to 100.0), 95% confidence interval 79.8 to 89.984.8%(79.8–89.9)- Mean score
- 84.8%
- 111AlibabaDossier
none
Accuracy (best score across scorers)
Qwen3.6 35B A3B, Accuracy (best score across scorers): 84.8% (scale 0.0 to 100.0), 95% confidence interval 79.8 to 89.984.8%(79.8–89.9)- Mean score
- 84.8%
- 117Moonshot AIDossier
Accuracy (best score across scorers)
Kimi K2 Thinking Turbo, Accuracy (best score across scorers): 84.2% (scale 0.0 to 100.0), 95% confidence interval 80.1 to 88.384.2%(80.1–88.3)- Mean score
- 84.2%
- 118GoogleDossier
Accuracy (best score across scorers)
Gemini 2.5 Pro Experimental (03-25), Accuracy (best score across scorers): 83.8% (scale 0.0 to 100.0), 95% confidence interval 78.7 to 89.083.8%(78.7–89.0)- Mean score
- 83.8%
- 118AlibabaDossier
Accuracy (best score across scorers)
Qwen3.6 35B A3B, Accuracy (best score across scorers): 83.8% (scale 0.0 to 100.0), 95% confidence interval 78.7 to 89.083.8%(78.7–89.0)- Mean score
- 83.8%
- 120OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5.4 mini, Accuracy (best score across scorers): 83.6% (scale 0.0 to 100.0), 95% confidence interval 79.2 to 88.083.6%(79.2–88.0)- Mean score
- 83.6%
- 121AlibabaDossier
Accuracy (best score across scorers)
Qwen3.5 35B A3B, Accuracy (best score across scorers): 83.5% (scale 0.0 to 100.0), 95% confidence interval 79.2 to 87.783.5%(79.2–87.7)- Mean score
- 83.5%
- 122DeepSeekDossier
Accuracy (best score across scorers)
DeepSeek V3.2, Accuracy (best score across scorers): 83.4% (scale 0.0 to 100.0), 95% confidence interval 79.5 to 87.483.4%(79.5–87.4)- Mean score
- 83.4%
- 123AnthropicDossier
high
Accuracy (best score across scorers)
Fable 5, Accuracy (best score across scorers): 83.3% (scale 0.0 to 100.0), 95% confidence interval 78.1 to 88.583.3%(78.1–88.5)- Mean score
- 83.3%
- 123GoogleDossier
high
Accuracy (best score across scorers)
Gemini 3.5 Flash-Lite, Accuracy (best score across scorers): 83.3% (scale 0.0 to 100.0), 95% confidence interval 78.1 to 88.583.3%(78.1–88.5)- Mean score
- 83.3%
- 123Z.aiDossier
Accuracy (best score across scorers)
GLM-4.7, Accuracy (best score across scorers): 83.3% (scale 0.0 to 100.0), 95% confidence interval 78.7 to 88.083.3%(78.7–88.0)- Mean score
- 83.3%
- 123AlibabaDossier
Accuracy (best score across scorers)
Qwen3.6 Flash, Accuracy (best score across scorers): 83.3% (scale 0.0 to 100.0), 95% confidence interval 78.1 to 88.583.3%(78.1–88.5)- Mean score
- 83.3%
- 123AnthropicDossier
high
Accuracy (best score across scorers)
Sonnet 4.6, Accuracy (best score across scorers): 83.3% (scale 0.0 to 100.0), 95% confidence interval 78.1 to 88.583.3%(78.1–88.5)- Mean score
- 83.3%
- 123AnthropicDossier
medium
Accuracy (best score across scorers)
Sonnet 4.6, Accuracy (best score across scorers): 83.3% (scale 0.0 to 100.0), 95% confidence interval 78.1 to 88.583.3%(78.1–88.5)- Mean score
- 83.3%
- 129GoogleDossier
Accuracy (best score across scorers)
Gemini 3 Flash Preview, Accuracy (best score across scorers): 83.2% (scale 0.0 to 100.0), 95% confidence interval 79.5 to 86.983.2%(79.5–86.9)- Mean score
- 83.2%
- 130OpenAIDossier
none
Accuracy (best score across scorers)
GPT-5.6 Sol, Accuracy (best score across scorers): 82.8% (scale 0.0 to 100.0), 95% confidence interval 77.6 to 88.182.8%(77.6–88.1)- Mean score
- 82.8%
- 131OpenAIDossier
low
Accuracy (best score across scorers)
GPT-5.2, Accuracy (best score across scorers): 82.7% (scale 0.0 to 100.0), 95% confidence interval 78.3 to 87.182.7%(78.3–87.1)- Mean score
- 82.7%
- 132OpenAIDossier
Accuracy (best score across scorers)
GPT-5.5 Instant, Accuracy (best score across scorers): 82.5% (scale 0.0 to 100.0), 95% confidence interval 78.1 to 86.982.5%(78.1–86.9)- Mean score
- 82.5%
- 133OpenAIDossier
low
Accuracy (best score across scorers)
GPT-5.6 Luna, Accuracy (best score across scorers): 82.3% (scale 0.0 to 100.0), 95% confidence interval 77.0 to 87.782.3%(77.0–87.7)- Mean score
- 82.3%
- 133AlibabaDossier
Accuracy (best score across scorers)
Qwen3.5 Flash, Accuracy (best score across scorers): 82.3% (scale 0.0 to 100.0), 95% confidence interval 77.0 to 87.782.3%(77.0–87.7)- Mean score
- 82.3%
- 133AlibabaDossier
Accuracy (best score across scorers)
Qwen3.7 Flash, Accuracy (best score across scorers): 82.3% (scale 0.0 to 100.0), 95% confidence interval 77.0 to 87.782.3%(77.0–87.7)- Mean score
- 82.3%
- 133AnthropicDossier
59,000 budget
Accuracy (best score across scorers)
Sonnet 4.5, Accuracy (best score across scorers): 82.3% (scale 0.0 to 100.0), 95% confidence interval 77.0 to 87.782.3%(77.0–87.7)- Mean score
- 82.3%
- 137GoogleDossier
high
Accuracy (best score across scorers)
Gemini 3.1 Flash-Lite Preview, Accuracy (best score across scorers): 81.8% (scale 0.0 to 100.0), 95% confidence interval 76.4 to 87.281.8%(76.4–87.2)- Mean score
- 81.8%
- 137OpenAIDossier
high
Accuracy (best score across scorers)
o3, Accuracy (best score across scorers): 81.8% (scale 0.0 to 100.0), 95% confidence interval 77.6 to 86.081.8%(77.6–86.0)- Mean score
- 81.8%
- 137AlibabaDossier
none
Accuracy (best score across scorers)
Qwen3.7 Plus, Accuracy (best score across scorers): 81.8% (scale 0.0 to 100.0), 95% confidence interval 76.4 to 87.281.8%(76.4–87.2)- Mean score
- 81.8%
- 140AnthropicDossier
32,000 budget
Accuracy (best score across scorers)
Sonnet 4.5, Accuracy (best score across scorers): 81.7% (scale 0.0 to 100.0), 95% confidence interval 76.3 to 87.181.7%(76.3–87.1)- Mean score
- 81.7%
- 141MiniMaxDossier
none
Accuracy (best score across scorers)
MiniMax M3, Accuracy (best score across scorers): 81.3% (scale 0.0 to 100.0), 95% confidence interval 75.9 to 86.881.3%(75.9–86.8)- Mean score
- 81.3%
- 142AlibabaDossier
none
Accuracy (best score across scorers)
Qwen3.5 35B A3B, Accuracy (best score across scorers): 81.2% (scale 0.0 to 100.0), 95% confidence interval 76.6 to 85.781.2%(76.6–85.7)- Mean score
- 81.2%
- 143OpenAIDossier
medium
Accuracy (best score across scorers)
o3, Accuracy (best score across scorers): 80.8% (scale 0.0 to 100.0), 95% confidence interval 75.3 to 86.380.8%(75.3–86.3)- Mean score
- 80.8%
- 143AlibabaDossier
none
Accuracy (best score across scorers)
Qwen3.7 Flash, Accuracy (best score across scorers): 80.8% (scale 0.0 to 100.0), 95% confidence interval 75.3 to 86.380.8%(75.3–86.3)- Mean score
- 80.8%
- 145AnthropicDossier
Accuracy (best score across scorers)
Opus 4.5, Accuracy (best score across scorers): 80.7% (scale 0.0 to 100.0), 95% confidence interval 76.0 to 85.480.7%(76.0–85.4)- Mean score
- 80.7%
- 146AnthropicDossier
max
Accuracy (best score across scorers)
Sonnet 5, Accuracy (best score across scorers): 80.3% (scale 0.0 to 100.0), 95% confidence interval 74.7 to 85.980.3%(74.7–85.9)- Mean score
- 80.3%
- 147AlibabaDossier
Accuracy (best score across scorers)
Qwen3 235B A22B Thinking (2507), Accuracy (best score across scorers): 80.1% (scale 0.0 to 100.0), 95% confidence interval 75.0 to 85.180.1%(75.0–85.1)- Mean score
- 80.1%
- 148OpenAIDossier
low
Accuracy (best score across scorers)
o3, Accuracy (best score across scorers): 79.8% (scale 0.0 to 100.0), 95% confidence interval 74.2 to 85.479.8%(74.2–85.4)- Mean score
- 79.8%
- 149AnthropicDossier
64,000 budget
Accuracy (best score across scorers)
Claude 3.7 Sonnet, Accuracy (best score across scorers): 79.7% (scale 0.0 to 100.0), 95% confidence interval 74.5 to 84.979.7%(74.5–84.9)- Mean score
- 78.5%
- 150OpenAIDossier
high
Accuracy (best score across scorers)
o4-mini, Accuracy (best score across scorers): 79.6% (scale 0.0 to 100.0), 95% confidence interval 74.9 to 84.379.6%(74.9–84.3)- Mean score
- 79.6%
- 151AnthropicDossier
59,000 budget
Accuracy (best score across scorers)
Claude Sonnet 4, Accuracy (best score across scorers): 79.2% (scale 0.0 to 100.0), 95% confidence interval 73.9 to 84.479.2%(73.9–84.4)- Mean score
- 77.8%
- 152AlibabaDossier
Accuracy (best score across scorers)
Qwen3.5 9B, Accuracy (best score across scorers): 79% (scale 0.0 to 100.0), 95% confidence interval 74.3 to 83.679%(74.3–83.6)- Mean score
- 79%
- 153AlibabaDossier
none
Accuracy (best score across scorers)
Qwen3.5 9B, Accuracy (best score across scorers): 78.9% (scale 0.0 to 100.0), 95% confidence interval 74.2 to 83.678.9%(74.2–83.6)- Mean score
- 78.9%
- 154AnthropicDossier
low
Accuracy (best score across scorers)
Fable 5, Accuracy (best score across scorers): 78.8% (scale 0.0 to 100.0), 95% confidence interval 73.1 to 84.578.8%(73.1–84.5)- Mean score
- 78.8%
- 154AnthropicDossier
16,000 budget
Accuracy (best score across scorers)
Sonnet 4.5, Accuracy (best score across scorers): 78.8% (scale 0.0 to 100.0), 95% confidence interval 73.1 to 84.578.8%(73.1–84.5)- Mean score
- 78.8%
- 154AnthropicDossier
max
Accuracy (best score across scorers)
Sonnet 4.6, Accuracy (best score across scorers): 78.8% (scale 0.0 to 100.0), 95% confidence interval 73.1 to 84.578.8%(73.1–84.5)- Mean score
- 78.8%
- 157OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5.4 nano, Accuracy (best score across scorers): 78.5% (scale 0.0 to 100.0), 95% confidence interval 73.7 to 83.378.5%(73.7–83.3)- Mean score
- 78.5%
- 158AnthropicDossier
32,000 budget
Accuracy (best score across scorers)
Claude Sonnet 4, Accuracy (best score across scorers): 78.3% (scale 0.0 to 100.0), 95% confidence interval 72.5 to 84.078.3%(72.5–84.0)- Mean score
- 78.3%
- 159OpenAIDossier
medium
Accuracy (best score across scorers)
o4-mini, Accuracy (best score across scorers): 77.8% (scale 0.0 to 100.0), 95% confidence interval 72.0 to 83.677.8%(72.0–83.6)- Mean score
- 77.8%
- 160ByteDanceDossier
Accuracy (best score across scorers)
Seed 1.6, Accuracy (best score across scorers): 77.6% (scale 0.0 to 100.0), 95% confidence interval 72.6 to 82.577.6%(72.6–82.5)- Mean score
- 77.6%
- 161OpenAIDossier
none
Accuracy (best score across scorers)
GPT-5.5, Accuracy (best score across scorers): 77.3% (scale 0.0 to 100.0), 95% confidence interval 71.4 to 83.177.3%(71.4–83.1)- Mean score
- 77.3%
- 161OpenAIDossier
none
Accuracy (best score across scorers)
GPT-5.6 Terra, Accuracy (best score across scorers): 77.3% (scale 0.0 to 100.0), 95% confidence interval 71.4 to 83.177.3%(71.4–83.1)- Mean score
- 77.3%
- 161AnthropicDossier
16,000 budget
Accuracy (best score across scorers)
Opus 4.1, Accuracy (best score across scorers): 77.3% (scale 0.0 to 100.0), 95% confidence interval 71.4 to 83.177.3%(71.4–83.1)- Mean score
- 77.3%
- 164OpenAIDossier
high
Accuracy (best score across scorers)
o3-mini, Accuracy (best score across scorers): 77% (scale 0.0 to 100.0), 95% confidence interval 72.0 to 82.177%(72.0–82.1)- Mean score
- 77%
- 165AnthropicDossier
32,000 budget
Accuracy (best score across scorers)
Claude 3.7 Sonnet, Accuracy (best score across scorers): 76.8% (scale 0.0 to 100.0), 95% confidence interval 70.9 to 82.776.8%(70.9–82.7)- Mean score
- 76.8%
- 165AnthropicDossier
16,000 budget
Accuracy (best score across scorers)
Claude 3.7 Sonnet, Accuracy (best score across scorers): 76.8% (scale 0.0 to 100.0), 95% confidence interval 70.9 to 82.776.8%(70.9–82.7)- Mean score
- 76.8%
- 165OpenAIDossier
high
Accuracy (best score across scorers)
o1, Accuracy (best score across scorers): 76.8% (scale 0.0 to 100.0), 95% confidence interval 70.9 to 82.776.8%(70.9–82.7)- Mean score
- 76.8%
- 165AnthropicDossier
27,000 budget
Accuracy (best score across scorers)
Opus 4.1, Accuracy (best score across scorers): 76.8% (scale 0.0 to 100.0), 95% confidence interval 70.9 to 82.776.8%(70.9–82.7)- Mean score
- 76.8%
- 169DeepSeekDossier
Accuracy (best score across scorers)
DeepSeek R1 0528, Accuracy (best score across scorers): 76.3% (scale 0.0 to 100.0), 95% confidence interval 71.6 to 81.076.3%(71.6–81.0)- Mean score
- 76.3%
- 170xAIDossier
low
Accuracy (best score across scorers)
Grok 3 Mini Beta, Accuracy (best score across scorers): 76.3% (scale 0.0 to 100.0), 95% confidence interval 70.3 to 82.276.3%(70.3–82.2)- Mean score
- 76.3%
- 170AnthropicDossier
16,000 budget
Accuracy (best score across scorers)
Opus 4, Accuracy (best score across scorers): 76.3% (scale 0.0 to 100.0), 95% confidence interval 70.3 to 82.276.3%(70.3–82.2)- Mean score
- 76.3%
- 172AnthropicDossier
16,000 budget
Accuracy (best score across scorers)
Claude Sonnet 4, Accuracy (best score across scorers): 75.8% (scale 0.0 to 100.0), 95% confidence interval 69.8 to 81.775.8%(69.8–81.7)- Mean score
- 75.8%
- 172GoogleDossier
low
Accuracy (best score across scorers)
Gemini 3.5 Flash-Lite, Accuracy (best score across scorers): 75.8% (scale 0.0 to 100.0), 95% confidence interval 69.8 to 81.775.8%(69.8–81.7)- Mean score
- 75.8%
- 172GoogleDossier
minimal
Accuracy (best score across scorers)
Gemma 4 31B, Accuracy (best score across scorers): 75.8% (scale 0.0 to 100.0), 95% confidence interval 69.8 to 81.775.8%(69.8–81.7)- Mean score
- 75.8%
- 172OpenAIDossier
high
Accuracy (best score across scorers)
gpt-oss-120b, Accuracy (best score across scorers): 75.8% (scale 0.0 to 100.0), 95% confidence interval 70.4 to 81.175.8%(70.4–81.1)- Mean score
- 75.8%
- 172xAIDossier
Accuracy (best score across scorers)
Grok 3 Beta, Accuracy (best score across scorers): 75.8% (scale 0.0 to 100.0), 95% confidence interval 70.5 to 81.075.8%(70.5–81.0)- Mean score
- 67.6%
- 172OpenAIDossier
medium
Accuracy (best score across scorers)
o1, Accuracy (best score across scorers): 75.8% (scale 0.0 to 100.0), 95% confidence interval 69.8 to 81.775.8%(69.8–81.7)- Mean score
- 75.8%
- 178xAIDossier
high
Accuracy (best score across scorers)
Grok 3 Mini Beta, Accuracy (best score across scorers): 75.5% (scale 0.0 to 100.0), 95% confidence interval 70.1 to 80.975.5%(70.1–80.9)- Mean score
- 74.6%
- 179OpenAIDossier
low
Accuracy (best score across scorers)
o4-mini, Accuracy (best score across scorers): 75.3% (scale 0.0 to 100.0), 95% confidence interval 69.2 to 81.375.3%(69.2–81.3)- Mean score
- 75.3%
- 180OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5 mini, Accuracy (best score across scorers): 75% (scale 0.0 to 100.0), 95% confidence interval 70.5 to 79.575%(70.5–79.5)- Mean score
- 75%
- 181OpenAIDossier
none
Accuracy (best score across scorers)
GPT-5.4, Accuracy (best score across scorers): 74.7% (scale 0.0 to 100.0), 95% confidence interval 68.7 to 80.874.7%(68.7–80.8)- Mean score
- 74.7%
- 182OpenAIDossier
medium
Accuracy (best score across scorers)
o3-mini, Accuracy (best score across scorers): 74.3% (scale 0.0 to 100.0), 95% confidence interval 69.3 to 79.374.3%(69.3–79.3)- Mean score
- 74.3%
- 183GoogleDossier
low
Accuracy (best score across scorers)
Gemini 3.1 Flash-Lite Preview, Accuracy (best score across scorers): 74.2% (scale 0.0 to 100.0), 95% confidence interval 68.1 to 80.374.2%(68.1–80.3)- Mean score
- 74.2%
- 183GoogleDossier
minimal
Accuracy (best score across scorers)
Gemini 3.5 Flash-Lite, Accuracy (best score across scorers): 74.2% (scale 0.0 to 100.0), 95% confidence interval 68.1 to 80.374.2%(68.1–80.3)- Mean score
- 74.2%
- 183OpenAIDossier
low
Accuracy (best score across scorers)
o1, Accuracy (best score across scorers): 74.2% (scale 0.0 to 100.0), 95% confidence interval 68.1 to 80.374.2%(68.1–80.3)- Mean score
- 74.2%
- 186GoogleDossier
minimal
Accuracy (best score across scorers)
Gemini 3.1 Flash-Lite Preview, Accuracy (best score across scorers): 73.7% (scale 0.0 to 100.0), 95% confidence interval 67.6 to 79.973.7%(67.6–79.9)- Mean score
- 73.7%
- 186AnthropicDossier
Accuracy (best score across scorers)
Sonnet 4.5, Accuracy (best score across scorers): 73.7% (scale 0.0 to 100.0), 95% confidence interval 67.6 to 79.973.7%(67.6–79.9)- Mean score
- 73.7%
- 188DeepSeekDossier
none
Accuracy (best score across scorers)
DeepSeek V4-Pro, Accuracy (best score across scorers): 73.2% (scale 0.0 to 100.0), 95% confidence interval 67.0 to 79.473.2%(67.0–79.4)- Mean score
- 73.2%
- 188GoogleDossier
minimal
Accuracy (best score across scorers)
Gemma 4 26B A4B, Accuracy (best score across scorers): 73.2% (scale 0.0 to 100.0), 95% confidence interval 67.0 to 79.473.2%(67.0–79.4)- Mean score
- 73.2%
- 188OpenAIDossier
none
Accuracy (best score across scorers)
GPT-5.2, Accuracy (best score across scorers): 73.2% (scale 0.0 to 100.0), 95% confidence interval 67.0 to 79.473.2%(67.0–79.4)- Mean score
- 73.2%
- 188AnthropicDossier
Accuracy (best score across scorers)
Opus 4.1, Accuracy (best score across scorers): 73.2% (scale 0.0 to 100.0), 95% confidence interval 67.0 to 79.473.2%(67.0–79.4)- Mean score
- 73.2%
- 192AlibabaDossier
Accuracy (best score across scorers)
Qwen3 Max, Accuracy (best score across scorers): 72.6% (scale 0.0 to 100.0), 95% confidence interval 67.2 to 78.072.6%(67.2–78.0)- Mean score
- 72.6%
- 193OpenAIDossier
low
Accuracy (best score across scorers)
GPT-5.4 nano, Accuracy (best score across scorers): 72.2% (scale 0.0 to 100.0), 95% confidence interval 66.0 to 78.572.2%(66.0–78.5)- Mean score
- 72.2%
- 194DeepSeekDossier
Accuracy (best score across scorers)
DeepSeek R1 (Jan '25), Accuracy (best score across scorers): 71.7% (scale 0.0 to 100.0), 95% confidence interval 65.7 to 77.771.7%(65.7–77.7)- Mean score
- 69.2%
- 194OpenAIDossier
minimal
Accuracy (best score across scorers)
GPT-5, Accuracy (best score across scorers): 71.7% (scale 0.0 to 100.0), 95% confidence interval 65.4 to 78.071.7%(65.4–78.0)- Mean score
- 71.7%
- 194OpenAIDossier
minimal
Accuracy (best score across scorers)
GPT-5 mini, Accuracy (best score across scorers): 71.7% (scale 0.0 to 100.0), 95% confidence interval 65.4 to 78.071.7%(65.4–78.0)- Mean score
- 71.7%
- 197OpenAIDossier
medium
Accuracy (best score across scorers)
GPT-5 mini, Accuracy (best score across scorers): 71.7% (scale 0.0 to 100.0), 95% confidence interval 67.1 to 76.271.7%(67.1–76.2)- Mean score
- 71.7%
- 198ByteDanceDossier
Accuracy (best score across scorers)
Seed-OSS 36B Instruct, Accuracy (best score across scorers): 71.5% (scale 0.0 to 100.0), 95% confidence interval 66.2 to 76.971.5%(66.2–76.9)- Mean score
- 71.5%
- 199DeepSeekDossier
Accuracy (best score across scorers)
DeepSeek V3.2, Accuracy (best score across scorers): 71.2% (scale 0.0 to 100.0), 95% confidence interval 64.9 to 77.571.2%(64.9–77.5)- Mean score
- 71.2%
- 199Z.aiDossier
none
Accuracy (best score across scorers)
GLM-5.2, Accuracy (best score across scorers): 71.2% (scale 0.0 to 100.0), 95% confidence interval 64.9 to 77.571.2%(64.9–77.5)- Mean score
- 71.2%
- 199AnthropicDossier
32,000 budget
Accuracy (best score across scorers)
Haiku 4.5, Accuracy (best score across scorers): 71.2% (scale 0.0 to 100.0), 95% confidence interval 64.9 to 77.571.2%(64.9–77.5)- Mean score
- 71.2%
- 202AlibabaDossier
Accuracy (best score across scorers)
Qwen3 235B A22B, Accuracy (best score across scorers): 70.7% (scale 0.0 to 100.0), 95% confidence interval 65.4 to 76.070.7%(65.4–76.0)- Mean score
- 70.7%
- 203AlibabaDossier
Accuracy (best score across scorers)
Qwen3 30B A3B Thinking 2507, Accuracy (best score across scorers): 70.1% (scale 0.0 to 100.0), 95% confidence interval 64.5 to 75.670.1%(64.5–75.6)- Mean score
- 70.1%
- 204OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5 nano, Accuracy (best score across scorers): 69.4% (scale 0.0 to 100.0), 95% confidence interval 64.0 to 74.969.4%(64.0–74.9)- Mean score
- 69.4%
- 205AnthropicDossier
Accuracy (best score across scorers)
Opus 4, Accuracy (best score across scorers): 69.2% (scale 0.0 to 100.0), 95% confidence interval 62.7 to 75.669.2%(62.7–75.6)- Mean score
- 69.2%
- 206OpenAIDossier
Accuracy (best score across scorers)
GPT-4.5 Preview, Accuracy (best score across scorers): 68.7% (scale 0.0 to 100.0), 95% confidence interval 62.2 to 75.268.7%(62.2–75.2)- Mean score
- 68.7%
- 207OpenAIDossier
low
Accuracy (best score across scorers)
o3-mini, Accuracy (best score across scorers): 68.2% (scale 0.0 to 100.0), 95% confidence interval 61.7 to 74.768.2%(61.7–74.7)- Mean score
- 68.2%
- 208DeepSeekDossier
Accuracy (best score across scorers)
DeepSeek V3 0324, Accuracy (best score across scorers): 67.6% (scale 0.0 to 100.0), 95% confidence interval 62.4 to 72.967.6%(62.4–72.9)- Mean score
- 67.6%
- 209OpenAIDossier
medium
Accuracy (best score across scorers)
GPT-5 nano, Accuracy (best score across scorers): 67.4% (scale 0.0 to 100.0), 95% confidence interval 62.2 to 72.667.4%(62.2–72.6)- Mean score
- 67.4%
- 210MetaDossier
Accuracy (best score across scorers)
Llama 4 Maverick 17B 128E Instruct (FP8), Accuracy (best score across scorers): 67% (scale 0.0 to 100.0), 95% confidence interval 61.4 to 72.567%(61.4–72.5)- Mean score
- 67%
- 211OpenAIDossier
Accuracy (best score across scorers)
GPT-4.1, Accuracy (best score across scorers): 66.9% (scale 0.0 to 100.0), 95% confidence interval 61.4 to 72.466.9%(61.4–72.4)- Mean score
- 66.9%
- 212AnthropicDossier
Accuracy (best score across scorers)
Claude Sonnet 4, Accuracy (best score across scorers): 66.7% (scale 0.0 to 100.0), 95% confidence interval 60.1 to 73.266.7%(60.1–73.2)- Mean score
- 66.7%
- 212GoogleDossier
Accuracy (best score across scorers)
Gemini 2.5 Pro Preview (May '25), Accuracy (best score across scorers): 66.7% (scale 0.0 to 100.0), 95% confidence interval 60.1 to 73.266.7%(60.1–73.2)- Mean score
- 66.7%
- 212OpenAIDossier
none
Accuracy (best score across scorers)
GPT-5.1, Accuracy (best score across scorers): 66.7% (scale 0.0 to 100.0), 95% confidence interval 60.1 to 73.266.7%(60.1–73.2)- Mean score
- 66.7%
- 215AnthropicDossier
Accuracy (best score across scorers)
Claude 3.7 Sonnet, Accuracy (best score across scorers): 66% (scale 0.0 to 100.0), 95% confidence interval 60.8 to 71.366%(60.8–71.3)- Mean score
- 66%
- 216OpenAIDossier
Accuracy (best score across scorers)
GPT-4.1 mini, Accuracy (best score across scorers): 65.8% (scale 0.0 to 100.0), 95% confidence interval 60.5 to 71.165.8%(60.5–71.1)- Mean score
- 65.8%
- 217AlibabaDossier
Accuracy (best score across scorers)
Qwen3 32B, Accuracy (best score across scorers): 65.7% (scale 0.0 to 100.0), 95% confidence interval 60.3 to 71.265.7%(60.3–71.2)- Mean score
- 65.7%
- 218GoogleDossier
Accuracy (best score across scorers)
Gemini 2.0 Pro Experimental (02-05), Accuracy (best score across scorers): 65.7% (scale 0.0 to 100.0), 95% confidence interval 59.0 to 72.365.7%(59.0–72.3)- Mean score
- 65.7%
- 219AlibabaDossier
Accuracy (best score across scorers)
QwQ Plus, Accuracy (best score across scorers): 65.4% (scale 0.0 to 100.0), 95% confidence interval 59.8 to 71.065.4%(59.8–71.0)- Mean score
- 65.4%
- 220AlibabaDossier
Accuracy (best score across scorers)
QwQ 32B, Accuracy (best score across scorers): 65.3% (scale 0.0 to 100.0), 95% confidence interval 59.8 to 70.965.3%(59.8–70.9)- Mean score
- 65.3%
- 221DeepSeekDossier
Accuracy (best score across scorers)
DeepSeek R1 Distill Qwen 32B, Accuracy (best score across scorers): 64.1% (scale 0.0 to 100.0), 95% confidence interval 58.7 to 69.564.1%(58.7–69.5)- Mean score
- 64.1%
- 221GoogleDossier
Accuracy (best score across scorers)
Gemini 2.0 Flash (001), Accuracy (best score across scorers): 64.1% (scale 0.0 to 100.0), 95% confidence interval 58.8 to 69.564.1%(58.8–69.5)- Mean score
- 64.1%
- 221OpenAIDossier
none
Accuracy (best score across scorers)
GPT-5.4 mini, Accuracy (best score across scorers): 64.1% (scale 0.0 to 100.0), 95% confidence interval 57.4 to 70.864.1%(57.4–70.8)- Mean score
- 64.1%
- 224AlibabaDossier
Accuracy (best score across scorers)
Qwen3 14B, Accuracy (best score across scorers): 63.8% (scale 0.0 to 100.0), 95% confidence interval 58.0 to 69.563.8%(58.0–69.5)- Mean score
- 63.8%
- 225OpenAIDossier
none
Accuracy (best score across scorers)
GPT-5.6 Luna, Accuracy (best score across scorers): 63.6% (scale 0.0 to 100.0), 95% confidence interval 56.9 to 70.463.6%(56.9–70.4)- Mean score
- 63.6%
- 226OpenAIDossier
high
Accuracy (best score across scorers)
o1-mini, Accuracy (best score across scorers): 62.4% (scale 0.0 to 100.0), 95% confidence interval 57.0 to 67.762.4%(57.0–67.7)- Mean score
- 62.4%
- 227AlibabaDossier
Accuracy (best score across scorers)
Qwen3 30B A3B, Accuracy (best score across scorers): 61.7% (scale 0.0 to 100.0), 95% confidence interval 55.9 to 67.561.7%(55.9–67.5)- Mean score
- 61.7%
- 228OpenAIDossier
medium
Accuracy (best score across scorers)
gpt-oss-20b, Accuracy (best score across scorers): 60.8% (scale 0.0 to 100.0), 95% confidence interval 55.9 to 65.760.8%(55.9–65.7)- Mean score
- 60.8%
- 229Z.aiDossier
Accuracy (best score across scorers)
GLM-4.7-Flash, Accuracy (best score across scorers): 60.5% (scale 0.0 to 100.0), 95% confidence interval 54.8 to 66.360.5%(54.8–66.3)- Mean score
- 60.5%
- 230AnthropicDossier
Accuracy (best score across scorers)
Haiku 4.5, Accuracy (best score across scorers): 60.5% (scale 0.0 to 100.0), 95% confidence interval 55.0 to 66.060.5%(55.0–66.0)- Mean score
- 60.5%
- 231MistralDossier
Accuracy (best score across scorers)
Mistral Medium 3, Accuracy (best score across scorers): 59.5% (scale 0.0 to 100.0), 95% confidence interval 54.0 to 65.159.5%(54.0–65.1)- Mean score
- 59.5%
- 232OpenAIDossier
medium
Accuracy (best score across scorers)
o1-mini, Accuracy (best score across scorers): 59.5% (scale 0.0 to 100.0), 95% confidence interval 54.0 to 65.059.5%(54.0–65.0)- Mean score
- 59.5%
- 233OpenAIDossier
low
Accuracy (best score across scorers)
GPT-5 nano, Accuracy (best score across scorers): 57.6% (scale 0.0 to 100.0), 95% confidence interval 50.7 to 64.557.6%(50.7–64.5)- Mean score
- 57.6%
- 234GoogleDossier
Accuracy (best score across scorers)
Gemini 1.5 Pro (002), Accuracy (best score across scorers): 57.2% (scale 0.0 to 100.0), 95% confidence interval 51.7 to 62.757.2%(51.7–62.7)- Mean score
- 57.2%
- 235GoogleDossier
Accuracy (best score across scorers)
Gemini 2.0 Flash Thinking Experimental (01-21), Accuracy (best score across scorers): 57.1% (scale 0.0 to 100.0), 95% confidence interval 50.2 to 64.057.1%(50.2–64.0)- Mean score
- 57.1%
- 236AlibabaDossier
Accuracy (best score across scorers)
Qwen3 8B, Accuracy (best score across scorers): 56.8% (scale 0.0 to 100.0), 95% confidence interval 51.0 to 62.556.8%(51.0–62.5)- Mean score
- 56.8%
- 237DeepSeekDossier
Accuracy (best score across scorers)
DeepSeek V3, Accuracy (best score across scorers): 56.5% (scale 0.0 to 100.0), 95% confidence interval 51.1 to 62.056.5%(51.1–62.0)- Mean score
- 56.5%
- 238AlibabaDossier
Accuracy (best score across scorers)
Qwen Max, Accuracy (best score across scorers): 56.1% (scale 0.0 to 100.0), 95% confidence interval 50.7 to 61.656.1%(50.7–61.6)- Mean score
- 56.1%
- 239MistralDossier
Accuracy (best score across scorers)
Magistral Small (2506), Accuracy (best score across scorers): 56.1% (scale 0.0 to 100.0), 95% confidence interval 49.1 to 63.056.1%(49.1–63.0)- Mean score
- 56.1%
- 239MicrosoftDossier
Accuracy (best score across scorers)
Phi-4, Accuracy (best score across scorers): 56.1% (scale 0.0 to 100.0), 95% confidence interval 51.0 to 61.156.1%(51.0–61.1)- Mean score
- 56.1%
- 241DeepSeekDossier
Accuracy (best score across scorers)
DeepSeek R1 Distill Llama 70B, Accuracy (best score across scorers): 55.7% (scale 0.0 to 100.0), 95% confidence interval 49.9 to 61.655.7%(49.9–61.6)- Mean score
- 55.7%
- 242AlibabaDossier
Accuracy (best score across scorers)
Qwen3 30B A3B 2507 Instruct, Accuracy (best score across scorers): 55.6% (scale 0.0 to 100.0), 95% confidence interval 50.0 to 61.355.6%(50.0–61.3)- Mean score
- 55.6%
- 243OpenAIDossier
none
Accuracy (best score across scorers)
GPT-5.4 nano, Accuracy (best score across scorers): 55.6% (scale 0.0 to 100.0), 95% confidence interval 48.6 to 62.555.6%(48.6–62.5)- Mean score
- 55.6%
- 244AnthropicDossier
Accuracy (best score across scorers)
Claude 3.5 Sonnet (Oct '24), Accuracy (best score across scorers): 55.3% (scale 0.0 to 100.0), 95% confidence interval 49.8 to 60.855.3%(49.8–60.8)- Mean score
- 55.3%
- 245AlibabaDossier
none
Accuracy (best score across scorers)
Qwen3 32B, Accuracy (best score across scorers): 54.1% (scale 0.0 to 100.0), 95% confidence interval 48.6 to 59.654.1%(48.6–59.6)- Mean score
- 54.1%
- 246AnthropicDossier
Accuracy (best score across scorers)
Claude 3.5 Sonnet (June '24), Accuracy (best score across scorers): 54% (scale 0.0 to 100.0), 95% confidence interval 48.6 to 59.454%(48.6–59.4)- Mean score
- 54%
- 247xAIDossier
Accuracy (best score across scorers)
Grok 2 (1212), Accuracy (best score across scorers): 53.8% (scale 0.0 to 100.0), 95% confidence interval 48.5 to 59.153.8%(48.5–59.1)- Mean score
- 53.8%
- 248AlibabaDossier
none
Accuracy (best score across scorers)
Qwen3 14B, Accuracy (best score across scorers): 53.4% (scale 0.0 to 100.0), 95% confidence interval 48.0 to 58.853.4%(48.0–58.8)- Mean score
- 53.4%
- 249OpenAIDossier
low
Accuracy (best score across scorers)
gpt-oss-20b, Accuracy (best score across scorers): 53.2% (scale 0.0 to 100.0), 95% confidence interval 48.3 to 58.153.2%(48.3–58.1)- Mean score
- 53.2%
- 250AlibabaDossier
Accuracy (best score across scorers)
Qwen3 4B, Accuracy (best score across scorers): 52.3% (scale 0.0 to 100.0), 95% confidence interval 46.4 to 58.252.3%(46.4–58.2)- Mean score
- 52.3%
- 251MetaDossier
Accuracy (best score across scorers)
Llama 4 Scout, Accuracy (best score across scorers): 51.8% (scale 0.0 to 100.0), 95% confidence interval 45.6 to 58.051.8%(45.6–58.0)- Mean score
- 51.8%
- 252MistralDossier
Accuracy (best score across scorers)
Mistral Large (2411), Accuracy (best score across scorers): 51.3% (scale 0.0 to 100.0), 95% confidence interval 46.1 to 56.651.3%(46.1–56.6)- Mean score
- 51.3%
- 253MetaDossier
Accuracy (best score across scorers)
Llama 3.1 405B Instruct, Accuracy (best score across scorers): 50.9% (scale 0.0 to 100.0), 95% confidence interval 45.8 to 56.050.9%(45.8–56.0)- Mean score
- 50.9%
- 254AlibabaDossier
none
Accuracy (best score across scorers)
Qwen3 30B A3B, Accuracy (best score across scorers): 50.4% (scale 0.0 to 100.0), 95% confidence interval 44.8 to 55.950.4%(44.8–55.9)- Mean score
- 50.4%
- 255OpenAIDossier
Accuracy (best score across scorers)
o1-preview, Accuracy (best score across scorers): 50.3% (scale 0.0 to 100.0), 95% confidence interval 44.8 to 55.950.3%(44.8–55.9)- Mean score
- 50.3%
- 256OpenAIDossier
Accuracy (best score across scorers)
GPT-4o (Aug '24), Accuracy (best score across scorers): 49.2% (scale 0.0 to 100.0), 95% confidence interval 44.1 to 54.349.2%(44.1–54.3)- Mean score
- 49.2%
- 257AlibabaDossier
Accuracy (best score across scorers)
Qwen2.5 72B Instruct, Accuracy (best score across scorers): 49.1% (scale 0.0 to 100.0), 95% confidence interval 43.9 to 54.449.1%(43.9–54.4)- Mean score
- 49.1%
- 258MistralDossier
Accuracy (best score across scorers)
Mistral Small 3.2, Accuracy (best score across scorers): 49.1% (scale 0.0 to 100.0), 95% confidence interval 43.6 to 54.549.1%(43.6–54.5)- Mean score
- 49.1%
- 259MistralDossier
Accuracy (best score across scorers)
Mistral Large 2 (Jul '24), Accuracy (best score across scorers): 49% (scale 0.0 to 100.0), 95% confidence interval 43.9 to 54.149%(43.9–54.1)- Mean score
- 49%
- 260OpenAIDossier
Accuracy (best score across scorers)
GPT-4.1 nano, Accuracy (best score across scorers): 48.9% (scale 0.0 to 100.0), 95% confidence interval 44.1 to 53.848.9%(44.1–53.8)- Mean score
- 48.9%
- 261OpenAIDossier
Accuracy (best score across scorers)
GPT-4o (May '24), Accuracy (best score across scorers): 48.9% (scale 0.0 to 100.0), 95% confidence interval 43.7 to 54.048.9%(43.7–54.0)- Mean score
- 48.9%
- 262OpenAIDossier
minimal
Accuracy (best score across scorers)
GPT-5 nano, Accuracy (best score across scorers): 48.5% (scale 0.0 to 100.0), 95% confidence interval 41.5 to 55.548.5%(41.5–55.5)- Mean score
- 48.5%
- 263AlibabaDossier
Accuracy (best score across scorers)
Qwen Plus, Accuracy (best score across scorers): 48.1% (scale 0.0 to 100.0), 95% confidence interval 42.8 to 53.448.1%(42.8–53.4)- Mean score
- 48.1%
- 264OpenAIDossier
Accuracy (best score across scorers)
GPT-4o (Nov '24), Accuracy (best score across scorers): 47.9% (scale 0.0 to 100.0), 95% confidence interval 42.7 to 53.047.9%(42.7–53.0)- Mean score
- 47.9%
- 265GoogleDossier
Accuracy (best score across scorers)
Gemma 3 27B, Accuracy (best score across scorers): 47.7% (scale 0.0 to 100.0), 95% confidence interval 42.4 to 53.047.7%(42.4–53.0)- Mean score
- 47.7%
- 266MistralDossier
Accuracy (best score across scorers)
Magistral Small 1.2, Accuracy (best score across scorers): 47.6% (scale 0.0 to 100.0), 95% confidence interval 42.5 to 52.747.6%(42.5–52.7)- Mean score
- 47.6%
- 267MistralDossier
Accuracy (best score across scorers)
Mistral Small 3.1, Accuracy (best score across scorers): 47.5% (scale 0.0 to 100.0), 95% confidence interval 42.1 to 52.947.5%(42.1–52.9)- Mean score
- 47.5%
- 267AlibabaDossier
none
Accuracy (best score across scorers)
Qwen3 8B, Accuracy (best score across scorers): 47.5% (scale 0.0 to 100.0), 95% confidence interval 42.3 to 52.647.5%(42.3–52.6)- Mean score
- 47.5%
- 269MetaDossier
Accuracy (best score across scorers)
Llama 3.3 Instruct 70B, Accuracy (best score across scorers): 47.4% (scale 0.0 to 100.0), 95% confidence interval 41.9 to 53.047.4%(41.9–53.0)- Mean score
- 47.4%
- 270GoogleDossier
Accuracy (best score across scorers)
Gemini 1.5 Flash (002), Accuracy (best score across scorers): 47.3% (scale 0.0 to 100.0), 95% confidence interval 41.9 to 52.847.3%(41.9–52.8)- Mean score
- 47.3%
- 271MistralDossier
Accuracy (best score across scorers)
Mistral Small 3, Accuracy (best score across scorers): 47.3% (scale 0.0 to 100.0), 95% confidence interval 41.8 to 52.847.3%(41.8–52.8)- Mean score
- 47.3%
- 272AnthropicDossier
Accuracy (best score across scorers)
Claude 3 Opus, Accuracy (best score across scorers): 47.2% (scale 0.0 to 100.0), 95% confidence interval 42.0 to 52.347.2%(42.0–52.3)- Mean score
- 47.2%
- 273OpenAIDossier
Accuracy (best score across scorers)
GPT-4 Turbo, Accuracy (best score across scorers): 46.6% (scale 0.0 to 100.0), 95% confidence interval 41.4 to 51.846.6%(41.4–51.8)- Mean score
- 46.6%
- 274Allen Institute for AIDossier
Accuracy (best score across scorers)
Llama 3.1 Tulu 3 70B DPO, Accuracy (best score across scorers): 46.3% (scale 0.0 to 100.0), 95% confidence interval 41.6 to 51.046.3%(41.6–51.0)- Mean score
- 46.3%
- 275AlibabaDossier
Accuracy (best score across scorers)
Qwen2.5 32B Instruct, Accuracy (best score across scorers): 46.1% (scale 0.0 to 100.0), 95% confidence interval 41.0 to 51.246.1%(41.0–51.2)- Mean score
- 46.1%
- 276OpenAIDossier
high
Accuracy (best score across scorers)
gpt-oss-20b, Accuracy (best score across scorers): 46% (scale 0.0 to 100.0), 95% confidence interval 39.0 to 52.946%(39.0–52.9)- Mean score
- 46%
- 277GoogleDossier
Accuracy (best score across scorers)
Gemini 1.5 Pro (001), Accuracy (best score across scorers): 45.9% (scale 0.0 to 100.0), 95% confidence interval 40.7 to 51.045.9%(40.7–51.0)- Mean score
- 45.9%
- 278AlibabaDossier
Accuracy (best score across scorers)
Qwen3 4B Instruct 2507, Accuracy (best score across scorers): 45.8% (scale 0.0 to 100.0), 95% confidence interval 40.2 to 51.445.8%(40.2–51.4)- Mean score
- 45.8%
- 279MistralDossier
Accuracy (best score across scorers)
Mistral Small 3, Accuracy (best score across scorers): 45.3% (scale 0.0 to 100.0), 95% confidence interval 40.4 to 50.245.3%(40.4–50.2)- Mean score
- 45.3%
- 280Z.aiDossier
none
Accuracy (best score across scorers)
GLM-4.7-Flash, Accuracy (best score across scorers): 45.1% (scale 0.0 to 100.0), 95% confidence interval 40.9 to 49.445.1%(40.9–49.4)- Mean score
- 45.1%
- 281DeepSeekDossier
Accuracy (best score across scorers)
DeepSeek R1 Distill Qwen 14B, Accuracy (best score across scorers): 44.7% (scale 0.0 to 100.0), 95% confidence interval 39.1 to 50.344.7%(39.1–50.3)- Mean score
- 44.7%
- 282MetaDossier
Accuracy (best score across scorers)
Llama 3.1 Instruct 70B, Accuracy (best score across scorers): 44.2% (scale 0.0 to 100.0), 95% confidence interval 39.4 to 49.044.2%(39.4–49.0)- Mean score
- 44.2%
- 283AlibabaDossier
none
Accuracy (best score across scorers)
Qwen3 4B, Accuracy (best score across scorers): 43.6% (scale 0.0 to 100.0), 95% confidence interval 38.3 to 48.843.6%(38.3–48.8)- Mean score
- 43.6%
- 284MicrosoftDossier
Accuracy (best score across scorers)
WizardLM-2 8x22B, Accuracy (best score across scorers): 43.4% (scale 0.0 to 100.0), 95% confidence interval 38.9 to 48.043.4%(38.9–48.0)- Mean score
- 43.4%
- 285OpenAIDossier
Accuracy (best score across scorers)
GPT-4 Turbo Preview (1106), Accuracy (best score across scorers): 42.4% (scale 0.0 to 100.0), 95% confidence interval 37.6 to 47.142.4%(37.6–47.1)- Mean score
- 42.4%
- 286OpenAIDossier
Accuracy (best score across scorers)
GPT-4 Turbo Preview (0125), Accuracy (best score across scorers): 42.3% (scale 0.0 to 100.0), 95% confidence interval 37.0 to 47.542.3%(37.0–47.5)- Mean score
- 42.3%
- 287MistralDossier
Accuracy (best score across scorers)
Mistral Small 3.1, Accuracy (best score across scorers): 41.9% (scale 0.0 to 100.0), 95% confidence interval 37.3 to 46.541.9%(37.3–46.5)- Mean score
- 41.9%
- 288AlibabaDossier
Accuracy (best score across scorers)
Qwen Turbo, Accuracy (best score across scorers): 41.8% (scale 0.0 to 100.0), 95% confidence interval 36.7 to 46.941.8%(36.7–46.9)- Mean score
- 41.8%
- 289MetaDossier
Accuracy (best score across scorers)
Llama 3.2 90B Vision Instruct, Accuracy (best score across scorers): 41% (scale 0.0 to 100.0), 95% confidence interval 36.5 to 45.541%(36.5–45.5)- Mean score
- 41%
- 290AlibabaDossier
Accuracy (best score across scorers)
Qwen2 72B Instruct, Accuracy (best score across scorers): 40.8% (scale 0.0 to 100.0), 95% confidence interval 35.7 to 45.840.8%(35.7–45.8)- Mean score
- 40.8%
- 291AnthropicDossier
Accuracy (best score across scorers)
Claude 3 Sonnet, Accuracy (best score across scorers): 40.6% (scale 0.0 to 100.0), 95% confidence interval 35.9 to 45.340.6%(35.9–45.3)- Mean score
- 40.6%
- 292MetaDossier
Accuracy (best score across scorers)
Llama 3 Instruct 70B, Accuracy (best score across scorers): 40.6% (scale 0.0 to 100.0), 95% confidence interval 35.4 to 45.740.6%(35.4–45.7)- Mean score
- 40.6%
- 293GoogleDossier
Accuracy (best score across scorers)
Gemini 1.5 Flash (001), Accuracy (best score across scorers): 40.4% (scale 0.0 to 100.0), 95% confidence interval 35.7 to 45.040.4%(35.7–45.0)- Mean score
- 40.4%
- 294GoogleDossier
Accuracy (best score across scorers)
Gemma 3 12B, Accuracy (best score across scorers): 39.5% (scale 0.0 to 100.0), 95% confidence interval 34.5 to 44.439.5%(34.5–44.4)- Mean score
- 39.5%
- 295MistralDossier
Accuracy (best score across scorers)
Mistral Large (Feb '24), Accuracy (best score across scorers): 38.8% (scale 0.0 to 100.0), 95% confidence interval 33.9 to 43.638.8%(33.9–43.6)- Mean score
- 38.8%
- 296AnthropicDossier
Accuracy (best score across scorers)
Claude 3.5 Haiku, Accuracy (best score across scorers): 38.1% (scale 0.0 to 100.0), 95% confidence interval 33.0 to 43.338.1%(33.0–43.3)- Mean score
- 38.1%
- 297AlibabaDossier
Accuracy (best score across scorers)
Qwen3 1.7B, Accuracy (best score across scorers): 38% (scale 0.0 to 100.0), 95% confidence interval 32.6 to 43.438%(32.6–43.4)- Mean score
- 38%
- 298OpenAIDossier
Accuracy (best score across scorers)
GPT-4o mini, Accuracy (best score across scorers): 37.7% (scale 0.0 to 100.0), 95% confidence interval 33.0 to 42.437.7%(33.0–42.4)- Mean score
- 37.7%
- 299Nous ResearchDossier
Accuracy (best score across scorers)
Hermes 2 Theta Llama 3 70B, Accuracy (best score across scorers): 37.5% (scale 0.0 to 100.0), 95% confidence interval 32.6 to 42.337.5%(32.6–42.3)- Mean score
- 37.5%
- 300GoogleDossier
Accuracy (best score across scorers)
Gemma 2 27B IT, Accuracy (best score across scorers): 36.5% (scale 0.0 to 100.0), 95% confidence interval 31.8 to 41.236.5%(31.8–41.2)- Mean score
- 36.5%
- 301AnthropicDossier
Accuracy (best score across scorers)
Claude 3 Haiku, Accuracy (best score across scorers): 36.3% (scale 0.0 to 100.0), 95% confidence interval 31.6 to 41.036.3%(31.6–41.0)- Mean score
- 36.3%
- 302OpenAIDossier
Accuracy (best score across scorers)
GPT-4 (0314), Accuracy (best score across scorers): 35.7% (scale 0.0 to 100.0), 95% confidence interval 31.1 to 40.435.7%(31.1–40.4)- Mean score
- 35.7%
- 303AlibabaDossier
Accuracy (best score across scorers)
Qwen2.5 7B Instruct, Accuracy (best score across scorers): 35.5% (scale 0.0 to 100.0), 95% confidence interval 31.0 to 40.035.5%(31.0–40.0)- Mean score
- 35.5%
- 304AnthropicDossier
Accuracy (best score across scorers)
Claude 2.0, Accuracy (best score across scorers): 34.7% (scale 0.0 to 100.0), 95% confidence interval 29.7 to 39.634.7%(29.7–39.6)- Mean score
- 34.7%
- 305MistralDossier
Accuracy (best score across scorers)
Mixtral 8x22B Instruct v0.1, Accuracy (best score across scorers): 34.1% (scale 0.0 to 100.0), 95% confidence interval 30.0 to 38.134.1%(30.0–38.1)- Mean score
- 34.1%
- 306GoogleDossier
Accuracy (best score across scorers)
Gemini 1.0 Pro (001), Accuracy (best score across scorers): 34% (scale 0.0 to 100.0), 95% confidence interval 30.0 to 37.934%(30.0–37.9)- Mean score
- 34%
- 307DeepSeekDossier
Accuracy (best score across scorers)
DeepSeek R1 Distill Qwen 1.5B, Accuracy (best score across scorers): 33.6% (scale 0.0 to 100.0), 95% confidence interval 29.5 to 37.633.6%(29.5–37.6)- Mean score
- 33.6%
- 308AnthropicDossier
Accuracy (best score across scorers)
Claude 2.1, Accuracy (best score across scorers): 33% (scale 0.0 to 100.0), 95% confidence interval 28.4 to 37.533%(28.4–37.5)- Mean score
- 33%
- 308GoogleDossier
Accuracy (best score across scorers)
Gemini 1.5 Flash-8B (001), Accuracy (best score across scorers): 33% (scale 0.0 to 100.0), 95% confidence interval 28.2 to 37.733%(28.2–37.7)- Mean score
- 33%
- 310DatabricksDossier
Accuracy (best score across scorers)
DBRX Instruct, Accuracy (best score across scorers): 32.9% (scale 0.0 to 100.0), 95% confidence interval 26.5 to 39.332.9%(26.5–39.3)- Mean score
- 32.9%
- 31101.AIDossier
Accuracy (best score across scorers)
Yi 1.5 34B Chat, Accuracy (best score across scorers): 32% (scale 0.0 to 100.0), 95% confidence interval 28.1 to 35.932%(28.1–35.9)- Mean score
- 32%
- 312AlibabaDossier
Accuracy (best score across scorers)
Qwen1.5 32B Chat, Accuracy (best score across scorers): 30.7% (scale 0.0 to 100.0), 95% confidence interval 26.8 to 34.730.7%(26.8–34.7)- Mean score
- 30.7%
- 313OpenAIDossier
Accuracy (best score across scorers)
GPT-4 (0613), Accuracy (best score across scorers): 30.7% (scale 0.0 to 100.0), 95% confidence interval 26.1 to 35.230.7%(26.1–35.2)- Mean score
- 30.7%
- 314AlibabaDossier
none
Accuracy (best score across scorers)
Qwen3 1.7B, Accuracy (best score across scorers): 30.6% (scale 0.0 to 100.0), 95% confidence interval 26.2 to 35.030.6%(26.2–35.0)- Mean score
- 30.6%
- 315MistralDossier
Accuracy (best score across scorers)
Mixtral 8x7B Instruct, Accuracy (best score across scorers): 30.6% (scale 0.0 to 100.0), 95% confidence interval 26.8 to 34.430.6%(26.8–34.4)- Mean score
- 30.6%
- 316MistralDossier
Accuracy (best score across scorers)
Mistral NeMo (2407), Accuracy (best score across scorers): 29.9% (scale 0.0 to 100.0), 95% confidence interval 25.8 to 34.029.9%(25.8–34.0)- Mean score
- 29.9%
- 317MistralDossier
Accuracy (best score across scorers)
Mixtral 8x7B Instruct, Accuracy (best score across scorers): 29.8% (scale 0.0 to 100.0), 95% confidence interval 26.1 to 33.629.8%(26.1–33.6)- Mean score
- 29.8%
- 318AlibabaDossier
Accuracy (best score across scorers)
Qwen1.5 72B Chat, Accuracy (best score across scorers): 28.8% (scale 0.0 to 100.0), 95% confidence interval 24.9 to 32.728.8%(24.9–32.7)- Mean score
- 28.8%
- 319IBMDossier
Accuracy (best score across scorers)
Granite 4.0 Micro, Accuracy (best score across scorers): 28.3% (scale 0.0 to 100.0), 95% confidence interval 24.7 to 31.928.3%(24.7–31.9)- Mean score
- 28.3%
- 320OpenAIDossier
Accuracy (best score across scorers)
GPT-3.5 Turbo (1106), Accuracy (best score across scorers): 28% (scale 0.0 to 100.0), 95% confidence interval 24.4 to 31.728%(24.4–31.7)- Mean score
- 28%
- 321MicrosoftDossier
Accuracy (best score across scorers)
Phi-3 Medium 128K Instruct, Accuracy (best score across scorers): 27.6% (scale 0.0 to 100.0), 95% confidence interval 24.4 to 30.727.6%(24.4–30.7)- Mean score
- 27.6%
- 322GoogleDossier
Accuracy (best score across scorers)
Gemma 2 9B IT, Accuracy (best score across scorers): 27.5% (scale 0.0 to 100.0), 95% confidence interval 22.9 to 32.027.5%(22.9–32.0)- Mean score
- 27.5%
- 323OpenAIDossier
Accuracy (best score across scorers)
GPT-3.5 Turbo (0125), Accuracy (best score across scorers): 27.2% (scale 0.0 to 100.0), 95% confidence interval 23.3 to 31.027.2%(23.3–31.0)- Mean score
- 27.2%
- 324MistralDossier
Accuracy (best score across scorers)
Ministral 8B (2410), Accuracy (best score across scorers): 27.1% (scale 0.0 to 100.0), 95% confidence interval 23.1 to 31.227.1%(23.1–31.2)- Mean score
- 27.1%
- 325MetaDossier
Accuracy (best score across scorers)
Llama 3.1 Instruct 8B, Accuracy (best score across scorers): 27% (scale 0.0 to 100.0), 95% confidence interval 23.3 to 30.627%(23.3–30.6)- Mean score
- 27%
- 326MetaDossier
Accuracy (best score across scorers)
Llama 2 70B Chat, Accuracy (best score across scorers): 26.3% (scale 0.0 to 100.0), 95% confidence interval 22.4 to 30.226.3%(22.4–30.2)- Mean score
- 26.3%
- 327MetaDossier
Accuracy (best score across scorers)
Llama 3 Instruct 8B, Accuracy (best score across scorers): 26.1% (scale 0.0 to 100.0), 95% confidence interval 22.7 to 29.426.1%(22.7–29.4)- Mean score
- 26.1%
- 328MistralDossier
Accuracy (best score across scorers)
Ministral 3B (2410), Accuracy (best score across scorers): 25.3% (scale 0.0 to 100.0), 95% confidence interval 21.5 to 29.025.3%(21.5–29.0)- Mean score
- 25.3%
- 329DeepSeekDossier
Accuracy (best score across scorers)
DeepSeek LLM 67B Chat, Accuracy (best score across scorers): 24.6% (scale 0.0 to 100.0), 95% confidence interval 22.0 to 27.324.6%(22.0–27.3)- Mean score
- 24.6%
- 330IBMDossier
Accuracy (best score across scorers)
Granite 4.0 1B, Accuracy (best score across scorers): 24% (scale 0.0 to 100.0), 95% confidence interval 20.8 to 27.224%(20.8–27.2)- Mean score
- 24%
- 331MetaDossier
Accuracy (best score across scorers)
Llama 3.2 1B Instruct, Accuracy (best score across scorers): 23.9% (scale 0.0 to 100.0), 95% confidence interval 20.7 to 27.123.9%(20.7–27.1)- Mean score
- 23.9%
- 332GoogleDossier
Accuracy (best score across scorers)
Gemma 3 4B, Accuracy (best score across scorers): 23.2% (scale 0.0 to 100.0), 95% confidence interval 19.6 to 26.923.2%(19.6–26.9)- Mean score
- 23.2%
- 333GoogleDossier
Accuracy (best score across scorers)
Gemma 3 1B, Accuracy (best score across scorers): 19.9% (scale 0.0 to 100.0), 95% confidence interval 17.0 to 22.919.9%(17.0–22.9)- Mean score
- 19.9%
- 334MistralDossier
Accuracy (best score across scorers)
Mistral 7B Instruct v0.3, Accuracy (best score across scorers): 15.2% (scale 0.0 to 100.0), 95% confidence interval 13.2 to 17.215.2%(13.2–17.2)- Mean score
- 15.2%
- 33501.AIDossier
Accuracy (best score across scorers)
Yi 34B Chat, Accuracy (best score across scorers): 14.7% (scale 0.0 to 100.0), 95% confidence interval 12.5 to 17.014.7%(12.5–17.0)- Mean score
- 14.7%
- 336MistralDossier
Accuracy (best score across scorers)
Mistral 7B Instruct, Accuracy (best score across scorers): 13.2% (scale 0.0 to 100.0), 95% confidence interval 11.3 to 15.213.2%(11.3–15.2)- Mean score
- 13.2%
- 337IBMDossier
Accuracy (best score across scorers)
Granite 4.0 350M, Accuracy (best score across scorers): 11.2% (scale 0.0 to 100.0), 95% confidence interval 9.5 to 13.011.2%(9.5–13.0)- Mean score
- 11.2%
- 338DeepSeekDossier
Accuracy (best score across scorers)
DeepSeek R1 0528 Qwen3 8B, Accuracy (best score across scorers): 9.3% (scale 0.0 to 100.0), 95% confidence interval 7.2 to 11.49.3%(7.2–11.4)- Mean score
- 9.3%
Unmatched upstream rows excluded from this board (2)
These upstream rows could not be mapped cleanly to the canonical model registry or reasoning effort enum.
| Raw name | Parsed model | Reason | Detail |
|---|---|---|---|
| Eurus-2-7B-PRIME | Eurus-2-7B-PRIME | no-alias | accuracy |
| Eurus-2-7B-PRIME | Eurus-2-7B-PRIME | no-alias | mean-score |