HLE
Humanity's Last Exam results that Epoch AI republishes from the original runner.
No filters. Showing 56 of 56 rows.
1 OpenAIDossier | No effort set | No harness | GPT-6 Astra, Accuracy: 54.8% (scale 0.0 to 100.0), 95% confidence interval 51.0 to 58.654.8%(51.0–58.6) | 39% | Runs unstated | Cost kind unknown | Date not stated |
2[2–6] AnthropicDossier | xhigh | No harness | Fable 5.1, Accuracy: 46.5% (scale 0.0 to 100.0), 95% confidence interval 42.6 to 50.446.5%(42.6–50.4) | 20% | Runs unstated | Cost kind unknown | Date not stated |
3[2–6] GoogleDossier | high | No harness | Gemini 3.1 Pro Preview, Accuracy: 46.4% (scale 0.0 to 100.0), 95% confidence interval 42.6 to 50.346.4%(42.6–50.3) | 51% | Runs unstated | Cost kind unknown | Date not stated |
4[2–6] AnthropicDossier | xhigh | No harness | Sonnet 5.5, Accuracy: 46.1% (scale 0.0 to 100.0), 95% confidence interval 42.3 to 49.946.1%(42.3–49.9) | Not tested | Runs unstated | Cost kind unknown | Date not stated |
5[2–7] GoogleDossier | No effort set | No harness | Gemini 3.8 Flash, Accuracy: 44.5% (scale 0.0 to 100.0), 95% confidence interval 40.7 to 48.444.5%(40.7–48.4) | 51% | Runs unstated | Cost kind unknown | Date not stated |
6[2–7] OpenAIDossier | No effort set | No harness | GPT-5.4 Pro, Accuracy: 44.3% (scale 0.0 to 100.0), 95% confidence interval 40.5 to 48.144.3%(40.5–48.1) | 38% | Runs unstated | Cost kind unknown | Date not stated |
7[6–9] MetaDossier | No effort set | No harness | Muse Spark, Accuracy: 40.6% (scale 0.0 to 100.0), 95% confidence interval 36.8 to 44.340.6%(36.8–44.3) | 50% | Runs unstated | Cost kind unknown | Date not stated |
8[7–11] GoogleDossier | No effort set | No harness | Gemini 3 Pro Preview, Accuracy: 37.5% (scale 0.0 to 100.0), 95% confidence interval 33.8 to 41.237.5%(33.8–41.2) | 57% | Runs unstated | Cost kind unknown | Date not stated |
9[7–11] OpenAIDossier | xhigh | No harness | GPT-5.4, Accuracy: 36.2% (scale 0.0 to 100.0), 95% confidence interval 32.6 to 39.936.2%(32.6–39.9) | 42% | Runs unstated | Cost kind unknown | Date not stated |
10[8–12] AnthropicDossier | No effort set | No harness | Opus 4.7, Accuracy: 36.2% (scale 0.0 to 100.0), 95% confidence interval 32.5 to 39.936.2%(32.5–39.9) | 47% | Runs unstated | Cost kind unknown | Date not stated |
11[8–13] AnthropicDossier | max | No harness | Opus 4.6, Accuracy: 34.4% (scale 0.0 to 100.0), 95% confidence interval 30.8 to 38.134.4%(30.8–38.1) | 46% | Runs unstated | Cost kind unknown | Date not stated |
12[10–14] OpenAIDossier | No effort set | No harness | GPT-5 Pro, Accuracy: 31.6% (scale 0.0 to 100.0), 95% confidence interval 28.1 to 35.231.6%(28.1–35.2) | 49% | Runs unstated | Cost kind unknown | Date not stated |
13[11–15] AnthropicDossier | No effort set | No harness | Haiku 5.5, Accuracy: 30.1% (scale 0.0 to 100.0), 95% confidence interval 26.5 to 33.730.1%(26.5–33.7) | Not tested | Runs unstated | Cost kind unknown | Date not stated |
14[12–18] OpenAIDossier | No effort set | No harness | GPT-5.2, Accuracy: 27.8% (scale 0.0 to 100.0), 95% confidence interval 24.4 to 31.227.8%(24.4–31.2) | 45% | Runs unstated | Cost kind unknown | Date not stated |
15[14–20] OpenAIDossier | high | No harness | GPT-5, Accuracy: 25.3% (scale 0.0 to 100.0), 95% confidence interval 22.0 to 28.725.3%(22.0–28.7) | 50% | Runs unstated | Cost kind unknown | Date not stated |
15[14–20] OpenAIDossier | No effort set | No harness | GPT-5, Accuracy: 25.3% (scale 0.0 to 100.0), 95% confidence interval 22.0 to 28.725.3%(22.0–28.7) | 50% | Runs unstated | Cost kind unknown | Date not stated |
17[14–20] AnthropicDossier | No effort set | No harness | Opus 4.5, Accuracy: 25.2% (scale 0.0 to 100.0), 95% confidence interval 21.9 to 28.525.2%(21.9–28.5) | 55% | Runs unstated | Cost kind unknown | Date not stated |
18[14–21] Moonshot AIDossier | No effort set | No harness | Kimi K2.5, Accuracy: 24.4% (scale 0.0 to 100.0), 95% confidence interval 20.8 to 27.924.4%(20.8–27.9) | 67% | Runs unstated | Cost kind unknown | Date not stated |
19[14–21] OpenAIDossier | No effort set | No harness | GPT-5.1, Accuracy: 23.7% (scale 0.0 to 100.0), 95% confidence interval 20.4 to 27.023.7%(20.4–27.0) | 55% | Runs unstated | Cost kind unknown | Date not stated |
20[16–24] GoogleDossier | No effort set | No harness | Gemini 2.5 Pro Preview (06-05), Accuracy: 21.6% (scale 0.0 to 100.0), 95% confidence interval 18.5 to 24.821.6%(18.5–24.8) | 72% | Runs unstated | Cost kind unknown | Date not stated |
21[19–26] OpenAIDossier | high | No harness | o3, Accuracy: 20.3% (scale 0.0 to 100.0), 95% confidence interval 17.2 to 23.420.3%(17.2–23.4) | 34% | Runs unstated | Cost kind unknown | Date not stated |
22[19–27] OpenAIDossier | No effort set | No harness | GPT-5 mini, Accuracy: 19.4% (scale 0.0 to 100.0), 95% confidence interval 16.4 to 22.519.4%(16.4–22.5) | 65% | Runs unstated | Cost kind unknown | Date not stated |
23[20–27] OpenAIDossier | medium | No harness | o3, Accuracy: 19.2% (scale 0.0 to 100.0), 95% confidence interval 16.2 to 22.219.2%(16.2–22.2) | 39% | Runs unstated | Cost kind unknown | Date not stated |
24[20–27] AnthropicDossier | No effort set | No harness | Opus 4.6, Accuracy: 19% (scale 0.0 to 100.0), 95% confidence interval 16.0 to 22.019%(16.0–22.0) | 44% | Runs unstated | Cost kind unknown | Date not stated |
25[21–28] GoogleDossier | No effort set | No harness | Gemini 2.5 Pro Experimental (03-25), Accuracy: 18.2% (scale 0.0 to 100.0), 95% confidence interval 15.2 to 21.118.2%(15.2–21.1) | 71% | Runs unstated | Cost kind unknown | Date not stated |
26[21–28] OpenAIDossier | high | No harness | o4-mini, Accuracy: 18.1% (scale 0.0 to 100.0), 95% confidence interval 15.1 to 21.018.1%(15.1–21.0) | 57% | Runs unstated | Cost kind unknown | Date not stated |
27[21–28] GoogleDossier | No effort set | No harness | Gemini 2.5 Pro Preview (May '25), Accuracy: 17.8% (scale 0.0 to 100.0), 95% confidence interval 14.9 to 20.717.8%(14.9–20.7) | 70% | Runs unstated | Cost kind unknown | Date not stated |
28[26–32] OpenAIDossier | medium | No harness | o4-mini, Accuracy: 14.3% (scale 0.0 to 100.0), 95% confidence interval 11.6 to 17.014.3%(11.6–17.0) | 59% | Runs unstated | Cost kind unknown | Date not stated |
29[27–33] AnthropicDossier | No effort set | No harness | Opus 4.5, Accuracy: 14.2% (scale 0.0 to 100.0), 95% confidence interval 11.5 to 16.814.2%(11.5–16.8) | 56% | Runs unstated | Cost kind unknown | Date not stated |
30[28–33] AnthropicDossier | No effort set | No harness | Sonnet 4.5, Accuracy: 13.7% (scale 0.0 to 100.0), 95% confidence interval 11.1 to 16.413.7%(11.1–16.4) | 65% | Runs unstated | Cost kind unknown | Date not stated |
31[28–35] GoogleDossier | No effort set | No harness | Gemini 2.5 Flash Preview (04-17), Accuracy: 12.1% (scale 0.0 to 100.0), 95% confidence interval 9.6 to 14.612.1%(9.6–14.6) | 80% | Runs unstated | Cost kind unknown | Date not stated |
32[29–38] AnthropicDossier | No effort set | No harness | Opus 4.1, Accuracy: 11.5% (scale 0.0 to 100.0), 95% confidence interval 9.1 to 14.011.5%(9.1–14.0) | 71% | Runs unstated | Cost kind unknown | Date not stated |
33[29–36] GoogleDossier | No effort set | No harness | Gemini 2.5 Flash Preview (05-20), Accuracy: 11% (scale 0.0 to 100.0), 95% confidence interval 8.6 to 13.411%(8.6–13.4) | 82% | Runs unstated | Cost kind unknown | Date not stated |
34[30–37] AnthropicDossier | thinking | No harness | Opus 4, Accuracy: 10.7% (scale 0.0 to 100.0), 95% confidence interval 8.3 to 13.110.7%(8.3–13.1) | 73% | Runs unstated | Cost kind unknown | Date not stated |
35[33–44] GoogleDossier | No effort set | No harness | Gemini 3.1 Flash-Lite Preview, Accuracy: 8.6% (scale 0.0 to 100.0), 95% confidence interval 6.5 to 10.88.6%(6.5–10.8) | 83% | Runs unstated | Cost kind unknown | Date not stated |
36[34–47] Z.aiDossier | No effort set | No harness | GLM-4.5, Accuracy: 8.3% (scale 0.0 to 100.0), 95% confidence interval 6.2 to 10.48.3%(6.2–10.4) | 79% | Runs unstated | Cost kind unknown | Date not stated |
37[34–47] Z.aiDossier | No effort set | No harness | GLM-4.5-Air, Accuracy: 8.1% (scale 0.0 to 100.0), 95% confidence interval 6.0 to 10.28.1%(6.0–10.2) | 77% | Runs unstated | Cost kind unknown | Date not stated |
37[34–46] OpenAIDossier | No effort set | No harness | o1-pro, Accuracy: 8.1% (scale 0.0 to 100.0), 95% confidence interval 6.0 to 10.28.1%(6.0–10.2) | 82% | Runs unstated | Cost kind unknown | Date not stated |
39[34–46] AnthropicDossier | thinking | No harness | Claude 3.7 Sonnet, Accuracy: 8% (scale 0.0 to 100.0), 95% confidence interval 5.9 to 10.18%(5.9–10.1) | 80% | Runs unstated | Cost kind unknown | Date not stated |
40[35–48] OpenAIDossier | No effort set | No harness | o1, Accuracy: 8% (scale 0.0 to 100.0), 95% confidence interval 5.9 to 10.08%(5.9–10.0) | 83% | Runs unstated | Cost kind unknown | Date not stated |
41[34–48] AnthropicDossier | No effort set | No harness | Opus 4.1, Accuracy: 7.9% (scale 0.0 to 100.0), 95% confidence interval 5.8 to 10.07.9%(5.8–10.0) | 70% | Runs unstated | Cost kind unknown | Date not stated |
42[34–48] AnthropicDossier | thinking | No harness | Claude Sonnet 4, Accuracy: 7.8% (scale 0.0 to 100.0), 95% confidence interval 5.7 to 9.87.8%(5.7–9.8) | 75% | Runs unstated | Cost kind unknown | Date not stated |
43[35–49] AnthropicDossier | No effort set | No harness | Sonnet 4.5, Accuracy: 7.5% (scale 0.0 to 100.0), 95% confidence interval 5.5 to 9.57.5%(5.5–9.5) | 70% | Runs unstated | Cost kind unknown | Date not stated |
44[37–52] OpenAIDossier | none | No harness | GPT-5.1, Accuracy: 6.8% (scale 0.0 to 100.0), 95% confidence interval 4.9 to 8.76.8%(4.9–8.7) | 69% | Runs unstated | Cost kind unknown | Date not stated |
45[38–51] AnthropicDossier | No effort set | No harness | Opus 4, Accuracy: 6.7% (scale 0.0 to 100.0), 95% confidence interval 4.8 to 8.66.7%(4.8–8.6) | 74% | Runs unstated | Cost kind unknown | Date not stated |
46[38–52] GoogleDossier | No effort set | No harness | Gemini 2.0 Flash Thinking Experimental (01-21), Accuracy: 6.6% (scale 0.0 to 100.0), 95% confidence interval 4.7 to 8.56.6%(4.7–8.5) | 82% | Runs unstated | Cost kind unknown | Date not stated |
47[41–53] MetaDossier | No effort set | No harness | Llama 4 Maverick, Accuracy: 5.7% (scale 0.0 to 100.0), 95% confidence interval 3.9 to 7.55.7%(3.9–7.5) | 83% | Runs unstated | Cost kind unknown | Date not stated |
48[42–54] AnthropicDossier | No effort set | No harness | Claude Sonnet 4, Accuracy: 5.5% (scale 0.0 to 100.0), 95% confidence interval 3.8 to 7.35.5%(3.8–7.3) | 76% | Runs unstated | Cost kind unknown | Date not stated |
49[42–55] OpenAIDossier | No effort set | No harness | GPT-4.5 Preview, Accuracy: 5.4% (scale 0.0 to 100.0), 95% confidence interval 3.7 to 7.25.4%(3.7–7.2) | 85% | Runs unstated | Cost kind unknown | Date not stated |
50[42–54] OpenAIDossier | No effort set | No harness | GPT-4.1, Accuracy: 5.4% (scale 0.0 to 100.0), 95% confidence interval 3.7 to 7.15.4%(3.7–7.1) | 89% | Runs unstated | Cost kind unknown | Date not stated |
51[46–55] GoogleDossier | No effort set | No harness | Gemini 1.5 Pro (002), Accuracy: 4.6% (scale 0.0 to 100.0), 95% confidence interval 3.0 to 6.24.6%(3.0–6.2) | 88% | Runs unstated | Cost kind unknown | Date not stated |
52[47–56] MistralDossier | No effort set | No harness | Mistral Medium 3, Accuracy: 4.5% (scale 0.0 to 100.0), 95% confidence interval 2.9 to 6.14.5%(2.9–6.1) | 77% | Runs unstated | Cost kind unknown | Date not stated |
53[47–55] AmazonDossier | No effort set | No harness | Nova Pro, Accuracy: 4.4% (scale 0.0 to 100.0), 95% confidence interval 2.8 to 6.04.4%(2.8–6.0) | 80% | Runs unstated | Cost kind unknown | Date not stated |
54[48–56] AnthropicDossier | No effort set | No harness | Claude 3.5 Sonnet (Oct '24), Accuracy: 4.1% (scale 0.0 to 100.0), 95% confidence interval 2.6 to 5.64.1%(2.6–5.6) | 84% | Runs unstated | Cost kind unknown | Date not stated |
55[50–56] AmazonDossier | No effort set | No harness | Nova Lite, Accuracy: 3.6% (scale 0.0 to 100.0), 95% confidence interval 2.2 to 5.13.6%(2.2–5.1) | 82% | Runs unstated | Cost kind unknown | Date not stated |
56[54–56] OpenAIDossier | No effort set | No harness | GPT-4o (Nov '24), Accuracy: 2.7% (scale 0.0 to 100.0), 95% confidence interval 1.5 to 4.02.7%(1.5–4.0) | 89% | Runs unstated | Cost kind unknown | Date not stated |
- 1OpenAIDossier
Accuracy
GPT-6 Astra, Accuracy: 54.8% (scale 0.0 to 100.0), 95% confidence interval 51.0 to 58.654.8%(51.0–58.6)- Calibration error
- 39%
- 2AnthropicDossier
xhigh
Accuracy
Fable 5.1, Accuracy: 46.5% (scale 0.0 to 100.0), 95% confidence interval 42.6 to 50.446.5%(42.6–50.4)- Calibration error
- 20%
- 3GoogleDossier
high
Accuracy
Gemini 3.1 Pro Preview, Accuracy: 46.4% (scale 0.0 to 100.0), 95% confidence interval 42.6 to 50.346.4%(42.6–50.3)- Calibration error
- 51%
- 4AnthropicDossier
xhigh
Accuracy
Sonnet 5.5, Accuracy: 46.1% (scale 0.0 to 100.0), 95% confidence interval 42.3 to 49.946.1%(42.3–49.9)- Calibration error
- Not tested
- 5GoogleDossier
Accuracy
Gemini 3.8 Flash, Accuracy: 44.5% (scale 0.0 to 100.0), 95% confidence interval 40.7 to 48.444.5%(40.7–48.4)- Calibration error
- 51%
- 6OpenAIDossier
Accuracy
GPT-5.4 Pro, Accuracy: 44.3% (scale 0.0 to 100.0), 95% confidence interval 40.5 to 48.144.3%(40.5–48.1)- Calibration error
- 38%
- 7MetaDossier
Accuracy
Muse Spark, Accuracy: 40.6% (scale 0.0 to 100.0), 95% confidence interval 36.8 to 44.340.6%(36.8–44.3)- Calibration error
- 50%
- 8GoogleDossier
Accuracy
Gemini 3 Pro Preview, Accuracy: 37.5% (scale 0.0 to 100.0), 95% confidence interval 33.8 to 41.237.5%(33.8–41.2)- Calibration error
- 57%
- 9OpenAIDossier
xhigh
Accuracy
GPT-5.4, Accuracy: 36.2% (scale 0.0 to 100.0), 95% confidence interval 32.6 to 39.936.2%(32.6–39.9)- Calibration error
- 42%
- 10AnthropicDossier
Accuracy
Opus 4.7, Accuracy: 36.2% (scale 0.0 to 100.0), 95% confidence interval 32.5 to 39.936.2%(32.5–39.9)- Calibration error
- 47%
- 11AnthropicDossier
max
Accuracy
Opus 4.6, Accuracy: 34.4% (scale 0.0 to 100.0), 95% confidence interval 30.8 to 38.134.4%(30.8–38.1)- Calibration error
- 46%
- 12OpenAIDossier
Accuracy
GPT-5 Pro, Accuracy: 31.6% (scale 0.0 to 100.0), 95% confidence interval 28.1 to 35.231.6%(28.1–35.2)- Calibration error
- 49%
- 13AnthropicDossier
Accuracy
Haiku 5.5, Accuracy: 30.1% (scale 0.0 to 100.0), 95% confidence interval 26.5 to 33.730.1%(26.5–33.7)- Calibration error
- Not tested
- 14OpenAIDossier
Accuracy
GPT-5.2, Accuracy: 27.8% (scale 0.0 to 100.0), 95% confidence interval 24.4 to 31.227.8%(24.4–31.2)- Calibration error
- 45%
- 15OpenAIDossier
high
Accuracy
GPT-5, Accuracy: 25.3% (scale 0.0 to 100.0), 95% confidence interval 22.0 to 28.725.3%(22.0–28.7)- Calibration error
- 50%
- 15OpenAIDossier
Accuracy
GPT-5, Accuracy: 25.3% (scale 0.0 to 100.0), 95% confidence interval 22.0 to 28.725.3%(22.0–28.7)- Calibration error
- 50%
- 17AnthropicDossier
Accuracy
Opus 4.5, Accuracy: 25.2% (scale 0.0 to 100.0), 95% confidence interval 21.9 to 28.525.2%(21.9–28.5)- Calibration error
- 55%
- 18Moonshot AIDossier
Accuracy
Kimi K2.5, Accuracy: 24.4% (scale 0.0 to 100.0), 95% confidence interval 20.8 to 27.924.4%(20.8–27.9)- Calibration error
- 67%
- 19OpenAIDossier
Accuracy
GPT-5.1, Accuracy: 23.7% (scale 0.0 to 100.0), 95% confidence interval 20.4 to 27.023.7%(20.4–27.0)- Calibration error
- 55%
- 20GoogleDossier
Accuracy
Gemini 2.5 Pro Preview (06-05), Accuracy: 21.6% (scale 0.0 to 100.0), 95% confidence interval 18.5 to 24.821.6%(18.5–24.8)- Calibration error
- 72%
- 21OpenAIDossier
high
Accuracy
o3, Accuracy: 20.3% (scale 0.0 to 100.0), 95% confidence interval 17.2 to 23.420.3%(17.2–23.4)- Calibration error
- 34%
- 22OpenAIDossier
Accuracy
GPT-5 mini, Accuracy: 19.4% (scale 0.0 to 100.0), 95% confidence interval 16.4 to 22.519.4%(16.4–22.5)- Calibration error
- 65%
- 23OpenAIDossier
medium
Accuracy
o3, Accuracy: 19.2% (scale 0.0 to 100.0), 95% confidence interval 16.2 to 22.219.2%(16.2–22.2)- Calibration error
- 39%
- 24AnthropicDossier
Accuracy
Opus 4.6, Accuracy: 19% (scale 0.0 to 100.0), 95% confidence interval 16.0 to 22.019%(16.0–22.0)- Calibration error
- 44%
- 25GoogleDossier
Accuracy
Gemini 2.5 Pro Experimental (03-25), Accuracy: 18.2% (scale 0.0 to 100.0), 95% confidence interval 15.2 to 21.118.2%(15.2–21.1)- Calibration error
- 71%
- 26OpenAIDossier
high
Accuracy
o4-mini, Accuracy: 18.1% (scale 0.0 to 100.0), 95% confidence interval 15.1 to 21.018.1%(15.1–21.0)- Calibration error
- 57%
- 27GoogleDossier
Accuracy
Gemini 2.5 Pro Preview (May '25), Accuracy: 17.8% (scale 0.0 to 100.0), 95% confidence interval 14.9 to 20.717.8%(14.9–20.7)- Calibration error
- 70%
- 28OpenAIDossier
medium
Accuracy
o4-mini, Accuracy: 14.3% (scale 0.0 to 100.0), 95% confidence interval 11.6 to 17.014.3%(11.6–17.0)- Calibration error
- 59%
- 29AnthropicDossier
Accuracy
Opus 4.5, Accuracy: 14.2% (scale 0.0 to 100.0), 95% confidence interval 11.5 to 16.814.2%(11.5–16.8)- Calibration error
- 56%
- 30AnthropicDossier
Accuracy
Sonnet 4.5, Accuracy: 13.7% (scale 0.0 to 100.0), 95% confidence interval 11.1 to 16.413.7%(11.1–16.4)- Calibration error
- 65%
- 31GoogleDossier
Accuracy
Gemini 2.5 Flash Preview (04-17), Accuracy: 12.1% (scale 0.0 to 100.0), 95% confidence interval 9.6 to 14.612.1%(9.6–14.6)- Calibration error
- 80%
- 32AnthropicDossier
Accuracy
Opus 4.1, Accuracy: 11.5% (scale 0.0 to 100.0), 95% confidence interval 9.1 to 14.011.5%(9.1–14.0)- Calibration error
- 71%
- 33GoogleDossier
Accuracy
Gemini 2.5 Flash Preview (05-20), Accuracy: 11% (scale 0.0 to 100.0), 95% confidence interval 8.6 to 13.411%(8.6–13.4)- Calibration error
- 82%
- 34AnthropicDossier
thinking
Accuracy
Opus 4, Accuracy: 10.7% (scale 0.0 to 100.0), 95% confidence interval 8.3 to 13.110.7%(8.3–13.1)- Calibration error
- 73%
- 35GoogleDossier
Accuracy
Gemini 3.1 Flash-Lite Preview, Accuracy: 8.6% (scale 0.0 to 100.0), 95% confidence interval 6.5 to 10.88.6%(6.5–10.8)- Calibration error
- 83%
- 36Z.aiDossier
Accuracy
GLM-4.5, Accuracy: 8.3% (scale 0.0 to 100.0), 95% confidence interval 6.2 to 10.48.3%(6.2–10.4)- Calibration error
- 79%
- 37Z.aiDossier
Accuracy
GLM-4.5-Air, Accuracy: 8.1% (scale 0.0 to 100.0), 95% confidence interval 6.0 to 10.28.1%(6.0–10.2)- Calibration error
- 77%
- 37OpenAIDossier
Accuracy
o1-pro, Accuracy: 8.1% (scale 0.0 to 100.0), 95% confidence interval 6.0 to 10.28.1%(6.0–10.2)- Calibration error
- 82%
- 39AnthropicDossier
thinking
Accuracy
Claude 3.7 Sonnet, Accuracy: 8% (scale 0.0 to 100.0), 95% confidence interval 5.9 to 10.18%(5.9–10.1)- Calibration error
- 80%
- 40OpenAIDossier
Accuracy
o1, Accuracy: 8% (scale 0.0 to 100.0), 95% confidence interval 5.9 to 10.08%(5.9–10.0)- Calibration error
- 83%
- 41AnthropicDossier
Accuracy
Opus 4.1, Accuracy: 7.9% (scale 0.0 to 100.0), 95% confidence interval 5.8 to 10.07.9%(5.8–10.0)- Calibration error
- 70%
- 42AnthropicDossier
thinking
Accuracy
Claude Sonnet 4, Accuracy: 7.8% (scale 0.0 to 100.0), 95% confidence interval 5.7 to 9.87.8%(5.7–9.8)- Calibration error
- 75%
- 43AnthropicDossier
Accuracy
Sonnet 4.5, Accuracy: 7.5% (scale 0.0 to 100.0), 95% confidence interval 5.5 to 9.57.5%(5.5–9.5)- Calibration error
- 70%
- 44OpenAIDossier
none
Accuracy
GPT-5.1, Accuracy: 6.8% (scale 0.0 to 100.0), 95% confidence interval 4.9 to 8.76.8%(4.9–8.7)- Calibration error
- 69%
- 45AnthropicDossier
Accuracy
Opus 4, Accuracy: 6.7% (scale 0.0 to 100.0), 95% confidence interval 4.8 to 8.66.7%(4.8–8.6)- Calibration error
- 74%
- 46GoogleDossier
Accuracy
Gemini 2.0 Flash Thinking Experimental (01-21), Accuracy: 6.6% (scale 0.0 to 100.0), 95% confidence interval 4.7 to 8.56.6%(4.7–8.5)- Calibration error
- 82%
- 47MetaDossier
Accuracy
Llama 4 Maverick, Accuracy: 5.7% (scale 0.0 to 100.0), 95% confidence interval 3.9 to 7.55.7%(3.9–7.5)- Calibration error
- 83%
- 48AnthropicDossier
Accuracy
Claude Sonnet 4, Accuracy: 5.5% (scale 0.0 to 100.0), 95% confidence interval 3.8 to 7.35.5%(3.8–7.3)- Calibration error
- 76%
- 49OpenAIDossier
Accuracy
GPT-4.5 Preview, Accuracy: 5.4% (scale 0.0 to 100.0), 95% confidence interval 3.7 to 7.25.4%(3.7–7.2)- Calibration error
- 85%
- 50OpenAIDossier
Accuracy
GPT-4.1, Accuracy: 5.4% (scale 0.0 to 100.0), 95% confidence interval 3.7 to 7.15.4%(3.7–7.1)- Calibration error
- 89%
- 51GoogleDossier
Accuracy
Gemini 1.5 Pro (002), Accuracy: 4.6% (scale 0.0 to 100.0), 95% confidence interval 3.0 to 6.24.6%(3.0–6.2)- Calibration error
- 88%
- 52MistralDossier
Accuracy
Mistral Medium 3, Accuracy: 4.5% (scale 0.0 to 100.0), 95% confidence interval 2.9 to 6.14.5%(2.9–6.1)- Calibration error
- 77%
- 53AmazonDossier
Accuracy
Nova Pro, Accuracy: 4.4% (scale 0.0 to 100.0), 95% confidence interval 2.8 to 6.04.4%(2.8–6.0)- Calibration error
- 80%
- 54AnthropicDossier
Accuracy
Claude 3.5 Sonnet (Oct '24), Accuracy: 4.1% (scale 0.0 to 100.0), 95% confidence interval 2.6 to 5.64.1%(2.6–5.6)- Calibration error
- 84%
- 55AmazonDossier
Accuracy
Nova Lite, Accuracy: 3.6% (scale 0.0 to 100.0), 95% confidence interval 2.2 to 5.13.6%(2.2–5.1)- Calibration error
- 82%
- 56OpenAIDossier
Accuracy
GPT-4o (Nov '24), Accuracy: 2.7% (scale 0.0 to 100.0), 95% confidence interval 1.5 to 4.02.7%(1.5–4.0)- Calibration error
- 89%