FrontierMath-2025-02-28-Private
FrontierMath, 2025-02-28 private set, run by Epoch AI. Superseded in benchmark_metadata by FrontierMath-Tiers-1-3-v2-Private.
No filters. Showing 101 of 101 rows.
1[1–6] OpenAIDossier | high | No harness | GPT-5.5 Pro Pre-release, Accuracy (best score across scorers): 52.4% (scale 0.0 to 100.0), 95% confidence interval 46.7 to 58.152.4%(46.7–58.1) | 52.4% | Runs unstated | Cost kind unknown |
2[1–6] OpenAIDossierTied #1 | xhigh | No harness | GPT-5.5 Pre-release, Accuracy (best score across scorers): 51.7% (scale 0.0 to 100.0), 95% confidence interval 46.0 to 57.451.7%(46.0–57.4) | 51.7% | Runs unstated | Cost kind unknown |
3[1–6] OpenAIDossierTied #1 | xhigh | No harness | GPT-5.5 Pro Pre-release, Accuracy (best score across scorers): 51% (scale 0.0 to 100.0), 95% confidence interval 45.3 to 56.751%(45.3–56.7) | 51% | Runs unstated | Cost kind unknown |
4[1–7] OpenAIDossierTied #1 | xhigh | No harness | GPT-5.4 Pro, Accuracy (best score across scorers): 50% (scale 0.0 to 100.0), 95% confidence interval 44.3 to 55.750%(44.3–55.7) | 50% | Runs unstated | Cost kind unknown |
5[1–10] OpenAIDossierTied #1 | xhigh | No harness | GPT-5.4, Accuracy (best score across scorers): 47.6% (scale 0.0 to 100.0), 95% confidence interval 41.9 to 53.347.6%(41.9–53.3) | 47.6% | Runs unstated | Cost kind unknown |
6[2–9] AnthropicDossierTied #1 | max | No harness | Opus 4.8, Accuracy (best score across scorers): 47.2% (scale 0.0 to 100.0), 95% confidence interval 41.5 to 53.047.2%(41.5–53.0) | 47.2% | Runs unstated | Cost kind unknown |
7[4–15] AnthropicDossier | xhigh | No harness | Opus 4.7, Accuracy (best score across scorers): 43.8% (scale 0.0 to 100.0), 95% confidence interval 38.1 to 49.543.8%(38.1–49.5) | 43.8% | Runs unstated | Cost kind unknown |
8[6–19] OpenAIDossier | xhigh | No harness | GPT-5.2, Accuracy (best score across scorers): 40.7% (scale 0.0 to 100.0), 95% confidence interval 35.0 to 46.440.7%(35.0–46.4) | 40.7% | Runs unstated | Cost kind unknown |
8[6–20] AnthropicDossier | max | No harness | Opus 4.6, Accuracy (best score across scorers): 40.7% (scale 0.0 to 100.0), 95% confidence interval 35.0 to 46.440.7%(35.0–46.4) | 40.7% | Runs unstated | Cost kind unknown |
10[6–21] OpenAIDossier | high | No harness | GPT-5.2, Accuracy (best score across scorers): 40.3% (scale 0.0 to 100.0), 95% confidence interval 34.7 to 46.040.3%(34.7–46.0) | 40.3% | Runs unstated | Cost kind unknown |
11[7–20] AnthropicDossier | 32,000 budget | No harness | Opus 4.6, Accuracy (best score across scorers): 40% (scale 0.0 to 100.0), 95% confidence interval 34.4 to 45.640%(34.4–45.6) | 40% | Runs unstated | Cost kind unknown |
12[7–21] AnthropicDossier | 64,000 budget | No harness | Opus 4.6, Accuracy (best score across scorers): 39.7% (scale 0.0 to 100.0), 95% confidence interval 34.0 to 45.339.7%(34.0–45.3) | 39.7% | Runs unstated | Cost kind unknown |
13[7–21] MetaDossier | No effort set | No harness | Muse Spark, Accuracy (best score across scorers): 39% (scale 0.0 to 100.0), 95% confidence interval 33.3 to 44.739%(33.3–44.7) | 39% | Runs unstated | Cost kind unknown |
14[7–21] GoogleDossier | high | No harness | Gemini 3.5 Flash, Accuracy (best score across scorers): 39% (scale 0.0 to 100.0), 95% confidence interval 33.3 to 44.639%(33.3–44.6) | 39% | Runs unstated | Cost kind unknown |
14[8–22] Moonshot AIDossier | No effort set | No harness | Kimi K2.6, Accuracy (best score across scorers): 39% (scale 0.0 to 100.0), 95% confidence interval 33.3 to 44.639%(33.3–44.6) | 39% | Runs unstated | Cost kind unknown |
16[8–22] AnthropicDossier | No effort set | No harness | Opus 4.6, Accuracy (best score across scorers): 38.3% (scale 0.0 to 100.0), 95% confidence interval 32.7 to 43.938.3%(32.7–43.9) | 38.3% | Runs unstated | Cost kind unknown |
17[8–22] GoogleDossier | No effort set | No harness | Gemini 3 Pro Preview, Accuracy (best score across scorers): 37.6% (scale 0.0 to 100.0), 95% confidence interval 32.0 to 43.237.6%(32.0–43.2) | 37.6% | Runs unstated | Cost kind unknown |
18[9–24] GoogleDossier | No effort set | No harness | Gemini 3.1 Pro Preview, Accuracy (best score across scorers): 36.9% (scale 0.0 to 100.0), 95% confidence interval 31.4 to 42.436.9%(31.4–42.4) | 36.9% | Runs unstated | Cost kind unknown |
19[9–23] OpenAIDossier | medium | No harness | GPT-5.2, Accuracy (best score across scorers): 36.9% (scale 0.0 to 100.0), 95% confidence interval 31.3 to 42.536.9%(31.3–42.5) | 36.9% | Runs unstated | Cost kind unknown |
20[9–25] GoogleDossier | No effort set | No harness | Gemini 3 Flash Preview, Accuracy (best score across scorers): 35.6% (scale 0.0 to 100.0), 95% confidence interval 30.1 to 41.235.6%(30.1–41.2) | 35.6% | Runs unstated | Cost kind unknown |
21[14–28] Z.aiDossier | No effort set | No harness | GLM-5.1, Accuracy (best score across scorers): 33.4% (scale 0.0 to 100.0), 95% confidence interval 28.0 to 38.933.4%(28.0–38.9) | 33.4% | Runs unstated | Cost kind unknown |
22[14–30] OpenAIDossier | high | No harness | GPT-5, Accuracy (best score across scorers): 32.4% (scale 0.0 to 100.0), 95% confidence interval 27.0 to 37.832.4%(27.0–37.8) | 32.4% | Runs unstated | Cost kind unknown |
23[16–30] AnthropicDossier | 16,000 budget | No harness | Sonnet 4.6, Accuracy (best score across scorers): 32.4% (scale 0.0 to 100.0), 95% confidence interval 26.9 to 37.932.4%(26.9–37.9) | 32.4% | Runs unstated | Cost kind unknown |
24[19–31] OpenAIDossier | high | No harness | GPT-5.1, Accuracy (best score across scorers): 31% (scale 0.0 to 100.0), 95% confidence interval 25.7 to 36.431%(25.7–36.4) | 31% | Runs unstated | Cost kind unknown |
25[20–35] GoogleDossier | No effort set | No harness | Gemini 2.5 Deep Think, Accuracy (best score across scorers): 29% (scale 0.0 to 100.0), 95% confidence interval 23.7 to 34.329%(23.7–34.3) | 29% | Runs unstated | Cost kind unknown |
26[21–33] OpenAIDossier | high | No harness | GPT-5.4 mini, Accuracy (best score across scorers): 28.3% (scale 0.0 to 100.0), 95% confidence interval 24.2 to 32.328.3%(24.2–32.3) | 28.3% | Runs unstated | Cost kind unknown |
27[22–36] Moonshot AIDossier | No effort set | No harness | Kimi K2.5, Accuracy (best score across scorers): 27.9% (scale 0.0 to 100.0), 95% confidence interval 22.8 to 33.027.9%(22.8–33.0) | 27.9% | Runs unstated | Cost kind unknown |
28[22–37] OpenAIDossier | medium | No harness | GPT-5, Accuracy (best score across scorers): 27.2% (scale 0.0 to 100.0), 95% confidence interval 22.1 to 32.427.2%(22.1–32.4) | 27.2% | Runs unstated | Cost kind unknown |
28[23–36] OpenAIDossier | high | No harness | GPT-5 mini, Accuracy (best score across scorers): 27.2% (scale 0.0 to 100.0), 95% confidence interval 22.1 to 32.427.2%(22.1–32.4) | 27.2% | Runs unstated | Cost kind unknown |
30[23–39] OpenAIDossier | medium | No harness | GPT-5.1, Accuracy (best score across scorers): 26.9% (scale 0.0 to 100.0), 95% confidence interval 21.8 to 32.026.9%(21.8–32.0) | 26.9% | Runs unstated | Cost kind unknown |
31[24–38] OpenAIDossier | low | No harness | GPT-5.2, Accuracy (best score across scorers): 26.6% (scale 0.0 to 100.0), 95% confidence interval 21.5 to 31.626.6%(21.5–31.6) | 26.6% | Runs unstated | Cost kind unknown |
32[23–39] AlibabaDossier | No effort set | No harness | Qwen3.6 Plus, Accuracy (best score across scorers): 26.2% (scale 0.0 to 100.0), 95% confidence interval 21.1 to 31.326.2%(21.1–31.3) | 26.2% | Runs unstated | Cost kind unknown |
33[24–40] OpenAIDossier | high | No harness | GPT-5.4 nano, Accuracy (best score across scorers): 25.9% (scale 0.0 to 100.0), 95% confidence interval 20.8 to 30.925.9%(20.8–30.9) | 25.9% | Runs unstated | Cost kind unknown |
34[25–41] OpenAIDossier | high | No harness | o4-mini, Accuracy (best score across scorers): 24.8% (scale 0.0 to 100.0), 95% confidence interval 19.8 to 29.824.8%(19.8–29.8) | 24.8% | Runs unstated | Cost kind unknown |
35[29–46] AlibabaDossier | No effort set | No harness | Qwen3.6 Max Preview, Accuracy (best score across scorers): 23.1% (scale 0.0 to 100.0), 95% confidence interval 18.2 to 28.023.1%(18.2–28.0) | 23.1% | Runs unstated | Cost kind unknown |
36[26–50] DeepSeekDossier | No effort set | No harness | DeepSeek V3.2, Accuracy (best score across scorers): 22.1% (scale 0.0 to 100.0), 95% confidence interval 14.1 to 30.122.1%(14.1–30.1) | 22.1% | Runs unstated | Cost kind unknown |
37[31–46] Moonshot AIDossier | No effort set | No harness | Kimi K2 Thinking, Accuracy (best score across scorers): 21.4% (scale 0.0 to 100.0), 95% confidence interval 16.6 to 26.221.4%(16.6–26.2) | 21.4% | Runs unstated | Cost kind unknown |
38[31–47] AlibabaDossier | No effort set | No harness | Qwen3.5 Plus, Accuracy (best score across scorers): 21% (scale 0.0 to 100.0), 95% confidence interval 16.3 to 25.721%(16.3–25.7) | 21% | Runs unstated | Cost kind unknown |
39[32–48] AnthropicDossier | No effort set | No harness | Opus 4.5, Accuracy (best score across scorers): 20.7% (scale 0.0 to 100.0), 95% confidence interval 16.0 to 25.420.7%(16.0–25.4) | 20.7% | Runs unstated | Cost kind unknown |
39[33–47] AnthropicDossier | 32,000 budget | No harness | Opus 4.5, Accuracy (best score across scorers): 20.7% (scale 0.0 to 100.0), 95% confidence interval 16.0 to 25.420.7%(16.0–25.4) | 20.7% | Runs unstated | Cost kind unknown |
41[33–48] OpenAIDossier | medium | No harness | GPT-5 mini, Accuracy (best score across scorers): 20.3% (scale 0.0 to 100.0), 95% confidence interval 15.7 to 25.020.3%(15.7–25.0) | 20.3% | Runs unstated | Cost kind unknown |
41[33–48] AnthropicDossier | 16,000 budget | No harness | Opus 4.5, Accuracy (best score across scorers): 20.3% (scale 0.0 to 100.0), 95% confidence interval 15.7 to 25.020.3%(15.7–25.0) | 20.3% | Runs unstated | Cost kind unknown |
43[34–49] xAIDossier | No effort set | No harness | Grok 4 (0709), Accuracy (best score across scorers): 19.7% (scale 0.0 to 100.0), 95% confidence interval 15.1 to 24.219.7%(15.1–24.2) | 19.7% | Runs unstated | Cost kind unknown |
44[35–49] OpenAIDossier | medium | No harness | o4-mini, Accuracy (best score across scorers): 19% (scale 0.0 to 100.0), 95% confidence interval 14.4 to 23.519%(14.4–23.5) | 19% | Runs unstated | Cost kind unknown |
45[36–50] OpenAIDossier | high | No harness | o3, Accuracy (best score across scorers): 18.7% (scale 0.0 to 100.0), 95% confidence interval 14.2 to 23.218.7%(14.2–23.2) | 18.7% | Runs unstated | Cost kind unknown |
46[38–51] OpenAIDossier | low | No harness | GPT-5.1, Accuracy (best score across scorers): 17.3% (scale 0.0 to 100.0), 95% confidence interval 12.9 to 21.717.3%(12.9–21.7) | 17.3% | Runs unstated | Cost kind unknown |
47[38–52] OpenAIDossier | medium | No harness | o3, Accuracy (best score across scorers): 16.9% (scale 0.0 to 100.0), 95% confidence interval 12.6 to 21.216.9%(12.6–21.2) | 16.9% | Runs unstated | Cost kind unknown |
48[38–52] Z.aiDossier | No effort set | No harness | GLM-5, Accuracy (best score across scorers): 16.4% (scale 0.0 to 100.0), 95% confidence interval 12.1 to 20.716.4%(12.1–20.7) | 16.4% | Runs unstated | Cost kind unknown |
49[42–54] AnthropicDossier | 32,000 budget | No harness | Sonnet 4.5, Accuracy (best score across scorers): 15.2% (scale 0.0 to 100.0), 95% confidence interval 11.1 to 19.415.2%(11.1–19.4) | 15.2% | Runs unstated | Cost kind unknown |
50[45–56] GoogleDossier | No effort set | No harness | Gemini 2.5 Pro, Accuracy (best score across scorers): 14.1% (scale 0.0 to 100.0), 95% confidence interval 10.1 to 18.214.1%(10.1–18.2) | 14.1% | Runs unstated | Cost kind unknown |
51[46–56] AnthropicDossier | 59,000 budget | No harness | Sonnet 4.5, Accuracy (best score across scorers): 13.5% (scale 0.0 to 100.0), 95% confidence interval 9.5 to 17.413.5%(9.5–17.4) | 13.5% | Runs unstated | Cost kind unknown |
52[47–59] OpenAIDossier | high | No harness | o3-mini, Accuracy (best score across scorers): 12.4% (scale 0.0 to 100.0), 95% confidence interval 8.6 to 16.212.4%(8.6–16.2) | 12.4% | Runs unstated | Cost kind unknown |
53[50–62] OpenAIDossier | low | No harness | o4-mini, Accuracy (best score across scorers): 10.7% (scale 0.0 to 100.0), 95% confidence interval 7.1 to 14.310.7%(7.1–14.3) | 10.7% | Runs unstated | Cost kind unknown |
54[50–62] GoogleDossier | No effort set | No harness | Gemini 2.5 Pro Preview (06-05), Accuracy (best score across scorers): 10.3% (scale 0.0 to 100.0), 95% confidence interval 6.8 to 13.910.3%(6.8–13.9) | 10.3% | Runs unstated | Cost kind unknown |
54[50–64] AlibabaDossier | No effort set | No harness | Qwen3.6 Flash, Accuracy (best score across scorers): 10.3% (scale 0.0 to 100.0), 95% confidence interval 6.8 to 13.910.3%(6.8–13.9) | 10.3% | Runs unstated | Cost kind unknown |
56[51–64] OpenAIDossier | low | No harness | o3, Accuracy (best score across scorers): 9.7% (scale 0.0 to 100.0), 95% confidence interval 6.3 to 13.19.7%(6.3–13.1) | 9.7% | Runs unstated | Cost kind unknown |
57[52–65] OpenAIDossier | high | No harness | o1, Accuracy (best score across scorers): 9.3% (scale 0.0 to 100.0), 95% confidence interval 6.0 to 12.79.3%(6.0–12.7) | 9.3% | Runs unstated | Cost kind unknown |
57[52–65] AnthropicDossier | No effort set | No harness | Sonnet 4.5, Accuracy (best score across scorers): 9.3% (scale 0.0 to 100.0), 95% confidence interval 6.0 to 12.79.3%(6.0–12.7) | 9.3% | Runs unstated | Cost kind unknown |
59[52–67] AlibabaDossier | No effort set | No harness | Qwen3 235B A22B Thinking (2507), Accuracy (best score across scorers): 8.5% (scale 0.0 to 100.0), 95% confidence interval 5.2 to 11.78.5%(5.2–11.7) | 8.5% | Runs unstated | Cost kind unknown |
60[53–69] OpenAIDossier | high | No harness | GPT-5 nano, Accuracy (best score across scorers): 8.3% (scale 0.0 to 100.0), 95% confidence interval 5.1 to 11.58.3%(5.1–11.5) | 8.3% | Runs unstated | Cost kind unknown |
61[55–66] OpenAIDossier | medium | No harness | o3-mini, Accuracy (best score across scorers): 8.1% (scale 0.0 to 100.0), 95% confidence interval 5.6 to 10.68.1%(5.6–10.6) | 8.1% | Runs unstated | Cost kind unknown |
62[55–73] OpenAIDossier | medium | No harness | GPT-5 nano, Accuracy (best score across scorers): 7.2% (scale 0.0 to 100.0), 95% confidence interval 4.3 to 10.27.2%(4.3–10.2) | 7.2% | Runs unstated | Cost kind unknown |
62[55–71] AnthropicDossier | 27,000 budget | No harness | Opus 4.1, Accuracy (best score across scorers): 7.2% (scale 0.0 to 100.0), 95% confidence interval 4.3 to 10.27.2%(4.3–10.2) | 7.2% | Runs unstated | Cost kind unknown |
64[58–77] AlibabaDossier | No effort set | No harness | Qwen3.5 Flash, Accuracy (best score across scorers): 6.2% (scale 0.0 to 100.0), 95% confidence interval 3.4 to 9.06.2%(3.4–9.0) | 6.2% | Runs unstated | Cost kind unknown |
65[59–76] AnthropicDossier | 32,000 budget | No harness | Haiku 4.5, Accuracy (best score across scorers): 5.9% (scale 0.0 to 100.0), 95% confidence interval 3.2 to 8.65.9%(3.2–8.6) | 5.9% | Runs unstated | Cost kind unknown |
66[58–77] xAIDossier | high | No harness | Grok 3 Mini Beta, Accuracy (best score across scorers): 5.9% (scale 0.0 to 100.0), 95% confidence interval 3.2 to 8.65.9%(3.2–8.6) | 5.9% | Runs unstated | Cost kind unknown |
66[59–77] AnthropicDossier | No effort set | No harness | Opus 4.1, Accuracy (best score across scorers): 5.9% (scale 0.0 to 100.0), 95% confidence interval 3.2 to 8.65.9%(3.2–8.6) | 5.9% | Runs unstated | Cost kind unknown |
68[60–78] OpenAIDossier | No effort set | No harness | GPT-4.1, Accuracy (best score across scorers): 5.5% (scale 0.0 to 100.0), 95% confidence interval 2.9 to 8.15.5%(2.9–8.1) | 5.5% | Runs unstated | Cost kind unknown |
69[62–80] GoogleDossier | No effort set | No harness | Gemini 2.5 Flash, Accuracy (best score across scorers): 4.8% (scale 0.0 to 100.0), 95% confidence interval 2.4 to 7.34.8%(2.4–7.3) | 4.8% | Runs unstated | Cost kind unknown |
70[63–83] OpenAIDossier | No effort set | No harness | GPT-4.1 mini, Accuracy (best score across scorers): 4.5% (scale 0.0 to 100.0), 95% confidence interval 2.1 to 6.94.5%(2.1–6.9) | 4.5% | Runs unstated | Cost kind unknown |
70[63–81] AnthropicDossier | No effort set | No harness | Opus 4, Accuracy (best score across scorers): 4.5% (scale 0.0 to 100.0), 95% confidence interval 2.1 to 6.94.5%(2.1–6.9) | 4.5% | Runs unstated | Cost kind unknown |
72[64–87] AnthropicDossier | 16,000 budget | No harness | Claude 3.7 Sonnet, Accuracy (best score across scorers): 4.1% (scale 0.0 to 100.0), 95% confidence interval 1.8 to 6.44.1%(1.8–6.4) | 4.1% | Runs unstated | Cost kind unknown |
72[65–85] AnthropicDossier | No effort set | No harness | Claude Sonnet 4, Accuracy (best score across scorers): 4.1% (scale 0.0 to 100.0), 95% confidence interval 1.8 to 6.44.1%(1.8–6.4) | 4.1% | Runs unstated | Cost kind unknown |
72[65–86] AnthropicDossier | No effort set | No harness | Haiku 4.5, Accuracy (best score across scorers): 4.1% (scale 0.0 to 100.0), 95% confidence interval 1.8 to 6.44.1%(1.8–6.4) | 4.1% | Runs unstated | Cost kind unknown |
72[65–84] AnthropicDossier | 27,000 budget | No harness | Opus 4, Accuracy (best score across scorers): 4.1% (scale 0.0 to 100.0), 95% confidence interval 1.8 to 6.44.1%(1.8–6.4) | 4.1% | Runs unstated | Cost kind unknown |
76[66–87] Z.aiDossier | No effort set | No harness | GLM-4.6, Accuracy (best score across scorers): 3.8% (scale 0.0 to 100.0), 95% confidence interval 1.6 to 6.03.8%(1.6–6.0) | 3.8% | Runs unstated | Cost kind unknown |
77[65–89] xAIDossier | No effort set | No harness | Grok 3 Beta, Accuracy (best score across scorers): 3.8% (scale 0.0 to 100.0), 95% confidence interval 1.6 to 6.03.8%(1.6–6.0) | 3.8% | Runs unstated | Cost kind unknown |
78[67–90] AnthropicDossier | 32,000 budget | No harness | Claude 3.7 Sonnet, Accuracy (best score across scorers): 3.4% (scale 0.0 to 100.0), 95% confidence interval 1.3 to 5.63.4%(1.3–5.6) | 3.4% | Runs unstated | Cost kind unknown |
79[69–89] AnthropicDossier | 64,000 budget | No harness | Claude 3.7 Sonnet, Accuracy (best score across scorers): 3.1% (scale 0.0 to 100.0), 95% confidence interval 1.1 to 5.13.1%(1.1–5.1) | 3.1% | Runs unstated | Cost kind unknown |
79[70–89] AnthropicDossier | No effort set | No harness | Claude 3.7 Sonnet, Accuracy (best score across scorers): 3.1% (scale 0.0 to 100.0), 95% confidence interval 1.1 to 5.13.1%(1.1–5.1) | 3.1% | Runs unstated | Cost kind unknown |
81[71–92] xAIDossier | low | No harness | Grok 3 Mini Beta, Accuracy (best score across scorers): 2.8% (scale 0.0 to 100.0), 95% confidence interval 0.9 to 4.62.8%(0.9–4.6) | 2.8% | Runs unstated | Cost kind unknown |
82[73–92] Z.aiDossier | No effort set | No harness | GLM-4.7, Accuracy (best score across scorers): 2.4% (scale 0.0 to 100.0), 95% confidence interval 0.7 to 4.22.4%(0.7–4.2) | 2.4% | Runs unstated | Cost kind unknown |
83[76–93] OpenAIDossier | none | No harness | GPT-5.1, Accuracy (best score across scorers): 2.1% (scale 0.0 to 100.0), 95% confidence interval 0.4 to 3.82.1%(0.4–3.8) | 2.1% | Runs unstated | Cost kind unknown |
84[76–95] AnthropicDossier | No effort set | No harness | Claude 3.5 Sonnet (Oct '24), Accuracy (best score across scorers): 2.1% (scale 0.0 to 100.0), 95% confidence interval 0.4 to 3.72.1%(0.4–3.7) | 2.1% | Runs unstated | Cost kind unknown |
85[79–96] DeepSeekDossier | No effort set | No harness | DeepSeek V3, Accuracy (best score across scorers): 1.7% (scale 0.0 to 100.0), 95% confidence interval 0.2 to 3.21.7%(0.2–3.2) | 1.7% | Runs unstated | Cost kind unknown |
85[78–95] GoogleDossier | No effort set | No harness | Gemini 2.0 Flash (001), Accuracy (best score across scorers): 1.7% (scale 0.0 to 100.0), 95% confidence interval 0.2 to 3.21.7%(0.2–3.2) | 1.7% | Runs unstated | Cost kind unknown |
85[78–98] OpenAIDossier | medium | No harness | o1-mini, Accuracy (best score across scorers): 1.7% (scale 0.0 to 100.0), 95% confidence interval 0.2 to 3.21.7%(0.2–3.2) | 1.7% | Runs unstated | Cost kind unknown |
85[77–96] AlibabaDossier | No effort set | No harness | Qwen Plus, Accuracy (best score across scorers): 1.7% (scale 0.0 to 100.0), 95% confidence interval 0.2 to 3.21.7%(0.2–3.2) | 1.7% | Runs unstated | Cost kind unknown |
89[81–99] OpenAIDossier | high | No harness | o1-mini, Accuracy (best score across scorers): 1.4% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 2.71.4%(0.0–2.7) | 1.4% | Runs unstated | Cost kind unknown |
90[81–100] AnthropicDossier | No effort set | No harness | Claude 3.5 Sonnet (June '24), Accuracy (best score across scorers): 1% (scale 0.0 to 100.0), 95% confidence interval 0.3 to 3.21%(0.3–3.2) | 1% | Runs unstated | Cost kind unknown |
90[82–101] OpenAIDossier | No effort set | No harness | GPT-4.1 nano, Accuracy (best score across scorers): 1% (scale 0.0 to 100.0), 95% confidence interval 0.3 to 3.21%(0.3–3.2) | 1% | Runs unstated | Cost kind unknown |
90[82–101] AlibabaDossier | No effort set | No harness | Qwen Max, Accuracy (best score across scorers): 1% (scale 0.0 to 100.0), 95% confidence interval 0.3 to 3.21%(0.3–3.2) | 1% | Runs unstated | Cost kind unknown |
93[84–101] xAIDossier | No effort set | No harness | Grok 2 (1212), Accuracy (best score across scorers): 0.7% (scale 0.0 to 100.0), 95% confidence interval 0.2 to 2.70.7%(0.2–2.7) | 0.7% | Runs unstated | Cost kind unknown |
93[86–101] MetaDossier | No effort set | No harness | Llama 4 Maverick 17B 128E Instruct (FP8), Accuracy (best score across scorers): 0.7% (scale 0.0 to 100.0), 95% confidence interval 0.2 to 2.70.7%(0.2–2.7) | 0.7% | Runs unstated | Cost kind unknown |
95[87–101] MistralDossier | No effort set | No harness | Mistral Medium 3, Accuracy (best score across scorers): 0.3% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 2.40.3%(0.0–2.4) | 0.3% | Runs unstated | Cost kind unknown |
96[87–101] AnthropicDossier | No effort set | No harness | Claude 3.5 Haiku, Accuracy (best score across scorers): 0.3% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 2.40.3%(0.0–2.4) | 0.3% | Runs unstated | Cost kind unknown |
96[87–101] OpenAIDossier | No effort set | No harness | GPT-4o (Aug '24), Accuracy (best score across scorers): 0.3% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 2.40.3%(0.0–2.4) | 0.3% | Runs unstated | Cost kind unknown |
96[88–101] OpenAIDossier | No effort set | No harness | GPT-4o (Nov '24), Accuracy (best score across scorers): 0.3% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 2.40.3%(0.0–2.4) | 0.3% | Runs unstated | Cost kind unknown |
96[89–101] MistralDossier | No effort set | No harness | Mistral Large (2411), Accuracy (best score across scorers): 0.3% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 2.40.3%(0.0–2.4) | 0.3% | Runs unstated | Cost kind unknown |
100[96–100] GoogleDossier | No effort set | No harness | Gemini 1.5 Flash (002), Accuracy (best score across scorers): 0% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 0.00%(0.0–0.0) | 0% | Runs unstated | Cost kind unknown |
100[97–101] MetaDossier | No effort set | No harness | Llama 4 Scout, Accuracy (best score across scorers): 0% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 0.00%(0.0–0.0) | 0% | Runs unstated | Cost kind unknown |
- 1OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5.5 Pro Pre-release, Accuracy (best score across scorers): 52.4% (scale 0.0 to 100.0), 95% confidence interval 46.7 to 58.152.4%(46.7–58.1)- Mean score
- 52.4%
Accuracy (best score across scorers)
GPT-5.5 Pre-release, Accuracy (best score across scorers): 51.7% (scale 0.0 to 100.0), 95% confidence interval 46.0 to 57.451.7%(46.0–57.4)- Mean score
- 51.7%
Accuracy (best score across scorers)
GPT-5.5 Pro Pre-release, Accuracy (best score across scorers): 51% (scale 0.0 to 100.0), 95% confidence interval 45.3 to 56.751%(45.3–56.7)- Mean score
- 51%
Accuracy (best score across scorers)
GPT-5.4 Pro, Accuracy (best score across scorers): 50% (scale 0.0 to 100.0), 95% confidence interval 44.3 to 55.750%(44.3–55.7)- Mean score
- 50%
Accuracy (best score across scorers)
GPT-5.4, Accuracy (best score across scorers): 47.6% (scale 0.0 to 100.0), 95% confidence interval 41.9 to 53.347.6%(41.9–53.3)- Mean score
- 47.6%
Accuracy (best score across scorers)
Opus 4.8, Accuracy (best score across scorers): 47.2% (scale 0.0 to 100.0), 95% confidence interval 41.5 to 53.047.2%(41.5–53.0)- Mean score
- 47.2%
- 7AnthropicDossier
xhigh
Accuracy (best score across scorers)
Opus 4.7, Accuracy (best score across scorers): 43.8% (scale 0.0 to 100.0), 95% confidence interval 38.1 to 49.543.8%(38.1–49.5)- Mean score
- 43.8%
- 8OpenAIDossier
xhigh
Accuracy (best score across scorers)
GPT-5.2, Accuracy (best score across scorers): 40.7% (scale 0.0 to 100.0), 95% confidence interval 35.0 to 46.440.7%(35.0–46.4)- Mean score
- 40.7%
- 8AnthropicDossier
max
Accuracy (best score across scorers)
Opus 4.6, Accuracy (best score across scorers): 40.7% (scale 0.0 to 100.0), 95% confidence interval 35.0 to 46.440.7%(35.0–46.4)- Mean score
- 40.7%
- 10OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5.2, Accuracy (best score across scorers): 40.3% (scale 0.0 to 100.0), 95% confidence interval 34.7 to 46.040.3%(34.7–46.0)- Mean score
- 40.3%
- 11AnthropicDossier
32,000 budget
Accuracy (best score across scorers)
Opus 4.6, Accuracy (best score across scorers): 40% (scale 0.0 to 100.0), 95% confidence interval 34.4 to 45.640%(34.4–45.6)- Mean score
- 40%
- 12AnthropicDossier
64,000 budget
Accuracy (best score across scorers)
Opus 4.6, Accuracy (best score across scorers): 39.7% (scale 0.0 to 100.0), 95% confidence interval 34.0 to 45.339.7%(34.0–45.3)- Mean score
- 39.7%
- 13MetaDossier
Accuracy (best score across scorers)
Muse Spark, Accuracy (best score across scorers): 39% (scale 0.0 to 100.0), 95% confidence interval 33.3 to 44.739%(33.3–44.7)- Mean score
- 39%
- 14GoogleDossier
high
Accuracy (best score across scorers)
Gemini 3.5 Flash, Accuracy (best score across scorers): 39% (scale 0.0 to 100.0), 95% confidence interval 33.3 to 44.639%(33.3–44.6)- Mean score
- 39%
- 14Moonshot AIDossier
Accuracy (best score across scorers)
Kimi K2.6, Accuracy (best score across scorers): 39% (scale 0.0 to 100.0), 95% confidence interval 33.3 to 44.639%(33.3–44.6)- Mean score
- 39%
- 16AnthropicDossier
Accuracy (best score across scorers)
Opus 4.6, Accuracy (best score across scorers): 38.3% (scale 0.0 to 100.0), 95% confidence interval 32.7 to 43.938.3%(32.7–43.9)- Mean score
- 38.3%
- 17GoogleDossier
Accuracy (best score across scorers)
Gemini 3 Pro Preview, Accuracy (best score across scorers): 37.6% (scale 0.0 to 100.0), 95% confidence interval 32.0 to 43.237.6%(32.0–43.2)- Mean score
- 37.6%
- 18GoogleDossier
Accuracy (best score across scorers)
Gemini 3.1 Pro Preview, Accuracy (best score across scorers): 36.9% (scale 0.0 to 100.0), 95% confidence interval 31.4 to 42.436.9%(31.4–42.4)- Mean score
- 36.9%
- 19OpenAIDossier
medium
Accuracy (best score across scorers)
GPT-5.2, Accuracy (best score across scorers): 36.9% (scale 0.0 to 100.0), 95% confidence interval 31.3 to 42.536.9%(31.3–42.5)- Mean score
- 36.9%
- 20GoogleDossier
Accuracy (best score across scorers)
Gemini 3 Flash Preview, Accuracy (best score across scorers): 35.6% (scale 0.0 to 100.0), 95% confidence interval 30.1 to 41.235.6%(30.1–41.2)- Mean score
- 35.6%
- 21Z.aiDossier
Accuracy (best score across scorers)
GLM-5.1, Accuracy (best score across scorers): 33.4% (scale 0.0 to 100.0), 95% confidence interval 28.0 to 38.933.4%(28.0–38.9)- Mean score
- 33.4%
- 22OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5, Accuracy (best score across scorers): 32.4% (scale 0.0 to 100.0), 95% confidence interval 27.0 to 37.832.4%(27.0–37.8)- Mean score
- 32.4%
- 23AnthropicDossier
16,000 budget
Accuracy (best score across scorers)
Sonnet 4.6, Accuracy (best score across scorers): 32.4% (scale 0.0 to 100.0), 95% confidence interval 26.9 to 37.932.4%(26.9–37.9)- Mean score
- 32.4%
- 24OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5.1, Accuracy (best score across scorers): 31% (scale 0.0 to 100.0), 95% confidence interval 25.7 to 36.431%(25.7–36.4)- Mean score
- 31%
- 25GoogleDossier
Accuracy (best score across scorers)
Gemini 2.5 Deep Think, Accuracy (best score across scorers): 29% (scale 0.0 to 100.0), 95% confidence interval 23.7 to 34.329%(23.7–34.3)- Mean score
- 29%
- 26OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5.4 mini, Accuracy (best score across scorers): 28.3% (scale 0.0 to 100.0), 95% confidence interval 24.2 to 32.328.3%(24.2–32.3)- Mean score
- 28.3%
- 27Moonshot AIDossier
Accuracy (best score across scorers)
Kimi K2.5, Accuracy (best score across scorers): 27.9% (scale 0.0 to 100.0), 95% confidence interval 22.8 to 33.027.9%(22.8–33.0)- Mean score
- 27.9%
- 28OpenAIDossier
medium
Accuracy (best score across scorers)
GPT-5, Accuracy (best score across scorers): 27.2% (scale 0.0 to 100.0), 95% confidence interval 22.1 to 32.427.2%(22.1–32.4)- Mean score
- 27.2%
- 28OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5 mini, Accuracy (best score across scorers): 27.2% (scale 0.0 to 100.0), 95% confidence interval 22.1 to 32.427.2%(22.1–32.4)- Mean score
- 27.2%
- 30OpenAIDossier
medium
Accuracy (best score across scorers)
GPT-5.1, Accuracy (best score across scorers): 26.9% (scale 0.0 to 100.0), 95% confidence interval 21.8 to 32.026.9%(21.8–32.0)- Mean score
- 26.9%
- 31OpenAIDossier
low
Accuracy (best score across scorers)
GPT-5.2, Accuracy (best score across scorers): 26.6% (scale 0.0 to 100.0), 95% confidence interval 21.5 to 31.626.6%(21.5–31.6)- Mean score
- 26.6%
- 32AlibabaDossier
Accuracy (best score across scorers)
Qwen3.6 Plus, Accuracy (best score across scorers): 26.2% (scale 0.0 to 100.0), 95% confidence interval 21.1 to 31.326.2%(21.1–31.3)- Mean score
- 26.2%
- 33OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5.4 nano, Accuracy (best score across scorers): 25.9% (scale 0.0 to 100.0), 95% confidence interval 20.8 to 30.925.9%(20.8–30.9)- Mean score
- 25.9%
- 34OpenAIDossier
high
Accuracy (best score across scorers)
o4-mini, Accuracy (best score across scorers): 24.8% (scale 0.0 to 100.0), 95% confidence interval 19.8 to 29.824.8%(19.8–29.8)- Mean score
- 24.8%
- 35AlibabaDossier
Accuracy (best score across scorers)
Qwen3.6 Max Preview, Accuracy (best score across scorers): 23.1% (scale 0.0 to 100.0), 95% confidence interval 18.2 to 28.023.1%(18.2–28.0)- Mean score
- 23.1%
- 36DeepSeekDossier
Accuracy (best score across scorers)
DeepSeek V3.2, Accuracy (best score across scorers): 22.1% (scale 0.0 to 100.0), 95% confidence interval 14.1 to 30.122.1%(14.1–30.1)- Mean score
- 22.1%
- 37Moonshot AIDossier
Accuracy (best score across scorers)
Kimi K2 Thinking, Accuracy (best score across scorers): 21.4% (scale 0.0 to 100.0), 95% confidence interval 16.6 to 26.221.4%(16.6–26.2)- Mean score
- 21.4%
- 38AlibabaDossier
Accuracy (best score across scorers)
Qwen3.5 Plus, Accuracy (best score across scorers): 21% (scale 0.0 to 100.0), 95% confidence interval 16.3 to 25.721%(16.3–25.7)- Mean score
- 21%
- 39AnthropicDossier
Accuracy (best score across scorers)
Opus 4.5, Accuracy (best score across scorers): 20.7% (scale 0.0 to 100.0), 95% confidence interval 16.0 to 25.420.7%(16.0–25.4)- Mean score
- 20.7%
- 39AnthropicDossier
32,000 budget
Accuracy (best score across scorers)
Opus 4.5, Accuracy (best score across scorers): 20.7% (scale 0.0 to 100.0), 95% confidence interval 16.0 to 25.420.7%(16.0–25.4)- Mean score
- 20.7%
- 41OpenAIDossier
medium
Accuracy (best score across scorers)
GPT-5 mini, Accuracy (best score across scorers): 20.3% (scale 0.0 to 100.0), 95% confidence interval 15.7 to 25.020.3%(15.7–25.0)- Mean score
- 20.3%
- 41AnthropicDossier
16,000 budget
Accuracy (best score across scorers)
Opus 4.5, Accuracy (best score across scorers): 20.3% (scale 0.0 to 100.0), 95% confidence interval 15.7 to 25.020.3%(15.7–25.0)- Mean score
- 20.3%
- 43xAIDossier
Accuracy (best score across scorers)
Grok 4 (0709), Accuracy (best score across scorers): 19.7% (scale 0.0 to 100.0), 95% confidence interval 15.1 to 24.219.7%(15.1–24.2)- Mean score
- 19.7%
- 44OpenAIDossier
medium
Accuracy (best score across scorers)
o4-mini, Accuracy (best score across scorers): 19% (scale 0.0 to 100.0), 95% confidence interval 14.4 to 23.519%(14.4–23.5)- Mean score
- 19%
- 45OpenAIDossier
high
Accuracy (best score across scorers)
o3, Accuracy (best score across scorers): 18.7% (scale 0.0 to 100.0), 95% confidence interval 14.2 to 23.218.7%(14.2–23.2)- Mean score
- 18.7%
- 46OpenAIDossier
low
Accuracy (best score across scorers)
GPT-5.1, Accuracy (best score across scorers): 17.3% (scale 0.0 to 100.0), 95% confidence interval 12.9 to 21.717.3%(12.9–21.7)- Mean score
- 17.3%
- 47OpenAIDossier
medium
Accuracy (best score across scorers)
o3, Accuracy (best score across scorers): 16.9% (scale 0.0 to 100.0), 95% confidence interval 12.6 to 21.216.9%(12.6–21.2)- Mean score
- 16.9%
- 48Z.aiDossier
Accuracy (best score across scorers)
GLM-5, Accuracy (best score across scorers): 16.4% (scale 0.0 to 100.0), 95% confidence interval 12.1 to 20.716.4%(12.1–20.7)- Mean score
- 16.4%
- 49AnthropicDossier
32,000 budget
Accuracy (best score across scorers)
Sonnet 4.5, Accuracy (best score across scorers): 15.2% (scale 0.0 to 100.0), 95% confidence interval 11.1 to 19.415.2%(11.1–19.4)- Mean score
- 15.2%
- 50GoogleDossier
Accuracy (best score across scorers)
Gemini 2.5 Pro, Accuracy (best score across scorers): 14.1% (scale 0.0 to 100.0), 95% confidence interval 10.1 to 18.214.1%(10.1–18.2)- Mean score
- 14.1%
- 51AnthropicDossier
59,000 budget
Accuracy (best score across scorers)
Sonnet 4.5, Accuracy (best score across scorers): 13.5% (scale 0.0 to 100.0), 95% confidence interval 9.5 to 17.413.5%(9.5–17.4)- Mean score
- 13.5%
- 52OpenAIDossier
high
Accuracy (best score across scorers)
o3-mini, Accuracy (best score across scorers): 12.4% (scale 0.0 to 100.0), 95% confidence interval 8.6 to 16.212.4%(8.6–16.2)- Mean score
- 12.4%
- 53OpenAIDossier
low
Accuracy (best score across scorers)
o4-mini, Accuracy (best score across scorers): 10.7% (scale 0.0 to 100.0), 95% confidence interval 7.1 to 14.310.7%(7.1–14.3)- Mean score
- 10.7%
- 54GoogleDossier
Accuracy (best score across scorers)
Gemini 2.5 Pro Preview (06-05), Accuracy (best score across scorers): 10.3% (scale 0.0 to 100.0), 95% confidence interval 6.8 to 13.910.3%(6.8–13.9)- Mean score
- 10.3%
- 54AlibabaDossier
Accuracy (best score across scorers)
Qwen3.6 Flash, Accuracy (best score across scorers): 10.3% (scale 0.0 to 100.0), 95% confidence interval 6.8 to 13.910.3%(6.8–13.9)- Mean score
- 10.3%
- 56OpenAIDossier
low
Accuracy (best score across scorers)
o3, Accuracy (best score across scorers): 9.7% (scale 0.0 to 100.0), 95% confidence interval 6.3 to 13.19.7%(6.3–13.1)- Mean score
- 9.7%
- 57OpenAIDossier
high
Accuracy (best score across scorers)
o1, Accuracy (best score across scorers): 9.3% (scale 0.0 to 100.0), 95% confidence interval 6.0 to 12.79.3%(6.0–12.7)- Mean score
- 9.3%
- 57AnthropicDossier
Accuracy (best score across scorers)
Sonnet 4.5, Accuracy (best score across scorers): 9.3% (scale 0.0 to 100.0), 95% confidence interval 6.0 to 12.79.3%(6.0–12.7)- Mean score
- 9.3%
- 59AlibabaDossier
Accuracy (best score across scorers)
Qwen3 235B A22B Thinking (2507), Accuracy (best score across scorers): 8.5% (scale 0.0 to 100.0), 95% confidence interval 5.2 to 11.78.5%(5.2–11.7)- Mean score
- 8.5%
- 60OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5 nano, Accuracy (best score across scorers): 8.3% (scale 0.0 to 100.0), 95% confidence interval 5.1 to 11.58.3%(5.1–11.5)- Mean score
- 8.3%
- 61OpenAIDossier
medium
Accuracy (best score across scorers)
o3-mini, Accuracy (best score across scorers): 8.1% (scale 0.0 to 100.0), 95% confidence interval 5.6 to 10.68.1%(5.6–10.6)- Mean score
- 8.1%
- 62OpenAIDossier
medium
Accuracy (best score across scorers)
GPT-5 nano, Accuracy (best score across scorers): 7.2% (scale 0.0 to 100.0), 95% confidence interval 4.3 to 10.27.2%(4.3–10.2)- Mean score
- 7.2%
- 62AnthropicDossier
27,000 budget
Accuracy (best score across scorers)
Opus 4.1, Accuracy (best score across scorers): 7.2% (scale 0.0 to 100.0), 95% confidence interval 4.3 to 10.27.2%(4.3–10.2)- Mean score
- 7.2%
- 64AlibabaDossier
Accuracy (best score across scorers)
Qwen3.5 Flash, Accuracy (best score across scorers): 6.2% (scale 0.0 to 100.0), 95% confidence interval 3.4 to 9.06.2%(3.4–9.0)- Mean score
- 6.2%
- 65AnthropicDossier
32,000 budget
Accuracy (best score across scorers)
Haiku 4.5, Accuracy (best score across scorers): 5.9% (scale 0.0 to 100.0), 95% confidence interval 3.2 to 8.65.9%(3.2–8.6)- Mean score
- 5.9%
- 66xAIDossier
high
Accuracy (best score across scorers)
Grok 3 Mini Beta, Accuracy (best score across scorers): 5.9% (scale 0.0 to 100.0), 95% confidence interval 3.2 to 8.65.9%(3.2–8.6)- Mean score
- 5.9%
- 66AnthropicDossier
Accuracy (best score across scorers)
Opus 4.1, Accuracy (best score across scorers): 5.9% (scale 0.0 to 100.0), 95% confidence interval 3.2 to 8.65.9%(3.2–8.6)- Mean score
- 5.9%
- 68OpenAIDossier
Accuracy (best score across scorers)
GPT-4.1, Accuracy (best score across scorers): 5.5% (scale 0.0 to 100.0), 95% confidence interval 2.9 to 8.15.5%(2.9–8.1)- Mean score
- 5.5%
- 69GoogleDossier
Accuracy (best score across scorers)
Gemini 2.5 Flash, Accuracy (best score across scorers): 4.8% (scale 0.0 to 100.0), 95% confidence interval 2.4 to 7.34.8%(2.4–7.3)- Mean score
- 4.8%
- 70OpenAIDossier
Accuracy (best score across scorers)
GPT-4.1 mini, Accuracy (best score across scorers): 4.5% (scale 0.0 to 100.0), 95% confidence interval 2.1 to 6.94.5%(2.1–6.9)- Mean score
- 4.5%
- 70AnthropicDossier
Accuracy (best score across scorers)
Opus 4, Accuracy (best score across scorers): 4.5% (scale 0.0 to 100.0), 95% confidence interval 2.1 to 6.94.5%(2.1–6.9)- Mean score
- 4.5%
- 72AnthropicDossier
16,000 budget
Accuracy (best score across scorers)
Claude 3.7 Sonnet, Accuracy (best score across scorers): 4.1% (scale 0.0 to 100.0), 95% confidence interval 1.8 to 6.44.1%(1.8–6.4)- Mean score
- 4.1%
- 72AnthropicDossier
Accuracy (best score across scorers)
Claude Sonnet 4, Accuracy (best score across scorers): 4.1% (scale 0.0 to 100.0), 95% confidence interval 1.8 to 6.44.1%(1.8–6.4)- Mean score
- 4.1%
- 72AnthropicDossier
Accuracy (best score across scorers)
Haiku 4.5, Accuracy (best score across scorers): 4.1% (scale 0.0 to 100.0), 95% confidence interval 1.8 to 6.44.1%(1.8–6.4)- Mean score
- 4.1%
- 72AnthropicDossier
27,000 budget
Accuracy (best score across scorers)
Opus 4, Accuracy (best score across scorers): 4.1% (scale 0.0 to 100.0), 95% confidence interval 1.8 to 6.44.1%(1.8–6.4)- Mean score
- 4.1%
- 76Z.aiDossier
Accuracy (best score across scorers)
GLM-4.6, Accuracy (best score across scorers): 3.8% (scale 0.0 to 100.0), 95% confidence interval 1.6 to 6.03.8%(1.6–6.0)- Mean score
- 3.8%
- 77xAIDossier
Accuracy (best score across scorers)
Grok 3 Beta, Accuracy (best score across scorers): 3.8% (scale 0.0 to 100.0), 95% confidence interval 1.6 to 6.03.8%(1.6–6.0)- Mean score
- 3.8%
- 78AnthropicDossier
32,000 budget
Accuracy (best score across scorers)
Claude 3.7 Sonnet, Accuracy (best score across scorers): 3.4% (scale 0.0 to 100.0), 95% confidence interval 1.3 to 5.63.4%(1.3–5.6)- Mean score
- 3.4%
- 79AnthropicDossier
64,000 budget
Accuracy (best score across scorers)
Claude 3.7 Sonnet, Accuracy (best score across scorers): 3.1% (scale 0.0 to 100.0), 95% confidence interval 1.1 to 5.13.1%(1.1–5.1)- Mean score
- 3.1%
- 79AnthropicDossier
Accuracy (best score across scorers)
Claude 3.7 Sonnet, Accuracy (best score across scorers): 3.1% (scale 0.0 to 100.0), 95% confidence interval 1.1 to 5.13.1%(1.1–5.1)- Mean score
- 3.1%
- 81xAIDossier
low
Accuracy (best score across scorers)
Grok 3 Mini Beta, Accuracy (best score across scorers): 2.8% (scale 0.0 to 100.0), 95% confidence interval 0.9 to 4.62.8%(0.9–4.6)- Mean score
- 2.8%
- 82Z.aiDossier
Accuracy (best score across scorers)
GLM-4.7, Accuracy (best score across scorers): 2.4% (scale 0.0 to 100.0), 95% confidence interval 0.7 to 4.22.4%(0.7–4.2)- Mean score
- 2.4%
- 83OpenAIDossier
none
Accuracy (best score across scorers)
GPT-5.1, Accuracy (best score across scorers): 2.1% (scale 0.0 to 100.0), 95% confidence interval 0.4 to 3.82.1%(0.4–3.8)- Mean score
- 2.1%
- 84AnthropicDossier
Accuracy (best score across scorers)
Claude 3.5 Sonnet (Oct '24), Accuracy (best score across scorers): 2.1% (scale 0.0 to 100.0), 95% confidence interval 0.4 to 3.72.1%(0.4–3.7)- Mean score
- 2.1%
- 85DeepSeekDossier
Accuracy (best score across scorers)
DeepSeek V3, Accuracy (best score across scorers): 1.7% (scale 0.0 to 100.0), 95% confidence interval 0.2 to 3.21.7%(0.2–3.2)- Mean score
- 1.7%
- 85GoogleDossier
Accuracy (best score across scorers)
Gemini 2.0 Flash (001), Accuracy (best score across scorers): 1.7% (scale 0.0 to 100.0), 95% confidence interval 0.2 to 3.21.7%(0.2–3.2)- Mean score
- 1.7%
- 85OpenAIDossier
medium
Accuracy (best score across scorers)
o1-mini, Accuracy (best score across scorers): 1.7% (scale 0.0 to 100.0), 95% confidence interval 0.2 to 3.21.7%(0.2–3.2)- Mean score
- 1.7%
- 85AlibabaDossier
Accuracy (best score across scorers)
Qwen Plus, Accuracy (best score across scorers): 1.7% (scale 0.0 to 100.0), 95% confidence interval 0.2 to 3.21.7%(0.2–3.2)- Mean score
- 1.7%
- 89OpenAIDossier
high
Accuracy (best score across scorers)
o1-mini, Accuracy (best score across scorers): 1.4% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 2.71.4%(0.0–2.7)- Mean score
- 1.4%
- 90AnthropicDossier
Accuracy (best score across scorers)
Claude 3.5 Sonnet (June '24), Accuracy (best score across scorers): 1% (scale 0.0 to 100.0), 95% confidence interval 0.3 to 3.21%(0.3–3.2)- Mean score
- 1%
- 90OpenAIDossier
Accuracy (best score across scorers)
GPT-4.1 nano, Accuracy (best score across scorers): 1% (scale 0.0 to 100.0), 95% confidence interval 0.3 to 3.21%(0.3–3.2)- Mean score
- 1%
- 90AlibabaDossier
Accuracy (best score across scorers)
Qwen Max, Accuracy (best score across scorers): 1% (scale 0.0 to 100.0), 95% confidence interval 0.3 to 3.21%(0.3–3.2)- Mean score
- 1%
- 93xAIDossier
Accuracy (best score across scorers)
Grok 2 (1212), Accuracy (best score across scorers): 0.7% (scale 0.0 to 100.0), 95% confidence interval 0.2 to 2.70.7%(0.2–2.7)- Mean score
- 0.7%
- 93MetaDossier
Accuracy (best score across scorers)
Llama 4 Maverick 17B 128E Instruct (FP8), Accuracy (best score across scorers): 0.7% (scale 0.0 to 100.0), 95% confidence interval 0.2 to 2.70.7%(0.2–2.7)- Mean score
- 0.7%
- 95MistralDossier
Accuracy (best score across scorers)
Mistral Medium 3, Accuracy (best score across scorers): 0.3% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 2.40.3%(0.0–2.4)- Mean score
- 0.3%
- 96AnthropicDossier
Accuracy (best score across scorers)
Claude 3.5 Haiku, Accuracy (best score across scorers): 0.3% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 2.40.3%(0.0–2.4)- Mean score
- 0.3%
- 96OpenAIDossier
Accuracy (best score across scorers)
GPT-4o (Aug '24), Accuracy (best score across scorers): 0.3% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 2.40.3%(0.0–2.4)- Mean score
- 0.3%
- 96OpenAIDossier
Accuracy (best score across scorers)
GPT-4o (Nov '24), Accuracy (best score across scorers): 0.3% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 2.40.3%(0.0–2.4)- Mean score
- 0.3%
- 96MistralDossier
Accuracy (best score across scorers)
Mistral Large (2411), Accuracy (best score across scorers): 0.3% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 2.40.3%(0.0–2.4)- Mean score
- 0.3%
- 100GoogleDossier
Accuracy (best score across scorers)
Gemini 1.5 Flash (002), Accuracy (best score across scorers): 0% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 0.00%(0.0–0.0)- Mean score
- 0%
- 100MetaDossier
Accuracy (best score across scorers)
Llama 4 Scout, Accuracy (best score across scorers): 0% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 0.00%(0.0–0.0)- Mean score
- 0%