Acropolis

FrontierMath-2025-02-28-Private

FrontierMath, 2025-02-28 private set, run by Epoch AI. Superseded in benchmark_metadata by FrontierMath-Tiers-1-3-v2-Private.

Upstream source
Math
Independent. Run by an operator independent of the model vendors.Epoch AI Benchmarking Hub by Epoch AIVersion 2025-02-28-PrivateData as of 17 Nov 2026

No filters. Showing 101 of 101 rows.

  1. 1
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    GPT-5.5 Pro Pre-release, Accuracy (best score across scorers): 52.4% (scale 0.0 to 100.0), 95% confidence interval 46.7 to 58.152.4%(46.7–58.1)
    Mean score
    52.4%
  2. 2
    OpenAIDossier

    xhigh

    Tied #1

    Accuracy (best score across scorers)

    GPT-5.5 Pre-release, Accuracy (best score across scorers): 51.7% (scale 0.0 to 100.0), 95% confidence interval 46.0 to 57.451.7%(46.0–57.4)
    Mean score
    51.7%
  3. 3
    OpenAIDossier

    xhigh

    Tied #1

    Accuracy (best score across scorers)

    GPT-5.5 Pro Pre-release, Accuracy (best score across scorers): 51% (scale 0.0 to 100.0), 95% confidence interval 45.3 to 56.751%(45.3–56.7)
    Mean score
    51%
  4. 4
    OpenAIDossier

    xhigh

    Tied #1

    Accuracy (best score across scorers)

    GPT-5.4 Pro, Accuracy (best score across scorers): 50% (scale 0.0 to 100.0), 95% confidence interval 44.3 to 55.750%(44.3–55.7)
    Mean score
    50%
  5. 5
    OpenAIDossier

    xhigh

    Tied #1

    Accuracy (best score across scorers)

    GPT-5.4, Accuracy (best score across scorers): 47.6% (scale 0.0 to 100.0), 95% confidence interval 41.9 to 53.347.6%(41.9–53.3)
    Mean score
    47.6%
  6. 6
    AnthropicDossier

    max

    Tied #1

    Accuracy (best score across scorers)

    Opus 4.8, Accuracy (best score across scorers): 47.2% (scale 0.0 to 100.0), 95% confidence interval 41.5 to 53.047.2%(41.5–53.0)
    Mean score
    47.2%
  7. 7
    AnthropicDossier

    xhigh

    Accuracy (best score across scorers)

    Opus 4.7, Accuracy (best score across scorers): 43.8% (scale 0.0 to 100.0), 95% confidence interval 38.1 to 49.543.8%(38.1–49.5)
    Mean score
    43.8%
  8. 8
    OpenAIDossier

    xhigh

    Accuracy (best score across scorers)

    GPT-5.2, Accuracy (best score across scorers): 40.7% (scale 0.0 to 100.0), 95% confidence interval 35.0 to 46.440.7%(35.0–46.4)
    Mean score
    40.7%
  9. 8
    AnthropicDossier

    max

    Accuracy (best score across scorers)

    Opus 4.6, Accuracy (best score across scorers): 40.7% (scale 0.0 to 100.0), 95% confidence interval 35.0 to 46.440.7%(35.0–46.4)
    Mean score
    40.7%
  10. 10
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    GPT-5.2, Accuracy (best score across scorers): 40.3% (scale 0.0 to 100.0), 95% confidence interval 34.7 to 46.040.3%(34.7–46.0)
    Mean score
    40.3%
  11. 11
    AnthropicDossier

    32,000 budget

    Accuracy (best score across scorers)

    Opus 4.6, Accuracy (best score across scorers): 40% (scale 0.0 to 100.0), 95% confidence interval 34.4 to 45.640%(34.4–45.6)
    Mean score
    40%
  12. 12
    AnthropicDossier

    64,000 budget

    Accuracy (best score across scorers)

    Opus 4.6, Accuracy (best score across scorers): 39.7% (scale 0.0 to 100.0), 95% confidence interval 34.0 to 45.339.7%(34.0–45.3)
    Mean score
    39.7%
  13. 13

    Accuracy (best score across scorers)

    Muse Spark, Accuracy (best score across scorers): 39% (scale 0.0 to 100.0), 95% confidence interval 33.3 to 44.739%(33.3–44.7)
    Mean score
    39%
  14. 14
    GoogleDossier

    high

    Accuracy (best score across scorers)

    Gemini 3.5 Flash, Accuracy (best score across scorers): 39% (scale 0.0 to 100.0), 95% confidence interval 33.3 to 44.639%(33.3–44.6)
    Mean score
    39%
  15. 14
    Moonshot AIDossier

    Accuracy (best score across scorers)

    Kimi K2.6, Accuracy (best score across scorers): 39% (scale 0.0 to 100.0), 95% confidence interval 33.3 to 44.639%(33.3–44.6)
    Mean score
    39%
  16. 16
    AnthropicDossier

    Accuracy (best score across scorers)

    Opus 4.6, Accuracy (best score across scorers): 38.3% (scale 0.0 to 100.0), 95% confidence interval 32.7 to 43.938.3%(32.7–43.9)
    Mean score
    38.3%
  17. 17
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 3 Pro Preview, Accuracy (best score across scorers): 37.6% (scale 0.0 to 100.0), 95% confidence interval 32.0 to 43.237.6%(32.0–43.2)
    Mean score
    37.6%
  18. 18
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 3.1 Pro Preview, Accuracy (best score across scorers): 36.9% (scale 0.0 to 100.0), 95% confidence interval 31.4 to 42.436.9%(31.4–42.4)
    Mean score
    36.9%
  19. 19
    OpenAIDossier

    medium

    Accuracy (best score across scorers)

    GPT-5.2, Accuracy (best score across scorers): 36.9% (scale 0.0 to 100.0), 95% confidence interval 31.3 to 42.536.9%(31.3–42.5)
    Mean score
    36.9%
  20. 20
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 3 Flash Preview, Accuracy (best score across scorers): 35.6% (scale 0.0 to 100.0), 95% confidence interval 30.1 to 41.235.6%(30.1–41.2)
    Mean score
    35.6%
  21. 21

    Accuracy (best score across scorers)

    GLM-5.1, Accuracy (best score across scorers): 33.4% (scale 0.0 to 100.0), 95% confidence interval 28.0 to 38.933.4%(28.0–38.9)
    Mean score
    33.4%
  22. 22
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    GPT-5, Accuracy (best score across scorers): 32.4% (scale 0.0 to 100.0), 95% confidence interval 27.0 to 37.832.4%(27.0–37.8)
    Mean score
    32.4%
  23. 23
    AnthropicDossier

    16,000 budget

    Accuracy (best score across scorers)

    Sonnet 4.6, Accuracy (best score across scorers): 32.4% (scale 0.0 to 100.0), 95% confidence interval 26.9 to 37.932.4%(26.9–37.9)
    Mean score
    32.4%
  24. 24
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    GPT-5.1, Accuracy (best score across scorers): 31% (scale 0.0 to 100.0), 95% confidence interval 25.7 to 36.431%(25.7–36.4)
    Mean score
    31%
  25. 25
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 2.5 Deep Think, Accuracy (best score across scorers): 29% (scale 0.0 to 100.0), 95% confidence interval 23.7 to 34.329%(23.7–34.3)
    Mean score
    29%
  26. 26
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    GPT-5.4 mini, Accuracy (best score across scorers): 28.3% (scale 0.0 to 100.0), 95% confidence interval 24.2 to 32.328.3%(24.2–32.3)
    Mean score
    28.3%
  27. 27
    Moonshot AIDossier

    Accuracy (best score across scorers)

    Kimi K2.5, Accuracy (best score across scorers): 27.9% (scale 0.0 to 100.0), 95% confidence interval 22.8 to 33.027.9%(22.8–33.0)
    Mean score
    27.9%
  28. 28
    OpenAIDossier

    medium

    Accuracy (best score across scorers)

    GPT-5, Accuracy (best score across scorers): 27.2% (scale 0.0 to 100.0), 95% confidence interval 22.1 to 32.427.2%(22.1–32.4)
    Mean score
    27.2%
  29. 28
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    GPT-5 mini, Accuracy (best score across scorers): 27.2% (scale 0.0 to 100.0), 95% confidence interval 22.1 to 32.427.2%(22.1–32.4)
    Mean score
    27.2%
  30. 30
    OpenAIDossier

    medium

    Accuracy (best score across scorers)

    GPT-5.1, Accuracy (best score across scorers): 26.9% (scale 0.0 to 100.0), 95% confidence interval 21.8 to 32.026.9%(21.8–32.0)
    Mean score
    26.9%
  31. 31
    OpenAIDossier

    low

    Accuracy (best score across scorers)

    GPT-5.2, Accuracy (best score across scorers): 26.6% (scale 0.0 to 100.0), 95% confidence interval 21.5 to 31.626.6%(21.5–31.6)
    Mean score
    26.6%
  32. 32
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3.6 Plus, Accuracy (best score across scorers): 26.2% (scale 0.0 to 100.0), 95% confidence interval 21.1 to 31.326.2%(21.1–31.3)
    Mean score
    26.2%
  33. 33
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    GPT-5.4 nano, Accuracy (best score across scorers): 25.9% (scale 0.0 to 100.0), 95% confidence interval 20.8 to 30.925.9%(20.8–30.9)
    Mean score
    25.9%
  34. 34
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    o4-mini, Accuracy (best score across scorers): 24.8% (scale 0.0 to 100.0), 95% confidence interval 19.8 to 29.824.8%(19.8–29.8)
    Mean score
    24.8%
  35. 35
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3.6 Max Preview, Accuracy (best score across scorers): 23.1% (scale 0.0 to 100.0), 95% confidence interval 18.2 to 28.023.1%(18.2–28.0)
    Mean score
    23.1%
  36. 36
    DeepSeekDossier

    Accuracy (best score across scorers)

    DeepSeek V3.2, Accuracy (best score across scorers): 22.1% (scale 0.0 to 100.0), 95% confidence interval 14.1 to 30.122.1%(14.1–30.1)
    Mean score
    22.1%
  37. 37
    Moonshot AIDossier

    Accuracy (best score across scorers)

    Kimi K2 Thinking, Accuracy (best score across scorers): 21.4% (scale 0.0 to 100.0), 95% confidence interval 16.6 to 26.221.4%(16.6–26.2)
    Mean score
    21.4%
  38. 38
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3.5 Plus, Accuracy (best score across scorers): 21% (scale 0.0 to 100.0), 95% confidence interval 16.3 to 25.721%(16.3–25.7)
    Mean score
    21%
  39. 39
    AnthropicDossier

    Accuracy (best score across scorers)

    Opus 4.5, Accuracy (best score across scorers): 20.7% (scale 0.0 to 100.0), 95% confidence interval 16.0 to 25.420.7%(16.0–25.4)
    Mean score
    20.7%
  40. 39
    AnthropicDossier

    32,000 budget

    Accuracy (best score across scorers)

    Opus 4.5, Accuracy (best score across scorers): 20.7% (scale 0.0 to 100.0), 95% confidence interval 16.0 to 25.420.7%(16.0–25.4)
    Mean score
    20.7%
  41. 41
    OpenAIDossier

    medium

    Accuracy (best score across scorers)

    GPT-5 mini, Accuracy (best score across scorers): 20.3% (scale 0.0 to 100.0), 95% confidence interval 15.7 to 25.020.3%(15.7–25.0)
    Mean score
    20.3%
  42. 41
    AnthropicDossier

    16,000 budget

    Accuracy (best score across scorers)

    Opus 4.5, Accuracy (best score across scorers): 20.3% (scale 0.0 to 100.0), 95% confidence interval 15.7 to 25.020.3%(15.7–25.0)
    Mean score
    20.3%
  43. 43

    Accuracy (best score across scorers)

    Grok 4 (0709), Accuracy (best score across scorers): 19.7% (scale 0.0 to 100.0), 95% confidence interval 15.1 to 24.219.7%(15.1–24.2)
    Mean score
    19.7%
  44. 44
    OpenAIDossier

    medium

    Accuracy (best score across scorers)

    o4-mini, Accuracy (best score across scorers): 19% (scale 0.0 to 100.0), 95% confidence interval 14.4 to 23.519%(14.4–23.5)
    Mean score
    19%
  45. 45
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    o3, Accuracy (best score across scorers): 18.7% (scale 0.0 to 100.0), 95% confidence interval 14.2 to 23.218.7%(14.2–23.2)
    Mean score
    18.7%
  46. 46
    OpenAIDossier

    low

    Accuracy (best score across scorers)

    GPT-5.1, Accuracy (best score across scorers): 17.3% (scale 0.0 to 100.0), 95% confidence interval 12.9 to 21.717.3%(12.9–21.7)
    Mean score
    17.3%
  47. 47
    OpenAIDossier

    medium

    Accuracy (best score across scorers)

    o3, Accuracy (best score across scorers): 16.9% (scale 0.0 to 100.0), 95% confidence interval 12.6 to 21.216.9%(12.6–21.2)
    Mean score
    16.9%
  48. 48

    Accuracy (best score across scorers)

    GLM-5, Accuracy (best score across scorers): 16.4% (scale 0.0 to 100.0), 95% confidence interval 12.1 to 20.716.4%(12.1–20.7)
    Mean score
    16.4%
  49. 49
    AnthropicDossier

    32,000 budget

    Accuracy (best score across scorers)

    Sonnet 4.5, Accuracy (best score across scorers): 15.2% (scale 0.0 to 100.0), 95% confidence interval 11.1 to 19.415.2%(11.1–19.4)
    Mean score
    15.2%
  50. 50
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 2.5 Pro, Accuracy (best score across scorers): 14.1% (scale 0.0 to 100.0), 95% confidence interval 10.1 to 18.214.1%(10.1–18.2)
    Mean score
    14.1%
  51. 51
    AnthropicDossier

    59,000 budget

    Accuracy (best score across scorers)

    Sonnet 4.5, Accuracy (best score across scorers): 13.5% (scale 0.0 to 100.0), 95% confidence interval 9.5 to 17.413.5%(9.5–17.4)
    Mean score
    13.5%
  52. 52
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    o3-mini, Accuracy (best score across scorers): 12.4% (scale 0.0 to 100.0), 95% confidence interval 8.6 to 16.212.4%(8.6–16.2)
    Mean score
    12.4%
  53. 53
    OpenAIDossier

    low

    Accuracy (best score across scorers)

    o4-mini, Accuracy (best score across scorers): 10.7% (scale 0.0 to 100.0), 95% confidence interval 7.1 to 14.310.7%(7.1–14.3)
    Mean score
    10.7%
  54. 54
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 2.5 Pro Preview (06-05), Accuracy (best score across scorers): 10.3% (scale 0.0 to 100.0), 95% confidence interval 6.8 to 13.910.3%(6.8–13.9)
    Mean score
    10.3%
  55. 54
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3.6 Flash, Accuracy (best score across scorers): 10.3% (scale 0.0 to 100.0), 95% confidence interval 6.8 to 13.910.3%(6.8–13.9)
    Mean score
    10.3%
  56. 56
    OpenAIDossier

    low

    Accuracy (best score across scorers)

    o3, Accuracy (best score across scorers): 9.7% (scale 0.0 to 100.0), 95% confidence interval 6.3 to 13.19.7%(6.3–13.1)
    Mean score
    9.7%
  57. 57
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    o1, Accuracy (best score across scorers): 9.3% (scale 0.0 to 100.0), 95% confidence interval 6.0 to 12.79.3%(6.0–12.7)
    Mean score
    9.3%
  58. 57
    AnthropicDossier

    Accuracy (best score across scorers)

    Sonnet 4.5, Accuracy (best score across scorers): 9.3% (scale 0.0 to 100.0), 95% confidence interval 6.0 to 12.79.3%(6.0–12.7)
    Mean score
    9.3%
  59. 59
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3 235B A22B Thinking (2507), Accuracy (best score across scorers): 8.5% (scale 0.0 to 100.0), 95% confidence interval 5.2 to 11.78.5%(5.2–11.7)
    Mean score
    8.5%
  60. 60
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    GPT-5 nano, Accuracy (best score across scorers): 8.3% (scale 0.0 to 100.0), 95% confidence interval 5.1 to 11.58.3%(5.1–11.5)
    Mean score
    8.3%
  61. 61
    OpenAIDossier

    medium

    Accuracy (best score across scorers)

    o3-mini, Accuracy (best score across scorers): 8.1% (scale 0.0 to 100.0), 95% confidence interval 5.6 to 10.68.1%(5.6–10.6)
    Mean score
    8.1%
  62. 62
    OpenAIDossier

    medium

    Accuracy (best score across scorers)

    GPT-5 nano, Accuracy (best score across scorers): 7.2% (scale 0.0 to 100.0), 95% confidence interval 4.3 to 10.27.2%(4.3–10.2)
    Mean score
    7.2%
  63. 62
    AnthropicDossier

    27,000 budget

    Accuracy (best score across scorers)

    Opus 4.1, Accuracy (best score across scorers): 7.2% (scale 0.0 to 100.0), 95% confidence interval 4.3 to 10.27.2%(4.3–10.2)
    Mean score
    7.2%
  64. 64
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3.5 Flash, Accuracy (best score across scorers): 6.2% (scale 0.0 to 100.0), 95% confidence interval 3.4 to 9.06.2%(3.4–9.0)
    Mean score
    6.2%
  65. 65
    AnthropicDossier

    32,000 budget

    Accuracy (best score across scorers)

    Haiku 4.5, Accuracy (best score across scorers): 5.9% (scale 0.0 to 100.0), 95% confidence interval 3.2 to 8.65.9%(3.2–8.6)
    Mean score
    5.9%
  66. 66

    high

    Accuracy (best score across scorers)

    Grok 3 Mini Beta, Accuracy (best score across scorers): 5.9% (scale 0.0 to 100.0), 95% confidence interval 3.2 to 8.65.9%(3.2–8.6)
    Mean score
    5.9%
  67. 66
    AnthropicDossier

    Accuracy (best score across scorers)

    Opus 4.1, Accuracy (best score across scorers): 5.9% (scale 0.0 to 100.0), 95% confidence interval 3.2 to 8.65.9%(3.2–8.6)
    Mean score
    5.9%
  68. 68
    OpenAIDossier

    Accuracy (best score across scorers)

    GPT-4.1, Accuracy (best score across scorers): 5.5% (scale 0.0 to 100.0), 95% confidence interval 2.9 to 8.15.5%(2.9–8.1)
    Mean score
    5.5%
  69. 69
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 2.5 Flash, Accuracy (best score across scorers): 4.8% (scale 0.0 to 100.0), 95% confidence interval 2.4 to 7.34.8%(2.4–7.3)
    Mean score
    4.8%
  70. 70
    OpenAIDossier

    Accuracy (best score across scorers)

    GPT-4.1 mini, Accuracy (best score across scorers): 4.5% (scale 0.0 to 100.0), 95% confidence interval 2.1 to 6.94.5%(2.1–6.9)
    Mean score
    4.5%
  71. 70
    AnthropicDossier

    Accuracy (best score across scorers)

    Opus 4, Accuracy (best score across scorers): 4.5% (scale 0.0 to 100.0), 95% confidence interval 2.1 to 6.94.5%(2.1–6.9)
    Mean score
    4.5%
  72. 72
    AnthropicDossier

    16,000 budget

    Accuracy (best score across scorers)

    Claude 3.7 Sonnet, Accuracy (best score across scorers): 4.1% (scale 0.0 to 100.0), 95% confidence interval 1.8 to 6.44.1%(1.8–6.4)
    Mean score
    4.1%
  73. 72
    AnthropicDossier

    Accuracy (best score across scorers)

    Claude Sonnet 4, Accuracy (best score across scorers): 4.1% (scale 0.0 to 100.0), 95% confidence interval 1.8 to 6.44.1%(1.8–6.4)
    Mean score
    4.1%
  74. 72
    AnthropicDossier

    Accuracy (best score across scorers)

    Haiku 4.5, Accuracy (best score across scorers): 4.1% (scale 0.0 to 100.0), 95% confidence interval 1.8 to 6.44.1%(1.8–6.4)
    Mean score
    4.1%
  75. 72
    AnthropicDossier

    27,000 budget

    Accuracy (best score across scorers)

    Opus 4, Accuracy (best score across scorers): 4.1% (scale 0.0 to 100.0), 95% confidence interval 1.8 to 6.44.1%(1.8–6.4)
    Mean score
    4.1%
  76. 76

    Accuracy (best score across scorers)

    GLM-4.6, Accuracy (best score across scorers): 3.8% (scale 0.0 to 100.0), 95% confidence interval 1.6 to 6.03.8%(1.6–6.0)
    Mean score
    3.8%
  77. 77

    Accuracy (best score across scorers)

    Grok 3 Beta, Accuracy (best score across scorers): 3.8% (scale 0.0 to 100.0), 95% confidence interval 1.6 to 6.03.8%(1.6–6.0)
    Mean score
    3.8%
  78. 78
    AnthropicDossier

    32,000 budget

    Accuracy (best score across scorers)

    Claude 3.7 Sonnet, Accuracy (best score across scorers): 3.4% (scale 0.0 to 100.0), 95% confidence interval 1.3 to 5.63.4%(1.3–5.6)
    Mean score
    3.4%
  79. 79
    AnthropicDossier

    64,000 budget

    Accuracy (best score across scorers)

    Claude 3.7 Sonnet, Accuracy (best score across scorers): 3.1% (scale 0.0 to 100.0), 95% confidence interval 1.1 to 5.13.1%(1.1–5.1)
    Mean score
    3.1%
  80. 79
    AnthropicDossier

    Accuracy (best score across scorers)

    Claude 3.7 Sonnet, Accuracy (best score across scorers): 3.1% (scale 0.0 to 100.0), 95% confidence interval 1.1 to 5.13.1%(1.1–5.1)
    Mean score
    3.1%
  81. 81

    low

    Accuracy (best score across scorers)

    Grok 3 Mini Beta, Accuracy (best score across scorers): 2.8% (scale 0.0 to 100.0), 95% confidence interval 0.9 to 4.62.8%(0.9–4.6)
    Mean score
    2.8%
  82. 82

    Accuracy (best score across scorers)

    GLM-4.7, Accuracy (best score across scorers): 2.4% (scale 0.0 to 100.0), 95% confidence interval 0.7 to 4.22.4%(0.7–4.2)
    Mean score
    2.4%
  83. 83
    OpenAIDossier

    none

    Accuracy (best score across scorers)

    GPT-5.1, Accuracy (best score across scorers): 2.1% (scale 0.0 to 100.0), 95% confidence interval 0.4 to 3.82.1%(0.4–3.8)
    Mean score
    2.1%
  84. 84
    AnthropicDossier

    Accuracy (best score across scorers)

    Claude 3.5 Sonnet (Oct '24), Accuracy (best score across scorers): 2.1% (scale 0.0 to 100.0), 95% confidence interval 0.4 to 3.72.1%(0.4–3.7)
    Mean score
    2.1%
  85. 85
    DeepSeekDossier

    Accuracy (best score across scorers)

    DeepSeek V3, Accuracy (best score across scorers): 1.7% (scale 0.0 to 100.0), 95% confidence interval 0.2 to 3.21.7%(0.2–3.2)
    Mean score
    1.7%
  86. 85
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 2.0 Flash (001), Accuracy (best score across scorers): 1.7% (scale 0.0 to 100.0), 95% confidence interval 0.2 to 3.21.7%(0.2–3.2)
    Mean score
    1.7%
  87. 85
    OpenAIDossier

    medium

    Accuracy (best score across scorers)

    o1-mini, Accuracy (best score across scorers): 1.7% (scale 0.0 to 100.0), 95% confidence interval 0.2 to 3.21.7%(0.2–3.2)
    Mean score
    1.7%
  88. 85
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen Plus, Accuracy (best score across scorers): 1.7% (scale 0.0 to 100.0), 95% confidence interval 0.2 to 3.21.7%(0.2–3.2)
    Mean score
    1.7%
  89. 89
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    o1-mini, Accuracy (best score across scorers): 1.4% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 2.71.4%(0.0–2.7)
    Mean score
    1.4%
  90. 90
    AnthropicDossier

    Accuracy (best score across scorers)

    Claude 3.5 Sonnet (June '24), Accuracy (best score across scorers): 1% (scale 0.0 to 100.0), 95% confidence interval 0.3 to 3.21%(0.3–3.2)
    Mean score
    1%
  91. 90
    OpenAIDossier

    Accuracy (best score across scorers)

    GPT-4.1 nano, Accuracy (best score across scorers): 1% (scale 0.0 to 100.0), 95% confidence interval 0.3 to 3.21%(0.3–3.2)
    Mean score
    1%
  92. 90
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen Max, Accuracy (best score across scorers): 1% (scale 0.0 to 100.0), 95% confidence interval 0.3 to 3.21%(0.3–3.2)
    Mean score
    1%
  93. 93

    Accuracy (best score across scorers)

    Grok 2 (1212), Accuracy (best score across scorers): 0.7% (scale 0.0 to 100.0), 95% confidence interval 0.2 to 2.70.7%(0.2–2.7)
    Mean score
    0.7%
  94. 93

    Accuracy (best score across scorers)

    Llama 4 Maverick 17B 128E Instruct (FP8), Accuracy (best score across scorers): 0.7% (scale 0.0 to 100.0), 95% confidence interval 0.2 to 2.70.7%(0.2–2.7)
    Mean score
    0.7%
  95. 95
    MistralDossier

    Accuracy (best score across scorers)

    Mistral Medium 3, Accuracy (best score across scorers): 0.3% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 2.40.3%(0.0–2.4)
    Mean score
    0.3%
  96. 96
    AnthropicDossier

    Accuracy (best score across scorers)

    Claude 3.5 Haiku, Accuracy (best score across scorers): 0.3% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 2.40.3%(0.0–2.4)
    Mean score
    0.3%
  97. 96
    OpenAIDossier

    Accuracy (best score across scorers)

    GPT-4o (Aug '24), Accuracy (best score across scorers): 0.3% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 2.40.3%(0.0–2.4)
    Mean score
    0.3%
  98. 96
    OpenAIDossier

    Accuracy (best score across scorers)

    GPT-4o (Nov '24), Accuracy (best score across scorers): 0.3% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 2.40.3%(0.0–2.4)
    Mean score
    0.3%
  99. 96
    MistralDossier

    Accuracy (best score across scorers)

    Mistral Large (2411), Accuracy (best score across scorers): 0.3% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 2.40.3%(0.0–2.4)
    Mean score
    0.3%
  100. 100
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 1.5 Flash (002), Accuracy (best score across scorers): 0% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 0.00%(0.0–0.0)
    Mean score
    0%
  101. 100

    Accuracy (best score across scorers)

    Llama 4 Scout, Accuracy (best score across scorers): 0% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 0.00%(0.0–0.0)
    Mean score
    0%