Acropolis

GPQA diamond

GPQA Diamond, run by Epoch AI.

Upstream source
Science
Independent. Run by an operator independent of the model vendors.Epoch AI Benchmarking Hub by Epoch AIVersion 1Data as of 17 Nov 2026

No filters. Showing 338 of 338 rows.

  1. 1
    GoogleDossier

    high

    Accuracy (best score across scorers)

    Gemini 4 Carbon, Accuracy (best score across scorers): 96.5% (scale 0.0 to 100.0), 95% confidence interval 94.2 to 98.996.5%(94.2–98.9)
    Mean score
    96.5%
  2. 2
    GoogleDossier

    high

    Tied #1

    Accuracy (best score across scorers)

    Gemini 4 Argon, Accuracy (best score across scorers): 96.4% (scale 0.0 to 100.0), 95% confidence interval 94.1 to 98.796.4%(94.1–98.7)
    Mean score
    96.4%
  3. 3
    AnthropicDossier

    max · fallback

    Tied #1

    Accuracy (best score across scorers)

    Fable 5.2, Accuracy (best score across scorers): 96% (scale 0.0 to 100.0), 95% confidence interval 93.4 to 98.696%(93.4–98.6)
    Mean score
    96%
  4. 4
    OpenAIDossier

    max

    Tied #1

    Accuracy (best score across scorers)

    GPT-6 Astra, Accuracy (best score across scorers): 95.8% (scale 0.0 to 100.0), 95% confidence interval 93.1 to 98.595.8%(93.1–98.5)
    Mean score
    95.8%
  5. 5
    AnthropicDossier

    max

    Tied #1

    Accuracy (best score across scorers)

    Sonnet 5.5, Accuracy (best score across scorers): 95.6% (scale 0.0 to 100.0), 95% confidence interval 92.9 to 98.395.6%(92.9–98.3)
    Mean score
    95.6%
  6. 6
    GoogleDossier

    high

    Tied #1

    Accuracy (best score across scorers)

    Gemini 3.8 Flash, Accuracy (best score across scorers): 95.4% (scale 0.0 to 100.0), 95% confidence interval 92.6 to 98.195.4%(92.6–98.1)
    Mean score
    95.4%
  7. 6
    OpenAIDossier

    max

    Tied #1

    Accuracy (best score across scorers)

    GPT-6.1 Sol, Accuracy (best score across scorers): 95.4% (scale 0.0 to 100.0), 95% confidence interval 92.7 to 98.195.4%(92.7–98.1)
    Mean score
    95.4%
  8. 8
    GoogleDossier

    high

    Tied #1

    Accuracy (best score across scorers)

    Gemini 3.7 Flash, Accuracy (best score across scorers): 94.8% (scale 0.0 to 100.0), 95% confidence interval 92.2 to 97.594.8%(92.2–97.5)
    Mean score
    94.8%
  9. 9
    OpenAIDossier

    xhigh

    Tied #1

    Accuracy (best score across scorers)

    GPT-5.4 Pro, Accuracy (best score across scorers): 94.6% (scale 0.0 to 100.0), 95% confidence interval 91.5 to 97.794.6%(91.5–97.7)
    Mean score
    94.6%
  10. 10
    GoogleDossier

    high

    Tied #1

    Accuracy (best score across scorers)

    Gemini 3.1 Pro Preview, Accuracy (best score across scorers): 94.4% (scale 0.0 to 100.0), 95% confidence interval 91.2 to 97.694.4%(91.2–97.6)
    Mean score
    94.4%
  11. 11
    OpenAIDossier

    max

    Tied #1

    Accuracy (best score across scorers)

    GPT-6 Sol, Accuracy (best score across scorers): 94.3% (scale 0.0 to 100.0), 95% confidence interval 91.4 to 97.294.3%(91.4–97.2)
    Mean score
    94.3%
  12. 11
    OpenAIDossier

    max

    Tied #1

    Accuracy (best score across scorers)

    GPT-6.1 Terra, Accuracy (best score across scorers): 94.3% (scale 0.0 to 100.0), 95% confidence interval 91.4 to 97.294.3%(91.4–97.2)
    Mean score
    94.3%
  13. 13
    Moonshot AIDossier

    max

    Tied #1

    Accuracy (best score across scorers)

    Kimi K3.1 Pro, Accuracy (best score across scorers): 94.2% (scale 0.0 to 100.0), 95% confidence interval 91.4 to 97.094.2%(91.4–97.0)
    Mean score
    94.2%
  14. 14
    GoogleDossier

    high

    Tied #1

    Accuracy (best score across scorers)

    Gemini 3.6 Flash, Accuracy (best score across scorers): 94.1% (scale 0.0 to 100.0), 95% confidence interval 91.4 to 96.994.1%(91.4–96.9)
    Mean score
    94.1%
  15. 15
    GoogleDossier
    Tied #1

    Accuracy (best score across scorers)

    Gemini 3.1 Pro Preview, Accuracy (best score across scorers): 94.1% (scale 0.0 to 100.0), 95% confidence interval 90.8 to 97.494.1%(90.8–97.4)
    Mean score
    94.1%
  16. 16
    AnthropicDossier

    max · fallback

    Tied #1

    Accuracy (best score across scorers)

    Opus 5.5, Accuracy (best score across scorers): 94.1% (scale 0.0 to 100.0), 95% confidence interval 91.1 to 97.094.1%(91.1–97.0)
    Mean score
    94.1%
  17. 17
    OpenAIDossier

    xhigh

    Tied #1

    Accuracy (best score across scorers)

    GPT-5.5 Pre-release, Accuracy (best score across scorers): 94% (scale 0.0 to 100.0), 95% confidence interval 91.0 to 97.094%(91.0–97.0)
    Mean score
    94%
  18. 17

    high

    Tied #1

    Accuracy (best score across scorers)

    Grok 4.6, Accuracy (best score across scorers): 94% (scale 0.0 to 100.0), 95% confidence interval 91.2 to 96.894%(91.2–96.8)
    Mean score
    94%
  19. 19
    OpenAIDossier

    xhigh

    Tied #1

    Accuracy (best score across scorers)

    GPT-5.5 Pro Pre-release, Accuracy (best score across scorers): 93.9% (scale 0.0 to 100.0), 95% confidence interval 90.8 to 97.093.9%(90.8–97.0)
    Mean score
    93.9%
  20. 20
    AnthropicDossier

    max · fallback

    Tied #1

    Accuracy (best score across scorers)

    Fable 5, Accuracy (best score across scorers): 93.9% (scale 0.0 to 100.0), 95% confidence interval 90.6 to 97.293.9%(90.6–97.2)
    Mean score
    93.9%
  21. 20
    AnthropicDossier

    max

    Tied #1

    Accuracy (best score across scorers)

    Fable 5.2, Accuracy (best score across scorers): 93.9% (scale 0.0 to 100.0), 95% confidence interval 90.7 to 97.193.9%(90.7–97.1)
    Mean score
    93.9%
  22. 20
    AnthropicDossier

    max

    Tied #1

    Accuracy (best score across scorers)

    Opus 5, Accuracy (best score across scorers): 93.9% (scale 0.0 to 100.0), 95% confidence interval 91.0 to 96.893.9%(91.0–96.8)
    Mean score
    93.9%
  23. 23
    OpenAIDossier

    max

    Tied #1

    Accuracy (best score across scorers)

    GPT-5.6 Sol, Accuracy (best score across scorers): 93.5% (scale 0.0 to 100.0), 95% confidence interval 90.4 to 96.693.5%(90.4–96.6)
    Mean score
    93.5%
  24. 24

    high

    Tied #1

    Accuracy (best score across scorers)

    Grok 4.5, Accuracy (best score across scorers): 93.4% (scale 0.0 to 100.0), 95% confidence interval 90.6 to 96.293.4%(90.6–96.2)
    Mean score
    93.4%
  25. 25
    OpenAIDossier

    max

    Tied #1

    Accuracy (best score across scorers)

    GPT-5.6 Terra, Accuracy (best score across scorers): 93.3% (scale 0.0 to 100.0), 95% confidence interval 90.3 to 96.393.3%(90.3–96.3)
    Mean score
    93.3%
  26. 26
    OpenAIDossier

    xhigh

    Tied #1

    Accuracy (best score across scorers)

    GPT-5.4, Accuracy (best score across scorers): 93.3% (scale 0.0 to 100.0), 95% confidence interval 89.8 to 96.893.3%(89.8–96.8)
    Mean score
    93.3%
  27. 27
    DeepSeekDossier

    max

    Tied #1

    Accuracy (best score across scorers)

    DeepSeek V4.1 Flash 1017, Accuracy (best score across scorers): 93.2% (scale 0.0 to 100.0), 95% confidence interval 90.1 to 96.393.2%(90.1–96.3)
    Mean score
    93.2%
  28. 27

    xhigh

    Tied #1

    Accuracy (best score across scorers)

    Grok 4.6, Accuracy (best score across scorers): 93.2% (scale 0.0 to 100.0), 95% confidence interval 90.2 to 96.293.2%(90.2–96.2)
    Mean score
    93.2%
  29. 29
    ByteDanceDossier

    high

    Tied #1

    Accuracy (best score across scorers)

    Doubao Seed Evolving, Accuracy (best score across scorers): 93.1% (scale 0.0 to 100.0), 95% confidence interval 90.0 to 96.293.1%(90.0–96.2)
    Mean score
    93.1%
  30. 29
    Moonshot AIDossier

    max

    Tied #1

    Accuracy (best score across scorers)

    Kimi K3, Accuracy (best score across scorers): 93.1% (scale 0.0 to 100.0), 95% confidence interval 90.2 to 96.093.1%(90.2–96.0)
    Mean score
    93.1%
  31. 31
    AnthropicDossier
    Tied #1

    Accuracy (best score across scorers)

    Opus 5, Accuracy (best score across scorers): 92.9% (scale 0.0 to 100.0), 95% confidence interval 89.3 to 96.592.9%(89.3–96.5)
    Mean score
    92.9%
  32. 32
    GoogleDossier

    high

    Tied #1

    Accuracy (best score across scorers)

    Gemini 3.5 Flash, Accuracy (best score across scorers): 92.8% (scale 0.0 to 100.0), 95% confidence interval 89.6 to 96.092.8%(89.6–96.0)
    Mean score
    92.8%
  33. 33

    xhigh

    Tied #1

    Accuracy (best score across scorers)

    Grok 4.7, Accuracy (best score across scorers): 92.7% (scale 0.0 to 100.0), 95% confidence interval 89.7 to 95.892.7%(89.7–95.8)
    Mean score
    92.7%
  34. 34
    AlibabaDossier

    xhigh

    Tied #1

    Accuracy (best score across scorers)

    Qwen3.8-Max, Accuracy (best score across scorers): 92.7% (scale 0.0 to 100.0), 95% confidence interval 89.4 to 96.092.7%(89.4–96.0)
    Mean score
    92.7%
  35. 35
    GoogleDossier
    Tied #1

    Accuracy (best score across scorers)

    Gemini 3 Pro Preview, Accuracy (best score across scorers): 92.6% (scale 0.0 to 100.0), 95% confidence interval 89.4 to 95.992.6%(89.4–95.9)
    Mean score
    92.6%
  36. 36
    AlibabaDossier

    xhigh

    Accuracy (best score across scorers)

    Qwen3.8 Max (0902), Accuracy (best score across scorers): 92.3% (scale 0.0 to 100.0), 95% confidence interval 88.9 to 95.792.3%(88.9–95.7)
    Mean score
    92.3%
  37. 37
    Moonshot AIDossier

    high

    Accuracy (best score across scorers)

    Kimi K3, Accuracy (best score across scorers): 91.9% (scale 0.0 to 100.0), 95% confidence interval 88.1 to 95.791.9%(88.1–95.7)
    Mean score
    91.9%
  38. 38

    max

    Accuracy (best score across scorers)

    GLM-5.2, Accuracy (best score across scorers): 91.9% (scale 0.0 to 100.0), 95% confidence interval 88.7 to 95.091.9%(88.7–95.0)
    Mean score
    91.9%
  39. 39
    DeepSeekDossier

    max

    Accuracy (best score across scorers)

    DeepSeek V4 Pro 0813, Accuracy (best score across scorers): 91.7% (scale 0.0 to 100.0), 95% confidence interval 88.7 to 94.791.7%(88.7–94.7)
    Mean score
    91.7%
  40. 40
    OpenAIDossier

    max

    Accuracy (best score across scorers)

    GPT-5.6 Luna, Accuracy (best score across scorers): 91.6% (scale 0.0 to 100.0), 95% confidence interval 88.2 to 95.091.6%(88.2–95.0)
    Mean score
    91.6%
  41. 41
    OpenAIDossier

    xhigh

    Accuracy (best score across scorers)

    GPT-5.2, Accuracy (best score across scorers): 91.4% (scale 0.0 to 100.0), 95% confidence interval 87.9 to 94.991.4%(87.9–94.9)
    Mean score
    91.4%
  42. 42
    DeepSeekDossier

    max

    Accuracy (best score across scorers)

    DeepSeek V4-Flash 0731, Accuracy (best score across scorers): 91% (scale 0.0 to 100.0), 95% confidence interval 87.6 to 94.591%(87.6–94.5)
    Mean score
    91%
  43. 42
    AnthropicDossier

    max

    Accuracy (best score across scorers)

    Opus 4.8, Accuracy (best score across scorers): 91% (scale 0.0 to 100.0), 95% confidence interval 87.3 to 94.891%(87.3–94.8)
    Mean score
    91%
  44. 44
    DeepSeekDossier

    high

    Accuracy (best score across scorers)

    DeepSeek V4-Pro, Accuracy (best score across scorers): 90.9% (scale 0.0 to 100.0), 95% confidence interval 86.9 to 94.990.9%(86.9–94.9)
    Mean score
    90.9%
  45. 44

    max

    Accuracy (best score across scorers)

    GLM-5.3, Accuracy (best score across scorers): 90.9% (scale 0.0 to 100.0), 95% confidence interval 87.8 to 94.090.9%(87.8–94.0)
    Mean score
    90.9%
  46. 44
    MiniMaxDossier

    Accuracy (best score across scorers)

    MiniMax M3, Accuracy (best score across scorers): 90.9% (scale 0.0 to 100.0), 95% confidence interval 86.9 to 94.990.9%(86.9–94.9)
    Mean score
    90.9%
  47. 44
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3.7 Max, Accuracy (best score across scorers): 90.9% (scale 0.0 to 100.0), 95% confidence interval 86.9 to 94.990.9%(86.9–94.9)
    Mean score
    90.9%
  48. 48
    Moonshot AIDossier

    Accuracy (best score across scorers)

    Kimi K2.6, Accuracy (best score across scorers): 90.8% (scale 0.0 to 100.0), 95% confidence interval 87.4 to 94.190.8%(87.4–94.1)
    Mean score
    90.8%
  49. 49
    OpenAIDossier

    low

    Accuracy (best score across scorers)

    GPT-5.5, Accuracy (best score across scorers): 90.7% (scale 0.0 to 100.0), 95% confidence interval 87.2 to 94.190.7%(87.2–94.1)
    Mean score
    90.7%
  50. 50
    AnthropicDossier

    max

    Accuracy (best score across scorers)

    Opus 5.5, Accuracy (best score across scorers): 90.6% (scale 0.0 to 100.0), 95% confidence interval 86.8 to 94.490.6%(86.8–94.4)
    Mean score
    90.6%
  51. 51
    AnthropicDossier

    32,000 budget

    Accuracy (best score across scorers)

    Opus 4.6, Accuracy (best score across scorers): 90.5% (scale 0.0 to 100.0), 95% confidence interval 87.2 to 93.890.5%(87.2–93.8)
    Mean score
    90.5%
  52. 51
    AnthropicDossier

    xhigh

    Accuracy (best score across scorers)

    Sonnet 5, Accuracy (best score across scorers): 90.5% (scale 0.0 to 100.0), 95% confidence interval 87.0 to 94.090.5%(87.0–94.0)
    Mean score
    90.5%
  53. 53
    OpenAIDossier

    max

    Accuracy (best score across scorers)

    GPT-6 Luna, Accuracy (best score across scorers): 90.5% (scale 0.0 to 100.0), 95% confidence interval 86.8 to 94.190.5%(86.8–94.1)
    Mean score
    90.5%
  54. 54

    max

    Accuracy (best score across scorers)

    GLM-5.3 Flash, Accuracy (best score across scorers): 90.2% (scale 0.0 to 100.0), 95% confidence interval 86.8 to 93.590.2%(86.8–93.5)
    Mean score
    90.2%
  55. 54
    MistralDossier

    high

    Accuracy (best score across scorers)

    Mistral Large 4 OSS, Accuracy (best score across scorers): 90.2% (scale 0.0 to 100.0), 95% confidence interval 86.6 to 93.790.2%(86.6–93.7)
    Mean score
    90.2%
  56. 54
    AnthropicDossier

    xhigh

    Accuracy (best score across scorers)

    Opus 4.7, Accuracy (best score across scorers): 90.2% (scale 0.0 to 100.0), 95% confidence interval 86.5 to 93.890.2%(86.5–93.8)
    Mean score
    90.2%
  57. 57

    Accuracy (best score across scorers)

    GLM-5.1, Accuracy (best score across scorers): 89.9% (scale 0.0 to 100.0), 95% confidence interval 85.7 to 94.189.9%(85.7–94.1)
    Mean score
    89.9%
  58. 57
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    GPT-5.4, Accuracy (best score across scorers): 89.9% (scale 0.0 to 100.0), 95% confidence interval 85.7 to 94.189.9%(85.7–94.1)
    Mean score
    89.9%
  59. 57
    OpenAIDossier

    low

    Accuracy (best score across scorers)

    GPT-5.6 Sol, Accuracy (best score across scorers): 89.9% (scale 0.0 to 100.0), 95% confidence interval 85.7 to 94.189.9%(85.7–94.1)
    Mean score
    89.9%
  60. 60
    DeepSeekDossier

    max

    Accuracy (best score across scorers)

    DeepSeek V4.1 Flash, Accuracy (best score across scorers): 89.8% (scale 0.0 to 100.0), 95% confidence interval 86.4 to 93.289.8%(86.4–93.2)
    Mean score
    89.8%
  61. 61

    Accuracy (best score across scorers)

    Muse Spark, Accuracy (best score across scorers): 89.8% (scale 0.0 to 100.0), 95% confidence interval 85.5 to 94.189.8%(85.5–94.1)
    Mean score
    89.8%
  62. 62
    DeepSeekDossier

    max

    Accuracy (best score across scorers)

    DeepSeek V4-Pro, Accuracy (best score across scorers): 89.6% (scale 0.0 to 100.0), 95% confidence interval 86.2 to 93.189.6%(86.2–93.1)
    Mean score
    89.6%
  63. 62
    AnthropicDossier

    max

    Accuracy (best score across scorers)

    Haiku 5.5, Accuracy (best score across scorers): 89.6% (scale 0.0 to 100.0), 95% confidence interval 85.8 to 93.589.6%(85.8–93.5)
    Mean score
    89.6%
  64. 64
    ByteDanceDossier

    high

    Accuracy (best score across scorers)

    Doubao Seed 2.1 Turbo (260628), Accuracy (best score across scorers): 89.6% (scale 0.0 to 100.0), 95% confidence interval 86.1 to 93.189.6%(86.1–93.1)
    Mean score
    89.6%
  65. 65
    GoogleDossier

    high

    Accuracy (best score across scorers)

    Gemini 3 Flash Preview, Accuracy (best score across scorers): 89.4% (scale 0.0 to 100.0), 95% confidence interval 85.1 to 93.789.4%(85.1–93.7)
    Mean score
    89.4%
  66. 66
    ByteDanceDossier

    high

    Accuracy (best score across scorers)

    Doubao Seed 2.1 Pro (260628), Accuracy (best score across scorers): 89.3% (scale 0.0 to 100.0), 95% confidence interval 85.7 to 92.989.3%(85.7–92.9)
    Mean score
    89.3%
  67. 66

    Accuracy (best score across scorers)

    Grok 4.20 0309, Accuracy (best score across scorers): 89.3% (scale 0.0 to 100.0), 95% confidence interval 85.7 to 93.089.3%(85.7–93.0)
    Mean score
    89.3%
  68. 68
    GoogleDossier

    low

    Accuracy (best score across scorers)

    Gemini 3.5 Flash, Accuracy (best score across scorers): 88.9% (scale 0.0 to 100.0), 95% confidence interval 84.5 to 93.388.9%(84.5–93.3)
    Mean score
    88.9%
  69. 68
    OpenAIDossier

    medium

    Accuracy (best score across scorers)

    GPT-5.4, Accuracy (best score across scorers): 88.9% (scale 0.0 to 100.0), 95% confidence interval 84.5 to 93.388.9%(84.5–93.3)
    Mean score
    88.9%
  70. 70

    high

    Accuracy (best score across scorers)

    Grok 4.3, Accuracy (best score across scorers): 88.8% (scale 0.0 to 100.0), 95% confidence interval 85.0 to 92.788.8%(85.0–92.7)
    Mean score
    88.8%
  71. 71
    AnthropicDossier

    64,000 budget

    Accuracy (best score across scorers)

    Opus 4.6, Accuracy (best score across scorers): 88.8% (scale 0.0 to 100.0), 95% confidence interval 85.0 to 92.688.8%(85.0–92.6)
    Mean score
    88.8%
  72. 72

    xhigh

    Accuracy (best score across scorers)

    Muse Spark Open, Accuracy (best score across scorers): 88.6% (scale 0.0 to 100.0), 95% confidence interval 84.9 to 92.288.6%(84.9–92.2)
    Mean score
    88.6%
  73. 73
    Thinking MachinesDossier

    xhigh

    Accuracy (best score across scorers)

    Inkling Small, Accuracy (best score across scorers): 88.5% (scale 0.0 to 100.0), 95% confidence interval 84.8 to 92.388.5%(84.8–92.3)
    Mean score
    88.5%
  74. 74
    AnthropicDossier

    max

    Accuracy (best score across scorers)

    Opus 4.6, Accuracy (best score across scorers): 88.4% (scale 0.0 to 100.0), 95% confidence interval 83.9 to 92.988.4%(83.9–92.9)
    Mean score
    88.4%
  75. 74
    AnthropicDossier

    low

    Accuracy (best score across scorers)

    Opus 4.8, Accuracy (best score across scorers): 88.4% (scale 0.0 to 100.0), 95% confidence interval 83.9 to 92.988.4%(83.9–92.9)
    Mean score
    88.4%
  76. 74
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3.6 Plus, Accuracy (best score across scorers): 88.4% (scale 0.0 to 100.0), 95% confidence interval 83.9 to 92.988.4%(83.9–92.9)
    Mean score
    88.4%
  77. 77
    Thinking MachinesDossier

    xhigh

    Accuracy (best score across scorers)

    Inkling, Accuracy (best score across scorers): 88.3% (scale 0.0 to 100.0), 95% confidence interval 84.5 to 92.188.3%(84.5–92.1)
    Mean score
    88.3%
  78. 78
    ByteDanceDossier

    high

    Accuracy (best score across scorers)

    Doubao Seed 2.0 Pro (260215), Accuracy (best score across scorers): 88.2% (scale 0.0 to 100.0), 95% confidence interval 84.4 to 92.088.2%(84.4–92.0)
    Mean score
    88.2%
  79. 78
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    GPT-5.2, Accuracy (best score across scorers): 88.2% (scale 0.0 to 100.0), 95% confidence interval 84.4 to 91.988.2%(84.4–91.9)
    Mean score
    88.2%
  80. 80
    ByteDanceDossier

    high

    Accuracy (best score across scorers)

    Doubao Seed 2.0 Lite (260428), Accuracy (best score across scorers): 88.1% (scale 0.0 to 100.0), 95% confidence interval 84.3 to 91.988.1%(84.3–91.9)
    Mean score
    88.1%
  81. 81
    MistralDossier

    high

    Accuracy (best score across scorers)

    Mistral Large 4, Accuracy (best score across scorers): 88.1% (scale 0.0 to 100.0), 95% confidence interval 84.3 to 91.988.1%(84.3–91.9)
    Mean score
    88.1%
  82. 82

    low

    Accuracy (best score across scorers)

    GLM-5.2, Accuracy (best score across scorers): 87.9% (scale 0.0 to 100.0), 95% confidence interval 83.3 to 92.487.9%(83.3–92.4)
    Mean score
    87.9%
  83. 82
    OpenAIDossier

    medium

    Accuracy (best score across scorers)

    GPT-5.2, Accuracy (best score across scorers): 87.9% (scale 0.0 to 100.0), 95% confidence interval 84.1 to 91.687.9%(84.1–91.6)
    Mean score
    87.9%
  84. 82
    Moonshot AIDossier

    Accuracy (best score across scorers)

    Kimi K2.7 Code, Accuracy (best score across scorers): 87.9% (scale 0.0 to 100.0), 95% confidence interval 83.3 to 92.487.9%(83.3–92.4)
    Mean score
    87.9%
  85. 82
    AnthropicDossier

    low

    Accuracy (best score across scorers)

    Opus 5, Accuracy (best score across scorers): 87.9% (scale 0.0 to 100.0), 95% confidence interval 83.3 to 92.487.9%(83.3–92.4)
    Mean score
    87.9%
  86. 82
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3.7 Plus, Accuracy (best score across scorers): 87.9% (scale 0.0 to 100.0), 95% confidence interval 83.3 to 92.487.9%(83.3–92.4)
    Mean score
    87.9%
  87. 87

    Accuracy (best score across scorers)

    GLM-5, Accuracy (best score across scorers): 87.8% (scale 0.0 to 100.0), 95% confidence interval 83.2 to 92.487.8%(83.2–92.4)
    Mean score
    87.8%
  88. 88
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    GPT-5.1, Accuracy (best score across scorers): 87.6% (scale 0.0 to 100.0), 95% confidence interval 84.0 to 91.387.6%(84.0–91.3)
    Mean score
    87.6%
  89. 89
    Moonshot AIDossier

    Accuracy (best score across scorers)

    Kimi K2.5, Accuracy (best score across scorers): 87.6% (scale 0.0 to 100.0), 95% confidence interval 83.9 to 91.387.6%(83.9–91.3)
    Mean score
    87.6%
  90. 90
    OpenAIDossier

    low

    Accuracy (best score across scorers)

    GPT-5.6 Terra, Accuracy (best score across scorers): 87.4% (scale 0.0 to 100.0), 95% confidence interval 82.7 to 92.087.4%(82.7–92.0)
    Mean score
    87.4%
  91. 90
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3.6 Max Preview, Accuracy (best score across scorers): 87.4% (scale 0.0 to 100.0), 95% confidence interval 82.7 to 92.087.4%(82.7–92.0)
    Mean score
    87.4%
  92. 90
    AnthropicDossier

    32,000 budget

    Accuracy (best score across scorers)

    Sonnet 4.6, Accuracy (best score across scorers): 87.4% (scale 0.0 to 100.0), 95% confidence interval 83.4 to 91.387.4%(83.4–91.3)
    Mean score
    87.4%
  93. 93

    Accuracy (best score across scorers)

    Grok 4 (0709), Accuracy (best score across scorers): 87% (scale 0.0 to 100.0), 95% confidence interval 83.1 to 90.987%(83.1–90.9)
    Mean score
    87%
  94. 94
    OpenAIDossier

    xhigh

    Accuracy (best score across scorers)

    GPT-5.4 mini, Accuracy (best score across scorers): 86.9% (scale 0.0 to 100.0), 95% confidence interval 82.2 to 91.686.9%(82.2–91.6)
    Mean score
    86.9%
  95. 95
    GoogleDossier

    minimal

    Accuracy (best score across scorers)

    Gemini 3.5 Flash, Accuracy (best score across scorers): 86.4% (scale 0.0 to 100.0), 95% confidence interval 81.6 to 91.286.4%(81.6–91.2)
    Mean score
    86.4%
  96. 95
    GoogleDossier

    low

    Accuracy (best score across scorers)

    Gemini 3.6 Flash, Accuracy (best score across scorers): 86.4% (scale 0.0 to 100.0), 95% confidence interval 81.6 to 91.286.4%(81.6–91.2)
    Mean score
    86.4%
  97. 95
    AnthropicDossier

    max

    Accuracy (best score across scorers)

    Opus 4.7, Accuracy (best score across scorers): 86.4% (scale 0.0 to 100.0), 95% confidence interval 81.6 to 91.286.4%(81.6–91.2)
    Mean score
    86.4%
  98. 95
    AlibabaDossier

    none

    Accuracy (best score across scorers)

    Qwen3.5 397B A17B, Accuracy (best score across scorers): 86.4% (scale 0.0 to 100.0), 95% confidence interval 81.6 to 91.286.4%(81.6–91.2)
    Mean score
    86.4%
  99. 99
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    GPT-5, Accuracy (best score across scorers): 86.2% (scale 0.0 to 100.0), 95% confidence interval 82.1 to 90.286.2%(82.1–90.2)
    Mean score
    86.2%
  100. 100
    AnthropicDossier

    32,000 budget

    Accuracy (best score across scorers)

    Opus 4.5, Accuracy (best score across scorers): 86% (scale 0.0 to 100.0), 95% confidence interval 82.0 to 90.186%(82.0–90.1)
    Mean score
    86%
  101. 101
    AnthropicDossier

    max

    Accuracy (best score across scorers)

    Fable 5, Accuracy (best score across scorers): 85.9% (scale 0.0 to 100.0), 95% confidence interval 81.0 to 90.785.9%(81.0–90.7)
    Mean score
    85.9%
  102. 101
    GoogleDossier

    minimal

    Accuracy (best score across scorers)

    Gemini 3.6 Flash, Accuracy (best score across scorers): 85.9% (scale 0.0 to 100.0), 95% confidence interval 81.0 to 90.785.9%(81.0–90.7)
    Mean score
    85.9%
  103. 101
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3.5 397B A17B, Accuracy (best score across scorers): 85.9% (scale 0.0 to 100.0), 95% confidence interval 81.0 to 90.785.9%(81.0–90.7)
    Mean score
    85.9%
  104. 101
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3.6 27B, Accuracy (best score across scorers): 85.9% (scale 0.0 to 100.0), 95% confidence interval 81.0 to 90.785.9%(81.0–90.7)
    Mean score
    85.9%
  105. 105
    AnthropicDossier

    16,000 budget

    Accuracy (best score across scorers)

    Opus 4.5, Accuracy (best score across scorers): 85.5% (scale 0.0 to 100.0), 95% confidence interval 81.3 to 89.785.5%(81.3–89.7)
    Mean score
    85.5%
  106. 106
    OpenAIDossier

    medium

    Accuracy (best score across scorers)

    GPT-5, Accuracy (best score across scorers): 85.4% (scale 0.0 to 100.0), 95% confidence interval 81.2 to 89.585.4%(81.2–89.5)
    Mean score
    85.4%
  107. 106
    NVIDIADossier

    Accuracy (best score across scorers)

    Nemotron 3 Ultra 550B A55B, Accuracy (best score across scorers): 85.4% (scale 0.0 to 100.0), 95% confidence interval 80.4 to 90.385.4%(80.4–90.3)
    Mean score
    85.4%
  108. 106
    AnthropicDossier

    none

    Accuracy (best score across scorers)

    Opus 4.8, Accuracy (best score across scorers): 85.4% (scale 0.0 to 100.0), 95% confidence interval 80.4 to 90.385.4%(80.4–90.3)
    Mean score
    85.4%
  109. 109
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 2.5 Pro, Accuracy (best score across scorers): 85.3% (scale 0.0 to 100.0), 95% confidence interval 81.1 to 89.485.3%(81.1–89.4)
    Mean score
    85.3%
  110. 110
    OpenAIDossier

    medium

    Accuracy (best score across scorers)

    GPT-5.1, Accuracy (best score across scorers): 85% (scale 0.0 to 100.0), 95% confidence interval 80.9 to 89.285%(80.9–89.2)
    Mean score
    85%
  111. 111
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 2.5 Pro Preview (06-05), Accuracy (best score across scorers): 84.8% (scale 0.0 to 100.0), 95% confidence interval 79.8 to 89.984.8%(79.8–89.9)
    Mean score
    84.8%
  112. 111
    OpenAIDossier

    low

    Accuracy (best score across scorers)

    GPT-5.4, Accuracy (best score across scorers): 84.8% (scale 0.0 to 100.0), 95% confidence interval 79.8 to 89.984.8%(79.8–89.9)
    Mean score
    84.8%
  113. 111
    Moonshot AIDossier

    low

    Accuracy (best score across scorers)

    Kimi K3, Accuracy (best score across scorers): 84.8% (scale 0.0 to 100.0), 95% confidence interval 79.8 to 89.984.8%(79.8–89.9)
    Mean score
    84.8%
  114. 111
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3.5 Plus, Accuracy (best score across scorers): 84.8% (scale 0.0 to 100.0), 95% confidence interval 79.8 to 89.984.8%(79.8–89.9)
    Mean score
    84.8%
  115. 111
    AlibabaDossier

    none

    Accuracy (best score across scorers)

    Qwen3.6 27B, Accuracy (best score across scorers): 84.8% (scale 0.0 to 100.0), 95% confidence interval 79.8 to 89.984.8%(79.8–89.9)
    Mean score
    84.8%
  116. 111
    AlibabaDossier

    none

    Accuracy (best score across scorers)

    Qwen3.6 35B A3B, Accuracy (best score across scorers): 84.8% (scale 0.0 to 100.0), 95% confidence interval 79.8 to 89.984.8%(79.8–89.9)
    Mean score
    84.8%
  117. 117
    Moonshot AIDossier

    Accuracy (best score across scorers)

    Kimi K2 Thinking Turbo, Accuracy (best score across scorers): 84.2% (scale 0.0 to 100.0), 95% confidence interval 80.1 to 88.384.2%(80.1–88.3)
    Mean score
    84.2%
  118. 118
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 2.5 Pro Experimental (03-25), Accuracy (best score across scorers): 83.8% (scale 0.0 to 100.0), 95% confidence interval 78.7 to 89.083.8%(78.7–89.0)
    Mean score
    83.8%
  119. 118
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3.6 35B A3B, Accuracy (best score across scorers): 83.8% (scale 0.0 to 100.0), 95% confidence interval 78.7 to 89.083.8%(78.7–89.0)
    Mean score
    83.8%
  120. 120
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    GPT-5.4 mini, Accuracy (best score across scorers): 83.6% (scale 0.0 to 100.0), 95% confidence interval 79.2 to 88.083.6%(79.2–88.0)
    Mean score
    83.6%
  121. 121
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3.5 35B A3B, Accuracy (best score across scorers): 83.5% (scale 0.0 to 100.0), 95% confidence interval 79.2 to 87.783.5%(79.2–87.7)
    Mean score
    83.5%
  122. 122
    DeepSeekDossier

    Accuracy (best score across scorers)

    DeepSeek V3.2, Accuracy (best score across scorers): 83.4% (scale 0.0 to 100.0), 95% confidence interval 79.5 to 87.483.4%(79.5–87.4)
    Mean score
    83.4%
  123. 123
    AnthropicDossier

    high

    Accuracy (best score across scorers)

    Fable 5, Accuracy (best score across scorers): 83.3% (scale 0.0 to 100.0), 95% confidence interval 78.1 to 88.583.3%(78.1–88.5)
    Mean score
    83.3%
  124. 123
    GoogleDossier

    high

    Accuracy (best score across scorers)

    Gemini 3.5 Flash-Lite, Accuracy (best score across scorers): 83.3% (scale 0.0 to 100.0), 95% confidence interval 78.1 to 88.583.3%(78.1–88.5)
    Mean score
    83.3%
  125. 123

    Accuracy (best score across scorers)

    GLM-4.7, Accuracy (best score across scorers): 83.3% (scale 0.0 to 100.0), 95% confidence interval 78.7 to 88.083.3%(78.7–88.0)
    Mean score
    83.3%
  126. 123
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3.6 Flash, Accuracy (best score across scorers): 83.3% (scale 0.0 to 100.0), 95% confidence interval 78.1 to 88.583.3%(78.1–88.5)
    Mean score
    83.3%
  127. 123
    AnthropicDossier

    high

    Accuracy (best score across scorers)

    Sonnet 4.6, Accuracy (best score across scorers): 83.3% (scale 0.0 to 100.0), 95% confidence interval 78.1 to 88.583.3%(78.1–88.5)
    Mean score
    83.3%
  128. 123
    AnthropicDossier

    medium

    Accuracy (best score across scorers)

    Sonnet 4.6, Accuracy (best score across scorers): 83.3% (scale 0.0 to 100.0), 95% confidence interval 78.1 to 88.583.3%(78.1–88.5)
    Mean score
    83.3%
  129. 129
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 3 Flash Preview, Accuracy (best score across scorers): 83.2% (scale 0.0 to 100.0), 95% confidence interval 79.5 to 86.983.2%(79.5–86.9)
    Mean score
    83.2%
  130. 130
    OpenAIDossier

    none

    Accuracy (best score across scorers)

    GPT-5.6 Sol, Accuracy (best score across scorers): 82.8% (scale 0.0 to 100.0), 95% confidence interval 77.6 to 88.182.8%(77.6–88.1)
    Mean score
    82.8%
  131. 131
    OpenAIDossier

    low

    Accuracy (best score across scorers)

    GPT-5.2, Accuracy (best score across scorers): 82.7% (scale 0.0 to 100.0), 95% confidence interval 78.3 to 87.182.7%(78.3–87.1)
    Mean score
    82.7%
  132. 132
    OpenAIDossier

    Accuracy (best score across scorers)

    GPT-5.5 Instant, Accuracy (best score across scorers): 82.5% (scale 0.0 to 100.0), 95% confidence interval 78.1 to 86.982.5%(78.1–86.9)
    Mean score
    82.5%
  133. 133
    OpenAIDossier

    low

    Accuracy (best score across scorers)

    GPT-5.6 Luna, Accuracy (best score across scorers): 82.3% (scale 0.0 to 100.0), 95% confidence interval 77.0 to 87.782.3%(77.0–87.7)
    Mean score
    82.3%
  134. 133
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3.5 Flash, Accuracy (best score across scorers): 82.3% (scale 0.0 to 100.0), 95% confidence interval 77.0 to 87.782.3%(77.0–87.7)
    Mean score
    82.3%
  135. 133
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3.7 Flash, Accuracy (best score across scorers): 82.3% (scale 0.0 to 100.0), 95% confidence interval 77.0 to 87.782.3%(77.0–87.7)
    Mean score
    82.3%
  136. 133
    AnthropicDossier

    59,000 budget

    Accuracy (best score across scorers)

    Sonnet 4.5, Accuracy (best score across scorers): 82.3% (scale 0.0 to 100.0), 95% confidence interval 77.0 to 87.782.3%(77.0–87.7)
    Mean score
    82.3%
  137. 137
    GoogleDossier

    high

    Accuracy (best score across scorers)

    Gemini 3.1 Flash-Lite Preview, Accuracy (best score across scorers): 81.8% (scale 0.0 to 100.0), 95% confidence interval 76.4 to 87.281.8%(76.4–87.2)
    Mean score
    81.8%
  138. 137
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    o3, Accuracy (best score across scorers): 81.8% (scale 0.0 to 100.0), 95% confidence interval 77.6 to 86.081.8%(77.6–86.0)
    Mean score
    81.8%
  139. 137
    AlibabaDossier

    none

    Accuracy (best score across scorers)

    Qwen3.7 Plus, Accuracy (best score across scorers): 81.8% (scale 0.0 to 100.0), 95% confidence interval 76.4 to 87.281.8%(76.4–87.2)
    Mean score
    81.8%
  140. 140
    AnthropicDossier

    32,000 budget

    Accuracy (best score across scorers)

    Sonnet 4.5, Accuracy (best score across scorers): 81.7% (scale 0.0 to 100.0), 95% confidence interval 76.3 to 87.181.7%(76.3–87.1)
    Mean score
    81.7%
  141. 141
    MiniMaxDossier

    none

    Accuracy (best score across scorers)

    MiniMax M3, Accuracy (best score across scorers): 81.3% (scale 0.0 to 100.0), 95% confidence interval 75.9 to 86.881.3%(75.9–86.8)
    Mean score
    81.3%
  142. 142
    AlibabaDossier

    none

    Accuracy (best score across scorers)

    Qwen3.5 35B A3B, Accuracy (best score across scorers): 81.2% (scale 0.0 to 100.0), 95% confidence interval 76.6 to 85.781.2%(76.6–85.7)
    Mean score
    81.2%
  143. 143
    OpenAIDossier

    medium

    Accuracy (best score across scorers)

    o3, Accuracy (best score across scorers): 80.8% (scale 0.0 to 100.0), 95% confidence interval 75.3 to 86.380.8%(75.3–86.3)
    Mean score
    80.8%
  144. 143
    AlibabaDossier

    none

    Accuracy (best score across scorers)

    Qwen3.7 Flash, Accuracy (best score across scorers): 80.8% (scale 0.0 to 100.0), 95% confidence interval 75.3 to 86.380.8%(75.3–86.3)
    Mean score
    80.8%
  145. 145
    AnthropicDossier

    Accuracy (best score across scorers)

    Opus 4.5, Accuracy (best score across scorers): 80.7% (scale 0.0 to 100.0), 95% confidence interval 76.0 to 85.480.7%(76.0–85.4)
    Mean score
    80.7%
  146. 146
    AnthropicDossier

    max

    Accuracy (best score across scorers)

    Sonnet 5, Accuracy (best score across scorers): 80.3% (scale 0.0 to 100.0), 95% confidence interval 74.7 to 85.980.3%(74.7–85.9)
    Mean score
    80.3%
  147. 147
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3 235B A22B Thinking (2507), Accuracy (best score across scorers): 80.1% (scale 0.0 to 100.0), 95% confidence interval 75.0 to 85.180.1%(75.0–85.1)
    Mean score
    80.1%
  148. 148
    OpenAIDossier

    low

    Accuracy (best score across scorers)

    o3, Accuracy (best score across scorers): 79.8% (scale 0.0 to 100.0), 95% confidence interval 74.2 to 85.479.8%(74.2–85.4)
    Mean score
    79.8%
  149. 149
    AnthropicDossier

    64,000 budget

    Accuracy (best score across scorers)

    Claude 3.7 Sonnet, Accuracy (best score across scorers): 79.7% (scale 0.0 to 100.0), 95% confidence interval 74.5 to 84.979.7%(74.5–84.9)
    Mean score
    78.5%
  150. 150
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    o4-mini, Accuracy (best score across scorers): 79.6% (scale 0.0 to 100.0), 95% confidence interval 74.9 to 84.379.6%(74.9–84.3)
    Mean score
    79.6%
  151. 151
    AnthropicDossier

    59,000 budget

    Accuracy (best score across scorers)

    Claude Sonnet 4, Accuracy (best score across scorers): 79.2% (scale 0.0 to 100.0), 95% confidence interval 73.9 to 84.479.2%(73.9–84.4)
    Mean score
    77.8%
  152. 152
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3.5 9B, Accuracy (best score across scorers): 79% (scale 0.0 to 100.0), 95% confidence interval 74.3 to 83.679%(74.3–83.6)
    Mean score
    79%
  153. 153
    AlibabaDossier

    none

    Accuracy (best score across scorers)

    Qwen3.5 9B, Accuracy (best score across scorers): 78.9% (scale 0.0 to 100.0), 95% confidence interval 74.2 to 83.678.9%(74.2–83.6)
    Mean score
    78.9%
  154. 154
    AnthropicDossier

    low

    Accuracy (best score across scorers)

    Fable 5, Accuracy (best score across scorers): 78.8% (scale 0.0 to 100.0), 95% confidence interval 73.1 to 84.578.8%(73.1–84.5)
    Mean score
    78.8%
  155. 154
    AnthropicDossier

    16,000 budget

    Accuracy (best score across scorers)

    Sonnet 4.5, Accuracy (best score across scorers): 78.8% (scale 0.0 to 100.0), 95% confidence interval 73.1 to 84.578.8%(73.1–84.5)
    Mean score
    78.8%
  156. 154
    AnthropicDossier

    max

    Accuracy (best score across scorers)

    Sonnet 4.6, Accuracy (best score across scorers): 78.8% (scale 0.0 to 100.0), 95% confidence interval 73.1 to 84.578.8%(73.1–84.5)
    Mean score
    78.8%
  157. 157
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    GPT-5.4 nano, Accuracy (best score across scorers): 78.5% (scale 0.0 to 100.0), 95% confidence interval 73.7 to 83.378.5%(73.7–83.3)
    Mean score
    78.5%
  158. 158
    AnthropicDossier

    32,000 budget

    Accuracy (best score across scorers)

    Claude Sonnet 4, Accuracy (best score across scorers): 78.3% (scale 0.0 to 100.0), 95% confidence interval 72.5 to 84.078.3%(72.5–84.0)
    Mean score
    78.3%
  159. 159
    OpenAIDossier

    medium

    Accuracy (best score across scorers)

    o4-mini, Accuracy (best score across scorers): 77.8% (scale 0.0 to 100.0), 95% confidence interval 72.0 to 83.677.8%(72.0–83.6)
    Mean score
    77.8%
  160. 160
    ByteDanceDossier

    Accuracy (best score across scorers)

    Seed 1.6, Accuracy (best score across scorers): 77.6% (scale 0.0 to 100.0), 95% confidence interval 72.6 to 82.577.6%(72.6–82.5)
    Mean score
    77.6%
  161. 161
    OpenAIDossier

    none

    Accuracy (best score across scorers)

    GPT-5.5, Accuracy (best score across scorers): 77.3% (scale 0.0 to 100.0), 95% confidence interval 71.4 to 83.177.3%(71.4–83.1)
    Mean score
    77.3%
  162. 161
    OpenAIDossier

    none

    Accuracy (best score across scorers)

    GPT-5.6 Terra, Accuracy (best score across scorers): 77.3% (scale 0.0 to 100.0), 95% confidence interval 71.4 to 83.177.3%(71.4–83.1)
    Mean score
    77.3%
  163. 161
    AnthropicDossier

    16,000 budget

    Accuracy (best score across scorers)

    Opus 4.1, Accuracy (best score across scorers): 77.3% (scale 0.0 to 100.0), 95% confidence interval 71.4 to 83.177.3%(71.4–83.1)
    Mean score
    77.3%
  164. 164
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    o3-mini, Accuracy (best score across scorers): 77% (scale 0.0 to 100.0), 95% confidence interval 72.0 to 82.177%(72.0–82.1)
    Mean score
    77%
  165. 165
    AnthropicDossier

    32,000 budget

    Accuracy (best score across scorers)

    Claude 3.7 Sonnet, Accuracy (best score across scorers): 76.8% (scale 0.0 to 100.0), 95% confidence interval 70.9 to 82.776.8%(70.9–82.7)
    Mean score
    76.8%
  166. 165
    AnthropicDossier

    16,000 budget

    Accuracy (best score across scorers)

    Claude 3.7 Sonnet, Accuracy (best score across scorers): 76.8% (scale 0.0 to 100.0), 95% confidence interval 70.9 to 82.776.8%(70.9–82.7)
    Mean score
    76.8%
  167. 165
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    o1, Accuracy (best score across scorers): 76.8% (scale 0.0 to 100.0), 95% confidence interval 70.9 to 82.776.8%(70.9–82.7)
    Mean score
    76.8%
  168. 165
    AnthropicDossier

    27,000 budget

    Accuracy (best score across scorers)

    Opus 4.1, Accuracy (best score across scorers): 76.8% (scale 0.0 to 100.0), 95% confidence interval 70.9 to 82.776.8%(70.9–82.7)
    Mean score
    76.8%
  169. 169
    DeepSeekDossier

    Accuracy (best score across scorers)

    DeepSeek R1 0528, Accuracy (best score across scorers): 76.3% (scale 0.0 to 100.0), 95% confidence interval 71.6 to 81.076.3%(71.6–81.0)
    Mean score
    76.3%
  170. 170

    low

    Accuracy (best score across scorers)

    Grok 3 Mini Beta, Accuracy (best score across scorers): 76.3% (scale 0.0 to 100.0), 95% confidence interval 70.3 to 82.276.3%(70.3–82.2)
    Mean score
    76.3%
  171. 170
    AnthropicDossier

    16,000 budget

    Accuracy (best score across scorers)

    Opus 4, Accuracy (best score across scorers): 76.3% (scale 0.0 to 100.0), 95% confidence interval 70.3 to 82.276.3%(70.3–82.2)
    Mean score
    76.3%
  172. 172
    AnthropicDossier

    16,000 budget

    Accuracy (best score across scorers)

    Claude Sonnet 4, Accuracy (best score across scorers): 75.8% (scale 0.0 to 100.0), 95% confidence interval 69.8 to 81.775.8%(69.8–81.7)
    Mean score
    75.8%
  173. 172
    GoogleDossier

    low

    Accuracy (best score across scorers)

    Gemini 3.5 Flash-Lite, Accuracy (best score across scorers): 75.8% (scale 0.0 to 100.0), 95% confidence interval 69.8 to 81.775.8%(69.8–81.7)
    Mean score
    75.8%
  174. 172
    GoogleDossier

    minimal

    Accuracy (best score across scorers)

    Gemma 4 31B, Accuracy (best score across scorers): 75.8% (scale 0.0 to 100.0), 95% confidence interval 69.8 to 81.775.8%(69.8–81.7)
    Mean score
    75.8%
  175. 172
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    gpt-oss-120b, Accuracy (best score across scorers): 75.8% (scale 0.0 to 100.0), 95% confidence interval 70.4 to 81.175.8%(70.4–81.1)
    Mean score
    75.8%
  176. 172

    Accuracy (best score across scorers)

    Grok 3 Beta, Accuracy (best score across scorers): 75.8% (scale 0.0 to 100.0), 95% confidence interval 70.5 to 81.075.8%(70.5–81.0)
    Mean score
    67.6%
  177. 172
    OpenAIDossier

    medium

    Accuracy (best score across scorers)

    o1, Accuracy (best score across scorers): 75.8% (scale 0.0 to 100.0), 95% confidence interval 69.8 to 81.775.8%(69.8–81.7)
    Mean score
    75.8%
  178. 178

    high

    Accuracy (best score across scorers)

    Grok 3 Mini Beta, Accuracy (best score across scorers): 75.5% (scale 0.0 to 100.0), 95% confidence interval 70.1 to 80.975.5%(70.1–80.9)
    Mean score
    74.6%
  179. 179
    OpenAIDossier

    low

    Accuracy (best score across scorers)

    o4-mini, Accuracy (best score across scorers): 75.3% (scale 0.0 to 100.0), 95% confidence interval 69.2 to 81.375.3%(69.2–81.3)
    Mean score
    75.3%
  180. 180
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    GPT-5 mini, Accuracy (best score across scorers): 75% (scale 0.0 to 100.0), 95% confidence interval 70.5 to 79.575%(70.5–79.5)
    Mean score
    75%
  181. 181
    OpenAIDossier

    none

    Accuracy (best score across scorers)

    GPT-5.4, Accuracy (best score across scorers): 74.7% (scale 0.0 to 100.0), 95% confidence interval 68.7 to 80.874.7%(68.7–80.8)
    Mean score
    74.7%
  182. 182
    OpenAIDossier

    medium

    Accuracy (best score across scorers)

    o3-mini, Accuracy (best score across scorers): 74.3% (scale 0.0 to 100.0), 95% confidence interval 69.3 to 79.374.3%(69.3–79.3)
    Mean score
    74.3%
  183. 183
    GoogleDossier

    low

    Accuracy (best score across scorers)

    Gemini 3.1 Flash-Lite Preview, Accuracy (best score across scorers): 74.2% (scale 0.0 to 100.0), 95% confidence interval 68.1 to 80.374.2%(68.1–80.3)
    Mean score
    74.2%
  184. 183
    GoogleDossier

    minimal

    Accuracy (best score across scorers)

    Gemini 3.5 Flash-Lite, Accuracy (best score across scorers): 74.2% (scale 0.0 to 100.0), 95% confidence interval 68.1 to 80.374.2%(68.1–80.3)
    Mean score
    74.2%
  185. 183
    OpenAIDossier

    low

    Accuracy (best score across scorers)

    o1, Accuracy (best score across scorers): 74.2% (scale 0.0 to 100.0), 95% confidence interval 68.1 to 80.374.2%(68.1–80.3)
    Mean score
    74.2%
  186. 186
    GoogleDossier

    minimal

    Accuracy (best score across scorers)

    Gemini 3.1 Flash-Lite Preview, Accuracy (best score across scorers): 73.7% (scale 0.0 to 100.0), 95% confidence interval 67.6 to 79.973.7%(67.6–79.9)
    Mean score
    73.7%
  187. 186
    AnthropicDossier

    Accuracy (best score across scorers)

    Sonnet 4.5, Accuracy (best score across scorers): 73.7% (scale 0.0 to 100.0), 95% confidence interval 67.6 to 79.973.7%(67.6–79.9)
    Mean score
    73.7%
  188. 188
    DeepSeekDossier

    none

    Accuracy (best score across scorers)

    DeepSeek V4-Pro, Accuracy (best score across scorers): 73.2% (scale 0.0 to 100.0), 95% confidence interval 67.0 to 79.473.2%(67.0–79.4)
    Mean score
    73.2%
  189. 188
    GoogleDossier

    minimal

    Accuracy (best score across scorers)

    Gemma 4 26B A4B, Accuracy (best score across scorers): 73.2% (scale 0.0 to 100.0), 95% confidence interval 67.0 to 79.473.2%(67.0–79.4)
    Mean score
    73.2%
  190. 188
    OpenAIDossier

    none

    Accuracy (best score across scorers)

    GPT-5.2, Accuracy (best score across scorers): 73.2% (scale 0.0 to 100.0), 95% confidence interval 67.0 to 79.473.2%(67.0–79.4)
    Mean score
    73.2%
  191. 188
    AnthropicDossier

    Accuracy (best score across scorers)

    Opus 4.1, Accuracy (best score across scorers): 73.2% (scale 0.0 to 100.0), 95% confidence interval 67.0 to 79.473.2%(67.0–79.4)
    Mean score
    73.2%
  192. 192
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3 Max, Accuracy (best score across scorers): 72.6% (scale 0.0 to 100.0), 95% confidence interval 67.2 to 78.072.6%(67.2–78.0)
    Mean score
    72.6%
  193. 193
    OpenAIDossier

    low

    Accuracy (best score across scorers)

    GPT-5.4 nano, Accuracy (best score across scorers): 72.2% (scale 0.0 to 100.0), 95% confidence interval 66.0 to 78.572.2%(66.0–78.5)
    Mean score
    72.2%
  194. 194
    DeepSeekDossier

    Accuracy (best score across scorers)

    DeepSeek R1 (Jan '25), Accuracy (best score across scorers): 71.7% (scale 0.0 to 100.0), 95% confidence interval 65.7 to 77.771.7%(65.7–77.7)
    Mean score
    69.2%
  195. 194
    OpenAIDossier

    minimal

    Accuracy (best score across scorers)

    GPT-5, Accuracy (best score across scorers): 71.7% (scale 0.0 to 100.0), 95% confidence interval 65.4 to 78.071.7%(65.4–78.0)
    Mean score
    71.7%
  196. 194
    OpenAIDossier

    minimal

    Accuracy (best score across scorers)

    GPT-5 mini, Accuracy (best score across scorers): 71.7% (scale 0.0 to 100.0), 95% confidence interval 65.4 to 78.071.7%(65.4–78.0)
    Mean score
    71.7%
  197. 197
    OpenAIDossier

    medium

    Accuracy (best score across scorers)

    GPT-5 mini, Accuracy (best score across scorers): 71.7% (scale 0.0 to 100.0), 95% confidence interval 67.1 to 76.271.7%(67.1–76.2)
    Mean score
    71.7%
  198. 198
    ByteDanceDossier

    Accuracy (best score across scorers)

    Seed-OSS 36B Instruct, Accuracy (best score across scorers): 71.5% (scale 0.0 to 100.0), 95% confidence interval 66.2 to 76.971.5%(66.2–76.9)
    Mean score
    71.5%
  199. 199
    DeepSeekDossier

    Accuracy (best score across scorers)

    DeepSeek V3.2, Accuracy (best score across scorers): 71.2% (scale 0.0 to 100.0), 95% confidence interval 64.9 to 77.571.2%(64.9–77.5)
    Mean score
    71.2%
  200. 199

    none

    Accuracy (best score across scorers)

    GLM-5.2, Accuracy (best score across scorers): 71.2% (scale 0.0 to 100.0), 95% confidence interval 64.9 to 77.571.2%(64.9–77.5)
    Mean score
    71.2%
  201. 199
    AnthropicDossier

    32,000 budget

    Accuracy (best score across scorers)

    Haiku 4.5, Accuracy (best score across scorers): 71.2% (scale 0.0 to 100.0), 95% confidence interval 64.9 to 77.571.2%(64.9–77.5)
    Mean score
    71.2%
  202. 202
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3 235B A22B, Accuracy (best score across scorers): 70.7% (scale 0.0 to 100.0), 95% confidence interval 65.4 to 76.070.7%(65.4–76.0)
    Mean score
    70.7%
  203. 203
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3 30B A3B Thinking 2507, Accuracy (best score across scorers): 70.1% (scale 0.0 to 100.0), 95% confidence interval 64.5 to 75.670.1%(64.5–75.6)
    Mean score
    70.1%
  204. 204
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    GPT-5 nano, Accuracy (best score across scorers): 69.4% (scale 0.0 to 100.0), 95% confidence interval 64.0 to 74.969.4%(64.0–74.9)
    Mean score
    69.4%
  205. 205
    AnthropicDossier

    Accuracy (best score across scorers)

    Opus 4, Accuracy (best score across scorers): 69.2% (scale 0.0 to 100.0), 95% confidence interval 62.7 to 75.669.2%(62.7–75.6)
    Mean score
    69.2%
  206. 206
    OpenAIDossier

    Accuracy (best score across scorers)

    GPT-4.5 Preview, Accuracy (best score across scorers): 68.7% (scale 0.0 to 100.0), 95% confidence interval 62.2 to 75.268.7%(62.2–75.2)
    Mean score
    68.7%
  207. 207
    OpenAIDossier

    low

    Accuracy (best score across scorers)

    o3-mini, Accuracy (best score across scorers): 68.2% (scale 0.0 to 100.0), 95% confidence interval 61.7 to 74.768.2%(61.7–74.7)
    Mean score
    68.2%
  208. 208
    DeepSeekDossier

    Accuracy (best score across scorers)

    DeepSeek V3 0324, Accuracy (best score across scorers): 67.6% (scale 0.0 to 100.0), 95% confidence interval 62.4 to 72.967.6%(62.4–72.9)
    Mean score
    67.6%
  209. 209
    OpenAIDossier

    medium

    Accuracy (best score across scorers)

    GPT-5 nano, Accuracy (best score across scorers): 67.4% (scale 0.0 to 100.0), 95% confidence interval 62.2 to 72.667.4%(62.2–72.6)
    Mean score
    67.4%
  210. 210

    Accuracy (best score across scorers)

    Llama 4 Maverick 17B 128E Instruct (FP8), Accuracy (best score across scorers): 67% (scale 0.0 to 100.0), 95% confidence interval 61.4 to 72.567%(61.4–72.5)
    Mean score
    67%
  211. 211
    OpenAIDossier

    Accuracy (best score across scorers)

    GPT-4.1, Accuracy (best score across scorers): 66.9% (scale 0.0 to 100.0), 95% confidence interval 61.4 to 72.466.9%(61.4–72.4)
    Mean score
    66.9%
  212. 212
    AnthropicDossier

    Accuracy (best score across scorers)

    Claude Sonnet 4, Accuracy (best score across scorers): 66.7% (scale 0.0 to 100.0), 95% confidence interval 60.1 to 73.266.7%(60.1–73.2)
    Mean score
    66.7%
  213. 212
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 2.5 Pro Preview (May '25), Accuracy (best score across scorers): 66.7% (scale 0.0 to 100.0), 95% confidence interval 60.1 to 73.266.7%(60.1–73.2)
    Mean score
    66.7%
  214. 212
    OpenAIDossier

    none

    Accuracy (best score across scorers)

    GPT-5.1, Accuracy (best score across scorers): 66.7% (scale 0.0 to 100.0), 95% confidence interval 60.1 to 73.266.7%(60.1–73.2)
    Mean score
    66.7%
  215. 215
    AnthropicDossier

    Accuracy (best score across scorers)

    Claude 3.7 Sonnet, Accuracy (best score across scorers): 66% (scale 0.0 to 100.0), 95% confidence interval 60.8 to 71.366%(60.8–71.3)
    Mean score
    66%
  216. 216
    OpenAIDossier

    Accuracy (best score across scorers)

    GPT-4.1 mini, Accuracy (best score across scorers): 65.8% (scale 0.0 to 100.0), 95% confidence interval 60.5 to 71.165.8%(60.5–71.1)
    Mean score
    65.8%
  217. 217
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3 32B, Accuracy (best score across scorers): 65.7% (scale 0.0 to 100.0), 95% confidence interval 60.3 to 71.265.7%(60.3–71.2)
    Mean score
    65.7%
  218. 218
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 2.0 Pro Experimental (02-05), Accuracy (best score across scorers): 65.7% (scale 0.0 to 100.0), 95% confidence interval 59.0 to 72.365.7%(59.0–72.3)
    Mean score
    65.7%
  219. 219
    AlibabaDossier

    Accuracy (best score across scorers)

    QwQ Plus, Accuracy (best score across scorers): 65.4% (scale 0.0 to 100.0), 95% confidence interval 59.8 to 71.065.4%(59.8–71.0)
    Mean score
    65.4%
  220. 220
    AlibabaDossier

    Accuracy (best score across scorers)

    QwQ 32B, Accuracy (best score across scorers): 65.3% (scale 0.0 to 100.0), 95% confidence interval 59.8 to 70.965.3%(59.8–70.9)
    Mean score
    65.3%
  221. 221
    DeepSeekDossier

    Accuracy (best score across scorers)

    DeepSeek R1 Distill Qwen 32B, Accuracy (best score across scorers): 64.1% (scale 0.0 to 100.0), 95% confidence interval 58.7 to 69.564.1%(58.7–69.5)
    Mean score
    64.1%
  222. 221
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 2.0 Flash (001), Accuracy (best score across scorers): 64.1% (scale 0.0 to 100.0), 95% confidence interval 58.8 to 69.564.1%(58.8–69.5)
    Mean score
    64.1%
  223. 221
    OpenAIDossier

    none

    Accuracy (best score across scorers)

    GPT-5.4 mini, Accuracy (best score across scorers): 64.1% (scale 0.0 to 100.0), 95% confidence interval 57.4 to 70.864.1%(57.4–70.8)
    Mean score
    64.1%
  224. 224
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3 14B, Accuracy (best score across scorers): 63.8% (scale 0.0 to 100.0), 95% confidence interval 58.0 to 69.563.8%(58.0–69.5)
    Mean score
    63.8%
  225. 225
    OpenAIDossier

    none

    Accuracy (best score across scorers)

    GPT-5.6 Luna, Accuracy (best score across scorers): 63.6% (scale 0.0 to 100.0), 95% confidence interval 56.9 to 70.463.6%(56.9–70.4)
    Mean score
    63.6%
  226. 226
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    o1-mini, Accuracy (best score across scorers): 62.4% (scale 0.0 to 100.0), 95% confidence interval 57.0 to 67.762.4%(57.0–67.7)
    Mean score
    62.4%
  227. 227
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3 30B A3B, Accuracy (best score across scorers): 61.7% (scale 0.0 to 100.0), 95% confidence interval 55.9 to 67.561.7%(55.9–67.5)
    Mean score
    61.7%
  228. 228
    OpenAIDossier

    medium

    Accuracy (best score across scorers)

    gpt-oss-20b, Accuracy (best score across scorers): 60.8% (scale 0.0 to 100.0), 95% confidence interval 55.9 to 65.760.8%(55.9–65.7)
    Mean score
    60.8%
  229. 229

    Accuracy (best score across scorers)

    GLM-4.7-Flash, Accuracy (best score across scorers): 60.5% (scale 0.0 to 100.0), 95% confidence interval 54.8 to 66.360.5%(54.8–66.3)
    Mean score
    60.5%
  230. 230
    AnthropicDossier

    Accuracy (best score across scorers)

    Haiku 4.5, Accuracy (best score across scorers): 60.5% (scale 0.0 to 100.0), 95% confidence interval 55.0 to 66.060.5%(55.0–66.0)
    Mean score
    60.5%
  231. 231
    MistralDossier

    Accuracy (best score across scorers)

    Mistral Medium 3, Accuracy (best score across scorers): 59.5% (scale 0.0 to 100.0), 95% confidence interval 54.0 to 65.159.5%(54.0–65.1)
    Mean score
    59.5%
  232. 232
    OpenAIDossier

    medium

    Accuracy (best score across scorers)

    o1-mini, Accuracy (best score across scorers): 59.5% (scale 0.0 to 100.0), 95% confidence interval 54.0 to 65.059.5%(54.0–65.0)
    Mean score
    59.5%
  233. 233
    OpenAIDossier

    low

    Accuracy (best score across scorers)

    GPT-5 nano, Accuracy (best score across scorers): 57.6% (scale 0.0 to 100.0), 95% confidence interval 50.7 to 64.557.6%(50.7–64.5)
    Mean score
    57.6%
  234. 234
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 1.5 Pro (002), Accuracy (best score across scorers): 57.2% (scale 0.0 to 100.0), 95% confidence interval 51.7 to 62.757.2%(51.7–62.7)
    Mean score
    57.2%
  235. 235
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 2.0 Flash Thinking Experimental (01-21), Accuracy (best score across scorers): 57.1% (scale 0.0 to 100.0), 95% confidence interval 50.2 to 64.057.1%(50.2–64.0)
    Mean score
    57.1%
  236. 236
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3 8B, Accuracy (best score across scorers): 56.8% (scale 0.0 to 100.0), 95% confidence interval 51.0 to 62.556.8%(51.0–62.5)
    Mean score
    56.8%
  237. 237
    DeepSeekDossier

    Accuracy (best score across scorers)

    DeepSeek V3, Accuracy (best score across scorers): 56.5% (scale 0.0 to 100.0), 95% confidence interval 51.1 to 62.056.5%(51.1–62.0)
    Mean score
    56.5%
  238. 238
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen Max, Accuracy (best score across scorers): 56.1% (scale 0.0 to 100.0), 95% confidence interval 50.7 to 61.656.1%(50.7–61.6)
    Mean score
    56.1%
  239. 239
    MistralDossier

    Accuracy (best score across scorers)

    Magistral Small (2506), Accuracy (best score across scorers): 56.1% (scale 0.0 to 100.0), 95% confidence interval 49.1 to 63.056.1%(49.1–63.0)
    Mean score
    56.1%
  240. 239
    MicrosoftDossier

    Accuracy (best score across scorers)

    Phi-4, Accuracy (best score across scorers): 56.1% (scale 0.0 to 100.0), 95% confidence interval 51.0 to 61.156.1%(51.0–61.1)
    Mean score
    56.1%
  241. 241
    DeepSeekDossier

    Accuracy (best score across scorers)

    DeepSeek R1 Distill Llama 70B, Accuracy (best score across scorers): 55.7% (scale 0.0 to 100.0), 95% confidence interval 49.9 to 61.655.7%(49.9–61.6)
    Mean score
    55.7%
  242. 242
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3 30B A3B 2507 Instruct, Accuracy (best score across scorers): 55.6% (scale 0.0 to 100.0), 95% confidence interval 50.0 to 61.355.6%(50.0–61.3)
    Mean score
    55.6%
  243. 243
    OpenAIDossier

    none

    Accuracy (best score across scorers)

    GPT-5.4 nano, Accuracy (best score across scorers): 55.6% (scale 0.0 to 100.0), 95% confidence interval 48.6 to 62.555.6%(48.6–62.5)
    Mean score
    55.6%
  244. 244
    AnthropicDossier

    Accuracy (best score across scorers)

    Claude 3.5 Sonnet (Oct '24), Accuracy (best score across scorers): 55.3% (scale 0.0 to 100.0), 95% confidence interval 49.8 to 60.855.3%(49.8–60.8)
    Mean score
    55.3%
  245. 245
    AlibabaDossier

    none

    Accuracy (best score across scorers)

    Qwen3 32B, Accuracy (best score across scorers): 54.1% (scale 0.0 to 100.0), 95% confidence interval 48.6 to 59.654.1%(48.6–59.6)
    Mean score
    54.1%
  246. 246
    AnthropicDossier

    Accuracy (best score across scorers)

    Claude 3.5 Sonnet (June '24), Accuracy (best score across scorers): 54% (scale 0.0 to 100.0), 95% confidence interval 48.6 to 59.454%(48.6–59.4)
    Mean score
    54%
  247. 247

    Accuracy (best score across scorers)

    Grok 2 (1212), Accuracy (best score across scorers): 53.8% (scale 0.0 to 100.0), 95% confidence interval 48.5 to 59.153.8%(48.5–59.1)
    Mean score
    53.8%
  248. 248
    AlibabaDossier

    none

    Accuracy (best score across scorers)

    Qwen3 14B, Accuracy (best score across scorers): 53.4% (scale 0.0 to 100.0), 95% confidence interval 48.0 to 58.853.4%(48.0–58.8)
    Mean score
    53.4%
  249. 249
    OpenAIDossier

    low

    Accuracy (best score across scorers)

    gpt-oss-20b, Accuracy (best score across scorers): 53.2% (scale 0.0 to 100.0), 95% confidence interval 48.3 to 58.153.2%(48.3–58.1)
    Mean score
    53.2%
  250. 250
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3 4B, Accuracy (best score across scorers): 52.3% (scale 0.0 to 100.0), 95% confidence interval 46.4 to 58.252.3%(46.4–58.2)
    Mean score
    52.3%
  251. 251

    Accuracy (best score across scorers)

    Llama 4 Scout, Accuracy (best score across scorers): 51.8% (scale 0.0 to 100.0), 95% confidence interval 45.6 to 58.051.8%(45.6–58.0)
    Mean score
    51.8%
  252. 252
    MistralDossier

    Accuracy (best score across scorers)

    Mistral Large (2411), Accuracy (best score across scorers): 51.3% (scale 0.0 to 100.0), 95% confidence interval 46.1 to 56.651.3%(46.1–56.6)
    Mean score
    51.3%
  253. 253

    Accuracy (best score across scorers)

    Llama 3.1 405B Instruct, Accuracy (best score across scorers): 50.9% (scale 0.0 to 100.0), 95% confidence interval 45.8 to 56.050.9%(45.8–56.0)
    Mean score
    50.9%
  254. 254
    AlibabaDossier

    none

    Accuracy (best score across scorers)

    Qwen3 30B A3B, Accuracy (best score across scorers): 50.4% (scale 0.0 to 100.0), 95% confidence interval 44.8 to 55.950.4%(44.8–55.9)
    Mean score
    50.4%
  255. 255
    OpenAIDossier

    Accuracy (best score across scorers)

    o1-preview, Accuracy (best score across scorers): 50.3% (scale 0.0 to 100.0), 95% confidence interval 44.8 to 55.950.3%(44.8–55.9)
    Mean score
    50.3%
  256. 256
    OpenAIDossier

    Accuracy (best score across scorers)

    GPT-4o (Aug '24), Accuracy (best score across scorers): 49.2% (scale 0.0 to 100.0), 95% confidence interval 44.1 to 54.349.2%(44.1–54.3)
    Mean score
    49.2%
  257. 257
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen2.5 72B Instruct, Accuracy (best score across scorers): 49.1% (scale 0.0 to 100.0), 95% confidence interval 43.9 to 54.449.1%(43.9–54.4)
    Mean score
    49.1%
  258. 258
    MistralDossier

    Accuracy (best score across scorers)

    Mistral Small 3.2, Accuracy (best score across scorers): 49.1% (scale 0.0 to 100.0), 95% confidence interval 43.6 to 54.549.1%(43.6–54.5)
    Mean score
    49.1%
  259. 259
    MistralDossier

    Accuracy (best score across scorers)

    Mistral Large 2 (Jul '24), Accuracy (best score across scorers): 49% (scale 0.0 to 100.0), 95% confidence interval 43.9 to 54.149%(43.9–54.1)
    Mean score
    49%
  260. 260
    OpenAIDossier

    Accuracy (best score across scorers)

    GPT-4.1 nano, Accuracy (best score across scorers): 48.9% (scale 0.0 to 100.0), 95% confidence interval 44.1 to 53.848.9%(44.1–53.8)
    Mean score
    48.9%
  261. 261
    OpenAIDossier

    Accuracy (best score across scorers)

    GPT-4o (May '24), Accuracy (best score across scorers): 48.9% (scale 0.0 to 100.0), 95% confidence interval 43.7 to 54.048.9%(43.7–54.0)
    Mean score
    48.9%
  262. 262
    OpenAIDossier

    minimal

    Accuracy (best score across scorers)

    GPT-5 nano, Accuracy (best score across scorers): 48.5% (scale 0.0 to 100.0), 95% confidence interval 41.5 to 55.548.5%(41.5–55.5)
    Mean score
    48.5%
  263. 263
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen Plus, Accuracy (best score across scorers): 48.1% (scale 0.0 to 100.0), 95% confidence interval 42.8 to 53.448.1%(42.8–53.4)
    Mean score
    48.1%
  264. 264
    OpenAIDossier

    Accuracy (best score across scorers)

    GPT-4o (Nov '24), Accuracy (best score across scorers): 47.9% (scale 0.0 to 100.0), 95% confidence interval 42.7 to 53.047.9%(42.7–53.0)
    Mean score
    47.9%
  265. 265
    GoogleDossier

    Accuracy (best score across scorers)

    Gemma 3 27B, Accuracy (best score across scorers): 47.7% (scale 0.0 to 100.0), 95% confidence interval 42.4 to 53.047.7%(42.4–53.0)
    Mean score
    47.7%
  266. 266
    MistralDossier

    Accuracy (best score across scorers)

    Magistral Small 1.2, Accuracy (best score across scorers): 47.6% (scale 0.0 to 100.0), 95% confidence interval 42.5 to 52.747.6%(42.5–52.7)
    Mean score
    47.6%
  267. 267
    MistralDossier

    Accuracy (best score across scorers)

    Mistral Small 3.1, Accuracy (best score across scorers): 47.5% (scale 0.0 to 100.0), 95% confidence interval 42.1 to 52.947.5%(42.1–52.9)
    Mean score
    47.5%
  268. 267
    AlibabaDossier

    none

    Accuracy (best score across scorers)

    Qwen3 8B, Accuracy (best score across scorers): 47.5% (scale 0.0 to 100.0), 95% confidence interval 42.3 to 52.647.5%(42.3–52.6)
    Mean score
    47.5%
  269. 269

    Accuracy (best score across scorers)

    Llama 3.3 Instruct 70B, Accuracy (best score across scorers): 47.4% (scale 0.0 to 100.0), 95% confidence interval 41.9 to 53.047.4%(41.9–53.0)
    Mean score
    47.4%
  270. 270
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 1.5 Flash (002), Accuracy (best score across scorers): 47.3% (scale 0.0 to 100.0), 95% confidence interval 41.9 to 52.847.3%(41.9–52.8)
    Mean score
    47.3%
  271. 271
    MistralDossier

    Accuracy (best score across scorers)

    Mistral Small 3, Accuracy (best score across scorers): 47.3% (scale 0.0 to 100.0), 95% confidence interval 41.8 to 52.847.3%(41.8–52.8)
    Mean score
    47.3%
  272. 272
    AnthropicDossier

    Accuracy (best score across scorers)

    Claude 3 Opus, Accuracy (best score across scorers): 47.2% (scale 0.0 to 100.0), 95% confidence interval 42.0 to 52.347.2%(42.0–52.3)
    Mean score
    47.2%
  273. 273
    OpenAIDossier

    Accuracy (best score across scorers)

    GPT-4 Turbo, Accuracy (best score across scorers): 46.6% (scale 0.0 to 100.0), 95% confidence interval 41.4 to 51.846.6%(41.4–51.8)
    Mean score
    46.6%
  274. 274
    Allen Institute for AIDossier

    Accuracy (best score across scorers)

    Llama 3.1 Tulu 3 70B DPO, Accuracy (best score across scorers): 46.3% (scale 0.0 to 100.0), 95% confidence interval 41.6 to 51.046.3%(41.6–51.0)
    Mean score
    46.3%
  275. 275
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen2.5 32B Instruct, Accuracy (best score across scorers): 46.1% (scale 0.0 to 100.0), 95% confidence interval 41.0 to 51.246.1%(41.0–51.2)
    Mean score
    46.1%
  276. 276
    OpenAIDossier

    high

    Accuracy (best score across scorers)

    gpt-oss-20b, Accuracy (best score across scorers): 46% (scale 0.0 to 100.0), 95% confidence interval 39.0 to 52.946%(39.0–52.9)
    Mean score
    46%
  277. 277
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 1.5 Pro (001), Accuracy (best score across scorers): 45.9% (scale 0.0 to 100.0), 95% confidence interval 40.7 to 51.045.9%(40.7–51.0)
    Mean score
    45.9%
  278. 278
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3 4B Instruct 2507, Accuracy (best score across scorers): 45.8% (scale 0.0 to 100.0), 95% confidence interval 40.2 to 51.445.8%(40.2–51.4)
    Mean score
    45.8%
  279. 279
    MistralDossier

    Accuracy (best score across scorers)

    Mistral Small 3, Accuracy (best score across scorers): 45.3% (scale 0.0 to 100.0), 95% confidence interval 40.4 to 50.245.3%(40.4–50.2)
    Mean score
    45.3%
  280. 280

    none

    Accuracy (best score across scorers)

    GLM-4.7-Flash, Accuracy (best score across scorers): 45.1% (scale 0.0 to 100.0), 95% confidence interval 40.9 to 49.445.1%(40.9–49.4)
    Mean score
    45.1%
  281. 281
    DeepSeekDossier

    Accuracy (best score across scorers)

    DeepSeek R1 Distill Qwen 14B, Accuracy (best score across scorers): 44.7% (scale 0.0 to 100.0), 95% confidence interval 39.1 to 50.344.7%(39.1–50.3)
    Mean score
    44.7%
  282. 282

    Accuracy (best score across scorers)

    Llama 3.1 Instruct 70B, Accuracy (best score across scorers): 44.2% (scale 0.0 to 100.0), 95% confidence interval 39.4 to 49.044.2%(39.4–49.0)
    Mean score
    44.2%
  283. 283
    AlibabaDossier

    none

    Accuracy (best score across scorers)

    Qwen3 4B, Accuracy (best score across scorers): 43.6% (scale 0.0 to 100.0), 95% confidence interval 38.3 to 48.843.6%(38.3–48.8)
    Mean score
    43.6%
  284. 284
    MicrosoftDossier

    Accuracy (best score across scorers)

    WizardLM-2 8x22B, Accuracy (best score across scorers): 43.4% (scale 0.0 to 100.0), 95% confidence interval 38.9 to 48.043.4%(38.9–48.0)
    Mean score
    43.4%
  285. 285
    OpenAIDossier

    Accuracy (best score across scorers)

    GPT-4 Turbo Preview (1106), Accuracy (best score across scorers): 42.4% (scale 0.0 to 100.0), 95% confidence interval 37.6 to 47.142.4%(37.6–47.1)
    Mean score
    42.4%
  286. 286
    OpenAIDossier

    Accuracy (best score across scorers)

    GPT-4 Turbo Preview (0125), Accuracy (best score across scorers): 42.3% (scale 0.0 to 100.0), 95% confidence interval 37.0 to 47.542.3%(37.0–47.5)
    Mean score
    42.3%
  287. 287
    MistralDossier

    Accuracy (best score across scorers)

    Mistral Small 3.1, Accuracy (best score across scorers): 41.9% (scale 0.0 to 100.0), 95% confidence interval 37.3 to 46.541.9%(37.3–46.5)
    Mean score
    41.9%
  288. 288
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen Turbo, Accuracy (best score across scorers): 41.8% (scale 0.0 to 100.0), 95% confidence interval 36.7 to 46.941.8%(36.7–46.9)
    Mean score
    41.8%
  289. 289

    Accuracy (best score across scorers)

    Llama 3.2 90B Vision Instruct, Accuracy (best score across scorers): 41% (scale 0.0 to 100.0), 95% confidence interval 36.5 to 45.541%(36.5–45.5)
    Mean score
    41%
  290. 290
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen2 72B Instruct, Accuracy (best score across scorers): 40.8% (scale 0.0 to 100.0), 95% confidence interval 35.7 to 45.840.8%(35.7–45.8)
    Mean score
    40.8%
  291. 291
    AnthropicDossier

    Accuracy (best score across scorers)

    Claude 3 Sonnet, Accuracy (best score across scorers): 40.6% (scale 0.0 to 100.0), 95% confidence interval 35.9 to 45.340.6%(35.9–45.3)
    Mean score
    40.6%
  292. 292

    Accuracy (best score across scorers)

    Llama 3 Instruct 70B, Accuracy (best score across scorers): 40.6% (scale 0.0 to 100.0), 95% confidence interval 35.4 to 45.740.6%(35.4–45.7)
    Mean score
    40.6%
  293. 293
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 1.5 Flash (001), Accuracy (best score across scorers): 40.4% (scale 0.0 to 100.0), 95% confidence interval 35.7 to 45.040.4%(35.7–45.0)
    Mean score
    40.4%
  294. 294
    GoogleDossier

    Accuracy (best score across scorers)

    Gemma 3 12B, Accuracy (best score across scorers): 39.5% (scale 0.0 to 100.0), 95% confidence interval 34.5 to 44.439.5%(34.5–44.4)
    Mean score
    39.5%
  295. 295
    MistralDossier

    Accuracy (best score across scorers)

    Mistral Large (Feb '24), Accuracy (best score across scorers): 38.8% (scale 0.0 to 100.0), 95% confidence interval 33.9 to 43.638.8%(33.9–43.6)
    Mean score
    38.8%
  296. 296
    AnthropicDossier

    Accuracy (best score across scorers)

    Claude 3.5 Haiku, Accuracy (best score across scorers): 38.1% (scale 0.0 to 100.0), 95% confidence interval 33.0 to 43.338.1%(33.0–43.3)
    Mean score
    38.1%
  297. 297
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen3 1.7B, Accuracy (best score across scorers): 38% (scale 0.0 to 100.0), 95% confidence interval 32.6 to 43.438%(32.6–43.4)
    Mean score
    38%
  298. 298
    OpenAIDossier

    Accuracy (best score across scorers)

    GPT-4o mini, Accuracy (best score across scorers): 37.7% (scale 0.0 to 100.0), 95% confidence interval 33.0 to 42.437.7%(33.0–42.4)
    Mean score
    37.7%
  299. 299
    Nous ResearchDossier

    Accuracy (best score across scorers)

    Hermes 2 Theta Llama 3 70B, Accuracy (best score across scorers): 37.5% (scale 0.0 to 100.0), 95% confidence interval 32.6 to 42.337.5%(32.6–42.3)
    Mean score
    37.5%
  300. 300
    GoogleDossier

    Accuracy (best score across scorers)

    Gemma 2 27B IT, Accuracy (best score across scorers): 36.5% (scale 0.0 to 100.0), 95% confidence interval 31.8 to 41.236.5%(31.8–41.2)
    Mean score
    36.5%
  301. 301
    AnthropicDossier

    Accuracy (best score across scorers)

    Claude 3 Haiku, Accuracy (best score across scorers): 36.3% (scale 0.0 to 100.0), 95% confidence interval 31.6 to 41.036.3%(31.6–41.0)
    Mean score
    36.3%
  302. 302
    OpenAIDossier

    Accuracy (best score across scorers)

    GPT-4 (0314), Accuracy (best score across scorers): 35.7% (scale 0.0 to 100.0), 95% confidence interval 31.1 to 40.435.7%(31.1–40.4)
    Mean score
    35.7%
  303. 303
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen2.5 7B Instruct, Accuracy (best score across scorers): 35.5% (scale 0.0 to 100.0), 95% confidence interval 31.0 to 40.035.5%(31.0–40.0)
    Mean score
    35.5%
  304. 304
    AnthropicDossier

    Accuracy (best score across scorers)

    Claude 2.0, Accuracy (best score across scorers): 34.7% (scale 0.0 to 100.0), 95% confidence interval 29.7 to 39.634.7%(29.7–39.6)
    Mean score
    34.7%
  305. 305
    MistralDossier

    Accuracy (best score across scorers)

    Mixtral 8x22B Instruct v0.1, Accuracy (best score across scorers): 34.1% (scale 0.0 to 100.0), 95% confidence interval 30.0 to 38.134.1%(30.0–38.1)
    Mean score
    34.1%
  306. 306
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 1.0 Pro (001), Accuracy (best score across scorers): 34% (scale 0.0 to 100.0), 95% confidence interval 30.0 to 37.934%(30.0–37.9)
    Mean score
    34%
  307. 307
    DeepSeekDossier

    Accuracy (best score across scorers)

    DeepSeek R1 Distill Qwen 1.5B, Accuracy (best score across scorers): 33.6% (scale 0.0 to 100.0), 95% confidence interval 29.5 to 37.633.6%(29.5–37.6)
    Mean score
    33.6%
  308. 308
    AnthropicDossier

    Accuracy (best score across scorers)

    Claude 2.1, Accuracy (best score across scorers): 33% (scale 0.0 to 100.0), 95% confidence interval 28.4 to 37.533%(28.4–37.5)
    Mean score
    33%
  309. 308
    GoogleDossier

    Accuracy (best score across scorers)

    Gemini 1.5 Flash-8B (001), Accuracy (best score across scorers): 33% (scale 0.0 to 100.0), 95% confidence interval 28.2 to 37.733%(28.2–37.7)
    Mean score
    33%
  310. 310
    DatabricksDossier

    Accuracy (best score across scorers)

    DBRX Instruct, Accuracy (best score across scorers): 32.9% (scale 0.0 to 100.0), 95% confidence interval 26.5 to 39.332.9%(26.5–39.3)
    Mean score
    32.9%
  311. 311
    01.AIDossier

    Accuracy (best score across scorers)

    Yi 1.5 34B Chat, Accuracy (best score across scorers): 32% (scale 0.0 to 100.0), 95% confidence interval 28.1 to 35.932%(28.1–35.9)
    Mean score
    32%
  312. 312
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen1.5 32B Chat, Accuracy (best score across scorers): 30.7% (scale 0.0 to 100.0), 95% confidence interval 26.8 to 34.730.7%(26.8–34.7)
    Mean score
    30.7%
  313. 313
    OpenAIDossier

    Accuracy (best score across scorers)

    GPT-4 (0613), Accuracy (best score across scorers): 30.7% (scale 0.0 to 100.0), 95% confidence interval 26.1 to 35.230.7%(26.1–35.2)
    Mean score
    30.7%
  314. 314
    AlibabaDossier

    none

    Accuracy (best score across scorers)

    Qwen3 1.7B, Accuracy (best score across scorers): 30.6% (scale 0.0 to 100.0), 95% confidence interval 26.2 to 35.030.6%(26.2–35.0)
    Mean score
    30.6%
  315. 315
    MistralDossier

    Accuracy (best score across scorers)

    Mixtral 8x7B Instruct, Accuracy (best score across scorers): 30.6% (scale 0.0 to 100.0), 95% confidence interval 26.8 to 34.430.6%(26.8–34.4)
    Mean score
    30.6%
  316. 316
    MistralDossier

    Accuracy (best score across scorers)

    Mistral NeMo (2407), Accuracy (best score across scorers): 29.9% (scale 0.0 to 100.0), 95% confidence interval 25.8 to 34.029.9%(25.8–34.0)
    Mean score
    29.9%
  317. 317
    MistralDossier

    Accuracy (best score across scorers)

    Mixtral 8x7B Instruct, Accuracy (best score across scorers): 29.8% (scale 0.0 to 100.0), 95% confidence interval 26.1 to 33.629.8%(26.1–33.6)
    Mean score
    29.8%
  318. 318
    AlibabaDossier

    Accuracy (best score across scorers)

    Qwen1.5 72B Chat, Accuracy (best score across scorers): 28.8% (scale 0.0 to 100.0), 95% confidence interval 24.9 to 32.728.8%(24.9–32.7)
    Mean score
    28.8%
  319. 319

    Accuracy (best score across scorers)

    Granite 4.0 Micro, Accuracy (best score across scorers): 28.3% (scale 0.0 to 100.0), 95% confidence interval 24.7 to 31.928.3%(24.7–31.9)
    Mean score
    28.3%
  320. 320
    OpenAIDossier

    Accuracy (best score across scorers)

    GPT-3.5 Turbo (1106), Accuracy (best score across scorers): 28% (scale 0.0 to 100.0), 95% confidence interval 24.4 to 31.728%(24.4–31.7)
    Mean score
    28%
  321. 321
    MicrosoftDossier

    Accuracy (best score across scorers)

    Phi-3 Medium 128K Instruct, Accuracy (best score across scorers): 27.6% (scale 0.0 to 100.0), 95% confidence interval 24.4 to 30.727.6%(24.4–30.7)
    Mean score
    27.6%
  322. 322
    GoogleDossier

    Accuracy (best score across scorers)

    Gemma 2 9B IT, Accuracy (best score across scorers): 27.5% (scale 0.0 to 100.0), 95% confidence interval 22.9 to 32.027.5%(22.9–32.0)
    Mean score
    27.5%
  323. 323
    OpenAIDossier

    Accuracy (best score across scorers)

    GPT-3.5 Turbo (0125), Accuracy (best score across scorers): 27.2% (scale 0.0 to 100.0), 95% confidence interval 23.3 to 31.027.2%(23.3–31.0)
    Mean score
    27.2%
  324. 324
    MistralDossier

    Accuracy (best score across scorers)

    Ministral 8B (2410), Accuracy (best score across scorers): 27.1% (scale 0.0 to 100.0), 95% confidence interval 23.1 to 31.227.1%(23.1–31.2)
    Mean score
    27.1%
  325. 325

    Accuracy (best score across scorers)

    Llama 3.1 Instruct 8B, Accuracy (best score across scorers): 27% (scale 0.0 to 100.0), 95% confidence interval 23.3 to 30.627%(23.3–30.6)
    Mean score
    27%
  326. 326

    Accuracy (best score across scorers)

    Llama 2 70B Chat, Accuracy (best score across scorers): 26.3% (scale 0.0 to 100.0), 95% confidence interval 22.4 to 30.226.3%(22.4–30.2)
    Mean score
    26.3%
  327. 327

    Accuracy (best score across scorers)

    Llama 3 Instruct 8B, Accuracy (best score across scorers): 26.1% (scale 0.0 to 100.0), 95% confidence interval 22.7 to 29.426.1%(22.7–29.4)
    Mean score
    26.1%
  328. 328
    MistralDossier

    Accuracy (best score across scorers)

    Ministral 3B (2410), Accuracy (best score across scorers): 25.3% (scale 0.0 to 100.0), 95% confidence interval 21.5 to 29.025.3%(21.5–29.0)
    Mean score
    25.3%
  329. 329
    DeepSeekDossier

    Accuracy (best score across scorers)

    DeepSeek LLM 67B Chat, Accuracy (best score across scorers): 24.6% (scale 0.0 to 100.0), 95% confidence interval 22.0 to 27.324.6%(22.0–27.3)
    Mean score
    24.6%
  330. 330

    Accuracy (best score across scorers)

    Granite 4.0 1B, Accuracy (best score across scorers): 24% (scale 0.0 to 100.0), 95% confidence interval 20.8 to 27.224%(20.8–27.2)
    Mean score
    24%
  331. 331

    Accuracy (best score across scorers)

    Llama 3.2 1B Instruct, Accuracy (best score across scorers): 23.9% (scale 0.0 to 100.0), 95% confidence interval 20.7 to 27.123.9%(20.7–27.1)
    Mean score
    23.9%
  332. 332
    GoogleDossier

    Accuracy (best score across scorers)

    Gemma 3 4B, Accuracy (best score across scorers): 23.2% (scale 0.0 to 100.0), 95% confidence interval 19.6 to 26.923.2%(19.6–26.9)
    Mean score
    23.2%
  333. 333
    GoogleDossier

    Accuracy (best score across scorers)

    Gemma 3 1B, Accuracy (best score across scorers): 19.9% (scale 0.0 to 100.0), 95% confidence interval 17.0 to 22.919.9%(17.0–22.9)
    Mean score
    19.9%
  334. 334
    MistralDossier

    Accuracy (best score across scorers)

    Mistral 7B Instruct v0.3, Accuracy (best score across scorers): 15.2% (scale 0.0 to 100.0), 95% confidence interval 13.2 to 17.215.2%(13.2–17.2)
    Mean score
    15.2%
  335. 335
    01.AIDossier

    Accuracy (best score across scorers)

    Yi 34B Chat, Accuracy (best score across scorers): 14.7% (scale 0.0 to 100.0), 95% confidence interval 12.5 to 17.014.7%(12.5–17.0)
    Mean score
    14.7%
  336. 336
    MistralDossier

    Accuracy (best score across scorers)

    Mistral 7B Instruct, Accuracy (best score across scorers): 13.2% (scale 0.0 to 100.0), 95% confidence interval 11.3 to 15.213.2%(11.3–15.2)
    Mean score
    13.2%
  337. 337

    Accuracy (best score across scorers)

    Granite 4.0 350M, Accuracy (best score across scorers): 11.2% (scale 0.0 to 100.0), 95% confidence interval 9.5 to 13.011.2%(9.5–13.0)
    Mean score
    11.2%
  338. 338
    DeepSeekDossier

    Accuracy (best score across scorers)

    DeepSeek R1 0528 Qwen3 8B, Accuracy (best score across scorers): 9.3% (scale 0.0 to 100.0), 95% confidence interval 7.2 to 11.49.3%(7.2–11.4)
    Mean score
    9.3%
Unmatched upstream rows excluded from this board (2)

These upstream rows could not be mapped cleanly to the canonical model registry or reasoning effort enum.

Raw nameParsed modelReasonDetail
Eurus-2-7B-PRIMEEurus-2-7B-PRIMEno-aliasaccuracy
Eurus-2-7B-PRIMEEurus-2-7B-PRIMEno-aliasmean-score