Acropolis

Evidence coverage and bias audits

Model evidence grades (A to D), rank probability per board, #1 tie bands, single-run exposure, and vendor self-preference DiD checks.

Rank probability

A board lists one rank per model, but each score carries error. We redraw every score 2,000 times within its 95% interval, rank the board each time, and count how often a model lands on each rank. Rows without an interval stay fixed, and errors are treated as independent.

Listed #1 below 50%
11 of 27 boards
Most certain leader
100%
GPT Image 2.5 Sunburst, LMArena Text-to-Image (85 rows)
Least certain leader
19%
Gemini 4 Carbon, GPQA diamond (338 rows)

Rows are listed best first on Accuracy (best score across scorers). Each cell is the share of draws in which the model holds that rank; the number is shown from 5%.

Model123456789101112P(#1)Exp. rankGemini 4 Carbon (high): P(#1) 19%, expected rank 5.5Gemini 4 Carbon (high)1915121076619%5.5Gemini 4 Argon (high): P(#1) 16%, expected rank 6.0Gemini 4 Argon (high)1614121086616%6.0Fable 5.2 (max): P(#1) 10%, expected rank 8.0Fable 5.2 (max)10109977610%8.0GPT-6 Astra (max): P(#1) 9%, expected rank 9.6GPT-6 Astra (max)9886675659%9.6Sonnet 5.5 (max): P(#1) 8%, expected rank 10.4Sonnet 5.5 (max)88776658%10.4Gemini 3.8 Flash (high): P(#1) 6%, expected rank 11.6Gemini 3.8 Flash (high)665656566%11.6GPT-6.1 Sol (max): P(#1) 5%, expected rank 11.7GPT-6.1 Sol (max)666655%11.7Gemini 3.7 Flash (high): P(#1) 2%, expected rank 15.1Gemini 3.7 Flash (high)2%15.1GPT-5.4 Pro (xhigh): P(#1) 3%, expected rank 17.1GPT-5.4 Pro (xhigh)3%17.1Gemini 3.1 Pro Preview (high): P(#1) 2%, expected rank 18.4Gemini 3.1 Pro Preview (high)2%18.4GPT-6.1 Terra (max): P(#1) 2%, expected rank 19.4GPT-6.1 Terra (max)2%19.4GPT-6 Sol (max): P(#1) 2%, expected rank 19.1GPT-6 Sol (max)2%19.1
Method, limits, and excluded inputs (27 boards, 2802 rows read)

Method rules

  • Only the board's rank metric is used, and hidden rows are left out.
  • A board is simulated when at least 5 of its rows have an interval (source CI, else derived from the standard error).
  • Each draw gives every row a normal score: mean is the listed score, spread is the interval half-width divided by 1.96.
  • Rows without an interval are held fixed at their listed score. Every row of the board takes part in the ranking, not only the top 12.
  • Each draw ranks the rows best first. Equal scores in a draw keep the listed order, so a fixed board reproduces its listed ranks.
  • P(rank = k) is the share of the 2,000 draws in which a row lands at rank k. The top 12 rows are reported, with P(#1), P(top 3) and the expected rank.
  • Draws are seeded from the board key, the metric and the row reference, so the numbers are identical on every run.

Known limits

  • Row errors are treated as independent. Models tested on the same questions have correlated errors, which would narrow the true rank uncertainty.
  • The spread comes from a normal approximation of the interval; intervals from few questions or near 0% and 100% are rough.
  • A row with no interval looks certain in the draws, which makes the rows around it look more certain than they are.
  • Probabilities describe sampling noise in the published scores. They do not cover test-set choice, prompt changes or contamination.

Exclusions

  • 25 boards excluded: board has fewer than 5 rows with an interval (not simulated)
  • 94 boards excluded: not the default slice of its source and version
  • 1 boards excluded: older version of a benchmark the source reports again

Statistical ties for #1 per board

BenchmarkRowsTied #1 countSingle-run rows
ARC-AGI-2Epoch AI Benchmarking Hub299Clear #10
Bug Hunt BenchBug Hunt Bench145Clear #163
CursorBenchCursorBench91Clear #10
FrontierCodeEpoch AI Benchmarking Hub51Clear #10
FrontierMath-2025-02-28-PrivateEpoch AI Benchmarking Hub101Tied #1 (6)0
GDPvalEpoch AI Benchmarking Hub11Clear #10
GPQA diamondEpoch AI Benchmarking Hub338Tied #1 (35)0
HLEEpoch AI Benchmarking Hub56Clear #10
LMArena AgentLMArena64Clear #10
LMArena Document (style control)LMArena46Tied #1 (6)0
LMArena DocumentLMArena46Tied #1 (4)0
LMArena Image EditLMArena20Clear #10
LMArena Image EditLMArena37Clear #10
LMArena Image EditLMArena14Clear #10
LMArena Image EditLMArena48Clear #10
LMArena Image-to-VideoLMArena2Clear #10
LMArena Search (style control)LMArena42Tied #1 (2)0
LMArena SearchLMArena42Tied #1 (4)0
LMArena Text (style control)LMArena395Tied #1 (5)0
LMArena Text (style control)LMArena416Tied #1 (6)0
LMArena Text (style control)LMArena419Tied #1 (3)0
LMArena Text (style control)LMArena421Tied #1 (4)0
LMArena Text (style control)LMArena421Tied #1 (2)0
LMArena Text (style control)LMArena371Tied #1 (6)0
LMArena Text (style control)LMArena301Tied #1 (6)0
LMArena Text (style control)LMArena320Clear #10
LMArena Text (style control)LMArena421Tied #1 (2)0
LMArena Text (style control)LMArena419Tied #1 (9)0
LMArena Text (style control)LMArena414Tied #1 (6)0
LMArena Text (style control)LMArena419Tied #1 (2)0
LMArena Text (style control)LMArena393Tied #1 (10)0
LMArena Text (style control)LMArena419Tied #1 (8)0
LMArena Text (style control)LMArena402Tied #1 (11)0
LMArena Text (style control)LMArena391Tied #1 (20)0
LMArena Text (style control)LMArena421Tied #1 (10)0
LMArena Text (style control)LMArena420Tied #1 (3)0
LMArena Text (style control)LMArena421Tied #1 (3)0
LMArena Text (style control)LMArena288Tied #1 (6)0
LMArena Text (style control)LMArena289Tied #1 (28)0
LMArena Text (style control)LMArena399Tied #1 (2)0
LMArena Text (style control)LMArena404Tied #1 (8)0
LMArena Text (style control)LMArena419Tied #1 (2)0
LMArena Text (style control)LMArena421Tied #1 (2)0
LMArena Text (style control)LMArena421Tied #1 (2)0
LMArena Text (style control)LMArena243Tied #1 (18)0
LMArena Text (style control)LMArena385Tied #1 (3)0
LMArena Text (style control)LMArena304Tied #1 (26)0
LMArena Text-to-ImageLMArena85Tied #1 (2)0
LMArena Text-to-ImageLMArena28Tied #1 (2)0
LMArena Text-to-ImageLMArena85Tied #1 (2)0
LMArena Text-to-ImageLMArena85Tied #1 (2)0
LMArena Text-to-ImageLMArena8Tied #1 (3)0
LMArena Text-to-ImageLMArena85Clear #10
LMArena Text-to-ImageLMArena8Clear #10
LMArena Text-to-ImageLMArena85Clear #10
LMArena Text-to-ImageLMArena85Clear #10
LMArena Text-to-ImageLMArena85Tied #1 (4)0
LMArena Text-to-ImageLMArena8Tied #1 (2)0
LMArena Text-to-ImageLMArena85Clear #10
LMArena Text-to-VideoLMArena2Clear #10
LMArena TextLMArena395Tied #1 (3)0
LMArena TextLMArena416Tied #1 (2)0
LMArena TextLMArena419Tied #1 (3)0
LMArena TextLMArena421Tied #1 (3)0
LMArena TextLMArena421Tied #1 (2)0
LMArena TextLMArena371Tied #1 (6)0
LMArena TextLMArena301Tied #1 (2)0
LMArena TextLMArena320Clear #10
LMArena TextLMArena421Tied #1 (2)0
LMArena TextLMArena419Tied #1 (5)0
LMArena TextLMArena414Tied #1 (4)0
LMArena TextLMArena419Tied #1 (4)0
LMArena TextLMArena393Tied #1 (13)0
LMArena TextLMArena419Tied #1 (4)0
LMArena TextLMArena402Tied #1 (8)0
LMArena TextLMArena391Tied #1 (8)0
LMArena TextLMArena421Tied #1 (2)0
LMArena TextLMArena420Tied #1 (3)0
LMArena TextLMArena421Tied #1 (4)0
LMArena TextLMArena288Tied #1 (5)0
LMArena TextLMArena289Tied #1 (13)0
LMArena TextLMArena399Tied #1 (2)0
LMArena TextLMArena404Tied #1 (7)0
LMArena TextLMArena419Tied #1 (2)0
LMArena TextLMArena421Tied #1 (2)0
LMArena TextLMArena421Tied #1 (2)0
LMArena TextLMArena243Tied #1 (15)0
LMArena TextLMArena385Tied #1 (2)0
LMArena TextLMArena304Tied #1 (9)0
LMArena Vision (style control)LMArena34Tied #1 (8)0
LMArena Vision (style control)LMArena135Tied #1 (18)0
LMArena Vision (style control)LMArena34Clear #10
LMArena Vision (style control)LMArena106Tied #1 (8)0
LMArena Vision (style control)LMArena123Tied #1 (3)0
LMArena Vision (style control)LMArena167Tied #1 (11)0
LMArena Vision (style control)LMArena52Tied #1 (10)0
LMArena Vision (style control)LMArena118Tied #1 (10)0
LMArena Vision (style control)LMArena102Tied #1 (8)0
LMArena Vision (style control)LMArena123Tied #1 (2)0
LMArena Vision (style control)LMArena167Tied #1 (3)0
LMArena VisionLMArena34Tied #1 (7)0
LMArena VisionLMArena135Tied #1 (8)0
LMArena VisionLMArena34Clear #10
LMArena VisionLMArena106Tied #1 (12)0
LMArena VisionLMArena123Tied #1 (9)0
LMArena VisionLMArena167Tied #1 (8)0
LMArena VisionLMArena52Tied #1 (6)0
LMArena VisionLMArena118Tied #1 (5)0
LMArena VisionLMArena102Tied #1 (8)0
LMArena VisionLMArena123Tied #1 (5)0
LMArena VisionLMArena167Tied #1 (6)0
LMArena WebDevLMArena64Tied #1 (2)0
LMArena WebDevLMArena156Tied #1 (2)0
LMArena WebDevLMArena156Tied #1 (2)0
LMArena WebDevLMArena155Tied #1 (4)0
LMArena WebDevLMArena140Tied #1 (2)0
METR time horizonMETR time horizons3Clear #10
METR time horizonMETR time horizons23Tied #1 (23)0
OSWorldEpoch AI Benchmarking Hub38Clear #10
SWE-Bench verifiedEpoch AI Benchmarking Hub35Tied #1 (5)0
Terminal BenchEpoch AI Benchmarking Hub202Tied #1 (8)0
Code MigrationVals AI benchmarks38Tied #1 (5)0
EMBVals AI benchmarks38Tied #1 (16)0
Finance AgentVals AI benchmarks38Tied #1 (15)0
Vals IndexVals AI benchmarks38Tied #1 (5)0
Legal Agent Benchmark (HLAB-Vals)Vals AI benchmarks38Tied #1 (8)0
Legal ResearchVals AI benchmarks38Tied #1 (12)0
Tax AgentVals AI benchmarks38Tied #1 (8)0
Terminal-Bench 4Vals AI benchmarks38Tied #1 (7)0
Vibe Code BenchVals AI benchmarks38Tied #1 (7)0
Vending-Bench 2Epoch AI Benchmarking Hub72Clear #10

Evidence coverage grades

ModelGradeIndependent sourcesVendor sourcesRecency
DeepSeek V4.1 Flash 1017Grade A. Evidence grade A of A to D: strongest.412d (recent)
Fable 5.2Grade A. Evidence grade A of A to D: strongest.412d (recent)
Gemini 3.8 FlashGrade A. Evidence grade A of A to D: strongest.412d (recent)
Gemini 4 ArgonGrade A. Evidence grade A of A to D: strongest.422d (recent)
Gemini 4 CarbonGrade A. Evidence grade A of A to D: strongest.412d (recent)
GLM-5.3Grade A. Evidence grade A of A to D: strongest.422d (recent)
GLM-5.3 FlashGrade A. Evidence grade A of A to D: strongest.422d (recent)
GPT-6 AstraGrade A. Evidence grade A of A to D: strongest.412d (recent)
GPT-6 LunaGrade A. Evidence grade A of A to D: strongest.412d (recent)
GPT-6.1 SolGrade A. Evidence grade A of A to D: strongest.412d (recent)
GPT-6.1 TerraGrade A. Evidence grade A of A to D: strongest.402d (recent)
Grok 4.7Grade A. Evidence grade A of A to D: strongest.422d (recent)
Haiku 5.5Grade A. Evidence grade A of A to D: strongest.422d (recent)
Kimi K3Grade A. Evidence grade A of A to D: strongest.412d (recent)
Kimi K3.1 ProGrade A. Evidence grade A of A to D: strongest.402d (recent)
Muse Wonder 1.0Grade A. Evidence grade A of A to D: strongest.412d (recent)
Opus 5.5Grade A. Evidence grade A of A to D: strongest.422d (recent)
Sonnet 5.5Grade A. Evidence grade A of A to D: strongest.422d (recent)
Beam 1.0Grade A. Evidence grade A of A to D: strongest.302d (recent)
Claude 3 OpusGrade A. Evidence grade A of A to D: strongest.306d (recent)
Claude 3.5 Sonnet (June '24)Grade A. Evidence grade A of A to D: strongest.306d (recent)
Claude 3.5 Sonnet (Oct '24)Grade A. Evidence grade A of A to D: strongest.306d (recent)
Claude 3.7 SonnetGrade A. Evidence grade A of A to D: strongest.306d (recent)
DeepSeek V4 Pro 0813Grade A. Evidence grade A of A to D: strongest.312d (recent)
DeepSeek V4-ProGrade A. Evidence grade A of A to D: strongest.312d (recent)
DeepSeek V4.1 FlashGrade A. Evidence grade A of A to D: strongest.312d (recent)
Fable 5Grade A. Evidence grade A of A to D: strongest.312d (recent)
Fable 5.1Grade A. Evidence grade A of A to D: strongest.322d (recent)
Gemini 3 Pro PreviewGrade A. Evidence grade A of A to D: strongest.312d (recent)
Gemini 3.1 Pro PreviewGrade A. Evidence grade A of A to D: strongest.302d (recent)
Gemini 3.7 FlashGrade A. Evidence grade A of A to D: strongest.312d (recent)
Gemini 4 HydrogenGrade A. Evidence grade A of A to D: strongest.312d (recent)
Gemma 4 31BGrade A. Evidence grade A of A to D: strongest.302d (recent)
GPT-4 (0314)Grade A. Evidence grade A of A to D: strongest.306d (recent)
GPT-4 TurboGrade A. Evidence grade A of A to D: strongest.306d (recent)

Vendor self-preference audit

Not enough data: 1 vendor-run board (need 2).

LabPaired modelsMedian vendor pctMedian independent pctSignal
alibaba14.473.7Insufficient sample
anthropic1572.890.9no-clear-difference
deepseek528.970.3lower-on-vendor-boards
google861.186.1lower-on-vendor-boards
meta247.581.2Insufficient sample
minimax111.156.0Insufficient sample
mistral140.064.1Insufficient sample
moonshot-ai248.954.4Insufficient sample
openai1451.767.0no-clear-difference
thinking-machines18.95.4Insufficient sample
xai454.066.5lower-on-vendor-boards
zai324.268.9lower-on-vendor-boards
Method, limits, and excluded inputs (131 boards, 28249 rows read)

Method rules

  • Grid: for each model and source, metric values on visible rows, boards, benchmarks, newest observation date, interval/stderr presence, and run counts.
  • Quality evidence counts a row on a board whose rank metric is a quality score; independent sources count once per model.
  • Recency uses observation dates (`observedAt`), never source snapshot dates (`lastListedAt`).
  • Grade A / strong: >= 3 independent sources, newest observation <= 90 days old, and precise. Grade B / good: >= 2 independent sources and <= 365 days. Grade C / thin: >= 1 independent source. Grade D / unverified: 0 independent quality evidence.

Known limits

  • The grade counts evidence; it does not rank capability.
  • Undated sources (e.g. Artificial Analysis, CursorBench) contribute to source count but cannot verify recency.

Exclusions

  • 75 rows excluded: hidden row (for example superseded)