Acropolis

Consensus and capability dispute

Paired-comparison Bradley-Terry standings across shared boards, the Arena preference gap, domain fingerprints, and Bug Hunt team coverage.

Bradley-Terry consensus capability standings

#ModelLatent score (0–100)Median pctIQR spreadAgreement class
1Gemini 4 Carbon

18 boards across 4 indep. sources

: 100.0 (scale 0.0 to 100.0), 95% confidence interval 93.4 to 100.0100.0(93.4–100.0)
98th9.5consistent
2Fable 5.2

15 boards across 4 indep. sources

: 95.3 (scale 0.0 to 100.0), 95% confidence interval 83.7 to 100.095.3(83.7–100.0)
99th4.0consistent
3Gemini 4 Argon

19 boards across 4 indep. sources

: 92.8 (scale 0.0 to 100.0), 95% confidence interval 88.1 to 95.592.8(88.1–95.5)
92th13.9consistent
4Opus 5.5

23 boards across 4 indep. sources

: 92.4 (scale 0.0 to 100.0), 95% confidence interval 84.8 to 99.092.4(84.8–99.0)
92th10.8consistent
5Sonnet 5.5

22 boards across 4 indep. sources

: 91.6 (scale 0.0 to 100.0), 95% confidence interval 83.7 to 96.891.6(83.7–96.8)
92th12.7consistent
6Gemini 4 Hydrogen

15 boards across 3 indep. sources

: 91.1 (scale 0.0 to 100.0), 95% confidence interval 84.1 to 97.391.1(84.1–97.3)
89th17.6consistent
7Muse Wonder Full 1.0

12 boards across 3 indep. sources

: 90.9 (scale 0.0 to 100.0), 95% confidence interval 85.0 to 94.690.9(85.0–94.6)
84th7.8consistent
8Fable 5.1

16 boards across 3 indep. sources

: 89.8 (scale 0.0 to 100.0), 95% confidence interval 81.4 to 94.289.8(81.4–94.2)
92th10.3consistent
9Muse Wonder 1.0

16 boards across 4 indep. sources

: 85.0 (scale 0.0 to 100.0), 95% confidence interval 78.7 to 88.385.0(78.7–88.3)
76th19.0consistent
10Fable 5

15 boards across 3 indep. sources

: 84.2 (scale 0.0 to 100.0), 95% confidence interval 76.6 to 90.884.2(76.6–90.8)
92th17.7consistent
11GPT-5.5 Pre-release

3 boards across 1 indep. sources

: 83.7 (scale 0.0 to 100.0), 95% confidence interval 72.0 to 89.783.7(72.0–89.7)
97th2.6consistent
12GPT-6 Astra

25 boards across 4 indep. sources

: 83.5 (scale 0.0 to 100.0), 95% confidence interval 74.5 to 90.283.5(74.5–90.2)
86th18.2consistent
13GPT-5.6 Sol

16 boards across 3 indep. sources

: 83.5 (scale 0.0 to 100.0), 95% confidence interval 76.6 to 90.383.5(76.6–90.3)
90th13.2consistent
14Opus 5

16 boards across 3 indep. sources

: 83.3 (scale 0.0 to 100.0), 95% confidence interval 77.2 to 91.283.3(77.2–91.2)
92th11.1consistent
15GPT-6.1 Sol

22 boards across 4 indep. sources

: 83.1 (scale 0.0 to 100.0), 95% confidence interval 73.3 to 89.683.1(73.3–89.6)
85th17.4consistent
16Qwen4 Max

9 boards across 1 indep. sources

: 82.7 (scale 0.0 to 100.0), 95% confidence interval 76.0 to 89.182.7(76.0–89.1)
73th13.5consistent
17DeepSeek V4.1 Flash 1017

15 boards across 4 indep. sources

: 82.4 (scale 0.0 to 100.0), 95% confidence interval 75.5 to 86.882.4(75.5–86.8)
68th26.7consistent
18Muse Ember

11 boards across 2 indep. sources

: 81.5 (scale 0.0 to 100.0), 95% confidence interval 73.8 to 87.481.5(73.8–87.4)
67th12.2consistent
19GPT-5.4 Pro

4 boards across 1 indep. sources

: 81.3 (scale 0.0 to 100.0), 95% confidence interval 71.5 to 85.381.3(71.5–85.3)
93th9.7consistent
20Opus 4.7

17 boards across 2 indep. sources

: 80.3 (scale 0.0 to 100.0), 95% confidence interval 73.8 to 87.480.3(73.8–87.4)
90th17.8consistent
21Opus 4.6

18 boards across 3 indep. sources

: 80.3 (scale 0.0 to 100.0), 95% confidence interval 71.7 to 87.480.3(71.7–87.4)
91th17.0consistent
22GPT-5.5

14 boards across 2 indep. sources

: 79.9 (scale 0.0 to 100.0), 95% confidence interval 74.3 to 84.879.9(74.3–84.8)
87th10.3consistent
23Qwen3.8-Max

10 boards across 3 indep. sources

: 79.1 (scale 0.0 to 100.0), 95% confidence interval 69.9 to 87.779.1(69.9–87.7)
92th8.6consistent
24Kimi K3.1 Pro

15 boards across 4 indep. sources

: 78.5 (scale 0.0 to 100.0), 95% confidence interval 69.5 to 84.278.5(69.5–84.2)
70th39.9within-domain-dispute. Evidence grade C of A to D: thin.
25Muse Spark 1.3

10 boards across 2 indep. sources

: 78.2 (scale 0.0 to 100.0), 95% confidence interval 72.4 to 84.778.2(72.4–84.7)
79th24.1consistent
26Qwen3.8 Max Internal

9 boards across 1 indep. sources

: 77.8 (scale 0.0 to 100.0), 95% confidence interval 64.5 to 86.677.8(64.5–86.6)
59th13.5consistent
27GPT-6 Sol

13 boards across 3 indep. sources

: 76.8 (scale 0.0 to 100.0), 95% confidence interval 69.5 to 84.976.8(69.5–84.9)
82th27.0consistent
28Grok 4.7

21 boards across 4 indep. sources

: 76.4 (scale 0.0 to 100.0), 95% confidence interval 70.6 to 80.576.4(70.6–80.5)
68th22.5consistent
29GPT-6.1 Terra

14 boards across 4 indep. sources

: 76.1 (scale 0.0 to 100.0), 95% confidence interval 63.8 to 86.276.1(63.8–86.2)
75th57.3within-domain-dispute. Evidence grade C of A to D: thin.
30Hy4

10 boards across 2 indep. sources

: 75.7 (scale 0.0 to 100.0), 95% confidence interval 66.6 to 81.275.7(66.6–81.2)
59th15.5consistent
31GLM-5.3

18 boards across 4 indep. sources

: 75.6 (scale 0.0 to 100.0), 95% confidence interval 69.0 to 79.975.6(69.0–79.9)
64th35.4within-domain-dispute. Evidence grade C of A to D: thin.
32Gemini 3.7 Flash

12 boards across 3 indep. sources

: 75.6 (scale 0.0 to 100.0), 95% confidence interval 65.9 to 82.175.6(65.9–82.1)
82th34.7domain-specialist. Evidence grade C of A to D: thin.
33Inkling 2 Small

14 boards across 3 indep. sources

: 74.8 (scale 0.0 to 100.0), 95% confidence interval 65.3 to 84.974.8(65.3–84.9)
68th36.7within-domain-dispute. Evidence grade C of A to D: thin.
34MiMo-V2.6-Pro

16 boards across 3 indep. sources

: 74.4 (scale 0.0 to 100.0), 95% confidence interval 67.1 to 79.974.4(67.1–79.9)
65th28.6consistent
35Gemini 3.8 Flash

23 boards across 4 indep. sources

: 73.8 (scale 0.0 to 100.0), 95% confidence interval 65.4 to 80.273.8(65.4–80.2)
64th47.9within-domain-dispute. Evidence grade C of A to D: thin.
36Muse Spark

9 boards across 2 indep. sources

: 73.3 (scale 0.0 to 100.0), 95% confidence interval 63.3 to 81.673.3(63.3–81.6)
89th16.5consistent
37GPT-5.6 Terra

14 boards across 3 indep. sources

: 73.0 (scale 0.0 to 100.0), 95% confidence interval 66.8 to 78.873.0(66.8–78.8)
74th21.5consistent
38Gemini 3.5 Flash

12 boards across 2 indep. sources

: 72.8 (scale 0.0 to 100.0), 95% confidence interval 63.9 to 78.672.8(63.9–78.6)
87th37.0within-domain-dispute. Evidence grade C of A to D: thin.
39Muse Spark 1.1

9 boards across 2 indep. sources

: 72.7 (scale 0.0 to 100.0), 95% confidence interval 63.5 to 79.372.7(63.5–79.3)
79th22.9consistent
40GPT-5.4

17 boards across 3 indep. sources

: 72.6 (scale 0.0 to 100.0), 95% confidence interval 65.2 to 77.572.6(65.2–77.5)
77th24.1consistent

Arena preference vs objective benchmarks

Compares each model's percentile on human preference arenas against its median percentile across objective verifiable boards.

Gap 15+ percentile pointsAlignedDiagonal shows equal Arena vs objective percentile.
00252550507575100100Human Arena percentileObjective median percentileGPT-5.3 CodexGemini 3.8 FlashGemini 3.1 Pro PreviewGPT-6 SolGLM-5.3 Flash
ModelArena pctObjective median pctDeltaClassification
GPT-5.3 Codex12.068.9-56.9capability-favored. Evidence grade C of A to D: thin.
Gemini 3.8 Flash85.243.8+41.4style-favored. Evidence grade C of A to D: thin.
Gemini 3.1 Pro Preview57.190.7-33.6capability-favored. Evidence grade C of A to D: thin.
GPT-6 Sol54.888.2-33.4capability-favored. Evidence grade C of A to D: thin.
GLM-5.3 Flash70.238.3+31.9style-favored. Evidence grade C of A to D: thin.
GPT-5.236.664.9-28.3capability-favored. Evidence grade C of A to D: thin.
GPT-4.5 Preview37.612.6+25.0style-favored. Evidence grade C of A to D: thin.
Inkling Small24.70.0+24.7style-favored. Evidence grade C of A to D: thin.
Kimi K380.956.3+24.6style-favored. Evidence grade C of A to D: thin.
Gemini 2.5 Flash35.511.1+24.4style-favored. Evidence grade C of A to D: thin.
Hy4 Preview52.528.1+24.4style-favored. Evidence grade C of A to D: thin.
Inkling27.23.1+24.1style-favored. Evidence grade C of A to D: thin.
Gemini 3 Flash Preview35.758.6-22.9capability-favored. Evidence grade C of A to D: thin.
Fable 590.268.5+21.7style-favored. Evidence grade C of A to D: thin.
Grok 4.648.770.4-21.7capability-favored. Evidence grade C of A to D: thin.
GPT-528.650.0-21.4capability-favored. Evidence grade C of A to D: thin.
Beam 1.027.97.8+20.1style-favored. Evidence grade C of A to D: thin.
Grok 4.560.240.4+19.8style-favored. Evidence grade C of A to D: thin.
Kimi K3.1 Pro75.856.3+19.5style-favored. Evidence grade C of A to D: thin.
Muse Spark Open44.025.0+19.0style-favored. Evidence grade C of A to D: thin.
Sonnet 5.572.891.7-18.9capability-favored. Evidence grade C of A to D: thin.
Haiku 5.529.748.4-18.7capability-favored. Evidence grade C of A to D: thin.
Opus 4.114.333.0-18.7capability-favored. Evidence grade C of A to D: thin.
GPT-4.129.010.3+18.7style-favored. Evidence grade C of A to D: thin.
GPT-6.1 Terra82.964.8+18.1style-favored. Evidence grade C of A to D: thin.
Grok 4.744.462.5-18.1capability-favored. Evidence grade C of A to D: thin.
Step 5 Preview48.731.3+17.4style-favored. Evidence grade C of A to D: thin.
GLM-4.738.721.3+17.4style-favored. Evidence grade C of A to D: thin.
DeepSeek V4 Pro 081357.040.6+16.4style-favored. Evidence grade C of A to D: thin.
Opus 4.693.377.8+15.5style-favored. Evidence grade C of A to D: thin.

Domain fingerprints

One median percentile per model and domain: flat profiles are generalists, jagged ones are specialists.

Each axis is a domain; a line is a model's median percentile on that domain's boards (100 is best). Domains need at least 12 models and models at least 4 domains, so 150 models and 9 domains are drawn. Neighbouring axes are the most correlated ones. Spread is the standard deviation of a model's domain values; the lists below keep models with a mean percentile of at least 60.

Top generalists

Flattest lines among strong models

  1. Qwen4 Maxspread 1.5, strength 73, 4 domains, best Finance, weakest Coding
  2. Gemini 4 Carbonspread 1.9, strength 97, 6 domains, best General text, weakest Legal
  3. GPT-5.5 Pre-releasespread 1.9, strength 96, 4 domains, best Math, weakest Science
  4. Muse Wonder Full 1.0spread 2.2, strength 85, 4 domains, best Legal, weakest Coding
  5. Qwen3.8 Max Internalspread 3.1, strength 60, 4 domains, best Legal, weakest Coding

Top specialists

Most jagged lines among strong models

  1. GPT-6.1 Terraspread 30.4, strength 62, 5 domains, best Science, weakest Legal
  2. Gemini 3.8 Flashspread 25.2, strength 67, 8 domains, best Vision, weakest Coding
  3. Gemini 3.1 Pro Previewspread 23.1, strength 72, 8 domains, best Science, weakest Finance
  4. Qwen3.8-Maxspread 21.9, strength 78, 6 domains, best Vision, weakest Agentic
  5. Gemini 3.7 Flashspread 21.8, strength 74, 5 domains, best Vision, weakest Agentic
0255075100General text136 modelsScience124 modelsMath66 modelsVision95 modelsCoding130 modelsAgentic128 modelsFinance96 modelsLegal38 modelsSearch33 models

Hover a line, or tab to one, to see its values. 30 of 150 lines highlighted; the rest are muted. Gaps are domains the model was not tested on.

Method, limits, and excluded inputs (32 boards, 3509 rows read)

Method rules

  • Boards and percentiles are those of the consensus map: one default board per benchmark and source, quality metrics only, boards with at least 8 models, best row per model.
  • A board counts in every domain its benchmark lists; a benchmark with no domain is left out.
  • Domain value = median percentile of the model over the boards of that domain.
  • A domain keeps its axis with at least 12 models; a model is kept with values on at least 4 kept domains. Both rules repeat until nothing more drops.
  • Strength = mean of the model's domain values. Specialization index = population standard deviation of those values, in percentile points. Strongest and weakest domain: highest and lowest value (ties go to the domain id that sorts first).
  • Axis order: start at the domain with most models, then always add the unplaced domain most correlated (Pearson, on models with both, at least 10 shared; fewer counts as 0) with the last one placed. Ties go to the domain id that sorts first.
  • Summary: among models with strength of at least 60, the 5 lowest indices are generalists and the 5 highest are specialists (ties: more domains, then higher strength, then model id).
  • Featured score = strength x domains covered / axes, so breadth and strength both count; models are listed by it.

Known limits

  • A domain value is a rank among the models tested on that domain's boards, so a model tested only against weaker rivals in one domain looks stronger there.
  • Missing domains are not tested, not zero; a model covering few domains shows a flatter or more jagged line by chance.
  • A board listed under two domains counts in both, so those two domains are correlated partly by construction.
  • Domains with many boards are steadier than those with one or two.

Exclusions

  • 16 boards excluded: board has fewer than 8 models
  • 94 boards excluded: not the default slice of its source and version
  • 1 boards excluded: older version of a benchmark the source reports again
  • 2 boards excluded: rank metric is not a quality score (price, speed, latency, or token count)
  • 484 models excluded: model has values on fewer than 4 kept domains
  • 75 rows excluded: hidden row (for example superseded)

Bug Hunt complementarity

Each model counts once, by its best current Bug Hunt row. A bug is as hard as the number of models that missed it, and a team covers a bug when any member fixed it. Rows averaged over several runs list the bugs of all their runs, so models tested more often look broader than they are.

Best single model
67 of 105 (64%)
Fable 5.2; 1 other set ties
Best pair
72 of 105 (69%)
Fable 5.2 + Gemini 4 Carbon; 2 other sets tie
Best trio
75 of 105 (71%)
Fable 5.2 + Gemini 4 Carbon + Grok 4.7; 3 other sets tie
Ceiling, all models
81 of 105
24 bugs no model fixed. Greedy reaches it with 9 models.

How hard each bug is

24 of 105 bugs were fixed by no model, and 5 by exactly one (Gemini 4 Carbon, GPT-6.1 Terra, GPT-5.6 Sol, GPT-5.6 Luna, GPT-6 Sol).

Never fixed (24)Fixed by one model (5)Fixed by two or more
02550Bug 1: fixed by 0 of 50 modelsBug 3: fixed by 0 of 50 modelsBug 4: fixed by 0 of 50 modelsBug 5: fixed by 0 of 50 modelsBug 7: fixed by 0 of 50 modelsBug 8: fixed by 0 of 50 modelsBug 9: fixed by 0 of 50 modelsBug 10: fixed by 0 of 50 modelsBug 11: fixed by 0 of 50 modelsBug 12: fixed by 0 of 50 modelsBug 14: fixed by 0 of 50 modelsBug 15: fixed by 0 of 50 modelsBug 18: fixed by 0 of 50 modelsBug 19: fixed by 0 of 50 modelsBug 48: fixed by 0 of 50 modelsBug 54: fixed by 0 of 50 modelsBug 58: fixed by 0 of 50 modelsBug 63: fixed by 0 of 50 modelsBug 72: fixed by 0 of 50 modelsBug 85: fixed by 0 of 50 modelsBug 86: fixed by 0 of 50 modelsBug 101: fixed by 0 of 50 modelsBug 103: fixed by 0 of 50 modelsBug 104: fixed by 0 of 50 modelsBug 2: fixed by 1 of 50 modelsBug 6: fixed by 1 of 50 modelsBug 31: fixed by 1 of 50 modelsBug 65: fixed by 1 of 50 modelsBug 75: fixed by 1 of 50 modelsBug 24: fixed by 2 of 50 modelsBug 67: fixed by 2 of 50 modelsBug 87: fixed by 2 of 50 modelsBug 89: fixed by 2 of 50 modelsBug 90: fixed by 2 of 50 modelsBug 52: fixed by 3 of 50 modelsBug 57: fixed by 3 of 50 modelsBug 73: fixed by 4 of 50 modelsBug 88: fixed by 4 of 50 modelsBug 105: fixed by 4 of 50 modelsBug 81: fixed by 5 of 50 modelsBug 53: fixed by 6 of 50 modelsBug 94: fixed by 6 of 50 modelsBug 17: fixed by 7 of 50 modelsBug 35: fixed by 7 of 50 modelsBug 42: fixed by 7 of 50 modelsBug 55: fixed by 7 of 50 modelsBug 68: fixed by 7 of 50 modelsBug 93: fixed by 7 of 50 modelsBug 64: fixed by 8 of 50 modelsBug 36: fixed by 9 of 50 modelsBug 38: fixed by 9 of 50 modelsBug 46: fixed by 9 of 50 modelsBug 56: fixed by 9 of 50 modelsBug 74: fixed by 10 of 50 modelsBug 32: fixed by 12 of 50 modelsBug 59: fixed by 12 of 50 modelsBug 61: fixed by 13 of 50 modelsBug 66: fixed by 14 of 50 modelsBug 78: fixed by 14 of 50 modelsBug 79: fixed by 14 of 50 modelsBug 96: fixed by 14 of 50 modelsBug 37: fixed by 16 of 50 modelsBug 51: fixed by 16 of 50 modelsBug 69: fixed by 16 of 50 modelsBug 95: fixed by 16 of 50 modelsBug 23: fixed by 20 of 50 modelsBug 28: fixed by 20 of 50 modelsBug 41: fixed by 20 of 50 modelsBug 80: fixed by 20 of 50 modelsBug 91: fixed by 20 of 50 modelsBug 26: fixed by 22 of 50 modelsBug 21: fixed by 24 of 50 modelsBug 70: fixed by 26 of 50 modelsBug 82: fixed by 26 of 50 modelsBug 40: fixed by 27 of 50 modelsBug 33: fixed by 28 of 50 modelsBug 34: fixed by 28 of 50 modelsBug 44: fixed by 28 of 50 modelsBug 13: fixed by 31 of 50 modelsBug 22: fixed by 31 of 50 modelsBug 77: fixed by 32 of 50 modelsBug 27: fixed by 33 of 50 modelsBug 29: fixed by 33 of 50 modelsBug 60: fixed by 33 of 50 modelsBug 100: fixed by 33 of 50 modelsBug 102: fixed by 33 of 50 modelsBug 45: fixed by 34 of 50 modelsBug 62: fixed by 34 of 50 modelsBug 49: fixed by 35 of 50 modelsBug 25: fixed by 36 of 50 modelsBug 97: fixed by 36 of 50 modelsBug 47: fixed by 37 of 50 modelsBug 39: fixed by 38 of 50 modelsBug 98: fixed by 38 of 50 modelsBug 16: fixed by 40 of 50 modelsBug 84: fixed by 40 of 50 modelsBug 92: fixed by 40 of 50 modelsBug 50: fixed by 42 of 50 modelsBug 43: fixed by 43 of 50 modelsBug 71: fixed by 43 of 50 modelsBug 30: fixed by 44 of 50 modelsBug 83: fixed by 44 of 50 modelsBug 20: fixed by 46 of 50 modelsBug 76: fixed by 46 of 50 modelsBug 99: fixed by 48 of 50 models105 bugs, hardest firstModels that fixed it

Adding models, one at a time

Each step adds the model that fixes the most bugs the team has missed so far.

0265379105Ceiling, all models: 81 of 105671722753764775786797808Models in the teamBugs covered
  1. 1 Fable 5.267
  2. 2 Gemini 4 Carbon+5
  3. 3 Grok 4.7+3
  4. 4 Sonnet 5.5+1
  5. 5 GPT-6.1 Terra+1
  6. 6 GPT-5.6 Sol+1
  7. 7 Muse Wonder 1.0+1
  8. 8 GPT-5.6 Luna+1

Which pairs fill each other's gaps

The 12 models that fix the most bugs on their own, paired.

Bugs fixed by either model5572The diagonal is each model on its own.
Fable 5.2Fable 5.2Gemini 4 CarbonGemini 4 CarbonSonnet 5.5Sonnet 5.5DeepSeek V4.1 Flash 1…DeepSeek V4.1 Flash 1…GPT-6.1 TerraGPT-6.1 TerraGPT-6 AstraGPT-6 AstraGPT-6.1 SolGPT-6.1 SolGPT-5.6 SolGPT-5.6 SolGemini 4 ArgonGemini 4 ArgonMuse Spark 1.3Muse Spark 1.3Opus 5.5Opus 5.5Gemini 4 HydrogenGemini 4 Hydrogen677271717069706968696869726770707270717168706967717064666967676967676566717066606965646667636264707269695664656362656362697067656455586464615961707167646558546567615963697169666364655262585959686867676264676251616155697067636561615861495658686965626359595961564859696766646261635955585947

Hover or focus a cell to compare two models. Arrow keys move between cells.

Method, limits, and excluded inputs (1 boards, 145 rows read)

Method rules

  • Only the Bug Hunt board is read. Where several boards carry a bug matrix, the newest version with the most rows is used and the others are listed in exclusions.
  • Rows marked superseded are dropped. Among a model's remaining rows, the best row is the one with the highest fixed count (the board's score), then the longer bug list, then the row id.
  • A model is one model id, whatever its effort or harness: its best row stands for it, and its other rows are listed in exclusions.
  • The bug universe is 1 to N, where N is the board's bug total (bugs fixed divided by fix rate, when every row agrees), widened to include any bug id seen. Bugs that no model fixed are the never-fixed bugs.
  • A bug's difficulty is the number of models that fixed it. Bugs fixed by exactly one model are credited to that model as its unique fixes.
  • The best set of 1, 2 and 3 models is found by exact search over every combination, scored by the number of bugs in the union. Ties go to the models that fix most on their own, then to the model id; the number of tied sets is reported.
  • The greedy curve adds, at each step, the model that adds the most new bugs (same tie rule). It is shown for 1 to 8 models; the step at which it reaches the ceiling is reported.
  • The ceiling is the union of every model's bugs.
  • The pair matrix covers the 12 models that fix most on their own: union size, and Jaccard overlap (shared bugs divided by the union; 0 when both are empty).

Known limits

  • A row averaged over several runs lists more bugs than its mean fixed count (for example 67 listed against a mean of 58), so its list appears to pool the runs. A model run more often has more chances to fix a bug, which favors it here. This was not checked against the source.
  • Coverage by a team of models is a union over single models. It does not say that a router could pick the right model for each bug in advance.
  • A bug that no model fixed may be broken, ambiguous or simply hard. The data cannot tell which.
  • Superseded rows are dropped, so bugs that only they fixed count as never fixed here; the result reports how many.
  • Only a model's best row counts. Bugs that only its other rows (other efforts or harnesses) fixed are lost, and the best row may come from a harness that suits the bug set.

Exclusions

  • 1 models excluded: model with only superseded rows
  • 95 rows excluded: another row of a model that has a better row
  • 75 rows excluded: superseded row
Method, limits, and excluded inputs (34 boards, 3509 rows read)

Method rules

  • Boards: one default board per benchmark and source (overall slice, newest version), excluding non-quality metrics and boards with fewer than 8 models.
  • To avoid double-counting composites, composite indices (such as vals-index and aa-intelligence-index) are excluded from the latent skill fit when two or more sub-boards are present, and each board is weighted inversely by the square root of its source's board count.
  • Latent capability (0-100) fits a weighted Bradley-Terry pairwise win model across all boards simultaneously with 120 seeded bootstrap resamples over boards for 95% confidence bounds.
  • Per model and board: take the best visible row; percentile = 100 x (beaten + 0.5 ties) / (n - 1). Models with >= 3 boards also report median percentile, IQR spread, and head-to-head win rates.

Known limits

  • Missing models are treated as not tested rather than zero, and model-selection patterns across benchmarks still reflect what sources chose to evaluate.
  • Bootstrap confidence intervals resample benchmarks as units and reflect between-benchmark variance more than item-sampling noise.
  • A specialist model strong in coding and weaker in math shows high cross-domain spread even when sources within each domain agree.

Exclusions

  • 16 boards excluded: board has fewer than 8 models
  • 94 boards excluded: not the default slice of its source and version
  • 1 boards excluded: older version of a benchmark the source reports again
  • 2 boards excluded: rank metric is not a quality score (price, speed, latency, or token count)
  • 372 models excluded: model has fewer than 3 boards
  • 9711 pairs excluded: pair shares fewer than 3 boards
  • 75 rows excluded: hidden row (for example superseded)