Bug Hunt Bench
105 real bugs planted in two production repos. Coding agents find and fix them; a blind judge grades the diff against a withheld answer key.
No filters. Showing 145 of 145 rows.
1 AnthropicDossierCaveat | max | Claude Code 2.1.321 | Fable 5.2, Bugs fixed: 58 (scale 0.0 to 105.0)58 | 55.2% | 0 | 142.8 min | $139.65 | 3 seeds |
2 GoogleDossierCaveat | high | Antigravity CLI 1.5.5 | Gemini 4 Carbon, Bugs fixed: 57.3 (scale 0.0 to 105.0)57.3 | 54.6% | 0 | 114 min | $31.32 | 3 seeds |
3 GoogleDossierCaveat | medium | Antigravity CLI 1.5.5 | Gemini 4 Carbon, Bugs fixed: 56 (scale 0.0 to 105.0)56 | 53.3% | 0 | 81.1 min | $24.95 | 3 seeds |
4 AnthropicDossierCaveat | max | Claude Code 2.1.283 | Sonnet 5.5, Bugs fixed: 51.3 (scale 0.0 to 105.0)51.3 | 48.9% | 0 | 287.1 min | $154.03 | 3 seeds |
5 AnthropicDossierCaveat | xhigh | Claude Code 2.1.321 | Fable 5.2, Bugs fixed: 49 (scale 0.0 to 105.0)49 | 46.7% | 0 | 57 min | $61.01 | 3 seeds |
6 GoogleDossierCaveat | low | Antigravity CLI | Gemini 4 Carbon, Bugs fixed: 46 (scale 0.0 to 105.0)46 | 43.8% | 0 | 60.4 min | $7.67 | 3 seeds |
7 OpenAIDossierCaveat | max | Codex CLI | GPT-6 Astra, Bugs fixed: 45 (scale 0.0 to 105.0)45 | 42.9% | 0 | 89.8 min | $33.03 | 3 seeds |
8 OpenAIDossierCaveat | max | Codex CLI 0.159.0 | GPT-6.1 Sol, Bugs fixed: 44.3 (scale 0.0 to 105.0)44.3 | 42.2% | 0 | 139.3 min | $5.88 | 3 seeds |
9 OpenAIDossierCaveat | max | Codex CLI 0.154.0 | GPT-5.6 Sol, Bugs fixed: 43.5 (scale 0.0 to 105.0)43.5 | 41.4% | 0 | 253.1 min | $95.35 | 2 seeds |
10 AnthropicDossier | max | Claude Code 2.1.251 | Fable 5.1, Bugs fixed: 43 (scale 0.0 to 105.0), single run, no repeated seeds43(1 seed) | 41% | 0 | 73.1 min | $87.18 | 1 seed. Based on one run, so no interval can be computed. |
10 AnthropicDossierCaveat | high | Claude Code 2.1.321 | Fable 5.2, Bugs fixed: 43 (scale 0.0 to 105.0)43 | 41% | 0 | 39.5 min | $46.67 | 2 seeds |
10 OpenAIDossier | xhigh | Codex CLI | GPT-6 Astra, Bugs fixed: 43 (scale 0.0 to 105.0), single run, no repeated seeds43(1 seed) | 41% | 0 | 58.8 min | $24.22 | 1 seed. Based on one run, so no interval can be computed. |
13 DeepSeekDossierCaveat | max | DeepSeek Harness 0.2.1-rc.2 | DeepSeek V4.1 Flash 1017, Bugs fixed: 42.7 (scale 0.0 to 105.0)42.7 | 40.7% | 0 | 62.1 min | $1.84 | 7 seeds |
13 OpenAIDossierCaveat | xhigh | Codex CLI 0.159.0 | GPT-6.1 Sol, Bugs fixed: 42.7 (scale 0.0 to 105.0)42.7 | 40.7% | 0 | 98.7 min | $4.35 | 3 seeds |
15 DeepSeekDossierCaveat | max | Pi 1.3.0 | DeepSeek V4.1 Flash 1017, Bugs fixed: 42.2 (scale 0.0 to 105.0)42.2 | 40.2% | 0 | 54.1 min | $0.83 | 5 seeds |
16 MetaDossierCaveat | max | Muse Code 1.4.1 | Muse Wonder Full 1.0, Bugs fixed: 42 (scale 0.0 to 105.0)42 | 40% | 0 | 145.3 min | Not tested | 2 seeds |
17 AnthropicDossierCaveat | max | Claude Code 2.1.280 | Opus 5.5, Bugs fixed: 41.7 (scale 0.0 to 105.0)41.7 | 39.7% | 0 | 66.9 min | $58.53 | 3 seeds |
18 AnthropicDossierCaveat | medium | Claude Code 2.1.321 | Fable 5.2, Bugs fixed: 41.5 (scale 0.0 to 105.0)41.5 | 39.5% | 0 | 19.8 min | $26.55 | 2 seeds |
18 MetaDossierCaveat | xhigh | Muse Code 1.4.1 | Muse Wonder Full 1.0, Bugs fixed: 41.5 (scale 0.0 to 105.0)41.5 | 39.5% | 0 | 90.5 min | Not tested | 2 seeds |
20 AnthropicDossierCaveat | low | Claude Code 2.1.321 | Fable 5.2, Bugs fixed: 41 (scale 0.0 to 105.0)41 | 39% | 0 | 18.7 min | $24.32 | 2 seeds |
21 GoogleDossierCaveat | high | Antigravity CLI 1.3.5 | Gemini 4 Argon, Bugs fixed: 40.7 (scale 0.0 to 105.0)40.7 | 38.8% | 0 | 56.5 min | $7.16 | 3 seeds |
21 OpenAIDossierCaveat | max | Codex CLI 0.159.0 | GPT-6.1 Terra, Bugs fixed: 40.7 (scale 0.0 to 105.0)40.7 | 38.8% | 0 | 37.3 min | $3.88 | 3 seeds |
23 DeepSeekDossierCaveat | high | Pi 1.3.0 | DeepSeek V4.1 Flash 1017, Bugs fixed: 40.4 (scale 0.0 to 105.0)40.4 | 38.5% | 0 | 48.3 min | $0.64 | 5 seeds |
24 GoogleDossierCaveat | medium | Antigravity CLI | Gemini 4 Argon, Bugs fixed: 39.3 (scale 0.0 to 105.0)39.3 | 37.4% | 0 | 59.9 min | $7.50 | 3 seeds |
25 DeepSeekDossierCaveat | high | DeepSeek Harness 0.2.1-rc.2 | DeepSeek V4.1 Flash 1017, Bugs fixed: 39.2 (scale 0.0 to 105.0)39.2 | 37.3% | 0 | 53 min | $1.27 | 5 seeds |
26 DeepSeekDossierCaveat | max | Claude Code 2.1.302 | DeepSeek V4.1 Flash 1017, Bugs fixed: 39.1 (scale 0.0 to 105.0)39.1 | 37.2% | 0 | 57.7 min | $1.01 | 7 seeds |
27 OpenAIDossier | xhigh | Codex CLI | GPT-5.6 Sol, Bugs fixed: 39 (scale 0.0 to 105.0), single run, no repeated seeds39(1 seed) | 37.1% | 0 | 126.2 min | $52.75 | 1 seed. Based on one run, so no interval can be computed. |
28 DeepSeekDossierCaveat | max | OpenCode 1.19.3 | DeepSeek V4.1 Flash 1017, Bugs fixed: 37.6 (scale 0.0 to 105.0)37.6 | 35.8% | 0 | 55.7 min | $0.96 | 5 seeds |
29 DeepSeekDossierCaveat | high | OpenCode 1.19.3 | DeepSeek V4.1 Flash 1017, Bugs fixed: 36.6 (scale 0.0 to 105.0)36.6 | 34.9% | 0 | 41.6 min | $0.79 | 5 seeds |
30 OpenAIDossierCaveat | high | Codex CLI 0.159.0 | GPT-6.1 Sol, Bugs fixed: 36.5 (scale 0.0 to 105.0)36.5 | 34.8% | 0 | 70.3 min | $3.26 | 2 seeds |
31 DeepSeekDossierCaveat | high | Claude Code 2.1.302 | DeepSeek V4.1 Flash 1017, Bugs fixed: 36.4 (scale 0.0 to 105.0)36.4 | 34.7% | 0 | 54.5 min | $0.82 | 5 seeds |
32 Moonshot AIDossierCaveat | max | Claude Code 2.1.309 | Kimi K3.1 Pro, Bugs fixed: 36 (scale 0.0 to 105.0)36 | 34.3% | 0 | 64.8 min | $13.76 | 3 seeds |
32 AnthropicDossierCaveat | xhigh | Claude Code 2.1.280 | Opus 5.5, Bugs fixed: 36 (scale 0.0 to 105.0)36 | 34.3% | 0 | 42.5 min | $34.98 | 3 seeds |
32 AnthropicDossierCaveat | xhigh | Claude Code 2.1.283 | Sonnet 5.5, Bugs fixed: 36 (scale 0.0 to 105.0)36 | 34.3% | 0 | 72.7 min | $42.86 | 3 seeds |
35 GoogleDossierCaveat | low | Antigravity CLI | Gemini 4 Argon, Bugs fixed: 35.3 (scale 0.0 to 105.0)35.3 | 33.6% | 0 | 59.6 min | $6.47 | 3 seeds |
36 OpenAIDossier | high | Codex CLI | GPT-6 Astra, Bugs fixed: 35 (scale 0.0 to 105.0), single run, no repeated seeds35(1 seed) | 33.3% | 0 | 40.5 min | $20.60 | 1 seed. Based on one run, so no interval can be computed. |
37 DeepSeekDossierCaveat | low | DeepSeek Harness 0.2.1-rc.2 | DeepSeek V4.1 Flash 1017, Bugs fixed: 34.8 (scale 0.0 to 105.0)34.8 | 33.1% | 0 | 33.8 min | $1.06 | 5 seeds |
38 DeepSeekDossierCaveat | low | Claude Code 2.1.302 | DeepSeek V4.1 Flash 1017, Bugs fixed: 34.4 (scale 0.0 to 105.0)34.4 | 32.8% | 0 | 44.5 min | $0.71 | 5 seeds |
39 AnthropicDossierCaveat | high | Claude Code 2.1.283 | Sonnet 5.5, Bugs fixed: 34.3 (scale 0.0 to 105.0)34.3 | 32.7% | 0 | 27.9 min | $19.85 | 3 seeds |
40 OpenAIDossier | high | Codex CLI | GPT-5.6 Sol, Bugs fixed: 34 (scale 0.0 to 105.0), single run, no repeated seeds34(1 seed) | 32.4% | 0 | 66.7 min | $33.92 | 1 seed. Based on one run, so no interval can be computed. |
40 OpenAIDossier | medium | Codex CLI | GPT-6 Astra, Bugs fixed: 34 (scale 0.0 to 105.0), single run, no repeated seeds34(1 seed) | 32.4% | 0 | 28 min | $15.78 | 1 seed. Based on one run, so no interval can be computed. |
42 DeepSeekDossierCaveat | low | Pi 1.3.0 | DeepSeek V4.1 Flash 1017, Bugs fixed: 33.8 (scale 0.0 to 105.0)33.8 | 32.2% | 0 | 33.2 min | $0.46 | 5 seeds |
43 AnthropicDossier | high | Claude Code 2.1.251 | Fable 5.1, Bugs fixed: 33 (scale 0.0 to 105.0), single run, no repeated seeds33(1 seed) | 31.4% | 0 | 36.3 min | $48.58 | 1 seed. Based on one run, so no interval can be computed. |
44 DeepSeekDossierCaveat | low | OpenCode 1.19.3 | DeepSeek V4.1 Flash 1017, Bugs fixed: 32.8 (scale 0.0 to 105.0)32.8 | 31.2% | 0 | 41.2 min | $0.66 | 5 seeds |
45 GoogleDossierCaveat | xhigh | Antigravity CLI 1.5.8 | Gemini 4 Hydrogen, Bugs fixed: 32.7 (scale 0.0 to 105.0)32.7 | 31.1% | 0 | 35.8 min | $7.07 | 3 seeds |
46 MetaDossierCaveat | max | Muse Code 1.1.1 / 1.2.1 / 1.3.0 | Muse Spark 1.3, Bugs fixed: 32.2 (scale 0.0 to 105.0)32.2 | 30.7% | 0 | 85.6 min | $18.11 | 5 seeds |
47 OpenAIDossier | max | Codex CLI | GPT-5.6 Terra, Bugs fixed: 32 (scale 0.0 to 105.0), single run, no repeated seeds32(1 seed) | 30.5% | 0 | 159.7 min | $27.98 | 1 seed. Based on one run, so no interval can be computed. |
48 MetaDossierCaveat | max | Muse Code 1.4.1 | Muse Wonder 1.0, Bugs fixed: 31.7 (scale 0.0 to 105.0)31.7 | 30.2% | 0 | 96.2 min | $27.87 | 3 seeds |
48 AnthropicDossierCaveat | high | Claude Code 2.1.280 | Opus 5.5, Bugs fixed: 31.7 (scale 0.0 to 105.0)31.7 | 30.2% | 0 | 23.9 min | $22.25 | 3 seeds |
50 OpenAIDossierCaveat | max | Codex CLI | GPT-5.6 Luna, Bugs fixed: 31.3 (scale 0.0 to 105.0)31.3 | 29.8% | 0 | 187.1 min | $3.15 | 3 seeds |
51 UnbiasedDossier | No effort set | Claude Code 2.1.274 | Pareto (ex-Union Alpha), Bugs fixed: 30.7 (scale 0.0 to 105.0)30.7 | 29.2% | 0 | 38.2 min | $4.81 | 3 seeds |
52 AnthropicDossierCaveat | medium | Claude Code 2.1.280 | Opus 5.5, Bugs fixed: 30.3 (scale 0.0 to 105.0)30.3 | 28.9% | 0 | 17 min | $15.68 | 3 seeds |
53 OpenAIDossierCaveat | max | Codex CLI 0.155.1 | GPT-6 Sol, Bugs fixed: 29.3 (scale 0.0 to 105.0)29.3 | 27.9% | 0 | 63.4 min | $9.33 | 3 seeds |
54 AnthropicDossier | max | Claude Code 2.1.220 | Fable 5, Bugs fixed: 29 (scale 0.0 to 105.0), single run, no repeated seeds29(1 seed) | 27.6% | 0 | 57.3 min | $112.40 | 1 seed. Based on one run, so no interval can be computed. |
54 AnthropicDossierCaveat | low | Claude Code 2.1.251 | Fable 5.1, Bugs fixed: 29 (scale 0.0 to 105.0), single run, no repeated seeds29(1 seed) | 27.6% | 0 | 33.2 min | $33.00 | 1 seed. Based on one run, so no interval can be computed. |
54 AnthropicDossierCaveat | xhigh | Claude Code 2.1.267 | Fable 5.1, Bugs fixed: 29 (scale 0.0 to 105.0), single run, no repeated seeds29(1 seed) | 27.6% | 0 | 59.5 min | $53.61 | 1 seed. Based on one run, so no interval can be computed. |
54 OpenAIDossier | medium | Codex CLI | GPT-5.6 Sol, Bugs fixed: 29 (scale 0.0 to 105.0), single run, no repeated seeds29(1 seed) | 27.6% | 0 | 47.6 min | $15.77 | 1 seed. Based on one run, so no interval can be computed. |
54 OpenAIDossierCaveat | medium | Codex CLI 0.159.0 | GPT-6.1 Sol, Bugs fixed: 29 (scale 0.0 to 105.0)29 | 27.6% | 0 | 29.8 min | $1.76 | 2 seeds |
54 xAIDossierCaveat | xhigh | Grok Build CLI (ACP) 1.0.30 | Grok 4.7, Bugs fixed: 29 (scale 0.0 to 105.0)29 | 27.6% | 0 | 44.7 min | $22.96 | 5 seeds |
60 xAIDossierCaveat | xhigh | Grok Build CLI (ACP) 1.0.5 | Grok 4.6, Bugs fixed: 28.7 (scale 0.0 to 105.0)28.7 | 27.3% | 0 | 43.1 min | $18.60 | 3 seeds |
61 GoogleDossierCaveat | high | Antigravity CLI 1.5.8 | Gemini 4 Hydrogen, Bugs fixed: 28.3 (scale 0.0 to 105.0)28.3 | 27% | 0 | 22.4 min | $4.42 | 3 seeds |
62 OpenAIDossier | low | Codex CLI | GPT-6 Astra, Bugs fixed: 27 (scale 0.0 to 105.0), single run, no repeated seeds27(1 seed) | 25.7% | 0 | 32.1 min | $11.69 | 1 seed. Based on one run, so no interval can be computed. |
62 AnthropicDossier | max | Claude Code 2.1.220 | Opus 5, Bugs fixed: 27 (scale 0.0 to 105.0), single run, no repeated seeds27(1 seed) | 25.7% | 0 | 60 min | $54.94 | 1 seed. Based on one run, so no interval can be computed. |
64 xAIDossierCaveat | medium | Grok Build CLI (ACP) | Grok 4.7, Bugs fixed: 26.7 (scale 0.0 to 105.0)26.7 | 25.4% | 0 | 30.7 min | $16.76 | 3 seeds |
65 AnthropicDossier | xhigh | Claude Code 2.1.248 | Opus 5, Bugs fixed: 26 (scale 0.0 to 105.0), single run, no repeated seeds26(1 seed) | 24.8% | 0 | 49.9 min | $63.25 | 1 seed. Based on one run, so no interval can be computed. |
65 AlibabaDossierCaveat | max | Claude Code 2.1.267 | Qwen3.8-Flash, Bugs fixed: 26 (scale 0.0 to 105.0), single run, no repeated seeds26(1 seed) | 24.8% | 0 | 97.3 min | $1.81 | 1 seed. Based on one run, so no interval can be computed. |
67 AlibabaDossierCaveat | max | Claude Code 2.1.267 / 2.1.272 | Qwen3.8-Max, Bugs fixed: 25.7 (scale 0.0 to 105.0)25.7 | 24.5% | 0 | 117.1 min | $26.76 | 3 seeds |
68 OpenAIDossierCaveat | xhigh | Codex CLI 0.155.1 | GPT-6 Sol, Bugs fixed: 25 (scale 0.0 to 105.0), single run, no repeated seeds25(1 seed) | 23.8% | 0 | 51.4 min | $7.67 | 1 seed. Based on one run, so no interval can be computed. |
69 GoogleDossierCaveat | medium | Antigravity CLI 1.5.8 | Gemini 4 Hydrogen, Bugs fixed: 24.7 (scale 0.0 to 105.0)24.7 | 23.5% | 0 | 19.9 min | $2.10 | 3 seeds |
70 AnthropicDossier | high | Claude Code 2.1.215 / 2.1.218 | Fable 5, Bugs fixed: 24 (scale 0.0 to 105.0), single run, no repeated seeds24(1 seed) | 22.9% | 0 | 31.4 min | $74.52 | 1 seed. Based on one run, so no interval can be computed. |
70 AnthropicDossier | medium | Claude Code 2.1.247 | Opus 5, Bugs fixed: 24 (scale 0.0 to 105.0), single run, no repeated seeds24(1 seed) | 22.9% | 0 | 30.5 min | $37.40 | 1 seed. Based on one run, so no interval can be computed. |
72 XiaomiDossierCaveat | No effort set | Claude Code 2.1.278 / 2.1.280 | MiMo-V2.6-Flash, Bugs fixed: 23.3 (scale 0.0 to 105.0)23.3 | 22.2% | 0 | 70.7 min | $0.49 | 3 seeds |
72 AnthropicDossierCaveat | medium | Claude Code 2.1.283 | Sonnet 5.5, Bugs fixed: 23.3 (scale 0.0 to 105.0)23.3 | 22.2% | 0 | 17.4 min | $11.11 | 3 seeds |
74 OpenAIDossier | xhigh | Codex CLI | GPT-5.6 Luna, Bugs fixed: 23 (scale 0.0 to 105.0), single run, no repeated seeds23(1 seed) | 21.9% | 0 | 135.6 min | $2.50 | 1 seed. Based on one run, so no interval can be computed. |
74 xAIDossier | high | Grok Build CLI (ACP) | Grok 4.6, Bugs fixed: 23 (scale 0.0 to 105.0), single run, no repeated seeds23(1 seed) | 21.9% | 0 | 33 min | $15.73 | 1 seed. Based on one run, so no interval can be computed. |
74 AlibabaDossierCaveat | low | Claude Code 2.1.267 | Qwen3.8-Flash, Bugs fixed: 23 (scale 0.0 to 105.0), single run, no repeated seeds23(1 seed) | 21.9% | 0 | 98.1 min | $1.37 | 1 seed. Based on one run, so no interval can be computed. |
77 XiaomiDossierCaveat | No effort set | Claude Code 2.1.278 | MiMo-V2.6-Pro, Bugs fixed: 22.7 (scale 0.0 to 105.0)22.7 | 21.6% | 0 | 90.9 min | $0.86 | 3 seeds |
78 OpenAIDossierCaveat | low | Codex CLI 0.159.0 | GPT-6.1 Sol, Bugs fixed: 22.5 (scale 0.0 to 105.0)22.5 | 21.4% | 0 | 25.4 min | $1.34 | 2 seeds |
79 AnthropicDossierCaveat | low | Claude Code 2.1.280 | Opus 5.5, Bugs fixed: 22.3 (scale 0.0 to 105.0)22.3 | 21.2% | 0 | 11.5 min | $8.34 | 3 seeds |
80 GoogleDossierCaveat | high | Gemini CLI (retired) + model-pinning gateway 0.56.0 | Gemini 3.7 Flash, Bugs fixed: 22 (scale 0.0 to 105.0), single run, no repeated seeds22(1 seed) | 21% | 0 | 96.8 min | $8.43 | 1 seed. Based on one run, so no interval can be computed. |
80 xAIDossier | medium | Grok Build CLI (ACP) 1.0.5 | Grok 4.6, Bugs fixed: 22 (scale 0.0 to 105.0), single run, no repeated seeds22(1 seed) | 21% | 0 | 26.5 min | $5.80 | 1 seed. Based on one run, so no interval can be computed. |
82 DeepSeekDossierCaveat | max | Claude Code 2.1.267 / 2.1.270 | DeepSeek V4.1 Flash, Bugs fixed: 21.7 (scale 0.0 to 105.0)21.7 | 20.7% | 0 | 55.9 min | $0.78 | 3 seeds |
83 AnthropicDossierCaveat | max | Claude Code 2.1.292 | Haiku 5.5, Bugs fixed: 21.5 (scale 0.0 to 105.0)21.5 | 20.5% | 0 | 32.5 min | $5.83 | 2 seeds |
84 AnthropicDossierCaveat | medium | Claude Code 2.1.267 | Fable 5.1, Bugs fixed: 21 (scale 0.0 to 105.0), single run, no repeated seeds21(1 seed) | 20% | 0 | 23 min | $23.73 | 1 seed. Based on one run, so no interval can be computed. |
84 Moonshot AIDossierinert | No effort set | Claude Code 2.1.215 / 2.1.218 | Kimi K3, Bugs fixed: 21 (scale 0.0 to 105.0), single run, no repeated seeds21(1 seed) | 20% | 0 | 107.8 min | $25.27 | 1 seed. Based on one run, so no interval can be computed. |
84 AnthropicDossier | high | Claude Code 2.1.218 | Opus 5, Bugs fixed: 21 (scale 0.0 to 105.0), single run, no repeated seeds21(1 seed) | 20% | 0 | 37.4 min | $41.51 | 1 seed. Based on one run, so no interval can be computed. |
87 MetaDossierCaveat | xhigh | Muse Code 1.2.1 | Muse Spark 1.3, Bugs fixed: 20.3 (scale 0.0 to 105.0)20.3 | 19.3% | 0 | 63.8 min | $13.71 | 3 seeds |
88 OpenAIDossier | xhigh | Codex CLI | GPT-5.6 Terra, Bugs fixed: 20 (scale 0.0 to 105.0), single run, no repeated seeds20(1 seed) | 19% | 0 | 56.6 min | $8.98 | 1 seed. Based on one run, so no interval can be computed. |
88 OpenAIDossierCaveat | high | Codex CLI 0.155.1 | GPT-6 Sol, Bugs fixed: 20 (scale 0.0 to 105.0), single run, no repeated seeds20(1 seed) | 19% | 0 | 33.9 min | $4.16 | 1 seed. Based on one run, so no interval can be computed. |
90 xAIDossierCaveat | high | Grok Build CLI (ACP) | Grok 4.7, Bugs fixed: 19.3 (scale 0.0 to 105.0)19.3 | 18.4% | 0 | 37.9 min | $12.29 | 3 seeds |
91 DeepSeekDossierCaveat | high | Claude Code 2.1.267 | DeepSeek V4.1 Flash, Bugs fixed: 19 (scale 0.0 to 105.0), single run, no repeated seeds19(1 seed) | 18.1% | 0 | 26.1 min | $0.31 | 1 seed. Based on one run, so no interval can be computed. |
91 Z.aiDossierCaveat | max | Claude Code 2.1.267 | GLM-5.3, Bugs fixed: 19 (scale 0.0 to 105.0), single run, no repeated seeds19(1 seed) | 18.1% | 0 | 40.4 min | $15.93 | 1 seed. Based on one run, so no interval can be computed. |
91 AnthropicDossierCaveat | low | Claude Code 2.1.283 | Sonnet 5.5, Bugs fixed: 19 (scale 0.0 to 105.0)19 | 18.1% | 0 | 8.5 min | $5.73 | 3 seeds |
94 MetaDossierCaveat | high | Muse Code 1.2.1 | Muse Spark 1.3, Bugs fixed: 18.7 (scale 0.0 to 105.0)18.7 | 17.8% | 0 | 75.6 min | $12.94 | 3 seeds |
95 OpenAIDossierCaveat | max | Codex CLI 0.155.1 | GPT-6 Luna, Bugs fixed: 18.3 (scale 0.0 to 105.0)18.3 | 17.4% | 0 | 99.2 min | $0.52 | 3 seeds |
95 AnthropicDossierCaveat | xhigh | Claude Code 2.1.292 | Haiku 5.5, Bugs fixed: 18.3 (scale 0.0 to 105.0)18.3 | 17.4% | 0 | 24.1 min | $4.80 | 3 seeds |
97 GoogleDossierCaveat | high | Antigravity CLI 1.1.22 | Gemini 3.8 Flash, Bugs fixed: 18 (scale 0.0 to 105.0)18 | 17.1% | 0 | 38.7 min | $11.03 | 3 seeds |
97 OpenAIDossier | high | Codex CLI | GPT-5.6 Terra, Bugs fixed: 18 (scale 0.0 to 105.0), single run, no repeated seeds18(1 seed) | 17.1% | 0 | 28.1 min | $5.89 | 1 seed. Based on one run, so no interval can be computed. |
97 TencentDossierinert | No effort set | Claude Code 2.1.250 | Hy4 Preview, Bugs fixed: 18 (scale 0.0 to 105.0), single run, no repeated seeds18(1 seed) | 17.1% | 0 | 67.5 min | $3.13 | 1 seed. Based on one run, so no interval can be computed. |
97 XiaomiDossierCaveat | thinking | Claude Code 2.1.278 | MiMo-V2.6-Pro, Bugs fixed: 18 (scale 0.0 to 105.0), single run, no repeated seeds18(1 seed) | 17.1% | 0 | 74.9 min | $0.59 | 1 seed. Based on one run, so no interval can be computed. |
101 Z.aiDossierCaveat | max | Claude Code 2.1.269 / 2.1.270 | GLM-5.3 Flash, Bugs fixed: 17.7 (scale 0.0 to 105.0)17.7 | 16.9% | 0 | 63.2 min | $1.03 | 3 seeds |
102 xAIDossier | high | Grok Build CLI (ACP) | Grok 4.5, Bugs fixed: 17 (scale 0.0 to 105.0), single run, no repeated seeds17(1 seed) | 16.2% | 0 | 27.9 min | $8.50 | 1 seed. Based on one run, so no interval can be computed. |
102 MetaDossierCaveat | xhigh | Claude Code 2.1.222 | Muse Spark 1.2, Bugs fixed: 17 (scale 0.0 to 105.0), single run, no repeated seeds17(1 seed) | 16.2% | 0 | 35.7 min | $13.99 | 1 seed. Based on one run, so no interval can be computed. |
104 DeepSeekDossierCaveat | max | Claude Code 2.1.267 | DeepSeek V4-Pro, Bugs fixed: 16 (scale 0.0 to 105.0), single run, no repeated seeds16(1 seed) | 15.2% | 0 | 37.2 min | $1.89 | 1 seed. Based on one run, so no interval can be computed. |
104 GoogleDossier | high | Antigravity CLI 1.1.20 | Gemini 3.7 Flash, Bugs fixed: 16 (scale 0.0 to 105.0), single run, no repeated seeds16(1 seed) | 15.2% | 0 | 22.7 min | $6.36 | 1 seed. Based on one run, so no interval can be computed. |
104 Z.aiDossierCaveat | high | Claude Code 2.1.269 | GLM-5.3 Flash, Bugs fixed: 16 (scale 0.0 to 105.0), single run, no repeated seeds16(1 seed) | 15.2% | 0 | 62.3 min | $0.94 | 1 seed. Based on one run, so no interval can be computed. |
104 AnthropicDossierCaveat | high | Claude Code 2.1.292 | Haiku 5.5, Bugs fixed: 16 (scale 0.0 to 105.0)16 | 15.2% | 0 | 19.3 min | $3.56 | 2 seeds |
104 Z.aiDossierinert | No effort set | Claude Code 2.1.243 | Ox Alpha (stealth), Bugs fixed: 16 (scale 0.0 to 105.0), single run, no repeated seeds16(1 seed) | 15.2% | 0 | 59.4 min | $0.00 | 1 seed. Based on one run, so no interval can be computed. |
109 xAIDossierCaveat | low | Grok Build CLI (ACP) | Grok 4.7, Bugs fixed: 15.7 (scale 0.0 to 105.0)15.7 | 15% | 0 | 22.2 min | $9.36 | 3 seeds |
110 OpenAIDossier | medium | Codex CLI | GPT-5.6 Terra, Bugs fixed: 15 (scale 0.0 to 105.0), single run, no repeated seeds15(1 seed) | 14.3% | 0 | 20.2 min | $3.87 | 1 seed. Based on one run, so no interval can be computed. |
110 xAIDossier | low | Grok Build CLI (ACP) 1.0.5 | Grok 4.6, Bugs fixed: 15 (scale 0.0 to 105.0), single run, no repeated seeds15(1 seed) | 14.3% | 0 | 18 min | $4.14 | 1 seed. Based on one run, so no interval can be computed. |
110 MistralDossierCaveat | high | Claude Code 2.1.291 | Mistral Large 4, Bugs fixed: 15 (scale 0.0 to 105.0)15 | 14.3% | 0 | 111.1 min | $11.95 | 3 seeds |
110 AnthropicDossierCaveat | max | Claude Code 2.1.267 | Opus 4.8, Bugs fixed: 15 (scale 0.0 to 105.0), single run, no repeated seeds15(1 seed) | 14.3% | 0 | 108.8 min | $58.11 | 1 seed. Based on one run, so no interval can be computed. |
110 AlibabaDossierCaveat | xhigh | Claude Code 2.1.268 / 2.1.272 | Qwen3.8-27B, Bugs fixed: 15 (scale 0.0 to 105.0)15 | 14.3% | 0 | 56.8 min | $5.55 | 3 seeds |
115 DeepSeekDossierinert | No effort set | Claude Code 2.1.220 | DeepSeek V4-Flash 0731, Bugs fixed: 14 (scale 0.0 to 105.0), single run, no repeated seeds14(1 seed) | 13.3% | 0 | 48.5 min | $1.52 | 1 seed. Based on one run, so no interval can be computed. |
115 OpenAIDossierCaveat | xhigh | Codex CLI 0.155.1 | GPT-6 Luna, Bugs fixed: 14 (scale 0.0 to 105.0), single run, no repeated seeds14(1 seed) | 13.3% | 0 | 57 min | $0.39 | 1 seed. Based on one run, so no interval can be computed. |
115 OpenAIDossierCaveat | medium | Codex CLI 0.155.1 | GPT-6 Sol, Bugs fixed: 14 (scale 0.0 to 105.0), single run, no repeated seeds14(1 seed) | 13.3% | 0 | 24.5 min | $2.39 | 1 seed. Based on one run, so no interval can be computed. |
118 DeepSeekDossierCaveat | high | Claude Code 2.1.267 | DeepSeek V4-Pro, Bugs fixed: 13 (scale 0.0 to 105.0), single run, no repeated seeds13(1 seed) | 12.4% | 0 | 26.6 min | $1.08 | 1 seed. Based on one run, so no interval can be computed. |
118 Z.aiDossierinert | No effort set | Claude Code 2.1.247 | GLM-5.3 Flash, Bugs fixed: 13 (scale 0.0 to 105.0), single run, no repeated seeds13(1 seed) | 12.4% | 0 | 57.1 min | $0.79 | 1 seed. Based on one run, so no interval can be computed. |
118 OpenAIDossier | high | Codex CLI | GPT-5.6 Luna, Bugs fixed: 13 (scale 0.0 to 105.0), single run, no repeated seeds13(1 seed) | 12.4% | 0 | 64.1 min | $0.57 | 1 seed. Based on one run, so no interval can be computed. |
118 MetaDossierCaveat | medium | Muse Code 1.2.1 | Muse Spark 1.3, Bugs fixed: 13 (scale 0.0 to 105.0)13 | 12.4% | 0 | 58.3 min | $9.24 | 3 seeds |
122 AnthropicDossierCaveat | medium | Claude Code 2.1.292 | Haiku 5.5, Bugs fixed: 12 (scale 0.0 to 105.0), single run, no repeated seeds12(1 seed) | 11.4% | 0 | 15.5 min | $2.66 | 1 seed. Based on one run, so no interval can be computed. |
123 OpenAIDossierCaveat | xhigh | Codex CLI 0.159.0 | GPT-6.1 Terra, Bugs fixed: 11.3 (scale 0.0 to 105.0)11.3 | 10.8% | 0 | 15.9 min | $0.75 | 3 seeds |
124 AnthropicDossierCaveat | low | Claude Code 2.1.292 | Haiku 5.5, Bugs fixed: 11 (scale 0.0 to 105.0), single run, no repeated seeds11(1 seed) | 10.5% | 0 | 8.7 min | $1.61 | 1 seed. Based on one run, so no interval can be computed. |
125 AlibabaDossierinertCaveat | 8-bit | Claude Code 2.1.272 | Qwen3.8-27B, Bugs fixed: 10.7 (scale 0.0 to 105.0)10.7 | 10.2% | 0 | 56.6 min | $2.97 | 3 seeds |
126 GoogleDossierCaveat | low | Antigravity CLI 1.5.8 | Gemini 4 Hydrogen, Bugs fixed: 10.3 (scale 0.0 to 105.0)10.3 | 9.8% | 0 | 7.2 min | $0.43 | 3 seeds |
127 OpenAIDossierCaveat | high | Codex CLI 0.159.0 | GPT-6.1 Terra, Bugs fixed: 9.7 (scale 0.0 to 105.0)9.7 | 9.2% | 0 | 12.6 min | $0.56 | 3 seeds |
127 MetaDossierCaveat | low | Muse Code 1.2.1 | Muse Spark 1.3, Bugs fixed: 9.7 (scale 0.0 to 105.0)9.7 | 9.2% | 0 | 36.4 min | $5.61 | 3 seeds |
129 Z.aiDossierCaveat | low | Claude Code 2.1.269 | GLM-5.3 Flash, Bugs fixed: 9 (scale 0.0 to 105.0), single run, no repeated seeds9(1 seed) | 8.6% | 0 | 29 min | $0.42 | 1 seed. Based on one run, so no interval can be computed. |
129 OpenAIDossier | medium | Codex CLI 0.150.0-alpha.8 | GPT-5.6 Luna, Bugs fixed: 9 (scale 0.0 to 105.0), single run, no repeated seeds9(1 seed) | 8.6% | 0 | 15.4 min | $0.33 | 1 seed. Based on one run, so no interval can be computed. |
129 OpenAIDossierCaveat | high | Codex CLI 0.155.1 | GPT-6 Luna, Bugs fixed: 9 (scale 0.0 to 105.0), single run, no repeated seeds9(1 seed) | 8.6% | 0 | 24.4 min | $0.13 | 1 seed. Based on one run, so no interval can be computed. |
129 AnthropicDossier | high | Claude Code 2.1.215 / 2.1.218 | Opus 4.8, Bugs fixed: 9 (scale 0.0 to 105.0), single run, no repeated seeds9(1 seed) | 8.6% | 0 | 34.8 min | $21.30 | 1 seed. Based on one run, so no interval can be computed. |
129 AnthropicDossier | high | Claude Code 2.1.218 | Sonnet 5, Bugs fixed: 9 (scale 0.0 to 105.0), single run, no repeated seeds9(1 seed) | 8.6% | 0 | 32.8 min | $11.38 | 1 seed. Based on one run, so no interval can be computed. |
129 AnthropicDossierCaveat | max | Claude Code 2.1.267 | Sonnet 5, Bugs fixed: 9 (scale 0.0 to 105.0), single run, no repeated seeds9(1 seed) | 8.6% | 0 | 61.3 min | $17.96 | 1 seed. Based on one run, so no interval can be computed. |
135 OpenAIDossierCaveat | medium | Codex CLI 0.159.0 | GPT-6.1 Terra, Bugs fixed: 7.3 (scale 0.0 to 105.0)7.3 | 7% | 0 | 9.6 min | $0.31 | 3 seeds |
136 OpenAIDossierCaveat | low | Codex CLI 0.155.1 | GPT-6 Sol, Bugs fixed: 6 (scale 0.0 to 105.0), single run, no repeated seeds6(1 seed) | 5.7% | 0 | 11.1 min | $1.08 | 1 seed. Based on one run, so no interval can be computed. |
136 OpenAIDossierCaveat | low | Codex CLI 0.159.0 | GPT-6.1 Terra, Bugs fixed: 6 (scale 0.0 to 105.0)6 | 5.7% | 0 | 10.1 min | $0.30 | 3 seeds |
138 GoogleDossierinertCaveat | No effort set | Claude Code 2.1.272 | Gemma 4 31B, Bugs fixed: 4 (scale 0.0 to 105.0), single run, no repeated seeds4(1 seed) | 3.8% | 0 | 221.8 min | $8.61 | 1 seed. Based on one run, so no interval can be computed. |
138 OpenAIDossier | low | Codex CLI 0.150.0-alpha.8 | GPT-5.6 Luna, Bugs fixed: 4 (scale 0.0 to 105.0), single run, no repeated seeds4(1 seed) | 3.8% | 0 | 5.8 min | $0.10 | 1 seed. Based on one run, so no interval can be computed. |
138 OpenAIDossierCaveat | medium | Codex CLI 0.155.1 | GPT-6 Luna, Bugs fixed: 4 (scale 0.0 to 105.0), single run, no repeated seeds4(1 seed) | 3.8% | 0 | 8.6 min | $0.060 | 1 seed. Based on one run, so no interval can be computed. |
138 OpenAIDossierCaveat | low | Codex CLI 0.155.1 | GPT-6 Luna, Bugs fixed: 4 (scale 0.0 to 105.0), single run, no repeated seeds4(1 seed) | 3.8% | 0 | 19.3 min | $0.20 | 1 seed. Based on one run, so no interval can be computed. |
142 GoogleDossierinertCaveat | 4-bit | Claude Code 2.1.272 | Gemma 4 31B, Bugs fixed: 3 (scale 0.0 to 105.0), single run, no repeated seeds3(1 seed) | 2.9% | 0 | 24.9 min | $0.56 | 1 seed. Based on one run, so no interval can be computed. |
143 AnthropicDossierinertCaveat | No effort set | Claude Code 2.1.292 | Haiku 4.5, Bugs fixed: 1.5 (scale 0.0 to 105.0)1.5 | 1.4% | 0 | 16.9 min | $2.22 | 2 seeds |
144 OpenAIDossierinertCaveat | No effort set | Claude Code 2.1.269 / 2.1.272 | gpt-oss-20b, Bugs fixed: 0 (scale 0.0 to 105.0)0 | 0% | 0 | 7.4 min | $0.11 | 3 seeds |
144 OpenAIDossierinertCaveat | No effort set | Claude Code 2.1.269 / 2.1.272 | gpt-oss-120b, Bugs fixed: 0 (scale 0.0 to 105.0)0 | 0% | 0 | 3.7 min | $0.11 | 3 seeds |
Bugs fixed
Fable 5.2, Bugs fixed: 58 (scale 0.0 to 105.0)58- Fix rate
- 55.2%
- False-positive fixes
- 0
- Wall time
- 142.8 min
- Cost per run
- $139.65
Bugs fixed
Gemini 4 Carbon, Bugs fixed: 57.3 (scale 0.0 to 105.0)57.3- Fix rate
- 54.6%
- False-positive fixes
- 0
- Wall time
- 114 min
- Cost per run
- $31.32
Bugs fixed
Gemini 4 Carbon, Bugs fixed: 56 (scale 0.0 to 105.0)56- Fix rate
- 53.3%
- False-positive fixes
- 0
- Wall time
- 81.1 min
- Cost per run
- $24.95
Bugs fixed
Sonnet 5.5, Bugs fixed: 51.3 (scale 0.0 to 105.0)51.3- Fix rate
- 48.9%
- False-positive fixes
- 0
- Wall time
- 287.1 min
- Cost per run
- $154.03
Bugs fixed
Fable 5.2, Bugs fixed: 49 (scale 0.0 to 105.0)49- Fix rate
- 46.7%
- False-positive fixes
- 0
- Wall time
- 57 min
- Cost per run
- $61.01
Bugs fixed
Gemini 4 Carbon, Bugs fixed: 46 (scale 0.0 to 105.0)46- Fix rate
- 43.8%
- False-positive fixes
- 0
- Wall time
- 60.4 min
- Cost per run
- $7.67
Bugs fixed
GPT-6 Astra, Bugs fixed: 45 (scale 0.0 to 105.0)45- Fix rate
- 42.9%
- False-positive fixes
- 0
- Wall time
- 89.8 min
- Cost per run
- $33.03
Bugs fixed
GPT-6.1 Sol, Bugs fixed: 44.3 (scale 0.0 to 105.0)44.3- Fix rate
- 42.2%
- False-positive fixes
- 0
- Wall time
- 139.3 min
- Cost per run
- $5.88
Bugs fixed
GPT-5.6 Sol, Bugs fixed: 43.5 (scale 0.0 to 105.0)43.5- Fix rate
- 41.4%
- False-positive fixes
- 0
- Wall time
- 253.1 min
- Cost per run
- $95.35
- 10AnthropicDossier
max · Claude Code 2.1.251
1 seed. Based on one run, so no interval can be computed.Bugs fixed
Fable 5.1, Bugs fixed: 43 (scale 0.0 to 105.0), single run, no repeated seeds43(1 seed)- Fix rate
- 41%
- False-positive fixes
- 0
- Wall time
- 73.1 min
- Cost per run
- $87.18
Bugs fixed
Fable 5.2, Bugs fixed: 43 (scale 0.0 to 105.0)43- Fix rate
- 41%
- False-positive fixes
- 0
- Wall time
- 39.5 min
- Cost per run
- $46.67
Bugs fixed
GPT-6 Astra, Bugs fixed: 43 (scale 0.0 to 105.0), single run, no repeated seeds43(1 seed)- Fix rate
- 41%
- False-positive fixes
- 0
- Wall time
- 58.8 min
- Cost per run
- $24.22
Bugs fixed
DeepSeek V4.1 Flash 1017, Bugs fixed: 42.7 (scale 0.0 to 105.0)42.7- Fix rate
- 40.7%
- False-positive fixes
- 0
- Wall time
- 62.1 min
- Cost per run
- $1.84
Bugs fixed
GPT-6.1 Sol, Bugs fixed: 42.7 (scale 0.0 to 105.0)42.7- Fix rate
- 40.7%
- False-positive fixes
- 0
- Wall time
- 98.7 min
- Cost per run
- $4.35
Bugs fixed
DeepSeek V4.1 Flash 1017, Bugs fixed: 42.2 (scale 0.0 to 105.0)42.2- Fix rate
- 40.2%
- False-positive fixes
- 0
- Wall time
- 54.1 min
- Cost per run
- $0.83
Bugs fixed
Muse Wonder Full 1.0, Bugs fixed: 42 (scale 0.0 to 105.0)42- Fix rate
- 40%
- False-positive fixes
- 0
- Wall time
- 145.3 min
- Cost per run
- Not tested
Bugs fixed
Opus 5.5, Bugs fixed: 41.7 (scale 0.0 to 105.0)41.7- Fix rate
- 39.7%
- False-positive fixes
- 0
- Wall time
- 66.9 min
- Cost per run
- $58.53
Bugs fixed
Fable 5.2, Bugs fixed: 41.5 (scale 0.0 to 105.0)41.5- Fix rate
- 39.5%
- False-positive fixes
- 0
- Wall time
- 19.8 min
- Cost per run
- $26.55
Bugs fixed
Muse Wonder Full 1.0, Bugs fixed: 41.5 (scale 0.0 to 105.0)41.5- Fix rate
- 39.5%
- False-positive fixes
- 0
- Wall time
- 90.5 min
- Cost per run
- Not tested
Bugs fixed
Fable 5.2, Bugs fixed: 41 (scale 0.0 to 105.0)41- Fix rate
- 39%
- False-positive fixes
- 0
- Wall time
- 18.7 min
- Cost per run
- $24.32
Bugs fixed
Gemini 4 Argon, Bugs fixed: 40.7 (scale 0.0 to 105.0)40.7- Fix rate
- 38.8%
- False-positive fixes
- 0
- Wall time
- 56.5 min
- Cost per run
- $7.16
Bugs fixed
GPT-6.1 Terra, Bugs fixed: 40.7 (scale 0.0 to 105.0)40.7- Fix rate
- 38.8%
- False-positive fixes
- 0
- Wall time
- 37.3 min
- Cost per run
- $3.88
Bugs fixed
DeepSeek V4.1 Flash 1017, Bugs fixed: 40.4 (scale 0.0 to 105.0)40.4- Fix rate
- 38.5%
- False-positive fixes
- 0
- Wall time
- 48.3 min
- Cost per run
- $0.64
Bugs fixed
Gemini 4 Argon, Bugs fixed: 39.3 (scale 0.0 to 105.0)39.3- Fix rate
- 37.4%
- False-positive fixes
- 0
- Wall time
- 59.9 min
- Cost per run
- $7.50
Bugs fixed
DeepSeek V4.1 Flash 1017, Bugs fixed: 39.2 (scale 0.0 to 105.0)39.2- Fix rate
- 37.3%
- False-positive fixes
- 0
- Wall time
- 53 min
- Cost per run
- $1.27
Bugs fixed
DeepSeek V4.1 Flash 1017, Bugs fixed: 39.1 (scale 0.0 to 105.0)39.1- Fix rate
- 37.2%
- False-positive fixes
- 0
- Wall time
- 57.7 min
- Cost per run
- $1.01
Bugs fixed
GPT-5.6 Sol, Bugs fixed: 39 (scale 0.0 to 105.0), single run, no repeated seeds39(1 seed)- Fix rate
- 37.1%
- False-positive fixes
- 0
- Wall time
- 126.2 min
- Cost per run
- $52.75
Bugs fixed
DeepSeek V4.1 Flash 1017, Bugs fixed: 37.6 (scale 0.0 to 105.0)37.6- Fix rate
- 35.8%
- False-positive fixes
- 0
- Wall time
- 55.7 min
- Cost per run
- $0.96
Bugs fixed
DeepSeek V4.1 Flash 1017, Bugs fixed: 36.6 (scale 0.0 to 105.0)36.6- Fix rate
- 34.9%
- False-positive fixes
- 0
- Wall time
- 41.6 min
- Cost per run
- $0.79
Bugs fixed
GPT-6.1 Sol, Bugs fixed: 36.5 (scale 0.0 to 105.0)36.5- Fix rate
- 34.8%
- False-positive fixes
- 0
- Wall time
- 70.3 min
- Cost per run
- $3.26
Bugs fixed
DeepSeek V4.1 Flash 1017, Bugs fixed: 36.4 (scale 0.0 to 105.0)36.4- Fix rate
- 34.7%
- False-positive fixes
- 0
- Wall time
- 54.5 min
- Cost per run
- $0.82
Bugs fixed
Kimi K3.1 Pro, Bugs fixed: 36 (scale 0.0 to 105.0)36- Fix rate
- 34.3%
- False-positive fixes
- 0
- Wall time
- 64.8 min
- Cost per run
- $13.76
Bugs fixed
Opus 5.5, Bugs fixed: 36 (scale 0.0 to 105.0)36- Fix rate
- 34.3%
- False-positive fixes
- 0
- Wall time
- 42.5 min
- Cost per run
- $34.98
Bugs fixed
Sonnet 5.5, Bugs fixed: 36 (scale 0.0 to 105.0)36- Fix rate
- 34.3%
- False-positive fixes
- 0
- Wall time
- 72.7 min
- Cost per run
- $42.86
Bugs fixed
Gemini 4 Argon, Bugs fixed: 35.3 (scale 0.0 to 105.0)35.3- Fix rate
- 33.6%
- False-positive fixes
- 0
- Wall time
- 59.6 min
- Cost per run
- $6.47
Bugs fixed
GPT-6 Astra, Bugs fixed: 35 (scale 0.0 to 105.0), single run, no repeated seeds35(1 seed)- Fix rate
- 33.3%
- False-positive fixes
- 0
- Wall time
- 40.5 min
- Cost per run
- $20.60
Bugs fixed
DeepSeek V4.1 Flash 1017, Bugs fixed: 34.8 (scale 0.0 to 105.0)34.8- Fix rate
- 33.1%
- False-positive fixes
- 0
- Wall time
- 33.8 min
- Cost per run
- $1.06
Bugs fixed
DeepSeek V4.1 Flash 1017, Bugs fixed: 34.4 (scale 0.0 to 105.0)34.4- Fix rate
- 32.8%
- False-positive fixes
- 0
- Wall time
- 44.5 min
- Cost per run
- $0.71
Bugs fixed
Sonnet 5.5, Bugs fixed: 34.3 (scale 0.0 to 105.0)34.3- Fix rate
- 32.7%
- False-positive fixes
- 0
- Wall time
- 27.9 min
- Cost per run
- $19.85
Bugs fixed
GPT-5.6 Sol, Bugs fixed: 34 (scale 0.0 to 105.0), single run, no repeated seeds34(1 seed)- Fix rate
- 32.4%
- False-positive fixes
- 0
- Wall time
- 66.7 min
- Cost per run
- $33.92
Bugs fixed
GPT-6 Astra, Bugs fixed: 34 (scale 0.0 to 105.0), single run, no repeated seeds34(1 seed)- Fix rate
- 32.4%
- False-positive fixes
- 0
- Wall time
- 28 min
- Cost per run
- $15.78
Bugs fixed
DeepSeek V4.1 Flash 1017, Bugs fixed: 33.8 (scale 0.0 to 105.0)33.8- Fix rate
- 32.2%
- False-positive fixes
- 0
- Wall time
- 33.2 min
- Cost per run
- $0.46
- 43AnthropicDossier
high · Claude Code 2.1.251
1 seed. Based on one run, so no interval can be computed.Bugs fixed
Fable 5.1, Bugs fixed: 33 (scale 0.0 to 105.0), single run, no repeated seeds33(1 seed)- Fix rate
- 31.4%
- False-positive fixes
- 0
- Wall time
- 36.3 min
- Cost per run
- $48.58
Bugs fixed
DeepSeek V4.1 Flash 1017, Bugs fixed: 32.8 (scale 0.0 to 105.0)32.8- Fix rate
- 31.2%
- False-positive fixes
- 0
- Wall time
- 41.2 min
- Cost per run
- $0.66
Bugs fixed
Gemini 4 Hydrogen, Bugs fixed: 32.7 (scale 0.0 to 105.0)32.7- Fix rate
- 31.1%
- False-positive fixes
- 0
- Wall time
- 35.8 min
- Cost per run
- $7.07
Bugs fixed
Muse Spark 1.3, Bugs fixed: 32.2 (scale 0.0 to 105.0)32.2- Fix rate
- 30.7%
- False-positive fixes
- 0
- Wall time
- 85.6 min
- Cost per run
- $18.11
Bugs fixed
GPT-5.6 Terra, Bugs fixed: 32 (scale 0.0 to 105.0), single run, no repeated seeds32(1 seed)- Fix rate
- 30.5%
- False-positive fixes
- 0
- Wall time
- 159.7 min
- Cost per run
- $27.98
Bugs fixed
Muse Wonder 1.0, Bugs fixed: 31.7 (scale 0.0 to 105.0)31.7- Fix rate
- 30.2%
- False-positive fixes
- 0
- Wall time
- 96.2 min
- Cost per run
- $27.87
Bugs fixed
Opus 5.5, Bugs fixed: 31.7 (scale 0.0 to 105.0)31.7- Fix rate
- 30.2%
- False-positive fixes
- 0
- Wall time
- 23.9 min
- Cost per run
- $22.25
Bugs fixed
GPT-5.6 Luna, Bugs fixed: 31.3 (scale 0.0 to 105.0)31.3- Fix rate
- 29.8%
- False-positive fixes
- 0
- Wall time
- 187.1 min
- Cost per run
- $3.15
- 51UnbiasedDossier
Claude Code 2.1.274
Bugs fixed
Pareto (ex-Union Alpha), Bugs fixed: 30.7 (scale 0.0 to 105.0)30.7- Fix rate
- 29.2%
- False-positive fixes
- 0
- Wall time
- 38.2 min
- Cost per run
- $4.81
Bugs fixed
Opus 5.5, Bugs fixed: 30.3 (scale 0.0 to 105.0)30.3- Fix rate
- 28.9%
- False-positive fixes
- 0
- Wall time
- 17 min
- Cost per run
- $15.68
Bugs fixed
GPT-6 Sol, Bugs fixed: 29.3 (scale 0.0 to 105.0)29.3- Fix rate
- 27.9%
- False-positive fixes
- 0
- Wall time
- 63.4 min
- Cost per run
- $9.33
- 54AnthropicDossier
max · Claude Code 2.1.220
1 seed. Based on one run, so no interval can be computed.Bugs fixed
Fable 5, Bugs fixed: 29 (scale 0.0 to 105.0), single run, no repeated seeds29(1 seed)- Fix rate
- 27.6%
- False-positive fixes
- 0
- Wall time
- 57.3 min
- Cost per run
- $112.40
- 54AnthropicDossier
low · Claude Code 2.1.251
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
Fable 5.1, Bugs fixed: 29 (scale 0.0 to 105.0), single run, no repeated seeds29(1 seed)- Fix rate
- 27.6%
- False-positive fixes
- 0
- Wall time
- 33.2 min
- Cost per run
- $33.00
- 54AnthropicDossier
xhigh · Claude Code 2.1.267
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
Fable 5.1, Bugs fixed: 29 (scale 0.0 to 105.0), single run, no repeated seeds29(1 seed)- Fix rate
- 27.6%
- False-positive fixes
- 0
- Wall time
- 59.5 min
- Cost per run
- $53.61
Bugs fixed
GPT-5.6 Sol, Bugs fixed: 29 (scale 0.0 to 105.0), single run, no repeated seeds29(1 seed)- Fix rate
- 27.6%
- False-positive fixes
- 0
- Wall time
- 47.6 min
- Cost per run
- $15.77
Bugs fixed
GPT-6.1 Sol, Bugs fixed: 29 (scale 0.0 to 105.0)29- Fix rate
- 27.6%
- False-positive fixes
- 0
- Wall time
- 29.8 min
- Cost per run
- $1.76
Bugs fixed
Grok 4.7, Bugs fixed: 29 (scale 0.0 to 105.0)29- Fix rate
- 27.6%
- False-positive fixes
- 0
- Wall time
- 44.7 min
- Cost per run
- $22.96
Bugs fixed
Grok 4.6, Bugs fixed: 28.7 (scale 0.0 to 105.0)28.7- Fix rate
- 27.3%
- False-positive fixes
- 0
- Wall time
- 43.1 min
- Cost per run
- $18.60
Bugs fixed
Gemini 4 Hydrogen, Bugs fixed: 28.3 (scale 0.0 to 105.0)28.3- Fix rate
- 27%
- False-positive fixes
- 0
- Wall time
- 22.4 min
- Cost per run
- $4.42
Bugs fixed
GPT-6 Astra, Bugs fixed: 27 (scale 0.0 to 105.0), single run, no repeated seeds27(1 seed)- Fix rate
- 25.7%
- False-positive fixes
- 0
- Wall time
- 32.1 min
- Cost per run
- $11.69
- 62AnthropicDossier
max · Claude Code 2.1.220
1 seed. Based on one run, so no interval can be computed.Bugs fixed
Opus 5, Bugs fixed: 27 (scale 0.0 to 105.0), single run, no repeated seeds27(1 seed)- Fix rate
- 25.7%
- False-positive fixes
- 0
- Wall time
- 60 min
- Cost per run
- $54.94
Bugs fixed
Grok 4.7, Bugs fixed: 26.7 (scale 0.0 to 105.0)26.7- Fix rate
- 25.4%
- False-positive fixes
- 0
- Wall time
- 30.7 min
- Cost per run
- $16.76
- 65AnthropicDossier
xhigh · Claude Code 2.1.248
1 seed. Based on one run, so no interval can be computed.Bugs fixed
Opus 5, Bugs fixed: 26 (scale 0.0 to 105.0), single run, no repeated seeds26(1 seed)- Fix rate
- 24.8%
- False-positive fixes
- 0
- Wall time
- 49.9 min
- Cost per run
- $63.25
- 65AlibabaDossier
max · Claude Code 2.1.267
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
Qwen3.8-Flash, Bugs fixed: 26 (scale 0.0 to 105.0), single run, no repeated seeds26(1 seed)- Fix rate
- 24.8%
- False-positive fixes
- 0
- Wall time
- 97.3 min
- Cost per run
- $1.81
Bugs fixed
Qwen3.8-Max, Bugs fixed: 25.7 (scale 0.0 to 105.0)25.7- Fix rate
- 24.5%
- False-positive fixes
- 0
- Wall time
- 117.1 min
- Cost per run
- $26.76
- 68OpenAIDossier
xhigh · Codex CLI 0.155.1
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
GPT-6 Sol, Bugs fixed: 25 (scale 0.0 to 105.0), single run, no repeated seeds25(1 seed)- Fix rate
- 23.8%
- False-positive fixes
- 0
- Wall time
- 51.4 min
- Cost per run
- $7.67
Bugs fixed
Gemini 4 Hydrogen, Bugs fixed: 24.7 (scale 0.0 to 105.0)24.7- Fix rate
- 23.5%
- False-positive fixes
- 0
- Wall time
- 19.9 min
- Cost per run
- $2.10
- 70AnthropicDossier
high · Claude Code 2.1.215 / 2.1.218
1 seed. Based on one run, so no interval can be computed.Bugs fixed
Fable 5, Bugs fixed: 24 (scale 0.0 to 105.0), single run, no repeated seeds24(1 seed)- Fix rate
- 22.9%
- False-positive fixes
- 0
- Wall time
- 31.4 min
- Cost per run
- $74.52
- 70AnthropicDossier
medium · Claude Code 2.1.247
1 seed. Based on one run, so no interval can be computed.Bugs fixed
Opus 5, Bugs fixed: 24 (scale 0.0 to 105.0), single run, no repeated seeds24(1 seed)- Fix rate
- 22.9%
- False-positive fixes
- 0
- Wall time
- 30.5 min
- Cost per run
- $37.40
Bugs fixed
MiMo-V2.6-Flash, Bugs fixed: 23.3 (scale 0.0 to 105.0)23.3- Fix rate
- 22.2%
- False-positive fixes
- 0
- Wall time
- 70.7 min
- Cost per run
- $0.49
Bugs fixed
Sonnet 5.5, Bugs fixed: 23.3 (scale 0.0 to 105.0)23.3- Fix rate
- 22.2%
- False-positive fixes
- 0
- Wall time
- 17.4 min
- Cost per run
- $11.11
Bugs fixed
GPT-5.6 Luna, Bugs fixed: 23 (scale 0.0 to 105.0), single run, no repeated seeds23(1 seed)- Fix rate
- 21.9%
- False-positive fixes
- 0
- Wall time
- 135.6 min
- Cost per run
- $2.50
Bugs fixed
Grok 4.6, Bugs fixed: 23 (scale 0.0 to 105.0), single run, no repeated seeds23(1 seed)- Fix rate
- 21.9%
- False-positive fixes
- 0
- Wall time
- 33 min
- Cost per run
- $15.73
- 74AlibabaDossier
low · Claude Code 2.1.267
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
Qwen3.8-Flash, Bugs fixed: 23 (scale 0.0 to 105.0), single run, no repeated seeds23(1 seed)- Fix rate
- 21.9%
- False-positive fixes
- 0
- Wall time
- 98.1 min
- Cost per run
- $1.37
Bugs fixed
MiMo-V2.6-Pro, Bugs fixed: 22.7 (scale 0.0 to 105.0)22.7- Fix rate
- 21.6%
- False-positive fixes
- 0
- Wall time
- 90.9 min
- Cost per run
- $0.86
Bugs fixed
GPT-6.1 Sol, Bugs fixed: 22.5 (scale 0.0 to 105.0)22.5- Fix rate
- 21.4%
- False-positive fixes
- 0
- Wall time
- 25.4 min
- Cost per run
- $1.34
Bugs fixed
Opus 5.5, Bugs fixed: 22.3 (scale 0.0 to 105.0)22.3- Fix rate
- 21.2%
- False-positive fixes
- 0
- Wall time
- 11.5 min
- Cost per run
- $8.34
- 80GoogleDossier
high · Gemini CLI (retired) + model-pinning gateway 0.56.0
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
Gemini 3.7 Flash, Bugs fixed: 22 (scale 0.0 to 105.0), single run, no repeated seeds22(1 seed)- Fix rate
- 21%
- False-positive fixes
- 0
- Wall time
- 96.8 min
- Cost per run
- $8.43
- 80xAIDossier
medium · Grok Build CLI (ACP) 1.0.5
1 seed. Based on one run, so no interval can be computed.Bugs fixed
Grok 4.6, Bugs fixed: 22 (scale 0.0 to 105.0), single run, no repeated seeds22(1 seed)- Fix rate
- 21%
- False-positive fixes
- 0
- Wall time
- 26.5 min
- Cost per run
- $5.80
Bugs fixed
DeepSeek V4.1 Flash, Bugs fixed: 21.7 (scale 0.0 to 105.0)21.7- Fix rate
- 20.7%
- False-positive fixes
- 0
- Wall time
- 55.9 min
- Cost per run
- $0.78
Bugs fixed
Haiku 5.5, Bugs fixed: 21.5 (scale 0.0 to 105.0)21.5- Fix rate
- 20.5%
- False-positive fixes
- 0
- Wall time
- 32.5 min
- Cost per run
- $5.83
- 84AnthropicDossier
medium · Claude Code 2.1.267
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
Fable 5.1, Bugs fixed: 21 (scale 0.0 to 105.0), single run, no repeated seeds21(1 seed)- Fix rate
- 20%
- False-positive fixes
- 0
- Wall time
- 23 min
- Cost per run
- $23.73
- 84Moonshot AIDossier
Claude Code 2.1.215 / 2.1.218
1 seed. Based on one run, so no interval can be computed.Bugs fixed
Kimi K3, Bugs fixed: 21 (scale 0.0 to 105.0), single run, no repeated seeds21(1 seed)- Fix rate
- 20%
- False-positive fixes
- 0
- Wall time
- 107.8 min
- Cost per run
- $25.27
- 84AnthropicDossier
high · Claude Code 2.1.218
1 seed. Based on one run, so no interval can be computed.Bugs fixed
Opus 5, Bugs fixed: 21 (scale 0.0 to 105.0), single run, no repeated seeds21(1 seed)- Fix rate
- 20%
- False-positive fixes
- 0
- Wall time
- 37.4 min
- Cost per run
- $41.51
Bugs fixed
Muse Spark 1.3, Bugs fixed: 20.3 (scale 0.0 to 105.0)20.3- Fix rate
- 19.3%
- False-positive fixes
- 0
- Wall time
- 63.8 min
- Cost per run
- $13.71
Bugs fixed
GPT-5.6 Terra, Bugs fixed: 20 (scale 0.0 to 105.0), single run, no repeated seeds20(1 seed)- Fix rate
- 19%
- False-positive fixes
- 0
- Wall time
- 56.6 min
- Cost per run
- $8.98
- 88OpenAIDossier
high · Codex CLI 0.155.1
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
GPT-6 Sol, Bugs fixed: 20 (scale 0.0 to 105.0), single run, no repeated seeds20(1 seed)- Fix rate
- 19%
- False-positive fixes
- 0
- Wall time
- 33.9 min
- Cost per run
- $4.16
Bugs fixed
Grok 4.7, Bugs fixed: 19.3 (scale 0.0 to 105.0)19.3- Fix rate
- 18.4%
- False-positive fixes
- 0
- Wall time
- 37.9 min
- Cost per run
- $12.29
- 91DeepSeekDossier
high · Claude Code 2.1.267
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
DeepSeek V4.1 Flash, Bugs fixed: 19 (scale 0.0 to 105.0), single run, no repeated seeds19(1 seed)- Fix rate
- 18.1%
- False-positive fixes
- 0
- Wall time
- 26.1 min
- Cost per run
- $0.31
- 91Z.aiDossier
max · Claude Code 2.1.267
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
GLM-5.3, Bugs fixed: 19 (scale 0.0 to 105.0), single run, no repeated seeds19(1 seed)- Fix rate
- 18.1%
- False-positive fixes
- 0
- Wall time
- 40.4 min
- Cost per run
- $15.93
Bugs fixed
Sonnet 5.5, Bugs fixed: 19 (scale 0.0 to 105.0)19- Fix rate
- 18.1%
- False-positive fixes
- 0
- Wall time
- 8.5 min
- Cost per run
- $5.73
Bugs fixed
Muse Spark 1.3, Bugs fixed: 18.7 (scale 0.0 to 105.0)18.7- Fix rate
- 17.8%
- False-positive fixes
- 0
- Wall time
- 75.6 min
- Cost per run
- $12.94
Bugs fixed
GPT-6 Luna, Bugs fixed: 18.3 (scale 0.0 to 105.0)18.3- Fix rate
- 17.4%
- False-positive fixes
- 0
- Wall time
- 99.2 min
- Cost per run
- $0.52
Bugs fixed
Haiku 5.5, Bugs fixed: 18.3 (scale 0.0 to 105.0)18.3- Fix rate
- 17.4%
- False-positive fixes
- 0
- Wall time
- 24.1 min
- Cost per run
- $4.80
Bugs fixed
Gemini 3.8 Flash, Bugs fixed: 18 (scale 0.0 to 105.0)18- Fix rate
- 17.1%
- False-positive fixes
- 0
- Wall time
- 38.7 min
- Cost per run
- $11.03
Bugs fixed
GPT-5.6 Terra, Bugs fixed: 18 (scale 0.0 to 105.0), single run, no repeated seeds18(1 seed)- Fix rate
- 17.1%
- False-positive fixes
- 0
- Wall time
- 28.1 min
- Cost per run
- $5.89
Bugs fixed
Hy4 Preview, Bugs fixed: 18 (scale 0.0 to 105.0), single run, no repeated seeds18(1 seed)- Fix rate
- 17.1%
- False-positive fixes
- 0
- Wall time
- 67.5 min
- Cost per run
- $3.13
- 97XiaomiDossier
thinking · Claude Code 2.1.278
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
MiMo-V2.6-Pro, Bugs fixed: 18 (scale 0.0 to 105.0), single run, no repeated seeds18(1 seed)- Fix rate
- 17.1%
- False-positive fixes
- 0
- Wall time
- 74.9 min
- Cost per run
- $0.59
Bugs fixed
GLM-5.3 Flash, Bugs fixed: 17.7 (scale 0.0 to 105.0)17.7- Fix rate
- 16.9%
- False-positive fixes
- 0
- Wall time
- 63.2 min
- Cost per run
- $1.03
Bugs fixed
Grok 4.5, Bugs fixed: 17 (scale 0.0 to 105.0), single run, no repeated seeds17(1 seed)- Fix rate
- 16.2%
- False-positive fixes
- 0
- Wall time
- 27.9 min
- Cost per run
- $8.50
- 102MetaDossier
xhigh · Claude Code 2.1.222
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
Muse Spark 1.2, Bugs fixed: 17 (scale 0.0 to 105.0), single run, no repeated seeds17(1 seed)- Fix rate
- 16.2%
- False-positive fixes
- 0
- Wall time
- 35.7 min
- Cost per run
- $13.99
- 104DeepSeekDossier
max · Claude Code 2.1.267
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
DeepSeek V4-Pro, Bugs fixed: 16 (scale 0.0 to 105.0), single run, no repeated seeds16(1 seed)- Fix rate
- 15.2%
- False-positive fixes
- 0
- Wall time
- 37.2 min
- Cost per run
- $1.89
- 104GoogleDossier
high · Antigravity CLI 1.1.20
1 seed. Based on one run, so no interval can be computed.Bugs fixed
Gemini 3.7 Flash, Bugs fixed: 16 (scale 0.0 to 105.0), single run, no repeated seeds16(1 seed)- Fix rate
- 15.2%
- False-positive fixes
- 0
- Wall time
- 22.7 min
- Cost per run
- $6.36
- 104Z.aiDossier
high · Claude Code 2.1.269
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
GLM-5.3 Flash, Bugs fixed: 16 (scale 0.0 to 105.0), single run, no repeated seeds16(1 seed)- Fix rate
- 15.2%
- False-positive fixes
- 0
- Wall time
- 62.3 min
- Cost per run
- $0.94
Bugs fixed
Haiku 5.5, Bugs fixed: 16 (scale 0.0 to 105.0)16- Fix rate
- 15.2%
- False-positive fixes
- 0
- Wall time
- 19.3 min
- Cost per run
- $3.56
Bugs fixed
Ox Alpha (stealth), Bugs fixed: 16 (scale 0.0 to 105.0), single run, no repeated seeds16(1 seed)- Fix rate
- 15.2%
- False-positive fixes
- 0
- Wall time
- 59.4 min
- Cost per run
- $0.00
Bugs fixed
Grok 4.7, Bugs fixed: 15.7 (scale 0.0 to 105.0)15.7- Fix rate
- 15%
- False-positive fixes
- 0
- Wall time
- 22.2 min
- Cost per run
- $9.36
Bugs fixed
GPT-5.6 Terra, Bugs fixed: 15 (scale 0.0 to 105.0), single run, no repeated seeds15(1 seed)- Fix rate
- 14.3%
- False-positive fixes
- 0
- Wall time
- 20.2 min
- Cost per run
- $3.87
- 110xAIDossier
low · Grok Build CLI (ACP) 1.0.5
1 seed. Based on one run, so no interval can be computed.Bugs fixed
Grok 4.6, Bugs fixed: 15 (scale 0.0 to 105.0), single run, no repeated seeds15(1 seed)- Fix rate
- 14.3%
- False-positive fixes
- 0
- Wall time
- 18 min
- Cost per run
- $4.14
Bugs fixed
Mistral Large 4, Bugs fixed: 15 (scale 0.0 to 105.0)15- Fix rate
- 14.3%
- False-positive fixes
- 0
- Wall time
- 111.1 min
- Cost per run
- $11.95
- 110AnthropicDossier
max · Claude Code 2.1.267
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
Opus 4.8, Bugs fixed: 15 (scale 0.0 to 105.0), single run, no repeated seeds15(1 seed)- Fix rate
- 14.3%
- False-positive fixes
- 0
- Wall time
- 108.8 min
- Cost per run
- $58.11
Bugs fixed
Qwen3.8-27B, Bugs fixed: 15 (scale 0.0 to 105.0)15- Fix rate
- 14.3%
- False-positive fixes
- 0
- Wall time
- 56.8 min
- Cost per run
- $5.55
Bugs fixed
DeepSeek V4-Flash 0731, Bugs fixed: 14 (scale 0.0 to 105.0), single run, no repeated seeds14(1 seed)- Fix rate
- 13.3%
- False-positive fixes
- 0
- Wall time
- 48.5 min
- Cost per run
- $1.52
- 115OpenAIDossier
xhigh · Codex CLI 0.155.1
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
GPT-6 Luna, Bugs fixed: 14 (scale 0.0 to 105.0), single run, no repeated seeds14(1 seed)- Fix rate
- 13.3%
- False-positive fixes
- 0
- Wall time
- 57 min
- Cost per run
- $0.39
- 115OpenAIDossier
medium · Codex CLI 0.155.1
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
GPT-6 Sol, Bugs fixed: 14 (scale 0.0 to 105.0), single run, no repeated seeds14(1 seed)- Fix rate
- 13.3%
- False-positive fixes
- 0
- Wall time
- 24.5 min
- Cost per run
- $2.39
- 118DeepSeekDossier
high · Claude Code 2.1.267
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
DeepSeek V4-Pro, Bugs fixed: 13 (scale 0.0 to 105.0), single run, no repeated seeds13(1 seed)- Fix rate
- 12.4%
- False-positive fixes
- 0
- Wall time
- 26.6 min
- Cost per run
- $1.08
Bugs fixed
GLM-5.3 Flash, Bugs fixed: 13 (scale 0.0 to 105.0), single run, no repeated seeds13(1 seed)- Fix rate
- 12.4%
- False-positive fixes
- 0
- Wall time
- 57.1 min
- Cost per run
- $0.79
Bugs fixed
GPT-5.6 Luna, Bugs fixed: 13 (scale 0.0 to 105.0), single run, no repeated seeds13(1 seed)- Fix rate
- 12.4%
- False-positive fixes
- 0
- Wall time
- 64.1 min
- Cost per run
- $0.57
Bugs fixed
Muse Spark 1.3, Bugs fixed: 13 (scale 0.0 to 105.0)13- Fix rate
- 12.4%
- False-positive fixes
- 0
- Wall time
- 58.3 min
- Cost per run
- $9.24
- 122AnthropicDossier
medium · Claude Code 2.1.292
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
Haiku 5.5, Bugs fixed: 12 (scale 0.0 to 105.0), single run, no repeated seeds12(1 seed)- Fix rate
- 11.4%
- False-positive fixes
- 0
- Wall time
- 15.5 min
- Cost per run
- $2.66
Bugs fixed
GPT-6.1 Terra, Bugs fixed: 11.3 (scale 0.0 to 105.0)11.3- Fix rate
- 10.8%
- False-positive fixes
- 0
- Wall time
- 15.9 min
- Cost per run
- $0.75
- 124AnthropicDossier
low · Claude Code 2.1.292
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
Haiku 5.5, Bugs fixed: 11 (scale 0.0 to 105.0), single run, no repeated seeds11(1 seed)- Fix rate
- 10.5%
- False-positive fixes
- 0
- Wall time
- 8.7 min
- Cost per run
- $1.61
Bugs fixed
Qwen3.8-27B, Bugs fixed: 10.7 (scale 0.0 to 105.0)10.7- Fix rate
- 10.2%
- False-positive fixes
- 0
- Wall time
- 56.6 min
- Cost per run
- $2.97
Bugs fixed
Gemini 4 Hydrogen, Bugs fixed: 10.3 (scale 0.0 to 105.0)10.3- Fix rate
- 9.8%
- False-positive fixes
- 0
- Wall time
- 7.2 min
- Cost per run
- $0.43
Bugs fixed
GPT-6.1 Terra, Bugs fixed: 9.7 (scale 0.0 to 105.0)9.7- Fix rate
- 9.2%
- False-positive fixes
- 0
- Wall time
- 12.6 min
- Cost per run
- $0.56
Bugs fixed
Muse Spark 1.3, Bugs fixed: 9.7 (scale 0.0 to 105.0)9.7- Fix rate
- 9.2%
- False-positive fixes
- 0
- Wall time
- 36.4 min
- Cost per run
- $5.61
- 129Z.aiDossier
low · Claude Code 2.1.269
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
GLM-5.3 Flash, Bugs fixed: 9 (scale 0.0 to 105.0), single run, no repeated seeds9(1 seed)- Fix rate
- 8.6%
- False-positive fixes
- 0
- Wall time
- 29 min
- Cost per run
- $0.42
- 129OpenAIDossier
medium · Codex CLI 0.150.0-alpha.8
1 seed. Based on one run, so no interval can be computed.Bugs fixed
GPT-5.6 Luna, Bugs fixed: 9 (scale 0.0 to 105.0), single run, no repeated seeds9(1 seed)- Fix rate
- 8.6%
- False-positive fixes
- 0
- Wall time
- 15.4 min
- Cost per run
- $0.33
- 129OpenAIDossier
high · Codex CLI 0.155.1
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
GPT-6 Luna, Bugs fixed: 9 (scale 0.0 to 105.0), single run, no repeated seeds9(1 seed)- Fix rate
- 8.6%
- False-positive fixes
- 0
- Wall time
- 24.4 min
- Cost per run
- $0.13
- 129AnthropicDossier
high · Claude Code 2.1.215 / 2.1.218
1 seed. Based on one run, so no interval can be computed.Bugs fixed
Opus 4.8, Bugs fixed: 9 (scale 0.0 to 105.0), single run, no repeated seeds9(1 seed)- Fix rate
- 8.6%
- False-positive fixes
- 0
- Wall time
- 34.8 min
- Cost per run
- $21.30
- 129AnthropicDossier
high · Claude Code 2.1.218
1 seed. Based on one run, so no interval can be computed.Bugs fixed
Sonnet 5, Bugs fixed: 9 (scale 0.0 to 105.0), single run, no repeated seeds9(1 seed)- Fix rate
- 8.6%
- False-positive fixes
- 0
- Wall time
- 32.8 min
- Cost per run
- $11.38
- 129AnthropicDossier
max · Claude Code 2.1.267
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
Sonnet 5, Bugs fixed: 9 (scale 0.0 to 105.0), single run, no repeated seeds9(1 seed)- Fix rate
- 8.6%
- False-positive fixes
- 0
- Wall time
- 61.3 min
- Cost per run
- $17.96
Bugs fixed
GPT-6.1 Terra, Bugs fixed: 7.3 (scale 0.0 to 105.0)7.3- Fix rate
- 7%
- False-positive fixes
- 0
- Wall time
- 9.6 min
- Cost per run
- $0.31
- 136OpenAIDossier
low · Codex CLI 0.155.1
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
GPT-6 Sol, Bugs fixed: 6 (scale 0.0 to 105.0), single run, no repeated seeds6(1 seed)- Fix rate
- 5.7%
- False-positive fixes
- 0
- Wall time
- 11.1 min
- Cost per run
- $1.08
Bugs fixed
GPT-6.1 Terra, Bugs fixed: 6 (scale 0.0 to 105.0)6- Fix rate
- 5.7%
- False-positive fixes
- 0
- Wall time
- 10.1 min
- Cost per run
- $0.30
Bugs fixed
Gemma 4 31B, Bugs fixed: 4 (scale 0.0 to 105.0), single run, no repeated seeds4(1 seed)- Fix rate
- 3.8%
- False-positive fixes
- 0
- Wall time
- 221.8 min
- Cost per run
- $8.61
- 138OpenAIDossier
low · Codex CLI 0.150.0-alpha.8
1 seed. Based on one run, so no interval can be computed.Bugs fixed
GPT-5.6 Luna, Bugs fixed: 4 (scale 0.0 to 105.0), single run, no repeated seeds4(1 seed)- Fix rate
- 3.8%
- False-positive fixes
- 0
- Wall time
- 5.8 min
- Cost per run
- $0.10
- 138OpenAIDossier
medium · Codex CLI 0.155.1
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
GPT-6 Luna, Bugs fixed: 4 (scale 0.0 to 105.0), single run, no repeated seeds4(1 seed)- Fix rate
- 3.8%
- False-positive fixes
- 0
- Wall time
- 8.6 min
- Cost per run
- $0.060
- 138OpenAIDossier
low · Codex CLI 0.155.1
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
GPT-6 Luna, Bugs fixed: 4 (scale 0.0 to 105.0), single run, no repeated seeds4(1 seed)- Fix rate
- 3.8%
- False-positive fixes
- 0
- Wall time
- 19.3 min
- Cost per run
- $0.20
- 142GoogleDossier
4-bit · Claude Code 2.1.272
Caveat1 seed. Based on one run, so no interval can be computed.Bugs fixed
Gemma 4 31B, Bugs fixed: 3 (scale 0.0 to 105.0), single run, no repeated seeds3(1 seed)- Fix rate
- 2.9%
- False-positive fixes
- 0
- Wall time
- 24.9 min
- Cost per run
- $0.56
Bugs fixed
Haiku 4.5, Bugs fixed: 1.5 (scale 0.0 to 105.0)1.5- Fix rate
- 1.4%
- False-positive fixes
- 0
- Wall time
- 16.9 min
- Cost per run
- $2.22
Bugs fixed
gpt-oss-20b, Bugs fixed: 0 (scale 0.0 to 105.0)0- Fix rate
- 0%
- False-positive fixes
- 0
- Wall time
- 7.4 min
- Cost per run
- $0.11
Bugs fixed
gpt-oss-120b, Bugs fixed: 0 (scale 0.0 to 105.0)0- Fix rate
- 0%
- False-positive fixes
- 0
- Wall time
- 3.7 min
- Cost per run
- $0.11
Per-bug fix matrix
Individual bug IDs resolved per run across 82 distinct bugs observed in this snapshot.
| Model | Variant | Fixed | Fixed bug IDs |
|---|---|---|---|
| anthropic/claude-fable-5-2 | max · Claude Code | 58 | #13#16#17#20#21#22#23#24#25#26#27#28#29#30#32#33#34#35#36#38#39#40#41#42+43 more |
| google/gemini-4-carbon | high · Antigravity CLI | 57.3 | #13#16#20#21#22#23#24#25#26#27#28#29#30#32#33#34#37#38#39#40#42#43#44#45+43 more |
| google/gemini-4-carbon | medium · Antigravity CLI | 56 | #13#16#20#21#22#23#24#25#26#27#28#29#30#32#33#34#37#38#39#40#43#44#45#46+40 more |
| anthropic/claude-sonnet-5-5 | max · Claude Code | 51.3 | #13#16#17#20#21#22#23#25#26#27#28#29#30#32#33#34#36#37#38#39#40#42#43#44+40 more |
| anthropic/claude-fable-5-2 | xhigh · Claude Code | 49 | #13#16#17#20#21#22#24#25#26#27#28#29#30#32#33#34#35#36#38#39#40#41#42#43+36 more |
| google/gemini-4-carbon | low · Antigravity CLI | 46 | #13#16#20#21#22#23#25#26#27#28#29#30#32#33#34#37#38#39#40#43#44#45#46#47+33 more |
| openai/gpt-6-astra | max · Codex CLI | 45 | #13#16#20#21#22#23#25#26#27#28#29#30#33#34#35#36#37#39#40#41#42#43#44#45+31 more |
| openai/gpt-6.1-sol | max · Codex CLI | 44.3 | #13#16#17#20#21#22#23#25#26#27#28#29#30#32#33#34#35#36#37#39#40#41#42#43+30 more |
| openai/gpt-5.6-sol | max · Codex CLI | 43.5 | #13#16#17#20#21#22#25#26#27#28#29#30#33#34#35#38#39#40#41#43#44#45#46#47+28 more |
| openai/gpt-6-astra | xhigh · Codex CLI | 43 | #13#16#20#21#22#23#25#26#27#28#29#30#33#34#35#36#37#39#40#41#43#44#45#47+19 more |
| anthropic/claude-fable-5-2 | high · Claude Code | 43 | #13#16#20#22#25#26#27#28#29#30#33#34#35#36#38#39#40#41#42#43#44#45#46#47+24 more |
| anthropic/claude-fable-5-1 | max · Claude Code | 43 | #13#16#20#22#23#25#26#27#29#30#32#33#34#36#39#40#41#43#45#47#49#50#59#60+19 more |
| deepseek/deepseek-v4.1-flash-1017 | max · DeepSeek Harness | 42.7 | #13#16#17#20#21#22#23#25#26#27#28#29#30#32#33#34#37#38#39#40#41#42#43#44+36 more |
| openai/gpt-6.1-sol | xhigh · Codex CLI | 42.7 | #13#16#17#20#21#22#23#25#26#27#28#29#30#32#33#34#35#36#37#39#40#41#43#44+28 more |
| deepseek/deepseek-v4.1-flash-1017 | max · Pi | 42.2 | #13#16#17#20#21#22#23#25#26#27#28#29#30#33#34#36#37#39#40#41#42#43#44#45+33 more |