SWE-Bench verified
SWE-bench Verified, run by Epoch AI.
No filters. Showing 35 of 35 rows.
1[1–3] AnthropicDossier | max | No harness | Opus 4.7, Accuracy (best score across scorers): 83.5% (scale 0.0 to 100.0), 95% confidence interval 80.2 to 86.883.5%(80.2–86.8) | 83.5% | Runs unstated | Cost kind unknown |
2[1–10] OpenAIDossierTied #1 | xhigh | No harness | GPT-5.5 Pre-release, Accuracy (best score across scorers): 80.6% (scale 0.0 to 100.0), 95% confidence interval 77.1 to 84.180.6%(77.1–84.1) | 80.6% | Runs unstated | Cost kind unknown |
3[1–12] GoogleDossierTied #1 | high | No harness | Gemini 3.5 Flash, Accuracy (best score across scorers): 79.3% (scale 0.0 to 100.0), 95% confidence interval 75.7 to 82.979.3%(75.7–82.9) | 79.3% | Runs unstated | Cost kind unknown |
4[2–14] AnthropicDossierTied #1 | No effort set | No harness | Opus 4.6, Accuracy (best score across scorers): 78.7% (scale 0.0 to 100.0), 95% confidence interval 75.1 to 82.478.7%(75.1–82.4) | 78.7% | Runs unstated | Cost kind unknown |
5[2–16] Z.aiDossierTied #1 | max | No harness | GLM-5.2, Accuracy (best score across scorers): 78.7% (scale 0.0 to 100.0), 95% confidence interval 75.0 to 82.478.7%(75.0–82.4) | 78.7% | Runs unstated | Cost kind unknown |
6[2–18] DeepSeekDossier | max | No harness | DeepSeek V4-Pro, Accuracy (best score across scorers): 77.6% (scale 0.0 to 100.0), 95% confidence interval 73.9 to 81.477.6%(73.9–81.4) | 77.6% | Runs unstated | Cost kind unknown |
7[3–20] AlibabaDossier | No effort set | No harness | Qwen3.7 Max, Accuracy (best score across scorers): 77.3% (scale 0.0 to 100.0), 95% confidence interval 73.5 to 81.077.3%(73.5–81.0) | 77.3% | Runs unstated | Cost kind unknown |
8[3–21] OpenAIDossier | high | No harness | GPT-5.4, Accuracy (best score across scorers): 76.9% (scale 0.0 to 100.0), 95% confidence interval 73.1 to 80.676.9%(73.1–80.6) | 76.9% | Runs unstated | Cost kind unknown |
9[3–20] Moonshot AIDossier | No effort set | No harness | Kimi K2.6, Accuracy (best score across scorers): 76.7% (scale 0.0 to 100.0), 95% confidence interval 72.9 to 80.476.7%(72.9–80.4) | 76.7% | Runs unstated | Cost kind unknown |
9[3–21] AnthropicDossier | No effort set | No harness | Opus 4.5, Accuracy (best score across scorers): 76.7% (scale 0.0 to 100.0), 95% confidence interval 72.9 to 80.476.7%(72.9–80.4) | 76.7% | Runs unstated | Cost kind unknown |
9[3–20] AlibabaDossier | No effort set | No harness | Qwen3.6 Max Preview, Accuracy (best score across scorers): 76.7% (scale 0.0 to 100.0), 95% confidence interval 72.9 to 80.476.7%(72.9–80.4) | 76.7% | Runs unstated | Cost kind unknown |
12[4–22] GoogleDossier | No effort set | No harness | Gemini 3.1 Pro Preview (Custom Tools), Accuracy (best score across scorers): 75.6% (scale 0.0 to 100.0), 95% confidence interval 71.8 to 79.475.6%(71.8–79.4) | 75.6% | Runs unstated | Cost kind unknown |
12[4–23] AnthropicDossier | No effort set | No harness | Opus 4.6, Accuracy (best score across scorers): 75.6% (scale 0.0 to 100.0), 95% confidence interval 71.8 to 79.475.6%(71.8–79.4) | 75.6% | Runs unstated | Cost kind unknown |
14[5–24] GoogleDossier | No effort set | No harness | Gemini 3 Flash Preview, Accuracy (best score across scorers): 75.4% (scale 0.0 to 100.0), 95% confidence interval 71.6 to 79.375.4%(71.6–79.3) | 75.4% | Runs unstated | Cost kind unknown |
15[4–24] AnthropicDossier | No effort set | No harness | Sonnet 4.6, Accuracy (best score across scorers): 75.2% (scale 0.0 to 100.0), 95% confidence interval 71.4 to 79.175.2%(71.4–79.1) | 75.2% | Runs unstated | Cost kind unknown |
16[6–25] OpenAIDossier | high | No harness | GPT-5.3 Codex, Accuracy (best score across scorers): 74.8% (scale 0.0 to 100.0), 95% confidence interval 70.9 to 78.774.8%(70.9–78.7) | 74.8% | Runs unstated | Cost kind unknown |
17[7–25] Z.aiDossier | No effort set | No harness | GLM-5.1, Accuracy (best score across scorers): 74.2% (scale 0.0 to 100.0), 95% confidence interval 70.3 to 78.174.2%(70.3–78.1) | 74.2% | Runs unstated | Cost kind unknown |
18[8–26] OpenAIDossier | high | No harness | GPT-5.2, Accuracy (best score across scorers): 73.8% (scale 0.0 to 100.0), 95% confidence interval 69.8 to 77.773.8%(69.8–77.7) | 73.8% | Runs unstated | Cost kind unknown |
18[7–25] Moonshot AIDossier | No effort set | No harness | Kimi K2.5, Accuracy (best score across scorers): 73.8% (scale 0.0 to 100.0), 95% confidence interval 69.8 to 77.773.8%(69.8–77.7) | 73.8% | Runs unstated | Cost kind unknown |
20[8–26] OpenAIDossier | high | No harness | GPT-5, Accuracy (best score across scorers): 73.6% (scale 0.0 to 100.0), 95% confidence interval 69.6 to 77.573.6%(69.6–77.5) | 73.6% | Runs unstated | Cost kind unknown |
21[9–26] AnthropicDossier | No effort set | No harness | Opus 4.1, Accuracy (best score across scorers): 73.3% (scale 0.0 to 100.0), 95% confidence interval 69.4 to 77.373.3%(69.4–77.3) | 73.3% | Runs unstated | Cost kind unknown |
22[10–26] GoogleDossier | No effort set | No harness | Gemini 3 Pro Preview, Accuracy (best score across scorers): 72.9% (scale 0.0 to 100.0), 95% confidence interval 69.0 to 76.972.9%(69.0–76.9) | 72.9% | Runs unstated | Cost kind unknown |
23[10–27] Z.aiDossier | No effort set | No harness | GLM-5, Accuracy (best score across scorers): 72.1% (scale 0.0 to 100.0), 95% confidence interval 68.0 to 76.272.1%(68.0–76.2) | 72.1% | Runs unstated | Cost kind unknown |
24[12–27] OpenAIDossier | medium | No harness | GPT-5, Accuracy (best score across scorers): 71.5% (scale 0.0 to 100.0), 95% confidence interval 67.5 to 75.571.5%(67.5–75.5) | 71.5% | Runs unstated | Cost kind unknown |
25[13–27] AnthropicDossier | No effort set | No harness | Sonnet 4.5, Accuracy (best score across scorers): 71.3% (scale 0.0 to 100.0), 95% confidence interval 67.2 to 75.371.3%(67.2–75.3) | 71.3% | Runs unstated | Cost kind unknown |
26[15–28] AnthropicDossier | No effort set | No harness | Opus 4, Accuracy (best score across scorers): 70.7% (scale 0.0 to 100.0), 95% confidence interval 66.6 to 74.770.7%(66.6–74.7) | 70.7% | Runs unstated | Cost kind unknown |
27[21–29] OpenAIDossier | high | No harness | GPT-5.1, Accuracy (best score across scorers): 68% (scale 0.0 to 100.0), 95% confidence interval 63.8 to 72.168%(63.8–72.1) | 68% | Runs unstated | Cost kind unknown |
28[26–30] OpenAIDossier | high | No harness | GPT-5.1, Accuracy (best score across scorers): 65.9% (scale 0.0 to 100.0), 95% confidence interval 61.7 to 70.165.9%(61.7–70.1) | 65.9% | Runs unstated | Cost kind unknown |
29[26–31] OpenAIDossier | medium | No harness | GPT-5 mini, Accuracy (best score across scorers): 64.7% (scale 0.0 to 100.0), 95% confidence interval 60.4 to 68.964.7%(60.4–68.9) | 64.7% | Runs unstated | Cost kind unknown |
30[28–32] OpenAIDossier | medium | No harness | o3, Accuracy (best score across scorers): 62.3% (scale 0.0 to 100.0), 95% confidence interval 58.0 to 66.662.3%(58.0–66.6) | 62.3% | Runs unstated | Cost kind unknown |
31[28–33] AnthropicDossier | No effort set | No harness | Claude 3.7 Sonnet, Accuracy (best score across scorers): 61% (scale 0.0 to 100.0), 95% confidence interval 56.6 to 65.361%(56.6–65.3) | 61% | Runs unstated | Cost kind unknown |
32[30–33] AlibabaDossier | No effort set | No harness | Qwen3.6 Plus, Accuracy (best score across scorers): 57.9% (scale 0.0 to 100.0), 95% confidence interval 53.4 to 62.357.9%(53.4–62.3) | 57.9% | Runs unstated | Cost kind unknown |
33[31–33] GoogleDossier | No effort set | No harness | Gemini 2.5 Pro, Accuracy (best score across scorers): 57.6% (scale 0.0 to 100.0), 95% confidence interval 53.1 to 62.057.6%(53.1–62.0) | 57.6% | Runs unstated | Cost kind unknown |
34 OpenAIDossier | No effort set | No harness | GPT-4.1, Accuracy (best score across scorers): 48.5% (scale 0.0 to 100.0), 95% confidence interval 44.1 to 53.048.5%(44.1–53.0) | 48.5% | Runs unstated | Cost kind unknown |
35 OpenAIDossier | No effort set | No harness | GPT-4o (Nov '24), Accuracy (best score across scorers): 31% (scale 0.0 to 100.0), 95% confidence interval 26.9 to 35.131%(26.9–35.1) | 31% | Runs unstated | Cost kind unknown |
- 1AnthropicDossier
max
Accuracy (best score across scorers)
Opus 4.7, Accuracy (best score across scorers): 83.5% (scale 0.0 to 100.0), 95% confidence interval 80.2 to 86.883.5%(80.2–86.8)- Mean score
- 83.5%
Accuracy (best score across scorers)
GPT-5.5 Pre-release, Accuracy (best score across scorers): 80.6% (scale 0.0 to 100.0), 95% confidence interval 77.1 to 84.180.6%(77.1–84.1)- Mean score
- 80.6%
Accuracy (best score across scorers)
Gemini 3.5 Flash, Accuracy (best score across scorers): 79.3% (scale 0.0 to 100.0), 95% confidence interval 75.7 to 82.979.3%(75.7–82.9)- Mean score
- 79.3%
Accuracy (best score across scorers)
Opus 4.6, Accuracy (best score across scorers): 78.7% (scale 0.0 to 100.0), 95% confidence interval 75.1 to 82.478.7%(75.1–82.4)- Mean score
- 78.7%
Accuracy (best score across scorers)
GLM-5.2, Accuracy (best score across scorers): 78.7% (scale 0.0 to 100.0), 95% confidence interval 75.0 to 82.478.7%(75.0–82.4)- Mean score
- 78.7%
- 6DeepSeekDossier
max
Accuracy (best score across scorers)
DeepSeek V4-Pro, Accuracy (best score across scorers): 77.6% (scale 0.0 to 100.0), 95% confidence interval 73.9 to 81.477.6%(73.9–81.4)- Mean score
- 77.6%
- 7AlibabaDossier
Accuracy (best score across scorers)
Qwen3.7 Max, Accuracy (best score across scorers): 77.3% (scale 0.0 to 100.0), 95% confidence interval 73.5 to 81.077.3%(73.5–81.0)- Mean score
- 77.3%
- 8OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5.4, Accuracy (best score across scorers): 76.9% (scale 0.0 to 100.0), 95% confidence interval 73.1 to 80.676.9%(73.1–80.6)- Mean score
- 76.9%
- 9Moonshot AIDossier
Accuracy (best score across scorers)
Kimi K2.6, Accuracy (best score across scorers): 76.7% (scale 0.0 to 100.0), 95% confidence interval 72.9 to 80.476.7%(72.9–80.4)- Mean score
- 76.7%
- 9AnthropicDossier
Accuracy (best score across scorers)
Opus 4.5, Accuracy (best score across scorers): 76.7% (scale 0.0 to 100.0), 95% confidence interval 72.9 to 80.476.7%(72.9–80.4)- Mean score
- 76.7%
- 9AlibabaDossier
Accuracy (best score across scorers)
Qwen3.6 Max Preview, Accuracy (best score across scorers): 76.7% (scale 0.0 to 100.0), 95% confidence interval 72.9 to 80.476.7%(72.9–80.4)- Mean score
- 76.7%
- 12GoogleDossier
Accuracy (best score across scorers)
Gemini 3.1 Pro Preview (Custom Tools), Accuracy (best score across scorers): 75.6% (scale 0.0 to 100.0), 95% confidence interval 71.8 to 79.475.6%(71.8–79.4)- Mean score
- 75.6%
- 12AnthropicDossier
Accuracy (best score across scorers)
Opus 4.6, Accuracy (best score across scorers): 75.6% (scale 0.0 to 100.0), 95% confidence interval 71.8 to 79.475.6%(71.8–79.4)- Mean score
- 75.6%
- 14GoogleDossier
Accuracy (best score across scorers)
Gemini 3 Flash Preview, Accuracy (best score across scorers): 75.4% (scale 0.0 to 100.0), 95% confidence interval 71.6 to 79.375.4%(71.6–79.3)- Mean score
- 75.4%
- 15AnthropicDossier
Accuracy (best score across scorers)
Sonnet 4.6, Accuracy (best score across scorers): 75.2% (scale 0.0 to 100.0), 95% confidence interval 71.4 to 79.175.2%(71.4–79.1)- Mean score
- 75.2%
- 16OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5.3 Codex, Accuracy (best score across scorers): 74.8% (scale 0.0 to 100.0), 95% confidence interval 70.9 to 78.774.8%(70.9–78.7)- Mean score
- 74.8%
- 17Z.aiDossier
Accuracy (best score across scorers)
GLM-5.1, Accuracy (best score across scorers): 74.2% (scale 0.0 to 100.0), 95% confidence interval 70.3 to 78.174.2%(70.3–78.1)- Mean score
- 74.2%
- 18OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5.2, Accuracy (best score across scorers): 73.8% (scale 0.0 to 100.0), 95% confidence interval 69.8 to 77.773.8%(69.8–77.7)- Mean score
- 73.8%
- 18Moonshot AIDossier
Accuracy (best score across scorers)
Kimi K2.5, Accuracy (best score across scorers): 73.8% (scale 0.0 to 100.0), 95% confidence interval 69.8 to 77.773.8%(69.8–77.7)- Mean score
- 73.8%
- 20OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5, Accuracy (best score across scorers): 73.6% (scale 0.0 to 100.0), 95% confidence interval 69.6 to 77.573.6%(69.6–77.5)- Mean score
- 73.6%
- 21AnthropicDossier
Accuracy (best score across scorers)
Opus 4.1, Accuracy (best score across scorers): 73.3% (scale 0.0 to 100.0), 95% confidence interval 69.4 to 77.373.3%(69.4–77.3)- Mean score
- 73.3%
- 22GoogleDossier
Accuracy (best score across scorers)
Gemini 3 Pro Preview, Accuracy (best score across scorers): 72.9% (scale 0.0 to 100.0), 95% confidence interval 69.0 to 76.972.9%(69.0–76.9)- Mean score
- 72.9%
- 23Z.aiDossier
Accuracy (best score across scorers)
GLM-5, Accuracy (best score across scorers): 72.1% (scale 0.0 to 100.0), 95% confidence interval 68.0 to 76.272.1%(68.0–76.2)- Mean score
- 72.1%
- 24OpenAIDossier
medium
Accuracy (best score across scorers)
GPT-5, Accuracy (best score across scorers): 71.5% (scale 0.0 to 100.0), 95% confidence interval 67.5 to 75.571.5%(67.5–75.5)- Mean score
- 71.5%
- 25AnthropicDossier
Accuracy (best score across scorers)
Sonnet 4.5, Accuracy (best score across scorers): 71.3% (scale 0.0 to 100.0), 95% confidence interval 67.2 to 75.371.3%(67.2–75.3)- Mean score
- 71.3%
- 26AnthropicDossier
Accuracy (best score across scorers)
Opus 4, Accuracy (best score across scorers): 70.7% (scale 0.0 to 100.0), 95% confidence interval 66.6 to 74.770.7%(66.6–74.7)- Mean score
- 70.7%
- 27OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5.1, Accuracy (best score across scorers): 68% (scale 0.0 to 100.0), 95% confidence interval 63.8 to 72.168%(63.8–72.1)- Mean score
- 68%
- 28OpenAIDossier
high
Accuracy (best score across scorers)
GPT-5.1, Accuracy (best score across scorers): 65.9% (scale 0.0 to 100.0), 95% confidence interval 61.7 to 70.165.9%(61.7–70.1)- Mean score
- 65.9%
- 29OpenAIDossier
medium
Accuracy (best score across scorers)
GPT-5 mini, Accuracy (best score across scorers): 64.7% (scale 0.0 to 100.0), 95% confidence interval 60.4 to 68.964.7%(60.4–68.9)- Mean score
- 64.7%
- 30OpenAIDossier
medium
Accuracy (best score across scorers)
o3, Accuracy (best score across scorers): 62.3% (scale 0.0 to 100.0), 95% confidence interval 58.0 to 66.662.3%(58.0–66.6)- Mean score
- 62.3%
- 31AnthropicDossier
Accuracy (best score across scorers)
Claude 3.7 Sonnet, Accuracy (best score across scorers): 61% (scale 0.0 to 100.0), 95% confidence interval 56.6 to 65.361%(56.6–65.3)- Mean score
- 61%
- 32AlibabaDossier
Accuracy (best score across scorers)
Qwen3.6 Plus, Accuracy (best score across scorers): 57.9% (scale 0.0 to 100.0), 95% confidence interval 53.4 to 62.357.9%(53.4–62.3)- Mean score
- 57.9%
- 33GoogleDossier
Accuracy (best score across scorers)
Gemini 2.5 Pro, Accuracy (best score across scorers): 57.6% (scale 0.0 to 100.0), 95% confidence interval 53.1 to 62.057.6%(53.1–62.0)- Mean score
- 57.6%
- 34OpenAIDossier
Accuracy (best score across scorers)
GPT-4.1, Accuracy (best score across scorers): 48.5% (scale 0.0 to 100.0), 95% confidence interval 44.1 to 53.048.5%(44.1–53.0)- Mean score
- 48.5%
- 35OpenAIDossier
Accuracy (best score across scorers)
GPT-4o (Nov '24), Accuracy (best score across scorers): 31% (scale 0.0 to 100.0), 95% confidence interval 26.9 to 35.131%(26.9–35.1)- Mean score
- 31%