Terminal-Bench 4
Terminal-Bench 4, a component of the Vals Index.
No filters. Showing 38 of 38 rows.
1[1–6] AnthropicDossier | high | No harness | Fable 5.2, Accuracy: 65.2% (scale 0.0 to 100.0), 95% confidence interval 60.6 to 69.765.2%(60.6–69.7) | 3,307.2 s | $14.56 | Runs unstated | Cost kind unknown |
1[1–4] AnthropicDossier | max | No harness | Opus 5.5, Accuracy: 65.2% (scale 0.0 to 100.0), 95% confidence interval 65.2 to 65.265.2%(65.2–65.2) | 3,810.9 s | $12.04 | Runs unstated | Cost kind unknown |
3[1–5] AnthropicDossierTied #1 | max | No harness | Sonnet 5.5, Accuracy: 64.1% (scale 0.0 to 100.0), 95% confidence interval 62.2 to 66.164.1%(62.2–66.1) | 4,928.9 s | $10.75 | Runs unstated | Cost kind unknown |
4[1–6] Moonshot AIDossierTied #1 | max | No harness | Kimi K3.1 Pro, Accuracy: 63.6% (scale 0.0 to 100.0), 95% confidence interval 60.2 to 67.163.6%(60.2–67.1) | 2,960.7 s | $4.57 | Runs unstated | Cost kind unknown |
5[1–7] GoogleDossierTied #1 | high | No harness | Gemini 4 Carbon, Accuracy: 63.1% (scale 0.0 to 100.0), 95% confidence interval 58.8 to 67.463.1%(58.8–67.4) | 2,706.8 s | $7.45 | Runs unstated | Cost kind unknown |
6[1–11] OpenAIDossierTied #1 | max | No harness | GPT-6 Astra, Accuracy: 59.6% (scale 0.0 to 100.0), 95% confidence interval 51.0 to 68.259.6%(51.0–68.2) | 2,142.1 s | $9.58 | Runs unstated | Cost kind unknown |
7[3–10] MetaDossierTied #1 | max | No harness | Muse Wonder Full 1.0, Accuracy: 58.1% (scale 0.0 to 100.0), 95% confidence interval 51.6 to 64.658.1%(51.6–64.6) | 4,330.7 s | $10.53 | Runs unstated | Cost kind unknown |
8[5–10] GoogleDossier | high | No harness | Gemini 4 Argon, Accuracy: 57.6% (scale 0.0 to 100.0), 95% confidence interval 53.0 to 62.157.6%(53.0–62.1) | 3,214.8 s | $17.64 | Runs unstated | Cost kind unknown |
9[7–10] OpenAIDossier | max | No harness | GPT-6.1 Sol, Accuracy: 55.1% (scale 0.0 to 100.0), 95% confidence interval 51.5 to 58.655.1%(51.5–58.6) | 2,711.7 s | $1.72 | Runs unstated | Cost kind unknown |
10[6–13] GoogleDossier | xhigh | No harness | Gemini 4 Hydrogen, Accuracy: 54% (scale 0.0 to 100.0), 95% confidence interval 47.5 to 60.554%(47.5–60.5) | 1,839.9 s | $7.56 | Runs unstated | Cost kind unknown |
11[9–13] OpenAIDossier | max | No harness | GPT-6.1 Terra, Accuracy: 49.5% (scale 0.0 to 100.0), 95% confidence interval 45.9 to 53.149.5%(45.9–53.1) | 2,032.8 s | $2.16 | Runs unstated | Cost kind unknown |
11[9–13] MetaDossier | max | No harness | Muse Wonder 1.0, Accuracy: 49.5% (scale 0.0 to 100.0), 95% confidence interval 45.2 to 53.849.5%(45.2–53.8) | 3,280.7 s | $14.66 | Runs unstated | Cost kind unknown |
13[11–13] AlibabaDossier | xhigh | No harness | Qwen4 Max, Accuracy: 48.5% (scale 0.0 to 100.0), 95% confidence interval 46.8 to 50.248.5%(46.8–50.2) | 4,489.8 s | $3.90 | Runs unstated | Cost kind unknown |
14[14–18] TencentDossier | No effort set | No harness | Hy4, Accuracy: 39.4% (scale 0.0 to 100.0), 95% confidence interval 37.7 to 41.139.4%(37.7–41.1) | 2,600.4 s | $1.28 | Runs unstated | Cost kind unknown |
14[14–20] MetaDossier | max | No harness | Muse Ember, Accuracy: 39.4% (scale 0.0 to 100.0), 95% confidence interval 34.9 to 43.939.4%(34.9–43.9) | 1,581.4 s | $0.22 | Runs unstated | Cost kind unknown |
16[14–19] Z.aiDossier | max | No harness | GLM-5.3, Accuracy: 38.9% (scale 0.0 to 100.0), 95% confidence interval 35.3 to 42.538.9%(35.3–42.5) | 4,952.9 s | $9.37 | Runs unstated | Cost kind unknown |
16[14–19] AlibabaDossier | No effort set | No harness | Qwen3.8 Max Internal, Accuracy: 38.9% (scale 0.0 to 100.0), 95% confidence interval 36.3 to 41.538.9%(36.3–41.5) | 8,591.4 s | $12.15 | Runs unstated | Cost kind unknown |
18[14–21] Z.aiDossier | max | No harness | GLM-5.3 Flash, Accuracy: 37.4% (scale 0.0 to 100.0), 95% confidence interval 32.1 to 42.637.4%(32.1–42.6) | 3,681.1 s | $1.22 | Runs unstated | Cost kind unknown |
19[14–21] AnthropicDossier | max | No harness | Haiku 5.5, Accuracy: 36.4% (scale 0.0 to 100.0), 95% confidence interval 31.8 to 40.936.4%(31.8–40.9) | 3,116.4 s | $3.12 | Runs unstated | Cost kind unknown |
20[18–24] DeepSeekDossier | max | No harness | DeepSeek V4.1 Flash 1017, Accuracy: 31.8% (scale 0.0 to 100.0), 95% confidence interval 26.7 to 37.031.8%(26.7–37.0) | 2,470.4 s | $0.54 | Runs unstated | Cost kind unknown |
20[18–24] StepFunDossier | No effort set | No harness | Step 5 Preview, Accuracy: 31.8% (scale 0.0 to 100.0), 95% confidence interval 26.7 to 37.031.8%(26.7–37.0) | 7,067 s | $3.11 | Runs unstated | Cost kind unknown |
22[17–25] XiaomiDossier | No effort set | No harness | MiMo-V2.6-Pro, Accuracy: 31.3% (scale 0.0 to 100.0), 95% confidence interval 25.3 to 37.331.3%(25.3–37.3) | 9,720 s | $0.46 | Runs unstated | Cost kind unknown |
23[21–25] xAIDossier | xhigh | No harness | Grok 4.7, Accuracy: 28.8% (scale 0.0 to 100.0), 95% confidence interval 24.3 to 33.328.8%(24.3–33.3) | 3,007.5 s | $18.01 | Runs unstated | Cost kind unknown |
24[19–28] MistralDossier | high | No harness | Mistral Large 4 OSS, Accuracy: 26.3% (scale 0.0 to 100.0), 95% confidence interval 16.2 to 36.326.3%(16.2–36.3) | 3,872.7 s | $2.73 | Runs unstated | Cost kind unknown |
25[23–26] MiniMaxDossier | max | No harness | MiniMax M3.1 Flash, Accuracy: 25.3% (scale 0.0 to 100.0), 95% confidence interval 23.3 to 27.225.3%(23.3–27.2) | 3,030.6 s | $0.73 | Runs unstated | Cost kind unknown |
26[24–26] XiaomiDossier | No effort set | No harness | MiMo-V2.6-Flash, Accuracy: 24.2% (scale 0.0 to 100.0), 95% confidence interval 21.3 to 27.224.2%(21.3–27.2) | 8,712.2 s | $0.21 | Runs unstated | Cost kind unknown |
27[26–29] GoogleDossier | high | No harness | Gemini 3.8 Flash, Accuracy: 19.7% (scale 0.0 to 100.0), 95% confidence interval 15.2 to 24.219.7%(15.2–24.2) | 6,427.6 s | $8.65 | Runs unstated | Cost kind unknown |
28[27–29] Moonshot AIDossier | max | No harness | Kimi K3, Accuracy: 17.2% (scale 0.0 to 100.0), 95% confidence interval 16.2 to 18.217.2%(16.2–18.2) | 11,353.5 s | $12.48 | Runs unstated | Cost kind unknown |
29[28–31] DeepSeekDossier | max | No harness | DeepSeek V4 Pro 0813, Accuracy: 14.1% (scale 0.0 to 100.0), 95% confidence interval 10.2 to 18.114.1%(10.2–18.1) | 5,493.6 s | $3.31 | Runs unstated | Cost kind unknown |
30[29–31] OpenAIDossier | max | No harness | GPT-6 Luna, Accuracy: 13.6% (scale 0.0 to 100.0), 95% confidence interval 10.7 to 16.613.6%(10.7–16.6) | 1,965.6 s | $0.35 | Runs unstated | Cost kind unknown |
31[28–32] Thinking MachinesDossier | No effort set | No harness | Inkling 2 Small, Accuracy: 12.6% (scale 0.0 to 100.0), 95% confidence interval 6.6 to 18.612.6%(6.6–18.6) | 6,062.7 s | $8.22 | Runs unstated | Cost kind unknown |
32[31–33] TencentDossier | No effort set | No harness | Hy4 Preview, Accuracy: 8.1% (scale 0.0 to 100.0), 95% confidence interval 5.5 to 10.78.1%(5.5–10.7) | 7,798.7 s | $3.07 | Runs unstated | Cost kind unknown |
33[32–33] MetaDossier | xhigh | No harness | Muse Spark Open, Accuracy: 6.6% (scale 0.0 to 100.0), 95% confidence interval 5.6 to 7.66.6%(5.6–7.6) | 7,466.4 s | $7.26 | Runs unstated | Cost kind unknown |
34[33–35] ReflectionDossier | No effort set | No harness | Beam 1.0, Accuracy: 4% (scale 0.0 to 100.0), 95% confidence interval 1.4 to 6.74%(1.4–6.7) | 1,154.6 s | Not tested | Runs unstated | Cost kind unknown |
35[34–35] InceptionDossier | high | No harness | Mercury 3, Accuracy: 3% (scale 0.0 to 100.0), 95% confidence interval 3.0 to 3.03%(3.0–3.0) | 1,987.2 s | $0.42 | Runs unstated | Cost kind unknown |
36[36–38] Thinking MachinesDossier | No effort set | No harness | Inkling, Accuracy: 0.5% (scale 0.0 to 100.0), 95% confidence interval 0.1 to 3.50.5%(0.1–3.5) | 1,378.8 s | $2.50 | Runs unstated | Cost kind unknown |
37[36–37] Thinking MachinesDossier | No effort set | No harness | Inkling Small, Accuracy: 0% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 0.00%(0.0–0.0) | 705.1 s | $0.32 | Runs unstated | Cost kind unknown |
37[37–38] NVIDIADossier | high | No harness | Nemotron 3.5 Super VL, Accuracy: 0% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 0.00%(0.0–0.0) | 895.6 s | $0.86 | Runs unstated | Cost kind unknown |
- 1AnthropicDossier
high
Accuracy
Fable 5.2, Accuracy: 65.2% (scale 0.0 to 100.0), 95% confidence interval 60.6 to 69.765.2%(60.6–69.7)- Duration per test
- 3,307.2 s
- Cost per test
- $14.56
- 1AnthropicDossier
max
Accuracy
Opus 5.5, Accuracy: 65.2% (scale 0.0 to 100.0), 95% confidence interval 65.2 to 65.265.2%(65.2–65.2)- Duration per test
- 3,810.9 s
- Cost per test
- $12.04
Accuracy
Sonnet 5.5, Accuracy: 64.1% (scale 0.0 to 100.0), 95% confidence interval 62.2 to 66.164.1%(62.2–66.1)- Duration per test
- 4,928.9 s
- Cost per test
- $10.75
Accuracy
Kimi K3.1 Pro, Accuracy: 63.6% (scale 0.0 to 100.0), 95% confidence interval 60.2 to 67.163.6%(60.2–67.1)- Duration per test
- 2,960.7 s
- Cost per test
- $4.57
Accuracy
Gemini 4 Carbon, Accuracy: 63.1% (scale 0.0 to 100.0), 95% confidence interval 58.8 to 67.463.1%(58.8–67.4)- Duration per test
- 2,706.8 s
- Cost per test
- $7.45
Accuracy
GPT-6 Astra, Accuracy: 59.6% (scale 0.0 to 100.0), 95% confidence interval 51.0 to 68.259.6%(51.0–68.2)- Duration per test
- 2,142.1 s
- Cost per test
- $9.58
Accuracy
Muse Wonder Full 1.0, Accuracy: 58.1% (scale 0.0 to 100.0), 95% confidence interval 51.6 to 64.658.1%(51.6–64.6)- Duration per test
- 4,330.7 s
- Cost per test
- $10.53
- 8GoogleDossier
high
Accuracy
Gemini 4 Argon, Accuracy: 57.6% (scale 0.0 to 100.0), 95% confidence interval 53.0 to 62.157.6%(53.0–62.1)- Duration per test
- 3,214.8 s
- Cost per test
- $17.64
- 9OpenAIDossier
max
Accuracy
GPT-6.1 Sol, Accuracy: 55.1% (scale 0.0 to 100.0), 95% confidence interval 51.5 to 58.655.1%(51.5–58.6)- Duration per test
- 2,711.7 s
- Cost per test
- $1.72
- 10GoogleDossier
xhigh
Accuracy
Gemini 4 Hydrogen, Accuracy: 54% (scale 0.0 to 100.0), 95% confidence interval 47.5 to 60.554%(47.5–60.5)- Duration per test
- 1,839.9 s
- Cost per test
- $7.56
- 11OpenAIDossier
max
Accuracy
GPT-6.1 Terra, Accuracy: 49.5% (scale 0.0 to 100.0), 95% confidence interval 45.9 to 53.149.5%(45.9–53.1)- Duration per test
- 2,032.8 s
- Cost per test
- $2.16
- 11MetaDossier
max
Accuracy
Muse Wonder 1.0, Accuracy: 49.5% (scale 0.0 to 100.0), 95% confidence interval 45.2 to 53.849.5%(45.2–53.8)- Duration per test
- 3,280.7 s
- Cost per test
- $14.66
- 13AlibabaDossier
xhigh
Accuracy
Qwen4 Max, Accuracy: 48.5% (scale 0.0 to 100.0), 95% confidence interval 46.8 to 50.248.5%(46.8–50.2)- Duration per test
- 4,489.8 s
- Cost per test
- $3.90
- 14TencentDossier
Accuracy
Hy4, Accuracy: 39.4% (scale 0.0 to 100.0), 95% confidence interval 37.7 to 41.139.4%(37.7–41.1)- Duration per test
- 2,600.4 s
- Cost per test
- $1.28
- 14MetaDossier
max
Accuracy
Muse Ember, Accuracy: 39.4% (scale 0.0 to 100.0), 95% confidence interval 34.9 to 43.939.4%(34.9–43.9)- Duration per test
- 1,581.4 s
- Cost per test
- $0.22
- 16Z.aiDossier
max
Accuracy
GLM-5.3, Accuracy: 38.9% (scale 0.0 to 100.0), 95% confidence interval 35.3 to 42.538.9%(35.3–42.5)- Duration per test
- 4,952.9 s
- Cost per test
- $9.37
- 16AlibabaDossier
Accuracy
Qwen3.8 Max Internal, Accuracy: 38.9% (scale 0.0 to 100.0), 95% confidence interval 36.3 to 41.538.9%(36.3–41.5)- Duration per test
- 8,591.4 s
- Cost per test
- $12.15
- 18Z.aiDossier
max
Accuracy
GLM-5.3 Flash, Accuracy: 37.4% (scale 0.0 to 100.0), 95% confidence interval 32.1 to 42.637.4%(32.1–42.6)- Duration per test
- 3,681.1 s
- Cost per test
- $1.22
- 19AnthropicDossier
max
Accuracy
Haiku 5.5, Accuracy: 36.4% (scale 0.0 to 100.0), 95% confidence interval 31.8 to 40.936.4%(31.8–40.9)- Duration per test
- 3,116.4 s
- Cost per test
- $3.12
- 20DeepSeekDossier
max
Accuracy
DeepSeek V4.1 Flash 1017, Accuracy: 31.8% (scale 0.0 to 100.0), 95% confidence interval 26.7 to 37.031.8%(26.7–37.0)- Duration per test
- 2,470.4 s
- Cost per test
- $0.54
- 20StepFunDossier
Accuracy
Step 5 Preview, Accuracy: 31.8% (scale 0.0 to 100.0), 95% confidence interval 26.7 to 37.031.8%(26.7–37.0)- Duration per test
- 7,067 s
- Cost per test
- $3.11
- 22XiaomiDossier
Accuracy
MiMo-V2.6-Pro, Accuracy: 31.3% (scale 0.0 to 100.0), 95% confidence interval 25.3 to 37.331.3%(25.3–37.3)- Duration per test
- 9,720 s
- Cost per test
- $0.46
- 23xAIDossier
xhigh
Accuracy
Grok 4.7, Accuracy: 28.8% (scale 0.0 to 100.0), 95% confidence interval 24.3 to 33.328.8%(24.3–33.3)- Duration per test
- 3,007.5 s
- Cost per test
- $18.01
- 24MistralDossier
high
Accuracy
Mistral Large 4 OSS, Accuracy: 26.3% (scale 0.0 to 100.0), 95% confidence interval 16.2 to 36.326.3%(16.2–36.3)- Duration per test
- 3,872.7 s
- Cost per test
- $2.73
- 25MiniMaxDossier
max
Accuracy
MiniMax M3.1 Flash, Accuracy: 25.3% (scale 0.0 to 100.0), 95% confidence interval 23.3 to 27.225.3%(23.3–27.2)- Duration per test
- 3,030.6 s
- Cost per test
- $0.73
- 26XiaomiDossier
Accuracy
MiMo-V2.6-Flash, Accuracy: 24.2% (scale 0.0 to 100.0), 95% confidence interval 21.3 to 27.224.2%(21.3–27.2)- Duration per test
- 8,712.2 s
- Cost per test
- $0.21
- 27GoogleDossier
high
Accuracy
Gemini 3.8 Flash, Accuracy: 19.7% (scale 0.0 to 100.0), 95% confidence interval 15.2 to 24.219.7%(15.2–24.2)- Duration per test
- 6,427.6 s
- Cost per test
- $8.65
- 28Moonshot AIDossier
max
Accuracy
Kimi K3, Accuracy: 17.2% (scale 0.0 to 100.0), 95% confidence interval 16.2 to 18.217.2%(16.2–18.2)- Duration per test
- 11,353.5 s
- Cost per test
- $12.48
- 29DeepSeekDossier
max
Accuracy
DeepSeek V4 Pro 0813, Accuracy: 14.1% (scale 0.0 to 100.0), 95% confidence interval 10.2 to 18.114.1%(10.2–18.1)- Duration per test
- 5,493.6 s
- Cost per test
- $3.31
- 30OpenAIDossier
max
Accuracy
GPT-6 Luna, Accuracy: 13.6% (scale 0.0 to 100.0), 95% confidence interval 10.7 to 16.613.6%(10.7–16.6)- Duration per test
- 1,965.6 s
- Cost per test
- $0.35
- 31Thinking MachinesDossier
Accuracy
Inkling 2 Small, Accuracy: 12.6% (scale 0.0 to 100.0), 95% confidence interval 6.6 to 18.612.6%(6.6–18.6)- Duration per test
- 6,062.7 s
- Cost per test
- $8.22
- 32TencentDossier
Accuracy
Hy4 Preview, Accuracy: 8.1% (scale 0.0 to 100.0), 95% confidence interval 5.5 to 10.78.1%(5.5–10.7)- Duration per test
- 7,798.7 s
- Cost per test
- $3.07
- 33MetaDossier
xhigh
Accuracy
Muse Spark Open, Accuracy: 6.6% (scale 0.0 to 100.0), 95% confidence interval 5.6 to 7.66.6%(5.6–7.6)- Duration per test
- 7,466.4 s
- Cost per test
- $7.26
- 34ReflectionDossier
Accuracy
Beam 1.0, Accuracy: 4% (scale 0.0 to 100.0), 95% confidence interval 1.4 to 6.74%(1.4–6.7)- Duration per test
- 1,154.6 s
- Cost per test
- Not tested
- 35InceptionDossier
high
Accuracy
Mercury 3, Accuracy: 3% (scale 0.0 to 100.0), 95% confidence interval 3.0 to 3.03%(3.0–3.0)- Duration per test
- 1,987.2 s
- Cost per test
- $0.42
- 36Thinking MachinesDossier
Accuracy
Inkling, Accuracy: 0.5% (scale 0.0 to 100.0), 95% confidence interval 0.1 to 3.50.5%(0.1–3.5)- Duration per test
- 1,378.8 s
- Cost per test
- $2.50
- 37Thinking MachinesDossier
Accuracy
Inkling Small, Accuracy: 0% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 0.00%(0.0–0.0)- Duration per test
- 705.1 s
- Cost per test
- $0.32
- 37NVIDIADossier
high
Accuracy
Nemotron 3.5 Super VL, Accuracy: 0% (scale 0.0 to 100.0), 95% confidence interval 0.0 to 0.00%(0.0–0.0)- Duration per test
- 895.6 s
- Cost per test
- $0.86