Acropolis

Cost frontier and effort scaling

Reasoning effort curves from low to high compute budgets alongside Pareto cost-frontier win counts.

Non-monotone regressionMonotone gain
04998Reasoning effort step (low to high across 5 steps)Qwen3.8-27BFable 5.2Opus 5.5Fable 5Fable 5.1Haiku 5.5

Pareto cost-frontier share across boards

Method, limits, and excluded inputs (90 boards, 896 rows read)

Method rules

  • A curve is one model on one board with one harness and at least two effort levels along the lab's canonical effort scale.
  • Each neighboring step computes score gain, cost gain, and dollars per point when costs are positive and comparable.
  • Upper concave hull ICER extracts the non-dominated efficiency envelope along (cost, score), omitting noisily dominated intermediate steps.
  • Thinking-budget curves group explicit token budgets across models; harness comparisons hold (model, benchmark, effort) constant across multiple scaffolds.

Known limits

  • Non-monotone dips can reflect single-seed sampling noise rather than algorithmic degradation.
  • Effort labels vary by lab and are ordered within each lab's own scale rather than equated across providers.
  • Rows whose effort status is clamped or inert are annotated when sources report reasoning effort diagnostics.

Exclusions

  • 55 boards excluded: board has no model with two or more effort levels
  • 2 boards excluded: rank metric is not a quality score (price, speed, latency, or token count)
  • 4117 curves excluded: model has one effort level on this board and harness
  • 75 rows excluded: hidden row (for example superseded)
  • 22891 rows excluded: row has no effort level (default setting)
  • 324 rows excluded: row sets a thinking budget instead of an effort
  • 21 rows excluded: second row for the same effort in one curve (best row kept)