Acropolis

Statistical specification

Method

The scoring, uncertainty, consensus, and data redistribution rules behind every table, scatter chart, and insight study in Acropolis.

Independent. Run by an operator independent of the model vendors.Vendor-run. Run by the vendor of at least one model on the list.Single run. Based on one run, so no interval can be computed.Grade A. Evidence grade A of A to D: strongest.
  1. 01

    Competition ranking and metric directionality

    Every board carries explicit metric direction (higher-is-better or lower-is-better) from the canonical catalog rather than hardcoded guesses.

    Within a filtered table or leaderboard view, rows are assigned 1224 competition ranks (tied values share the same rank and skip subsequent positions).

    Filtered rows keep their table position while displaying their overall board rank beside the filtered rank whenever filters hide higher-scoring rows.

  2. 02

    Uncertainty intervals and single-run flagging

    Source-reported 95% confidence bounds or standard errors are preserved verbatim when valid.

    When a benchmark reports only a binary percentage pass rate (0–100%) and item count n without a confidence interval, Acropolis derives a 95% binomial interval. Inside interior bounds it uses linear Wald bounds; near 0% or 100% where linear intervals would overshoot, it switches smoothly to a logit-transformed interval bounded within [0, 100].

    Rows backed by a single evaluation seed (runCount = 1) or lacking uncertainty estimates never receive fabricated error bars and render with dashed single-seed score bars.

  3. 03

    Monte Carlo rank bands and #1 statistical ties

    Leaderboard #1 rankings are unstable when the top several models have overlapping 95% intervals.

    Acropolis draws 250 deterministic Monte Carlo samples per row (seeded cleanly by board key and row reference) to report a 2.5th to 97.5th percentile rank band.

    A row is marked Tied #1 whenever its simulated 95% rank band includes rank 1.

  4. 04

    Per-cost-kind Pareto frontiers

    Cost numbers are heterogeneous across sources: some report published API list prices, others report measured episode billing, open-weight floor estimates, or $0 free tiers.

    Pareto frontiers are computed strictly within each cost kind (list, billed, floor, free) so optimistic floor estimates never eclipse verified billed runs.

    In scatter charts using a logarithmic cost axis, verified $0 free runs occupy a dedicated 44px left gutter lane instead of being dropped at log(0).

  5. 05

    Bradley-Terry consensus and disagreement taxonomy

    On quality boards sharing overlapping models, Acropolis solves a board-normalized weighted Bradley-Terry latent capability model scaled onto 0–100 with 120 seeded bootstrap resamples for 95% confidence bounds.

    Boards from the same source are down-weighted by inverse square root of source board count, and composite benchmarks are excluded when two or more component boards are present.

    Models with high cross-board interquartile spread (>= 30 percentile points) are classified as domain specialists (variance across domains) or within-domain disputes (disagreement inside the same domain).

  6. 06

    Evidence coverage grades and vendor audits

    Each model receives an evidence coverage grade from A to D reflecting how many independent operators evaluate it, how recently runs were observed, and whether intervals are reported.

    When sufficient own-lab and peer models (>= 3 each) appear on both an operator-run board and independent domain-matched baselines, Acropolis computes paired percentile differences and difference-in-differences estimates.

  7. 07

    Rights gate and open exports

    Every upstream source is tagged with a redistribution status in the source manifest (open, permission-granted, none-stated, or restricted).

    Public production builds (--mode public-prod) emit link-only stub boards with empty scores for restricted sources and omit restricted data from bulk CSV and JSONL exports.