A 2025 paper presenting a statistical framework that places disparate AI benchmarks onto a single numerical scale by treating model capability and benchmark difficulty as latent variables jointly inferred from observed performance data. Named after the Rosetta Stone, the method lets any model and any benchmark be located on one difficulty/capability scale even when they were never evaluated against each other directly. The paper documents three applications: measuring the speed of AI progress, estimating algorithmic-efficiency improvements, and detecting accelerations in progress.
Authors: Anson Ho (Epoch AI), Jean-Stanislas Denain (Epoch AI), David Atanasov (Epoch AI), Samuel Albanie (Google DeepMind), Rohin Shah (Google DeepMind) Funder: Google DeepMind Code: github.com Source: Epoch AI / arXiv
Method
The framework stitches disparate AI benchmarks onto a single numerical scale by treating model capability and benchmark difficulty as latent variables, jointly inferred from observed performance data. It is presented as a response to three measurement problems the authors identify:
- Most AI benchmarks saturate within months or years after introduction.
- Models can be compared only if they share a benchmark, but in practice they rarely do across long time horizons.
- No assumption is made about how capabilities evolve with time or training compute.
The output is a single difficulty/capability scale on which any model and any benchmark can be located, regardless of whether they were ever evaluated against each other directly. The authors note the method is particularly suited to cross-time comparisons (for example GPT-3 versus GPT-5, or 2020 versus 2026) where the underlying benchmark sets have drifted entirely.
Applications
The paper documents three applications of the framework:
| Application | What the framework does | ||
|---|---|---|---|
| Speed of AI progress | Measures the rate of capability-frontier advance over calendar time and forecasts future capability levels. | ||
| Algorithmic-efficiency improvements | Estimates how much capability per unit of compute is being added year-over-year, finding estimates higher than but broadly consistent with prior work. | ||
| Detecting accelerations | Identifies regime changes where AI progress speeds up beyond prior trend, relevant to the [[concepts/scaling-laws | scaling-law]] saturation debate and to the [[ai-2027 | AI 2027 forecasting]] tradition. |
Stitched values
Sample stitched capability scores reported in the paper's figures:
| Model | Stitched capability score |
|---|---|
| GPT-5 | 2.6 |
| Grok 4 | 2.5 |
| Gemini 2.5 Pro | 2.4 |
| Claude Sonnet 4.5 | 2.2 |
| DeepSeek-R1 | 2.2 |
| GPT-4.5 | 2.1 |
| Llama 3.1-405B | 1.8 |
Sample benchmark difficulties:
- Cybench: 3.1
- Terminal Bench: 2.9
- FrontierMath: 2.8
- SWE-Bench Verified: 2.3
- GPQA Diamond: 1.8
The roughly 1-point separation between Cybench (3.1) and SWE-Bench Verified (2.3) provides a quantitative answer to how much harder one benchmark is than another.
Relation to other measurement work
The framework relates to several other strands of AI-progress measurement. The Stanford HAI AI Index reports many benchmarks but does not stitch them; the method described here would let such an index produce a single frontier-capability-over-time curve (Stanford HAI AI Index Report 2026). The algorithmic-efficiency estimate is an input to scaling-law debates (Scaling Laws). Capability forecasts in AI 2027 and Broad Timelines depend implicitly on cross-benchmark stitching, which the paper formalizes into a defensible methodology. The companion Epoch AI paper Trends in AI Supercomputers — Pilz, Sanders, Rahman, Heim (Epoch AI / Georgetown / GovAI, 2025) addresses the input side (compute trends) of the same measurement problem, with this paper addressing the output side (capability stitching). The paper is positioned as a methodological anchor for planned wiki content at AI Progress Measurement. The joint Epoch AI and Google DeepMind authorship adds Rohin Shah and Samuel Albanie as DeepMind co-authors of the methodology (Epoch AI).
The method supports cross-time capability comparisons (GPT-3 versus GPT-5, 2020 versus 2026) where benchmark sets have drifted entirely, allowing comparison against a quantitative scale rather than per-benchmark scores in isolation.
Confidence note
Confidence is high for the methodology and the published stitched values. It is lower for application claims that depend on the model-class assumptions: the acceleration-detection application requires adopting the framework's regime-change definition, which is a methodological choice rather than a settled fact.
Relationships
- supports: AI Progress Measurement, Scaling Laws (algorithmic-efficiency input), AI 2027 (capability-forecasting methodology), Broad Timelines
- related: Trends in AI Supercomputers — Pilz, Sanders, Rahman, Heim (Epoch AI / Georgetown / GovAI, 2025) (companion Epoch paper, input-side), Stanford HAI AI Index Report 2026 (would benefit from this methodology), Epoch AI, Google DeepMind (co-authorship)
- instance-of: Compute Governance (methodology cousin)