AI progress measurement is the set of methodologies and benchmarks used to track frontier-AI capability over time. It is distinct from safety frameworks, which use such measurements as inputs to release decisions, and from alignment auditing, which measures how a model reaches its outputs rather than the level of capability itself. As of the May 2026 capability landscape, different labs cite different benchmarks for different framing purposes, which makes cross-lab comparison difficult.
Benchmarks and measurement methods
Several distinct benchmarks and measurement methods are in active use as of the May 2026 capability landscape, each capturing a different dimension of capability:
| Benchmark | Measures |
|---|---|
| METR HCAST time-horizon | Longest task a model can complete autonomously |
| GPQA | Graduate-level Q&A; reasoning on hard scientific questions |
| ARC-AGI-2 | François Chollet's general-intelligence benchmark |
| AIME 2025 | Math-olympiad-level problems |
| HumanEval / BigCodeBench | Coding tasks |
| GDPval | Frontier-model economic-value evaluation |
| MMLU-Pro | Multi-domain knowledge and reasoning |
| Anthropic Economic Index | Real-world task-substitution rate |
| METR Task Substitution and Uplift | Methodological counterweight to the Anthropic Economic Index |
The METR HCAST time-horizon measure is referenced in connection with Claude Opus 4.5, Claude Opus 4.7, and AGI Timelines. ARC-AGI-2 is likewise associated with AGI Timelines. On GPQA, GPT-5.5 Instant scored 85.6%; on AIME 2025, the same model scored 81.2%. HumanEval and BigCodeBench coding tasks are associated with Clawed (Claude Code), and GDPval with Claude Opus 4.7. MMLU-Pro is used as a standard cross-lab measure. The Anthropic Economic Index connects to AI and Productivity. METR's Task Substitution and Uplift work, presented as a methodological counterweight to the Anthropic Economic Index, was queued for ingest as of May 2026.
Methodological tensions
Four recurring tensions complicate capability measurement:
- Saturation. Most benchmarks reach roughly 90% or higher accuracy within two generations of a frontier release, so new benchmarks are required almost continuously.
- Contamination. Pre-training data overlap with benchmark sets is a recurring problem, and cross-lab reproducibility suffers as a result.
- Selection by labs. Each lab cites benchmarks favorable to its own model, so cross-lab comparison requires careful base-rate adjustment.
- Reasoning versus capability gap. Benchmarks measure outputs, whereas alignment auditing measures how those outputs are produced. The NLA paper (May 6, 2026) shows that the two can diverge.
Relationships
- related: AI Safety Cases and Frameworks, AGI Timelines, Alignment Auditing.
- related: METR, Epoch AI.
- related: Scaling Laws, AI Software Progress.
Sources
Stub created 2026-05-11 during the v4.0 backlog-close pass. Foundational ingest candidates: METR HCAST methodology paper; Anthropic Economic Index quarterly publication; Epoch AI capability-trajectory analyses.