This paper introduces an assessment framework for the quality of AI benchmarks and applies it to 24 of them. Its authors are Anka Reuel and Amelia Hardy (equal contribution, both corresponding) of Stanford University, Chandler Smith of Northeastern University, and Max Lamparth, Malcolm Hardy and Mykel J. Kochenderfer of Stanford. It was posted to arXiv as 2411.12990 in November 2024 and published at NeurIPS 2024. Results are maintained at betterbench.stanford.edu under a CC BY 4.0 licence.
The paper adopts a definition of a benchmark as a combination of datasets and a metric, conceptualised as representing tasks or abilities and adopted by a research community as a shared framework for comparison, and defines a high-quality benchmark as one that is interpretable, clear about its purpose and scope, and usable. It notes that benchmarks now carry policy weight, citing the UK AI Safety Institute's Inspect framework and Article 51 of the EU AI Act, and states that no structured assessment of benchmark quality had been published.
Summary
Method
The framework was built in six steps. A landscape survey identified five stakeholder groups, followed by unstructured interviews with more than 20 policymakers, model developers, benchmark developers, model users and researchers, which produced a five-stage benchmark lifecycle. Tasks and objectives at each stage were translated into criteria, sorted into those under the developer's control with normative consensus, those under the developer's control but context-dependent or hard to assess externally, and those outside the developer's control; only the first group, 46 criteria, forms the scored assessment.
Validation involved three authors independently scoring one benchmark to calibrate, revising guidelines, circulating the assessment to stakeholders, and checking that the assessment itself met its own criteria where applicable. Sixteen foundation-model and eight non-foundation-model benchmarks were then scored manually, prioritising those recently reported by model developers, using only materials the benchmark developers themselves released; at least two authors reviewed each benchmark and had to reach consensus.
Scoring is a discrete 0/5/10/15 scale per criterion: 15 for fully meeting, 10 for partially, 5 for mentioning without fulfilling, 0 for neither. Stage averages are computed for design, implementation, documentation and maintenance, plus a usability score that is the weighted average of the last three. The authors treat a mean of 10 or higher as a reasonably good result for a category.
Criteria by stage
Design criteria cover stating goals and scope, defining the capability tested, explaining how it translates into the benchmark task and why it matters in real applications, involving domain experts and literature, distinguishing the benchmark from related ones, and making measurements interpretable through metric choice and guidance on how scores should and should not be read.
Implementation criteria cover working evaluation code, accessible evaluation data, prompts or test environment, a script to replicate the published results, support for both API-based and local models, and contamination and gaming defences such as a globally unique identifier, encrypted evaluation instances, and a task for detecting whether a model trained on the test set.
Documentation criteria cover the construction process, data collection or environment design, task rationale, metric documentation, reported statistical significance, stated normative assumptions and limitations, plus codebase requirements, a quick-start guide, file-structure description and in-line comments; peer review is treated as a signal of external scrutiny.
Results
Average scores by stage across the 24 benchmarks were design 10.8, implementation 6.2, documentation 10.1 and maintenance 9.6; foundation-model benchmarks scored 10.6 / 5.5 / 10.3 / 9.1 and non-foundation-model benchmarks 11.2 / 7.5 / 9.9 / 10.6. Overall average design was 10.7 and weighted usability 8.7, with the difference between the two benchmark types not statistically significant at 95% confidence. Design and usability correlated at Pearson 0.730 (p = 0.001) for foundation-model benchmarks and 0.693 (p < 0.001) across all, which the authors read as poorly designed benchmarks also tending to be less usable.
Two criteria scored especially low: including a script to replicate results averaged 3.75 and reporting statistical significance averaged 5.62. Seventeen of 24 benchmarks provided no easy-to-run replication script and four more covered only part of their results; 14 of 24 neither ran repeated evaluations of the same model nor reported statistical significance or uncertainty.
The discussion opens with the observation that model developers report performance on benchmarks of markedly different quality without acknowledging the difference: MMLU scored lowest of those assessed at a weighted average of 5.5, while GPQA scored 11.0, yet GPT-4, Claude 3 and Gemini announcements reported both. The authors recommend that developers re-run evaluations with different seeds or temperatures and report mean and variance, since users need intra-model variance to judge whether differences between models are real. They note that several of their criteria, such as code documentation and a point of contact, are cheap to add.
Open challenges and limitations
Four issues are listed as beyond a benchmark developer's unilateral control: rapid saturation, sometimes within months of release; contamination, since developers cannot make model builders honour canary strings, making it a shared responsibility; poor construct validity, including properties that arise from the interaction between model and user population rather than the model alone; and the absence of standardised benchmark reporting, so that developers report whichever benchmarks they choose without rationale.
The authors' stated limitations are that all criteria carry equal weight despite differing effort and contribution, that four score categories miss nuance within a category, and that the scheme could be gamed by targeting easy criteria — though they argue a benchmark meeting even the easy criteria is better than one meeting none. The impact statement notes the risk of misinterpretation: the assessment provides minimum quality assurance and is not sufficient to judge a benchmark's suitability for a particular use.
Key claims
- 46 criteria from expert interviews and literature, scored 0/5/10/15 across a five-stage lifecycle; 24 benchmarks assessed (16 foundation-model, 8 non-foundation-model).
- Stage averages: design 10.8, implementation 6.2, documentation 10.1, maintenance 9.6; overall usability 8.7.
- Replication script averaged 3.75 and statistical significance 5.62 across benchmarks.
- 17 of 24 lacked an easy replication script; 14 of 24 reported no repeat runs or uncertainty.
- MMLU scored lowest (5.5 weighted average), GPQA 11.0, both reported side by side in model announcements.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering (arxiv.org); abstract page arxiv.org. Living repository at betterbench.stanford.edu.
Relationships
- supports: AI Benchmarks and Evaluation — the benchmark-quality assessment the concept page cites.
- related: Measuring what Matters: Construct Validity in Large Language Model Benchmarks (Bean et al., 2025) — a later and larger review focused on construct validity.
- related: MMLU, GPQA — the two benchmarks the paper contrasts by quality score.
- related: EU AI Act (Regulation 2024/1689) — cited as a policy use of benchmarks (Article 51).