This paper reports a systematic review of construct validity in language-model benchmarks. Its first author is Andrew M. Bean of the University of Oxford, with Ryan Othniel Kearns, Franziska Sofia Hafner and Harry Mayne (Oxford), Angelika Romanou and Negar Foroutan (EPFL), Jan Batzner (Weizenbaum Institute Berlin and TU Munich) and further co-authors, with Adam Mahdi as co-corresponding author at the Oxford Internet Institute. It was posted to arXiv as 2511.04703 in November 2025 and appeared at NeurIPS 2025.
The paper's frame is that measuring abstract phenomena such as safety and robustness requires construct validity — "having measures that represent what matters to the phenomenon" — and that the reviewed literature shows recurring patterns that undermine the validity of the claims benchmarks support.
Summary
Method
The corpus was 46,114 articles from ICML, ICLR and NeurIPS proceedings between 2018 and 2024 and from ACL, NAACL and EMNLP between 2020 and 2024, the latter range limited by abstract availability. Titles or abstracts containing "benchmark" plus either "LLM" or "language model" gave 2,189 articles, with only 14 from 2018 and 2019.
Four inclusion criteria applied: the article must concern LLM capabilities rather than only technical properties such as inference speed; it must introduce an empirical benchmark and report performance, excluding opinions, reviews and policy frameworks; the benchmark must work with text and vision models; and it must be a novel benchmark or a substantial modification rather than a repackaging. GPT-4o mini screened against the first three criteria, validated against human labels on 50 articles at an F1 of 84% and used only to exclude, reducing the set to 522. Twenty-nine reviewers matched on expertise applied all four criteria manually, giving 445 articles.
The codebook treats a benchmark as a task and a metric used together to represent a phenomenon of interest, assessed alongside how the authors interpret the results, and produced 21 coded items per article.
Findings
On phenomena: the most common were reasoning (18.5%), alignment (8.1%) and code generation (5.7%). Most articles defined their phenomenon (78.2%), but of those definitions 52.2% were widely agreed and 47.8% contested. Definitions treated the phenomenon as a composite in 61.2% of cases and as a single unified whole in 36.5%.
On tasks: fewer than 10% used complete real-world tasks. Constructed tasks appeared in 40.7% of benchmarks (28.5% exclusively), partially real-world tasks in 32.3% and representative tasks such as exam-style questions in 36.9%. Only 33.6% relied on a single source of task items; authors handcrafted new items in 43.3% of cases, reused existing benchmark data in 42.6%, generated data with models in 31.2%, and used human exams or other pre-existing sources in 38.2%, with 7.7% sourcing tasks solely from another benchmark. On sampling, 12.3% used convenience sampling exclusively and 27.0% partially, 55.2% used at least partially targeted sampling, and 17.1% used random or stratified sampling. Response formats were free response (46.4% partially, 17.8% exclusively), multiple choice (40.0%, 18.5%), short free response (38.5%, 13.2%) and structured formats such as JSON (21.1%, 8.6%).
On metrics: exact matching dominated, used at least partially by 81.3% and exclusively by 40.7%, followed by soft-match scoring (20.9% / 0.9%), model-as-judge (17.1% / 3.1%) and human ratings (13.0% / 1.8%). Only 16.0% used uncertainty estimates or statistical tests when comparing results.
On claims: 53.4% presented evidence for their benchmark's construct validity, 35.2% compared to other benchmarks of similar phenomena, 32.4% to a human baseline and 31.2% to a more realistic setting.
Recommendations
Eight recommendations, each with a checklist: define the phenomenon, including its scope and sub-components, with an apophatic definition where no consensus exists; measure the phenomenon and only the phenomenon, controlling for confounds such as output-format difficulty and validating any automated parsing; construct a representative dataset through sampling strategies rather than convenience, noting that a smaller well-sampled dataset can have higher validity at lower cost; acknowledge the limitations of reusing datasets and report performance against the original; prepare for contamination with detection tests, a held-out set and checks on source-material exposure; use statistical methods, reporting sample size, power and uncertainty, and describing rater demographics where human ratings are used; conduct an error analysis to see whether failures track the intended construct; and justify construct validity explicitly, including design trade-offs such as multiple choice being easy to score but gameable against free text being realistic but costly.
The paper works the checklist through GSM8K as a demonstration, concluding that it is "a generally valid benchmark for measuring performance on grade school maths questions," that contamination has likely raised scores over time, that no error analysis was conducted, and that interpretations stretching from maths problems to logical reasoning would benefit from greater clarity.
Limitations stated by the authors
Restricting the corpus to leading conference proceedings excludes benchmarks released by industry labs without peer review and those in domain-specific venues. The model-assisted screening step, though used only to exclude and validated against human annotation, may have introduced undetected false negatives, and shifts in language use may have depressed inclusion of older papers. The scale of the review limited the number of reviewers per paper, reducing robustness.
Key claims
- 445 benchmark articles reviewed by 29 experts, drawn from 46,114 conference papers through a four-criterion filter.
- 78.2% define their phenomenon; 47.8% of those definitions are contested.
- Fewer than 10% of benchmarks use complete real-world tasks; 39.3% involve convenience sampling in whole or part.
- Exact matching used at least partially by 81.3%; statistical testing by 16.0%; construct-validity evidence by 53.4%.
- Eight recommendations with an operational checklist, recommended for reporting as an appendix with explanations for skipped items.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering (arxiv.org); abstract page arxiv.org.
Relationships
- supports: AI Benchmarks and Evaluation — the systematic evidence behind the construct-validity section.
- related: BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices (Reuel et al., 2024) — an earlier assessment of benchmarks against best practices.
- related: AI and the Everything in the Whole Wide World Benchmark (Raji et al., 2021) — the earlier construct-validity argument this review tests empirically.
- related: GSM8K — the benchmark used as the paper's worked example.