This is a position paper by Inioluwa Deborah Raji (Mozilla Foundation and UC Berkeley), Emily M. Bender and Amandalynne Paullada (Department of Linguistics, University of Washington), and Emily Denton and Alex Hanna (Google Research). It was posted to arXiv as 2111.15366 on 26 November 2021 and presented at the NeurIPS 2021 Datasets and Benchmarks track.
Its title refers to the 1974 Sesame Street storybook in which Grover visits a museum claiming to hold "everything in the whole wide world," finds it composed of arbitrary rooms, and opens a door marked "Everything Else" onto the outside. The authors argue that benchmarks presented as measures of progress toward general ability in vague tasks such as "visual understanding" or "language understanding" fail for the same reason the museum does: they are inherently specific, finite and contextual.
Summary
The claim
The paper's target is a construct-validity problem: benchmarks like GLUE and ImageNet are "elevated to become definitions of the essential common tasks to validate the performance of any given model," and the claims justified through them extend past the tasks they were designed for and past their creators' ambitions. The authors note that the Common Task Framework was developed for tightly scoped practical tasks such as speech recognition and machine translation, where validation asks whether the benchmark reflects a real-world task, and argue that current practice extends it to performance in the abstract.
Two case studies anchor the argument. ImageNet is treated as a general benchmark because its creators described it as "the most comprehensive and diverse coverage of the image world" and later as an attempt to map the entire world of objects, and because gains on it are cited as evidence of progress toward general-purpose AI. GLUE and SuperGLUE are presented by their creators as evaluation frameworks for research toward general-purpose language-understanding technologies, on the reasoning that a system with linguistic knowledge should apply it across tasks.
Limited task design
The authors argue that if these were legitimate tests of general cognitive ability, their component tasks would be chosen systematically or with reference to theories of the abilities modelled; instead they look like convenience samples. They cite Wagstaff's observation that heavy work on the UCI iris and mushroom classifiers had no visible effect on botany or mycology. ImageNet's classes range from specific dog breeds to "New Zealand beach" and derive from twelve WordNet subtrees adopted nearly whole cloth from a project with a different purpose. GLUE's tasks were curated from roughly 30 proposals gathered in an informal survey of NLP colleagues and filtered on practical heuristics such as licensing and format.
A second objection concerns domain knowledge: GLUE and SuperGLUE combine linguistic competence with commonsense and world reasoning as if they were equivalently scoped problems, and the authors note that language understanding also depends on world knowledge, commonsense reasoning and modelling the interlocutor's state of mind, none of which text-only tasks can thoroughly test.
De-contextualised data and reporting
The authors argue that no dataset is neutral and that the generality claim lets benchmark developers avoid reporting limitations. On scope: ImageNet's objects tend to be centred in the frame, unlike natural images, and its creators themselves note a trade-off between annotation quality and scale. GLUE's stated diversity amounts to two single-sentence tasks, two similarity and paraphrase tasks and four inference tasks, and SuperGLUE's own authors note GLUE's formats are limited to sentence and sentence-pair classification. On subjectivity: GLUE and SuperGLUE target English specifically rather than language in the abstract, and geo-location analysis of the 2011 ImageNet found 45% of images sourced from the United States and over 60% from a small set of Western countries, with 1% from China and 1.2% from India.
Inappropriate community use
Benchmarks shape what a field works on. The paper cites NIST investing over $6.5 million in the FERET facial-recognition challenge between 1993 and 1998 to draw research toward a technology of interest to its Department of Defense sponsor, and the 1960s chess focus that concentrated work on tree search and minimax at the expense of other approaches. It also notes benchmark scores appearing in commercial marketing as evidence of real-world achievement, quoting Microsoft's January 2021 statement that "DeBERTa surpassing human performance on SuperGLUE marks an important milestone toward general AI," and warns that aggregate scores can hide poor performance on minority subgroups.
What the authors propose instead
The authors state that the answer is not to fix or expand general benchmarks: "the solution to addressing the issues inherent to the 'Everything in the Whole Wide World' museum is not to add more rooms." They propose two paths. Benchmarks should be developed, presented and understood as evaluating concrete, well-scoped, contextualised tasks. For broader objectives and behaviours, they list four families of alternative methods: systematic development of test items (test suites, audits, adversarial testing); system-output and behavioural analysis (error analysis, disaggregated analysis, counterfactual analysis); ablation testing; and analysis of model properties orthogonal to outputs, such as energy consumption, memory requirements and stability under training-data perturbation.
The conclusion states that "open-world, universal and neutral datasets don't exist, and current methods of benchmarking do not offer meaningful measures of general capabilities," and that what makes a benchmark effective is not a claim to generality but how well it helps researchers understand how systems work and how they fail — "not about winning a contest but more about surveying a landscape."
Key claims
- Benchmarks framed as general measures of progress have a construct-validity problem: they cannot capture what they are taken to capture.
- The component tasks of ImageNet and GLUE were selected by convenience rather than from a theory of the abilities being modelled.
- ImageNet's 2011 geo-location analysis: 45% US-sourced, over 60% from North America and Europe, 1% China, 1.2% India.
- GLUE's task diversity amounts to eight tasks in three formats, all in English.
- The remedy proposed is scoping benchmarks to concrete tasks plus four families of non-benchmark evaluation methods.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering (arxiv.org); abstract page arxiv.org.
Relationships
- supports: AI Benchmarks and Evaluation — the construct-validity critique the concept page attributes to these authors.
- related: Dynabench: Rethinking Benchmarking in NLP (Kiela et al., 2021) — a contemporaneous response to benchmark saturation that the authors' critique of dataset-defined tasks bears on.
- related: MMLU, HELM (Holistic Evaluation of Language Models)