"Are We Done with MMLU?" is a paper by Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken and Pasquale Minervini, with affiliations at the University of Edinburgh, Sapienza University of Rome, Polytechnic University of Bari, University College London, University of Trento, AssemblyAI, the UK Health Security Agency and Miniml.AI. It was posted to arXiv as 2406.04127 in June 2024 and revised through version 3. It documents ground-truth and question errors in MMLU and releases a re-annotated subset, MMLU-Redux. The abstract answers the title's question: "Maybe not."
Summary
The authors state that MMLU, "despite its popularity," contains errors "that can mislead evaluation and model comparison," ranging from parsing and scraping mistakes to missing context, and that its under-documented annotation procedure makes their causes hard to trace. They develop a hierarchical annotation protocol and apply it to 100 randomly selected questions from each of the 57 MMLU subjects, 5,700 questions in all, annotated by 14 human experts, confirming errors against the original source where it could be found. Annotators' suggested corrections are recorded but "not used to replace the existing MMLU labels."
Error protocol
Errors fall into two groups. Type 1, question assessment, covers (1a) bad question clarity (poorly presented, insufficient information, references to a previous question) and (1b) bad options clarity (unclear, similar or irrelevant options, mostly from mis-parsed sources, such as one option split into two). Type 2, ground-truth verification, covers (2a) no correct answer (for instance when the correct option was dropped in reducing five options to four), (2b) multiple correct answers (such as a synonym of the label among the options), and (2c) wrong ground truth. Inter-annotator agreement among three annotators on five error-prone subjects gave Cohen's kappa from 0.64 (Human Sexuality) to 0.96 (Formal Logic) on exact error type.
Findings
Using stratified sampling, the authors "estimate that 6.49% of the questions in the complete MMLU dataset contain errors"; within the sampled subsets from the most erroneous subjects, "more than 9% of the examples are incorrect." Virology has the highest rate, with 57% of analysed instances erroneous, including 33% with a wrong ground truth (given as 30% elsewhere in the text) and 14–15% unclear questions; one item suggests sending the American army to West Africa to prevent Ebola outbreaks. Logical Fallacies (26%) and College Chemistry exceed 20%, and Professional Law, Business Ethics, Formal Logic, Human Aging, Global Facts, Machine Learning, Miscellaneous and Public Relations exceed 10%. Subject-level causes include parsing errors from the Human Virology textbook, multi-line chemistry questions split into options, unstated US jurisdiction in Professional Law, incorrect Formal Logic answers sourced from Oxford Learning Link, and Global Facts items whose sources conflict.
Re-evaluating leading models on the five most erroneous subjects with and without the erroneous items changes rankings. In Virology, Llama 3.1 Instruct Turbo 405B ranks 16th on all instances and first on correct instances only; in Human Sexuality, GPT-4 (0613) scores 0.91 on all instances (5th) but 0.43 on correct instances only (last of the top ten), while Gemini 1.5 Pro (001) moves from 0.37 to 0.94. Across the ten best models on HELM's ranking, performance correlates with the share of non-erroneous instances per subject. The authors also observe that on some erroneous instances (Professional Law, Formal Logic) models score as well as or better than on correct ones, which, since a correct answer against a wrong key should be impossible by reasoning, "may be evidence of memorisation."
Automatic error detection
Tested as a benchmark for automatic error detection, MMLU-Redux proved difficult. With zero-shot, few-shot and chain-of-thought prompting, the best F2 was 40.29 (Claude 3 Opus, few-shot chain-of-thought); with retrieval-augmented prompting from Wikipedia, Claude 3 Opus reached an F2 of 41.92 and recall of 83.91 on the MS MARCO index. Fine-tuning Llama 3 8B-Instruct was also tried. The authors conclude that even the best combination "is still insufficient to produce a high-quality dataset."
Limitations and recommendations
The remaining 8,342 MMLU questions are unannotated, and the protocol "might still be prone to the personal biases of the annotators." The authors "strongly advocate for revising MMLU's error-ridden questions" and open the dataset to further annotation, with a maintenance plan and contribution guidelines in the appendices. The dataset is released as mmlu-redux-2.0 on Hugging Face.
Key claims
- MMLU-Redux: 5,700 questions, 100 per subject across 57 subjects, 14 expert annotators, hierarchical error taxonomy.
- Estimated overall MMLU error rate 6.49%; Virology 57%, Logical Fallacies 26%, College Chemistry above 20%.
- Errors change model rankings on the most affected subjects.
- LLM-based error detection peaks at F2 41.92 (Claude 3 Opus with retrieval).
- The authors' position: MMLU's erroneous questions should be revised for it to remain a reliable benchmark.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 3 (arxiv.org); the abstract page is arxiv.org. Dataset at huggingface.co.
Relationships
- supports: MMLU — source of the error-rate estimate and ranking-change findings.
- related: MMLU-Pro — the contemporaneous harder replacement discussed in the paper's related work.
- related: FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI (Glazer et al., Epoch AI, 2024) — cites this paper's MMLU error rate as a comparison point for its own.
- related: AI Benchmarks and Evaluation