This is the paper introducing the MMLU-Pro benchmark, written by Yubo Wang, Xueguang Ma and Wenhu Chen of the University of Waterloo (the corresponding authors) with 14 co-authors from Waterloo, the University of Toronto and Carnegie Mellon University, and posted to arXiv as 2406.01574 in June 2024. It presents MMLU-Pro as a harder, more reasoning-oriented and more prompt-stable replacement for MMLU, which the authors argue had saturated.
Summary
Motivation
The authors state that MMLU had become "the de facto standard" for evaluating language models but that progress on it had stalled: since GPT-4 reached 86.4% in March 2023, frontier models released in early-to-mid 2024 (GPT-4-Turbo, Gemini 1.5 Pro, Claude, Llama-3-400B) all settled between 86% and 87%, and GPT-4o improved by only about 1 point to 87.4% while gaining more than 10 points on MATH and Chatbot Arena. They also cite prior work showing MMLU scores are highly sensitive to prompt and scoring choices. The paper attributes these problems to three causes: only three distractors per question, allowing shortcuts; questions that are "mostly knowledge-driven" so that most models score better without chain-of-thought; and a portion of questions that are unanswerable or mislabelled, which lowers the ceiling.
Dataset construction
MMLU-Pro contains 12,032 questions in 14 disciplines (the 57 MMLU subjects merged into 14 broader categories). Construction proceeded in four stages. First, MMLU questions answered correctly by more than four of eight small models (Llama-2-7B and 13B in base and chat form, Mistral-7B, Gemma-7B, Yi-6B and Yi-6B-Chat) were classed as too easy and removed; 5,886 questions were filtered out this way. Second, questions were added from three sources: a website of worked STEM problems (stemez.com), TheoremQA (human-annotated questions requiring a theorem) and SciBench (college-examination science questions), with GPT-4-Turbo used to extract short answers from worked solutions and to generate three distractors, followed by manual checking. Third, GPT-4-Turbo generated six further distractors to bring each question from four options to ten; the authors report that GPT-4-Turbo gained no advantage from having written the distractors. Fourth, a two-phase expert review checked answer correctness and removed questions needing images, tables or a non-multiple-choice format, then used Gemini 1.5 Pro to flag options that might be correct but labelled as distractors ("false negatives") for human re-examination. The paper's Table 1 records the issues found: among MMLU-derived items, 350 incorrect answers, 1,953 false-negative options and 385 bad questions; among STEM Website items, 483, 293 and 862; among SciBench items, 11, 15 and 15; among TheoremQA items, 0, 5 and 1. After removing false-negative options, 83% of questions have ten options, 17% have fewer, and the average is 9.47.
Evaluation protocol
The standard setting is 5-shot chain-of-thought prompting, with five demonstration examples selected per discipline (Gemini 1.5 Pro and Flash were run 0-shot). Answers are extracted with a regular expression matching "answer is (X)", with a second pattern as fallback; if neither matches, a random option is recorded.
Results
The paper evaluated more than 50 models. In its Table 2, GPT-4o led with 72.6% overall, followed by Gemini 1.5 Pro at 69.0%, Claude 3 Opus at 68.5%, GPT-4-Turbo at 63.7%, Gemini 1.5 Flash at 59.1%, Yi-large at 58.1% and Claude 3 Sonnet at 56.8%. Among open-weight models, Llama-3-70B-Instruct scored 56.2%, Phi-3-medium-4k-instruct 55.7%, DeepSeek-V2-Chat 54.8%, Llama-3-70B 52.8%, Qwen1.5-72B-Chat 52.6% and Yi-1.5-34B-Chat 52.3%; the smallest model listed, Gemma-2B, scored 15.9%. GPT-4o's subject scores ranged from 79.2% (psychology) and 76.1% (mathematics) to 55.0% (engineering) and 51.0% (law). The authors note that Engineering and Law scored lowest across models, attributing the Engineering result to new STEM Website questions requiring multi-step calculation.
Three comparisons with MMLU carry the paper's argument. On difficulty, accuracy falls by 16 to 33 points from MMLU to MMLU-Pro; four models that score between 78% and 82% on MMLU spread over roughly 10 points on MMLU-Pro, and the gap among GPT-4o, Claude 3 Opus and GPT-4-Turbo widens from about 2 points to about 9. The authors also note that the best score leaves 27.4 points of headroom against about 11.3 on MMLU. On reasoning, chain-of-thought helps far more on MMLU-Pro than on MMLU: GPT-4o gained 19.1 points (72.6% against 53.5% direct) on MMLU-Pro but 1.5 points on MMLU (88.7% against 87.2%); GPT-4-Turbo gained 15.3 points on MMLU-Pro and lost 0.2 on MMLU; Llama-3-8B and Gemma-7B lost 3.9 and 3.6 points from chain-of-thought on MMLU but gained 3.9 and 6.7 on MMLU-Pro. On robustness, across 24 prompt styles the score range on MMLU was generally 4–5% with a peak of 10.98%, and on MMLU-Pro around 2% with a maximum of 3.74%.
An error analysis of 120 randomly selected GPT-4o mistakes, judged by expert annotators, attributed 39% to reasoning errors, 35% to lack of specific domain knowledge, 12% to calculation errors, 5% to no selection made, 4% to question-understanding errors, 2% to generation issues, 2% to annotation errors and 1% to answer-extraction errors.
Limitations stated by the authors
The paper names two: the multiple-choice format "may not capture the depth of comprehension and creative response generation as effectively as open-ended answers," and the benchmark evaluates text-only language models and not multimodal systems.
Key claims
- 12,032 questions in 14 disciplines; up to ten options per question (average 9.47); 5,886 easy MMLU questions removed; additional items from stemez.com, TheoremQA and SciBench.
- Expert review found 350 incorrect answers and 385 bad questions among the MMLU-derived items examined.
- GPT-4o 72.6%, Gemini 1.5 Pro 69.0%, Claude 3 Opus 68.5%, GPT-4-Turbo 63.7%, Llama-3-70B-Instruct 56.2% (5-shot chain-of-thought, 0-shot for Gemini).
- Accuracy drops 16–33 points relative to MMLU; the leading closed models spread from about 2 points apart on MMLU to about 9 on MMLU-Pro.
- Chain-of-thought adds 19.1 points for GPT-4o on MMLU-Pro against 1.5 on MMLU; prompt-induced variation is about 2% (maximum 3.74%) against 4–5% (maximum 10.98%) on MMLU.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of 2406.01574 (arxiv.org); the abstract page is arxiv.org. The leaderboard and dataset are hosted at huggingface.co. The arXiv text does not state a publication venue.
Relationships
- supports: MMLU-Pro — the primary text for the benchmark's design and release-time results.
- supports: MMLU — the source of the saturation, prompt-sensitivity and chain-of-thought comparisons cited there.
- depends-on: MMLU — the benchmark is built from filtered MMLU items.
- instance-of: AI Benchmarks and Evaluation