MMLU-Pro

MMLU-Pro is a multiple-choice benchmark of 12,032 questions in 14 subject areas, each with up to ten answer options. It was released in June 2024 by TIGER-Lab, a research group at the University of Waterloo, and published at NeurIPS 2024 in the Datasets and Benchmarks track MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (Wang et al., 2024). Its authors built it because scores on MMLU had begun "to plateau, making it increasingly difficult to discern differences in model capabilities." GPT-4o, the strongest model at release, scored 72.6%; by 2026 the leading models on the maintainers' leaderboard were at 87–91% (Source: MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (Wang et al., 2024); huggingface.co).

What it measures

The authors describe MMLU-Pro as extending "the mostly knowledge-driven MMLU benchmark by integrating more challenging, reasoning-focused questions." Their evidence that it tests reasoning more than recall is the effect of chain-of-thought prompting, in which a model writes out intermediate steps before answering: GPT-4o gained 19.1 points from chain-of-thought on MMLU-Pro and 1.5 points on MMLU MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (Wang et al., 2024). Hugging Face adopted it for the second version of its Open LLM Leaderboard in 2024 as the replacement for MMLU, which it called "the reference multichoice knowledge dataset" (Source: huggingface.co).

Design

Dataset construction

The dataset has four sources. Just over half of the questions (6,810) come from MMLU; the rest (5,222) come from a website of worked STEM problems (stemez.com), from TheoremQA, a set of human-annotated questions that require applying a theorem, and from SciBench, a set of science questions from college examinations (Source: MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (Wang et al., 2024); huggingface.co). Construction ran in four steps MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (Wang et al., 2024):

1. Filtering MMLU. The 57 MMLU subjects were merged into 14 categories. Eight small open models (versions of Llama 2, Mistral, Gemma and Yi) answered every MMLU question, and the 5,886 questions that more than four of them got right were discarded as too easy.
2. Adding new questions. The STEM-website and TheoremQA items were problem statements with worked solutions rather than multiple-choice questions. GPT-4-Turbo extracted a short answer from each solution and wrote three wrong options, and the authors manually removed items where the extracted answer was incomplete or wrong.
3. Option augmentation. GPT-4-Turbo wrote six more wrong options for every question, taking the total from four to ten. The authors report that GPT-4-Turbo gained no advantage on the benchmark from having written them.
4. Expert review. Human experts checked each answer and removed questions unsuited to multiple choice or dependent on images or tables. Gemini-1.5-Pro then re-examined all options to flag wrong options that were in fact correct, and experts reviewed the flagged cases. The review found 350 incorrect answers and 385 bad questions among the MMLU-derived items and 483 and 862 among the STEM-website items, and removed 1,953 and 293 mistakenly "wrong" options respectively. After removals, 83% of questions have ten options and the average is 9.47.

The 14 categories are biology, business, chemistry, computer science, economics, engineering, health, history, law, mathematics, philosophy, physics, psychology and other (Source: huggingface.co).

Task format and an example

An item is a question followed by options lettered A to J. Many of the added questions require calculation. One physics-style item in the mathematics category asks for the work done by a muscle fibre that contracts by 3.5 cm while obeying Hooke's law with a force constant of 750 N/m, with ten numerical options between 0.25 J and 1.00 J (Source: kaitchup.substack.com).

Scoring

The score is the percentage of questions answered correctly, micro-averaged over the whole set. The authors' standard setting is 5-shot chain-of-thought: five worked examples per category in the prompt, with the answer extracted from the model's text by a regular expression that looks for "answer is (X)". If no answer can be extracted, a random option is recorded. With up to ten options, random guessing yields roughly 10%. Developers also report zero-shot results, and the maintainers note that "normally we use 5-shot, some models like Gemini use 0-shot" (Source: MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (Wang et al., 2024); huggingface.co).

Human baseline, splits and contamination

The paper reports no human baseline. The full test set with answers is public on Hugging Face under an MIT licence, and there is no held-out portion; the MMLU-derived half inherits whatever exposure MMLU already had (Source: huggingface.co).

Versions and variants

The dataset has been corrected in place rather than versioned. The maintainers' changelog records a consolidated note on problems in the Health category in October 2025 and a fix in January 2026 for a formatting flaw described under Limitations (Source: huggingface.co). Derivatives by other groups include MMLU-Pro+ (Taghanaki and colleagues, September 2024), which adds questions with more than one correct answer to test whether models take shortcuts (Source: arxiv.org), and MMLU-ProX (March 2025), which translates 11,829 of the questions into 29 languages (Source: arxiv.org). OpenAI's CoT-Control suite draws some of its tasks from MMLU-Pro (Reasoning models struggle to control their chains of thought, and that's good (OpenAI, March 2026)).

Development and maintainers

The paper lists 17 authors led by Yubo Wang, Xueguang Ma and Wenhu Chen of the University of Waterloo MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (Wang et al., 2024). TIGER-Lab hosts the dataset, evaluation scripts and a leaderboard. Each leaderboard entry is marked either "TIGER-Lab", meaning the maintainers ran the model, or "Self-Reported", meaning the figure was taken from the developer; most frontier-model entries are self-reported and give only an overall score. The maintainers state that submissions "must provide substantial evidence demonstrating that their system is a genuine language model." The most recent batch of additions was in March 2026 (Source: huggingface.co; huggingface.co).

Independent measurement has narrowed. Artificial Analysis ran MMLU-Pro itself as part of its Intelligence Index, with Gemini 3 Pro Preview highest at 89.8%, but removed it from the index in version 4.0 and lists it as "retired from our active reporting" (Source: artificialanalysis.ai; artificialanalysis.ai).

Results

At release in June 2024 GPT-4o led at 72.6%, followed by Gemini 1.5 Pro at 69.0% and Claude 3 Opus at 68.5%; the best open-weight model, Llama-3-70B-Instruct, scored 56.2%. The authors noted that the three leading closed models, about 2 points apart on MMLU, were about 9 points apart on MMLU-Pro MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (Wang et al., 2024). Reasoning models closed much of the remaining room: OpenAI's o1 is listed at 89.3% and DeepSeek-R1 was reported at 84.0% in January 2025 (Source: huggingface.co; DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning). On the leaderboard as of September 2026, Gemini 3.1 Pro leads at 91.16% in the maintainers' own run, and most other frontier entries lie between 85% and 90% (Source: huggingface.co).

Overall accuracy

Accuracy on the full set for instruction-tuned and reasoning models. Rows sourced to the TIGER-Lab leaderboard carry the leaderboard's own label ("self-reported" or "run by TIGER-Lab") and are dated by the model's release, since the leaderboard does not date entries.

Base models (5-shot)

Accuracy of pretrained models before instruction tuning, which developers report to compare pretraining quality.

MMLU Pro — 0-shot CoT

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

MMLU-Pro — 5-shot, CoT

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

MMLU-Pro — Pre-trained

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

MMLU Pro — Instruction tuned

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

Interpreting scores

Chance performance is about 10%, against 25% on MMLU, so a given score reflects less guessing (Source: huggingface.co). The authors found scores less sensitive to prompt wording than MMLU's: across 24 prompt styles the spread was generally around 2%, with a maximum of 3.74%, against 4–5% and a maximum of 10.98% on MMLU MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (Wang et al., 2024). Scores are highly sensitive, however, to whether the model reasons before answering: GPT-4o scored 72.6% with chain-of-thought and 53.5% answering directly, and DeepSeek reports 87.5% for V4-Pro at its maximum thinking setting against 82.9% with thinking off (Source: MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (Wang et al., 2024); huggingface.co). Base-model scores in the 60s and 70s and reasoning-model scores near 90 are therefore measurements under different conditions and are kept in separate tracks above.

Performance differs by subject. In the paper's results, GPT-4o scored 79.2% on psychology and 76.1% on mathematics but 55.0% on engineering and 51.0% on law, and the authors' analysis of GPT-4o's errors attributed 39% to reasoning errors, 35% to missing domain knowledge and 12% to calculation errors MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (Wang et al., 2024).

Because most leaderboard entries for frontier models are self-reported overall scores, differences of a point or two between them reflect each developer's prompt, sampling and reasoning settings as well as the model (Source: huggingface.co). Third-party leaderboards have also treated developer figures with caution: independent reproductions of Meta's reported Llama 4 numbers on MMLU-Pro and other benchmarks came in below the company's figures (Llama 4 (Meta)).

Limitations and criticism

The authors name two limitations: the multiple-choice format "may not capture the depth of comprehension and creative response generation as effectively as open-ended answers," and the benchmark is text-only MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (Wang et al., 2024).

Machine-written options and residual errors

Six of every ten options were written by GPT-4-Turbo and screened by Gemini-1.5-Pro before human review, and the review statistics above show how many faulty items the pipeline produced before filtering. Errors have continued to surface after release, including the Health-category issues the maintainers documented in October 2025 (Source: MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (Wang et al., 2024); huggingface.co).

Answer leakage through formatting

In January 2026 users found that in some chemistry, physics and other STEM items the correct option began with a space character that the wrong options lacked. The newsletter The Kaitchup, following a post by Eric W. Tramel, reproduced the flaw and argued that "if it's visible to humans, it's visible to LLMs during evaluation, too." The maintainers fixed the dataset on January 18, 2026, acknowledging that the inconsistency "could have been exploited as a shortcut" (Source: kaitchup.substack.com; huggingface.co). Scores computed before and after the fix are not strictly comparable.

Narrowing headroom

The MMLU-Pro+ authors wrote within three months of release that existing benchmarks "increasingly struggle to differentiate between top-performing models" (Source: arxiv.org). With leading models at 87–91% on the maintainers' leaderboard, Artificial Analysis dropped the benchmark from its composite index in version 4.0 (Source: huggingface.co; artificialanalysis.ai). Several 2026 frontier launch tables, including OpenAI's for GPT-5.5 and GPT-5.6, do not report it (Source: openai.com; openai.com).

Contamination

The test set is public, and half of it descends from MMLU, whose test items the study by Deng and colleagues found commercial models could partly reproduce (see MMLU).

Use in policy and safety frameworks

MMLU-Pro is a general capability measure and does not figure in frontier safety thresholds. Its policy-relevant use is as a common yardstick in comparisons of open-weight and Chinese models with US frontier models: Chinese developers including DeepSeek, Z.ai and Moonshot AI report it routinely, and forecasts about whether Chinese or open-weight models will overtake US closed models have been written with MMLU-Pro as one of the resolving benchmarks (US-China AI Competition: Different Races, Different Metrics; Reflection AI). Thomson Reuters used an average of GPQA Diamond, Humanity's Last Exam and MMLU-Pro as the "Reasoning" row when benchmarking its legal-domain model against general models (Thomson Reuters Built Its Own AI Model That Now Ranks Among the World's Best). OpenAI's chain-of-thought controllability evaluation, reported in its GPT-6 Astra system card, draws tasks from it (Reasoning models struggle to control their chains of thought, and that's good (OpenAI, March 2026)).

Relationships

• depends-on: MMLU — source of about half the questions and the benchmark it was built to succeed.
• instance-of: AI Benchmarks and Evaluation
• related: GPQA, Humanity's Last Exam — harder closed-ended knowledge benchmarks reported alongside it.
• related: Artificial Analysis Intelligence Index — composite index that included MMLU-Pro until version 4.0.
• related: Hugging Face — hosts the dataset and leaderboard and used the benchmark in its Open LLM Leaderboard.

See also

• HELM (Holistic Evaluation of Language Models)
• SimpleQA
• AI Progress Measurement