This is the paper by Xiang Yue, Tianyu Zheng, Yuansheng Ni (equal contributors), Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen and Graham Neubig, writing as the MMMU Team, that introduces MMMU-Pro, a revised version of the MMMU benchmark. It was submitted to arXiv as 2409.02813 on 4 September 2024; the captured text is version 3 of 22 May 2025. The paper's premise is that some MMMU questions can be answered without their images, and it describes a three-step procedure to remove that shortcut.
Summary
Problems identified in MMMU
The authors report that text-only language models can answer some MMMU questions correctly, for two reasons: some questions are independent of their images, and models can exploit correlations among the candidate options using pre-training knowledge even where humans would need the image. They give two examples answered correctly by Llama-3-70B Instruct without vision.
Construction
Filtering. Four open-source text-only models (Llama3-70B-Instruct, Qwen2-72B-Instruct, Yi-1.5-34B-Chat and Mixtral-8×22B-Instruct) answered every MMMU question ten times without images, required to answer even when they said an image was needed. A question was treated as answerable by a model if it was correct more than five times, and excluded if at least three of the four models answered it correctly across the majority of trials. From the remainder, 1,800 questions were sampled evenly, 60 per subject.
Augmenting options. Because some questions remained answerable from hints in the options, the number of candidate options was increased from four to ten. GPT-4o generated options, Claude 3.5 filtered them, and two rounds of human review refined and verified them. During this step experts also removed 70 questions lacking a clear connection to their image, leaving 1,730.
Vision-only setting. Human annotators captured photos and screenshots of each question displayed in a simulated environment with varied backgrounds, fonts and sizes, so that the model receives the question only as an image. The authors motivate this from real usage, where users screenshot questions rather than typing them, and from the human ability to integrate visual and textual content. The final benchmark has 3,460 items: 1,730 in the standard ten-option format and 1,730 as screenshots or photos.
Results
Increasing options from four to ten (Δ1) reduced every model's accuracy; GPT-4o (0513) fell from 64.7% on MMMU validation to 54.0%. The vision-only setting reduced accuracy further (Δ2): GPT-4o dropped another 4.3 points to 49.7%, and LLaVA-OneVision-72B dropped 14.0 points. Overall declines from MMMU validation to MMMU-Pro ranged from 16.8% (Claude 3.5 Sonnet) to 26.9% (VILA-1.5-40B), with Gemini 1.5 Pro (0801) falling 18.9%. Table 1 lists, for the standard ten-option and vision settings respectively, GPT-4o (0513) 54.0 and 49.7, Claude 3.5 Sonnet 55.0 and 48.0, Gemini 1.5 Pro (0801) 49.4 and 44.4, Qwen2-VL-72B 49.2 and 43.3, GPT-4o mini 39.9 and 35.2, InternVL2-Llama3-76B 41.9 and 38.0, and InternVL2-8B 32.5 and 25.4.
Chain-of-thought prompting generally helped, with Claude 3.5 Sonnet rising from 42.7% to 55.0% in the standard setting, though some models such as VILA1.5-40B fell, which the authors attribute to weak instruction following. By discipline, chain-of-thought gave GPT-4o a 14.49-point gain in Tech and Engineering and 8.22 in Science but only 1.58 in Art and Design, where LLaVA-OneVision 72B lost 17.12 points. OCR prompts had minimal effect in the vision setting. An analysis of 60 GPT-4o errors in the vision setting found reasoning errors at 46% of cases, up from 26% in the original MMMU analysis, with text recognition not the main bottleneck. GPT-4o's responses in the vision setting were shorter and more descriptive than analytical.
Human performance
Rather than re-running human evaluation, the authors approximate expert performance from the original MMMU data for the 577 MMMU-Pro questions drawn from the MMMU validation set, counting answers with documented solution processes and applying a conservative correction for guesses. The resulting estimates are 73.0% (low), 80.8% (medium) and 85.4% (high) overall, against 76.2%, 82.6% and 88.6% on MMMU. The limitations section names this reliance on approximated rather than directly measured human performance as a potential source of bias, alongside the possibility of remaining statistical shortcuts and the vision-only setting not fully capturing human perception.
Key claims
- Three-step construction: text-only filtering by four open-source LLMs; options expanded from 4 to 10; vision-only screenshots. 1,730 questions × 2 formats = 3,460 items.
- GPT-4o (0513): 64.7% MMMU validation → 54.0% ten-option → 49.7% vision-only. Drops of 16.8% to 26.9% across models.
- Approximated human expert performance: 73.0% / 80.8% / 85.4% (low / medium / high).
- Chain-of-thought generally improves scores; OCR prompts have minimal effect; reasoning errors rise to 46% of GPT-4o errors in the vision setting.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 3 (arxiv.org); abstract page arxiv.org. Leaderboard: mmmu-benchmark.github.io.
Relationships
- supports: MMMU — the primary text for the MMMU-Pro variant and its results.
- supersedes: MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (Yue et al., 2023) — in scope of the questions retained and the input formats; the original benchmark remains in use.
- instance-of: AI Benchmarks and Evaluation
- related: Multimodality