This is the paper by Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng and 18 co-authors, with Yue, Ni, Zhang, Zheng, Huan Sun, Yu Su and Wenhu Chen listed as core contributors, that introduces the MMMU benchmark. The authors are affiliated with IN.AI Research, the University of Waterloo, The Ohio State University, Carnegie Mellon University, the University of Victoria and Princeton University. It was submitted to arXiv as 2311.16502 on 27 November 2023; the captured text is version 4 of 13 June 2024, whose change log records the addition of Gemini and other results after version 1. The paper frames the benchmark as a test of "expert-level" multimodal understanding, requiring college-level subject knowledge and deliberate reasoning over images.
Summary
The benchmark
MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning) contains 11.5K questions collected from college exams, quizzes and textbooks, covering six disciplines (Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, Tech & Engineering), 30 subjects and 183 subfields, with 30 heterogeneous image types including charts, diagrams, maps, tables, music sheets and chemical structures. The questions are split into a few-shot development set of 5 questions per subject, a validation set of about 900 questions and a test set of 10.5K questions. Questions may be multiple-choice or open-ended and interleave text and images. The authors say the benchmark measures three skills, perception, knowledge and reasoning, and contrast it with prior benchmarks such as VQA, GQA, TextVQA, MMBench and ScienceQA that focus on daily knowledge, common sense and a limited range of image formats.
Data curation
Subjects were chosen from common university majors on the principle that visual inputs should be commonly used in the subject, which excluded fields such as law and linguistics. More than 50 university students, including co-authors, collected questions from textbooks and online resources, creating new questions where necessary, under instructions to respect copyright and licensing and to avoid data from sites prohibiting redistribution. Quality control removed duplicates identified by lexical overlap and source-URL similarity, checked format and typos, and classified problems into four difficulty levels; about 10% of problems, rated "very easy," were excluded.
Results
The paper evaluates 28 open-source large multimodal models plus GPT-4V(ision) and Gemini. On the validation set, GPT-4V (Playground) scored 56.8% and on the test set 55.7%; the abstract summarises GPT-4V at 56% and Gemini Ultra at 59%, and the results table lists Gemini Ultra and Claude 3 Opus at 59.4% on validation from developer-provided results. Random choice scores 22.1% on validation and 23.9% on test, and frequent choice 26.8% and 25.8%. Human expert performance on validation was 76.2% (worst), 82.6% (medium) and 88.6% (best). Leading open-source models at submission time, BLIP-2 FLAN-T5-XXL and LLaVA-1.5, reached about 34%; later versions of the paper add LLaVA-1.6-34B at 44.7% and InternVL-Chat-V1.2 at 46.2% on the test set. OCR and captioning enhancements did not significantly improve text-only language models, which the authors read as evidence that the questions require integrating text and image.
By discipline, models did relatively better in Art & Design and Humanities & Social Science, where images are more "natural" and questions involve less reasoning, and worse in Science, Health & Medicine and Tech & Engineering. Across image types, all models scored close to random on geometric shapes, music sheets and chemical structures. By difficulty, GPT-4V scored 76.1% on easy, 55.6% on medium and 31.2% on hard test questions, with the gap to open-source models almost disappearing on the hard set.
Error analysis
Expert annotators examined 150 randomly sampled GPT-4V errors. Perceptual errors accounted for 35% (basic misreadings of the image, and a bias toward text over visual input), lack of knowledge for 29%, reasoning errors for 26%, and textual-understanding errors, refusals, annotation errors and answer-extraction errors for 6%, 3%, 2% and 1%. The authors identify grounding, complex reasoning chains and the interplay of language and vision as open challenges. The appendix notes that annotators were advised to choose questions not immediately findable online to limit contamination, and that a live leaderboard is maintained at the project site.
Key claims
- 11.5K questions; 6 disciplines, 30 subjects, 183 subfields, 30 image types; 5-per-subject development set, ~900 validation, 10.5K test.
- GPT-4V: 56.8% validation, 55.7% test; Gemini Ultra 59.4% validation (developer-provided); best open-source at submission ~34%.
- Human experts on validation: 76.2% worst, 82.6% medium, 88.6% best; random choice 22.1%.
- GPT-4V by difficulty: 76.1% easy, 55.6% medium, 31.2% hard.
- GPT-4V error breakdown: 35% perception, 29% knowledge, 26% reasoning.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 4 (arxiv.org); abstract page arxiv.org. Project site and leaderboard: mmmu-benchmark.github.io.
Relationships
- supports: MMMU — the primary text for the benchmark's construction, human baseline and release-time results.
- related: MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (Yue et al., 2024) — the 2024 revision that filters text-answerable questions and adds a vision-only setting.
- instance-of: AI Benchmarks and Evaluation
- related: Multimodality