This is the paper by Chaoyou Fu of Nanjing University and 20 co-authors from institutions including the State Key Laboratory of Cognitive Intelligence, Xiamen University, the University of Hong Kong, Peking University, the Chinese University of Hong Kong, East China Normal University and CASIA, with Xing Sun as corresponding author, that introduces the Video-MME benchmark. It was submitted to arXiv as 2405.21075 on 31 May 2024; the captured text is version 3 of 30 May 2025. The paper presents the benchmark as the first "full-spectrum" evaluation of multimodal large language models on video, covering a range of video types, durations and input modalities.
Summary
Design
The authors state four distinguishing features: diversity across 6 visual domains and 30 subfields; durations spanning short, medium and long videos from 11 seconds to 1 hour; breadth of modalities, with subtitles and audio alongside frames; and manual annotation by expert annotators.
Videos were sourced from YouTube under a domain hierarchy (Knowledge, Film & Television, Sports Competition, Artistic Performance, Life Record and Multilingual), each subdivided into tags such as football and basketball, giving 30 categories. For each category the team collected short (under 2 minutes), medium (4–15 minutes) and long (30–60 minutes) videos, together with subtitles where available and audio. The dataset has 900 videos, 744 with subtitles and all 900 with audio.
Annotators with strong English proficiency and research experience in vision-language learning watched each entire video and wrote three questions with four options each, for 2,700 question-answer pairs across 12 task types covering perception, reasoning and information synthesis. Each pair was reviewed by a different annotator for clarity and answerability. To ensure the video is needed, the text-only questions were given to Gemini 1.5 Pro and pairs it could answer from text alone were returned for revision; the model reached under 15% accuracy in the text-only setup. Option letters are near-uniformly distributed (25.1/27.2/25.3/22.4%). Questions, options and answers average 12.7, 22.9 and 5.5 words, while subtitles average 198.6 words for short videos and 6,515.6 for long ones.
Using the "certificate length" measure from EgoSchema (the total length of the sub-clips needed to convince a verifier of the answer), the median certificate is 26.0 seconds for short, 164.7 seconds for medium and 890.7 seconds for long videos, which the authors say makes Video-MME "the most challenging Video QA dataset to date."
Results
Four commercial models (GPT-4V, GPT-4o, Gemini 1.5 Flash, Gemini 1.5 Pro), nine open-source video models and three image models were evaluated with and without subtitles. Overall accuracies without and with subtitles were: Gemini 1.5 Pro 75.0% and 81.3%; GPT-4o 71.9% and 77.2%; Gemini 1.5 Flash 70.3% and 75.0%; GPT-4V 59.9% and 63.3%; VILA-1.5 (34B) 59.0% and 59.4%; VITA-1.5 (7B) 56.1% and 58.7%; LLaVA-NeXT-Video (34B) 52.0% and 54.9%. Gemini 1.5 Pro outperformed GPT-4V and GPT-4o by 15.1 and 3.1 points with frames alone, did best in Artistic Performance and worst in Sports Competition, and lost 14.3 points from short to long videos. Adding audio raised its long-video accuracy by 6.2 points and its Multilingual accuracy by up to 16.7 points; subtitles raised its overall score by 6.2 points.
Among open-source models, VITA-1.5 led the 7B class and VILA-1.5 with a 34B language model scored 59.0%, with the largest gaps to Gemini 1.5 Pro in counting, action recognition and temporal perception. Subtitles consistently helped open-source models; most did not support audio input. Image models Qwen-VL-Max and InternVL-Chat-V1.5 performed comparably to LLaVA-NeXT-Video, which the authors read as showing that image understanding underlies video understanding and that the benchmark applies to both kinds of model. Performance declined with video duration for every model.
Discussion
The authors identify long-context modelling as the main weakness revealed, noting that restricted input frames bottleneck open-source models on long videos, and call for context-extension techniques, architectures that select key frames or compress video tokens, and instruction-tuning datasets focused on temporal reasoning in long videos.
Key claims
- 900 YouTube videos (11 seconds to 1 hour), 6 domains, 30 categories; 744 with subtitles, 900 with audio; 2,700 four-option questions across 12 task types.
- Text-only filter: Gemini 1.5 Pro under 15% on questions without video.
- Median certificate lengths 26.0 s / 164.7 s / 890.7 s for short / medium / long videos.
- Overall accuracy (frames only / with subtitles): Gemini 1.5 Pro 75.0 / 81.3; GPT-4o 71.9 / 77.2; Gemini 1.5 Flash 70.3 / 75.0; GPT-4V 59.9 / 63.3; VILA-1.5 59.0 / 59.4.
- Accuracy declines with video length for all models; audio and subtitles improve Gemini 1.5 Pro.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 3 (arxiv.org); abstract page arxiv.org. Project page: video-mme.github.io.
Relationships
- supports: Video-MME — the primary text for the benchmark's construction and release-time results.
- instance-of: AI Benchmarks and Evaluation
- related: Multimodality, Context Length and Long Context