"Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities" is a technical report by the Gemini Team at Google, first posted to arXiv on July 7, 2025 and last revised on December 19, 2025 (version 6, summarised here) (Source: https://arxiv.org/abs/2507.06261). It introduces what it calls the Gemini 2.X model family: Gemini 2.5 Pro and Gemini 2.5 Flash, together with the earlier Gemini 2.0 Flash and Flash-Lite. The shorter Gemini 2.5 Pro model card refers readers to this report for detail. Most of the document's length is its contributor list.
Models, architecture and training
The report describes the Gemini 2.5 models as sparse mixture-of-experts transformers with native multimodal support for text, vision and audio input. Per its Table 1, Gemini 2.5 Pro and Flash accept text, image, video and audio with a 1M-token input length, produce up to 64K tokens of output, have a knowledge cutoff of January 2025, support tool use, and use "dynamic" thinking; the Gemini 2.0 models have an 8K output length and a June 2024 cutoff. Parameter counts are not given. Pre-training data is described as publicly available web documents, code, images, audio and video, with improved filtering and deduplication relative to Gemini 1.5; post-training uses vetted instruction-tuning, human-preference and tool-use data, with more training compute allocated to reinforcement learning and a focus on verifiable and model-based generative rewards.
The training-infrastructure section reports that the models were trained on TPUv5p accelerators across multiple datacenters, with slice-granularity elasticity, under which training continues at around 97% throughput while a failed slice recovers, and split-phase detection of silent data corruption; 93.4% of the run's time was spent on TPU computation. The thinking section states that the models are trained with reinforcement learning to use additional inference-time compute and can spend "tens of thousands of forward passes" before responding, with a controllable thinking budget. The report also describes Gemini 2.5 Pro Deep Think, a parallel-thinking approach announced at Google I/O and released experimentally to trusted testers in June 2025, without scores.
Capability evaluation
The methodology section states that all Gemini scores are pass@1 in a "single attempt" setting unless marked, run through the AI Studio API with default sampling, and that results for other developers' models are the providers' self-reported numbers unless stated; it cautions that SWE-bench Verified numbers across providers "aren't directly comparable" because scaffolds differ. Google describes n-gram, semantic-similarity and model-based decontamination, and continues to report an internal non-public benchmark, HiddenMath.
Table 3 compares six Gemini models (1.5 Flash and Pro, 2.0 Flash-Lite and Flash, 2.5 Flash and Pro) on LiveCodeBench, Aider Polyglot, SWE-bench Verified (single and multiple attempts), GPQA diamond, Humanity's Last Exam, SimpleQA, FACTS Grounding, Global MMLU (Lite), ECLeKTic, AIME 2025, HiddenMath-Hard, LOFT, MRCR-V2 (8-needle), MMMU, Vibe-Eval, ZeroBench and BetterChartQA. Table 4 compares Gemini 2.5 Pro with OpenAI o3 and o4-mini at high reasoning, Claude Sonnet 4 and Opus 4, Grok 3 Beta with extended thinking, and DeepSeek R1 0528. The report states that Gemini 2.5 Pro achieves the highest score among those models on Aider Polyglot, Humanity's Last Exam, GPQA diamond, SimpleQA and FACTS Grounding, and on LOFT and MRCR at 128k context, and that it is the only model in the table supporting context lengths of 1M tokens or more; in the same table the Claude models are higher on SWE-bench Verified, o4-mini on AIME 2025 and LiveCodeBench, and o3 on MMMU. Tables 5 and 6 report audio (FLEURS, CoVoST2) and video (fourteen benchmark settings including VideoMME, VideoMMMU and LVBench) results against GPT-4o audio models and GPT-4.1, which Google ran itself with the same prompts and inputs. The report also states that Gemini Deep Research, built on Gemini 2.5 Pro, rose on Humanity's Last Exam from 7.95% in December 2024 to 26.9%, and 32.4% with higher compute, in June 2025. A case study describes Gemini 2.5 Pro playing Pokémon Blue through an agent harness on a Twitch stream set up by an independent developer, Joel Zhang, and the failure modes observed.
Safety, security and memorization
Automated safety evaluations are reported as changes relative to Gemini 1.5 baselines (Table 7). Google states that the 2.0 models were substantially safer than 1.5 but over-refused benign requests, and that Gemini 2.5 was trained to reduce such refusals; for Gemini 2.5 Pro against Gemini 1.5 Pro 002 it reports fewer text-to-text policy violations, more image-to-text violations marked as not egregious, and gains in tone and in helpfulness and instruction following. Automated red teaming, framed as a multi-agent game between Gemini attackers and the target model, found dangerous-content policy violations falling from 43.5% for Gemini 1.5 Pro 002 to 24.3% for Gemini 2.5 Pro and helpfulness violations from 8.9% to 6.1% (Table 8). For indirect prompt injection, the report gives best attack success rates over 500 held-out scenarios for three attack methods (Actor Critic, Beam Search and Tree of Attacks with Pruning); Gemini 2.5 Flash and Pro were more resilient than their 1.5 baselines on all three after adversarial training, while the report notes that Gemini 2.5 Pro "is still less resilient compared to Gemini 2.5 Flash" (Table 9).
On memorization, Google sampled over 700,000 training documents and found the 2.X family memorizes long-form text at a much lower rate than earlier models, with no personal information observed in outputs classified as memorized. A divergence attack succeeded roughly 69% of the time against the 2.0 models and 59% against the 2.5 family, with about 0.2% of diverged text classified as memorization.
Frontier Safety Framework
Gemini 2.5 Pro was evaluated against the critical capability levels (CCLs) in Google's Frontier Safety Framework for CBRN, cybersecurity, machine learning R&D and deceptive alignment, and the report concludes that no CCL was reached. The evaluations "did reach an alert threshold for the Cyber Uplift 1 CCL", and Google says it put in place a response plan of more frequent cyber testing and accelerated mitigations. Reported results include solve counts on an autonomous cyber offense suite and on a new "key skills" benchmark of 48 challenges, where Gemini 2.5 Pro solved 6 of 12 hard challenges; RE-Bench, where the best agent solutions scored between 50% and 125% of the best expert-written solutions and the alert threshold, set above human performance, was not reached; and deceptive-alignment evaluations run on Gemini 2.5 Pro Preview 03-25, in which the model solved 2 of 5 stealth and 2 of 11 situational-awareness challenges. CBRN multiple-choice results (SecureBio VMQA, LAB-Bench subsets and WMDP) are shown only as a chart. External safety testing covered an early version, Gemini 2.5 Pro Preview 05-06, with findings reported qualitatively; the Flash models were not externally tested.
Reported results
Evaluation results the document states for which there is no separate benchmark page; results on benchmarks that have a page are recorded there and cite this page. Scores and settings are as printed in the document.
| Evaluation | Model | Score | Setting | Section |
|---|---|---|---|---|
| FACTS Grounding | Gemini 1.5 Flash | 82.9% | sourced from the Kaggle FACTS Grounding leaderboard; pass@1, AI Studio API model id gemini-1.5-flash-002, default sampling; developer-reported | 3.2 (Table 3) |
| FACTS Grounding | Gemini 1.5 Pro | 80.0% | sourced from the Kaggle FACTS Grounding leaderboard; pass@1, AI Studio API model id gemini-1.5-pro-002, default sampling; developer-reported | 3.2 (Table 3) |
| FACTS Grounding | Gemini 2.0 Flash-Lite | 82.4% | sourced from the Kaggle FACTS Grounding leaderboard; pass@1, AI Studio API model id gemini-2.0-flash-lite-001, default sampling; developer-reported | 3.2 (Table 3) |
| FACTS Grounding | Gemini 2.0 Flash | 84.6% | sourced from the Kaggle FACTS Grounding leaderboard; pass@1, AI Studio API model id gemini-2.0-flash-001, default sampling; developer-reported | 3.2 (Table 3) |
| FACTS Grounding | Gemini 2.5 Flash | 85.3% | sourced from the Kaggle FACTS Grounding leaderboard; pass@1, AI Studio API model id gemini-2.5-flash, default sampling; developer-reported | 3.2 (Table 3) |
| FACTS Grounding | Gemini 2.5 Pro | 87.8% | sourced from the Kaggle FACTS Grounding leaderboard; pass@1, AI Studio API model id gemini-2.5-pro, default sampling; developer-reported | 3.2 (Table 3) |
| FACTS Grounding | OpenAI o3 (high) | 69.9% | reported by Google in the Gemini 2.5 technical report; sourced from the Kaggle FACTS Grounding leaderboard | 3.3 (Table 4) |
| FACTS Grounding | OpenAI o4-mini (high) | 62.1% | reported by Google in the Gemini 2.5 technical report; sourced from the Kaggle FACTS Grounding leaderboard | 3.3 (Table 4) |
| FACTS Grounding | Claude Sonnet 4 | 79.1% | reported by Google in the Gemini 2.5 technical report; sourced from the Kaggle FACTS Grounding leaderboard | 3.3 (Table 4) |
| FACTS Grounding | Claude Opus 4 | 77.7% | reported by Google in the Gemini 2.5 technical report; sourced from the Kaggle FACTS Grounding leaderboard | 3.3 (Table 4) |
| FACTS Grounding | Grok 3 Beta (Extended Thinking) | 74.8% | reported by Google in the Gemini 2.5 technical report; sourced from the Kaggle FACTS Grounding leaderboard | 3.3 (Table 4) |
| FACTS Grounding | DeepSeek R1 0528 | 82.4% | reported by Google in the Gemini 2.5 technical report; sourced from the Kaggle FACTS Grounding leaderboard | 3.3 (Table 4) |
| Global MMLU (Lite) | Gemini 1.5 Flash | 72.5% | 15 languages; 200 culturally sensitive and 200 culturally agnostic samples per language; pass@1, AI Studio API model id gemini-1.5-flash-002, default sampling; developer-reported | 3.2 (Table 3) |
| Global MMLU (Lite) | Gemini 1.5 Pro | 80.8% | 15 languages; 200 culturally sensitive and 200 culturally agnostic samples per language; pass@1, AI Studio API model id gemini-1.5-pro-002, default sampling; developer-reported | 3.2 (Table 3) |
| Global MMLU (Lite) | Gemini 2.0 Flash-Lite | 78.0% | 15 languages; 200 culturally sensitive and 200 culturally agnostic samples per language; pass@1, AI Studio API model id gemini-2.0-flash-lite-001, default sampling; developer-reported | 3.2 (Table 3) |
| Global MMLU (Lite) | Gemini 2.0 Flash | 83.4% | 15 languages; 200 culturally sensitive and 200 culturally agnostic samples per language; pass@1, AI Studio API model id gemini-2.0-flash-001, default sampling; developer-reported | 3.2 (Table 3) |
| Global MMLU (Lite) | Gemini 2.5 Flash | 88.4% | 15 languages; 200 culturally sensitive and 200 culturally agnostic samples per language; pass@1, AI Studio API model id gemini-2.5-flash, default sampling; developer-reported | 3.2 (Table 3) |
| Global MMLU (Lite) | Gemini 2.5 Pro | 89.2% | 15 languages; 200 culturally sensitive and 200 culturally agnostic samples per language; pass@1, AI Studio API model id gemini-2.5-pro, default sampling; developer-reported | 3.2 (Table 3) |
| ECLeKTic | Gemini 1.5 Flash | 16.4% | closed-book cross-lingual knowledge transfer QA; pass@1, AI Studio API model id gemini-1.5-flash-002, default sampling; developer-reported | 3.2 (Table 3) |
| ECLeKTic | Gemini 1.5 Pro | 27.0% | closed-book cross-lingual knowledge transfer QA; pass@1, AI Studio API model id gemini-1.5-pro-002, default sampling; developer-reported | 3.2 (Table 3) |
| ECLeKTic | Gemini 2.0 Flash-Lite | 27.7% | closed-book cross-lingual knowledge transfer QA; pass@1, AI Studio API model id gemini-2.0-flash-lite-001, default sampling; developer-reported | 3.2 (Table 3) |
| ECLeKTic | Gemini 2.0 Flash | 33.6% | closed-book cross-lingual knowledge transfer QA; pass@1, AI Studio API model id gemini-2.0-flash-001, default sampling; developer-reported | 3.2 (Table 3) |
| ECLeKTic | Gemini 2.5 Flash | 36.8% | closed-book cross-lingual knowledge transfer QA; pass@1, AI Studio API model id gemini-2.5-flash, default sampling; developer-reported | 3.2 (Table 3) |
| ECLeKTic | Gemini 2.5 Pro | 46.8% | closed-book cross-lingual knowledge transfer QA; pass@1, AI Studio API model id gemini-2.5-pro, default sampling; developer-reported | 3.2 (Table 3) |
| HiddenMath-Hard | Gemini 1.5 Flash | 36.8% | internal held-out competition-level math set; pass@1, AI Studio API model id gemini-1.5-flash-002, default sampling; developer-reported | 3.2 (Table 3) |
| HiddenMath-Hard | Gemini 1.5 Pro | 44.3% | internal held-out competition-level math set; pass@1, AI Studio API model id gemini-1.5-pro-002, default sampling; developer-reported | 3.2 (Table 3) |
| HiddenMath-Hard | Gemini 2.0 Flash-Lite | 47.4% | internal held-out competition-level math set; pass@1, AI Studio API model id gemini-2.0-flash-lite-001, default sampling; developer-reported | 3.2 (Table 3) |
| HiddenMath-Hard | Gemini 2.0 Flash | 53.7% | internal held-out competition-level math set; pass@1, AI Studio API model id gemini-2.0-flash-001, default sampling; developer-reported | 3.2 (Table 3) |
| HiddenMath-Hard | Gemini 2.5 Flash | 75.5% | internal held-out competition-level math set; pass@1, AI Studio API model id gemini-2.5-flash, default sampling; developer-reported | 3.2 (Table 3) |
| HiddenMath-Hard | Gemini 2.5 Pro | 80.5% | internal held-out competition-level math set; pass@1, AI Studio API model id gemini-2.5-pro, default sampling; developer-reported | 3.2 (Table 3) |
| LOFT (hard retrieval) — ≤128K | Gemini 1.5 Flash | 67.3% | 300 queries; contexts up to 128K (average); pass@1, AI Studio API model id gemini-1.5-flash-002, default sampling; developer-reported | 3.2 (Table 3) |
| LOFT (hard retrieval) — ≤128K | Gemini 1.5 Pro | 75.9% | 300 queries; contexts up to 128K (average); pass@1, AI Studio API model id gemini-1.5-pro-002, default sampling; developer-reported | 3.2 (Table 3) |
| LOFT (hard retrieval) — ≤128K | Gemini 2.0 Flash-Lite | 50.7% | 300 queries; contexts up to 128K (average); pass@1, AI Studio API model id gemini-2.0-flash-lite-001, default sampling; developer-reported | 3.2 (Table 3) |
| LOFT (hard retrieval) — ≤128K | Gemini 2.0 Flash | 58.0% | 300 queries; contexts up to 128K (average); pass@1, AI Studio API model id gemini-2.0-flash-001, default sampling; developer-reported | 3.2 (Table 3) |
| LOFT (hard retrieval) — ≤128K | Gemini 2.5 Flash | 82.1% | 300 queries; contexts up to 128K (average); pass@1, AI Studio API model id gemini-2.5-flash, default sampling; developer-reported | 3.2 (Table 3) |
| LOFT (hard retrieval) — ≤128K | Gemini 2.5 Pro | 87.0% | 300 queries; contexts up to 128K (average); pass@1, AI Studio API model id gemini-2.5-pro, default sampling; developer-reported | 3.2 (Table 3) |
| LOFT (hard retrieval) — 1M | Gemini 1.5 Flash | 36.7% | 300 queries; pointwise at exactly 1M context; pass@1, AI Studio API model id gemini-1.5-flash-002, default sampling; developer-reported | 3.2 (Table 3) |
| LOFT (hard retrieval) — 1M | Gemini 1.5 Pro | 47.1% | 300 queries; pointwise at exactly 1M context; pass@1, AI Studio API model id gemini-1.5-pro-002, default sampling; developer-reported | 3.2 (Table 3) |
| LOFT (hard retrieval) — 1M | Gemini 2.0 Flash-Lite | 7.6% | 300 queries; pointwise at exactly 1M context; pass@1, AI Studio API model id gemini-2.0-flash-lite-001, default sampling; developer-reported | 3.2 (Table 3) |
| LOFT (hard retrieval) — 1M | Gemini 2.0 Flash | 7.6% | 300 queries; pointwise at exactly 1M context; pass@1, AI Studio API model id gemini-2.0-flash-001, default sampling; developer-reported | 3.2 (Table 3) |
| LOFT (hard retrieval) — 1M | Gemini 2.5 Flash | 58.9% | 300 queries; pointwise at exactly 1M context; pass@1, AI Studio API model id gemini-2.5-flash, default sampling; developer-reported | 3.2 (Table 3) |
| LOFT (hard retrieval) — 1M | Gemini 2.5 Pro | 69.8% | 300 queries; pointwise at exactly 1M context; pass@1, AI Studio API model id gemini-2.5-pro, default sampling; developer-reported | 3.2 (Table 3) |
| LOFT (hard retrieval) — ≤128K | OpenAI o3 (high) | 77.0% | contexts up to 128K (average); reported by Google in the Gemini 2.5 technical report; source of the comparison figure not stated | 3.3 (Table 4) |
| LOFT (hard retrieval) — ≤128K | OpenAI o4-mini (high) | 60.5% | contexts up to 128K (average); reported by Google in the Gemini 2.5 technical report; source of the comparison figure not stated | 3.3 (Table 4) |
| LOFT (hard retrieval) — ≤128K | Claude Sonnet 4 | 81.6% | contexts up to 128K (average); reported by Google in the Gemini 2.5 technical report; source of the comparison figure not stated | 3.3 (Table 4) |
| LOFT (hard retrieval) — ≤128K | Grok 3 Beta (Extended Thinking) | 73.1% | contexts up to 128K (average); reported by Google in the Gemini 2.5 technical report; source of the comparison figure not stated | 3.3 (Table 4) |
| Vibe-Eval (Reka) | Gemini 1.5 Flash | 52.3% | Gemini used as judge; pass@1, AI Studio API model id gemini-1.5-flash-002, default sampling; developer-reported | 3.2 (Table 3) |
| Vibe-Eval (Reka) | Gemini 1.5 Pro | 55.9% | Gemini used as judge; pass@1, AI Studio API model id gemini-1.5-pro-002, default sampling; developer-reported | 3.2 (Table 3) |
| Vibe-Eval (Reka) | Gemini 2.0 Flash-Lite | 51.5% | Gemini used as judge; pass@1, AI Studio API model id gemini-2.0-flash-lite-001, default sampling; developer-reported | 3.2 (Table 3) |
| Vibe-Eval (Reka) | Gemini 2.0 Flash | 55.4% | Gemini used as judge; pass@1, AI Studio API model id gemini-2.0-flash-001, default sampling; developer-reported | 3.2 (Table 3) |
| Vibe-Eval (Reka) | Gemini 2.5 Flash | 65.4% | Gemini used as judge; pass@1, AI Studio API model id gemini-2.5-flash, default sampling; developer-reported | 3.2 (Table 3) |
| Vibe-Eval (Reka) | Gemini 2.5 Pro | 67.2% | Gemini used as judge; pass@1, AI Studio API model id gemini-2.5-pro, default sampling; developer-reported | 3.2 (Table 3) |
| ZeroBench | Gemini 1.5 Flash | 0.5% | Gemini used as judge; average over 4 runs; pass@1, AI Studio API model id gemini-1.5-flash-002, default sampling; developer-reported | 3.2 (Table 3) |
| ZeroBench | Gemini 1.5 Pro | 1.0% | Gemini used as judge; average over 4 runs; pass@1, AI Studio API model id gemini-1.5-pro-002, default sampling; developer-reported | 3.2 (Table 3) |
| ZeroBench | Gemini 2.0 Flash-Lite | 0.75% | Gemini used as judge; average over 4 runs; pass@1, AI Studio API model id gemini-2.0-flash-lite-001, default sampling; developer-reported | 3.2 (Table 3) |
| ZeroBench | Gemini 2.0 Flash | 1.25% | Gemini used as judge; average over 4 runs; pass@1, AI Studio API model id gemini-2.0-flash-001, default sampling; developer-reported | 3.2 (Table 3) |
| ZeroBench | Gemini 2.5 Flash | 2.0% | Gemini used as judge; average over 4 runs; pass@1, AI Studio API model id gemini-2.5-flash, default sampling; developer-reported | 3.2 (Table 3) |
| ZeroBench | Gemini 2.5 Pro | 4.5% | Gemini used as judge; average over 4 runs; pass@1, AI Studio API model id gemini-2.5-pro, default sampling; developer-reported | 3.2 (Table 3) |
| BetterChartQA | Gemini 1.5 Flash | 59.0% | Gemini used as judge; pass@1, AI Studio API model id gemini-1.5-flash-002, default sampling; developer-reported | 3.2 (Table 3) |
| BetterChartQA | Gemini 1.5 Pro | 65.8% | Gemini used as judge; pass@1, AI Studio API model id gemini-1.5-pro-002, default sampling; developer-reported | 3.2 (Table 3) |
| BetterChartQA | Gemini 2.0 Flash-Lite | 52.3% | Gemini used as judge; pass@1, AI Studio API model id gemini-2.0-flash-lite-001, default sampling; developer-reported | 3.2 (Table 3) |
| BetterChartQA | Gemini 2.0 Flash | 57.8% | Gemini used as judge; pass@1, AI Studio API model id gemini-2.0-flash-001, default sampling; developer-reported | 3.2 (Table 3) |
| BetterChartQA | Gemini 2.5 Flash | 67.3% | Gemini used as judge; pass@1, AI Studio API model id gemini-2.5-flash, default sampling; developer-reported | 3.2 (Table 3) |
| BetterChartQA | Gemini 2.5 Pro | 72.4% | Gemini used as judge; pass@1, AI Studio API model id gemini-2.5-pro, default sampling; developer-reported | 3.2 (Table 3) |
| FLEURS — 53 languages, WER (lower is better) | Gemini 1.5 Flash | 12.71 | automatic speech recognition, 0-shot queries to public APIs, 53 of 102 languages; word error rate (character error rate for Chinese, Japanese, Korean and Thai); AI Studio API model id gemini-1.5-flash-002; developer-reported | 3.3 Audio Understanding (Table 5) |
| FLEURS — 53 languages, WER (lower is better) | Gemini 1.5 Pro | 7.14 | automatic speech recognition, 0-shot queries to public APIs, 53 of 102 languages; word error rate (character error rate for Chinese, Japanese, Korean and Thai); AI Studio API model id gemini-1.5-pro-002; developer-reported | 3.3 Audio Understanding (Table 5) |
| FLEURS — 53 languages, WER (lower is better) | Gemini 2.0 Flash-Lite | 9.60 | automatic speech recognition, 0-shot queries to public APIs, 53 of 102 languages; word error rate (character error rate for Chinese, Japanese, Korean and Thai); AI Studio API model id gemini-2.0-flash-lite-001; developer-reported | 3.3 Audio Understanding (Table 5) |
| FLEURS — 53 languages, WER (lower is better) | Gemini 2.0 Flash | 9.04 | automatic speech recognition, 0-shot queries to public APIs, 53 of 102 languages; word error rate (character error rate for Chinese, Japanese, Korean and Thai); AI Studio API model id gemini-2.0-flash-001; developer-reported | 3.3 Audio Understanding (Table 5) |
| FLEURS — 53 languages, WER (lower is better) | Gemini 2.5 Flash | 9.95 | automatic speech recognition, 0-shot queries to public APIs, 53 of 102 languages; word error rate (character error rate for Chinese, Japanese, Korean and Thai); AI Studio API model id gemini-2.5-flash; developer-reported | 3.3 Audio Understanding (Table 5) |
| FLEURS — 53 languages, WER (lower is better) | Gemini 2.5 Pro | 6.66 | automatic speech recognition, 0-shot queries to public APIs, 53 of 102 languages; word error rate (character error rate for Chinese, Japanese, Korean and Thai); AI Studio API model id gemini-2.5-pro; developer-reported | 3.3 Audio Understanding (Table 5) |
| FLEURS — 53 languages, WER (lower is better) | GPT-4o mini Audio Preview | 19.52 | automatic speech recognition, 0-shot queries to public APIs, 53 of 102 languages; word error rate (character error rate for Chinese, Japanese, Korean and Thai); run by Google with the same prompts and inputs — the report notes GPT metrics may differ from previously reported ones; reported by Google in the Gemini 2.5 technical report | 3.3 Audio Understanding (Table 5) |
| FLEURS — 53 languages, WER (lower is better) | GPT-4o Audio Preview | 12.16 | automatic speech recognition, 0-shot queries to public APIs, 53 of 102 languages; word error rate (character error rate for Chinese, Japanese, Korean and Thai); run by Google with the same prompts and inputs — the report notes GPT metrics may differ from previously reported ones; reported by Google in the Gemini 2.5 technical report | 3.3 Audio Understanding (Table 5) |
| FLEURS — 53 languages, WER (lower is better) | GPT-4o transcribe | 8.17 | automatic speech recognition, 0-shot queries to public APIs, 53 of 102 languages; word error rate (character error rate for Chinese, Japanese, Korean and Thai); run by Google with the same prompts and inputs — the report notes GPT metrics may differ from previously reported ones; reported by Google in the Gemini 2.5 technical report | 3.3 Audio Understanding (Table 5) |
| CoVoST2 — 21 languages, BLEU (higher is better) | Gemini 1.5 Flash | 34.81 | speech-to-text translation of 21 languages into English, 0-shot queries to public APIs; BLEU; AI Studio API model id gemini-1.5-flash-002; developer-reported | 3.3 Audio Understanding (Table 5) |
| CoVoST2 — 21 languages, BLEU (higher is better) | Gemini 1.5 Pro | 37.53 | speech-to-text translation of 21 languages into English, 0-shot queries to public APIs; BLEU; AI Studio API model id gemini-1.5-pro-002; developer-reported | 3.3 Audio Understanding (Table 5) |
| CoVoST2 — 21 languages, BLEU (higher is better) | Gemini 2.0 Flash-Lite | 34.74 | speech-to-text translation of 21 languages into English, 0-shot queries to public APIs; BLEU; AI Studio API model id gemini-2.0-flash-lite-001; developer-reported | 3.3 Audio Understanding (Table 5) |
| CoVoST2 — 21 languages, BLEU (higher is better) | Gemini 2.0 Flash | 36.35 | speech-to-text translation of 21 languages into English, 0-shot queries to public APIs; BLEU; AI Studio API model id gemini-2.0-flash-001; developer-reported | 3.3 Audio Understanding (Table 5) |
| CoVoST2 — 21 languages, BLEU (higher is better) | Gemini 2.5 Flash | 36.15 | speech-to-text translation of 21 languages into English, 0-shot queries to public APIs; BLEU; AI Studio API model id gemini-2.5-flash; developer-reported | 3.3 Audio Understanding (Table 5) |
| CoVoST2 — 21 languages, BLEU (higher is better) | Gemini 2.5 Pro | 38.48 | speech-to-text translation of 21 languages into English, 0-shot queries to public APIs; BLEU; AI Studio API model id gemini-2.5-pro; developer-reported | 3.3 Audio Understanding (Table 5) |
| CoVoST2 — 21 languages, BLEU (higher is better) | GPT-4o mini Audio Preview | 29.5 | speech-to-text translation of 21 languages into English, 0-shot queries to public APIs; BLEU; run by Google with the same prompts and inputs — the report notes GPT metrics may differ from previously reported ones; reported by Google in the Gemini 2.5 technical report | 3.3 Audio Understanding (Table 5) |
| CoVoST2 — 21 languages, BLEU (higher is better) | GPT-4o Audio Preview | 35.89 | speech-to-text translation of 21 languages into English, 0-shot queries to public APIs; BLEU; run by Google with the same prompts and inputs — the report notes GPT metrics may differ from previously reported ones; reported by Google in the Gemini 2.5 technical report | 3.3 Audio Understanding (Table 5) |
| ActivityNet-QA — visual-only | Gemini 1.5 Flash | 56.2 | modality: visual-only; test subset, 0-shot; 1 fps, max 1024 frames (500 frames for GPT 4.1); AI Studio API model id gemini-1.5-flash-002; developer-reported | 3.3 Video Understanding (Table 6) |
| ActivityNet-QA — visual-only | Gemini 1.5 Pro | 57.3 | modality: visual-only; test subset, 0-shot; 1 fps, max 1024 frames (500 frames for GPT 4.1); AI Studio API model id gemini-1.5-pro-002; developer-reported | 3.3 Video Understanding (Table 6) |
| ActivityNet-QA — visual-only | Gemini 2.0 Flash-Lite | 55.3 | modality: visual-only; test subset, 0-shot; 1 fps, max 1024 frames (500 frames for GPT 4.1); AI Studio API model id gemini-2.0-flash-lite-001; developer-reported | 3.3 Video Understanding (Table 6) |
| ActivityNet-QA — visual-only | Gemini 2.0 Flash | 56.4 | modality: visual-only; test subset, 0-shot; 1 fps, max 1024 frames (500 frames for GPT 4.1); AI Studio API model id gemini-2.0-flash-001; developer-reported | 3.3 Video Understanding (Table 6) |
| ActivityNet-QA — visual-only | Gemini 2.5 Flash | 65.1 | modality: visual-only; test subset, 0-shot; 1 fps, max 1024 frames (500 frames for GPT 4.1); AI Studio API model id gemini-2.5-flash; developer-reported | 3.3 Video Understanding (Table 6) |
| ActivityNet-QA — visual-only | Gemini 2.5 Pro | 66.7 | modality: visual-only; test subset, 0-shot; 1 fps, max 1024 frames (500 frames for GPT 4.1); AI Studio API model id gemini-2.5-pro; developer-reported | 3.3 Video Understanding (Table 6) |
| ActivityNet-QA — visual-only | GPT-4.1 | 60.4 | modality: visual-only; test subset, 0-shot; 1 fps, max 1024 frames (500 frames for GPT 4.1); run by Google with the same prompt and video frames; reported by Google in the Gemini 2.5 technical report | 3.3 Video Understanding (Table 6) |
| EgoTempo — visual-only | Gemini 1.5 Flash | 34.5 | modality: visual-only; test subset, 0-shot; max 256 frames; AI Studio API model id gemini-1.5-flash-002; developer-reported | 3.3 Video Understanding (Table 6) |
| EgoTempo — visual-only | Gemini 1.5 Pro | 36.3 | modality: visual-only; test subset, 0-shot; max 256 frames; AI Studio API model id gemini-1.5-pro-002; developer-reported | 3.3 Video Understanding (Table 6) |
| EgoTempo — visual-only | Gemini 2.0 Flash-Lite | 30.1 | modality: visual-only; test subset, 0-shot; max 256 frames; AI Studio API model id gemini-2.0-flash-lite-001; developer-reported | 3.3 Video Understanding (Table 6) |
| EgoTempo — visual-only | Gemini 2.0 Flash | 39.3 | modality: visual-only; test subset, 0-shot; max 256 frames; AI Studio API model id gemini-2.0-flash-001; developer-reported | 3.3 Video Understanding (Table 6) |
| EgoTempo — visual-only | Gemini 2.5 Flash | 36.7 | modality: visual-only; test subset, 0-shot; max 256 frames; AI Studio API model id gemini-2.5-flash; developer-reported | 3.3 Video Understanding (Table 6) |
| EgoTempo — visual-only | Gemini 2.5 Pro | 44.3 | modality: visual-only; test subset, 0-shot; max 256 frames; AI Studio API model id gemini-2.5-pro; developer-reported | 3.3 Video Understanding (Table 6) |
| EgoTempo — visual-only | GPT-4.1 | 40.3 | modality: visual-only; test subset, 0-shot; max 256 frames; run by Google with the same prompt and video frames; reported by Google in the Gemini 2.5 technical report | 3.3 Video Understanding (Table 6) |
| Perception Test — visual-only | Gemini 1.5 Flash | 66.5 | modality: visual-only; test subset, 0-shot; max 256 frames; AI Studio API model id gemini-1.5-flash-002; developer-reported | 3.3 Video Understanding (Table 6) |
| Perception Test — visual-only | Gemini 1.5 Pro | 69.4 | modality: visual-only; test subset, 0-shot; max 256 frames; AI Studio API model id gemini-1.5-pro-002; developer-reported | 3.3 Video Understanding (Table 6) |
| Perception Test — visual-only | Gemini 2.0 Flash-Lite | 67.5 | modality: visual-only; test subset, 0-shot; max 256 frames; AI Studio API model id gemini-2.0-flash-lite-001; developer-reported | 3.3 Video Understanding (Table 6) |
| Perception Test — visual-only | Gemini 2.0 Flash | 68.8 | modality: visual-only; test subset, 0-shot; max 256 frames; AI Studio API model id gemini-2.0-flash-001; developer-reported | 3.3 Video Understanding (Table 6) |
| Perception Test — visual-only | Gemini 2.5 Flash | 75.1 | modality: visual-only; test subset, 0-shot; max 256 frames; AI Studio API model id gemini-2.5-flash; developer-reported | 3.3 Video Understanding (Table 6) |
| Perception Test — visual-only | Gemini 2.5 Pro | 78.4 | modality: visual-only; test subset, 0-shot; max 256 frames; AI Studio API model id gemini-2.5-pro; developer-reported | 3.3 Video Understanding (Table 6) |
| Perception Test — visual-only | GPT-4.1 | 64.8 | modality: visual-only; test subset, 0-shot; max 256 frames; run by Google with the same prompt and video frames; reported by Google in the Gemini 2.5 technical report | 3.3 Video Understanding (Table 6) |
| QVHighlights — visual-only | Gemini 1.5 Flash | 64.4 | modality: visual-only; validation subset, 4-shot; R1@0.5; max 256 frames; AI Studio API model id gemini-1.5-flash-002; developer-reported | 3.3 Video Understanding (Table 6) |
| QVHighlights — visual-only | Gemini 1.5 Pro | 68.7 | modality: visual-only; validation subset, 4-shot; R1@0.5; max 256 frames; AI Studio API model id gemini-1.5-pro-002; developer-reported | 3.3 Video Understanding (Table 6) |
| QVHighlights — visual-only | Gemini 2.0 Flash-Lite | 25.7 | modality: visual-only; validation subset, 4-shot; R1@0.5; max 256 frames; AI Studio API model id gemini-2.0-flash-lite-001; developer-reported | 3.3 Video Understanding (Table 6) |
| QVHighlights — visual-only | Gemini 2.0 Flash | 63.9 | modality: visual-only; validation subset, 4-shot; R1@0.5; max 256 frames; AI Studio API model id gemini-2.0-flash-001; developer-reported | 3.3 Video Understanding (Table 6) |
| QVHighlights — visual-only | Gemini 2.5 Flash | 52.4 | modality: visual-only; validation subset, 4-shot; R1@0.5; max 256 frames; AI Studio API model id gemini-2.5-flash; developer-reported | 3.3 Video Understanding (Table 6) |
| QVHighlights — visual-only | Gemini 2.5 Pro | 75.0 | modality: visual-only; validation subset, 4-shot; R1@0.5; max 256 frames; AI Studio API model id gemini-2.5-pro; developer-reported | 3.3 Video Understanding (Table 6) |
| QVHighlights — visual-only | GPT-4.1 | 71.4 | modality: visual-only; validation subset, 4-shot; R1@0.5; max 256 frames; run by Google with the same prompt and video frames; reported by Google in the Gemini 2.5 technical report | 3.3 Video Understanding (Table 6) |
| VideoMMMU — visual-only | Gemini 1.5 Flash | 64.8 | modality: visual-only; test subset, 0-shot; max 256 frames; AI Studio API model id gemini-1.5-flash-002; developer-reported | 3.3 Video Understanding (Table 6) |
| VideoMMMU — visual-only | Gemini 1.5 Pro | 70.4 | modality: visual-only; test subset, 0-shot; max 256 frames; AI Studio API model id gemini-1.5-pro-002; developer-reported | 3.3 Video Understanding (Table 6) |
| VideoMMMU — visual-only | Gemini 2.0 Flash-Lite | 64.3 | modality: visual-only; test subset, 0-shot; max 256 frames; AI Studio API model id gemini-2.0-flash-lite-001; developer-reported | 3.3 Video Understanding (Table 6) |
| VideoMMMU — visual-only | Gemini 2.0 Flash | 68.5 | modality: visual-only; test subset, 0-shot; max 256 frames; AI Studio API model id gemini-2.0-flash-001; developer-reported | 3.3 Video Understanding (Table 6) |
| VideoMMMU — visual-only | Gemini 2.5 Flash | 79.2 | modality: visual-only; test subset, 0-shot; max 256 frames; AI Studio API model id gemini-2.5-flash; developer-reported | 3.3 Video Understanding (Table 6) |
| VideoMMMU — visual-only | Gemini 2.5 Pro | 83.6 | modality: visual-only; test subset, 0-shot; max 256 frames; AI Studio API model id gemini-2.5-pro; developer-reported | 3.3 Video Understanding (Table 6) |
| VideoMMMU — visual-only | GPT-4.1 | 60.9 | modality: visual-only; test subset, 0-shot; max 256 frames; run by Google with the same prompt and video frames; reported by Google in the Gemini 2.5 technical report | 3.3 Video Understanding (Table 6) |
| 1H-VideoQA — visual-only | Gemini 1.5 Flash | 61.9 | modality: visual-only; test subset, 0-shot; max 7200 frames; AI Studio API model id gemini-1.5-flash-002; developer-reported | 3.3 Video Understanding (Table 6) |
| 1H-VideoQA — visual-only | Gemini 1.5 Pro | 72.2 | modality: visual-only; test subset, 0-shot; max 7200 frames; AI Studio API model id gemini-1.5-pro-002; developer-reported | 3.3 Video Understanding (Table 6) |
| 1H-VideoQA — visual-only | Gemini 2.0 Flash-Lite | 55.6 | modality: visual-only; test subset, 0-shot; max 7200 frames; AI Studio API model id gemini-2.0-flash-lite-001; developer-reported | 3.3 Video Understanding (Table 6) |
| 1H-VideoQA — visual-only | Gemini 2.0 Flash | 67.5 | modality: visual-only; test subset, 0-shot; max 7200 frames; AI Studio API model id gemini-2.0-flash-001; developer-reported | 3.3 Video Understanding (Table 6) |
| 1H-VideoQA — visual-only | Gemini 2.5 Flash | 67.5 | modality: visual-only; test subset, 0-shot; max 7200 frames; AI Studio API model id gemini-2.5-flash; developer-reported | 3.3 Video Understanding (Table 6) |
| 1H-VideoQA — visual-only | Gemini 2.5 Pro | 81.0 | modality: visual-only; test subset, 0-shot; max 7200 frames; AI Studio API model id gemini-2.5-pro; developer-reported | 3.3 Video Understanding (Table 6) |
| 1H-VideoQA — visual-only | GPT-4.1 | 56.8 | modality: visual-only; test subset, 0-shot; max 7200 frames; run by Google with the same prompt and video frames; reported by Google in the Gemini 2.5 technical report | 3.3 Video Understanding (Table 6) |
| LVBench — audio + visual | Gemini 1.5 Flash | 61.9 | modality: audio + visual; test subset, 0-shot; max 1024 frames; AI Studio API model id gemini-1.5-flash-002; developer-reported | 3.3 Video Understanding (Table 6) |
| LVBench — audio + visual | Gemini 1.5 Pro | 65.7 | modality: audio + visual; test subset, 0-shot; max 1024 frames; AI Studio API model id gemini-1.5-pro-002; developer-reported | 3.3 Video Understanding (Table 6) |
| LVBench — audio + visual | Gemini 2.0 Flash-Lite | 52 | modality: audio + visual; test subset, 0-shot; max 1024 frames; AI Studio API model id gemini-2.0-flash-lite-001; developer-reported | 3.3 Video Understanding (Table 6) |
| LVBench — audio + visual | Gemini 2.0 Flash | 61.8 | modality: audio + visual; test subset, 0-shot; max 1024 frames; AI Studio API model id gemini-2.0-flash-001; developer-reported | 3.3 Video Understanding (Table 6) |
| LVBench — audio + visual | Gemini 2.5 Flash | 62.7 | modality: audio + visual; test subset, 0-shot; max 1024 frames; AI Studio API model id gemini-2.5-flash; developer-reported | 3.3 Video Understanding (Table 6) |
| LVBench — audio + visual | Gemini 2.5 Pro | 78.7 | modality: audio + visual; test subset, 0-shot; max 1024 frames; AI Studio API model id gemini-2.5-pro; developer-reported | 3.3 Video Understanding (Table 6) |
| LVBench — audio + visual | GPT-4.1 | 63.4 | modality: audio + visual; test subset, 0-shot; max 1024 frames; run by Google with the same prompt and video frames; reported by Google in the Gemini 2.5 technical report | 3.3 Video Understanding (Table 6) |
| VATEX — audio + visual | Gemini 1.5 Flash | 56.9 | modality: audio + visual; test subset, 4-shot; CIDEr; max 64 frames; AI Studio API model id gemini-1.5-flash-002; developer-reported | 3.3 Video Understanding (Table 6) |
| VATEX — audio + visual | Gemini 1.5 Pro | 55.5 | modality: audio + visual; test subset, 4-shot; CIDEr; max 64 frames; AI Studio API model id gemini-1.5-pro-002; developer-reported | 3.3 Video Understanding (Table 6) |
| VATEX — audio + visual | Gemini 2.0 Flash-Lite | 58.5 | modality: audio + visual; test subset, 4-shot; CIDEr; max 64 frames; AI Studio API model id gemini-2.0-flash-lite-001; developer-reported | 3.3 Video Understanding (Table 6) |
| VATEX — audio + visual | Gemini 2.0 Flash | 56.9 | modality: audio + visual; test subset, 4-shot; CIDEr; max 64 frames; AI Studio API model id gemini-2.0-flash-001; developer-reported | 3.3 Video Understanding (Table 6) |
| VATEX — audio + visual | Gemini 2.5 Flash | 65.2 | modality: audio + visual; test subset, 4-shot; CIDEr; max 64 frames; AI Studio API model id gemini-2.5-flash; developer-reported | 3.3 Video Understanding (Table 6) |
| VATEX — audio + visual | Gemini 2.5 Pro | 71.3 | modality: audio + visual; test subset, 4-shot; CIDEr; max 64 frames; AI Studio API model id gemini-2.5-pro; developer-reported | 3.3 Video Understanding (Table 6) |
| VATEX — audio + visual | GPT-4.1 | 64.1 | modality: audio + visual; test subset, 4-shot; CIDEr; max 64 frames; run by Google with the same prompt and video frames; reported by Google in the Gemini 2.5 technical report | 3.3 Video Understanding (Table 6) |
| VATEX-ZH — audio + visual | Gemini 1.5 Flash | 46.2 | modality: audio + visual; validation subset, 4-shot; CIDEr; max 64 frames; AI Studio API model id gemini-1.5-flash-002; developer-reported | 3.3 Video Understanding (Table 6) |
| VATEX-ZH — audio + visual | Gemini 1.5 Pro | 52.2 | modality: audio + visual; validation subset, 4-shot; CIDEr; max 64 frames; AI Studio API model id gemini-1.5-pro-002; developer-reported | 3.3 Video Understanding (Table 6) |
| VATEX-ZH — audio + visual | Gemini 2.0 Flash-Lite | 43.2 | modality: audio + visual; validation subset, 4-shot; CIDEr; max 64 frames; AI Studio API model id gemini-2.0-flash-lite-001; developer-reported | 3.3 Video Understanding (Table 6) |
| VATEX-ZH — audio + visual | Gemini 2.0 Flash | 48.5 | modality: audio + visual; validation subset, 4-shot; CIDEr; max 64 frames; AI Studio API model id gemini-2.0-flash-001; developer-reported | 3.3 Video Understanding (Table 6) |
| VATEX-ZH — audio + visual | Gemini 2.5 Flash | 43.9 | modality: audio + visual; validation subset, 4-shot; CIDEr; max 64 frames; AI Studio API model id gemini-2.5-flash; developer-reported | 3.3 Video Understanding (Table 6) |
| VATEX-ZH — audio + visual | Gemini 2.5 Pro | 59.7 | modality: audio + visual; validation subset, 4-shot; CIDEr; max 64 frames; AI Studio API model id gemini-2.5-pro; developer-reported | 3.3 Video Understanding (Table 6) |
| VATEX-ZH — audio + visual | GPT-4.1 | 48.7 | modality: audio + visual; validation subset, 4-shot; CIDEr; max 64 frames; run by Google with the same prompt and video frames; reported by Google in the Gemini 2.5 technical report | 3.3 Video Understanding (Table 6) |
| YouCook2 Cap — audio + visual | Gemini 1.5 Flash | 153.2 | modality: audio + visual; validation subset, 4-shot; CIDEr; max 256 frames; AI Studio API model id gemini-1.5-flash-002; developer-reported | 3.3 Video Understanding (Table 6) |
| YouCook2 Cap — audio + visual | Gemini 1.5 Pro | 170.0 | modality: audio + visual; validation subset, 4-shot; CIDEr; max 256 frames; AI Studio API model id gemini-1.5-pro-002; developer-reported | 3.3 Video Understanding (Table 6) |
| YouCook2 Cap — audio + visual | Gemini 2.0 Flash-Lite | 78.6 | modality: audio + visual; validation subset, 4-shot; CIDEr; max 256 frames; AI Studio API model id gemini-2.0-flash-lite-001; developer-reported | 3.3 Video Understanding (Table 6) |
| YouCook2 Cap — audio + visual | Gemini 2.0 Flash | 129.0 | modality: audio + visual; validation subset, 4-shot; CIDEr; max 256 frames; AI Studio API model id gemini-2.0-flash-001; developer-reported | 3.3 Video Understanding (Table 6) |
| YouCook2 Cap — audio + visual | Gemini 2.5 Flash | 177.6 | modality: audio + visual; validation subset, 4-shot; CIDEr; max 256 frames; AI Studio API model id gemini-2.5-flash; developer-reported | 3.3 Video Understanding (Table 6) |
| YouCook2 Cap — audio + visual | Gemini 2.5 Pro | 188.3 | modality: audio + visual; validation subset, 4-shot; CIDEr; max 256 frames; AI Studio API model id gemini-2.5-pro; developer-reported | 3.3 Video Understanding (Table 6) |
| YouCook2 Cap — audio + visual | GPT-4.1 | 127.6 | modality: audio + visual; validation subset, 4-shot; CIDEr; max 256 frames; run by Google with the same prompt and video frames; reported by Google in the Gemini 2.5 technical report | 3.3 Video Understanding (Table 6) |
| Minerva — visual + subtitles | Gemini 1.5 Flash | 49.6 | modality: visual + subtitles; test subset, 0-shot; max 1024 frames; AI Studio API model id gemini-1.5-flash-002; developer-reported | 3.3 Video Understanding (Table 6) |
| Minerva — visual + subtitles | Gemini 1.5 Pro | 52.8 | modality: visual + subtitles; test subset, 0-shot; max 1024 frames; AI Studio API model id gemini-1.5-pro-002; developer-reported | 3.3 Video Understanding (Table 6) |
| Minerva — visual + subtitles | Gemini 2.0 Flash-Lite | 46.8 | modality: visual + subtitles; test subset, 0-shot; max 1024 frames; AI Studio API model id gemini-2.0-flash-lite-001; developer-reported | 3.3 Video Understanding (Table 6) |
| Minerva — visual + subtitles | Gemini 2.0 Flash | 52.4 | modality: visual + subtitles; test subset, 0-shot; max 1024 frames; AI Studio API model id gemini-2.0-flash-001; developer-reported | 3.3 Video Understanding (Table 6) |
| Minerva — visual + subtitles | Gemini 2.5 Flash | 60.7 | modality: visual + subtitles; test subset, 0-shot; max 1024 frames; AI Studio API model id gemini-2.5-flash; developer-reported | 3.3 Video Understanding (Table 6) |
| Minerva — visual + subtitles | Gemini 2.5 Pro | 67.6 | modality: visual + subtitles; test subset, 0-shot; max 1024 frames; AI Studio API model id gemini-2.5-pro; developer-reported | 3.3 Video Understanding (Table 6) |
| Minerva — visual + subtitles | GPT-4.1 | 54.0 | modality: visual + subtitles; test subset, 0-shot; max 1024 frames; run by Google with the same prompt and video frames; reported by Google in the Gemini 2.5 technical report | 3.3 Video Understanding (Table 6) |
| Neptune — visual + subtitles | Gemini 1.5 Flash | 78.7 | modality: visual + subtitles; test subset, 0-shot; max 1024 frames; AI Studio API model id gemini-1.5-flash-002; developer-reported | 3.3 Video Understanding (Table 6) |
| Neptune — visual + subtitles | Gemini 1.5 Pro | 82.7 | modality: visual + subtitles; test subset, 0-shot; max 1024 frames; AI Studio API model id gemini-1.5-pro-002; developer-reported | 3.3 Video Understanding (Table 6) |
| Neptune — visual + subtitles | Gemini 2.0 Flash-Lite | 81.5 | modality: visual + subtitles; test subset, 0-shot; max 1024 frames; AI Studio API model id gemini-2.0-flash-lite-001; developer-reported | 3.3 Video Understanding (Table 6) |
| Neptune — visual + subtitles | Gemini 2.0 Flash | 83.1 | modality: visual + subtitles; test subset, 0-shot; max 1024 frames; AI Studio API model id gemini-2.0-flash-001; developer-reported | 3.3 Video Understanding (Table 6) |
| Neptune — visual + subtitles | Gemini 2.5 Flash | 84.3 | modality: visual + subtitles; test subset, 0-shot; max 1024 frames; AI Studio API model id gemini-2.5-flash; developer-reported | 3.3 Video Understanding (Table 6) |
| Neptune — visual + subtitles | Gemini 2.5 Pro | 87.3 | modality: visual + subtitles; test subset, 0-shot; max 1024 frames; AI Studio API model id gemini-2.5-pro; developer-reported | 3.3 Video Understanding (Table 6) |
| Neptune — visual + subtitles | GPT-4.1 | 85.2 | modality: visual + subtitles; test subset, 0-shot; max 1024 frames; run by Google with the same prompt and video frames; reported by Google in the Gemini 2.5 technical report | 3.3 Video Understanding (Table 6) |
| EN text-to-text Policy Violations — Change vs. Gemini 1.5 Flash 002 | Gemini 2.0 Flash-Lite | ↓14.3% | automated internal evaluation, change relative to Gemini 1.5 Flash 002; down arrow = fewer violations (better); evaluation recently updated, not comparable with past reports; developer-reported | 5.4 (Table 7) |
| i18n text-to-text Policy Violations — Change vs. Gemini 1.5 Flash 002 | Gemini 2.0 Flash-Lite | ↓7.3% | automated internal evaluation, change relative to Gemini 1.5 Flash 002; down arrow = fewer violations (better); evaluation recently updated, not comparable with past reports; developer-reported | 5.4 (Table 7) |
| Image-to-text Policy Violations — Change vs. Gemini 1.5 Flash 002 | Gemini 2.0 Flash-Lite | ↑4.6% | automated internal evaluation, change relative to Gemini 1.5 Flash 002; up arrow = more violations; report marks these '*No egregious losses reported'; marked * (no egregious losses reported); developer-reported | 5.4 (Table 7) |
| Tone — Change vs. Gemini 1.5 Flash 002 | Gemini 2.0 Flash-Lite | ↑8.4% | automated internal evaluation, change relative to Gemini 1.5 Flash 002; up arrow = improvement; developer-reported | 5.4 (Table 7) |
| Helpfulness / Instruction Following — Change vs. Gemini 1.5 Flash 002 | Gemini 2.0 Flash-Lite | ↓19.7% | automated internal evaluation, change relative to Gemini 1.5 Flash 002; up arrow = improvement; developer-reported | 5.4 (Table 7) |
| EN text-to-text Policy Violations — Change vs. Gemini 1.5 Flash 002 | Gemini 2.0 Flash | ↓12.7% | automated internal evaluation, change relative to Gemini 1.5 Flash 002; down arrow = fewer violations (better); evaluation recently updated, not comparable with past reports; developer-reported | 5.4 (Table 7) |
| i18n text-to-text Policy Violations — Change vs. Gemini 1.5 Flash 002 | Gemini 2.0 Flash | ↓7.8% | automated internal evaluation, change relative to Gemini 1.5 Flash 002; down arrow = fewer violations (better); evaluation recently updated, not comparable with past reports; developer-reported | 5.4 (Table 7) |
| Image-to-text Policy Violations — Change vs. Gemini 1.5 Flash 002 | Gemini 2.0 Flash | ↑5.2% | automated internal evaluation, change relative to Gemini 1.5 Flash 002; up arrow = more violations; report marks these '*No egregious losses reported'; marked * (no egregious losses reported); developer-reported | 5.4 (Table 7) |
| Tone — Change vs. Gemini 1.5 Flash 002 | Gemini 2.0 Flash | ↑1.5% | automated internal evaluation, change relative to Gemini 1.5 Flash 002; up arrow = improvement; developer-reported | 5.4 (Table 7) |
| Helpfulness / Instruction Following — Change vs. Gemini 1.5 Flash 002 | Gemini 2.0 Flash | ↓13.2% | automated internal evaluation, change relative to Gemini 1.5 Flash 002; up arrow = improvement; developer-reported | 5.4 (Table 7) |
| EN text-to-text Policy Violations — Change vs. Gemini 1.5 Flash 002 | Gemini 2.5 Flash | ↓8.2% | automated internal evaluation, change relative to Gemini 1.5 Flash 002; down arrow = fewer violations (better); evaluation recently updated, not comparable with past reports; developer-reported | 5.4 (Table 7) |
| i18n text-to-text Policy Violations — Change vs. Gemini 1.5 Flash 002 | Gemini 2.5 Flash | ↑1.1% | automated internal evaluation, change relative to Gemini 1.5 Flash 002; down arrow = fewer violations (better); evaluation recently updated, not comparable with past reports; marked * (no egregious losses reported); developer-reported | 5.4 (Table 7) |
| Image-to-text Policy Violations — Change vs. Gemini 1.5 Flash 002 | Gemini 2.5 Flash | ↑6.4% | automated internal evaluation, change relative to Gemini 1.5 Flash 002; up arrow = more violations; report marks these '*No egregious losses reported'; marked * (no egregious losses reported); developer-reported | 5.4 (Table 7) |
| Tone — Change vs. Gemini 1.5 Flash 002 | Gemini 2.5 Flash | ↑7.9% | automated internal evaluation, change relative to Gemini 1.5 Flash 002; up arrow = improvement; developer-reported | 5.4 (Table 7) |
| Helpfulness / Instruction Following — Change vs. Gemini 1.5 Flash 002 | Gemini 2.5 Flash | ↑13.6% | automated internal evaluation, change relative to Gemini 1.5 Flash 002; up arrow = improvement; developer-reported | 5.4 (Table 7) |
| EN text-to-text Policy Violations — Change vs. Gemini 1.5 Pro 002 | Gemini 2.5 Pro | ↓0.9% | automated internal evaluation, change relative to Gemini 1.5 Pro 002; down arrow = fewer violations (better); evaluation recently updated, not comparable with past reports; developer-reported | 5.4 (Table 7) |
| i18n text-to-text Policy Violations — Change vs. Gemini 1.5 Pro 002 | Gemini 2.5 Pro | ↓3.5% | automated internal evaluation, change relative to Gemini 1.5 Pro 002; down arrow = fewer violations (better); evaluation recently updated, not comparable with past reports; developer-reported | 5.4 (Table 7) |
| Image-to-text Policy Violations — Change vs. Gemini 1.5 Pro 002 | Gemini 2.5 Pro | ↑1.8% | automated internal evaluation, change relative to Gemini 1.5 Pro 002; up arrow = more violations; report marks these '*No egregious losses reported'; marked * (no egregious losses reported); developer-reported | 5.4 (Table 7) |
| Tone — Change vs. Gemini 1.5 Pro 002 | Gemini 2.5 Pro | ↑18.4% | automated internal evaluation, change relative to Gemini 1.5 Pro 002; up arrow = improvement; developer-reported | 5.4 (Table 7) |
| Helpfulness / Instruction Following — Change vs. Gemini 1.5 Pro 002 | Gemini 2.5 Pro | ↑14.8% | automated internal evaluation, change relative to Gemini 1.5 Pro 002; up arrow = improvement; developer-reported | 5.4 (Table 7) |
| Automated Red Teaming (ART) — Dangerous Content policy violations | Gemini 1.5 Flash 002 | 38.3% | violations discovered by automated red teaming (Gemini attackers and judges, black-box); lower is better; developer-reported | 5.5 For Safety (Table 8) |
| Automated Red Teaming (ART) — Helpfulness violations | Gemini 1.5 Flash 002 | 9.5% | violations discovered by automated red teaming; lower is better; developer-reported | 5.5 For Safety (Table 8) |
| Automated Red Teaming (ART) — Dangerous Content policy violations | Gemini 1.5 Pro 002 | 43.5% | violations discovered by automated red teaming (Gemini attackers and judges, black-box); lower is better; developer-reported | 5.5 For Safety (Table 8) |
| Automated Red Teaming (ART) — Helpfulness violations | Gemini 1.5 Pro 002 | 8.9% | violations discovered by automated red teaming; lower is better; developer-reported | 5.5 For Safety (Table 8) |
| Automated Red Teaming (ART) — Dangerous Content policy violations | Gemini 2.0 Flash | 25.2% | violations discovered by automated red teaming (Gemini attackers and judges, black-box); lower is better; developer-reported | 5.5 For Safety (Table 8) |
| Automated Red Teaming (ART) — Helpfulness violations | Gemini 2.0 Flash | 8.1% | violations discovered by automated red teaming; lower is better; developer-reported | 5.5 For Safety (Table 8) |
| Automated Red Teaming (ART) — Dangerous Content policy violations | Gemini 2.5 Flash | 26.9% | violations discovered by automated red teaming (Gemini attackers and judges, black-box); lower is better; developer-reported | 5.5 For Safety (Table 8) |
| Automated Red Teaming (ART) — Helpfulness violations | Gemini 2.5 Flash | 6.6% | violations discovered by automated red teaming; lower is better; developer-reported | 5.5 For Safety (Table 8) |
| Automated Red Teaming (ART) — Dangerous Content policy violations | Gemini 2.5 Pro | 24.3% | violations discovered by automated red teaming (Gemini attackers and judges, black-box); lower is better; developer-reported | 5.5 For Safety (Table 8) |
| Automated Red Teaming (ART) — Helpfulness violations | Gemini 2.5 Pro | 6.1% | violations discovered by automated red teaming; lower is better; developer-reported | 5.5 For Safety (Table 8) |
| Indirect prompt injection attack success rate — Actor Critic | Gemini 2.0 Flash-Lite | 52.0% | best attack success rate over 500 held-out scenarios in which the injection exfiltrated synthetic private data (passport numbers); lower is better; change vs. Gemini 1.5 Flash 002: ↓44.2%; developer-reported | 5.5 For Security (Table 9) |
| Indirect prompt injection attack success rate — Beam Search | Gemini 2.0 Flash-Lite | 75.4% | best attack success rate over 500 held-out scenarios in which the injection exfiltrated synthetic private data (passport numbers); lower is better; change vs. Gemini 1.5 Flash 002: ↓9.0%; developer-reported | 5.5 For Security (Table 9) |
| Indirect prompt injection attack success rate — Tree of Attacks w/ Pruning (TAP) | Gemini 2.0 Flash-Lite | 64.8% | best attack success rate over 500 held-out scenarios in which the injection exfiltrated synthetic private data (passport numbers); lower is better; change vs. Gemini 1.5 Flash 002: ↓17.4%; developer-reported | 5.5 For Security (Table 9) |
| Indirect prompt injection attack success rate — Actor Critic | Gemini 2.0 Flash | 68.0% | best attack success rate over 500 held-out scenarios in which the injection exfiltrated synthetic private data (passport numbers); lower is better; change vs. Gemini 1.5 Flash 002: ↓28.2%; developer-reported | 5.5 For Security (Table 9) |
| Indirect prompt injection attack success rate — Beam Search | Gemini 2.0 Flash | 67.2% | best attack success rate over 500 held-out scenarios in which the injection exfiltrated synthetic private data (passport numbers); lower is better; change vs. Gemini 1.5 Flash 002: ↓17.2%; developer-reported | 5.5 For Security (Table 9) |
| Indirect prompt injection attack success rate — Tree of Attacks w/ Pruning (TAP) | Gemini 2.0 Flash | 98.4% | best attack success rate over 500 held-out scenarios in which the injection exfiltrated synthetic private data (passport numbers); lower is better; change vs. Gemini 1.5 Flash 002: ↑16.2%; developer-reported | 5.5 For Security (Table 9) |
| Indirect prompt injection attack success rate — Actor Critic | Gemini 2.5 Flash | 40.8% | best attack success rate over 500 held-out scenarios in which the injection exfiltrated synthetic private data (passport numbers); lower is better; change vs. Gemini 1.5 Flash 002: ↓55.4%; developer-reported | 5.5 For Security (Table 9) |
| Indirect prompt injection attack success rate — Beam Search | Gemini 2.5 Flash | 4.2% | best attack success rate over 500 held-out scenarios in which the injection exfiltrated synthetic private data (passport numbers); lower is better; change vs. Gemini 1.5 Flash 002: ↓80.2%; developer-reported | 5.5 For Security (Table 9) |
| Indirect prompt injection attack success rate — Tree of Attacks w/ Pruning (TAP) | Gemini 2.5 Flash | 53.6% | best attack success rate over 500 held-out scenarios in which the injection exfiltrated synthetic private data (passport numbers); lower is better; change vs. Gemini 1.5 Flash 002: ↓28.6%; developer-reported | 5.5 For Security (Table 9) |
| Indirect prompt injection attack success rate — Actor Critic | Gemini 2.5 Pro | 61.4% | best attack success rate over 500 held-out scenarios in which the injection exfiltrated synthetic private data (passport numbers); lower is better; change vs. Gemini 1.5 Pro 002: ↓36.8%; developer-reported | 5.5 For Security (Table 9) |
| Indirect prompt injection attack success rate — Beam Search | Gemini 2.5 Pro | 63.8% | best attack success rate over 500 held-out scenarios in which the injection exfiltrated synthetic private data (passport numbers); lower is better; change vs. Gemini 1.5 Pro 002: ↓35.6%; developer-reported | 5.5 For Security (Table 9) |
| Indirect prompt injection attack success rate — Tree of Attacks w/ Pruning (TAP) | Gemini 2.5 Pro | 30.8% | best attack success rate over 500 held-out scenarios in which the injection exfiltrated synthetic private data (passport numbers); lower is better; change vs. Gemini 1.5 Pro 002: ↓57.0%; developer-reported | 5.5 For Security (Table 9) |
| Divergence attack (Nasr et al. 2023) — Divergence rate | Gemini 2.0 Flash and Flash-Lite | 69% | 'roughly'; 3750 prompts split across 125 single-token characters; developer-reported | 5.6 Extractable Memorization and Divergence |
| Divergence attack (Nasr et al. 2023) — Divergence rate | Gemini 2.5 model family | 59% | 'roughly'; 3750 prompts split across 125 single-token characters; developer-reported | 5.6 Extractable Memorization and Divergence |
| Divergence attack (Nasr et al. 2023) — Emitted text classified as memorization, given divergence | Gemini 2.X model family | 0.2% | 'roughly'; developer-reported | 5.6 Extractable Memorization and Divergence |
| Autonomous cyber offense suite — Easy (InterCode-CTF) | Gemini 2.5 Pro | 74/76 | challenges solved; solved if at least one of N attempts captures the flag (N between 5 and 30); Bash and Python tools; Gemini 2.5 Pro up to version 06-05; Cyber Autonomy Level 1 CCL not reached; developer-reported | 5.7 (Table 10; Cybersecurity) |
| Autonomous cyber offense suite — Medium (in-house CTFs) | Gemini 2.5 Pro | 11/13 | challenges solved; solved if at least one of N attempts captures the flag (N between 5 and 30); Bash and Python tools; Gemini 2.5 Pro up to version 06-05; Cyber Autonomy Level 1 CCL not reached; developer-reported | 5.7 (Table 10; Cybersecurity) |
| Autonomous cyber offense suite — Hard (Hack the Box) | Gemini 2.5 Pro | 1/13 | challenges solved; solved if at least one of N attempts captures the flag (N between 5 and 30); Bash and Python tools; Gemini 2.5 Pro up to version 06-05; Cyber Autonomy Level 1 CCL not reached; developer-reported | 5.7 (Table 10; Cybersecurity) |
| Cybersecurity key skills benchmark — Easy | Gemini 2.5 Pro | 7/8 | challenges solved of 48 mapped from an external vendor; solved if at least one attempt captures the flag (10–50 attempts); Gemini 2.5 Pro up to version 06-05; alert threshold reached, Cyber Uplift Level 1 CCL not reached; developer-reported | 5.7 (Table 10; Cybersecurity) |
| Cybersecurity key skills benchmark — Medium | Gemini 2.5 Pro | 14/28 | challenges solved of 48 mapped from an external vendor; solved if at least one attempt captures the flag (10–50 attempts); Gemini 2.5 Pro up to version 06-05; alert threshold reached, Cyber Uplift Level 1 CCL not reached; developer-reported | 5.7 (Table 10; Cybersecurity) |
| Cybersecurity key skills benchmark — Hard | Gemini 2.5 Pro | 6/12 | challenges solved of 48 mapped from an external vendor; solved if at least one attempt captures the flag (10–50 attempts); Gemini 2.5 Pro up to version 06-05; alert threshold reached, Cyber Uplift Level 1 CCL not reached; developer-reported | 5.7 (Table 10; Cybersecurity) |
| Stealth challenges (Phuong et al. 2025) | Gemini 2.5 Pro | 2/5 | challenges solved; Gemini 2.5 Pro Preview 03-25, 50 retries, scaffold with chain-of-thought and reflection; developer-reported | 5.7 (Table 10; Deceptive Alignment) |
| Situational awareness challenges (Phuong et al. 2025) | Gemini 2.5 Pro | 2/11 | challenges solved; Gemini 2.5 Pro Preview 03-25, 50 retries; developer-reported | 5.7 (Table 10; Deceptive Alignment) |
| Situational awareness challenges (Phuong et al. 2025) — Disabled Tool | Gemini 2.5 Pro | 42% | success rate across 50 trials; highest of the 11 situational-awareness evaluations; no successes on 8 of 11; developer-reported | 5.7 Deceptive Alignment |
| Stealth challenges (Phuong et al. 2025) — Decision Sabotage | Gemini 2.5 Pro | ~44% | success rate; developer-reported | 5.7 Deceptive Alignment |
| Stealth challenges (Phuong et al. 2025) — Oversight Pattern | Gemini 2.5 Pro | ~35% | success rate; report says this 'is similar to human performance'; developer-reported | 5.7 Deceptive Alignment |
Provenance
The page summarises version 6 of the report as rendered by arXiv's HTML service on September 19, 2026. Figures are images and their values are not reproduced. The report's cybersecurity counts and its LiveCodeBench figure for Gemini 2.5 Pro differ from those in the June 27, 2025 model card, which describes different model versions and dates. All results are reported by Google; comparison figures for other developers' models are mostly those developers' own numbers or public leaderboards. Confidence is medium.