Gemma 3 Technical Report (Google DeepMind, March 2025)

The Gemma 3 Technical Report is a paper by the Gemma Team at Google DeepMind, posted to arXiv on March 25, 2025 (arXiv 2503.19786), accompanying the release of the third generation of the Gemma open-weight model family (Source: https://arxiv.org/abs/2503.19786). It describes four models of 1, 4, 12 and 27 billion parameters, released in both pre-trained (PT) and instruction-tuned (IT) form, and states that the version adds vision understanding, wider language coverage and a context window of at least 128K tokens. The report gives architecture, training and evaluation detail, an audit of training-data memorization, and a section on responsibility and safety that reports no numeric results.

Models and architecture

The report states that the models are co-designed with the Gemini family, target consumer hardware such as phones, laptops and high-end GPUs, and use the same tokenizer as Gemini 2.0. All Gemma 3 models are trained with knowledge distillation. The 4B, 12B and 27B models pair the language model with a tailored SigLIP vision encoder that produces a fixed 256 soft tokens per image; a "Pan and Scan" (P&S) method, adapted from LLaVA, allows flexible resolutions at inference. The context window is raised to 128K tokens (32K for the 1B model), and to limit KV-cache memory the architecture interleaves five local attention layers (span 1,024 tokens) for every global layer. The report also describes quantization-aware-training checkpoints and the TPU training infrastructure. Post-training uses an updated recipe that the authors say improves mathematics, chat, instruction-following and multilingual ability; the abstract states that Gemma3-4B-IT is "competitive with Gemma2-27B-IT and Gemma3-27B-IT comparable to Gemini-1.5-Pro across benchmarks".

Evaluation approach

Section 4 evaluates the IT models on LMSYS Chatbot Arena and on a set of zero-shot benchmarks. The report says it does not compare directly with external models "that often report their own evaluation settings", since running them under Gemma's setting "does not guarantee a fair comparison", and it directs readers to third-party leaderboards; the comparison columns in its headline table are therefore Google's own Gemini 1.5 and 2.0 models and the Gemma 2 models, run in Google's setting. Appendix tables give pre-trained results on factuality and common-sense, STEM and code, multimodal, multilingual and long-context benchmarks, and IT results on further text, multimodal and video benchmarks. Tables 19–21 record the per-benchmark protocol (shots, chain-of-thought, normalisation, metric).

Findings

On Chatbot Arena, the report places Gemma 3 27B IT at 1338 Elo, which it describes as among the top ten models at the time and above other non-thinking open models such as DeepSeek-V3 (1318), Llama 3 405B (1257) and Qwen2.5-70B (1257); Gemma 2 27B IT is listed at 1220. The Arena table has no snapshot date. On the headline zero-shot table (Table 6), Gemma 3 27B IT scores 67.5 on MMLU-Pro, 29.7 on LiveCodeBench, 42.4 on GPQA Diamond, 89.0 on MATH and 64.9 on MMMU (val); the report presents these as improvements over Gemma 2 and, for the larger sizes, in the range of Gemini 1.5 Flash and Pro. Appendix tables report the 27B IT model at 76.9 on MMLU, 87.8 on HumanEval, 95.9 on GSM8K and 90.4 on IFEval, and give RULER and MRCR long-context scores at 32K and 128K for PT and IT checkpoints.

The ablation section (Section 5) examines the local-to-global attention ratio and sliding-window span, the effect of enabling 128K context, distillation from small versus large teachers, and vision-encoder resolution and P&S; most of these results are shown in figures rather than as numbers.

Memorization and privacy

Section 6 measures exact and approximate memorization of training data following the Gemma 2 methodology and reports, in a figure, that Gemma 3 models memorize long-form text "at a much lower rate than prior models", with only marginal differences between the 4B, 12B and 27B sizes and the 1B memorizing less. Using the Google Cloud Sensitive Data Protection service, which the report says has high recall and many false positives, it observed no personal information in the outputs it classified as memorized.

Responsibility, safety and security

Section 7 describes governance and assessment, safety policies (child sexual abuse material, harmful personal-information disclosure, hate speech and harassment, dangerous content, sexually explicit content, and medical advice contrary to consensus), train-time mitigations by pre-training data filtering and SFT/RLHF, and assurance evaluations. The baseline assurance evaluation uses synthetic adversarial queries rated by humans for policy violation; the report states that the violation rate is "significantly low overall" without giving a figure. Because of stronger STEM performance, the authors evaluated chemical, biological, radiological and nuclear knowledge with internal closed-ended multiple-choice questions and a chemical-hazard set, and state that the models' knowledge in these domains "is low". The report says in-depth dangerous-capability assessments were reserved for models presenting a potentially heightened risk, under the heuristic that evaluating a more capable model provides assurance for less capable ones, while upholding the commitments of Google's Frontier Safety Framework. No CBRN, cyber or autonomy scores are reported.

Reported results

Evaluation results the document states for which there is no separate benchmark page; results on benchmarks that have a page are recorded there and cite this page. Scores and settings are as printed in the document.

Provenance

The page summarises the arXiv HTML version (v1, 25 March 2025) as captured on September 21, 2026. All results are reported by Google DeepMind; the Gemini comparison figures are Google's own runs under Gemma's evaluation setting, not those models' published numbers. Confidence is medium: a single developer-authored source with fast-decay benchmark claims.

Relationships

• supports: Gemma (Google open-weight models)
• instance-of: AI System Cards
• related: Google DeepMind, Gemma 4 Technical Report (Google DeepMind, July 2026), LMArena (Chatbot Arena), MMLU-Pro, GPQA, MRCR