Nemotron 3 Ultra Technical Report (NVIDIA, June 2026)

Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning is the technical report NVIDIA published for the largest model in its Nemotron 3 family, dated 9 June 2026 on its first page (Source: research.nvidia.com). The report describes a 550 billion total and 55 billion active parameter Mixture-of-Experts hybrid Mamba-Attention language model, pre-trained in NVFP4 on 20 trillion text tokens, extended to a 1M-token context, and post-trained with supervised fine-tuning, reinforcement learning and a method the authors call Multi-teacher On-Policy Distillation (MOPD). NVIDIA states that the base, post-trained and quantized checkpoints are released on Hugging Face together with the training data and recipe. The report contains no safety, dangerous-capability or alignment evaluations; its evaluation chapters cover base-model benchmarks, post-trained benchmarks against open-weight peers, quantization fidelity and inference throughput.

Model and training

The report presents Nemotron 3 Ultra as the largest and most capable model in the Nemotron 3 family, building on the family white paper (Blakeman et al., 2025). Its stated design goal is a better inference-throughput-to-accuracy frontier for long-running agents: the Mixture-of-Experts layers are credited with accuracy per active parameter, and the hybrid Mamba-Attention layout with reduced attention cost and KV-cache footprint. The authors report that Nemotron 3 Ultra achieves 5.9×, 4.8× and 1.6× the inference throughput of GLM-5.1-754B-A40B, Kimi-K2.6-1T-A32B and Qwen-3.5-397B-17B respectively in an 8K-input / 64K-output setting, at what they describe as on-par accuracy (§1). The named technologies are LatentMoE, Multi Token Prediction (MTP), NVFP4 pre-training, multi-environment reinforcement learning with verifiable rewards, MOPD, and reasoning budget control.

Pre-training used a Warmup-Stable-Decay learning-rate schedule in two phases: 15 trillion tokens aimed at diversity and breadth, then 5 trillion tokens of higher-quality data (§1, §2). The pre-training chapter documents new data sets (a code refresh, multiple-choice and generative synthetic sets, fact-seeking, moral-scenario and legal data), the data mixture and ordering, hyperparameters, long-context extension and a section on training stability (§2.3–§2.7). Post-training (§3) covers SFT data and packing, RL, the MOPD algorithm with specialised teacher models, MTP boosting, reasoning-efficiency control and the RL infrastructure. A quantization chapter (§4) describes bits-per-element selection, FP4 algorithm experiments, the final weight and GEMM recipe, SSM-cache optimisation and a single NVFP4 checkpoint; an inference chapter (§5) reports throughput across serving regimes.

Evaluation approach

Base-model results (§2.6, Table 2) were collected with the Nemo Evaluator SDK and NVIDIA's container of the LM Evaluation Harness, with the report's comparison set drawn from publicly available base checkpoints: DeepSeek-V3.2-Exp-Base, Mistral-Large-3-675B-Base-2512, Kimi-K2-Base and GLM-4.5-Base. The table spans general knowledge (MMLU, MMLU-Pro, AGIEval-En, GPQA), mathematics (GSM8K, MATH), code (HumanEval, MBPP-Sanitized), commonsense, reading comprehension, multilingual (MMLU Global Lite, MGSM) and RULER at 64K to 1M tokens, each with its shot count and metric printed. The authors state that the base model "achieves significantly higher accuracy than other publicly available base models" (§1).

Post-trained results (§3.7, Table 10) use three harnesses — Nemo Gym, Nemo Skills and Harbor — under the Nemo Evaluator SDK, with BrowseComp, Tau Bench 3, ProfBench, PinchBench, Vals.ai Financial Agent and LongBench v2 run through official implementations or internal scaffolding. The report states that all models, baselines included, were evaluated under the same settings (agentic resources, input data, prompt templates, number of repeats and metric implementation), with inference parameters taken from each model's own card; the baseline figures are therefore NVIDIA's own runs rather than numbers quoted from other developers. The suite spans agentic tasks (Terminal-Bench 2.1, GDPVal, SWE-Bench Verified and Multilingual, TauBench V3, BrowseComp, Vals.ai Financial Agent, PinchBench, ProfBench), reasoning and knowledge (GPQA Diamond, HLE, MMLU-Pro, LiveCodeBench v6, IOI 2025, IMOAnswerBench, MathArena Apex-Shortlist, SciCode, CritPt), instruction following and conversation (IFBench, Multi-Challenge), long context (RULER 1M, AA-LCR, LongBench v2), reliability (AA-Omniscience) and multilingual tasks (MMLU-ProX, WMT24++). Table 18 in the quantization chapter repeats a subset of Table 10 for the NVFP4 checkpoint against BF16.

The authors describe PinchBench and ProfBench as held-out generalisation gates: not used for training-time monitoring or checkpoint selection, and evaluated once after the final model was produced (§3.7.2). A separate section (§3.7.3, Table 11) reports Olympiad-level competition mathematics under a high-compute generate–verify–refine test-time-scaling pipeline following Shao et al. (2025) and Nemotron-Cascade-2, starting from 128 proof attempts per problem with a 512K context, graded by human experts except for USAMO 2026.

Findings as stated by the authors

The report characterises Nemotron 3 Ultra as "a strong, agentic-first, and well-rounded post-trained model" that "remains competitive with leading open models, including several with substantially larger total parameter counts" (§3.7.2). Among the figures it foregrounds in prose: 90.0 on PinchBench, described as within 1.3 points of the best model in the comparison; 56.0 on ProfBench, tying Kimi-K2.6; a score of 570.0 on IOI 2025, which the authors place between the second- and third-ranked human contestants on the official scoreboard; 92.3 on IMOAnswerBench with tools; and 78.7 non-hallucination on AA-Omniscience, which the report calls the highest in its comparison. Under test-time scaling, Table 11 reports 82.3% (173/210) on IMO-ProofBench Advanced, 83.3% (35/42) on IMO 2025, 96.7% (116/120) on Putnam 2025 and 97.6% (41/42) on USAMO 2026, with Figure 13 plotting the IMO-ProofBench score by refinement round against the Aletheia agent's 91.9%.

The conclusion (§6) restates the throughput claim as roughly 5× that of other state-of-the-art open LLMs at on-par accuracy, and the open release of pre-trained, post-trained and quantized checkpoints with training data. The MOPD section includes its own limitations and open problems (§3.3.5).

Provenance and confidence

The raw file is the text layer of NVIDIA's PDF; figure values and some multi-column table layouts are flattened in the capture, and Table 10 prints no units. All results, for baselines as well as for Nemotron 3 Ultra, are NVIDIA's own measurements under its own harness and have not been independently replicated; third-party comparisons are recorded at NVIDIA Nemotron. Confidence is medium: a single developer-authored source, with fast-decay benchmark claims. The report is dated 2026-6-9 on its first page, while NVIDIA's research site gives 4 June 2026 as the model's publication date.

Reported results

Evaluation results the document states for which there is no separate benchmark page; results on benchmarks that have a page are recorded there and cite this page. Scores and settings are as printed in the document.

Relationships

• supports: NVIDIA Nemotron (primary reference for Nemotron 3 Ultra's architecture, training and evaluation)
• related: Kimi K2, GLM-5 and GLM-5.1, DeepSeek V4 Pro / V4 Flash, Qwen3 (comparison baselines in Tables 2 and 10), Terminal-Bench, SWE-bench, τ-bench, GDPval, Humanity's Last Exam
• instance-of: Open-Weight Frontier Models