The DeepSeek-V3 Technical Report is the primary technical paper for DeepSeek-V3, a Mixture-of-Experts (MoE) language model with 671B total parameters and 37B activated per token, pre-trained on 14.8T tokens. It was authored by DeepSeek-AI (200+ authors; lead names include Aixin Liu, Bei Feng, and Bing Xue) and published as arXiv 2412.19437 on December 27, 2024, with a v2 revision on February 18, 2025. The report states that pre-training required 2.788M H800 GPU hours with no irrecoverable loss spikes, an efficiency result that figured in the early-2025 reassessment of the United States compute advantage.
Summary
The paper describes the architecture, training methods, and reported performance of DeepSeek-V3. It carries forward Multi-head Latent Attention (MLA) and the DeepSeekMoE architecture, both validated in the earlier V2, and introduces an auxiliary-loss-free load-balancing strategy and a multi-token prediction training objective. On the systems side, it documents FP8 mixed-precision training and the DualPipe algorithm for reducing pipeline-parallelism communication overhead. The paper also describes distilling reasoning capability from the long chain-of-thought DeepSeek-R1 series into the V3 base model, the direct precursor relationship to DeepSeek-R1. The paper states that V3 "outperforms other open-source models and achieves performance comparable to leading closed-source models."
As the primary technical source for DeepSeek-V3 — previously sourced only from secondary coverage ((Source: epochai.substack.com), China and the US Are Running Different AI Races) — the report documents the architectural levers behind the model's reported training efficiency: MLA, MoE sparsity, FP8 mixed precision, and DualPipe. It also provides direct evidence for distillation as a training strategy: the R1-to-V3 transfer is distillation within the DeepSeek lab, distinct from the Anthropic/Claude controversy.
Key claims
- Architecture: MLA and the DeepSeekMoE architecture, both validated in V2; new in V3 are the auxiliary-loss-free load-balancing strategy and the multi-token prediction training objective. (high)
- FP8 mixed-precision training at this scale is described as a first for a model of V3's size. (high)
- DualPipe reduces pipeline-parallelism communication overhead. (high)
- DeepSeek-V3 distills reasoning capability from the long chain-of-thought DeepSeek-R1 series into the base model, the direct precursor relationship to DeepSeek-R1. (high)
- V3 "outperforms other open-source models and achieves performance comparable to leading closed-source models." (high, per the paper)
Reception and political context
A March 2025 CSIS report by Gregory Allen (Allen's CSIS report) reads the V3 paper in the context of export-control policy. Allen argues that DeepSeek's algorithmic and architectural innovations — MoE scaling, MLA, FP8, and DualPipe — are genuinely novel and validated by Western replication, rather than propaganda or artifacts of undisclosed GPU stockpiles. Allen also argues that the $5.576M V3 training cost (2.788M H800 GPU-hours × $2/hour) widely quoted from this paper does not overturn the compute-gap account, citing three points:
- DeepSeek CEO Liang Wenfeng describes export controls as "the greatest challenge," with Chinese firms needing "two to four times" the compute of unrestricted access.
- DeepSeek/High-Flyer's total GPU server capex is estimated at $1.63B, against which the V3 training cost is marginal rather than a $6M bottom line.
- Citing the Jevons Paradox, Allen notes that after V3 and R1, major U.S. firms announced roughly 50% capex increases, an argument that efficiency gains raise rather than lower chip demand.
Allen characterizes the V3 paper as evidence of constraint-driven efficiency within a binding constraint, not as proof the constraint failed.
Relationships
- supports: DeepSeek-V3, DeepSeek, Distillation, Fast-Follow Problem, China and the US Are Running Different AI Races
- related: Scaling Laws, AI Software Progress, Compute Governance, CSIS — DeepSeek, Huawei, Export Controls, and the Future of the U.S.-China AI Race (Allen, March 2025)
- depends-on: Distillation