DeepSeek-V3 is an open-weights Mixture-of-Experts (MoE) language model released by DeepSeek in December 2024. It has 671 billion total parameters but activates only 37 billion per token, and was trained for 2.788M H800 GPU-hours. It is frequently cited as an example of competitive model performance achieved on a fraction of the compute used by US frontier models.
| Field | Value | |
|---|---|---|
| Developer | [[deepseek-company | DeepSeek]] |
| Released | December 2024 | |
| Architecture | Mixture-of-Experts (MoE) with MLA | |
| Total parameters | 671 billion | |
| Active per token | 37 billion | |
| Training tokens | 14.8T | |
| Training compute | 2.788M H800 GPU-hours | |
| Key innovations | MLA, auxiliary-loss-free load balancing, multi-token prediction, FP8 mixed-precision, DualPipe |
Capabilities and efficiency
DeepSeek-V3 was designed to be competitive while using a fraction of the compute of US frontier models. Its MoE architecture activates only 37B of 671B parameters per token, reducing inference costs. The primary technical source is DeepSeek-V3 Technical Report (arXiv 2412.19437), which reports full pre-training in 2.788M H800 GPU-hours with no irrecoverable loss spikes or rollbacks, a training-stability result that competing US labs have not publicly matched at equivalent scale (Source: DeepSeek-V3 Technical Report, China and the US Are Running Different AI Races). The technical report prices the full training run at $5.576 million, valuing the 2.788M GPU-hours at $2 per GPU-hour, a figure it notes covers only the official training and excludes prior research and ablation experiments (Source: DeepSeek-V3 Technical Report).
In the framing of US-China AI competition, DeepSeek-V3 is presented as embodying a Chinese strategy under which efficiency expands the addressable market when capital is scarce and customers are price-sensitive. The model is also cited in discussions of how software progress is measured: efficiency gains may reflect data-quality improvements and scale-dependent innovations more than raw algorithmic breakthroughs (Source: The Least Understood Driver of AI Progress). It is referenced in this context alongside AI Software Progress and the broader State of AI Report 2025.
Training and architecture
The model was trained on 14.8 trillion tokens. Its architectural and training innovations, drawn primarily from the technical report, include:
- Multi-head Latent Attention (MLA): A more efficient attention variant, validated in DeepSeek-V2 and retained for V3 (Source: DeepSeek-V3 Technical Report, epochai.substack.com).
- Auxiliary-loss-free load balancing: An MoE load-balancing strategy that avoids the performance penalty of auxiliary losses (Source: DeepSeek-V3 Technical Report).
- Multi-token prediction objective: Adds a dense training signal beyond next-token prediction (Source: DeepSeek-V3 Technical Report).
- FP8 mixed-precision training: First documented use of FP8 at this scale for a language model (Source: DeepSeek-V3 Technical Report).
- DualPipe algorithm: A pipeline-parallelism scheme that reduces communication overhead and improves device utilization (Source: DeepSeek-V3 Technical Report).
- GRPO: A reinforcement-learning approach that Epoch AI characterized as showing "great research taste" (Source: epochai.substack.com).
Reasoning distillation
DeepSeek-V3 distills reasoning capability from the long-CoT DeepSeek-R1 series into the base model via supervised fine-tuning on R1-generated traces (Source: DeepSeek-V3 Technical Report). This R1 → V3 feedback loop is distinct from the external distillation-controversy narrative associated with DeepSeek-R1, and is treated in Distillation as an example of same-lab reasoning distillation.
Checkpoint updates and licensing
On March 25, 2025 DeepSeek released DeepSeek-V3-0324, an updated checkpoint of the model, and moved the release to the MIT License, matching the license already used for DeepSeek-R1 (Source: api-docs.deepseek.com). Simon Willison noted the naming convention bakes the release date into the model name and described the updated open-weights release as a 641 GB download (Source: simonwillison.net).
Related models
- DeepSeek-R1 — the long-CoT reasoning series whose traces are distilled into V3.
- DeepSeek V4 Pro / V4 Flash — DeepSeek's subsequent model generation.
- DeepSeek-V3 Technical Report — the primary technical source (arXiv 2412.19437).
Relationships
- instance-of: General-Purpose AI (GPAI)
- depends-on: Scaling Laws
- supports: China and the US Are Running Different AI Races — cited as the exemplar of constraint-driven efficiency
- related: DeepSeek-R1, DeepSeek V4 Pro / V4 Flash, DeepSeek, Distillation, Open-Source AI / Open-Weight Models