DeepSeek-V3 Technical Report

The DeepSeek-V3 Technical Report is the primary technical paper for DeepSeek-V3, a Mixture-of-Experts (MoE) language model with 671B total parameters and 37B activated per token, pre-trained on 14.8T tokens. It was authored by DeepSeek-AI and published as arXiv 2412.19437 on December 27, 2024, with a v2 revision on February 18, 2025. The arXiv listing begins with Aixin Liu, Bei Feng, and Bing Xue; the report's contributions appendix credits 196 named individuals (150 in research and engineering, 31 in data annotation, 15 in business and compliance), listed alphabetically within each role, 10 of them marked as having left the team. The report states that the full training run required 2.788M H800 GPU hours with no irrecoverable loss spikes and no rollbacks, an efficiency result that figured in the early-2025 reassessment of the United States compute advantage.

Summary

The paper describes the architecture, training infrastructure, pre-training, post-training, and reported performance of DeepSeek-V3. It carries forward Multi-head Latent Attention (MLA) and the DeepSeekMoE architecture, both validated in the earlier V2, and introduces an auxiliary-loss-free load-balancing strategy and a multi-token prediction (MTP) training objective. On the systems side, it documents FP8 mixed-precision training, the DualPipe algorithm for reducing pipeline-parallelism communication overhead, custom cross-node all-to-all communication kernels, and memory optimizations that allowed training without tensor parallelism. In post-training, the paper describes distilling reasoning capability from a long chain-of-thought model of the DeepSeek-R1 series into DeepSeek-V3, which ties V3 to the DeepSeek-R1 line; the R1 paper in turn trains R1 from DeepSeek-V3-Base (DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning). The paper states that V3 "outperforms other open-source models and achieves performance comparable to leading closed-source models," naming GPT-4o and Claude-3.5-Sonnet as the comparison points.

As the primary technical source for DeepSeek-V3 — previously sourced only from secondary coverage ((Source: epochai.substack.com), China and the US Are Running Different AI Races) — the report documents the architectural levers behind the model's reported training efficiency: MLA, MoE sparsity, FP8 mixed precision, and DualPipe. It also provides direct evidence for distillation as a training strategy: the R1-to-V3 transfer is distillation within the DeepSeek lab, distinct from the Anthropic/Claude controversy.

Training cost

Table 1 of the report gives the cost of the official training run, valued at an assumed H800 rental price of $2 per GPU hour:

The report states that each trillion training tokens required 180K H800 GPU hours, or 3.7 days on its cluster of 2,048 H800 GPUs, so that pre-training finished in less than two months. It notes that the figures cover only the official training of DeepSeek-V3 and exclude "the costs associated with prior research and ablation experiments on architectures, algorithms, or data." The authors attribute the cost to co-design of algorithms, frameworks and hardware, and describe the per-trillion-token cost as "much cheaper than training 72B or 405B dense models."

Architecture

The model remains a Transformer. For details not stated, it follows DeepSeek-V2.

• Multi-head Latent Attention. Keys and values are jointly compressed into a low-rank latent vector, and only that vector and a decoupled key carrying rotary position embedding are cached during generation. The report states this "results in significantly reduced KV cache while maintaining performance comparable to standard Multi-Head Attention." Queries are also low-rank compressed to reduce activation memory in training.
• DeepSeekMoE. Feed-forward layers use finer-grained experts than conventional MoE designs such as GShard, with some experts isolated as shared experts. Unlike V2, V3 computes token-to-expert affinity with a sigmoid function and normalizes over the selected experts' scores.
• Auxiliary-loss-free load balancing. Conventional MoE training adds an auxiliary loss to prevent unbalanced expert load, which the authors state can impair performance when too large. V3 instead adds a per-expert bias term used only for routing: after each step the bias is lowered for overloaded experts and raised for underloaded ones by a fixed bias update speed γ. A complementary sequence-wise balance loss with a very small weight guards against extreme imbalance within a single sequence. The report calls this approach one that DeepSeek-V3 "pioneers," citing Wang et al. (2024).
• Node-limited routing and no token dropping. Each token is sent to at most 4 nodes. The report states that load stayed balanced throughout training, so no tokens were dropped in training or, given its deployment strategy, in inference.
• Multi-token prediction. Inspired by Gloeckle et al. (2024), V3 adds sequential MTP modules that predict additional future tokens while keeping the full causal chain at each depth, sharing the embedding layer and output head with the main model. The authors state MTP densifies the training signal and may let the model pre-plan its representations. The modules can be discarded at inference or reused for speculative decoding.

The configuration has 61 Transformer layers and a hidden dimension of 7,168. MLA uses 128 heads of dimension 128, a KV compression dimension of 512, a query compression dimension of 1,536, and a decoupled per-head dimension of 64. All feed-forward layers except the first three are MoE layers, each with 1 shared and 256 routed experts of intermediate dimension 2,048; 8 routed experts are activated per token. The MTP depth is 1, so each token predicts one additional token. This yields 671B total parameters, 37B activated per token.

Training infrastructure

Training ran on a cluster of 2,048 NVIDIA H800 GPUs, 8 per node connected by NVLink and NVSwitch, with InfiniBand between nodes, using DeepSeek's in-house HAI-LLM framework. The parallelism layout was 16-way pipeline parallelism, 64-way expert parallelism across 8 nodes, and ZeRO-1 data parallelism.

• DualPipe. Cross-node expert parallelism gave a computation-to-communication ratio of roughly 1:1. DualPipe splits each forward and backward chunk into attention, all-to-all dispatch, MLP and all-to-all combine, rearranges a paired forward and backward chunk so that communication is hidden behind computation, and feeds micro-batches from both ends of the pipeline. Compared with 1F1B and ZB1P, the report states it has fewer pipeline bubbles at the cost of keeping two copies of the parameters and a peak activation memory higher by 1/PP (Table 2). The authors state that as long as the computation-to-communication ratio is held constant, the design allows fine-grained experts across nodes at "near-zero all-to-all communication overhead."
• All-to-all kernels. NVLink bandwidth (160 GB/s) is about 3.2 times InfiniBand's (50 GB/s). Tokens are sent over InfiniBand to at most 4 nodes and then forwarded over NVLink, letting each token reach an average of 3.2 experts per node; the report states the design could scale to 13 routed experts at the same communication cost. Twenty streaming multiprocessors (SMs), partitioned into 10 communication channels using warp specialization and customized PTX instructions, were enough to saturate both links.
• Memory. RMSNorm and MLA up-projections are recomputed in back-propagation; an exponential moving average of the parameters is kept in CPU memory; and the shallowest and deepest layers share a pipeline rank so the MTP modules physically share the embedding and output head. These measures let the model be trained without tensor parallelism.

FP8 mixed-precision training

The report introduces an FP8 mixed-precision framework and states that it validates FP8 training "for the first time" on an extremely large-scale model. Most general matrix multiplications (the forward pass and both backward passes of linear layers) run in FP8, which the report says theoretically doubles speed relative to BF16; the embedding, output head, MoE gating, normalization and attention operators stay in BF16 or FP32, as do master weights, weight gradients and optimizer states.

Four techniques target accuracy. Fine-grained quantization scales activations per 1×128 tile and weights per 128×128 block to limit the effect of outliers, an approach the report relates to the microscaling formats announced for NVIDIA's Blackwell GPUs. Because FP8 accumulation on the H800's Tensor Cores retains only about 14 bits, giving a maximum relative error of nearly 2% in a test with inner dimension 4,096, partial results are promoted to FP32 on CUDA cores every 128 elements. All tensors use the E4M3 format rather than a hybrid of E4M3 and E5M2. Scaling factors are computed online rather than from a history of prior iterations. To save memory and bandwidth, AdamW moments are stored in BF16, most cached activations and the activations dispatched to MoE layers are held in FP8, and the combine steps stay in BF16.

The framework was validated on two model scales similar to DeepSeek-V2-Lite and DeepSeek-V2, with the loss of the FP8 models staying within 0.25% relative error of BF16 baselines. Appendix B.1 describes these as MoE models of about 16B parameters trained on 1.33T tokens and about 230B parameters trained on about 0.9T tokens. Appendix B.2 reports that quantizing activation gradients block-wise, as done for weights, caused an MoE model of about 16B parameters to diverge after about 300B tokens, which the authors hypothesize reflects token-correlated outliers.

Inference and deployment

Serving separates prefilling from decoding. The minimum prefilling unit is 4 nodes (32 GPUs), with 4-way tensor parallelism plus sequence parallelism and 8-way data parallelism for attention and 32-way expert parallelism for MoE layers; 32 redundant copies of high-load experts, re-selected periodically (for example every 10 minutes), balance load. The minimum decoding unit is 40 nodes (320 GPUs), with 320-way expert parallelism, one expert per GPU, 64 GPUs hosting redundant and shared experts, and point-to-point InfiniBand transfers using IBGDA. In decoding the shared expert is treated as a routed expert, so each token selects 9 experts. The report describes dynamic expert redundancy and two-micro-batch overlap in decoding as work in progress.

Suggestions on hardware design

Section 3.5 addresses AI hardware vendors directly. It notes that communication currently occupies 20 of the H800's 132 SMs and asks for communication co-processors that offload this work and unify the InfiniBand (scale-out) and NVLink (scale-up) networks behind simple primitives. For compute, it asks for higher FP8 accumulation precision in Tensor Cores, describing the Hopper architecture's 14-bit accumulation as a "hardware deficiency"; native support for tile- and block-wise quantization with group scaling; fusing FP8 casting with memory transfers (or near-memory computing, which it estimates would cut off-chip memory access by roughly 50%) to support online quantization; and direct transposed reads for GEMM operations.

Pre-training

Relative to V2, the corpus raises the share of mathematics and programming samples and extends multilingual coverage beyond English and Chinese, for 14.8T tokens. Documents are packed without cross-sample attention masking, and the Fill-in-Middle strategy in Prefix-Suffix-Middle form is applied at a rate of 0.1. The tokenizer uses byte-level BPE with a 128K vocabulary; to counter token-boundary bias from new tokens combining punctuation and line breaks, a proportion of such tokens is split at random during training.

Training used AdamW (β1 = 0.9, β2 = 0.95, weight decay 0.1), a 4K sequence length, and a learning rate that rose to 2.2×10⁻⁴ over 2K steps, held until 10T tokens, decayed on a cosine curve to 2.2×10⁻⁵ over 4.3T tokens, and was held at 2.2×10⁻⁵ for 333B tokens and then 7.3×10⁻⁶ for the final 167B. The batch size increased from 3,072 to 15,360 over the first 469B tokens. The bias update speed γ was 0.001 for the first 14.3T tokens and 0 thereafter; the balance-loss weight α was 0.0001; and the MTP loss weight was 0.3 for the first 10T tokens and 0.1 for the remaining 4.8T.

Context was then extended with YaRN in two 1,000-step phases, from 4K to 32K (batch size 1,920) and then to 128K (batch size 480), at a learning rate of 7.3×10⁻⁶. The report shows Needle In A Haystack results for the post-SFT model as a figure and states that performance is consistent across context lengths up to 128K.

Evaluation of the base model

DeepSeek-V3-Base is compared in Table 3 with DeepSeek-V2-Base, Qwen2.5 72B Base and LLaMA-3.1 405B Base, all run in DeepSeek's internal evaluation framework under the same setting, with scores within 0.3 of each other treated as equivalent. The report notes that DeepSeek-V2-Base scores differ slightly from its earlier publications because the framework changed. It describes DeepSeek-V3-Base as the strongest open-source base model, especially in code and mathematics, and as ahead of Qwen2.5 72B Base on Chinese benchmarks except CMMLU. Individual results are recorded below and on the benchmark pages cited there.

Ablations

• Multi-token prediction (Table 4). An MTP module of depth 1 was added to MoE baselines of 15.7B total parameters (1.33T tokens) and 228.7B total parameters (540B tokens), with the module discarded at inference so inference cost is unchanged. The report states that MTP "consistently enhances the model performance on most of the evaluation benchmarks."
• Auxiliary-loss-free balancing (Table 5). Baselines of 15.7B (1.33T tokens) and 228.7B (578B tokens) total parameters trained with auxiliary losses at the strengths used for DeepSeek-V2-Lite and DeepSeek-V2 were compared with the same models using only the bias-based strategy; the report states the auxiliary-loss-free version is better on most benchmarks.
• Batch-wise versus sequence-wise balance. The authors attribute the gain to balancing across a batch rather than within each sequence, which lets experts specialize by domain, and show greater expert specialization on the Pile test set for a 16B auxiliary-loss-free model (figures only). A batch-wise auxiliary loss matched the auxiliary-loss-free method: validation losses were 2.258 (sequence-wise auxiliary loss), 2.253 (auxiliary-loss-free) and 2.253 (batch-wise auxiliary loss) for 1B MoE models, and at 3B, 2.085 with the sequence-wise auxiliary loss against 2.080 for both the auxiliary-loss-free method and the batch-wise auxiliary loss. The report names two efficiency risks of batch-wise balancing, imbalance within small batches and domain shift at inference, and states that large-scale parallelism and redundant-expert deployment address them.

Post-training

• Supervised fine-tuning. The instruction data comprise 1.5M instances. Reasoning data (mathematics, competition code, logic puzzles) were generated with an internal DeepSeek-R1 model; because R1 output suffered from "overthinking, poor formatting, and excessive length," domain expert models were trained with SFT and RL on both original responses and R1 responses under a system prompt encouraging reflection and verification, then used as generators with rejection sampling to produce concise final data. Non-reasoning data (creative writing, role-play, simple question answering) were generated by DeepSeek-V2.5 and checked by human annotators. DeepSeek-V3-Base was fine-tuned for two epochs with a learning rate decaying from 5×10⁻⁶ to 1×10⁻⁶.
• Reward models. Questions verifiable by rule (boxed maths answers, LeetCode problems checked by a compiler) use a rule-based reward, which the report describes as "resistant to manipulation or exploitation." Other questions use a model-based reward model trained from DeepSeek-V3 SFT checkpoints on preference data that include the chain of thought leading to each reward, which the authors state helps mitigate reward hacking.
• Reinforcement learning. As in V2, V3 uses Group Relative Policy Optimization (GRPO), which drops the separate critic model and estimates the baseline from the scores of a group of sampled outputs. Prompts span coding, mathematics, writing, role-play and question answering.
• Self-rewarding. For tasks without external verification, DeepSeek used a constitutional AI approach in which DeepSeek-V3's own voting evaluations served as the feedback source. The report states this "produced notable alignment effects" in subjective evaluations.

Evaluation of the chat model

Table 6 compares DeepSeek-V3 with DeepSeek-V2-0506, DeepSeek-V2.5-0905, Qwen2.5 72B Instruct, LLaMA-3.1 405B Instruct, Claude-3.5-Sonnet-1022 and GPT-4o-0513, with closed models evaluated through their APIs and all outputs capped at 8,192 tokens. MMLU, DROP, GPQA and SimpleQA use simple-evals prompts; MMLU-Redux uses the Zero-Eval format zero-shot; SWE-bench Verified uses the Agentless framework; Aider benchmarks use the diff format; AIME and CNMO 2024 are sampled at temperature 0.7 and averaged over 16 runs; and MATH-500 uses greedy decoding. The report's introduction highlights 88.5 on MMLU, 75.9 on MMLU-Pro and 59.1 on GPQA.

In the report's reading, DeepSeek-V3 is the best-performing open-source model and competitive with GPT-4o and Claude-3.5-Sonnet. It ranks second to Claude 3.5 Sonnet on GPQA-Diamond and trails Claude-Sonnet-3.5-1022 on engineering coding tasks (SWE-bench Verified, Aider), while leading the compared models on HumanEval-Mul, LiveCodeBench and the mathematics benchmarks, which the authors attribute to distillation from DeepSeek-R1. It trails GPT-4o and Claude-Sonnet on SimpleQA, which the report attributes to allocating more training tokens to Chinese knowledge, and leads on Chinese SimpleQA, 16.4 points above Qwen2.5-72B although Qwen2.5 was trained on 18T tokens. On the open-ended Arena-Hard and AlpacaEval 2.0 (Table 7, judged by GPT-4-Turbo-1106), the report calls V3 the first open-source model to exceed 85% on Arena-Hard. Used as a generative reward model on RewardBench (Table 8), V3 is described as on par with GPT-4o-0806 and Claude-3.5-Sonnet-1022, with further gains from majority voting over six samples.

Distillation from DeepSeek-R1 and MTP decoding

An ablation on DeepSeek-V2.5 (Table 9) compared a baseline trained on short chain-of-thought data with one trained on data from the R1-derived expert checkpoints. Distillation raised LiveCodeBench-CoT and MATH-500 scores but also lengthened responses, the average MATH-500 response rising from 769 to 1,510 tokens, and the report states that DeepSeek selected settings for V3 to balance accuracy against length. The authors describe distillation from reasoning models as "a promising direction for post-training optimization" and note that their work so far covers only mathematics and coding data.

Used for speculative decoding, the MTP module's second-token prediction is accepted 85% to 90% of the time across generation topics, which the report states gives 1.8 times the tokens per second.

Limitations and future directions

The report names two limitations, both concerning deployment: the recommended deployment unit is large, which "might pose a burden for small-sized teams," and although end-to-end generation is more than twice as fast as DeepSeek-V2, there is room for improvement; it expects both to ease with more advanced hardware. It restates DeepSeek's commitment to "the route of open-source models with longtermism" toward AGI and lists future work on more efficient architectures approaching infinite context and going beyond the Transformer, on scaling training data and signal sources, on longer and deeper reasoning, and on broader evaluation methods to avoid "optimizing a fixed set of benchmarks," which it says may "create a misleading impression of the model capabilities."

Key claims

• Architecture: MLA and the DeepSeekMoE architecture, both validated in V2; new in V3 are the auxiliary-loss-free load-balancing strategy and the multi-token prediction training objective. (high)
• FP8 mixed-precision training at this scale is described as a first for a model of V3's size. (high)
• DualPipe reduces pipeline-parallelism communication overhead. (high)
• DeepSeek-V3 distills reasoning capability from the long chain-of-thought DeepSeek-R1 series during post-training, generating SFT data with R1-derived expert models; this is the V3–DeepSeek-R1 link described above. (high)
• V3 "outperforms other open-source models and achieves performance comparable to leading closed-source models." (high, per the paper)
• The full training run cost 2.788M H800 GPU hours, or $5.576M at $2 per GPU hour, excluding prior research and ablations. (high, per the paper)

Reception and political context

A March 2025 CSIS report by Gregory Allen (Allen's CSIS report) reads the V3 paper in the context of export-control policy. Allen argues that DeepSeek's algorithmic and architectural innovations — MoE scaling, MLA, FP8, and DualPipe — are genuinely novel and validated by Western replication, rather than propaganda or artifacts of undisclosed GPU stockpiles. Allen also argues that the $5.576M V3 training cost (2.788M H800 GPU-hours × $2/hour) widely quoted from this paper does not overturn the compute-gap account, citing three points:

• DeepSeek CEO Liang Wenfeng describes export controls as "the greatest challenge," with Chinese firms needing "two to four times" the compute of unrestricted access.
• DeepSeek/High-Flyer's total GPU server capex is estimated at $1.63B, against which the V3 training cost is marginal rather than a $6M bottom line.
• Citing the Jevons Paradox, Allen notes that after V3 and R1, major U.S. firms announced roughly 50% capex increases, an argument that efficiency gains raise rather than lower chip demand.

Allen characterizes the V3 paper as evidence of constraint-driven efficiency within a binding constraint, not as proof the constraint failed. The paper itself states that the $5.576M figure excludes research and ablation costs (see Training cost above).

Discrepancies noted

• Arena-Hard. Section 5.3.3 states that DeepSeek-V3 "achieves an impressive win rate of over 86% against the baseline GPT-4-0314"; Table 7 gives 85.5, and the same section calls V3 the first open-source model "to surpass 85%."
• Mathematics margins. Section 5.3.2 states that on AIME, MATH-500 and CNMO 2024 DeepSeek-V3 "outperforms the second-best model, Qwen2.5 72B, by approximately 10% in absolute scores." In Table 6 the margins over Qwen2.5 72B Instruct are 15.9 points on AIME 2024 (39.2 against 23.3, a score LLaMA-3.1 405B Instruct also reaches), 10.2 on MATH-500 (90.2 against 80.0) and 27.3 on CNMO 2024 (43.2 against 15.9).
• AlpacaEval 2.0. Section 5.3.3 states V3 surpasses DeepSeek-V2.5-0905 "by a significant margin of 20%"; Table 7 gives 70.0 against 50.5, a difference of 19.5 points.
• FP8 validation runs. Section 3.3 describes the FP8 validation models as trained "for approximately 1 trillion tokens"; Appendix B.1 gives 1.33T tokens for the smaller model and about 0.9T for the larger.

Figure-only results

Values for the following appear only in figures and are not recorded: the benchmark overview chart (Figure 1), Needle In A Haystack results to 128K (Figure 8), relative expert load of the auxiliary-loss-based and auxiliary-loss-free 16B models (Figures 9 and 11), and FP8 versus BF16 loss curves (Figure 10).

Reported results

Evaluation results the document states for which there is no separate benchmark page; results on benchmarks that have a page are recorded there and cite this page. Scores and settings are as printed in the document. Rows for the Table 4 and Table 5 ablations describe internal checkpoints, not released models; their MMLU, HumanEval, GSM8K and MATH rows are on those benchmarks' pages, as are the Table 9 distillation rows (LiveCodeBench, MATH-500).

Relationships

• supports: DeepSeek-V3, DeepSeek, Distillation, Fast-Follow Problem, China and the US Are Running Different AI Races
• related: Scaling Laws, AI Software Progress, Compute Governance, CSIS — DeepSeek, Huawei, Export Controls, and the Future of the U.S.-China AI Race (Allen, March 2025), Mixture of Experts (MoE), Export Controls (AI), DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• depends-on: Distillation

Provenance

Two raw files back this page. The full text is arXiv v2 (February 18, 2025), captured unaltered on 2026-09-19 from arxiv.org and held in Raw Sources/DeepSeek-V3 Technical Report - full text (DeepSeek, 2024-12-27).md; section and table numbers on this page follow that version. The earlier raw, Raw Sources/DeepSeek-V3 Technical Report.md, holds the arXiv abstract-page metadata (submission and revision dates, DOI 10.48550/arXiv.2412.19437, the abstract, and the checkpoint repository github.com) but not the paper body. Figures are not reproduced in the capture.