DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" is a paper by DeepSeek-AI that introduces the company's first-generation reasoning models, DeepSeek-R1-Zero and DeepSeek-R1, together with six smaller dense models distilled from DeepSeek-R1. It was posted to arXiv as 2501.12948 on January 22, 2025, with a second version on January 4, 2026, and was published in Nature vol. 645, 633–638 (2025). The arXiv v1 text lists DeepSeek-AI as author; its appendix names 18 core contributors, beginning with Daya Guo, Dejian Yang and Haowei Zhang, and 177 further contributors. The paper reports that reasoning capabilities in large language models can be incentivized through large-scale reinforcement learning (RL) without human-labeled reasoning trajectories, that the resulting DeepSeek-R1 performs comparably to OpenAI-o1-1217 on reasoning tasks, and that distilling DeepSeek-R1's outputs into small models works better than applying the same RL to those models directly.

Summary

The paper introduces two training regimes, both starting from DeepSeek-V3-Base. DeepSeek-R1-Zero is trained by large-scale RL applied directly to the base model with no supervised fine-tuning (SFT). The paper reports that reasoning behaviors such as self-verification, reflection and long chains of thought emerge from this RL with rule-based rewards, and DeepSeek describes it as the first open research to validate that reasoning in LLMs can be incentivized purely through RL. DeepSeek-R1-Zero's outputs were hard to read and mixed languages. DeepSeek-R1 addresses these problems, and improves performance further, with a four-stage pipeline: a small cold-start SFT set of long chain-of-thought examples, reasoning-oriented RL, a round of rejection sampling and SFT on about 800,000 samples, and a final RL stage covering all scenarios.

The paper then fine-tunes six open models from the Qwen2.5 and Llama 3 families (1.5B, 7B, 8B, 14B, 32B and 70B parameters) on the 800,000 samples curated with DeepSeek-R1. It open-sources DeepSeek-R1-Zero, DeepSeek-R1 and the six distilled models. A comparison on Qwen2.5-32B finds that distillation from DeepSeek-R1 outperforms more than 10,000 steps of large-scale RL on the same base. The paper also reports two approaches that did not work for the authors, process reward models and Monte Carlo tree search, and closes with four limitations.

DeepSeek-R1-Zero: reinforcement learning on the base model

Group Relative Policy Optimization

To save RL training costs, DeepSeek uses Group Relative Policy Optimization (GRPO), introduced in its earlier DeepSeekMath work (Shao et al., 2024). GRPO drops the critic model, which is typically the same size as the policy model, and instead estimates the baseline from group scores. For each question it samples a group of outputs from the old policy and computes each output's advantage by normalizing its reward against the mean and standard deviation of the group's rewards. The objective uses a clipped probability ratio, as in PPO, and a KL-divergence penalty against a reference policy, with the clipping range and KL weight as hyperparameters.

Rule-based rewards and training template

DeepSeek-R1-Zero is trained with a rule-based reward system of two kinds:

• Accuracy rewards check whether a response is correct. For math problems with deterministic answers the model must give the final answer in a specified format, such as inside a box, so correctness can be verified by rule; for LeetCode problems a compiler checks the code against predefined test cases.
• Format rewards require the model to place its reasoning between <think> and </think> tags.

The authors did not use an outcome or process neural reward model for DeepSeek-R1-Zero. They state that a neural reward model may suffer from reward hacking in large-scale RL, and that retraining it takes additional resources and complicates the pipeline.

The training template asks the model to reason first and then answer, with the reasoning and the answer enclosed in <think> and <answer> tags. DeepSeek limited the template to this structural format and avoided content-specific instructions, such as requiring reflective reasoning or particular problem-solving strategies, so that it could observe how the model developed during RL.

Performance and self-evolution

Over thousands of RL steps, DeepSeek-R1-Zero's average pass@1 on AIME 2024, measured by sampling 16 responses per question, rose from 15.6% to 71.0%. With majority voting over 64 samples its score was 86.7%, which the paper reports as exceeding OpenAI-o1-0912. On the paper's reasoning comparison (Table 2), DeepSeek-R1-Zero scored 95.9% on MATH-500, 73.3% on GPQA Diamond, 50.0% on LiveCodeBench and a Codeforces rating of 1,444, against 94.8%, 77.3%, 63.4% and 1,843 for OpenAI-o1-0912 and 90.0%, 60.0%, 53.8% and 1,820 for OpenAI-o1-mini. These results are recorded on the benchmark pages listed under Evaluation below.

The average response length on the training set grew steadily during RL, from hundreds to thousands of reasoning tokens. The authors present this as an intrinsic development rather than the result of external adjustment: with more test-time computation the model began reflecting on and re-evaluating earlier steps and exploring alternative approaches, behaviors that were not explicitly programmed.

The "aha moment"

The paper describes an "aha moment" in an intermediate version of DeepSeek-R1-Zero, in which the model learned to allocate more thinking time to a problem by re-evaluating its initial approach. In the example it gives (Table 3), the model stops partway through solving an equation and writes "Wait, wait. Wait. That's an aha moment I can flag here," before rechecking its work step by step. The authors note the anthropomorphic tone and describe the episode as an "aha moment" for the researchers as well, attributing it to RL supplying incentives rather than explicit instructions on how to solve problems.

Drawbacks

DeepSeek-R1-Zero had poor readability and mixed languages within its responses, which often lacked markdown formatting to highlight answers. DeepSeek developed DeepSeek-R1 to make the reasoning readable and shareable.

DeepSeek-R1: reinforcement learning with cold start

The paper frames DeepSeek-R1 around two questions: whether a small amount of high-quality cold-start data could improve reasoning or speed convergence, and how to train a user-friendly model that produces clear chains of thought and has strong general capabilities. The pipeline has four stages, two RL stages and two SFT stages.

Cold start

To avoid the unstable early phase of RL from a base model, DeepSeek fine-tuned DeepSeek-V3-Base on "thousands" of long chain-of-thought examples to serve as the initial RL actor. The data were collected by few-shot prompting with a long chain-of-thought example, by prompting models directly for detailed answers with reflection and verification, by gathering DeepSeek-R1-Zero outputs in a readable format, and by post-processing from human annotators. The cold-start format places a summary at the end of each response, in the pattern |special_token|<reasoning_process>|special_token|<summary>, and responses that were not reader-friendly were filtered out. The authors report better performance than DeepSeek-R1-Zero from designing this pattern with human priors and state that they believe iterative training is better for reasoning models.

Reasoning-oriented reinforcement learning

The cold-started model then undergoes the same large-scale RL as DeepSeek-R1-Zero, focused on coding, mathematics, science and logic problems with well-defined solutions. Because chains of thought often mixed languages when RL prompts involved several languages, DeepSeek added a language-consistency reward, calculated as the proportion of target-language words in the chain of thought. Ablations showed a slight performance degradation from this reward, which the authors accepted because it aligns with human preferences and makes output more readable. The final reward is the sum of reasoning accuracy and language consistency, and RL continues until convergence on reasoning tasks.

Rejection sampling and supervised fine-tuning

When reasoning-oriented RL converged, DeepSeek used the checkpoint to generate SFT data for another round:

• Reasoning data. Reasoning trajectories were produced by rejection sampling from the RL checkpoint. The prompt set was expanded beyond problems verifiable by rule, with some data judged by a generative reward model in which DeepSeek-V3 compared model predictions with ground truth. Chains of thought with mixed languages, long paragraphs or code blocks were filtered out, and of multiple sampled responses per prompt only correct ones were kept, giving about 600,000 reasoning samples.
• Non-reasoning data. For writing, factual QA, self-cognition and translation, DeepSeek reused the DeepSeek-V3 pipeline and parts of the DeepSeek-V3 SFT dataset. For some tasks DeepSeek-V3 was prompted to generate a chain of thought before answering; simple queries such as "hello" received none. This gave about 200,000 samples.

DeepSeek-V3-Base was fine-tuned on the combined set of about 800,000 samples for two epochs.

Reinforcement learning for all scenarios

A second RL stage aims to improve helpfulness and harmlessness while continuing to refine reasoning. Reasoning data keep the rule-based rewards used for DeepSeek-R1-Zero in math, code and logic. General data use reward models to capture human preferences, following the DeepSeek-V3 pipeline with a similar distribution of preference pairs and training prompts (see RLHF (Reinforcement Learning from Human Feedback)). Helpfulness is assessed on the final summary only, to limit interference with the reasoning process; harmlessness is assessed on the whole response, reasoning and summary, to identify risks, biases or harmful content.

Distillation into smaller dense models

DeepSeek fine-tuned six open models directly on the 800,000 samples curated with DeepSeek-R1: Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, Qwen2.5-14B and Qwen2.5-32B from Qwen2.5, Llama-3.1-8B, and Llama-3.3-70B-Instruct from Llama 3. Llama-3.3 was chosen over Llama-3.1 for the 70B model because its reasoning capability was slightly better. The distilled models received SFT only. The paper states that adding RL could substantially boost their performance, and reports finding significant further gains from applying RL to them, but presents only the SFT results in order to demonstrate the distillation technique, leaving RL on distilled models to the research community (see Distillation).

Evaluation

Setup

DeepSeek evaluated on MMLU, MMLU-Redux, MMLU-Pro, C-Eval, CMMLU, IFEval, FRAMES, GPQA Diamond, SimpleQA, Chinese SimpleQA, SWE-bench Verified, Aider, LiveCodeBench (problems from August 2024 to January 2025), Codeforces, CNMO 2024 (the Chinese National High School Mathematics Olympiad) and AIME 2024, and on open-ended generation with AlpacaEval 2.0 and Arena-Hard, which use GPT-4-Turbo-1106 as a pairwise judge. For the two judged benchmarks only the final summary was scored, to avoid length bias. Following the DeepSeek-V3 setup, MMLU, DROP, GPQA Diamond and SimpleQA used prompts from OpenAI's simple-evals framework; MMLU-Redux used the Zero-Eval prompt format zero-shot; and the few-shot prompts of MMLU-Pro, C-Eval and CLUE-WSC were modified to zero-shot, because chain-of-thought examples in few-shot prompts may hurt DeepSeek-R1. LiveCodeBench was evaluated in chain-of-thought format. Codeforces performance was estimated on problems from 10 Division 2 contests with expert-crafted test cases, from which expected ratings and percentiles of competitors were calculated. SWE-bench Verified results were obtained with the Agentless framework, and Aider results in the "diff" edit format. The paper also describes a multilingual HumanEval-Mul set covering eight programming languages, for which it reports no results.

Outputs were capped at 32,768 tokens. Because greedy decoding of long-output reasoning models produced higher repetition and large variation between checkpoints, the paper reports pass@1 estimated from k sampled responses per question (k between 4 and 64 depending on test-set size) at temperature 0.6 and top-p 0.95, as the mean correctness of the k responses. For AIME 2024 it also reports consensus (majority-vote) accuracy over 64 samples, written cons@64.

The baselines were DeepSeek-V3, Claude-3.5-Sonnet-1022, GPT-4o-0513, OpenAI-o1-mini and OpenAI-o1-1217, with QwQ-32B-Preview added for the distilled models. Because the OpenAI-o1-1217 API was difficult to access from mainland China, its figures are taken from OpenAI's official reports.

DeepSeek-R1 against comparison models

DeepSeek-R1 is a Mixture-of-Experts model with 37 billion activated and 671 billion total parameters, the same as DeepSeek-V3. On the knowledge benchmarks it scored 90.8% on MMLU, 84.0% on MMLU-Pro and 71.5% on GPQA Diamond, above DeepSeek-V3 and below OpenAI-o1-1217 on MMLU and GPQA Diamond; the paper attributes the gain over DeepSeek-V3 mainly to better accuracy on STEM questions from large-scale RL. It scored 82.5% on FRAMES, a long-context question-answering task, and 30.1% on SimpleQA against 24.9% for DeepSeek-V3, a pattern the paper compares to o1 outperforming GPT-4o on the same benchmark. On Chinese SimpleQA it scored 63.7% against DeepSeek-V3's 68.0%, which the paper attributes to DeepSeek-R1 refusing certain queries after safety RL; without safety RL, it states, accuracy would exceed 70%.

DeepSeek-R1 scored 83.3% on IFEval, which the paper links to instruction-following data added in the final SFT and RL stages. It reached a length-controlled win rate of 87.6% on AlpacaEval 2.0 and 92.3% on Arena-Hard. Its summaries averaged 689 tokens on Arena-Hard and 2,218 characters on AlpacaEval 2.0, which the authors present as evidence that the scores are not driven by length bias in the GPT-based judges.

In mathematics DeepSeek-R1 scored 79.8% pass@1 on AIME 2024 (OpenAI-o1-1217: 79.2%), 97.3% on MATH-500 (96.4%) and 78.8% on CNMO 2024. In competitive programming it scored 65.9% on LiveCodeBench and a Codeforces rating of 2,029, above 96.3% of human participants, against 2,061 and the 96.6th percentile for OpenAI-o1-1217. On engineering-oriented coding it scored 53.3% on Aider-Polyglot against 61.7% for OpenAI-o1-1217 and 49.2% on SWE-bench Verified against 48.9%; the authors expect engineering performance to improve in the next version because the amount of RL training data for such tasks was still very limited.

Distilled models

On the paper's reasoning comparison (Table 5), DeepSeek-R1-Distill-Qwen-7B outperforms GPT-4o-0513 on every benchmark, DeepSeek-R1-Distill-Qwen-14B surpasses QwQ-32B-Preview on every metric, and the 32B and 70B distilled models exceed OpenAI-o1-mini on most benchmarks. DeepSeek-R1-Distill-Qwen-32B scored 72.6% on AIME 2024, 94.3% on MATH-500 and 57.2% on LiveCodeBench, results the paper describes as comparable to o1-mini; the paper states that the 32B and 70B models set a new record on reasoning benchmarks among dense models. DeepSeek-R1-Distill-Qwen-7B scored 55.5% on AIME 2024, above QwQ-32B-Preview, and DeepSeek-R1-Distill-Qwen-1.5B scored 28.9% on AIME 2024 and 83.9% on MATH-500, above GPT-4o and Claude-3.5-Sonnet on those math benchmarks. The paper also states that the distilled models significantly outperform other instruction-tuned models built on the same underlying checkpoints.

Distillation versus reinforcement learning

To test whether a small model could match distillation through the paper's own large-scale RL, DeepSeek trained Qwen-32B-Base with RL on math, code and STEM data for more than 10,000 steps. The resulting DeepSeek-R1-Zero-Qwen-32B (Table 6) scored 47.0% pass@1 and 60.0% cons@64 on AIME 2024, 91.6% on MATH-500, 55.0% on GPQA Diamond and 40.2% on LiveCodeBench, on par with QwQ-32B-Preview (50.0%, 60.0%, 90.6%, 54.5% and 41.9%). DeepSeek-R1-Distill-Qwen-32B scored higher on every benchmark (72.6%, 83.3%, 94.3%, 62.1% and 57.2%). The paper draws two conclusions: that distilling more powerful models into smaller ones yields strong results, whereas smaller models relying on the paper's large-scale RL need enormous computational power and may not reach the performance of distillation; and that although distillation is economical and effective, advancing "beyond the boundaries of intelligence" may still require more powerful base models and larger-scale RL.

Results by benchmark

The paper's scores, with their settings, are listed on the pages for MMLU (with MMLU-Redux), MMLU-Pro, GPQA (Diamond), SimpleQA (with Chinese SimpleQA), LiveCodeBench (with Codeforces ratings), SWE-bench (Verified), Aider Polyglot, AIME (AIME 2024) and MATH (MATH-500), and under Reported results below for the other evaluations. Several comparison figures repeat numbers first published by the models' developers. The OpenAI-o1-0912 figures for AIME 2024 (74.4% pass@1, 83.3% cons@64), MATH-500 (94.8%) and GPQA Diamond (77.3%) are those OpenAI reported for o1 in September 2024 (Source: openai.com).

Unsuccessful attempts

The paper reports two approaches that did not succeed in DeepSeek's early development work, adding that this does not mean they cannot produce effective reasoning models.

• Process reward model (PRM). The paper identifies three limitations: a fine-grained step is hard to define explicitly in general reasoning; judging whether an intermediate step is correct is difficult, since automated annotation by models gave unsatisfactory results and manual annotation does not scale; and a model-based PRM leads to reward hacking, with retraining adding resources and complexity. PRMs were useful for reranking the top-N responses or assisting guided search, but in DeepSeek's experiments their advantages were limited relative to the computational overhead they added during large-scale RL.
• Monte Carlo tree search (MCTS). Inspired by AlphaGo and AlphaZero, DeepSeek broke answers into smaller parts, prompted the model to generate tags corresponding to reasoning steps, used MCTS guided by a pre-trained value model to find answers, and trained the actor and value models iteratively on the resulting question-answer pairs. Scaling up ran into two problems: token generation has an exponentially larger search space than chess, and the per-node extension limit used to contain it left the model stuck in local optima; and a fine-grained value model, on which generation quality depends, is hard to train. The paper concludes that MCTS can improve inference-time performance when paired with a pre-trained value model, but that iteratively improving a model through self-search remains a significant challenge.

Limitations and future work

The paper lists four areas for further work:

• General capability. DeepSeek-R1 falls short of DeepSeek-V3 on function calling, multi-turn dialogue, complex role-playing and JSON output. DeepSeek plans to explore how long chains of thought can improve these tasks.
• Language mixing. DeepSeek-R1 is optimized for Chinese and English and may reason and respond in English to queries in other languages.
• Prompt sensitivity. Few-shot prompting consistently degrades performance; the paper recommends that users describe the problem and specify the output format in a zero-shot setting.
• Software engineering. Long evaluation times limited the use of large-scale RL on software-engineering tasks, so DeepSeek-R1 shows little improvement over DeepSeek-V3 on those benchmarks. DeepSeek plans to apply rejection sampling to software-engineering data or to use asynchronous evaluation during RL.

Key claims

• Reasoning can be incentivized through pure RL, "obviating the need for human-labeled reasoning trajectories" (abstract of the later arXiv and Nature version) (high, per paper).
• Advanced reasoning patterns, including self-reflection, verification and long chains of thought, emerge during RL rather than being taught through SFT (high, per paper).
• DeepSeek-R1 achieves performance comparable to OpenAI-o1-1217 on reasoning tasks. In the paper's Table 4 it scores higher than o1-1217 on AIME 2024, MATH-500, LiveCodeBench, DROP and SWE-bench Verified, and lower on MMLU, GPQA Diamond, SimpleQA, the Codeforces rating and Aider-Polyglot (high, per paper; the o1-1217 figures are taken from OpenAI's reports).
• Small dense models distilled from DeepSeek-R1 outperform much larger models trained without RL, with DeepSeek-R1-Distill-Qwen-14B above QwQ-32B-Preview and DeepSeek-R1-Distill-Qwen-32B comparable to o1-mini (high, per paper).
• On a 32B base, distillation from DeepSeek-R1 outperforms large-scale RL on the base model alone (high, per paper; a single comparison on Qwen2.5-32B).

Reception and political context

The paper is the primary technical source for DeepSeek-R1; Epoch AI's account of the release is among the secondary coverage of the model (Source: epochai.substack.com). The paper is cited in discussions of the Fast-Follow Problem as an instance of a model matching o1 within months without comparable compute; in discussions of Distillation, where the distillation of DeepSeek-R1 into small dense models serves as a reference demonstration that reasoning can be distilled; and in discussions of Open-Source AI / Open-Weight Models, since DeepSeek-R1 and its distilled models were released with open weights. On Scaling Laws, the paper places the reasoning gains in post-training RL on a strong pretrained base, and its discussion of distillation versus RL concludes that further progress may require both more powerful base models and larger-scale RL.

DeepSeek-R1's public launch on January 20, 2025 and its rise in the App Store on January 27, 2025 prompted a brief "export controls have failed" narrative that Gregory Allen's CSIS report of March 7, 2025 was written to rebut. Allen characterizes R1's innovations, pure-RL reasoning emergence and distillation into dense models, as real and already being replicated in U.S. labs, some of them internally known but undisclosed and others new. He argues that the correct policy reading is not that controls failed but that controls combined with constraint produce the efficiency R1 exemplifies, which he frames as consistent with the Sullivan/Feldman "implicit subsidy" theory on the Chinese side.

Allen's most-cited conclusion is that the U.S. lead is unlikely to exceed "more than a year or two, even with extremely aggressive export controls," and he reads R1 as consistent with, not a refutation of, that ceiling. The main R1-related risk he identifies is that DeepSeek's open-source community may accelerate the maturation of Huawei's CANN ecosystem by contributing optimized kernels and support, partially closing the CUDA moat that he describes as a larger practical barrier to Chinese frontier training than silicon itself.

Reported results

Evaluation results the document states for which there is no separate benchmark page; results on benchmarks that have a page are recorded there and cite this page. Scores and settings are as printed in the document.

Provenance

Two raw files back this page. The full text is arXiv v1 (January 22, 2025), captured unaltered on 2026-09-19 from arxiv.org; all figures and section numbers on this page are from v1. The earlier raw holds the arXiv abstract-page metadata for the later version (v2, January 4, 2026), including the Nature publication details (vol. 645, pp. 633–638, 2025) and excerpts of the revised abstract, but not the paper body. The v2 and Nature texts have not been captured in full, and figures there may differ from v1.

Relationships

• supports: DeepSeek-R1, Fast-Follow Problem, Distillation, Open-Source AI / Open-Weight Models, China and the US Are Running Different AI Races, Reasoning Models and Chain-of-Thought
• depends-on: DeepSeek-V3 Technical Report (V3 is R1's base), DeepSeek-V3
• related: Scaling Laws, AI Software Progress, AI Benchmarks and Evaluation, CSIS — DeepSeek, Huawei, Export Controls, and the Future of the U.S.-China AI Race (Allen, March 2025), Dario Amodei — On DeepSeek and Export Controls, Reward Hacking (reason given for avoiding neural reward models and PRMs), RLHF (Reinforcement Learning from Human Feedback) (preference reward models in the final RL stage), DeepSeek (developer)
• related: OpenAI o-series (o1 → o4-mini) (comparison baseline), Qwen 2.5 and Llama 3 (Meta) (base models for the distilled checkpoints)
• related: AIME (as a model benchmark), MATH and MATH-500, GPQA, LiveCodeBench, SWE-bench, Aider Polyglot, MMLU, MMLU-Pro, SimpleQA (benchmarks on which the paper's results are recorded)