Drafted with Claude from the paper on 15 October 2026; reviewed by Zeeshan Arif. Read from arxiv.org (arXiv version 2 (January 2026); the paper was published in Nature (volume 645, September 2025)).
What it changed
It showed in the open that reinforcement learning with simple right-or-wrong rewards, and no human-written reasoning examples, can teach a language model to reason at length, to the level of OpenAI’s o1 on maths and coding tests.
The problem before it
Chain-of-thought prompting and fine-tuning on worked solutions had improved reasoning. But the authors name two problems. Human-written reasoning examples are costly to collect at scale, and a model trained to copy them is limited by them: it cannot find better, non-human ways to reason.
o1 had shown that a model trained to think for longer before answering does much better on hard problems, and work on test-time compute supported spending more computation at inference. But OpenAI did not publish how o1 was trained.
The idea
The mechanism. The paper trains two models from DeepSeek-V3’s base model.
DeepSeek-R1-Zero uses reinforcement learning only, with no supervised fine-tuning first. The authors’ bet was that human-defined reasoning patterns would limit what the model explores.
- The template. The model is asked to write its reasoning inside
<think> tags and then its answer inside <answer> tags. Nothing else about how to reason is specified.
- The rewards are rules, not a learned model: an accuracy reward (is the final maths answer right, do the test cases pass?) and a format reward (is the reasoning inside the tags?). The authors avoid neural reward models for reasoning because the model learns to exploit them (“reward hacking”).
- The algorithm is GRPO. For each question, sample 16 answers, score them, and push the model towards the answers that scored above the group’s average. Unlike PPO, it needs no separate model to estimate values.
During training, R1-Zero’s answers grew from hundreds to thousands of tokens without being asked to. It started to check its work, reflect and try other approaches. At one point it wrote “Wait, wait. Wait. That’s an aha moment I can flag here.” and started again, which the authors call an “aha moment”.
DeepSeek-R1 fixes R1-Zero’s problems, which were hard-to-read reasoning and mixing English and Chinese. It is trained in several stages:
- Fine-tune on thousands of “cold-start” examples of readable, conversational reasoning.
- Reinforcement learning as for R1-Zero, plus a reward for staying in one language.
- Generate many answers, keep the good ones (rejection sampling), and fine-tune again on these plus non-reasoning data such as writing.
- A second round of reinforcement learning, adding reward models for helpfulness and safety.
Finally, about 800,000 examples generated by R1 were used to fine-tune smaller open models (Qwen and Llama, from 1.5 to 70 billion parameters), with no reinforcement learning.
The notation. GRPO samples a group of outputs o_1, ..., o_G for a question and gives each the advantage A_i = (r_i - mean(r)) / std(r), its reward compared with the group. It then maximises a clipped objective, as in PPO, minus β times the KL divergence from a reference model. The rule reward for R1-Zero is Reward = Reward_acc + Reward_format.
What it showed
Reported in the paper. We read version 2 of the arXiv paper. A version was published in Nature, a peer-reviewed journal, in September 2025. The comparisons were run by DeepSeek.
- Reasoning from reinforcement learning alone. R1-Zero’s AIME 2024 maths score rose from 15.6% to 77.9% during training, and to 86.7% with majority voting over many answers. The authors say this is above the average human competitor.
- R1 against o1. On AIME 2024, R1 scored 79.8% and OpenAI’s o1 (December 2024 version) 79.2%. On MATH-500, 97.3% against 96.4%. On Codeforces, R1 reached the 96.3rd percentile (rating 2029), and o1 the 96.6th (2061).
- Where o1 was ahead. On GPQA Diamond (hard science questions), R1 scored 71.5% and o1 75.7%. On SimpleQA (facts), 30.1% against 47.0%. On the Aider coding test, 53.3% against 61.7%. On SWE-bench Verified, the two were close (49.2% and 48.9%).
- General use. The final reinforcement stage raised R1’s AlpacaEval 2.0 score from 62.1 to 87.6 and its ArenaHard score from 75.6 to 92.3. On Chatbot Arena on 24 January 2025, with style control, R1 shared first place with o1 and Gemini-Exp-1206.
- Small models learn from it. A 1.5 billion-parameter Qwen model fine-tuned on R1’s outputs scored 28.9% on AIME 2024, above GPT-4o (9.3%) and Claude 3.5 Sonnet (16.0%). The 32B version scored 72.6%. For small models, this distillation worked better than reinforcement learning alone.
- Cost. The reinforcement learning and data creation for R1-Zero and R1 took 147,000 H800 GPU hours, about $294,000 at an assumed $2 per GPU hour. This does not include training the DeepSeek-V3 base model.
- Safety. The authors judge R1’s built-in safety as moderate, comparable to GPT-4o (May 2024), and higher when combined with a risk-control system.
Limits, and what came later
The limits come from the paper:
- It is weaker than other models at structured output, and cannot use tools such as search engines or calculators.
- It sometimes “overthinks”, writing far more than needed for simple questions.
- It is tuned for Chinese and English, and may reason or answer in English when asked in another language.
- It is sensitive to prompts. Giving it examples (few-shot prompting) made it worse, so the authors recommend describing the problem directly.
- It improved little over DeepSeek-V3 on software engineering, because those tests were too slow to use much in reinforcement learning.
- Pure reinforcement learning needs reliable rewards. For tasks like writing, a learned reward model can be exploited, so R1 used human-annotated data there and only a few hundred steps of reinforcement learning.
- It can be jailbroken into giving dangerous plans, and its better reasoning can make those plans more practical. As an open model, it can also be fine-tuned to remove its safety training.
Later work built on it. Qwen3 cited the open reasoning trend in designing its thinking mode.
What to know first
How to read it
The paper is 86 pages, but the main text is only 11. The rest is supplementary material.
- Read section 1.
- Read section 2 with Figure 1 and Table 2 (the “aha moment”). This is the core result.
- Read section 3 with Figure 2 for the multi-stage pipeline, then Table 3.
- Read section 6 for the authors’ own limits.
- In the supplement, read section B.5 on reward hacking and section F on distillation.
Expect about 1.5 hours for the parts above (an estimate).
Links