Drafted with Claude from the paper on 15 October 2026; reviewed by Zeeshan Arif. Read from arxiv.org (arXiv version 3 (July 2024) of the NeurIPS 2023 conference paper).
What it changed
It showed that a language model can be trained on human preferences with one simple classification loss, with no separate reward model and no reinforcement learning, while aiming at the same goal as RLHF.
The problem before it
Reinforcement learning from human feedback (RLHF), as used for summarisation and in InstructGPT, works in three stages:
- Fine-tune a pretrained model on good examples (supervised fine-tuning).
- Show people pairs of the model’s answers, ask which is better, and train a separate reward model to predict their choices.
- Use reinforcement learning, usually PPO, to make the model produce answers with a high reward, while a penalty stops it from moving too far from the starting model.
The authors call this “complex and often unstable”. Stage 3 trains several models, and the model must keep generating new text during training, which is costly. It also needs careful tuning.
The idea
The mechanism. The goal in stage 3 (high reward, but not too far from the starting model) has a known best solution: the starting model’s probabilities, multiplied by a factor that grows with the reward. The authors turn this around. If you know the best model, you can read off the reward: it is the log of how much more likely the model makes an answer than the starting model does, times a constant.
So the language model itself can stand for the reward model. Put this “implicit reward” into the standard formula for preferences between two answers, and an awkward normalising term cancels out. What remains is a loss that depends only on the model being trained and the frozen starting model.
DPO (Direct Preference Optimization) trains on a fixed set of preference pairs: a prompt, a preferred answer and a rejected answer. For each pair, it raises the probability of the preferred answer and lowers that of the rejected one, both measured relative to the starting model. Pairs that the model currently ranks the wrong way get more weight. The authors show that without this weighting, a simpler version can make the model degenerate.
There is no reward model to train, no sampling from the model during training, and no reinforcement learning. The appendix gives the whole loss in about 10 lines of PyTorch.
The notation. RLHF maximises E[r(x, y)] - β · KL(π_θ || π_ref): the reward of answers y to prompts x, minus a penalty for moving away from the reference model π_ref. Its best solution is π*(y | x) ∝ π_ref(y | x) · exp(r(x, y) / β). Rearranged, r(x, y) = β log(π*(y | x) / π_ref(y | x)) + β log Z(x). The DPO loss, for a preferred answer y_w and a rejected answer y_l, is -log σ(β log(π_θ(y_w | x) / π_ref(y_w | x)) - β log(π_θ(y_l | x) / π_ref(y_l | x))), where σ is the logistic function. Z(x) has cancelled.
What it showed
Reported in the paper. We read version 3 of the arXiv paper. It was published at NeurIPS 2023, a peer-reviewed conference. The models have up to 6 billion parameters. For summaries and dialogue, quality is judged by GPT-4, which the authors checked against people.
- Controlled test (positive movie reviews). Here the true reward is known, because a sentiment classifier provides it. Over 22 training runs, DPO got the highest reward for any given distance from the starting model. It beat PPO, even a version of PPO given the true reward instead of a learned one.
- Summaries (Reddit TL;DR). Against the human-written reference summaries, DPO won about 61% of the time at its best setting, against about 57% for PPO at its best. PPO got much worse at higher sampling temperatures, falling to the level of the untrained base model. DPO stayed steadier.
- People’s view. In a direct comparison, people preferred DPO’s summaries to PPO’s 58% of the time.
- Dialogue (Anthropic HH). DPO was the only efficient method that beat the preferred answers in the dataset. It did as well as or better than picking the best of 128 sampled answers with a reward model, which is far more costly.
- New kinds of input. On news articles (CNN/DailyMail), with models trained only on Reddit posts, DPO won 36% against the reference summaries and PPO 26%.
- GPT-4 as a judge. GPT-4 agreed with people about as often as people agreed with each other. With a simple prompt, GPT-4 preferred longer, more repetitive summaries than people did. So the authors used a prompt that also asks about conciseness.
- Little tuning. The authors note they did not meaningfully tune DPO’s one main setting,
β.
Limits, and what came later
The limits come from the paper:
- The models have at most 6 billion parameters. Scaling DPO to much larger models is left to future work.
- It is not yet clear how well DPO generalises to new inputs compared with training on an explicit reward. The news-article test is only a first sign.
- PPO can also learn from extra prompts that have no labels. Whether DPO can do the same, by labelling its own outputs, is an open question.
- Reward over-optimisation, where the model exploits flaws in what it is trained on, may still happen. A slight drop late in training may be an example.
- GPT-4’s verdicts depend on the prompt used to ask it.
Later work built on it directly. Llama 3.1 used direct preference optimisation in its post-training.
What to know first
- RLHF and reward models, as in InstructGPT.
- The Bradley-Terry model: the chance that people prefer answer A to answer B depends on the difference between their rewards.
- KL divergence, a measure of how different two probability distributions are.
- The log-probability a language model gives to a whole answer.
How to read it
The main text is 10 pages. The appendix has the derivations and the code.
- Read section 1 with Figure 1, which compares the RLHF and DPO pipelines.
- Read section 3 for the RLHF stages.
- Read section 4 slowly. Equations 4 to 7 are the core, and “What does the DPO update do?” explains the gradient in words.
- Read section 6 with Figures 2 and 3.
- Look at the code in Appendix B.
Section 5 and the proofs can wait. Expect about 1.5 to 2 hours for the parts above (an estimate).
Links
- The paper: arXiv 2305.18290
- Code: the paper gives the loss as code in Appendix B, but does not link a repository.
- Video: none is linked here yet.