Direct preference optimization
Direct preference optimization (DPO) is a way to align a language model with human preferences by training directly on pairs of preferred and rejected answers, using a simple classification-style loss instead of a reward model and reinforcement learning.
Also called: DPO
By Divya Sharma · Checked Nov 16, 2026 · How we review
DPO was introduced by Rafailov et al. in 2023 as a simpler route to the same goal as RLHF. Both start from preference data: for each prompt, one answer people liked better and one they liked less. RLHF first trains a reward model on that data and then runs reinforcement learning against it. DPO shows that the same objective can be rewritten so the language model itself is trained on the pairs, with a loss that raises the probability of the chosen answer relative to the rejected one, measured against a frozen reference copy of the model.
Why it caught on
There is no second model to train and no sampling from the model during training, so DPO runs like ordinary supervised fine-tuning. The paper describes it as stable, performant and computationally lightweight, and reports results that match or beat PPO-based RLHF on the tasks tested. Many open fine-tuning tools, including Unsloth and Axolotl, support it.
Limits
DPO only learns from the pairs you give it, so data quality matters a great deal: noisy or inconsistent preferences produce a confused model. It is usually applied after supervised fine-tuning, not instead of it. Because it is offline, it cannot explore new answers the way an RL loop can, and several later variants try to address that. For the wider picture of adapting a model, see fine-tuning.
Related terms
- RLHF
- RLHF (reinforcement learning from human feedback) fine-tunes a language model using human rankings of its answers: a reward model learns those preferences, then reinforcement learning pushes the model toward answers that score well.
- Reward model
- A reward model is a separate model trained on human preference data to score how good a response is. Reinforcement learning then uses that score to tune a language model towards answers people prefer.
- Fine-tuning
- Fine-tuning is further training of a pretrained model on a smaller, task-specific dataset, so the model adopts a particular skill, format, tone or domain without being trained from scratch.
- Supervised fine-tuning
- Supervised fine-tuning (SFT) trains a pretrained model further on example prompts paired with the responses you want. It is the usual first step that turns a base model into one that follows instructions.
- AI alignment
- AI alignment is the work of making an AI system's behaviour match what its users and developers actually intend, so it is helpful, honest and avoids harm, rather than just predicting likely text or optimising a narrow score.