Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Direct preference optimization

Direct preference optimization (DPO) is a way to align a language model with human preferences by training directly on pairs of preferred and rejected answers, using a simple classification-style loss instead of a reward model and reinforcement learning.

Also called: DPO

By Divya Sharma · Checked Nov 16, 2026 · How we review

DPO was introduced by Rafailov et al. in 2023 as a simpler route to the same goal as RLHF. Both start from preference data: for each prompt, one answer people liked better and one they liked less. RLHF first trains a reward model on that data and then runs reinforcement learning against it. DPO shows that the same objective can be rewritten so the language model itself is trained on the pairs, with a loss that raises the probability of the chosen answer relative to the rejected one, measured against a frozen reference copy of the model.

Why it caught on

There is no second model to train and no sampling from the model during training, so DPO runs like ordinary supervised fine-tuning. The paper describes it as stable, performant and computationally lightweight, and reports results that match or beat PPO-based RLHF on the tasks tested. Many open fine-tuning tools, including Unsloth and Axolotl, support it.

Limits

DPO only learns from the pairs you give it, so data quality matters a great deal: noisy or inconsistent preferences produce a confused model. It is usually applied after supervised fine-tuning, not instead of it. Because it is offline, it cannot explore new answers the way an RL loop can, and several later variants try to address that. For the wider picture of adapting a model, see fine-tuning.

Related terms

RLHF
RLHF (reinforcement learning from human feedback) fine-tunes a language model using human rankings of its answers: a reward model learns those preferences, then reinforcement learning pushes the model toward answers that score well.
Reward model
A reward model is a separate model trained on human preference data to score how good a response is. Reinforcement learning then uses that score to tune a language model towards answers people prefer.
Fine-tuning
Fine-tuning is further training of a pretrained model on a smaller, task-specific dataset, so the model adopts a particular skill, format, tone or domain without being trained from scratch.
Supervised fine-tuning
Supervised fine-tuning (SFT) trains a pretrained model further on example prompts paired with the responses you want. It is the usual first step that turns a base model into one that follows instructions.
AI alignment
AI alignment is the work of making an AI system's behaviour match what its users and developers actually intend, so it is helpful, honest and avoids harm, rather than just predicting likely text or optimising a narrow score.

Sources