RLHF
RLHF (reinforcement learning from human feedback) fine-tunes a language model using human rankings of its answers: a reward model learns those preferences, then reinforcement learning pushes the model toward answers that score well.
Also called: reinforcement learning from human feedback
By Divya Sharma · Checked Nov 16, 2026 · How we review
A pretrained model predicts likely text, which is not the same as giving a helpful, honest answer. RLHF is one way to close that gap. It was made widely known by OpenAI's InstructGPT paper (Ouyang et al., 2022), which turned GPT-3 into a model that follows instructions.
How it works
The InstructGPT recipe has three steps. First, people write good answers to prompts and the model is supervised fine-tuned on them. Second, the model writes several answers per prompt and labelers rank them; a separate reward model is trained to predict those rankings. Third, the model is trained with reinforcement learning (PPO in the paper) to produce answers the reward model scores highly, while a penalty keeps it from drifting too far from the starting model.
In the paper's human evaluations, outputs from a 1.3B parameter InstructGPT model were preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters.
Limits
RLHF is costly and fiddly. It needs a lot of human labelling, a second model, and an RL loop that can be unstable. The model can learn to exploit weaknesses in the reward model rather than truly improve. Labelers' preferences also shape the result, which is part of the wider AI alignment debate.
RLHF and DPO
Direct preference optimization uses the same kind of preference data but skips the separate reward model and the RL loop, which is why many open models now use it or a variant instead.
Related terms
- Direct preference optimization
- Direct preference optimization (DPO) is a way to align a language model with human preferences by training directly on pairs of preferred and rejected answers, using a simple classification-style loss instead of a reward model and reinforcement learning.
- Reward model
- A reward model is a separate model trained on human preference data to score how good a response is. Reinforcement learning then uses that score to tune a language model towards answers people prefer.
- Supervised fine-tuning
- Supervised fine-tuning (SFT) trains a pretrained model further on example prompts paired with the responses you want. It is the usual first step that turns a base model into one that follows instructions.
- Fine-tuning
- Fine-tuning is further training of a pretrained model on a smaller, task-specific dataset, so the model adopts a particular skill, format, tone or domain without being trained from scratch.
- AI alignment
- AI alignment is the work of making an AI system's behaviour match what its users and developers actually intend, so it is helpful, honest and avoids harm, rather than just predicting likely text or optimising a narrow score.