Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Reward model

A reward model is a separate model trained on human preference data to score how good a response is. Reinforcement learning then uses that score to tune a language model towards answers people prefer.

Also called: preference model

By Divya Sharma · Checked Nov 15, 2026 · How we review

People cannot rate millions of model outputs during training, so a reward model stands in for them. It is usually a language model with its final layer replaced by a single number output. It learns from comparisons: shown a prompt and two responses, plus which one a person preferred, it is trained to give the preferred one a higher score.

Where it is used

Reward models are the core of RLHF. In OpenAI's summarisation work and later InstructGPT, people compared or ranked outputs, a model learned to predict their preference, and that model then served as the reward function for reinforcement learning on the policy model. The policy is usually the result of supervised fine-tuning.

They are also used outside training: to pick the best of several sampled answers at inference time, which is one form of test-time compute, and to filter or rank synthetic data.

Reward model versus DPO

DPO skips the separate reward model and trains the language model on preference pairs directly. It is simpler to run, which is why many open models use it. A separate reward model is still useful when you want a reusable scorer or online reinforcement learning.

Limits

A reward model is only an approximation of what people want. A policy optimised hard against it can find outputs that score well but are not actually better, such as long or flattering answers. This is called reward hacking, and training pipelines limit it by penalising the policy for drifting too far from its starting point.

Related terms

RLHF
RLHF (reinforcement learning from human feedback) fine-tunes a language model using human rankings of its answers: a reward model learns those preferences, then reinforcement learning pushes the model toward answers that score well.
Direct preference optimization
Direct preference optimization (DPO) is a way to align a language model with human preferences by training directly on pairs of preferred and rejected answers, using a simple classification-style loss instead of a reward model and reinforcement learning.
Supervised fine-tuning
Supervised fine-tuning (SFT) trains a pretrained model further on example prompts paired with the responses you want. It is the usual first step that turns a base model into one that follows instructions.
LLM-as-a-judge
LLM-as-a-judge is an evaluation method where a strong language model grades the outputs of another model or application, using a rubric, a reference answer or a side-by-side comparison, in place of or alongside human reviewers.
AI alignment
AI alignment is the work of making an AI system's behaviour match what its users and developers actually intend, so it is helpful, honest and avoids harm, rather than just predicting likely text or optimising a narrow score.

Sources