Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

AI alignment

AI alignment is the work of making an AI system's behaviour match what its users and developers actually intend, so it is helpful, honest and avoids harm, rather than just predicting likely text or optimising a narrow score.

Also called: alignment, model alignment

By Divya Sharma · Checked Nov 15, 2026 · How we review

A model trained only to predict the next word learns to imitate the internet, including its errors and its worst content. The InstructGPT paper put the problem plainly: making language models bigger does not by itself make them better at following a user's intent, and such models can be untruthful, toxic or unhelpful. In that sense they are not aligned with their users.

How models are aligned today

Most chat models go through post-training after pretraining. First, supervised fine-tuning on examples of good answers. Then preference training: people rank pairs of answers, and the model is tuned towards the preferred ones, through RLHF with a reward model or directly with methods like DPO. Labs also write rules or principles the model is trained to follow, and test the result through red teaming.

Two senses of the word

In product work, alignment usually means the practical goal above: a model that follows instructions, refuses clearly harmful requests and admits uncertainty. In AI safety research it also covers a longer-term question: how to keep much more capable systems pursuing the goals people intended, when those goals are hard to specify and the system may find shortcuts. The two overlap but are not the same.

Limits

Alignment training changes behaviour, not underlying knowledge, and it can be bypassed. A jailbreak is evidence of an alignment gap. Preference training can also teach a model to please raters, for example by being agreeable or verbose, rather than to be right. That is why applications still add their own guardrails and evaluations on top.

Related terms

RLHF
RLHF (reinforcement learning from human feedback) fine-tunes a language model using human rankings of its answers: a reward model learns those preferences, then reinforcement learning pushes the model toward answers that score well.
AI red teaming
AI red teaming is the practice of deliberately attacking a model or AI application before and after release, with human testers or automated tools, to find harmful outputs, jailbreaks, injections and other failures so they can be fixed.
Direct preference optimization
Direct preference optimization (DPO) is a way to align a language model with human preferences by training directly on pairs of preferred and rejected answers, using a simple classification-style loss instead of a reward model and reinforcement learning.
Reward model
A reward model is a separate model trained on human preference data to score how good a response is. Reinforcement learning then uses that score to tune a language model towards answers people prefer.
Jailbreak (LLM)
A jailbreak is a prompt or technique that gets a language model to produce output its safety training is meant to refuse, such as harmful instructions, by role-play, obfuscation, odd encodings or long adversarial text.

Sources