AI alignment
AI alignment is the work of making an AI system's behaviour match what its users and developers actually intend, so it is helpful, honest and avoids harm, rather than just predicting likely text or optimising a narrow score.
Also called: alignment, model alignment
By Divya Sharma · Checked Nov 15, 2026 · How we review
A model trained only to predict the next word learns to imitate the internet, including its errors and its worst content. The InstructGPT paper put the problem plainly: making language models bigger does not by itself make them better at following a user's intent, and such models can be untruthful, toxic or unhelpful. In that sense they are not aligned with their users.
How models are aligned today
Most chat models go through post-training after pretraining. First, supervised fine-tuning on examples of good answers. Then preference training: people rank pairs of answers, and the model is tuned towards the preferred ones, through RLHF with a reward model or directly with methods like DPO. Labs also write rules or principles the model is trained to follow, and test the result through red teaming.
Two senses of the word
In product work, alignment usually means the practical goal above: a model that follows instructions, refuses clearly harmful requests and admits uncertainty. In AI safety research it also covers a longer-term question: how to keep much more capable systems pursuing the goals people intended, when those goals are hard to specify and the system may find shortcuts. The two overlap but are not the same.
Limits
Alignment training changes behaviour, not underlying knowledge, and it can be bypassed. A jailbreak is evidence of an alignment gap. Preference training can also teach a model to please raters, for example by being agreeable or verbose, rather than to be right. That is why applications still add their own guardrails and evaluations on top.
Related terms
- RLHF
- RLHF (reinforcement learning from human feedback) fine-tunes a language model using human rankings of its answers: a reward model learns those preferences, then reinforcement learning pushes the model toward answers that score well.
- AI red teaming
- AI red teaming is the practice of deliberately attacking a model or AI application before and after release, with human testers or automated tools, to find harmful outputs, jailbreaks, injections and other failures so they can be fixed.
- Direct preference optimization
- Direct preference optimization (DPO) is a way to align a language model with human preferences by training directly on pairs of preferred and rejected answers, using a simple classification-style loss instead of a reward model and reinforcement learning.
- Reward model
- A reward model is a separate model trained on human preference data to score how good a response is. Reinforcement learning then uses that score to tune a language model towards answers people prefer.
- Jailbreak (LLM)
- A jailbreak is a prompt or technique that gets a language model to produce output its safety training is meant to refuse, such as harmful instructions, by role-play, obfuscation, odd encodings or long adversarial text.