Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Supervised fine-tuning

Supervised fine-tuning (SFT) trains a pretrained model further on example prompts paired with the responses you want. It is the usual first step that turns a base model into one that follows instructions.

Also called: SFT, instruction tuning

By Divya Sharma · Checked Nov 15, 2026 · How we review

A base model fresh from pretraining predicts the next token of internet text. It does not reliably answer questions or follow a format. Supervised fine-tuning fixes that by showing the model many demonstrations: a prompt, and the exact response a person would want. The model is trained to reproduce those responses.

How it fits the training pipeline

In the InstructGPT recipe that most chat models follow, SFT comes first. Labelers wrote demonstrations of the desired behaviour and the model was fine-tuned on them. Only then came RLHF: people ranked model outputs, a reward model learned from those rankings, and reinforcement learning tuned the SFT model further. Newer pipelines often swap RLHF for DPO, but SFT is still the starting point. The output is an instruct model.

For builders

When you fine-tune an open model on your own data, you are almost always doing SFT. The data is a file of input and output pairs, often in a chat format. Quality matters more than volume: a few thousand clean, consistent examples usually beat a large noisy set. Libraries such as TRL provide an SFT trainer, and SFT is often combined with a parameter-efficient method such as LoRA.

Limits

SFT teaches the model to imitate. It cannot easily express "this answer is better than that one", which is what preference methods add. It can also make a model confidently copy errors or gaps in the training examples.

Related terms

Fine-tuning
Fine-tuning is further training of a pretrained model on a smaller, task-specific dataset, so the model adopts a particular skill, format, tone or domain without being trained from scratch.
Instruct model
An instruct model is a base language model that has been further trained on instructions paired with good answers, and often on human feedback, so it follows requests instead of just continuing text.
RLHF
RLHF (reinforcement learning from human feedback) fine-tunes a language model using human rankings of its answers: a reward model learns those preferences, then reinforcement learning pushes the model toward answers that score well.
Direct preference optimization
Direct preference optimization (DPO) is a way to align a language model with human preferences by training directly on pairs of preferred and rejected answers, using a simple classification-style loss instead of a reward model and reinforcement learning.
Parameter-efficient fine-tuning
Parameter-efficient fine-tuning (PEFT) adapts a pretrained model by training only a small number of new or selected parameters, leaving the rest frozen. It cuts the memory, compute and storage that full fine-tuning needs.
Base model
A base model is a language model straight out of pretraining, before instruction tuning or human-feedback training. It continues whatever text it is given rather than following instructions or holding a conversation.

Sources