Knowledge distillation
Knowledge distillation is a training method in which a smaller student model learns to imitate a larger teacher model's outputs, producing a distilled model that keeps much of the teacher's skill at lower cost.
Also called: distilled model, distillation
By Divya Sharma · Checked Nov 15, 2026 · How we review
Distillation is how many small language models get good. Instead of learning only from raw data, the student learns from what a stronger model does.
How it works
The idea was set out by Hinton, Vinyals and Dean in 2015. They showed that the knowledge in a large model, or an ensemble of models, can be compressed into a single smaller model that is easier to deploy. In the classic form the student is trained to match the teacher's full probability distribution over answers, its soft targets, not just the single right label. Those probabilities carry extra signal about which wrong answers are nearly right.
With language models, a common modern form is simpler: the teacher generates answers or reasoning traces, and the student is fine-tuned on them as synthetic data.
Example
Model names that include "Distill" usually mean a small open model was fine-tuned on the outputs of a larger one. DeepSeek, for example, reports distilling the reasoning patterns of its large R1 model into smaller models.
Limits
A student rarely matches its teacher on hard or unusual tasks. Distilling from a commercial model's outputs may also break that provider's terms of service, so check them first.
Related terms
- Small language model
- A small language model (SLM) is a language model with relatively few parameters, usually a few billion or fewer, built to run on a laptop, phone or single GPU at lower cost than a large model.
- Fine-tuning
- Fine-tuning is further training of a pretrained model on a smaller, task-specific dataset, so the model adopts a particular skill, format, tone or domain without being trained from scratch.
- Synthetic data
- Synthetic data is training or test data generated by software, often by another AI model, rather than collected from real-world sources. It is used to fill gaps, add examples cheaply, or avoid exposing real personal data.
- Supervised fine-tuning
- Supervised fine-tuning (SFT) trains a pretrained model further on example prompts paired with the responses you want. It is the usual first step that turns a base model into one that follows instructions.
- Reasoning model
- A reasoning model is a language model trained to work through a problem in a stream of intermediate thinking tokens before giving its final answer, which tends to help on maths, coding and multi-step tasks.