Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Knowledge distillation

Knowledge distillation is a training method in which a smaller student model learns to imitate a larger teacher model's outputs, producing a distilled model that keeps much of the teacher's skill at lower cost.

Also called: distilled model, distillation

By Divya Sharma · Checked Nov 15, 2026 · How we review

Distillation is how many small language models get good. Instead of learning only from raw data, the student learns from what a stronger model does.

How it works

The idea was set out by Hinton, Vinyals and Dean in 2015. They showed that the knowledge in a large model, or an ensemble of models, can be compressed into a single smaller model that is easier to deploy. In the classic form the student is trained to match the teacher's full probability distribution over answers, its soft targets, not just the single right label. Those probabilities carry extra signal about which wrong answers are nearly right.

With language models, a common modern form is simpler: the teacher generates answers or reasoning traces, and the student is fine-tuned on them as synthetic data.

Example

Model names that include "Distill" usually mean a small open model was fine-tuned on the outputs of a larger one. DeepSeek, for example, reports distilling the reasoning patterns of its large R1 model into smaller models.

Limits

A student rarely matches its teacher on hard or unusual tasks. Distilling from a commercial model's outputs may also break that provider's terms of service, so check them first.

Related terms

Small language model
A small language model (SLM) is a language model with relatively few parameters, usually a few billion or fewer, built to run on a laptop, phone or single GPU at lower cost than a large model.
Fine-tuning
Fine-tuning is further training of a pretrained model on a smaller, task-specific dataset, so the model adopts a particular skill, format, tone or domain without being trained from scratch.
Synthetic data
Synthetic data is training or test data generated by software, often by another AI model, rather than collected from real-world sources. It is used to fill gaps, add examples cheaply, or avoid exposing real personal data.
Supervised fine-tuning
Supervised fine-tuning (SFT) trains a pretrained model further on example prompts paired with the responses you want. It is the usual first step that turns a base model into one that follows instructions.
Reasoning model
A reasoning model is a language model trained to work through a problem in a stream of intermediate thinking tokens before giving its final answer, which tends to help on maths, coding and multi-step tasks.

Sources