Speculative decoding
Speculative decoding speeds up text generation by letting a small, fast draft model propose several tokens, which the large model then checks in a single pass, keeping the accepted ones. The output matches what the large model alone would produce.
Also called: draft model
By Divya Sharma · Checked Nov 16, 2026 · How we review
A large language model normally generates one token per forward pass, and each pass reads all of its weights from memory. That makes decoding slow even when the GPU has spare compute. Speculative decoding, described by Leviathan et al. in 2022, uses that spare compute.
How it works
A cheaper draft model guesses the next few tokens. The large target model then scores all of those positions at once, in parallel. Tokens it agrees with are kept; at the first disagreement the target model's own choice is used and the rest of the draft is thrown away. A special sampling rule ensures the final text follows the same distribution as the target model, so quality does not change. The paper reports a 2X-3X speedup on T5-XXL with identical outputs.
When it helps
The gain depends on how often the draft is right. It works best on predictable text such as code, structured output, or answers that repeat parts of the prompt. Variants skip the separate draft model: some add extra prediction heads to the main model, others look up likely continuations in the prompt itself. Serving engines such as vLLM and SGLang support several methods.
Costs
The draft model needs its own memory, and wrong guesses waste compute. Under heavy batched load, where the GPU is already busy, the benefit shrinks. Measure tokens per second on your own traffic before and after. For the wider picture, see inference and model serving.
Related terms
- Inference
- Inference is running a trained model to produce output from new input, such as an LLM generating a reply to a prompt. It is the stage users pay for every time they call a model, as opposed to training.
- Tokens per second
- Tokens per second (tok/s) measures how fast a language model produces output. It is quoted either per user, the speed one reply streams at, or per system, the total output across all concurrent requests.
- Model serving
- Model serving is running a trained model as a service that applications call over a network, with software that loads the weights onto GPUs, queues and batches requests, manages memory and returns results.
- Autoregressive model
- An autoregressive model generates a sequence one piece at a time, predicting each next token from all the tokens before it and feeding that output back in. Most large language models work this way.
- Knowledge distillation
- Knowledge distillation is a training method in which a smaller student model learns to imitate a larger teacher model's outputs, producing a distilled model that keeps much of the teacher's skill at lower cost.