Autoregressive model
An autoregressive model generates a sequence one piece at a time, predicting each next token from all the tokens before it and feeding that output back in. Most large language models work this way.
Also called: next-token prediction, ARM, causal language model
By Divya Sharma · Checked Nov 15, 2026 · How we review
Autoregressive means the model's output becomes part of its next input. To write a sentence, the model predicts one token, appends it, and predicts again, until it produces a stop token or hits a limit. GPT-3, for example, is described by its authors as an autoregressive language model.
Next-token prediction
Next-token prediction is the training goal behind this. During pretraining the model sees huge amounts of text and learns to give a probability to every possible next token. At generation time, a sampling setting such as temperature picks one token from that distribution. A transformer uses a mask so each position can only attend to earlier positions, which keeps training consistent with how it generates.
What it means in practice
Because tokens come out one after another, response time depends on output length. That is one reason output tokens are often priced higher than input tokens, and why streaming is useful: you can show text as it arrives. Speed-ups such as speculative decoding and the KV cache all work around this one-at-a-time loop.
The model also cannot go back. Once a token is chosen it stays, so an early mistake can carry through the answer. Reasoning models partly work around this by writing out steps before the final answer.
The alternative
Diffusion language models start from a fully masked or noisy sequence and fill in many positions in parallel over several steps. The LLaDA paper challenges the common view that core LLM abilities depend on being autoregressive.
Related terms
- Diffusion language model
- A diffusion language model (dLLM) generates text by starting from a fully masked or noisy sequence and refining many token positions in parallel over several steps, instead of writing one token at a time.
- Transformer (architecture)
- The transformer is a neural network architecture, introduced in 2017, that processes sequences using attention instead of recurrence. It is the basis of nearly all current large language models and many image and audio models.
- Token (LLM)
- A token (LLM) is the basic unit of text a language model reads and writes, often a word, part of a word, a punctuation mark or a byte. Prices, speed and context limits are all counted in tokens.
- Speculative decoding
- Speculative decoding speeds up text generation by letting a small, fast draft model propose several tokens, which the large model then checks in a single pass, keeping the accepted ones. The output matches what the large model alone would produce.
- Output tokens
- Output tokens are the tokens a language model generates in its reply. They are counted and billed separately from input tokens, and a max_tokens setting caps how many one response may contain.