Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Top-p sampling

Top-p sampling, or nucleus sampling, picks each next token only from the smallest set of candidates whose combined probability reaches a threshold p, cutting off the long tail of unlikely tokens.

Also called: nucleus sampling, top-k, top-k sampling

By Divya Sharma · Checked Nov 15, 2026 · How we review

When a model generates text, every possible next token has some probability. Sampling from all of them lets rare, odd tokens slip in. Always taking the single most likely token tends to produce bland, repetitive text. Top-p sampling sits between the two.

How it works

Sort the candidate tokens from most to least likely. Add up their probabilities until the total passes p, for example 0.9. Keep that set, discard the rest, and sample from what is kept. When the model is confident, the set may hold only a few tokens. When it is unsure, the set grows. That adaptive size is the main difference from top-k.

Where it comes from

Holtzman and colleagues introduced nucleus sampling in "The Curious Case of Neural Text Degeneration" (2019). They found that using likelihood as the decoding objective leads to bland, repetitive text, and that sampling from the dynamic nucleus of the distribution keeps variety while removing the less reliable tail.

Top-k sampling

Top-k keeps a fixed number of candidates, the k most likely tokens, however the probability is spread. It is simpler but can keep too many options when the model is sure and too few when it is not. Some APIs expose both settings.

How it differs from temperature

Temperature reshapes the whole distribution; top-p truncates it. Both run at inference time and change nothing in the model itself.

Why it matters

For most builders these are tuning knobs, not design decisions. Defaults are usually sensible. Lower values suit tasks with one right answer; higher values suit open-ended writing.

Related terms

Temperature (sampling)
Temperature is a sampling setting that controls how random a language model's word choices are. Low values make it favour the most likely tokens; higher values spread choices across less likely ones.
Large language model
A large language model (LLM) is a neural network with billions of parameters, trained on large amounts of text to predict the next token, which lets it write, answer questions, summarise and follow instructions.
Token (LLM)
A token (LLM) is the basic unit of text a language model reads and writes, often a word, part of a word, a punctuation mark or a byte. Prices, speed and context limits are all counted in tokens.
Inference
Inference is running a trained model to produce output from new input, such as an LLM generating a reply to a prompt. It is the stage users pay for every time they call a model, as opposed to training.

Sources