Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Inference

Inference is running a trained model to produce output from new input, such as an LLM generating a reply to a prompt. It is the stage users pay for every time they call a model, as opposed to training.

Also called: model inference

By Divya Sharma · Checked Nov 16, 2026 · How we review

Training sets a model's weights; inference uses them. For a large language model, inference means taking a prompt, splitting it into tokens, and predicting the next token again and again, each time feeding the new token back in, until the model stops or hits a length limit.

Two phases

LLM inference has a prefill phase and a decode phase. Prefill reads the whole prompt at once and builds the KV cache; it decides time to first token. Decode then produces one token per step, and its speed is measured in tokens per second. Prefill is mostly limited by compute, decode mostly by how fast the GPU can read weights from memory.

Why it matters to builders

Training happens once; inference happens on every request, so it usually dominates the running cost of an AI product. You can buy it per token from an inference provider, or run it yourself with a model serving engine such as vLLM. Choices like quantization, batching and speculative decoding trade speed, cost and quality against each other. The inference serving guide compares options.

Not the same as reasoning

"Inference" here is an engineering word for running a model. It does not mean the model is drawing logical inferences. Reasoning models spend more inference compute per answer, which is called test-time compute.

Related terms

Model serving
Model serving is running a trained model as a service that applications call over a network, with software that loads the weights onto GPUs, queues and batches requests, manages memory and returns results.
Time to first token
Time to first token (TTFT) is how long a user waits between sending a request to a language model and receiving the first token of its reply, covering queuing, prompt processing and network time.
Tokens per second
Tokens per second (tok/s) measures how fast a language model produces output. It is quoted either per user, the speed one reply streams at, or per system, the total output across all concurrent requests.
KV cache
A KV cache stores the attention keys and values a transformer has already computed for earlier tokens, so each new token only needs work for itself. It speeds up generation but takes GPU memory that grows with context length.
Inference provider
An inference provider is a company that runs AI models on its own hardware and sells access to them through an API, usually charged per token. You send prompts and get outputs without managing GPUs yourself.

Sources