Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Tokens per second

Tokens per second (tok/s) measures how fast a language model produces output. It is quoted either per user, the speed one reply streams at, or per system, the total output across all concurrent requests.

Also called: tok/s, throughput

By Divya Sharma · Checked Nov 16, 2026 · How we review

Once the first token has arrived, tokens per second decides how long the rest of the reply takes. A token is a chunk of text, often part of a word, so tok/s is not words per second, and the same text can give different counts on different tokenizers.

Per user and per system

These two numbers answer different questions. Per-user speed is what one person sees as a reply streams in. Per-system throughput, as NVIDIA's benchmarking documentation defines it, is total output token throughput across all simultaneous requests. Adding more concurrent requests usually raises system throughput until the GPU is saturated, while each user's stream gets slower. Providers and benchmarks do not always say which one they mean.

What limits it

During decoding, each new token needs the model's weights to be read from memory, so speed depends heavily on memory bandwidth and model size. That is why quantization, smaller models and mixture-of-experts models, which use only part of their weights per token, often generate faster. Speculative decoding and continuous batching are common ways to raise speed in a serving engine.

Using the number

For local models, our GPU calculator estimates whether a model fits in VRAM and how fast it may run. For hosted models, measure with your own prompts, since output length, prompt length and server load all change the result. Pair it with time to first token to estimate total response time.

Related terms

Time to first token
Time to first token (TTFT) is how long a user waits between sending a request to a language model and receiving the first token of its reply, covering queuing, prompt processing and network time.
VRAM
VRAM is the memory on a graphics card. For AI work it holds the model weights, the KV cache and working data, and whether a model fits in VRAM usually decides which GPU you need to run it.
Inference
Inference is running a trained model to produce output from new input, such as an LLM generating a reply to a prompt. It is the stage users pay for every time they call a model, as opposed to training.
Speculative decoding
Speculative decoding speeds up text generation by letting a small, fast draft model propose several tokens, which the large model then checks in a single pass, keeping the accepted ones. The output matches what the large model alone would produce.
Model serving
Model serving is running a trained model as a service that applications call over a network, with software that loads the weights onto GPUs, queues and batches requests, manages memory and returns results.

Sources