Time to first token
Time to first token (TTFT) is how long a user waits between sending a request to a language model and receiving the first token of its reply, covering queuing, prompt processing and network time.
Also called: TTFT, latency
By Divya Sharma · Checked Nov 16, 2026 · How we review
TTFT is the delay before anything appears on screen. For chat, voice and coding assistants it is often what makes a model feel fast or slow, even more than total generation time, because a streaming response lets users start reading as soon as the first token arrives.
What goes into it
NVIDIA's benchmarking documentation describes TTFT as generally including request queuing time, prefill time and network latency. Prefill is the step where the model reads the whole prompt and builds its KV cache, so longer prompts raise TTFT. A busy server adds queuing time. Distance to the data centre adds network time.
How it differs from throughput
After the first token, speed is measured as inter-token latency or tokens per second. A model can have a low TTFT and slow generation, or the reverse. Total response time is roughly TTFT plus the number of output tokens times the time per token. Reasoning models may also spend time on hidden thinking tokens before the visible answer begins, which users experience as a long TTFT.
How to lower it
Keep prompts short or reuse a fixed prefix with prompt caching, which skips most of the prefill work. Choose a smaller model or a provider region near users. When self-hosting, avoid overloading the server, since queuing quickly dominates. When comparing providers, measure TTFT at your own prompt lengths and at realistic load, and look at percentiles rather than the average. See the inference serving guide for engines and providers.
Related terms
- Tokens per second
- Tokens per second (tok/s) measures how fast a language model produces output. It is quoted either per user, the speed one reply streams at, or per system, the total output across all concurrent requests.
- Inference
- Inference is running a trained model to produce output from new input, such as an LLM generating a reply to a prompt. It is the stage users pay for every time they call a model, as opposed to training.
- KV cache
- A KV cache stores the attention keys and values a transformer has already computed for earlier tokens, so each new token only needs work for itself. It speeds up generation but takes GPU memory that grows with context length.
- Prompt caching
- Prompt caching is an API feature that stores the processed form of a repeated prompt prefix, such as a long system prompt or document, so later requests that start with the same prefix are cheaper and faster.
- Streaming (LLM output)
- Streaming sends an LLM's output to the client piece by piece as tokens are generated, instead of waiting for the full response. Users see text appear within moments, even when the complete answer takes much longer.