Streaming (LLM output)
Streaming sends an LLM's output to the client piece by piece as tokens are generated, instead of waiting for the full response. Users see text appear within moments, even when the complete answer takes much longer.
Also called: token streaming, streaming responses, server-sent events, SSE
By Divya Sharma · Checked Nov 15, 2026 · How we review
Language models generate one token at a time. A long answer can take many seconds to finish, and without streaming the user stares at a spinner for all of it. With streaming, the first words arrive as soon as they exist, so the app feels fast even though the total time is the same.
How it works
You set a flag such as `"stream": true` on the request. The provider keeps the HTTP connection open and sends a sequence of small events, usually as server-sent events (SSE), each carrying the next fragment of text. Separate events mark the start and end of the message, and some APIs also stream tool calls and reasoning. Official SDKs collect the fragments for you. Most self-hosted servers that offer an OpenAI-compatible API support the same pattern.
What it changes
Two numbers describe a streamed response: time to first token, which is what users perceive as responsiveness, and tokens per second, which decides how fast the rest arrives. Streaming improves the first without changing the second.
Things to handle
Your interface has to cope with partial text: rendering markdown that is half written, and handling a connection that drops halfway. Errors can arrive in the middle of a stream after you have already shown text. If you need structured output such as JSON, you either wait for the end to parse it or use a parser that accepts partial input. Content filters and guardrails that check the whole answer are harder to apply when text is already on screen.
Related terms
- Time to first token
- Time to first token (TTFT) is how long a user waits between sending a request to a language model and receiving the first token of its reply, covering queuing, prompt processing and network time.
- Tokens per second
- Tokens per second (tok/s) measures how fast a language model produces output. It is quoted either per user, the speed one reply streams at, or per system, the total output across all concurrent requests.
- Output tokens
- Output tokens are the tokens a language model generates in its reply. They are counted and billed separately from input tokens, and a max_tokens setting caps how many one response may contain.
- OpenAI-compatible API
- An OpenAI-compatible API is an endpoint from another provider or tool that accepts the same request and response format as OpenAI's API, so code written for OpenAI can switch to it by changing the base URL, key and model name.
- Structured output
- Structured output is an API feature that makes a language model's reply follow a format you define, usually a JSON schema, so that code can parse it reliably instead of scraping free text.