Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Continuous batching

Continuous batching is a model-serving technique that adds new requests to the running GPU batch and removes finished ones after every generation step, instead of waiting for a whole batch to complete. It keeps the GPU busy and raises throughput.

Also called: iteration-level scheduling, in-flight batching, dynamic batching

By Divya Sharma · Checked Nov 15, 2026 · How we review

A GPU is most efficient when it processes many requests at once. The simple way to do that, static batching, collects a group of requests, runs them together, and returns them when all are done. With language models that wastes a lot: one request may need a one-line answer and another a long essay, so short requests sit finished but unreturned while new arrivals wait for the batch to empty.

How it works

Generation happens one token per step. The Orca paper (OSDI 2022) proposed scheduling at the level of these steps rather than whole requests. After each step, the server returns any request that has finished and slots waiting requests into the free places. The batch is always full of active work.

The technique is now standard. vLLM lists continuous batching of incoming requests among its core features, and SGLang, TensorRT-LLM (which calls it in-flight batching) and Hugging Face TGI do it too.

Why it matters

For anyone self-hosting a model, continuous batching is a large part of why a modern model server handles many more users per GPU than a naive loop over `generate()`. It works best together with PagedAttention, because adding and removing requests every step needs flexible KV cache memory.

Not the same as a batch API

A batch API is a discounted, offline way to buy inference from a provider. Continuous batching is a scheduling method inside the server, used for live traffic.

Trade-off

More concurrent requests share the GPU, so each one generates a little slower. Servers expose settings for the maximum batch size and tokens per step so you can balance throughput against time to first token.

Related terms

Model serving
Model serving is running a trained model as a service that applications call over a network, with software that loads the weights onto GPUs, queues and batches requests, manages memory and returns results.
PagedAttention
PagedAttention is a method for storing a language model's KV cache in small fixed-size blocks that need not sit next to each other in GPU memory, like pages in an operating system. It cuts wasted memory so a server can run more requests at once.
KV cache
A KV cache stores the attention keys and values a transformer has already computed for earlier tokens, so each new token only needs work for itself. It speeds up generation but takes GPU memory that grows with context length.
Batch API
A batch API lets you submit a large set of model requests in one job and collect the results later, instead of waiting on each call. Providers run the job when capacity allows and usually charge less for it.
Tokens per second
Tokens per second (tok/s) measures how fast a language model produces output. It is quoted either per user, the speed one reply streams at, or per system, the total output across all concurrent requests.

Sources