Model serving
Model serving is running a trained model as a service that applications call over a network, with software that loads the weights onto GPUs, queues and batches requests, manages memory and returns results.
Also called: inference server, serving engine
By Divya Sharma · Checked Nov 16, 2026 · How we review
Calling `generate()` in a notebook works for one user. A product needs a server that many users can hit at once without running out of memory or making everyone wait. That software is a serving engine, also called an inference server. Common open-source examples for LLMs are vLLM, SGLang and llama.cpp's server.
What a serving engine does
It loads model weights into VRAM, often split across several GPUs with tensor parallelism. It accepts requests, usually through an OpenAI-compatible API, and streams tokens back. The main job is throughput: the PagedAttention paper notes that high-throughput serving requires batching many requests at once. Engines use continuous batching so new requests join a running batch, and manage the KV cache carefully, since wasted cache memory limits how many requests fit.
Self-hosted or managed
You can run a serving engine on your own machines or rented GPUs (self-hosting), on a serverless GPU platform, or skip it and pay an inference provider per token. Self-hosting gives control over models, data and cost at high volume, but you take on capacity planning, upgrades and monitoring. The inference serving guide compares options.
What to measure
Track time to first token, tokens per second per user and per system, error rate, and GPU memory use. Tuning for one, such as larger batches for throughput, often worsens another, such as latency for each user.
Related terms
- Inference
- Inference is running a trained model to produce output from new input, such as an LLM generating a reply to a prompt. It is the stage users pay for every time they call a model, as opposed to training.
- KV cache
- A KV cache stores the attention keys and values a transformer has already computed for earlier tokens, so each new token only needs work for itself. It speeds up generation but takes GPU memory that grows with context length.
- Continuous batching
- Continuous batching is a model-serving technique that adds new requests to the running GPU batch and removes finished ones after every generation step, instead of waiting for a whole batch to complete. It keeps the GPU busy and raises throughput.
- Self-hosting (AI models)
- Self-hosting an AI model means running it on hardware you control, such as your own servers, rented GPUs or a laptop, instead of calling a provider's API. You choose the model and keep the data, but you run the infrastructure.
- Inference provider
- An inference provider is a company that runs AI models on its own hardware and sells access to them through an API, usually charged per token. You send prompts and get outputs without managing GPUs yourself.
- OpenAI-compatible API
- An OpenAI-compatible API is an endpoint from another provider or tool that accepts the same request and response format as OpenAI's API, so code written for OpenAI can switch to it by changing the base URL, key and model name.