Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Model serving

Model serving is running a trained model as a service that applications call over a network, with software that loads the weights onto GPUs, queues and batches requests, manages memory and returns results.

Also called: inference server, serving engine

By Divya Sharma · Checked Nov 16, 2026 · How we review

Calling `generate()` in a notebook works for one user. A product needs a server that many users can hit at once without running out of memory or making everyone wait. That software is a serving engine, also called an inference server. Common open-source examples for LLMs are vLLM, SGLang and llama.cpp's server.

What a serving engine does

It loads model weights into VRAM, often split across several GPUs with tensor parallelism. It accepts requests, usually through an OpenAI-compatible API, and streams tokens back. The main job is throughput: the PagedAttention paper notes that high-throughput serving requires batching many requests at once. Engines use continuous batching so new requests join a running batch, and manage the KV cache carefully, since wasted cache memory limits how many requests fit.

Self-hosted or managed

You can run a serving engine on your own machines or rented GPUs (self-hosting), on a serverless GPU platform, or skip it and pay an inference provider per token. Self-hosting gives control over models, data and cost at high volume, but you take on capacity planning, upgrades and monitoring. The inference serving guide compares options.

What to measure

Track time to first token, tokens per second per user and per system, error rate, and GPU memory use. Tuning for one, such as larger batches for throughput, often worsens another, such as latency for each user.

Related terms

Inference
Inference is running a trained model to produce output from new input, such as an LLM generating a reply to a prompt. It is the stage users pay for every time they call a model, as opposed to training.
KV cache
A KV cache stores the attention keys and values a transformer has already computed for earlier tokens, so each new token only needs work for itself. It speeds up generation but takes GPU memory that grows with context length.
Continuous batching
Continuous batching is a model-serving technique that adds new requests to the running GPU batch and removes finished ones after every generation step, instead of waiting for a whole batch to complete. It keeps the GPU busy and raises throughput.
Self-hosting (AI models)
Self-hosting an AI model means running it on hardware you control, such as your own servers, rented GPUs or a laptop, instead of calling a provider's API. You choose the model and keep the data, but you run the infrastructure.
Inference provider
An inference provider is a company that runs AI models on its own hardware and sells access to them through an API, usually charged per token. You send prompts and get outputs without managing GPUs yourself.
OpenAI-compatible API
An OpenAI-compatible API is an endpoint from another provider or tool that accepts the same request and response format as OpenAI's API, so code written for OpenAI can switch to it by changing the base URL, key and model name.

Sources