Skip to content
ReviewRamble

vLLM

High-throughput LLM serving engine for GPUs, with an OpenAI-compatible server and wide model support.

Reviewed by Sheela · Updated

View on GitHub Open source

Our rating

4.7/5

Still the engine to put an open model behind a multi-user API, provided someone on the team owns its upgrades and its network exposure.

What it is

vLLM is a serving engine for running language models in production. It started in UC Berkeley's Sky Computing Lab and is known for PagedAttention, a way of managing GPU memory that lets one server handle many requests at once. More than 2,000 people have contributed.

It provides an OpenAI-compatible API server (plus Anthropic Messages and gRPC), continuous batching, prefix caching, speculative decoding, many quantisation formats and tensor, pipeline and expert parallelism for spreading large models over several GPUs. It runs on Linux with NVIDIA GPUs of compute capability 7.5 or newer, AMD GPUs on ROCm 6.3 or newer, and Intel GPUs, with plugins for TPUs and other accelerators.

The release cadence is fast and breaking: v0.32.0, published on 18 October 2026, bundled 814 commits from 363 contributors and dropped Python 3.10.

Our review

vLLM is the standard way to serve open-weight models to many concurrent users, and the question for most teams is no longer whether it is fast enough but whether they can run it safely. It has about 95,200 GitHub stars, supports more than 200 model architectures, and has shipped a new minor version roughly every two weeks since July 2026.

PagedAttention stores the attention cache in small blocks so GPU memory is not stranded, which raises the number of requests a card can hold at once. Continuous batching, chunked prefill and prefix caching keep it busy. The serving API speaks OpenAI's format, so most client code needs only a new base URL, and quantisation options from FP8 and NVFP4 to GPTQ, AWQ and GGUF let a model fit on fewer cards.

The security model needs reading before deployment. Per vLLM's own security guide, --api-key protects only the /v1, /v2, /inference and /cohere routes; /invocations gives unauthenticated inference, /pause can take the server down, and traffic between nodes in a multi-node setup is unencrypted and unauthenticated by default. GitHub lists 86 advisories for the project in 2026 alone, most of them single-request crashes of the engine, plus a critical OpenAI API authentication bypass in June. Anonymous usage statistics are also collected by default.

The other cost is churn. v0.32.0 removed cascade attention, dropped Python 3.10 and changed max_tokens from a reserved budget to an upper bound, any of which can alter a production deployment. SGLang's authors report up to 6.4 times the throughput of other engines on prefix-heavy workloads, and NVIDIA tunes TensorRT-LLM for its own GPUs first; neither claim replaces a benchmark on your own traffic. vLLM's lasting advantage is breadth: new open-weight models usually document vLLM support at launch, and that matters more week to week than a few per cent of throughput.

Where it earns its place

  • A platform engineer putting an open-weight model behind an internal OpenAI-compatible endpoint

    Replace a hosted API for an internal assistant without rewriting client code

    vllm serve exposes /v1/chat/completions, so existing OpenAI SDK code changes only its base URL. One on-demand H100 PCIe on Lambda costs $3.29 an hour, about $2,400 a month if left running, plus a reverse proxy that allowlists only the routes users need.

  • An ML lead at a SaaS company paying per token for a high-volume extraction job

    Cut inference spend on millions of short, similar requests

    Continuous batching and prefix caching keep a GPU saturated across many short prompts that share a system prompt. Savings depend on keeping the card busy around the clock, so measure utilisation on a week of real traffic before cancelling the API contract.

  • A research group evaluating many fine-tuned checkpoints

    Run large offline evaluation batches on a shared GPU cluster

    The offline LLM class processes a dataset without running a server, and multi-LoRA serving lets one base model carry many adapters. Pin the vLLM version per study: v0.32.0 changed how max_tokens interacts with the context window.

Before you commit

Setup
uv pip install vllm, then vllm download-kernels, on Linux with Python 3.11 to 3.14 and an NVIDIA GPU of compute capability 7.5 or newer (T4 upward). AMD ROCm wheels exist only for Python 3.12; Windows needs WSL.
Integrations
OpenAI-compatible and Anthropic Messages APIs, gRPC, tool-calling and reasoning parsers, structured output via xgrammar or guidance, and Prometheus metrics. Hugging Face model IDs load directly.
Data and privacy
Collects anonymous usage statistics on hardware and model configuration by default; set VLLM_NO_USAGE_STATS=1 or DO_NOT_TRACK=1 to stop it. Prompts stay on your servers, but the API key does not cover every route, and 86 GitHub advisories were published in 2026, including a critical auth bypass.
Cost at scale
No paid edition from the project. The cost is GPUs: on Lambda an H100 PCIe is $3.29 per hour, an 8x H100 SXM node $3.99 per GPU-hour and a B200 from $6.69 per GPU-hour, plus engineering time for tuning and upgrades.
Limits
9,343 open issues and pull requests. Minor releases about every two weeks routinely remove flags and change API behaviour; multi-node traffic is insecure unless you isolate the network yourself.

What's good

  • PagedAttention and continuous batching give high throughput per GPU
  • OpenAI- and Anthropic-compatible APIs
  • 200+ model architectures, usually supported at launch
  • NVIDIA, AMD and Intel GPUs from one codebase

What's not

  • --api-key leaves several endpoints unauthenticated; needs a reverse proxy
  • Breaking changes in most minor releases
  • Linux only; no native Windows, laptops are awkward

Best for

Teams serving open models to many concurrent users on GPU servers.

Alternatives

SGLang

Run models locallyPython38k stars

4.4/5

Fast serving framework for LLMs and multimodal models, built around RadixAttention prefix caching.

TensorRT-LLM

Run models locallyPython15k stars

4.0/5

NVIDIA's library for optimized LLM inference on its GPUs, with a Python API and production runtimes.

llama.cpp

Run models locallyC++132k stars

4.8/5

C/C++ engine for running LLMs locally on CPUs, Apple silicon and most GPUs, home of the GGUF model format.

vLLM: common questions

Is vLLM worth it?
Still the engine to put an open model behind a multi-user API, provided someone on the team owns its upgrades and its network exposure. We rate it 4.7 out of 5.
How much does vLLM cost?
Free and open source under the Apache 2.0 licence. You pay for the GPUs it runs on.
What are the downsides of vLLM?
--api-key leaves several endpoints unauthenticated; needs a reverse proxy; breaking changes in most minor releases; linux only; no native Windows, laptops are awkward.
Who is vLLM best for?
Teams serving open models to many concurrent users on GPU servers.
How much does vLLM cost for a team or in production?
No paid edition from the project. The cost is GPUs: on Lambda an H100 PCIe is $3.29 per hour, an 8x H100 SXM node $3.99 per GPU-hour and a B200 from $6.69 per GPU-hour, plus engineering time for tuning and upgrades.
Is vLLM safe for company data?
Collects anonymous usage statistics on hardware and model configuration by default; set VLLM_NO_USAGE_STATS=1 or DO_NOT_TRACK=1 to stop it. Prompts stay on your servers, but the API key does not cover every route, and 86 GitHub advisories were published in 2026, including a critical auth bypass.
What are the best vLLM alternatives?
The closest alternatives we have reviewed are SGLang, TensorRT-LLM and llama.cpp.

Answers checked Oct 23, 2026.

Make vLLM? Show this rating on your sitevLLM rated 4.7 out of 5 on ReviewRamble