What it is
vLLM Semantic Router sits in front of a pool of models and decides which one should answer each request. Applications call one virtual model name, such as vllm-sr/auto; the router extracts signals from the request (topic, modality, safety, PII, user preferences), applies rules you write and sends the call to a backend that suits it on quality, cost, latency or location.
It is written in Go, Apache 2.0-licensed and part of the vLLM project. Version 0.4, named Hermes, was announced on 24 September 2026 with 982 commits from 130 contributors since the previous release. The project also trains its own small classifier models (Vela) for domain, jailbreak, PII and hallucination checks, and larger decision models, reviewed separately as d3.
Backends can be vLLM, SGLang or any OpenAI-compatible API, including closed models from OpenAI and Anthropic.
Our review
Semantic routing is the right idea for a narrow set of teams, and this is the strongest open implementation of it. If most of your traffic is simple and a small open model can answer it, sending only hard requests to an expensive frontier model saves money. The router makes that decision per request and can keep private data on your own GPUs while other calls go to the cloud.
The project moves quickly. It reached v0.1 in January 2026 and v0.4 in September, and the main branch changes daily. The cost is churn: the release notes describe new signals, plugins and experimental features (Fusion, Workflows) in each version, and the decision models the project promotes are not yet wired into the router itself, by its own account. Its benchmark numbers come from the project's own evaluations, and 490 issues are open.
The operational surface is larger than a gateway's. A local install runs the router, a dashboard on port 8700, an OpenAI-compatible API on 8899 and support services under Docker, and the hardening guide asks operators to authenticate the inference and management listeners and bind internal ports privately. On 18 October 2026 the project published a high-severity advisory: under Istio, a client header could bypass the router entirely because of a reused filter name.
Most teams do not need this. A static rule, such as one model for chat and another for code, or a gateway's fallback list, covers the common case with far less machinery. Routing by meaning only pays when traffic is large and varied enough that per-request choices change the bill, and when evaluations confirm cheaper routes do not lower quality. RouteLLM, the LMSYS research router often cited for the same idea, has had no commits since August 2024, which leaves this project with little open competition.
Where it earns its place
A platform team serving 10 million requests a month across a vLLM fleet and a frontier API
Send easy prompts to an in-house open model and only hard ones to the paid API
Applications call vllm-sr/auto and rules route by domain and difficulty. Savings depend on the share of traffic a small model can take, which needs an evaluation set to prove before switching.
A healthcare IT lead with patient data in prompts
Keep PII-bearing requests on-premise while other calls use cloud models
The PII classifier flags requests and routing rules pin them to local vLLM backends. The router is free; the cost is running it, the classifiers and the private GPUs.
An engineering manager at a 40-developer company using coding agents
Give agents an automatic model choice like Cursor's auto mode
Point the agent harness at the router's OpenAI-compatible endpoint. The project's OpenCode guide shows the pattern; expect config changes between releases.
Before you commit
- Setup
- curl -fsSL https://vllm-sr.ai/install.sh | bash, or pip install vllm-sr then vllm-sr serve; needs Linux, macOS or WSL2, Python 3.10+, and Docker or Podman. Kubernetes runs through a Helm chart or operator.
- Integrations
- Backends on vLLM, SGLang or any OpenAI-compatible API, including OpenAI and Anthropic; Envoy ext_proc gateway mode; Valkey for agentic memory; an experimental NVIDIA Dynamo example.
- Data and privacy
- Apache 2.0 with no hosted paid service. The router can store replay, response cache, memory and logs, which operators must secure and set retention for. One high-severity advisory (GHSA-xr66-pq9r-2c8j) covers a header-based bypass under Istio.
- Cost at scale
- Free software. The router runs on x86-64 CPUs, with optional CUDA or ROCm for its classifier models; the docs do not publish sizing, so cost is your containers plus whatever backends you route to.
- Limits
- 490 open issues and new features in each release; ARM, DGX Spark and AMD NPU support are build-qualified or experimental. Decision models are not yet integrated into the router itself.
What's good
- Routes by request content, not static rules
- Built-in safety, jailbreak and PII classifiers
- Part of the vLLM project; Apache 2.0
- Can keep sensitive calls on private hardware
What's not
- Changes quickly between releases
- More services to secure than a plain gateway
- Benchmarks are self-reported
- High-severity Istio bypass advisory in October 2026
Best for
Teams serving several models, often on their own vLLM fleet, with enough varied traffic to justify choosing a model per request.
Alternatives
vLLM Semantic Router: common questions
- Is vLLM Semantic Router worth it?
- An ambitious open router for choosing models per request, suited to teams already running vLLM who can absorb a fast-moving project. We rate it 3.8 out of 5.
- How much does vLLM Semantic Router cost?
- Free and open source under Apache 2.0. No hosted paid service.
- What are the downsides of vLLM Semantic Router?
- Changes quickly between releases; more services to secure than a plain gateway; benchmarks are self-reported; high-severity Istio bypass advisory in October 2026.
- Who is vLLM Semantic Router best for?
- Teams serving several models, often on their own vLLM fleet, with enough varied traffic to justify choosing a model per request.
- How much does vLLM Semantic Router cost for a team or in production?
- Free software. The router runs on x86-64 CPUs, with optional CUDA or ROCm for its classifier models; the docs do not publish sizing, so cost is your containers plus whatever backends you route to.
- Is vLLM Semantic Router safe for company data?
- Apache 2.0 with no hosted paid service. The router can store replay, response cache, memory and logs, which operators must secure and set retention for. One high-severity advisory (GHSA-xr66-pq9r-2c8j) covers a header-based bypass under Istio.
- What are the best vLLM Semantic Router alternatives?
- The closest alternatives we have reviewed are LiteLLM and Portkey AI Gateway.
Answers checked Oct 23, 2026.