Step 10 of 22
LLM routing & gateways
Send each request to the right model, with fallbacks, budgets and keys in one place. No single model is best and cheapest for every request. Routing cuts cost and survives outages.
How to choose
- Many providers, one API key: a hosted router.
- Self-hosted control over keys, budgets and logs: an open-source gateway.
- Routing by request type (easy vs hard, safe vs risky): a small decision model in front of the expensive one.
What to look for
The questions that separate the options, in order of how much they matter.
01
Do you need a router at all?
If one service calls one or two providers, the providers' own SDKs plus a retry wrapper are simpler and remove a network hop. A gateway pays for itself when several teams share provider keys, finance wants spend by team, or you need automatic failover between providers. Per-request routing by meaning only pays when traffic is large and varied enough that sending easy requests to a cheaper model changes the bill, and you have evaluations to prove quality holds.
02
Hosted router, self-hosted gateway or decision model
A hosted router (OpenRouter) gives one key and one bill for hundreds of models with nothing to run, but every request passes through a third party and you pay a fee on credits. A self-hosted gateway (LiteLLM, Portkey, Claude Code Router) keeps traffic and keys in your network and adds budgets and logging, but you operate it. A decision-model router (vLLM Semantic Router, or a decision model such as Jev, pplx-decider or d3 called from your code) reads each request and picks the model; it is a layer on top of a gateway, not a replacement.
03
Fallbacks, retries and provider coverage
Check that the router can retry on rate limits and timeouts and fall back to a different provider or model, not just a different key. Coverage claims (100+ providers, 1,600+ models) matter less than whether your specific providers, including Azure, Bedrock and Vertex, are supported with their quirks: tool calls, streaming, prompt caching headers and vision inputs. Test a failover before you rely on it.
04
Keys, budgets and spend tracking
For teams, the core feature is issuing virtual keys with budgets and rate limits per team, user or project, and logging spend against them. Check which of these features are in the open-source build and which are paid: LiteLLM's documentation and enterprise page disagree on whether budgets are enterprise-only. Hosted routers handle billing for you but give coarser controls.
05
Security posture and supply chain
A gateway holds every provider key you own, so its own security record matters as much as its features. LiteLLM's PyPI releases 1.82.7 and 1.82.8 were poisoned in March 2026 to steal credentials. Pin exact versions, install from verified images, isolate the gateway and keep local routers bound to localhost. Ownership matters too: Portkey is now part of Palo Alto Networks, which sets its roadmap.
06
Latency overhead and caching
Gateways advertise small overheads (LiteLLM cites 8 ms at P95 at 1,000 requests a second; Portkey under 1 ms), but these are the projects' own benchmarks; measure on your traffic. A hosted router adds a network hop and possibly a region change. Semantic routers add a classifier call per request; decision models report tens to hundreds of milliseconds. Response caching and pass-through of provider prompt caching can save more than the router costs.
07
Routing method, licence and maintenance
Rules-based routing (model lists, weights, conditions) is predictable and easy to audit. Semantic or learned routing is cheaper in theory but harder to debug; check whether it reports why a model was chosen. Licences differ: MIT (Portkey, Claude Code Router), MIT with a commercial enterprise folder (LiteLLM), Apache 2.0 (vLLM Semantic Router). Check release activity: Portkey's gateway had no stable release between January and October 2026.
How the options differ
Three kinds of product share this stage. Hosted routers (OpenRouter) sell access to many models through one API and bill you, adding a fee; you run nothing. Self-hosted gateways (LiteLLM, Portkey, and Claude Code Router for coding agents) sit in your network, hold your provider keys, and handle fallbacks, budgets and logs with rules you write.
Semantic routers and decision models choose the model per request. vLLM Semantic Router classifies each request and applies your rules; decision models such as Jev, pplx-decider, Laya and d3 return a typed choice with probabilities that your code or router acts on. They decide which model to call; you still need a gateway or SDK to call it.
9 tools, compared
Ranked by our rating. Open any review for sources, prices and the full verdict.
| Tool | Rating | Our verdict | Best for | Price |
|---|---|---|---|---|
| OpenRoutertool | 4.6/5 | OpenRouter is the sensible first integration for any app that needs more than one model; at steady high volume on a single model, its credit fee becomes the thing to engineer away. | Developers and small teams who want to test, compare or offer many AI models through a single integration. | Pay provider list prices for models. Fee on credit purchases: 5.5% (Standard) or 8% (Business). Bring-your-own-key free for the first $25,000/month of usage, then 5%. Free models: 50 requests/day, or 1,000/day after a $10 purchase. |
| Perplexity Decider (pplx-decider)model | 4.5/5 | A decision model you can download, read images with and run for under half the price of Jev, scoring well above Jev on tests nobody can train on. | Teams that want strong typed decisions over text and images, either through a cheap API or on their own GPUs. | API: pplx-decider-v1.2 $0.02 and Flash $0.01 per 1M input tokens; output free, no per-request fee. Weights free under Apache 2.0. |
| LiteLLMrepo | 4.3/5 | The default self-hosted gateway for teams that need one API, per-team budgets and fallbacks across many providers, provided they pin versions and treat it as security-critical infrastructure. | Platform teams that must give many developers governed access to many model providers from their own infrastructure. | Free and open source under MIT, except the enterprise/ folder, which is under a separate commercial licence. Enterprise is custom-priced with a free 30-day trial key. |
| d3 (vLLM Semantic Router)model | 4.0/5 | Strong, open and easy to serve on vLLM, but nine days after launch its own d3-turbo made it the version to migrate from. | Teams already running vLLM who want an open decision model for text, image or video checks on their own GPUs and have validated against this checkpoint. | Free weights under Apache 2.0. No paid hosted API; the free public demo now serves d3-turbo. |
| Claude Code Routerrepo | 4.0/5 | A capable local switchboard for running coding agents on cheaper models, as long as you accept that Anthropic will not support the result and the edits will be worse. | Developers who want to run Claude Code or other agents on cheaper or local models. | Free and open source (MIT). You pay each model provider you route to. |
| vLLM Semantic Routerrepo | 3.8/5 | An ambitious open router for choosing models per request, suited to teams already running vLLM who can absorb a fast-moving project. | Teams serving several models, often on their own vLLM fleet, with enough varied traffic to justify choosing a model per request. | Free and open source under Apache 2.0. No hosted paid service. |
| Jevmodel | 3.8/5 | Jev turned typed decisions into a product, and open models have since passed it on both score and price. | Teams that want a hosted, low-cost API for text routing, triage and yes/no gates without running their own GPUs. | $0.042 per 1M input tokens ($42 per billion); output tokens are free. Same price at long context. Enterprise limits through sales. |
| Portkey AI Gatewayrepo | 3.7/5 | A neat, MIT-licensed router that has stood still since Palo Alto Networks bought it; fine to run today, hard to build a roadmap on. | Teams on edge runtimes, or enterprises already buying Palo Alto security, who want a light routing layer. | Free and open source under MIT. Hosted plans: Developer free (10,000 recorded logs a month, 3-day log retention); Production $49/month (100,000 logs, then $9 per 100,000); Enterprise custom. |
| Layamodel | 3.4/5 | Very fast and genuinely open, but close to guessing on hard decisions out of the box: a base to fine-tune, not a Jev replacement. | Teams with their own labelled decision data who want a tiny, self-hosted model to fine-tune for fast routing or guardrails. | Free weights and Python package under Apache 2.0. No hosted paid API; you run it yourself. |
Side by side
The specifications that matter for this step. Checked 2026-10-19; confirm on the vendor’s site before you commit.
| Tool | Type | Routing method | Provider coverage | Keys and budgets | Licence |
|---|---|---|---|---|---|
| OpenRouter | Hosted router | Provider fallback; model choice | 500+ models, 80+ providers | One key; BYOK; credit balance | Proprietary service |
| Perplexity Decider (pplx-decider) | Decision model (API or weights) | Typed choice with probabilities | You define the options | n/a | Apache 2.0 weights |
| LiteLLM | Self-hosted gateway and SDK | Rules, load balancing, fallbacks | 100+ providers | Virtual keys, budgets, rate limits | MIT; enterprise/ commercial |
| d3 (vLLM Semantic Router) | Decision model (self-host) | Typed choice with probabilities | You define the options | n/a | Apache 2.0 weights |
| Claude Code Router | Local gateway for coding agents | Rules by request type | OpenAI, Gemini, OpenRouter, more | Client keys with limits; key pools | MIT |
| vLLM Semantic Router | Semantic router | Request signals plus rules | vLLM, SGLang, OpenAI-compatible | Not a billing gateway | Apache 2.0 |
| Jev | Decision model (hosted API) | Typed choice with probabilities | You define the options | n/a | Closed; $0.042/1M input |
| Portkey AI Gateway | Self-hosted gateway | Conditional routing, fallbacks | 1,600+ models (README) | In hosted plans | MIT |
| Laya | Decision model (small, self-host) | Typed choice; fine-tune first | You define the options | n/a | Apache 2.0 weights |
Where this stage fits
- Customer support agent: we would start with Jev.
- Coding agent: we would start with OpenRouter.
- Voice agent: we would start with Laya.
- Document processing: we would start with GLiNER2.5-Decide.
- Workflow automation: we would start with Perplexity Decider (pplx-decider).