Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

LLM routing

LLM routing is the practice of sending each request to the language model best suited to it, often a cheap model for easy prompts and a stronger one for hard prompts, to balance cost, speed and quality.

Also called: model routing, router

By Divya Sharma · Checked Nov 15, 2026 · How we review

Most applications do not need their most capable model for every request. A greeting, a short lookup or a formatting job can go to a small, fast model, while a tricky coding or reasoning task goes to a larger one. A router makes that choice per request, so the bill reflects the actual mix of work.

How it works

A router can be a set of rules (send anything with code to model A, anything over a certain length to model B), a classifier trained on past outcomes, or a small model that reads the prompt and picks a destination. The RouteLLM paper trained routers on human preference data to choose between a stronger and a weaker model, and reports cutting cost by more than half in some cases without a drop in answer quality. A decision model that returns a typed choice is another way to build the picking step.

Routing usually lives in an LLM gateway or a service such as OpenRouter, or in an open-source router like vLLM Semantic Router. The LLM routing stage of the AI stack lists more options.

Routing versus fallback

Routing picks a model before the call based on the request. A fallback model is used after a call fails or times out. Many systems do both.

Limits

A router adds its own latency and can misjudge a hard prompt as easy, which shows up as worse answers rather than an error. Test it on your own traffic and log which model handled each request.

Related terms

LLM gateway
An LLM gateway is a proxy server between your applications and model providers that exposes one API and handles keys, routing, fallbacks, rate limits, logging and spend tracking in a single place.
Decision model
A decision model is a model built to return a typed answer, such as a choice from a list, a score on a rubric or a yes/no probability, instead of generating free text.
Model fallback
Model fallback is a rule that sends a request to a backup model or provider when the first choice fails, times out, hits a rate limit or rejects the input. It keeps an AI feature working through outages.
Token pricing
Token pricing is how most AI model APIs charge: a set price per million input tokens you send and a separate, usually higher, price per million output tokens the model generates, with discounts for cached input and batch jobs.
Inference provider
An inference provider is a company that runs AI models on its own hardware and sells access to them through an API, usually charged per token. You send prompts and get outputs without managing GPUs yourself.

Sources