Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

LLM gateway

An LLM gateway is a proxy server between your applications and model providers that exposes one API and handles keys, routing, fallbacks, rate limits, logging and spend tracking in a single place.

Also called: AI gateway

By Divya Sharma · Checked Nov 15, 2026 · How we review

Once a team calls more than one model provider, the same plumbing gets rebuilt in every service: API keys, retries, cost tracking, per-team limits. A gateway moves that work into one shared layer. Applications send every request to the gateway, and the gateway forwards it to the right provider.

What a gateway usually does

Most gateways present an OpenAI-compatible API, so existing client code works with many providers by changing the base URL. Behind that endpoint they hand out virtual keys per user or team, enforce budgets and rate limits, record the cost of each call, retry or fail over when a provider has an outage, and send logs and traces to observability tools. Some also do LLM routing, caching and guard checks.

Open-source examples include LiteLLM, Portkey Gateway and Bifrost. Hosted services such as OpenRouter offer a similar single endpoint as a paid service.

Gateway versus router

A router answers one question: which model should take this request. A gateway is the wider control point that may contain a router along with keys, limits and logging.

Limits

A gateway is a new piece of infrastructure in the request path. It adds a hop of latency, becomes a single point of failure if it is not run with redundancy, and holds every provider key, so it needs the same care as any other secrets store.

Related terms

LLM routing
LLM routing is the practice of sending each request to the language model best suited to it, often a cheap model for easy prompts and a stronger one for hard prompts, to balance cost, speed and quality.
OpenAI-compatible API
An OpenAI-compatible API is an endpoint from another provider or tool that accepts the same request and response format as OpenAI's API, so code written for OpenAI can switch to it by changing the base URL, key and model name.
Model fallback
Model fallback is a rule that sends a request to a backup model or provider when the first choice fails, times out, hits a rate limit or rejects the input. It keeps an AI feature working through outages.
Rate limit (API)
A rate limit caps how much an account can use an API in a time window, typically as requests per minute (RPM) and tokens per minute (TPM). Requests over the limit are rejected, usually with an HTTP 429 error.
LLM observability
LLM observability is recording what an AI application does in production, including prompts, responses, tool calls, latency, token use and cost, as traces you can search, so you can debug failures and see how the system behaves over time.
API key
An API key is a secret string an application sends with each request so that a service can tell which account is calling, apply its limits and bill it. Anyone holding the key can usually use it.

Sources