Model fallback
Model fallback is a rule that sends a request to a backup model or provider when the first choice fails, times out, hits a rate limit or rejects the input. It keeps an AI feature working through outages.
Also called: failover, fallback model, provider failover
By Divya Sharma · Checked Nov 15, 2026 · How we review
Hosted models fail more often than most web APIs. Providers have outages, return rate-limit and overload errors at busy times, and reject requests that are too long or trip a content filter. A fallback chain turns those errors into a slower or slightly different answer instead of a broken feature.
How it works
You list models in order of preference, for example a primary model, then the same model from a second inference provider, then a cheaper model from another company. The client or an LLM gateway tries the first; if it fails after a set number of retries, the request moves down the list. Tools such as LiteLLM, Portkey and OpenRouter support this. LiteLLM also lets you set separate chains per error type, such as one for prompts that exceed the context window and one for content policy rejections, and puts a failing deployment into a cooldown so traffic stops hitting it.
Fallback versus routing
LLM routing chooses a model up front for each request, for example by difficulty or price. Fallback only acts after something has gone wrong. Many gateways do both.
Things to watch
A backup model can format output differently, support different tools, or have a smaller context window, so test every model in the chain against your prompts and structured output schema. Log which model answered each request; otherwise quality changes from silent fallbacks are hard to trace. Do not retry requests with side effects, such as agent tool calls, without making them safe to repeat.
Related terms
- LLM gateway
- An LLM gateway is a proxy server between your applications and model providers that exposes one API and handles keys, routing, fallbacks, rate limits, logging and spend tracking in a single place.
- LLM routing
- LLM routing is the practice of sending each request to the language model best suited to it, often a cheap model for easy prompts and a stronger one for hard prompts, to balance cost, speed and quality.
- Rate limit (API)
- A rate limit caps how much an account can use an API in a time window, typically as requests per minute (RPM) and tokens per minute (TPM). Requests over the limit are rejected, usually with an HTTP 429 error.
- Inference provider
- An inference provider is a company that runs AI models on its own hardware and sells access to them through an API, usually charged per token. You send prompts and get outputs without managing GPUs yourself.
- LLM observability
- LLM observability is recording what an AI application does in production, including prompts, responses, tool calls, latency, token use and cost, as traces you can search, so you can debug failures and see how the system behaves over time.