Rate limit (API)
A rate limit caps how much an account can use an API in a time window, typically as requests per minute (RPM) and tokens per minute (TPM). Requests over the limit are rejected, usually with an HTTP 429 error.
Also called: rate limiting, RPM, TPM
By Divya Sharma · Checked Nov 16, 2026 · How we review
AI providers limit usage to share capacity fairly and to prevent abuse. Limits are usually set per organisation or project and per model, and they rise as your account builds usage history. Anthropic's documentation, for example, measures Messages API limits in requests per minute, input tokens per minute and output tokens per minute, and returns a 429 error saying which limit was exceeded.
How limits behave
A per-minute limit is often enforced over shorter intervals, so a burst can fail even if the minute's total would be fine. Anthropic says its API uses a token bucket, so capacity is continuously replenished rather than reset at fixed times. Some limits count only uncached input, which makes prompt caching a way to fit more work under the same limit. Separate spend caps can also block requests once a monthly budget is reached.
Handling 429 errors
Read the `retry-after` header when present and wait that long. Otherwise retry with exponential backoff and some random jitter, and cap the number of retries so a failing job does not loop forever. Response headers usually report how much of each limit is left, which lets a client slow down before hitting it. For large offline jobs, a batch API often has its own, higher limits.
Planning around limits
Spread traffic across time, keep prompts lean, and set your own per-workspace limits so one service cannot starve others. An LLM gateway can queue requests and route to a fallback model or a second provider when one is limited. Rate limits are separate from token pricing.
Related terms
- Token pricing
- Token pricing is how most AI model APIs charge: a set price per million input tokens you send and a separate, usually higher, price per million output tokens the model generates, with discounts for cached input and batch jobs.
- Batch API
- A batch API lets you submit a large set of model requests in one job and collect the results later, instead of waiting on each call. Providers run the job when capacity allows and usually charge less for it.
- LLM gateway
- An LLM gateway is a proxy server between your applications and model providers that exposes one API and handles keys, routing, fallbacks, rate limits, logging and spend tracking in a single place.
- Model fallback
- Model fallback is a rule that sends a request to a backup model or provider when the first choice fails, times out, hits a rate limit or rejects the input. It keeps an AI feature working through outages.
- API key
- An API key is a secret string an application sends with each request so that a service can tell which account is calling, apply its limits and bill it. Anyone holding the key can usually use it.