Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Batch API

A batch API lets you submit a large set of model requests in one job and collect the results later, instead of waiting on each call. Providers run the job when capacity allows and usually charge less for it.

Also called: batch inference, batch processing, Message Batches API

By Divya Sharma · Checked Nov 15, 2026 · How we review

Most model calls are synchronous: you send a prompt and wait for the answer. Plenty of work does not need an answer in seconds, such as classifying a million support tickets, generating product descriptions, running an evaluation suite or labelling training data. A batch API is built for that.

How it works

You upload a file or list of requests, each with its own ID. The provider queues the job, processes it in the background, and gives you a status to poll. When it finishes you download the results and match them to your IDs. Anthropic's Message Batches API, for example, charges 50 percent less than standard calls and says a batch can take up to 24 hours, though many finish sooner. OpenAI and other providers offer similar endpoints with their own discounts and windows; check current figures on AI model pricing.

Why it is cheaper

The provider can run batch work in quiet periods and pack requests together efficiently. Batch jobs also typically have their own rate limits, separate from your live traffic, so a large job does not starve your product.

Batch API versus batching in a server

"Batching" also describes how a model server groups requests on the GPU, as in continuous batching. That is an internal speed technique and happens for live traffic too. A batch API is a product: a slower, discounted way to buy inference.

Limits

You give up latency and some control. Individual requests can fail or expire and need resubmitting, results arrive out of order, and you cannot stream. Batches are a poor fit for anything a user is waiting on.

Related terms

Token pricing
Token pricing is how most AI model APIs charge: a set price per million input tokens you send and a separate, usually higher, price per million output tokens the model generates, with discounts for cached input and batch jobs.
Rate limit (API)
A rate limit caps how much an account can use an API in a time window, typically as requests per minute (RPM) and tokens per minute (TPM). Requests over the limit are rejected, usually with an HTTP 429 error.
Inference
Inference is running a trained model to produce output from new input, such as an LLM generating a reply to a prompt. It is the stage users pay for every time they call a model, as opposed to training.
LLM evaluation
LLM evaluation, often called evals, is measuring how well a model or AI application performs a task by running it on a set of test cases and scoring the outputs against defined criteria, by code, by humans or by another model.
Continuous batching
Continuous batching is a model-serving technique that adds new requests to the running GPU batch and removes finished ones after every generation step, instead of waiting for a whole batch to complete. It keeps the GPU busy and raises throughput.

Sources