Inference provider
An inference provider is a company that runs AI models on its own hardware and sells access to them through an API, usually charged per token. You send prompts and get outputs without managing GPUs yourself.
Also called: model host, inference API, model API provider
By Divya Sharma · Checked Nov 15, 2026 · How we review
Running a model takes GPUs, a model server, scaling and monitoring. An inference provider does all of that and exposes the model as an API. Some, like the big labs, serve only their own models. Others, such as Together AI, Fireworks AI, Groq, Baseten and Replicate, host many open-weight models, so you can use the same model from more than one company.
How they differ
Providers serving the same model can differ a lot in price, speed, context window, rate limits and quantization, since some run lower-precision versions to save cost. Many expose an OpenAI-compatible API, so switching is often a change of base URL and key. Compare current token prices on our AI model pricing page.
Aggregators
Routers sit in front of many providers. Hugging Face's Inference Providers, for example, gives access to models across partner providers through one token and an OpenAI-compatible endpoint, and can pick the fastest or cheapest provider per model. OpenRouter works in a similar way. These make model fallback between providers simpler.
Inference provider versus GPU cloud
A GPU cloud rents you machines and you run the model yourself. An inference provider rents you the model's output. The first gives more control; the second is less work.
What to check
Where requests are processed and whether they are kept or used for training matter for privacy and data residency. Read each provider's data terms and check whether it offers zero data retention.
Related terms
- Model serving
- Model serving is running a trained model as a service that applications call over a network, with software that loads the weights onto GPUs, queues and batches requests, manages memory and returns results.
- OpenAI-compatible API
- An OpenAI-compatible API is an endpoint from another provider or tool that accepts the same request and response format as OpenAI's API, so code written for OpenAI can switch to it by changing the base URL, key and model name.
- Token pricing
- Token pricing is how most AI model APIs charge: a set price per million input tokens you send and a separate, usually higher, price per million output tokens the model generates, with discounts for cached input and batch jobs.
- GPU cloud
- A GPU cloud is a service that rents out GPU machines by the hour or second for AI training, fine-tuning and inference. Specialist providers, sometimes called neoclouds, focus on GPUs and often charge less than the large general clouds.
- Model fallback
- Model fallback is a rule that sends a request to a backup model or provider when the first choice fails, times out, hits a rate limit or rejects the input. It keeps an AI feature working through outages.