Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Serverless GPU

Serverless GPU is a cloud model where you deploy code or a model and the platform starts GPU workers only when requests arrive, billing for the time they run and scaling to zero when idle.

Also called: serverless inference

By Divya Sharma · Checked Nov 16, 2026 · How we review

With a rented GPU server you pay by the hour whether it is busy or not, and you manage the machine. Serverless GPU platforms, such as Runpod's Serverless product, Modal, Replicate and Baseten, take care of the machines. You package a model and a handler, the platform runs containers on GPUs when traffic arrives, adds more under load, and removes them when traffic stops. Runpod's documentation, for example, says you pay only while workers run and that workers scale down to zero after a short idle timeout.

When it fits

It suits spiky or low traffic: an internal tool, a batch job that runs nightly, an image model used a few times an hour, or a prototype. You avoid paying for idle GPUs and do not need to run your own model serving cluster.

Cold starts

The main cost is the cold start. When no worker is running, the platform has to start a container and load the model into GPU memory before it can answer. Large models take longer to load, so the first request after a quiet period can be slow. Platforms offer options such as keeping a minimum number of warm workers, at extra cost, or caching model files near the GPU.

Compared with other options

At steady high volume, a dedicated GPU from an AI GPU cloud or self-hosting is often cheaper per request. If you only need popular models, a per-token inference provider avoids deployment work entirely. See the deployment guide for options.

Related terms

Model serving
Model serving is running a trained model as a service that applications call over a network, with software that loads the weights onto GPUs, queues and batches requests, manages memory and returns results.
GPU cloud
A GPU cloud is a service that rents out GPU machines by the hour or second for AI training, fine-tuning and inference. Specialist providers, sometimes called neoclouds, focus on GPUs and often charge less than the large general clouds.
Inference provider
An inference provider is a company that runs AI models on its own hardware and sells access to them through an API, usually charged per token. You send prompts and get outputs without managing GPUs yourself.
Self-hosting (AI models)
Self-hosting an AI model means running it on hardware you control, such as your own servers, rented GPUs or a laptop, instead of calling a provider's API. You choose the model and keep the data, but you run the infrastructure.
GPU
A GPU (graphics processing unit) is a processor with many small cores built to run many calculations in parallel. Its speed at matrix maths makes it the standard hardware for training and running AI models.

Sources