Serverless GPU
Serverless GPU is a cloud model where you deploy code or a model and the platform starts GPU workers only when requests arrive, billing for the time they run and scaling to zero when idle.
Also called: serverless inference
By Divya Sharma · Checked Nov 16, 2026 · How we review
With a rented GPU server you pay by the hour whether it is busy or not, and you manage the machine. Serverless GPU platforms, such as Runpod's Serverless product, Modal, Replicate and Baseten, take care of the machines. You package a model and a handler, the platform runs containers on GPUs when traffic arrives, adds more under load, and removes them when traffic stops. Runpod's documentation, for example, says you pay only while workers run and that workers scale down to zero after a short idle timeout.
When it fits
It suits spiky or low traffic: an internal tool, a batch job that runs nightly, an image model used a few times an hour, or a prototype. You avoid paying for idle GPUs and do not need to run your own model serving cluster.
Cold starts
The main cost is the cold start. When no worker is running, the platform has to start a container and load the model into GPU memory before it can answer. Large models take longer to load, so the first request after a quiet period can be slow. Platforms offer options such as keeping a minimum number of warm workers, at extra cost, or caching model files near the GPU.
Compared with other options
At steady high volume, a dedicated GPU from an AI GPU cloud or self-hosting is often cheaper per request. If you only need popular models, a per-token inference provider avoids deployment work entirely. See the deployment guide for options.
Related terms
- Model serving
- Model serving is running a trained model as a service that applications call over a network, with software that loads the weights onto GPUs, queues and batches requests, manages memory and returns results.
- GPU cloud
- A GPU cloud is a service that rents out GPU machines by the hour or second for AI training, fine-tuning and inference. Specialist providers, sometimes called neoclouds, focus on GPUs and often charge less than the large general clouds.
- Inference provider
- An inference provider is a company that runs AI models on its own hardware and sells access to them through an API, usually charged per token. You send prompts and get outputs without managing GPUs yourself.
- Self-hosting (AI models)
- Self-hosting an AI model means running it on hardware you control, such as your own servers, rented GPUs or a laptop, instead of calling a provider's API. You choose the model and keep the data, but you run the infrastructure.
- GPU
- A GPU (graphics processing unit) is a processor with many small cores built to run many calculations in parallel. Its speed at matrix maths makes it the standard hardware for training and running AI models.