CPU offloading
CPU offloading runs a model that does not fit in GPU memory by keeping some of its layers in system RAM, or on disk, and using the GPU for the rest. The model runs, but more slowly.
Also called: offload layers, GPU offloading, partial offload
By Divya Sharma · Checked Nov 15, 2026 · How we review
A model's weights normally live in VRAM. When a model is bigger than your GPU, offloading puts as many layers as fit on the GPU and the remainder in ordinary RAM. That is how people run large local LLMs on a gaming PC.
Two ways it is done
In Hugging Face Accelerate and Transformers, setting `device_map="auto"` fills the GPU first, then the CPU, and finally the disk. Layers kept in RAM are moved to the GPU when needed, which works as long as the largest single layer fits on the GPU.
In llama.cpp and tools built on it, such as Ollama and LM Studio, you choose how many layers go to the GPU (the "GPU layers" setting). The rest are computed by the CPU directly from RAM. This hybrid mode is the common one for GGUF models.
The cost
System RAM is much slower than VRAM, and moving data over the PCIe bus is slower still. Each offloaded layer cuts tokens per second, sometimes sharply; disk offload is slowest of all. Offloading suits trying a model, or batch jobs where speed matters little, more than interactive use.
Alternatives
Before offloading, try a smaller quantization, which shrinks every layer. Mixture-of-experts models are a good fit for partial offload, since only a few experts run per token. Machines with unified memory avoid the problem because the CPU and GPU share one pool. The GPU calculator estimates how much of a model fits on your card.
Related terms
- VRAM
- VRAM is the memory on a graphics card. For AI work it holds the model weights, the KV cache and working data, and whether a model fits in VRAM usually decides which GPU you need to run it.
- Unified memory
- Unified memory is a design in which the CPU and GPU share one pool of memory instead of each having its own. On Apple Silicon Macs and similar chips, it lets a local AI model use most of the system RAM as GPU memory.
- Local LLM
- A local LLM is a language model that runs on your own computer, phone or server instead of a provider's cloud API, so prompts and data stay on your hardware and it works without an internet connection.
- Quantization
- Quantization is a technique that stores a model's weights, and sometimes activations, with fewer bits, such as 8 or 4 instead of 16, so the model needs less memory and often runs faster, at some cost in accuracy.
- GGUF
- GGUF is a binary file format for storing a model's weights, tokenizer and metadata in one file, built for fast loading and local inference with GGML-based tools such as llama.cpp, often in quantized form.