Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

CPU offloading

CPU offloading runs a model that does not fit in GPU memory by keeping some of its layers in system RAM, or on disk, and using the GPU for the rest. The model runs, but more slowly.

Also called: offload layers, GPU offloading, partial offload

By Divya Sharma · Checked Nov 15, 2026 · How we review

A model's weights normally live in VRAM. When a model is bigger than your GPU, offloading puts as many layers as fit on the GPU and the remainder in ordinary RAM. That is how people run large local LLMs on a gaming PC.

Two ways it is done

In Hugging Face Accelerate and Transformers, setting `device_map="auto"` fills the GPU first, then the CPU, and finally the disk. Layers kept in RAM are moved to the GPU when needed, which works as long as the largest single layer fits on the GPU.

In llama.cpp and tools built on it, such as Ollama and LM Studio, you choose how many layers go to the GPU (the "GPU layers" setting). The rest are computed by the CPU directly from RAM. This hybrid mode is the common one for GGUF models.

The cost

System RAM is much slower than VRAM, and moving data over the PCIe bus is slower still. Each offloaded layer cuts tokens per second, sometimes sharply; disk offload is slowest of all. Offloading suits trying a model, or batch jobs where speed matters little, more than interactive use.

Alternatives

Before offloading, try a smaller quantization, which shrinks every layer. Mixture-of-experts models are a good fit for partial offload, since only a few experts run per token. Machines with unified memory avoid the problem because the CPU and GPU share one pool. The GPU calculator estimates how much of a model fits on your card.

Related terms

VRAM
VRAM is the memory on a graphics card. For AI work it holds the model weights, the KV cache and working data, and whether a model fits in VRAM usually decides which GPU you need to run it.
Unified memory
Unified memory is a design in which the CPU and GPU share one pool of memory instead of each having its own. On Apple Silicon Macs and similar chips, it lets a local AI model use most of the system RAM as GPU memory.
Local LLM
A local LLM is a language model that runs on your own computer, phone or server instead of a provider's cloud API, so prompts and data stay on your hardware and it works without an internet connection.
Quantization
Quantization is a technique that stores a model's weights, and sometimes activations, with fewer bits, such as 8 or 4 instead of 16, so the model needs less memory and often runs faster, at some cost in accuracy.
GGUF
GGUF is a binary file format for storing a model's weights, tokenizer and metadata in one file, built for fast loading and local inference with GGML-based tools such as llama.cpp, often in quantized form.

Sources