Running giant AI models on an ordinary PC: Colibri, Strata, AirLLM and FreeToken compared

You can now run a model with hundreds of billions of parameters on a desktop PC. Whether you should depends on one question: which parts of the model have to move between your SSD, your RAM and your graphics card, and how often. Four open-source engines answer it differently, and the answer decides whether you get a reply in seconds or one word every quarter of a minute.

Why this works at all

Most large open models are now mixture-of-experts (MoE) models. Each one is a crowd of small specialist networks, and for any given word only a few of them do the work. A 744-billion-parameter model might use a few per cent of its weights per token. If you can keep the busy experts close to the processor and fetch the rest when needed, you do not need a data-centre's worth of memory.

Think of your graphics memory as your desk, RAM as the shelf behind you and the SSD as the library down the corridor. Every expert that is not on the desk when it is needed costs a trip, and the engines below are different strategies for making fewer, shorter trips.

Colibri: any model, if you can wait

Colibri (4.2) is a pure-C engine that treats the disk as part of memory. It keeps the most used weights in RAM and streams the other experts from an NVMe drive. That makes enormous models start on modest machines, at a price its README states openly: GLM-5.2, a 744B model, decodes at 0.05 to 0.1 tokens a second cold on the 25 GB laptop where the project began, roughly one word every 10 to 20 seconds. On a 128 GB Ryzen AI Max+ 395 it reaches 1.83. Smaller MoE models are where it is pleasant: Qwen3.6-35B-A3B runs at 6.0 tokens a second on CPU and 30.0 with CUDA.

The catch: "cold" means nothing is cached yet, and first runs on a big model are slow enough to feel broken. Large models also mean downloads of hundreds of gigabytes. Colibri parses very large model files in C and published eight security advisories in August, so take model files only from sources you trust.

Strata: one model family, fast on a gaming PC

Strata (4.0) does the opposite: instead of running everything slowly, it runs Qwen3.8-Flash-Next and DeepSeek V4.1-Flash quickly. It pins the most used experts on the GPU, computes the rest on the CPU from RAM rather than shuttling them across, keeps a large lookup table on the SSD, and uses speculative decoding, where a small helper drafts words and the main model checks them. The project's own measurements on an RTX 5070 with 12 GB and a Ryzen 5 7600 show 93 tokens a second at 4K context with the Q2_0 pack and 53 with the higher-quality IQ3_S, and community reports on AMD cards are in the same range.

The catch: those packs need 37.6 to 54.8 GB of RAM and VRAM combined, so 64 GB of RAM is the realistic floor. Speeds at 4K context also flatter long coding sessions, and the project is largely one developer's work, with releases every few days.

AirLLM: least GPU memory, most patience

AirLLM keeps only one layer of the model on the GPU at a time, loading each from disk as it is needed. That is how its README can list a 70B model on a 4 GB card and Llama 3.1 405B on 8 GB, and its v3 line adds per-expert streaming for Kimi K3, which the project says ran in 3.72 GB of VRAM. It is a Python library, so a first experiment is a few lines with AutoModel.

The catch: the headline is a memory figure, not a speed figure. AirLLM first splits the checkpoint into per-layer files, which the README calls "very disk-consuming" (about 360 GB for Qwen3.8-Flash-Next), and names disk loading as the main bottleneck. It publishes no tokens-per-second numbers. Optional 4-bit or 8-bit compression makes it up to three times faster.

FreeToken: built for agents that come back for more

FreeToken, from a university research team (paper), keeps all weights in RAM and turns spare GPU memory into one shared cache of recently used experts. When an expert is missing, it decides per case whether copying it over PCIe or computing it on the CPU is faster, based on the bandwidth it measures. It can also change how GPU memory is split between expert cache and conversation memory without a restart.

Its distinctive feature is for coding agents. An agent edits tool arguments and injects command output mid-conversation, which normally forces an engine to recompute the whole prompt. FreeToken saves the model's internal state at logical task boundaries and reuses it, so the tenth turn of a session should not cost as much as the first. Better Stack's walkthrough covers setup and speeds against Ollama. It is Apache-2.0, supports NVIDIA RTX 30, 40 and 50 cards on Windows and Linux, and works with Claude Code, Codex and OpenCode through OpenAI- and Anthropic-compatible APIs.

The catch: NVIDIA only for now; AMD support exists only in a community fork. The project page publishes no benchmark figures of its own, and with 281 open issues it is moving fast.

Which to try first

Choose the model before the engine. A fast runtime cannot rescue a model that fails your task or carries a licence you cannot use.

For a quick, responsive assistant on a gaming PC with 64 GB of RAM, Strata is the shortest path, provided Qwen3.8-Flash-Next or DeepSeek V4.1-Flash is the model you want. For long agent sessions on an NVIDIA card, FreeToken is designed around exactly that workload; time a whole task, not one reply. For trying a model far too big for your machine, Colibri will run it, slowly, and tells you honestly how slowly. For a Python experiment on a small GPU where waiting is fine, AirLLM is the simplest to script.

If none of these appeal, llama.cpp and Ollama remain the dependable choice for models that fit in your memory.

Our take: the right engine is the one whose trips to the library match your patience. Measure the second and tenth turn of a real session, not the first answer to "hello".