What it is
colibri runs very large open mixture-of-experts (MoE) models on machines that cannot hold them in memory. An MoE model uses only a few of its "experts" for each token, so colibri keeps the always-used part in RAM and reads experts from disk as the model asks for them. It is written in C with one file per model family, needs no GPU, and uses CUDA or Vulkan when a card is there. The minimum is 8 GB of RAM and 22 GB of disk for the smallest model.
Thirteen engines run today, including GLM-5.2 and 5.3, Kimi K3, DeepSeek V4 and V4.1 Flash, Qwen3.6, Qwen3.8-Flash-Next and the image model Qwen-Image-2.1. It serves OpenAI- and Anthropic-compatible APIs and a web dashboard.
It also has a System One mode: POST /v1/systemone accepts TypeSafe Jev's request and returns Jev's reply format. Any chat model it runs can answer by scoring the allowed options, and three decision models run natively: Laya, GLiNER2.5-Decide and Cloudflare's Clef. An earlier /v1/brio route was folded into this endpoint after version 1.12.1. Vincenzo Fornaro maintains it under Apache 2.0.
Our review
colibri does something llama.cpp and Ollama do not set out to do: run models of hundreds of billions of parameters on a desktop by treating the disk as part of memory. Its own figures are honest about the cost. GLM-5.2, a 744B model, decodes at 0.05 to 0.1 tokens a second cold on the 25 GB laptop where the project started, 1.83 on a 128 GB Ryzen AI Max+ 395, and 9.0 to 9.2 on six RTX 5090s. Smaller MoE models are usable: Qwen3.6-35B-A3B reaches 6.0 tokens a second on a CPU and 30.0 with CUDA.
The project is popular and busy, with about 43,500 GitHub stars since July 2026. Version 2.0.0 on 6 October 2026 bundled 169 pull requests, 88 from outside contributors, adding Vulkan on every engine and up to 16 conversations at once. The README is unusually candid about its gaps: the maintainers say they have not timed a discrete GPU on Vulkan themselves, and the first figure, 17 to 19 tokens a second for Qwen3.6 on a Tesla V100, came from a user.
Its security record deserves attention, because it parses very large model files in C. On 5 August 2026 the project published eight advisories, six rated high, covering heap and stack overflows from crafted safetensors and config files, a negative max_tok that wrote past the KV cache, and an unauthenticated /profile endpoint. The server binds to localhost by default and enforces a key only when COLI_API_KEY is set, so anything beyond one machine needs that variable, and model files should come from sources you trust.
The System One gateway is a thoughtful extra. A Jev client switches by changing its base URL, and the docs say TypeSafe's two official SDKs are tested against it unmodified. The probabilities are your model's, not Jev's, and will not match. On a CPU, Laya answers in 219 ms and GLiNER2.5-Decide in 294 ms, but Clef takes 20.4 seconds per request in int8, so 27B decision models still want a GPU.
For many-user serving on real GPUs, vLLM and SGLang remain the tools; 16 KV slots suit a household or a small team, not a product. The fairer comparison is llama.cpp, which covers the smaller models from RAM but does not attempt to stream a 429 GB checkpoint from an NVMe drive.
Where it earns its place
A researcher with a 128 GB Ryzen AI Max+ 395 workstation
Run GLM-5.2 (744B) locally for offline experiments
Download the 429 GB int4 checkpoint to NVMe and run coli chat; the project measured 1.83 tokens a second warm on this machine. There is no API bill, but a long answer takes minutes, so it suits batch questions rather than interactive coding.
A developer on a 32 GB desktop with an integrated Radeon 780M
Point Claude Code or an OpenAI-compatible editor at a local model
Qwen3.6-35B-A3B (a 23 GB download) runs from RAM at 6.0 tokens a second on the CPU or 9.9 with Vulkan on the integrated GPU. Set ANTHROPIC_BASE_URL to the local server; nothing leaves the machine.
A product team prototyping Jev-based ticket routing before buying API capacity
Test decision prompts offline
Run Laya (1.7 GB of RAM, 219 ms per question on an i7 laptop) behind /v1/systemone and change only TYPESAFE_BASE_URL. Probabilities differ from hosted Jev, so thresholds need recalibrating before switching back.
Before you commit
- Setup
- ./start-here.sh on Linux or macOS, START-HERE.bat on Windows: it checks RAM, disk and GPU, recommends a model, builds or downloads the engine and resumes interrupted downloads. The minimum is 8 GB of RAM and 22 GB of disk; the large models need hundreds of GB of NVMe.
- Integrations
- OpenAI-compatible /v1/chat/completions, Anthropic-compatible /v1/messages, /v1/images/generations for Qwen-Image-2.1, /v1/systemone for Jev-style decisions, a web dashboard and an MCP server (coli mcp) for setup tasks.
- Data and privacy
- Runs locally; the README describes no remote telemetry, only a local routing-history file (.coli_usage). Binds to localhost and enforces a key only with COLI_API_KEY. The August 2026 advisories covered crafted model files and an unauthenticated /profile endpoint. Qwen-Image-2.1 weights are non-commercial.
- Cost at scale
- The software is free; the cost is hardware. GLM-5.2 decodes at 9.0 to 9.2 tokens a second with every expert resident on six RTX 5090s, and Kimi K3 is a 1.56 TB download at about 9.4 seconds per token. Disk speed sets throughput; a second NVMe drive measured +37.5% decode.
- Limits
- The largest models run at fractional to single-digit tokens a second; at most 16 concurrent conversations; Vulkan output can drift from the CPU's by a word; 141 open issues and pull requests, many about GPU detection and Windows CUDA builds.
What's good
- Runs 744B-class MoE models with 16 to 24 GB of RAM by streaming experts
- Pure C, no required GPU; CUDA, Vulkan and Metal when available
- Jev-compatible /v1/systemone endpoint with native Laya, GLiNER and Clef
- One-step setup that resumes downloads; v2.0.0 in October 2026
What's not
- Single-digit or fractional tokens per second on the largest models
- Eight security advisories on 5 August 2026, six high
- No API key enforced unless COLI_API_KEY is set
- Throughput depends heavily on disk speed
Best for
Enthusiasts and developers who want to run frontier-size open MoE models, or prototype Jev-style decisions, on their own machine.
Alternatives
colibri: common questions
- Is colibri worth it?
- A disk-streaming C engine that makes 744B-parameter open models run on a desktop, slowly, with an honest local stand-in for the Jev decision API on the side. We rate it 4.2 out of 5.
- How much does colibri cost?
- Free and open source (Apache 2.0). Each model keeps its own licence.
- What are the downsides of colibri?
- Single-digit or fractional tokens per second on the largest models; eight security advisories on 5 August 2026, six high; no API key enforced unless COLI_API_KEY is set; throughput depends heavily on disk speed.
- Who is colibri best for?
- Enthusiasts and developers who want to run frontier-size open MoE models, or prototype Jev-style decisions, on their own machine.
- How much does colibri cost for a team or in production?
- The software is free; the cost is hardware. GLM-5.2 decodes at 9.0 to 9.2 tokens a second with every expert resident on six RTX 5090s, and Kimi K3 is a 1.56 TB download at about 9.4 seconds per token. Disk speed sets throughput; a second NVMe drive measured +37.5% decode.
- Is colibri safe for company data?
- Runs locally; the README describes no remote telemetry, only a local routing-history file (.coli_usage). Binds to localhost and enforces a key only with COLI_API_KEY. The August 2026 advisories covered crafted model files and an unauthenticated /profile endpoint. Qwen-Image-2.1 weights are non-commercial.
- What are the best colibri alternatives?
- The closest alternatives we have reviewed are llama.cpp, Ollama, vLLM, SGLang and Laya.
Answers checked Oct 23, 2026.
Make colibri? Show this rating on your site
