What is llama-modchip?
llama-modchip is a set of patches to llama.cpp, a quantiser and a ./modchip wrapper script for running large mixture-of-experts (MoE) models on a CPU alone. An MoE model only uses a few of its experts for each token, so speed on a CPU is set mostly by how many bytes of weights must be read from memory per token. The project shrinks those bytes with two formats of its own: Fractal-64 at 3.75 bits per weight and LUT-VQ3-64 at 3.00, decoded with AVX-512 lookup instructions on recent AMD and Intel chips, with AVX2 and Apple Metal paths as well.
Its headline build is DeepSeek-V4-Flash-0731, a model with 284.33B parameters and 13.17B active per token, in 56.40 to 56.93 GiB files that run inside a 64 GB machine. It also ships NVIDIA Nemotron-3.5-30B-A3B at 13.30 GiB for 16 to 32 GB laptops. The ./modchip script downloads a prebuilt server, picks a model for your RAM and starts a terminal chat, a web chat, a benchmark or Claude Code pointed at the local engine.
The files only load in this runtime. LUT-VQ3-64 adds a new tensor type, 43, and Fractal-64 reuses llama.cpp's existing type number 42 with a different layout, so stock llama.cpp, Ollama, vLLM and LM Studio reject them. The project is under Apache 2.0, started on 29 October 2026, and almost all of its commits come from one developer.
Is llama-modchip worth it?
llama-modchip makes a specific promise and largely documents it: DeepSeek-V4-Flash, 284B parameters, on an ordinary 64 GB desktop with no graphics card. On a 16-core Zen 5 with dual-channel DDR5-5600, the README reports 8.66 tokens a second for the default MaxQ-Fractal build generating one reply, 36.1 to 41.1 tokens a second reading a cold prompt, and bursts of up to 25.3 tokens a second with its speculative decoder. The process takes 57.54 to 58.13 GiB of the roughly 62.5 GiB a 64 GB Linux box can use, which leaves very little for a browser.
The quality cost is in its own tables, which is to its credit. On AtomicChat's wiki-alt text, the 284B build's perplexity is 7.2977 against 4.5381 for the unquantised model, 60.8% higher, and it scores 70 of 80 on the project's functional test. The 30B Nemotron build is closer to the original: perplexity 7.4400 against 6.9582 on WikiText-2, and 100 of 100 on the same test. These are the maintainer's measurements on a test suite he wrote; outside reports so far are speed submissions to the leaderboard, such as 8.11 tokens a second on a Ryzen 9 9950X3D.
The formats are the main practical trade-off. Because Fractal-64 takes over an existing tensor type number, a file can be mistaken for an upstream one and fail with a size mismatch rather than a clear error, and you depend on this fork to keep pace with llama.cpp. The prebuilt Linux binaries come with SHA-256 checksums, but the checksums sit in the same release as the files, so they protect against a broken download, not a compromised release. On Windows, the build script is run with PowerShell's execution policy set to Bypass.
./modchip claude points Claude Code at the local gateway with a dummy key. In the current script it adds --dangerously-skip-permissions, which lets the agent run commands without asking, only when MODCHIP_CLAUDE_SKIP_PERMS=1 is set; leave it unset. The gateway and web chat bind to 127.0.0.1 and ask for a key only if GATEWAY_API_KEY is set. Two optional abliterated models, with refusals removed, install only after an explicit confirmation.
For most people with 64 GB, a smaller model in a standard GGUF file under llama.cpp or Ollama will be faster, better tested and portable. colibri takes the other route to huge MoE models, streaming experts from disk instead of compressing them to fit in RAM. llama-modchip suits the enthusiast who specifically wants the 284B model resident in memory and accepts a format tied to one project.
Where it earns its place
A developer with a 64 GB Ryzen 9000 desktop and no discrete GPU
Run a frontier-size open model locally for private coding questions
Clone the repository and run ./modchip chat; it fetches the matching AVX-512 binary and the 56.93 GiB MaxQ-Fractal model. Expect about 9 tokens a second with a 32,768-token context, close other heavy apps, and enable zram if the desktop starts to swap.
A student on a 32 GB laptop
Point Claude Code at a local model with no API bill
The default 13.30 GiB Nemotron-3.5-30B-A3B build runs at about 24 tokens a second on the project's Zen 5 test machine. ./modchip claude sets ANTHROPIC_BASE_URL to the local gateway; keep MODCHIP_CLAUDE_SKIP_PERMS unset so Claude Code still asks before running commands.
A quantisation researcher
Compare vector-quantised MoE formats against Q2_K and Q3_K
The README and benchmark files give signal-to-noise, cosine similarity, perplexity and KL divergence for each format on the same tensors, and ./modchip quantize converts your own GGUF files. The results are the maintainer's, so rerun them before citing.
Before you commit
- Setup
- git clone, then ./modchip chat on Linux or macOS; ./modchip download-bin selects the AVX-512 or AVX2 binary and checks its SHA-256. Windows needs a MinGW-w64 build through a PowerShell script. Model downloads are 13.30 GiB for Nemotron and about 57 GiB for DeepSeek-V4-Flash.
- Integrations
- llama.cpp's OpenAI-compatible server on port 8080, a gateway that Claude Code talks to as an Anthropic endpoint, a web chat at 127.0.0.1:8787, and Hugging Face downloads of the Synchopate GGUF files.
- Data and privacy
- Runs locally. The gateway binds to 127.0.0.1 and enforces a key only when GATEWAY_API_KEY is set. ./modchip claude sets a dummy Anthropic key and turns off Claude Code's non-essential traffic; it skips permission prompts only with MODCHIP_CLAUDE_SKIP_PERMS=1.
- Cost at scale
- The software is free; the cost is a 64 GB machine with fast memory. Speed scales with memory bandwidth: about 75 GB/s on dual-channel DDR5 gives roughly 9 tokens a second on the 284B model, and the project reports 111.62 tokens a second for the 30B model on an 8-channel Threadripper.
- Limits
- One request at a time is the measured case. Full 256K context on the 284B model needs a headless 64 GB machine with swap or 96 GB or more. The custom formats need AVX-512 VBMI for full speed and do not load in other runtimes.
What’s good
- DeepSeek-V4-Flash 284B in 56.93 GiB, resident in 64 GB of RAM
- About 9 tokens a second on a desktop CPU with no GPU
- Publishes perplexity and quality loss next to its speed figures
- Prebuilt Linux binaries with checksums; Apple Metal and Windows builds
What’s not
- Files load only in this fork, not stock llama.cpp, Ollama or LM Studio
- Perplexity 60.8% above the unquantised 284B model on its own test
- Leaves little memory free on a 64 GB desktop
- Two weeks old with one main maintainer and releases several times a day
Best for
Enthusiasts with a 64 GB AVX-512 desktop who want to run DeepSeek-V4-Flash 284B entirely on the CPU and accept a single-project file format.
llama-modchip alternatives
llama-modchip: common questions
- Is llama-modchip worth it?
- An impressive feat of packing that runs a 284B model on a 64 GB desktop at about 9 tokens a second, in file formats nothing else can read and with a real loss in quality. We rate it 3.6 out of 5.
- How much does llama-modchip cost?
- Free and open source (Apache 2.0). Each model keeps its own licence.
- What are the downsides of llama-modchip?
- Files load only in this fork, not stock llama.cpp, Ollama or LM Studio; perplexity 60.8% above the unquantised 284B model on its own test; leaves little memory free on a 64 GB desktop; two weeks old with one main maintainer and releases several times a day.
- Who is llama-modchip best for?
- Enthusiasts with a 64 GB AVX-512 desktop who want to run DeepSeek-V4-Flash 284B entirely on the CPU and accept a single-project file format.
- How much does llama-modchip cost for a team or in production?
- The software is free; the cost is a 64 GB machine with fast memory. Speed scales with memory bandwidth: about 75 GB/s on dual-channel DDR5 gives roughly 9 tokens a second on the 284B model, and the project reports 111.62 tokens a second for the 30B model on an 8-channel Threadripper.
- Is llama-modchip safe for company data?
- Runs locally. The gateway binds to 127.0.0.1 and enforces a key only when GATEWAY_API_KEY is set. ./modchip claude sets a dummy Anthropic key and turns off Claude Code's non-essential traffic; it skips permission prompts only with MODCHIP_CLAUDE_SKIP_PERMS=1.
- What are the best llama-modchip alternatives?
- The closest alternatives we have reviewed are colibri, llama.cpp and Ollama.
Answers checked Nov 13, 2026.