What is FreeToken?
FreeToken runs open-weight mixture-of-experts (MoE) models, the kind where only a few of many expert sub-networks fire for each token, on personal computers rather than datacentre servers. It treats the GPU, CPU, system RAM and the links between them as one pool: hot experts are cached on the GPU, the rest are streamed or computed on the CPU, and the split between expert cache and KV memory can change at runtime without a restart.
The accompanying arXiv paper, whose 11 authors include Ion Stoica, Matei Zaharia, Song Han and Kurt Keutzer, claims more than 20 supported MoE models and reports serving a 35B model on a laptop, a 284B model on a gaming desktop and the 753B GLM-5.2 on a single workstation GPU. FreeToken exposes OpenAI- and Anthropic-compatible APIs, can configure and launch Claude Code, Codex or OpenCode against itself, and has a desktop app for Windows and Linux. It is Apache 2.0 licensed, was created on 20 July 2026, has about 17,564 stars and released v0.2.2 on 9 November 2026.
Is FreeToken worth it?
Running a frontier-sized open model at home has meant either heavy quantisation in llama.cpp or a slow crawl through system memory. FreeToken's argument is that MoE models make a third path possible: because each token touches only a few experts, a consumer GPU can hold the busy ones while the CPU serves the rest, if the engine keeps re-deciding which is which. Its bandwidth-adaptive policy, global LRU expert cache and double-buffered prefill streaming are built for exactly that, and the paper, submitted on 17 August 2026, reports a 753B model on one workstation GPU and 284B on a gaming desktop.
The agent integration is the practical draw. FreeToken serves both the OpenAI and Anthropic APIs, so ft launch claude writes Claude Code's provider config, installs the CLI if it is missing and points it at your local server. Semantic anchor checkpoints save recurrent and KV state at tool-call and thinking-block boundaries, so an agent that edits its own context does not force the whole prompt to be recomputed. The v0.2.2 release added speculative decoding; on an H100 it lifted Qwen3.8-27B from 50.1 to 215.0 tokens a second with a DFlash 2 draft, a 4.29 times gain, though that is a datacentre card, not the consumer hardware the project is named for.
Security defaults are reasonable. The server binds to 127.0.0.1 on port 1919, and an API key is off unless you set --api-key or FREETOKEN_API_KEY; set one before binding to another address. Model weights come from Hugging Face or local files.
The limits are those of a young, fast-moving project. Support is NVIDIA RTX 30, 40 and 50 series only, so there is nothing for Macs or AMD cards. The repository was created on 20 July 2026 and has 492 open issues, the docs already include a guide to repairing old FTW checkpoints, and speculative decoding does not yet work above one GPU of tensor parallelism. Against Ollama it is far more capable with huge MoE models and far less polished for everything else.
Where it earns its place
A developer with an RTX 50-series gaming PC and 128 GB of RAM
Run Claude Code against a large open model without sending code to a provider
ft serve a supported MoE checkpoint, then ft launch claude to wire Claude Code to the local Anthropic-compatible endpoint. No token bill; speed depends on how many experts fit in VRAM.
A research engineer benchmarking local inference for an internal report
Compare MoE serving on consumer hardware against llama.cpp
Run ft bench bw so the hybrid CPU-GPU policy measures the machine, then serve the same GGUF in both engines and compare with the timings report enabled.
Before you commit
- Setup
- Install the desktop app for Windows or Linux from flashml.ai, or uv pip install "freetoken[accel]"; then ft serve --model with a Hugging Face id, local folder or GGUF. The server is ready when it logs 127.0.0.1:1919.
- Integrations
- OpenAI API (/v1/chat/completions, /v1/responses) and Anthropic API (/v1/messages); ft launch for Claude Code, Codex, OpenCode, OpenClaw, Hermes and DeepSeek Harness; GGUF, MXFP4, NVFP4, FP8 and BF16 checkpoints; a terminal chat shell.
- Data and privacy
- Runs entirely on your machine. The server listens on 127.0.0.1:1919 by default with no API key unless you set one, so add a key before exposing it on a network. ft launch rewrites the chosen agent's provider config; --dry-run shows the changes first.
- Cost at scale
- No licence cost. The bill is the hardware: system RAM and a recent NVIDIA GPU sized to the model, plus power.
- Limits
- NVIDIA RTX 30, 40 and 50 series only; speculative decoding unsupported above TP 1 and for some windowed-attention models; 492 open issues; Windows and WSL cap pinned memory, which needs the auto CPU-layer setting.
What’s good
- Runs very large MoE models by splitting experts across GPU, CPU and RAM
- OpenAI and Anthropic APIs, with one-command setup for Claude Code, Codex and OpenCode
- Context-edit-aware caching avoids recomputing agent prompts
- Localhost binding by default; optional API key on every route
What’s not
- NVIDIA RTX 30, 40 and 50 series only; no Mac or AMD support
- 492 open issues four months after launch
- Headline speed-ups measured on an H100, not consumer cards
- Docs already need a guide to repairing old checkpoints
Best for
Developers with a recent NVIDIA gaming or workstation GPU who want to run very large open MoE models locally behind their coding agents.
FreeToken alternatives
FreeToken: common questions
- Is FreeToken worth it?
- FreeToken is the most serious attempt yet to run 284B-class MoE models on a gaming PC, backed by a credible research team, but it is four months old, NVIDIA-only and carrying 492 open issues. We rate it 4.0 out of 5.
- How much does FreeToken cost?
- Free and open source (Apache 2.0), including the desktop app. Your only cost is the hardware and electricity.
- What are the downsides of FreeToken?
- NVIDIA RTX 30, 40 and 50 series only; no Mac or AMD support; 492 open issues four months after launch; headline speed-ups measured on an H100, not consumer cards; docs already need a guide to repairing old checkpoints.
- Who is FreeToken best for?
- Developers with a recent NVIDIA gaming or workstation GPU who want to run very large open MoE models locally behind their coding agents.
- How much does FreeToken cost for a team or in production?
- No licence cost. The bill is the hardware: system RAM and a recent NVIDIA GPU sized to the model, plus power.
- Is FreeToken safe for company data?
- Runs entirely on your machine. The server listens on 127.0.0.1:1919 by default with no API key unless you set one, so add a key before exposing it on a network. ft launch rewrites the chosen agent's provider config; --dry-run shows the changes first.
- What are the best FreeToken alternatives?
- The closest alternatives we have reviewed are llama.cpp, Ollama, AirLLM and SGLang.
Answers checked Nov 21, 2026.