Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

FreeToken

A serving engine for very large mixture-of-experts models on consumer NVIDIA GPUs, splitting experts across GPU, CPU and system memory.

Reviewed by ReviewRamble editors · Updated

View on GitHub Open source

Share this page:XLinkedInFacebookRedditWhatsAppEmail

Our rating

4.0/5

FreeToken is the most serious attempt yet to run 284B-class MoE models on a gaming PC, backed by a credible research team, but it is four months old, NVIDIA-only and carrying 492 open issues.

What is FreeToken?

FreeToken runs open-weight mixture-of-experts (MoE) models, the kind where only a few of many expert sub-networks fire for each token, on personal computers rather than datacentre servers. It treats the GPU, CPU, system RAM and the links between them as one pool: hot experts are cached on the GPU, the rest are streamed or computed on the CPU, and the split between expert cache and KV memory can change at runtime without a restart.

The accompanying arXiv paper, whose 11 authors include Ion Stoica, Matei Zaharia, Song Han and Kurt Keutzer, claims more than 20 supported MoE models and reports serving a 35B model on a laptop, a 284B model on a gaming desktop and the 753B GLM-5.2 on a single workstation GPU. FreeToken exposes OpenAI- and Anthropic-compatible APIs, can configure and launch Claude Code, Codex or OpenCode against itself, and has a desktop app for Windows and Linux. It is Apache 2.0 licensed, was created on 20 July 2026, has about 17,564 stars and released v0.2.2 on 9 November 2026.

Is FreeToken worth it?

Running a frontier-sized open model at home has meant either heavy quantisation in llama.cpp or a slow crawl through system memory. FreeToken's argument is that MoE models make a third path possible: because each token touches only a few experts, a consumer GPU can hold the busy ones while the CPU serves the rest, if the engine keeps re-deciding which is which. Its bandwidth-adaptive policy, global LRU expert cache and double-buffered prefill streaming are built for exactly that, and the paper, submitted on 17 August 2026, reports a 753B model on one workstation GPU and 284B on a gaming desktop.

The agent integration is the practical draw. FreeToken serves both the OpenAI and Anthropic APIs, so ft launch claude writes Claude Code's provider config, installs the CLI if it is missing and points it at your local server. Semantic anchor checkpoints save recurrent and KV state at tool-call and thinking-block boundaries, so an agent that edits its own context does not force the whole prompt to be recomputed. The v0.2.2 release added speculative decoding; on an H100 it lifted Qwen3.8-27B from 50.1 to 215.0 tokens a second with a DFlash 2 draft, a 4.29 times gain, though that is a datacentre card, not the consumer hardware the project is named for.

Security defaults are reasonable. The server binds to 127.0.0.1 on port 1919, and an API key is off unless you set --api-key or FREETOKEN_API_KEY; set one before binding to another address. Model weights come from Hugging Face or local files.

The limits are those of a young, fast-moving project. Support is NVIDIA RTX 30, 40 and 50 series only, so there is nothing for Macs or AMD cards. The repository was created on 20 July 2026 and has 492 open issues, the docs already include a guide to repairing old FTW checkpoints, and speculative decoding does not yet work above one GPU of tensor parallelism. Against Ollama it is far more capable with huge MoE models and far less polished for everything else.

Where it earns its place

  • A developer with an RTX 50-series gaming PC and 128 GB of RAM

    Run Claude Code against a large open model without sending code to a provider

    ft serve a supported MoE checkpoint, then ft launch claude to wire Claude Code to the local Anthropic-compatible endpoint. No token bill; speed depends on how many experts fit in VRAM.

  • A research engineer benchmarking local inference for an internal report

    Compare MoE serving on consumer hardware against llama.cpp

    Run ft bench bw so the hybrid CPU-GPU policy measures the machine, then serve the same GGUF in both engines and compare with the timings report enabled.

Before you commit

Setup
Install the desktop app for Windows or Linux from flashml.ai, or uv pip install "freetoken[accel]"; then ft serve --model with a Hugging Face id, local folder or GGUF. The server is ready when it logs 127.0.0.1:1919.
Integrations
OpenAI API (/v1/chat/completions, /v1/responses) and Anthropic API (/v1/messages); ft launch for Claude Code, Codex, OpenCode, OpenClaw, Hermes and DeepSeek Harness; GGUF, MXFP4, NVFP4, FP8 and BF16 checkpoints; a terminal chat shell.
Data and privacy
Runs entirely on your machine. The server listens on 127.0.0.1:1919 by default with no API key unless you set one, so add a key before exposing it on a network. ft launch rewrites the chosen agent's provider config; --dry-run shows the changes first.
Cost at scale
No licence cost. The bill is the hardware: system RAM and a recent NVIDIA GPU sized to the model, plus power.
Limits
NVIDIA RTX 30, 40 and 50 series only; speculative decoding unsupported above TP 1 and for some windowed-attention models; 492 open issues; Windows and WSL cap pinned memory, which needs the auto CPU-layer setting.

What’s good

  • Runs very large MoE models by splitting experts across GPU, CPU and RAM
  • OpenAI and Anthropic APIs, with one-command setup for Claude Code, Codex and OpenCode
  • Context-edit-aware caching avoids recomputing agent prompts
  • Localhost binding by default; optional API key on every route

What’s not

  • NVIDIA RTX 30, 40 and 50 series only; no Mac or AMD support
  • 492 open issues four months after launch
  • Headline speed-ups measured on an H100, not consumer cards
  • Docs already need a guide to repairing old checkpoints

Best for

Developers with a recent NVIDIA gaming or workstation GPU who want to run very large open MoE models locally behind their coding agents.

FreeToken alternatives

FreeToken: common questions

Is FreeToken worth it?
FreeToken is the most serious attempt yet to run 284B-class MoE models on a gaming PC, backed by a credible research team, but it is four months old, NVIDIA-only and carrying 492 open issues. We rate it 4.0 out of 5.
How much does FreeToken cost?
Free and open source (Apache 2.0), including the desktop app. Your only cost is the hardware and electricity.
What are the downsides of FreeToken?
NVIDIA RTX 30, 40 and 50 series only; no Mac or AMD support; 492 open issues four months after launch; headline speed-ups measured on an H100, not consumer cards; docs already need a guide to repairing old checkpoints.
Who is FreeToken best for?
Developers with a recent NVIDIA gaming or workstation GPU who want to run very large open MoE models locally behind their coding agents.
How much does FreeToken cost for a team or in production?
No licence cost. The bill is the hardware: system RAM and a recent NVIDIA GPU sized to the model, plus power.
Is FreeToken safe for company data?
Runs entirely on your machine. The server listens on 127.0.0.1:1919 by default with no API key unless you set one, so add a key before exposing it on a network. ft launch rewrites the chosen agent's provider config; --dry-run shows the changes first.
What are the best FreeToken alternatives?
The closest alternatives we have reviewed are llama.cpp, Ollama, AirLLM and SGLang.

Answers checked Nov 21, 2026.

Make FreeToken? Show this rating on your siteFreeToken rated 4.0 out of 5 on ReviewRamble