What is AirLLM?
AirLLM, by Gavin Li, wraps Hugging Face transformers so that only one layer of the model sits on the GPU at any moment. On first load it splits a checkpoint into per-layer files on disk; during generation it streams each layer in, runs it and drops it, so the memory needed depends on the size of a layer rather than the whole model. For mixture-of-experts models it can load only the experts each token is routed to.
The README lists Llama, Qwen, DeepSeek, Mistral, Phi, Gemma, ChatGLM and Kimi K3 among supported families, with optional 4-bit and 8-bit block-wise compression, a CPU path and an Apple silicon path limited to Llama-style models. Version 4.0.0, released in September 2026, added LoRA fine-tuning with the same layer streaming. The code is Apache-2.0.
Is AirLLM worth it?
AirLLM's memory claims hold up; speed is the price, and the README never states it. The headline says 70B models run on a 4GB GPU and Kimi K3, a 2.8T-parameter model, in under 4GB. The one speed figure the maintainers have published comes from the Kimi K3 release: on an RTX 6000 Ada, peak VRAM was 3.72 GB, set-up took 900 seconds and generation ran at 292 seconds per token, four tokens in 1,168 seconds. That is almost five minutes a token, and the notes call it disk-bound.
User reports on the issue tracker put smaller models in the same territory. One measured Qwen2.5-32B at 13.3 seconds per token on an NVIDIA GB10; another reported a 3B model at 28.6 seconds per token on an RTX 5060 Ti under Windows; a third waited about 30 minutes for a 120-token answer from gpt-oss-20b on an RTX 3090. The cause is structural: every new token passes every layer through the GPU again, so throughput is set by disk and PCIe bandwidth rather than by the card. The README also promises that 4-bit or 8-bit compression speeds inference up by as much as 3x. An open issue from August 2026 measured the reverse on Qwen2.5-32B: 8-bit was 3.8 times slower and 4-bit 8 times slower than no compression, because each layer is expanded back to 16-bit on every pass.
The catch: beyond speed, AirLLM needs a lot of disk. Splitting a model writes a second copy, so the README tells you to free space or set `delete_original`; Qwen3.8-Flash-Next needs about 360GB of checkpoint disk and a 64GB host, and Kimi K3 is a 1.56TB checkpoint that would need 3.12TB if split naively. The README's model table lists Flash-Next at about 180B parameters while its update notes say 125B. On macOS only Llama-style models run, and the README opens with promotional links to unrelated AI apps. Inference itself is local; the only network use described is downloading models, with a Hugging Face token for gated ones.
Our take: AirLLM suits one-off jobs where waiting is acceptable, such as checking how a 70B or 671B model answers a handful of test prompts before renting GPUs for it, or training a LoRA overnight on a gaming card. For anything interactive, a quantised model that fits in memory through llama.cpp or Ollama will serve you far better.
Where it earns its place
A PhD student with one RTX 3090
Compare how a 70B and a 671B model answer 20 evaluation prompts before asking for cluster time
Load each with `AutoModel.from_pretrained` and run the prompts as an overnight batch job; the README puts DeepSeek-V3 at about 12GB of VRAM. Leave disk for the split copy and expect seconds to minutes per token.
An ML engineer at a small start-up
Fine-tune a LoRA adapter on a 27B model without renting a GPU
Version 4.0.0 streams frozen weights from disk and keeps only the adapters on the GPU; the release notes put Qwen3.8-27B at about 2GB at a sequence length of 512. Run `train_qwen38_lora.py` against a JSONL file and leave it running.
A hobbyist with an Apple silicon Mac
Try a Llama-family model larger than the Mac's memory
Install mlx and torch, then use the same API; only `LlamaForCausalLM` and `MistralForCausalLM` architectures run on macOS. One user measured TinyLlama-1.1B at about 3 seconds per token on an M1 Pro, so keep prompts short.
Before you commit
- Setup
- `pip install airllm`, then `AutoModel.from_pretrained(<Hugging Face ID>)`. The first run downloads the checkpoint and splits it layer by layer, which needs roughly as much free disk again. Some models need a specific transformers version or flash-attn.
- Integrations
- Hugging Face transformers and Hub; bitsandbytes for optional compression; mlx on macOS. The README describes no server or OpenAI-compatible endpoint.
- Data and privacy
- Runs on your own machine; the only network use described is downloading models from Hugging Face, with an optional `hf_token` for gated models.
- Cost at scale
- Free; the real costs are disk (about 360GB for Qwen3.8-Flash-Next, a 1.56TB checkpoint for Kimi K3) and time, since every token streams the whole model through the GPU again.
- Limits
- Throughput is disk-bound; the Kimi K3 pull request says memory at long context was not tested; macOS supports Llama-style models only; in one user benchmark compression made generation slower, not faster.
What’s good
- Runs 70B to 2.8T-parameter models in single-digit gigabytes of VRAM, per the README and release notes
- One `AutoModel.from_pretrained` call for most Hugging Face models
- Streamed LoRA fine-tuning on small GPUs since version 4.0.0
- Apache-2.0 and fully local
What’s not
- Kimi K3 generates at 292 seconds per token; users report 13.3 to 28.6 seconds per token on 3B to 32B models
- The README's 3x compression speed-up is contradicted by a user benchmark that found 4-bit 8 times slower
- Needs disk for a second, split copy of every model
- macOS limited to Llama-style models
Best for
Researchers and hobbyists who need to run or fine-tune a model far larger than their GPU for a handful of prompts and can wait minutes for each answer.
AirLLM alternatives
AirLLM: common questions
- Is AirLLM worth it?
- AirLLM really does fit giant models into a few gigabytes of GPU memory, but at seconds to minutes per token it is a way to run a model once, not to chat with it. We rate it 3.3 out of 5.
- How much does AirLLM cost?
- Free and open source under Apache-2.0; no paid tier.
- What are the downsides of AirLLM?
- Kimi K3 generates at 292 seconds per token; users report 13.3 to 28.6 seconds per token on 3B to 32B models; the README's 3x compression speed-up is contradicted by a user benchmark that found 4-bit 8 times slower; needs disk for a second, split copy of every model; macOS limited to Llama-style models.
- Who is AirLLM best for?
- Researchers and hobbyists who need to run or fine-tune a model far larger than their GPU for a handful of prompts and can wait minutes for each answer.
- How much does AirLLM cost for a team or in production?
- Free; the real costs are disk (about 360GB for Qwen3.8-Flash-Next, a 1.56TB checkpoint for Kimi K3) and time, since every token streams the whole model through the GPU again.
- Is AirLLM safe for company data?
- Runs on your own machine; the only network use described is downloading models from Hugging Face, with an optional `hf_token` for gated models.
- What are the best AirLLM alternatives?
- The closest alternatives we have reviewed are llama.cpp, Ollama and Unsloth.
Answers checked Nov 18, 2026.