What is MiniMind?
MiniMind is a teaching project for building a small language model end to end. It walks through the whole pipeline on a single GPU: a custom BPE tokenizer, pretraining, supervised fine-tuning, LoRA, preference tuning (DPO), reinforcement learning (PPO, GRPO, CISPO), tool use and distillation, each written directly in PyTorch rather than called through a framework wrapper.
The smallest main-branch model, minimind-3, is about 64M parameters, roughly one 2,700th the size of GPT-3. Jingyao Gong has maintained it since 2024, and the April 2026 rewrite aligned the architecture with the Qwen3 and Qwen3-MoE families and added an agentic reinforcement-learning script. Spin-offs cover vision (MiniMind-V), any-to-any (MiniMind-O), a diffusion language model and a linear-attention variant. The code, datasets and trained weights are all open.
Is MiniMind worth it?
The value is the reading, not the resulting model. Because every stage is written out in native PyTorch, with the datasets and the trained weights published alongside, you can follow a tokenizer, a pretraining loop, SFT, DPO and GRPO line by line and run each on a single card. That is a gap most tutorials leave open: they fine-tune an existing model in a dozen lines of `transformers` and never show the loop underneath.
The headline claim is narrower than it sounds. The "2 hours, 3 CNY" figure is one epoch of the minimal pretrain-plus-SFT run on a rented NVIDIA 3090 at about 1.3 CNY an hour, not a full training. Fuller runs in the docs take far longer, and the 64M model scores near chance on knowledge benchmarks: 24.89 on C-Eval and 28.49 on ARC-Easy, close to what a larger SmolLM2-135M or TinyLlama beats comfortably. The project is honest that a model this size is for learning, not use.
It is also genuinely maintained, not a one-off: 65,230 stars, commits through late October 2026, and third-party benchmark support through lm-eval. Read the fine print on "from scratch" for the spin-offs, though; MiniMind-V continues from the base model's weights rather than starting from zero.
The catch: the write-up is clearest in Chinese and the English README is a translation, some benchmark datasets and parts of the synthetic SFT data carry a non-commercial licence even though the code is Apache 2.0, and you need a GPU. A 3090 or similar is assumed; the quick run is tuned for 24 GB.
Our take: nanochat teaches the same lesson in one script; MiniMind spreads it across the whole training menu, including the reinforcement-learning stages nanochat leaves out, and documents each one.
Where it earns its place
A computer science student new to language models
See what pretraining and fine-tuning actually do, not just call a library
Run the minimal pretrain plus SFT on a single 3090, about 2.31 hours and roughly 3 CNY of rented GPU time, then read the loop that produced the chat model. The published weights let you skip training and inspect a finished model too.
An ML engineer learning reinforcement learning for LLMs
Implement DPO and GRPO once without a framework abstracting them away
The repository implements DPO, PPO, GRPO and CISPO from scratch in PyTorch with the matching datasets, so you can step through a preference-tuning or agentic-RL run on a 64M model where a full-size one would need a cluster.
A workshop or course instructor
Give attendees a complete training pipeline they can run on their own laptops or a cheap cloud GPU
Point them at the open datasets and the quick config; a full "Zero" chat model trains from scratch in about two hours on one card, so everyone finishes in a session. Check dataset licences before any commercial reuse.
Before you commit
- Setup
- Clone the repository, `pip install -r requirements.txt`, download the dataset from ModelScope or Hugging Face into `./dataset/`, then run the pretrain and SFT scripts; a single 3090-class GPU is assumed.
- Integrations
- Native PyTorch with transformers, trl and peft compatibility; export to llama.cpp, vLLM and Ollama; DDP and DeepSpeed for multi-GPU; wandb or SwanLab logging; an OpenAI-compatible API server and a Streamlit web UI.
- Data and privacy
- All training data is published and runs locally on your own machine; nothing is sent anywhere. Sources are cleaned and deduplicated public and distilled corpora under mixed licences, including Apache 2.0 and CC-BY-NC.
- Cost at scale
- Only GPU time. The quick run is about 3 CNY on a rented 3090; the docs estimate roughly 50 CNY for the 26M model and about 159 CNY for the 0.1B model across the full pipeline.
- Limits
- A 64M model is for learning, not production; knowledge-benchmark scores sit near chance; the documentation is strongest in Chinese; some datasets are non-commercial.
What’s good
- Every stage in readable native PyTorch, from tokenizer to RL
- Open datasets and trained weights, not just code
- Runs the quick pretrain on a single 24 GB GPU
- Actively maintained since 2024 with spin-off models
What’s not
- The 64M model scores near chance on knowledge benchmarks
- "2 hours, 3 CNY" covers only the minimal run
- Clearest in Chinese; English is a translation
- Some datasets are non-commercial despite the Apache code
Best for
Engineers and students who want to understand the full LLM training pipeline by running every stage themselves on one GPU.
MiniMind alternatives
MiniMind: common questions
- Is MiniMind worth it?
- MiniMind is the clearest hands-on way to learn how a language model is trained, from the tokenizer to reinforcement learning, without a cluster or a framework hiding the parts. We rate it 4.4 out of 5.
- How much does MiniMind cost?
- Free and open source (Apache 2.0). The only cost is GPU time; the README's quick run is about 3 CNY of rented 3090 time.
- What are the downsides of MiniMind?
- The 64M model scores near chance on knowledge benchmarks; "2 hours, 3 CNY" covers only the minimal run; clearest in Chinese; English is a translation; some datasets are non-commercial despite the Apache code.
- Who is MiniMind best for?
- Engineers and students who want to understand the full LLM training pipeline by running every stage themselves on one GPU.
- How much does MiniMind cost for a team or in production?
- Only GPU time. The quick run is about 3 CNY on a rented 3090; the docs estimate roughly 50 CNY for the 26M model and about 159 CNY for the 0.1B model across the full pipeline.
- Is MiniMind safe for company data?
- All training data is published and runs locally on your own machine; nothing is sent anywhere. Sources are cleaned and deduplicated public and distilled corpora under mixed licences, including Apache 2.0 and CC-BY-NC.
- What are the best MiniMind alternatives?
- The closest alternatives we have reviewed are nanochat and Unsloth.
Answers checked Nov 14, 2026.
Make MiniMind? Show this rating on your site
