What it is
LLMs from Scratch is the code repository for Sebastian Raschka's book "Build a Large Language Model (From Scratch)" (Manning, September 2024, 368 pages). Its notebooks build tokenisation, attention, a GPT-style model, pretraining and instruction fine-tuning in plain PyTorch, without LLM libraries.
Bonus folders go well beyond the book, with from-scratch implementations of Llama 3.2, Qwen3 (dense and mixture-of-experts), Gemma 3, Olmo 3, Qwen3.5 and Gemma 4, plus preference tuning with DPO and evaluation through Ollama.
The code is free under Apache 2.0; the book lists at $59.99 in print or $54.99 as an ebook. The repository has no tagged releases; the last commit was on 2 October 2026.
Our review
This is the clearest route from knowing PyTorch to understanding how a GPT-style model works, because nothing is hidden behind a library call. With about 107,100 GitHub stars it is one of the most followed AI learning repositories, and Raschka still commits fixes, the latest on 2 October 2026.
The bonus material is what keeps it relevant two years after publication. Standalone notebooks rebuild recent open architectures, including Qwen3's mixture-of-experts layers and Gemma 4's E2B and E4B variants, so a reader can see exactly how a current model departs from the GPT-2-style design the book starts with. A 17-hour companion video course and a free 170-page quiz PDF from Manning suit different ways of learning.
Its limits are deliberate. The main chapters are designed to run on an ordinary laptop, so the models are small and pretraining is a demonstration rather than a usable assistant. It teaches internals, not application building: there is nothing on retrieval, agents or deployment. And while the notebooks run on their own, the reasoning behind each step lives in the paid book.
Karpathy's nanochat covers similar ground as one end-to-end training run on rented GPUs, and suits readers who want to watch a full pipeline produce a chat model. Raschka's version is slower and closer to a textbook, and its sequel repository, Reasoning From Scratch, carries on into inference-time scaling, reinforcement learning and distillation.
Where it earns its place
A backend developer moving into an ML team
Understand attention, tokenisation and fine-tuning well enough to debug real models
Work through a chapter a week on a laptop; chapter 7's instruction fine-tuning and its Ollama-based evaluation script close the loop. The cost is the $59.99 print book (often discounted at Manning) and evenings, not cloud GPUs.
A university lecturer running an NLP module
Build practical coursework around a working transformer
The Apache 2.0 licence allows reusing the notebooks in course materials, and Manning's free 170-page quiz PDF has about 30 questions per chapter. Students without GPUs can still run the main chapters, which use a GPU automatically when one is present.
An ML engineer assessing a new open model's architecture
See what changed in Qwen3 or Gemma 4 without reading a large framework
The standalone bonus notebooks implement each model in a single file, so differences in attention, normalisation or expert routing are visible line by line. Running the larger variants with pretrained weights needs a GPU with enough memory to hold them.
Before you commit
- Setup
- Clone the repository and install the requirements with pip or uv; the setup guide covers both. The main chapters need no specialised hardware and switch to a GPU automatically if one is present.
- Data and privacy
- Runs locally. Optional chapter 7 material calls the OpenAI API or a local Ollama server for evaluation and dataset generation; nothing else leaves the machine. The code is Apache 2.0 and allows commercial reuse; the book text is copyrighted.
- Cost at scale
- The code is free and has no hosted or paid edition. The book lists at $54.99 (ebook) or $59.99 (print with ebook) from Manning, audio adds $24.99, and the separate video course is sold by Manning.
- Limits
- Only 4 open issues at the time of checking. There are no tagged releases, so track commits; the material teaches internals at small scale and stops short of production training or serving.
What's good
- Builds every component step by step in plain PyTorch
- Main chapters run on an ordinary laptop
- Bonus notebooks on Qwen3, Gemma 4 and Olmo 3
- Apache 2.0 code reusable in courses and products
What's not
- The explanations live in the paid book
- Nothing on RAG, agents or deployment
- Laptop-scale models are demonstrations, not usable assistants
Best for
Developers and students who want to understand LLM internals by building one.
Alternatives
LLMs from Scratch: common questions
- Is LLMs from Scratch worth it?
- A rare learning resource that makes every line of a working LLM legible, kept current in 2026 by its bonus notebooks. We rate it 4.7 out of 5.
- How much does LLMs from Scratch cost?
- Code is free under the Apache 2.0 licence; the companion book lists at $54.99 (ebook) or $59.99 (print).
- What are the downsides of LLMs from Scratch?
- The explanations live in the paid book; nothing on RAG, agents or deployment; laptop-scale models are demonstrations, not usable assistants.
- Who is LLMs from Scratch best for?
- Developers and students who want to understand LLM internals by building one.
- How much does LLMs from Scratch cost for a team or in production?
- The code is free and has no hosted or paid edition. The book lists at $54.99 (ebook) or $59.99 (print with ebook) from Manning, audio adds $24.99, and the separate video course is sold by Manning.
- Is LLMs from Scratch safe for company data?
- Runs locally. Optional chapter 7 material calls the OpenAI API or a local Ollama server for evaluation and dataset generation; nothing else leaves the machine. The code is Apache 2.0 and allows commercial reuse; the book text is copyrighted.
- What are the best LLMs from Scratch alternatives?
- The closest alternatives we have reviewed are nanochat and LLM Course.
Answers checked Oct 23, 2026.
Make LLMs from Scratch? Show this rating on your site
