Pretraining
Pretraining is the first and largest stage of training a model, where it learns general patterns from a very large unlabelled dataset, usually by predicting the next token, before any task-specific tuning.
Also called: pre-training
By Divya Sharma · Checked Nov 15, 2026 · How we review
Pretraining is where a language model gets most of its knowledge and ability. The model reads a huge corpus of web pages, books, code and other text, and is trained to predict the next token at each position. No one labels the data; the text itself supplies the right answers. The result is a base model.
Why it matters
Pretraining is by far the most expensive step, so only a few organisations do it for large models. Almost everyone else starts from a pretrained checkpoint and adapts it. The GPT-3 paper describes this pattern: pretrain on a large corpus, then fine-tune for a task. The 2021 report that coined the term foundation model used it for models trained on broad data and then adapted to many tasks.
What goes into pretraining shapes everything after it: which languages the model handles, which coding styles it knows, what it repeats and its knowledge cutoff. See AI training data for where that data comes from.
How it differs from fine-tuning
Fine-tuning and post-training (such as instruction tuning and RLHF) use far smaller, curated datasets to change how the model behaves: following instructions, answering in a format, refusing some requests. They add relatively little new knowledge. If a model does not know a fact, fine-tuning is usually a poor fix; retrieval is often better.
Continued pretraining
Teams sometimes run more pretraining on a domain corpus, such as legal or medical text, before fine-tuning. This sits between the two stages and costs much less than pretraining from scratch.
Related terms
- Fine-tuning
- Fine-tuning is further training of a pretrained model on a smaller, task-specific dataset, so the model adopts a particular skill, format, tone or domain without being trained from scratch.
- Foundation model
- A foundation model is a large model trained on broad data at scale that can be adapted to many different downstream tasks, through prompting or fine-tuning, instead of being built for one task.
- AI training data
- AI training data is the collection of examples a model learns from, such as web text, code, images, transcripts or labelled pairs. Its size, quality, mix and licensing shape what the model can do and where it fails.
- Base model
- A base model is a language model straight out of pretraining, before instruction tuning or human-feedback training. It continues whatever text it is given rather than following instructions or holding a conversation.
- Knowledge cutoff
- A knowledge cutoff is the date after which a model has little or no information, because its training data was collected up to around then. Events after it are unknown unless supplied in the prompt.