Chunking
Chunking is the step that splits long documents into smaller passages before they are embedded and indexed, so a retrieval system can find and return only the parts of a document that answer a query.
Also called: document chunking, chunk size, chunk overlap, semantic chunking
By Divya Sharma · Checked Nov 15, 2026 · How we review
A language model cannot read a whole document library at once, and an embedding of a 200-page PDF says little about any single page. Chunking cuts the source into passages small enough to embed, search and paste into a prompt. It is the first design choice in a retrieval-augmented generation pipeline, and a bad choice here limits everything downstream.
How it works
The simplest method uses a fixed size: every N words, characters or tokens. The original RAG paper split each Wikipedia article into 100-word chunks. Most splitters add chunk overlap, repeating a few sentences at each boundary so an idea that spans two chunks is not lost. Recursive splitters try natural breaks first (sections, then paragraphs, then sentences) and fall back to smaller units only when a piece is still too long. Structure-aware chunking uses the headings and tables found during document parsing.
Semantic chunking
Semantic chunking places breaks by meaning instead of length. It embeds each sentence and starts a new chunk where the similarity between neighbouring sentences drops, so each chunk holds sentences about one topic. It costs an embedding call per sentence, and the breakpoint threshold needs tuning per corpus.
Trade-offs
Small chunks match queries precisely but lose context; large chunks keep context but dilute the embedding and use more of the context window. There is no universal best size, so test a few settings against real questions. The chunking stage guide compares the tools.
Related terms
- Retrieval-augmented generation
- Retrieval-augmented generation (RAG) is a pattern where a system first searches a collection of documents for passages relevant to a question, then gives those passages to a language model so its answer draws on them.
- Document parsing
- Document parsing converts files such as PDFs, scans, slides and spreadsheets into structured text that software can use, keeping headings, reading order, tables and images instead of a flat stream of characters.
- Embedding
- An embedding is a list of numbers (a vector) that represents a piece of text, an image or other data, produced so that items with similar meaning end up with vectors close together.
- Vector database
- A vector database stores embeddings alongside their source records and finds the stored vectors closest to a query vector, usually with an approximate index so search stays fast across millions of items.
- Context window
- A context window is the maximum amount of text, counted in tokens, that a language model can consider at once, covering the system prompt, conversation, documents, tool results and the response it writes.
- Reranking
- Reranking is a second search step that takes a short list of candidate documents and reorders them by relevance to the query, using a more accurate but slower model than the first search.