RAG chunking strategies compared: why your retriever finds half the answer

If your retrieval system finds part of the answer most of the time, it is probably failing more often than your dashboard says. A set of October papers on chunking, and one careful experiment that ran five of their ideas side by side, point to the same lesson: how you cut a document decides what can be found, and the best cut depends on how much context you can afford to send.

Why chunking is back on the agenda

Retrieval-augmented generation (RAG) cuts your documents into pieces, finds the pieces that match a question and hands them to a model. The model can only use what it is given. The Bottleneck Is Retrieval, Not Reading (arXiv, 18 October) tested ten frontier and three open-weight models and found the same pattern in all of them: retrieval reliably surfaces some of the chunks a multi-fact question needs, but not all of them, and chunking strips out the document-level context that links one passage to another.

Three October papers attack that from different sides.

QESChunker (27 October) treats chunking as optimisation. Given example questions and the sentences that answer them, it picks the cut points that keep the most evidence retrievable. Without overlap, dynamic programming finds the exact best answer; with overlap, the problem is NP-hard, and a greedy pass gets within a proven bound of the best.

StrucChunk (19 October) uses the structure the author already wrote. It never cuts across a clause, resolves cross-references such as "see Section 4.2" before indexing, and fuses keyword search (BM25) with dense embeddings. On 501 questions about India's Code of Criminal Procedure, Recall@5 rose from 0.557 to 0.804, and the authors attribute 88% of that gain to chunking alone.

SpectRAG (27 October) stops treating chunks as independent. It builds a weighted graph over them from adjacency, section hierarchy, shared entities and citations, spreads relevance from the first search hits through that graph, and assembles a context under a fixed token budget.

One experiment, five chunkers

Vishal Mysore ran these ideas against each other in a browser lab anyone can rerun. The document is a fictional observatory handbook written like a policy manual, with numbered sections, definitions and cross-references. Retrieval is BM25 only, so every number is deterministic. Each chunker gets the same limits: chunks of up to 450 characters and a context budget of 120 words.

The five chunkers were a fixed window that cuts every few hundred characters, a sentence window that packs whole sentences, a structure-aware chunker in the StrucChunk style (one chunk per numbered section, heading path attached, cross-referenced definitions pasted in), and QESChunker in its exact and overlapping forms.

Every chunker kept every evidence sentence intact somewhere. The differences are about whether the right pieces can be found and fitted into the budget.

What the results say

Partial recall flatters. Every strategy found at least one evidence sentence for 69 to 92% of questions. The default chunkers assembled the complete answer less than a quarter of the time. A pipeline judged on partial hits would look healthy while three out of four multi-fact answers were built on half the evidence.

Small, structured chunks win when context is tight. Section-sized chunks averaged 285 characters against about 400 for the windows, so two or three fit in 120 words where one window does. At a 40-word budget, only the structure-aware chunker scored above 10%. By 300 words the window chunkers had caught up, and by 400 they had passed it, while structure-aware levelled off at 77%. With a generous budget, almost any chunker drags in the evidence, because you are sending a large share of the document.

Chunk size and budget must be set together. At 800-character chunks, the fixed and sentence windows scored 0%: an 800-character window is about 130 words, more than the 120-word budget, so not one chunk fitted. That is the failure you get when one person tunes the chunker and another tunes the prompt.

The provably optimal chunker came last. QESChunker's exact version scored 15% on held-out questions. It is optimal for its objective, which rewards keeping evidence sentences whole. The cheapest way to keep a rule whole is to cut immediately before it, and in a handbook the line before a rule is its heading. The heading ended up at the end of one chunk and the rule at the start of the next, so a keyword search for the heading found the wrong chunk. None of its chunks began at a heading. The lesson is general: a chunker optimises what you write down, and intact evidence is not the same as findable evidence. The overlapping version did much better, 46% held-out and 83% on the questions it was tuned on. That gap is the warning for any chunker trained on example questions: the index is shaped around the questions you thought to ask.

A chunk graph helps where definitions are hubs. Switching structure-aware chunking to graph retrieval raised it from 62% to 69%. For "Is flash photography allowed on the dome floor?", plain BM25 ranked the definition of the red-light zone third, and it no longer fitted the budget. In graph mode, two other sections cross-reference that definition, so relevance flowed to it and it made the cut. Turning off cross-reference resolution erased the gain. On the window chunkers the graph did not help, and for sentence windows it dropped the score from 23% to 15%.

Some misses are not chunking problems. "My 8-year-old is coming. What rule applies on the terrace?" failed everywhere, because the rule says "children under 10" and shares no words with the question. That gap needs dense or hybrid retrieval, query rewriting or a reranker.

What to change in your pipeline

For documents with real structure, such as contracts, policies, manuals and specifications, chunk by section, attach the heading path, and resolve cross-references at index time so a rule travels with the definition it depends on. Tools such as Docling and Unstructured recover that structure from PDFs; Chonkie handles the chunking itself.

Measure complete answers, not partial hits. A multi-fact question answered from half its evidence is a wrong answer that looks right. Ragas can score context recall per question.

Set chunk size and context budget in the same place, and test at the budget you actually send. The best chunker at 120 words was not the best at 400.

If you try a supervised or graph-based method, check that its objective matches the retriever you run. Two September papers back this up from other angles: a 54-configuration study found parser and chunker choices interact, with no single retriever best everywhere, and a comparison of six pipelines over about 464,000 papers found reranking and late interaction move quality more than first-stage tweaks. A third, on adaptive retrieval depth, reports over 36% better F1 and 14% fewer tokens on easy queries by choosing how many chunks to fetch per type of question.

Our take: chunking decides what can be retrieved; the retriever decides what is. Fix the first with structure, the second with hybrid search and a reranker such as Cohere Rerank 5 or Jina Reranker v3.5, and judge both on whether the whole answer arrives.