Chunking is choosing the unit of meaning
Chunk size in RAG is not a random hyperparameter. It is a decision about information density: the smallest unit that preserves the meaning needed to answer real queries.
If you have built a RAG system, you have had this experience: you change chunk size from 300 to 600 tokens, the answers change dramatically, and nothing else in the pipeline changed. That feels like an arbitrary hyperparameter. It is not. Chunking is an information problem. You are choosing the unit of meaning your retrieval channel can transmit, and bad chunking reduces signal-to-noise in a way that eventually gets blamed on the model.
Retrieval is a noisy communication channel
The useful reframe is: RAG is an information channel from your corpus to the model. Your query is the signal. Retrieval is the channel. The chunks you return are the message delivered to the model. Channel problems show up as missing the right message, delivering the wrong message, or delivering too much irrelevant content. Chunking is how you define the alphabet of messages the channel can send.
A chunk that is too small loses context. Pronouns break: "it" and "this" become ambiguous. Definitions and setup go missing. Similarity search pulls in wrong neighbors because the fragment is too thin to carry a stable meaning. A chunk that is too large becomes an embedding of many meanings averaged together. Retrieval finds "kind of related" chunks rather than the right one, and the model must search inside the chunk at generation time, spending attention on noise you paid to inject.
Same support-doc paragraph, re-chunked by raw word count. Watch how often a boundary lands mid-sentence.
Semantic boundaries beat fixed counts
The reason is information density. A chunk aligned to a section boundary, a Q/A pair, or a heading-to-heading span tends to package one coherent idea. A fixed-token chunk is blind to meaning and regularly cuts across boundaries, producing fragments that are half of something.
The practical rule: if your documents have headings, chunk by heading and include the heading text in the chunk. That single change, embedding the heading with the content, often produces more retrieval improvement than any parameter tuning. A chunk without its header is a paragraph ripped from a book without the chapter title.
RAPTOR (Sarthi et al., 2024) is a concrete demonstration of structure-aware retrieval over fixed-size chunking: on the QuALITY benchmark it reached 62.4% accuracy, a 5.1-point improvement over a BM25 baseline and 2 points over dense passage retrieval, by indexing recursively summarized tree nodes instead of flat fixed-length chunks.
| Symptom | Root cause | Fix |
|---|---|---|
| Retrieval returns adjacent chunks from the same doc | Chunks too small; split across one idea | Merge to section-level boundaries |
| Retrieved chunks mention the topic once in passing | Chunks too large; high noise-to-signal | Split at heading or intent boundaries |
| Model answers 'kind of related' but not precise | Low retrieval precision | Add reranking + tighten chunk boundaries |
| High token cost per answer | Too many large chunks injected | Reduce chunk size + add top-k budget |
| Model ignores retrieved content | Context window overwhelmed by noise | Filter by metadata before retrieval, reduce k |
Chunking is not a one-time decision. It is a system parameter you maintain with an eval set. The right chunk is the smallest unit that preserves the meaning needed to answer real queries. That is an information problem, not a formatting problem, and it is measured with retrieval hit-rate on labeled questions, not eyeballed from spot-checks.
Practitioners building RAG systems keep arriving at the same diagnosis:
Your chunking strategy is sabotaging your retrieval quality 🗡️ Not even the best embeddings and models can save you! Raw documents, especially long ones, often make terrible retrieval units. When chunks are too large, too small, or arbitrarily split, your RAG system struggles to Show more
References: RAPTOR (Sarthi et al., 2024), Pinecone, Weaviate