Most of the attention in a RAG (Retrieval-Augmented Generation) build goes to the embedding model, the vector database, the reranker, or the LLM.
One decision gets far less attention but can quietly affect the whole pipeline:
How you chunk your documents.
Chunking is how you split documents into smaller pieces before embedding and indexing them. It looks like a minor preprocessing step.
It isn't.
If your chunks don't preserve the right context, retrieval can fail to surface information that's clearly sitting in your source documents — even when the embedding model, vector database, and LLM are all working as designed.
A realistic failure mode
Imagine a document containing:
"The platform experienced a 20% increase in transaction volume after the new release. This increase was primarily driven by..."
If the chunk boundary falls between those two sentences, the first chunk has the number but not what it refers to. The second has the explanation but not the number.
Neither chunk represents the complete idea.
The document hasn't lost any information. The retrieval system has simply been handed fragmented representations of it.
Vector similarity search may miss the relevant information because the embedding of each fragment no longer represents the complete idea. Lexical retrieval such as BM25 may suffer too, because important matching terms can end up split across different chunks.
This is an easy-to-miss failure mode in RAG systems, and it can sometimes be mistaken for a problem with the embedding model or the LLM when the problem actually occurred earlier, during chunking.
Why chunking matters
A chunk is the basic unit your retrieval system tries to match against a user's query.
Get the size or the boundary wrong, and everything downstream can suffer.
Too small, and a chunk loses context:
"It increased by 20%."
That tells you very little without knowing what "it" refers to.
Too large, and a chunk can mix unrelated ideas, product information, configuration steps, troubleshooting, and pricing, forcing one representation to cover several topics at once. That can make fine-grained retrieval less precise.
And even at the right size, a chunk can still be cut in the wrong place, splitting a sentence, procedure, table, or argument in half.
Good chunking means building retrieval units that are:
Relevant enough to match a query
Large enough to preserve necessary context
Small enough to stay focused
Aligned with the structure and meaning of the source material
Four chunking strategies
Different chunking approaches make different trade-offs.
1. Fixed-size chunking
Split the document into chunks of roughly equal size.
For example:
500-token chunks
50-token overlap
This approach is simple, fast, predictable, and relatively inexpensive.
The downside is that boundaries follow token or character counts rather than meaning. A sentence, paragraph, or related idea can be split across chunks.
Overlap can reduce the impact of those boundaries, but it doesn't eliminate the problem.
Fixed-size chunking is a solid baseline. The mistake is assuming one configuration will work equally well for every document type.
2. Hierarchical chunking
Hierarchical chunking maintains multiple levels of context.
For example:
Document → Section → Subsection → Chunk
Smaller child chunks can be used for precise retrieval, while larger parent chunks can provide additional context when constructing the response.
This gives you a useful combination of retrieval precision and broader context.
The trade-off is complexity. You need additional relationships or metadata, and context assembly becomes more involved.
This approach can be particularly useful for technical documentation, policies, manuals, and other structured content where a detail's surrounding section matters.
3. Semantic chunking
Semantic chunking attempts to create boundaries based on meaning rather than only document length.
Instead of:
"Split every 500 tokens."
The system tries to identify where the topic or semantic relationship between sentences changes.
This can help preserve semantic coherence and reduce arbitrary boundaries.
However, semantic chunking isn't automatically the "best" option.
Its practical constraints around input size, cost, latency, and implementation complexity can vary significantly depending on the system. More sophisticated chunking also doesn't necessarily outperform simpler approaches for every dataset.
The right approach is to test it against your own data rather than assume that more sophisticated means better.
4. No chunking
For short documents, treating the entire document as a single retrieval unit can be perfectly reasonable.
Nothing is split, so there is no chunk-boundary context loss.
For longer, multi-topic documents, however, fine-grained retrieval becomes more difficult because a single representation has to cover a much larger amount of information.
There can also be input-size limitations depending on the embedding model and ingestion system.
So the real question isn't:
"Should I chunk?"
It's:
"What should the retrieval unit be for this content?"
A legal contract, an FAQ page, a technical manual, and a chat transcript don't naturally break into the same kinds of retrieval units.
How to evaluate what you've built
One of the biggest mistakes in RAG evaluation is judging only the final LLM answer.
If the retriever never returns the right chunk, no prompt tweak or model swap can fix that. The information was never available to the model in the first place.
Treat retrieval as its own component and evaluate it separately from generation.
For a representative set of user questions, check:
Was the correct document retrieved?
Was the correct chunk retrieved?
Was it ranked highly enough?
Were relevant neighboring chunks also retrieved?
How many irrelevant chunks came back?
Does the answer depend on information spread across multiple chunks?
Standard information-retrieval metrics such as Recall@K, Precision@K, and Mean Reciprocal Rank (MRR) can help measure retrieval quality independently of the final generated response.
You can also compare different chunking strategies, for example:
Fixed-size
Fixed-size with overlap
Semantic
Hierarchical
Evaluate them against your own representative question set and consider operational characteristics such as latency, index size, storage, and ingestion cost.
The goal isn't to find the most sophisticated chunking strategy.
It's to find the strategy that works best for your workload.
Match chunking to the document
There is no universal "best chunk size."
Different types of content have different natural retrieval boundaries.
Legal contract
A clause may need to remain associated with its surrounding section.
Technical documentation
A procedure may need to remain associated with its prerequisites and configuration details.
FAQ
The question and answer are usually a natural retrieval unit.
Research paper
The relationship between a claim, its explanation, and supporting evidence may matter more than an arbitrary token boundary.
Chat transcript
Speaker, topic, and conversation boundaries may be more useful than raw token counts.
The chunking strategy should reflect the information architecture of the source data.
Practical takeaway
Chunking is usually treated as a small, forgettable preprocessing step:
Split the document, embed it, load it into the vector database, and move on.
In practice, it is one of the important design decisions in a RAG pipeline.
If your retrieval units don't preserve the information your users are asking about, the system may struggle to find the answer, regardless of how good your embedding model or LLM is.
The lesson isn't that semantic or hierarchical chunking is always better than fixed-size chunking.
It isn't.
The lesson is that chunking should be designed around your data, your retrieval strategy, and the actual questions your users ask, then evaluated rather than assumed.
Before tuning your embedding model, vector database, reranker, or prompt, ask the simpler question first:
Is my chunking strategy preserving the context my retrieval system needs to find?
Good RAG doesn't start with the LLM.
It starts with getting the right information into the retrieval system in a form it can actually find.
