RAG Chunking Strategies Compared: Fixed vs Recursive vs Semantic
Your retrieval ceiling is set the moment you decide where to cut the document — before a single embedding is generated.
Key takeaways
- Chunking happens before embedding, so it sets a hard ceiling on retrieval quality — no amount of prompt tuning recovers a fact that was split across two chunks.
- Recursive character splitting is the safest default for most production RAG systems: it respects document structure at close to zero extra cost.
- Semantic chunking can lift recall on long, topic-diverse content, but it roughly doubles embedding calls and can produce fragments too short to be useful on their own.
- The chunk size that matters is the one you validate against your own questions — there is no single correct number for every corpus.
Why chunking sets your retrieval ceiling
Every RAG pipeline embeds chunks, not documents. If a chunk boundary falls in the middle of the one sentence that answers the user's question, no embedding model and no reranker can fully recover it — the model can only work with what got embedded in the first place. That makes chunking one of the few decisions in a RAG system that acts as a hard ceiling rather than a tunable parameter: you can swap embedding models, add a reranker, or upgrade the generator, and none of it fixes a chunk that cut the answer in half before indexing ever happened.
Take a support-docs example: "Refunds are processed within 5–7 business days. This does not apply to enterprise accounts, which follow the terms in their contract." If a fixed cut lands between those two sentences, a question about enterprise refund timing retrieves only the first sentence — and the model confidently returns the wrong policy. The failure is invisible to any metric that only checks whether an answer was returned; it only shows up when someone checks whether it was the right answer for that user.
This is also why chunking is easy to get wrong quietly. A pipeline with bad chunking still returns results, still generates fluent answers, and still looks like it's working. See how embeddings represent meaning for the layer that sits directly underneath this decision.
The four strategies, mechanically
Fixed-size chunking
Split every N characters or tokens, usually with some overlap to avoid losing context at the boundary. It is the simplest option available: no parsing, no extra model calls, and entirely predictable chunk sizes going into the index.
Fixed-size chunking works well on homogeneous, low-structure text — logs, transcripts, plain data exports — where there is no paragraph or heading structure to preserve in the first place.
Recursive character splitting
Instead of cutting blindly at a character count, a recursive splitter tries a hierarchy of separators in order — paragraph breaks first, then line breaks, then sentence endings, then words — falling back only when a chunk is still too large after trying the more natural boundary. A 2,000-character paragraph gets split at a sentence boundary, not mid-word, and two short paragraphs that both fit inside the target size stay together as one chunk.
This is the default most production systems and popular retrieval frameworks reach for first, and for good reason: it respects the document's own structure at essentially the same computational cost as fixed-size chunking, with none of the downside.
Semantic chunking
Embed individual sentences, measure the similarity between adjacent ones, and cut where that similarity drops — the idea being that a genuine topic shift is a better boundary than an arbitrary character count. It can meaningfully improve recall on long, multi-topic prose where paragraph breaks don't line up with where the subject actually changes.
Watch out: semantic chunking roughly doubles your embedding calls at index time — one pass to find the boundaries, another to embed the resulting chunks — and without a minimum chunk-size floor it can produce fragments too short to answer anything on their own.
Structure-aware and hierarchical chunking
For documents that already carry real structure — Markdown headings, HTML sections, PDF headings, source files — split on that structure directly: one chunk per section, or per function, rather than by character count. A hierarchical variant takes this further with parent-child chunking: index small, precise chunks for matching, but return the larger parent section they belong to for generation, giving the model more surrounding context than the exact chunk that was matched.
What published benchmarks actually show
Independent comparisons of chunking strategies on real retrieval tasks generally agree on a few points, even though the exact numbers shift by corpus and embedding model. Recursive splitting is consistently a strong, low-cost baseline. Fixed-size chunking stays close behind on homogeneous text but falls off on documents with real structure. Semantic chunking can push recall higher on topic-diverse, unstructured prose, but the gain doesn't always survive contact with a full retrieve-then-generate pipeline once you account for its extra cost and its tendency to over-fragment short passages.
| Strategy | Typical strength | Typical weakness | Extra cost |
|---|---|---|---|
| Fixed-size | Fast, predictable, zero setup | Ignores structure, breaks mid-idea | None |
| Recursive | Respects structure at low cost | Still character-count driven at the leaves | None |
| Semantic | Follows real topic boundaries | Slower; can over-fragment | ~2x embedding calls |
| Structure-aware / hierarchical | Matches the document's own sections | Needs structured input to work | One parsing pass |
There is no globally optimal chunk size. There is only the chunk size that scored best against your own questions.
Contextual retrieval: fixing chunks after they're cut
A newer technique, often called contextual retrieval, attacks the same problem from the other direction: instead of finding a smarter place to cut, it uses an LLM to prepend a short, chunk-specific summary of where that chunk sits in the wider document, before embedding it. A chunk that reads "Revenue grew 12% year over year" becomes something like "This chunk is from Q3's earnings report, in the section on European sales. Revenue grew 12% year over year" — recovering context that any fixed cutting point would otherwise throw away.
This costs one extra LLM call per chunk at index time, which is meaningfully more expensive than plain chunking, but it's a genuinely different lever from choosing where to cut: it repairs the information loss that every chunking strategy leaves behind, rather than trying to pick a smarter boundary in the first place.
Treat contextual retrieval as an addition to a chunking strategy, not a replacement for one — you still need to decide chunk size and overlap before deciding whether to enrich each chunk with extra context.
Picking a chunk size and overlap
Start narrower than feels natural. 300–500 tokens with 10–20% overlap is a reasonable first pass for prose-heavy documentation; go smaller for FAQ-style content where each answer is genuinely self-contained, and larger for reference material where a fact needs several surrounding sentences to make sense on its own.
- Too small: retrieval finds the right neighbourhood but each chunk lacks enough context to answer alone, forcing the generator to stitch together several retrieved fragments.
- Too large: retrieval precision drops because each chunk covers multiple topics, and irrelevant text dilutes the embedding's usefulness.
- Overlap exists specifically to stop a fact from being split across a boundary. Past roughly 20%, you are mostly paying storage and embedding cost for duplicated text without a proportional recall gain.
Don't guess. Build a 30–50 question set from questions people actually asked, and measure recall@5 for two or three chunk sizes before picking one — this evaluation habit matters more than which chunking method you start with.
Chunking by document type
| Document type | Recommended approach |
|---|---|
| Markdown or HTML docs | Structure-aware — split on headings first, then recursive within each section |
| Long-form prose, reports | Recursive, or semantic if the topic shifts frequently within sections |
| PDFs with tables and figures | Page- or element-aware chunking; avoid naive text extraction that merges columns |
| Source code | Split by function or class, never by raw character count |
| FAQs, tickets, short records | One chunk per record — chunking often isn't needed at all |
The most common mistake is using one chunking configuration for an entire mixed corpus. A PDF manual, a support-ticket export and a code repository don't share a chunking strategy just because they end up in the same vector database.
A simple decision framework
- Does the corpus already have real structure — headings, sections, code? If yes, chunk on that structure first.
- If not, start with recursive character splitting at 300–500 tokens and 10–20% overlap.
- Only reach for semantic chunking after measuring recursive splitting underperforming on genuinely topic-diverse, unstructured prose — and only if you can afford the extra embedding cost.
- For long reference material, consider hierarchical retrieval: small chunks for matching, larger parent sections for generation.
- Re-run your evaluation set every time you change the strategy. A change that looks obviously better can still regress specific question types.
Where chunking fits in the pipeline
Chunking sits upstream of everything else in retrieval: the index and metadata filtering that stores your chunks, and the decision of whether to reach for RAG or fine-tuning in the first place, both assume the chunks going in are coherent. Fix chunking before tuning the prompt, swapping the embedding model, or adding a reranker — in most RAG debugging sessions, the chunk boundary turns out to be where the failure actually started.
None of the four strategies is universally "best" — each trades setup cost against how well it respects the document's own structure. Pick the cheapest one that fits your document type, measure it against real questions, and only add complexity where the numbers say it earned its keep.
Frequently asked questions
What chunk size should I start with for RAG?
A common starting point is 300–500 tokens with 10–20% overlap for prose-heavy documentation, adjusted down for short self-contained content and up for reference material that needs surrounding context — validate the choice against a real question set rather than treating any number as fixed.
Is semantic chunking always better than fixed-size chunking?
No. It can improve recall on long, topic-diverse prose, but it roughly doubles embedding calls and can produce fragments too short to be useful — recursive character splitting remains a strong, much cheaper default for most documents.
Does chunk overlap actually improve retrieval?
A moderate overlap of 10–20% helps by preventing a fact from being split exactly at a chunk boundary. Pushing overlap much higher mostly adds storage and embedding cost for duplicated text without a proportional recall gain.