10decoders
Why chunking determines the ceiling on your retrieval quality
Chunking is not a preprocessing detail. It determines what the embedding model has to work with when it encodes each chunk, and it determines whether a retrieved chunk contains enough context for the LLM to use it to answer a question. A chunk that starts mid-sentence has an embedding that partially reflects the preceding text, context the embedding model never actually saw. The mid-paragraph cut is equally damaging: it removes the explanation that would have made the retrieved text usable. These failures happen on every split at a fixed character boundary that does not align with a semantic unit.
The relationship between chunk size and retrieval quality is not monotonic. Smaller chunks produce more granular retrieval: the embedding is a better match to a narrow query because it represents a focused idea rather than a paragraph covering multiple points. But smaller chunks also strip context: a 50-token chunk that precisely matches a query may not contain enough information for the LLM to produce a complete answer, even when it is retrieved at rank 1. Larger chunks carry more context but their embeddings are diluted across multiple ideas, making them weaker matches for specific queries. The optimal chunk size for a given corpus and query distribution sits somewhere in this tension, and it is different for different document types.
Overlap between consecutive chunks partially addresses the context-stripping problem. By including the last 50 to 100 tokens of the previous chunk at the start of the next one, overlap ensures that sentences split across a boundary appear in at least one complete chunk. It does not solve the semantic unit problem: a table row that spans a chunk boundary will still be split regardless of overlap. And overlap increases index size by roughly the overlap-to-chunk-size ratio, which affects storage costs and retrieval throughput at scale.
Your retriever can only find what your chunking preserved. A 90th-percentile embedding model cannot recover meaning from a chunk where the key sentence was cut in half at the character boundary.
The 5 chunking approaches compared: where each wins and where each fails
No single chunking strategy outperforms the others across all document types and query distributions. Each approach optimizes for a different property of the retrieval problem, and the right choice depends on whether your corpus is continuous prose, paginated structured documents, or a mix.
| Chunking approach | Best for | Where it fails | Production failure mode | Risk level |
|---|---|---|---|---|
| Fixed-size splitting (no overlap) | Simple implementation; fast indexing; acceptable for short, uniform documents where splits rarely cross sentence boundaries | Splits mid-sentence and mid-paragraph. Table rows, numbered list items, and clause-heavy sentences get truncated at arbitrary character positions | Default framework setting of 1,000 characters ships to production; retrieval recall is acceptable on simple queries but precision fails on queries that require a complete clause or table row; team attributes failures to the embedding model rather than chunk boundaries | Critical |
| Recursive splitting with overlap | Continuous prose, knowledge bases, long-form documentation, technical articles. Strong baseline: 69% end-to-end accuracy on academic documents (FloTorch 2026) | Overlap increases index size by 10-20%. Does not respect table or list boundaries. Fails on PDFs where extracted text loses structural markers | Works well in development on clean Markdown docs; degrades in production on PDFs where extraction produces garbled text that the recursive splitter cannot find clean split points for | High |
| Semantic chunking | Dense analytical text where sentence-level semantic coherence matters more than context length. High retrieval recall when minimum size floor is set | Default implementations produce very small chunks (~43 tokens average). Tiny fragments have high recall but insufficient context for answer generation; end-to-end accuracy drops to 54% despite 91.9% retrieval recall (FloTorch 2026) | Team sets up semantic chunking, sees high recall@k metrics, ships to production; LLM answers are incomplete or generic because retrieved chunks are too small to contain the full answer; team diagnoses as a generation problem rather than a chunk-size problem | High |
| Structure-aware chunking (section, heading, page boundaries) | PDFs with defined page layouts, legal documents with numbered clauses, financial filings with table-per-page structure. Page-level chunking achieved 0.648 accuracy on FinanceBench with the lowest variance | Requires document structure detection: PDF parser must identify headings, sections, and table boundaries accurately. Poorly-scanned PDFs and complex multi-column layouts produce incorrect structure detection | Structure detection works on well-formatted source documents and fails silently on scanned PDFs and older filings where the parser cannot identify section boundaries; chunks revert to arbitrary splits on those documents with no alert to the pipeline | Moderate |
| Hierarchical parent-child chunking | Mixed-complexity corpora where queries range from specific fact-finding to broad topic retrieval. Retrieve by small child chunks for precision; return parent chunk for context. De facto pattern in enterprise production at scale | Doubles index complexity: two indexes (child for retrieval, parent for return), plus the mapping logic between them. Adds latency for the parent-lookup step after retrieval | Parent-child mapping breaks during index updates when documents are re-chunked after edits; child chunks point to stale or deleted parent chunks; retrieval returns the right child but the context fetch fails silently and returns a truncated or empty result | Lower |
The most common chunking mistake in enterprise RAG is using the same strategy across a mixed document corpus. A pipeline that processes both continuous policy documents and paginated financial PDFs cannot use a single chunking approach that serves both well. Policy documents need recursive splitting with overlap to preserve sentence continuity. Financial PDFs are a different problem: page-level or section-level splitting keeps tables and financial data together rather than scattering them across chunk boundaries. Applying recursive splitting to a financial PDF that contains a 20-row table will cut that table into two or three chunks, distribute the rows across them, and produce retrieval where a specific row may be in a different chunk than the header row that gives it meaning.
Not sure which chunking strategy fits your document corpus?
10decoders audits your RAG pipeline's document processing layer against your actual corpus and query distribution, identifies where your current chunking configuration is limiting retrieval quality, and builds the corpus-appropriate chunking and evaluation layer your pipeline needs. Our DocuFindr platform handles complex enterprise document types including multi-column PDFs, clinical notes, and financial filings with structure-aware processing.
Book a Free AI Assessment →Why semantic chunking's recall advantage disappears at answer quality
Semantic chunking is appealing in theory because it respects sentence and paragraph boundaries rather than splitting at arbitrary character counts. In practice, the default implementations split aggressively: the average semantic chunk in the FloTorch 2026 benchmark was approximately 43 tokens. At that size, individual chunks typically contain one or two complete sentences, which is enough for an embedding to match a query accurately but not enough for an LLM to produce a complete, contextually-grounded answer from the retrieved text alone.
The result is a retrieval quality metric that looks strong (high recall@10, high MRR) while end-to-end answer quality is substantially lower than a coarser chunking strategy would produce. A team evaluating their RAG pipeline at the retrieval layer will see that semantic chunking finds the right chunks. The answer quality problem only surfaces when they evaluate at the generation layer and find that the retrieved chunks are too small to anchor complete answers. At that point, the fix requires re-embedding the entire corpus with a minimum size floor on chunk creation, which was the configuration decision that should have been made before indexing.
Semantic chunking with a minimum size floor addresses this. Setting a floor of 200 to 300 tokens prevents the splitter from creating single-sentence fragments while still preserving semantic boundaries for longer passages. This configuration is not the default in any major framework; it requires explicit parameter tuning. Measure end-to-end answer accuracy, not retrieval recall alone, when evaluating chunking strategies. The two metrics can diverge substantially, and the gap between them is exactly where the context-stripping problem lives.
Fixed-size or framework default
Framework default applied without corpus-specific tuning. It splits at character boundaries with no overlap and no awareness of document structure. Recall@k may look acceptable; end-to-end accuracy will reflect the boundary artifacts.
Strategy matched to document type
Prose corpora get recursive splitting at 400-600 tokens with 10-15% overlap. Structured PDFs get page-level or section-level splits instead. Measure end-to-end accuracy on a labeled eval set before re-embedding the full corpus.
Parent-child + metadata + per-type routing
Parent-child indexing handles mixed-complexity corpora by retrieving against small child chunks and returning the parent for context. A document-type classifier routes each document to the appropriate chunking pipeline, with metadata enrichment adding source, date, and section tags for filtered retrieval.
The chunking configuration checklist
Each item below should be confirmed before a chunking configuration is used to build a production index. Re-embedding a large corpus is the most expensive corrective action in a RAG pipeline; getting the chunking right before indexing is substantially cheaper than fixing it after.
Chunking is the decision you cannot easily revisit in production. Tune it on a sample before you index the corpus, not after retrieval quality disappoints you on 10 million embedded chunks.
What to do this week
01 Sample 100 documents from your corpus and classify them by structure type
Pull a representative sample of 100 documents from your RAG corpus and manually classify each as: continuous prose, paginated PDF, structured document with tables, or mixed format. Record the proportions. If your corpus is more than 20% structured PDFs or contains any documents with significant table content, your current single-strategy chunking configuration is likely degrading retrieval on those document types. The classification takes two to three hours and determines whether you need corpus-type routing before re-evaluating your chunk settings.
02 Build an evaluation set that includes table-based and multi-paragraph queries
Most RAG evaluation sets are built from the documents that were easiest to write queries for, which are typically the continuous prose documents. These systematically underrepresent the query types where chunking choice matters most: table lookups, numbered list retrieval, and multi-sentence answers that require context from consecutive paragraphs. Build 20 table-based queries and 20 multi-paragraph queries into your evaluation set this week and run them against your current chunking configuration. The results will tell you whether your chunking boundary decisions are costing you accuracy on the hardest query types in your corpus.
03 Test recursive splitting at 400-600 tokens with 50-100 token overlap as a baseline before any other optimization
If you are on a framework default (typically 1,000 characters with no overlap), switching to recursive splitting at 400-600 tokens with 10-15% overlap is the single highest-return chunking change for a prose corpus. Index a representative 10% sample of your corpus with this configuration and run your evaluation set. The FloTorch 2026 benchmark puts this configuration at 69% end-to-end accuracy on academic prose, which represents a strong baseline for comparison. If your current configuration is below this, the configuration change is worth the re-indexing cost on your full corpus before any embedding model or retrieval architecture changes are considered.
04 Add chunk boundary inspection to your pipeline debug tooling
When retrieval fails on a specific query, the usual diagnostic path is to check what was retrieved and compare it to what should have been retrieved. Add one more step: check whether the correct chunk exists as a coherent unit in your index, or whether the relevant content was split across two or three chunks by the boundary logic. This inspection step is cheap to add and surfaces the class of failures where a better chunk configuration would directly fix the retrieval problem without any changes to embeddings, reranking, or query rewriting. Most teams who add this step find that 15-25% of their retrieval failures trace back to boundary splits rather than embedding or ranking quality.
Let 10decoders optimize your RAG pipeline's document processing layer
We audit your RAG corpus's document type distribution, evaluate your current chunking configuration against your actual query set, and implement the corpus-appropriate chunking strategy and metadata enrichment layer that your retrieval quality requires. Our DocuFindr platform handles complex enterprise document types at production scale including multi-column PDFs, clinical documentation, and financial filings with structure-aware processing.