10decoders
Why chunking is harder to get right than it looks
The chunking decision feels like a parameter choice: pick a size, split the documents, index the results. In practice it's an architecture decision that shapes every downstream component. The chunk size determines how much context each retrieved passage carries. The chunking method determines whether chunk boundaries respect meaning or cut across it. The overlap configuration determines how much content is duplicated between adjacent chunks to prevent boundary losses. And the choice of strategy determines whether the pipeline can be extended later with parent-child retrieval, contextual compression, or sentence-window expansion without re-indexing the full corpus.
The mismatch between corpus structure and chunking strategy is where most retrieval quality problems originate. A fixed-size strategy applied to a corpus of dense technical documentation, where a single paragraph may contain the only instance of a specific procedure, produces chunks that split the procedure across two or three passages. The retrieval layer returns partial context, the LLM synthesizes an incomplete answer, and the quality issue is attributed to the model rather than the chunking. A semantic strategy applied to a corpus of short FAQ entries, where each document is already a single coherent thought, produces unnecessary computation overhead for segmentation with no retrieval improvement over fixed-size. The right strategy depends on the corpus, and the corpus has to be analyzed before the strategy is chosen.
Chunk size also interacts with the embedding model in ways that teams often don't account for at selection time. Most embedding models have an input token limit, typically 512–8192 tokens depending on the model, and a performance window that's usually shorter than the maximum. A chunk that exceeds the model's effective encoding window gets truncated silently in most implementations. Chunks significantly shorter than the model's training distribution may not embed with enough semantic density to support reliable similarity search. The chunk size, the embedding model, and the retrieval k parameter all need to be tuned together, and changing any one of them after the corpus is indexed requires re-indexing.
"Chunking is the decision that's hardest to reverse after launch. Every other retrieval tuning choice can be adjusted without touching the index. A chunking change means re-processing the entire corpus."
Five chunking strategies and where each one fits your corpus
Five strategies cover the realistic decision space for enterprise RAG: fixed-size, sentence-window, semantic, hierarchical (parent-child), and structure-aware. Each one has a corpus type it's well-suited for, implementation complexity it requires, and retrieval failure modes that appear when it's applied to the wrong corpus. The table below maps each strategy to the corpus characteristics that make it a good or poor fit, along with the retrieval risk if the mismatch isn't caught before indexing.
| Strategy | How it works | Best corpus fit | Implementation complexity | Retrieval failure mode when mismatched | Retrieval risk if mismatched |
|---|---|---|---|---|---|
| Fixed-size | Splits documents into chunks of N tokens with an overlap of M tokens. No corpus analysis required. Chunk boundaries fall wherever the token count is reached, with no regard for sentence or paragraph structure | Short, uniform documents where content is densely packed and no single passage spans a complete procedure or argument. Support ticket corpora, product descriptions, changelog entries. Documents where every N tokens are roughly equally valuable context | Low | Chunk boundaries cut across sentences and paragraphs. Retrieved context is mid-thought. The LLM receives partial information and must either guess at completion or hedge. Answers contain the right topic but the wrong or incomplete details. This failure mode is invisible in low-volume testing, where queries happen to land on well-formed chunks, and surfaces at scale when the full query distribution hits boundary-split content | High for complex corpora |
| Sentence-window | Indexes individual sentences as the base unit but retrieves a window of surrounding sentences around each match. The embedding captures the sentence's specific meaning; the retrieval window provides surrounding context to the LLM. Window size is configurable (typically 3–7 sentences) | Corpora where the key retrievable fact lives in a single sentence but needs 2–4 surrounding sentences to be interpretable: legal texts, compliance documents, policy manuals, technical specifications where individual clauses reference context from adjacent sentences | Medium | Window size misconfiguration: a window that's too narrow returns a sentence with insufficient context; a window that's too wide returns so much surrounding text that the LLM's attention is diluted and the specific answer loses prominence. The optimal window size varies by corpus and is difficult to set correctly without testing against a representative query set | Moderate |
| Semantic | Uses embedding similarity to identify topic boundaries within documents, then splits at those boundaries rather than at fixed token counts. Adjacent sentences with high cosine similarity stay in the same chunk; a drop in similarity signals a topic change and triggers a split. Produces variable-length chunks that correspond to coherent topics | Long-form documents with multiple distinct topics per document: research reports, white papers, annual reports, technical documentation with multiple sections. Any corpus where fixed-size chunking would routinely split across topic boundaries. Works poorly on short, uniform documents where embedding-based boundary detection finds no meaningful variation | High | Similarity threshold misconfiguration: a threshold that's too sensitive creates micro-chunks that lose paragraph-level context; a threshold that's too lenient creates chunks that span multiple unrelated topics. Calibration requires sampling documents from the corpus and validating that chunk boundaries align with actual topic changes. Computational cost is also higher than fixed-size, which matters at re-indexing time | Moderate with calibration |
| Hierarchical (parent-child) | Indexes small child chunks (precise, narrow context) for retrieval but returns the parent chunk (broader surrounding context) to the LLM. Retrieval precision comes from the small chunk's focused embedding; answer completeness comes from the parent's wider scope. Requires maintaining parent-child relationships in the vector store's metadata | Corpora where precise retrieval and complete context are both required: product manuals, knowledge bases with procedure steps, legal contracts where the answer is a specific clause but the LLM needs the section containing it for complete interpretation. Works best in vector stores that support metadata filtering to traverse parent-child relationships at query time | High | Parent chunk size misconfiguration: if the parent is too large, the LLM's context window fills with surrounding content that dilutes the answer. If the parent is too small, it's effectively the same as returning the child. Dependency on vector store metadata means the strategy is coupled to the selected database's filtering capabilities and breaks if the store changes. Pipeline complexity is higher: indexing, retrieval, and the context assembly step all need to handle the two-level structure | Low with correct configuration |
| Structure-aware | Uses document structure signals (headers, section titles, list items, table rows, code blocks) to define chunk boundaries, rather than token count or embedding similarity. Requires document pre-processing to extract and preserve structure. Chunks correspond to semantically meaningful document units: a section, a procedure, a table | Structured corpora where document format encodes meaningful boundaries: Markdown documentation, HTML pages, PDF reports with consistent heading hierarchies, Word documents with defined section styles. Poor fit for unstructured text-heavy corpora where there are no consistent structural signals to use as boundaries | High | Structure inconsistency in the corpus: if documents don't use consistent heading levels or formatting conventions, the boundary detection produces chunks of wildly varying size and coherence. A corpus that looks structured in sample documents may have 20–30% of documents with non-standard formatting that breaks the parser. Requires a corpus audit before committing to this strategy | High for inconsistent corpora |
Not sure which chunking strategy fits your corpus?
10decoders runs two-week RAG architecture assessments that analyze your corpus structure, benchmark candidate chunking strategies against your actual query patterns, and implement the selected pipeline with retrieval quality evaluation from day one.
Book a Free AI Assessment →How corpus analysis drives the chunking decision
The chunking decision can't be made from a feature comparison table alone. It requires a corpus audit that answers four questions: How long are the documents, on average and at the extremes? Do documents have consistent internal structure (headers, sections, lists) or are they free-form text? Is knowledge distributed across many short passages or concentrated in long dense sections? And how varied is the document quality: are there formatting inconsistencies, scan artifacts, or encoding issues that would break structure-aware parsing?
A corpus audit for chunking purposes takes one to two days. Sample 200–300 documents across the full document length distribution, not just typical examples. Calculate the distribution of document lengths in tokens. Identify what percentage of documents have consistent structural markers. Note the average paragraph length and whether any content type (tables, code, lists) requires special handling. This analysis often reveals that the corpus has two or three distinct document types that would benefit from different chunking strategies, which is common in enterprise knowledge bases that aggregate content from multiple sources.
When the corpus contains multiple document types, a hybrid approach is often the right answer. Short FAQ-style documents get fixed-size chunking because their content is already compact and uniform. Long technical manuals get hierarchical chunking to preserve both retrieval precision and answer completeness. Legal or policy documents get sentence-window chunking where clause-level precision matters. The chunking layer becomes a routing step that applies the appropriate strategy based on document metadata, which adds pipeline complexity but produces better retrieval quality than forcing a single strategy across a structurally diverse corpus.
Default Fixed-Size Chunking
Fixed-size chunking applied to the full corpus with default parameters (typically 512 tokens, 50-token overlap). No corpus analysis. The strategy ships in a day. Retrieval quality issues surface gradually as the query distribution reveals boundary-split content. The chunk size gets adjusted post-launch (61% of teams make this change within 90 days). The adjustment requires re-indexing the full corpus, which in large enterprise corpora takes several hours to days and interrupts any production query traffic against the index during the rebuild. Teams at this stage often attribute quality problems to the LLM or prompt rather than the chunking, which delays the diagnosis.
Corpus-Analyzed Strategy Selection
Corpus audit completed before any indexing begins: document length distribution calculated, structural consistency assessed, content type inventory produced. Chunking strategy selected based on corpus characteristics, not default behavior. For uniform corpora, fixed-size with tuned parameters. For structured corpora, structure-aware or hierarchical. For mixed corpora, a routing approach that applies the appropriate strategy by document type. A 10–20% corpus sample is indexed with the candidate strategy, retrieval quality is scored against a representative query set, and the configuration is validated before full indexing. The evaluation query set is retained as the retrieval quality baseline for ongoing monitoring.
Continuously Monitored Chunking Architecture
Chunking strategy documented with the corpus analysis that justified it. Chunk boundary quality monitored via periodic spot-checks on new documents added to the corpus. Retrieval quality scores tracked against the evaluation baseline, with alerting when context relevance drops more than 10% from the established baseline. Corpus growth monitored for new document types that may fall outside the original chunking strategy's design assumptions. Re-evaluation triggered when the corpus composition changes significantly (more than 20% new content from a new source) or when retrieval quality metrics show sustained degradation. Chunking configuration treated as a versioned artifact that changes alongside corpus changes.
Chunking strategy readiness checklist
"The teams who re-index once at launch, and then leave it alone, didn't get lucky. They analyzed the corpus before writing the chunking code. That's the whole difference."
What to do this week
01 Analyze your corpus before touching any chunking configuration
If you're starting a new RAG project, don't set up the chunking pipeline until you've spent a day analyzing the corpus. Pull 200 documents across the length distribution. Calculate the token count percentiles. Note what percentage have consistent structural markers. Identify every special content type (tables, code, lists) that will need explicit handling. This analysis produces a one-page summary that makes the chunking strategy decision almost obvious: uniform short documents point to fixed-size, structured long documents point to hierarchical or structure-aware, mixed corpora point to a routing approach. The decision still needs to be validated against a query set, but the analysis gets you to the right two or three candidates before any indexing work starts.
02 Audit your current chunking configuration if you're already in production
If you have a RAG system running, pull 50 recent retrieved contexts and read them. Look for patterns: chunks that start or end mid-sentence, chunks that contain the topic label (a heading) but not the content it heads, table fragments without their column headers, code blocks split across two chunks. Each pattern points to a specific chunking configuration problem. If you find more than 10% of retrieved contexts showing at least one of these patterns, the chunking configuration is contributing to retrieval quality issues. Quantify the frequency before making changes: run your representative query set, score context relevance, and record the baseline. Changes to chunking require re-indexing, so you want the data to justify it before you commit to the work.
03 Test two chunk sizes against your representative query set before setting one
Pick the two chunk sizes that bracket what feels right for your corpus: one smaller (256–384 tokens) and one larger (768–1024 tokens). Index a 10% corpus sample at each size. Run your representative query set and score context relevance. The size that scores better on your queries is the right starting point. More often than not, one size performs noticeably better on the query types that matter most for the use case. This test costs one to two hours of indexing time and produces a data-backed parameter choice rather than a guess that gets revisited post-launch.
04 Identify your special content types and add explicit handling before indexing
Walk through a sample of 50–100 documents and list every content type that appears: prose paragraphs, numbered lists, bullet lists, tables, code blocks, mathematical formulas, captions, footnotes. For each type, decide: can it be split at a token boundary, or does splitting it destroy the information it contains? Tables and code blocks almost always need atomic treatment. Numbered procedure steps often need to stay together. Identify the content types that need special handling and add the rules to the chunking pipeline before indexing begins. This is a two to four hour task that prevents an entire category of retrieval errors from appearing in production.
Let 10decoders design your RAG chunking architecture
We analyze your corpus structure, benchmark candidate chunking strategies against your query patterns, and implement the configuration with retrieval quality evaluation built in from day one. No post-launch re-indexing surprises.