10decoders
Why context assembly is a distinct engineering problem from retrieval
Retrieval and context assembly are separate pipeline stages, and treating them as a single step is the most common source of token inefficiency in production RAG. Retrieval identifies which chunks are relevant to a query. What context assembly decides is different: which of those chunks the LLM actually receives, in what order, and how much of each chunk survives into the final prompt. A retriever that returns the right 20 chunks has done its job correctly. If the assembly step passes all 20 chunks to the LLM in a single context block, the LLM now faces a 20,000-token input where the answer it needs is at chunk position 11, in the middle of the window, receiving the least attention of any position in the context.
The token cost of this arrangement compounds at scale. A RAG pipeline that retrieves top-20 at 1,000 tokens per chunk and processes 10,000 queries per day consumes 200 million input tokens daily from retrieval context alone, before system prompts or conversation history. At 2026 frontier model pricing, that is a material line item. A conservative 50% context compression rate on the retrieved chunks cuts that to 100 million tokens, with benchmark evidence suggesting the accuracy loss is minimal when compression is query-aware. The question is not whether to manage context assembly; it is which approach fits the pipeline's latency budget and document types.
Latency is the other dimension that most teams underweight. Time to first token scales with total input length because the transformer must process the full context before generation begins. A pipeline with 15,000 tokens of retrieved context before the generation step will have meaningfully higher latency than one with 4,000 tokens of carefully assembled context delivering the same answer quality. For conversational applications where users wait for responses, the difference is perceptible. For batch pipelines, it compounds through the queue.
"Retrieval quality sets the ceiling on what the LLM can find. Context assembly determines whether the LLM can actually see it inside the window you gave it."
The 5 context assembly approaches: trade-offs and production failure patterns
Each approach below optimizes for a different balance of latency, cost, and accuracy. The right choice depends on query complexity, document types, latency requirements, and whether prompt caching is a priority in the pipeline's cost architecture.
| Context assembly approach | Token efficiency | Where it fails | Production failure pattern | Risk level |
|---|---|---|---|---|
| No token budget (pass all retrieved chunks) | None. All top-k chunks reach the LLM regardless of relevance score or query complexity | Lost-in-the-middle degrades accuracy on multi-chunk contexts. Token cost scales linearly with retrieval depth. Latency rises with context length. Prompt caching cannot stabilize a context that varies per query | Pipeline ships with top-20 retrieval and no context cap. Per-request input token cost stays invisible until billing arrives. Answer quality on complex queries is inconsistent because the relevant chunk is buried in position 12 of a 20-chunk context | Critical |
| Fixed top-k (always top-3 or top-5) | Moderate. Token count is capped but the cap is query-independent | Simple factoid queries overpay: top-1 would answer correctly. Complex multi-hop queries are underfed: top-5 may not contain all the evidence needed. Fixed k is a retrieval parameter, not a context budget | Team sets top-5 as a reasonable default and ships it. Answer quality on multi-document queries degrades because the fifth chunk rarely contains evidence for the query but the first needed chunk was cut off. The fix always feels like "just increase k" rather than fixing assembly logic | High |
| Similarity score threshold (dynamic k based on relevance) | Better. Passes only chunks above a minimum similarity score, letting k vary per query | Similarity scores are not calibrated to answer quality. A 0.75 cosine similarity does not mean 75% chance the chunk is useful. High-similarity chunks may be topically related but not answer the query; low-similarity chunks sometimes contain the exact fact needed | Threshold set at 0.8 cuts context on most queries. A clinical query returns 2 chunks above threshold and misses the dosage guideline in a 0.76-similarity chunk. Team debugs answer quality by lowering the threshold, which re-introduces most of the token overhead the threshold was meant to remove | High |
| Query-aware context compression (LLMLingua-2, RECOMP, selective passage extraction) | High. Strips query-irrelevant tokens from each retrieved chunk before assembly, achieving 50-70% token reduction with minimal accuracy loss | Compression latency adds 50-200ms per request depending on chunk count and compression model size. Query-specific compression breaks prompt caching: each request produces a different compressed context, so there is no stable cacheable prefix. Requires a compression model running in-pipeline | Team deploys LLMLingua-2 and reduces token costs by 55%. Latency budget is exceeded by 180ms on complex queries because the compression step runs synchronously over 15 chunks. P95 latency spikes on queries that retrieve more chunks than average | Moderate |
| Tiered token budget with graceful fallback | Highest. Set a token budget for context assembly, fill it with the highest-relevance chunks, stop when the budget is reached, and fall back to a reduced-context or no-context response with a clear label when the budget is insufficient | Fallback responses require clear labeling so users know the answer was generated without full context. Building and tuning the fallback path adds engineering complexity. Token budget must be calibrated per query type rather than set globally | Closest to production-optimal. Primary failure is a miscalibrated global budget: set too low for a document type that consistently requires more context, it triggers fallbacks on queries users expect to be fully answered. Per-query-type budget profiles solve this but require ongoing maintenance as the corpus evolves | Lower |
The compression-versus-caching tension deserves explicit attention. Prompt caching works by keeping a stable prefix in the KV cache across requests, dramatically reducing the cost of repeated context. Query-aware compression produces a different compressed output for each query, which means each request has a unique context block that cannot hit a warm cache. For pipelines where the same document set is queried repeatedly, prompt caching on a fixed, uncompressed context block may cost less than query-specific compression plus full input token pricing. Measure both options against your actual query distribution before committing to compression as the primary cost lever.
Not sure where your RAG pipeline's token budget is leaking?
10decoders audits your RAG pipeline's context assembly layer, measures per-request token spend against answer quality on your actual query distribution, and implements the token budget and context compression configuration that reduces cost and latency without degrading retrieval accuracy. Our DocuFindr platform handles complex document types including multi-column PDFs, clinical notes, and financial filings with query-aware context assembly built in.
Book a Free AI Assessment →Why the lost-in-the-middle problem makes token budgets mandatory for multi-document queries
Transformer attention mechanisms do not distribute attention uniformly across the input context. Research on this effect, documented across multiple model families, consistently shows that models attend more strongly to content at the beginning and end of the context window than to content in the middle. For a 15-chunk retrieved context, this means that the chunk at position 7 receives less attention than the chunks at positions 1 and 15. If the answer to a query lives in chunk 7, the LLM will produce a weaker or incomplete answer compared to what it would produce if that same chunk were at position 1 or 15.
The practical implication is that beyond a certain context depth, adding more retrieved chunks actively degrades answer quality while increasing cost. Teams running top-20 retrieval with no token budget are not getting better answers than they would with top-7 retrieval and a tighter assembly; they are paying for 20 chunks and getting the answer quality of a pipeline that placed the relevant chunk in a middle position. The optimal context depth varies by query type and model, and it is worth measuring explicitly: construct a small eval set of queries where the correct answer is in a known chunk, vary the retrieval depth and chunk position, and measure where answer quality peaks before the lost-in-the-middle effect takes over.
Ordering retrieved chunks within the assembled context also matters. Placing the highest-scoring chunks at the beginning and end of the context block rather than in sequential retrieval rank order measurably improves answer quality on some models. This is a zero-cost change that requires only sorting the assembled context by relevance score with alternating placement from the two ends of the window, rather than packing chunks in rank order from the top. Test it on your eval set before assuming it helps; the effect size varies by model and document type.
Top-k pass-through
All retrieved chunks pass to the LLM in retrieval rank order with no token cap. Token cost and latency both scale with retrieval depth, and lost-in-the-middle degrades answer quality on any query where the relevant chunk lands somewhere in the middle of the context window.
Token budget with relevance ordering
A per-request token cap limits how many retrieved chunks reach the LLM, with chunks ordered by relevance score so the highest-scoring ones land at the start and end of the context window. When the budget runs out before the relevant chunk is included, a fallback label tells the user the response may be incomplete.
Query-aware compression plus budget
Query-aware compression strips irrelevant tokens from each chunk before the context is assembled, with a token budget capping the total context size. Per-query-type profiles let you tune the budget differently for financial filings versus clinical notes versus policy documents. For latency-sensitive query types, compression can run asynchronously to avoid adding to time-to-first-token.
The context assembly configuration checklist
These checks should be applied before a RAG pipeline's context assembly configuration is locked for production, and re-evaluated when the query distribution changes materially or the corpus grows by more than 20 percent.
"Top-k retrieval is a retrieval decision. How many of those k chunks the LLM actually receives is a separate engineering decision, and most teams skip it entirely."
What to do this week
01 Measure your current per-request retrieved token count from production logs
Pull one day's production logs and compute the median and 95th percentile input token count per request, broken down into system prompt, retrieved context, conversation history, and user query. In most enterprise RAG pipelines, retrieved context accounts for 60 to 80 percent of total input tokens. If your logs do not currently record this breakdown, add per-component token logging this week before any other optimization work. Without this measurement, any token reduction effort is unvalidated and you cannot confirm whether a configuration change improved or degraded the pipeline's cost and quality profile.
02 Run your eval set at three retrieval depths and find where accuracy peaks
Take your existing retrieval eval set, or construct a minimal one of 30 to 50 labeled queries, and run it at top-3, top-7, and top-15 retrieval depths. Record answer accuracy at each depth. For most enterprise corpora, the accuracy curve peaks well below top-15 and the marginal gain from each additional chunk drops sharply after a point. Identifying that peak gives you the data to set a retrieval depth that maximizes answer quality per input token, which is a better optimization than any downstream compression applied to an unnecessarily large retrieved set.
03 Test relevance-weighted chunk ordering against rank-order on your multi-chunk queries
Select 20 to 30 queries from your eval set that consistently retrieve more than 5 chunks. For each, compare answer quality when chunks are assembled in retrieval rank order versus when the highest-scoring chunks are placed at the first and last positions in the context window. This test takes less than a day to run and requires no code changes to the retrieval or generation layers, only to the assembly sorting step. If relevance-weighted ordering improves accuracy on these queries, it is the highest-return change available in the context assembly layer, with zero additional latency or token cost.
04 Estimate your prompt caching hit rate before deciding between compression and caching
Review your production request logs for the past week and identify what fraction of requests query the same document set, the same document version, or the same set of retrieved chunks as a previous request within the cache TTL. If this fraction exceeds 30 percent, your pipeline has a substantial caching opportunity that may outperform compression on cost terms. If the fraction is below 10 percent, your query distribution is too diverse for caching to be a primary cost lever, and query-aware compression is worth evaluating. This measurement takes an afternoon and determines which cost optimization path to pursue without committing engineering time to the wrong one.
Let 10decoders tune your RAG pipeline's context assembly layer
We measure your current per-request token spend from retrieved context, run your query distribution against multiple assembly configurations, and implement the token budget and ordering strategy that reduces input token cost and latency without degrading answer quality. Our DocuFindr platform includes query-aware context assembly built for complex enterprise document types including multi-column PDFs, clinical documentation, and financial filings.