10decoders
Why retrieval failure is harder to catch than language model failure
When a language model produces a hallucination, it's often detectable: the output contains a fact that contradicts a known source or seems implausible in context. When a retrieval layer fails, the symptom is subtler. The model gets chunks that are topically adjacent to the question but don't contain the specific information needed to answer it. The model then does what it's trained to do: produce a fluent, confident-sounding answer from whatever it received. The output looks fine. It passes a quick read. It fails a domain expert who knows what the correct answer should have been.
This is why manual testing before launch catches very little retrieval failure. A developer testing with a handful of curated queries typically picks questions the system was designed for, checks that the output sounds reasonable, and moves on. The queries that expose retrieval gaps are the ones no one thought to test: edge cases, partial matches, queries that use different terminology from the indexed documents, questions that require synthesizing information across multiple chunks that the retrieval layer returned in the wrong order or didn't return at all.
The teams that catch retrieval failures before users do approach evaluation differently. A test set of 100–200 query-answer pairs gets built from real expected queries, not developer intuitions about what the system will be asked. That test set runs against the retrieval layer independently, before any language model is involved, measuring whether the retrieved chunks contain the information needed to answer each query. Retrieval quality is tracked as a separate metric from answer quality. When something degrades, the team knows whether the problem is in retrieval or generation, and fixes the right layer.
"Most RAG quality problems aren't language model problems. They're retrieval problems the model can't recover from, no matter how good it is."
Five retrieval evaluation metrics and what each one actually measures
Retrieval evaluation requires measuring multiple things independently: whether the retrieved chunks are relevant to the query, whether the generated answer is faithful to those chunks, whether the answer addresses what was actually asked, and whether the system finds answers when they exist. The table below maps each metric to what it catches and what happens when it's skipped.
| Metric | What it measures | How to measure it | What skipping it misses | Skipping risk |
|---|---|---|---|---|
| Context relevance | Whether the chunks returned by the retrieval layer contain information relevant to the query. A high context relevance score means the retrieval layer is returning chunks that bear on what was asked. A low score means the model is being handed content it can't use to answer the question | For each query in the test set, score each retrieved chunk on relevance to the query on a 0–3 scale. Average across chunks for a mean relevance score. Automate this with a smaller LLM used as a judge, or use human annotators for a gold-standard calibration run. Track score distribution across query types, not just the mean | Retrieval layer returning topically adjacent but non-answering chunks. The model produces confident output from irrelevant context. No signal that the retrieval configuration needs tuning until user complaints arrive three weeks into production | Critical |
| Answer faithfulness | Whether the generated answer is supported by the retrieved context, with no claims that go beyond what the chunks contain. Faithfulness measures whether the model hallucinated. Low faithfulness means the model added information not present in the retrieved chunks, which is the primary hallucination risk in RAG systems | For each generated answer, check each factual claim in the answer against the retrieved context. A claim is faithful if it can be grounded in a specific chunk. The faithfulness score is the fraction of claims that are grounded. Use an LLM judge or a human annotator on a sample. Flag any answer with faithfulness below 0.8 for manual review | Hallucinated answers that look correct on a quick read. The model confident-sounds its way through questions the retrieved context didn't fully support. Low-faithfulness answers are the hardest for users to catch and the most damaging to trust when they do catch them | Critical |
| Answer relevance | Whether the generated answer addresses the actual question that was asked. A system can have high context relevance and high faithfulness and still produce answers that don't quite answer the question, because the model interpreted the query differently than the user intended or the retrieved chunks led it in a tangential direction | For each query-answer pair, score whether the answer directly addresses what the query asked, on a 0–2 scale. Automate with an LLM judge prompted with the original query. Flag answers that score below 1.5. Review flagged answers for patterns: are certain query types consistently misinterpreted? Does the system handle implicit questions worse than explicit ones? | Answers that are technically accurate but miss the user's intent. A query asking "how do I cancel my subscription" that returns a faithful paragraph about subscription pricing is a relevance failure even if nothing it says is wrong. These failures are the leading cause of low user satisfaction scores in enterprise RAG deployments | High |
| Retrieval coverage | Whether the system can retrieve relevant information for the full range of queries it will receive in production, including edge cases and queries that use terminology different from the indexed documents. High context relevance on the easy queries doesn't mean the system handles the full query distribution. Coverage measures what percentage of production-realistic queries the retrieval layer can answer | Build a test set that includes queries from the tails of the expected distribution, not just the obvious use cases. For each query, determine whether the indexed corpus contains an answer (some queries may be genuinely unanswerable from the available documents). Measure what fraction of answerable queries the system retrieves useful context for. Low coverage on edge cases is the most common RAG failure pattern | Systematic gaps in what the system can answer that only become visible at scale. A system with 95% coverage on developer-curated test queries can have 60% coverage on the actual production query distribution if the test set was biased toward the happy path. Coverage gaps compound over time as users learn to avoid queries the system can't handle | High |
| Semantic similarity at rank | Whether the most relevant chunks are ranked at the top of the retrieval results, not buried at position 6 or 8 where they may be truncated by the context window limit. A retrieval layer can return relevant content but still fail if that content ranks below less relevant chunks that consume the available context space | For each query with known relevant chunks, measure where the most relevant chunk appears in the ranked retrieval results. Track mean reciprocal rank (MRR) and recall at k for different values of k. A low MRR means the system often has the answer but puts it in the wrong position. Tuning chunk size, embedding model, and reranking logic addresses this | Relevant information being consistently outranked by less relevant content. The model receives the top-k chunks, which may not include the most important one. The answer reflects what was ranked highest, not what was most relevant. This failure is invisible without per-query ranking analysis | Moderate |
Not sure where your RAG system's retrieval gaps are?
10decoders runs two-week AI engineering assessments that build retrieval evaluation suites specific to your query distribution, measure context relevance and faithfulness baselines, and identify the retrieval configuration changes that most improve production quality.
Book a Free AI Assessment →How retrieval evaluation fits into the RAG development lifecycle
Retrieval evaluation works best when it's built before the system goes live, not added as a diagnostic after quality problems appear. The sequence that works: build the retrieval pipeline, build the evaluation suite against a sample of expected production queries, run the suite to establish a baseline, then tune the retrieval configuration to improve the baseline before launch. By launch, you know exactly how the system performs across your query distribution. When quality changes post-launch, you re-run the suite and find out whether the change is in retrieval or generation.
The evaluation suite itself requires effort to build correctly. A test set of 50 developer-curated queries is not enough. The queries need to cover the realistic distribution of what users will ask: common queries, edge cases, queries that use different vocabulary from the source documents, queries that require synthesizing information across multiple chunks, and queries that are genuinely unanswerable from the available corpus. Building this test set takes one to two days. It's the highest-return investment in RAG quality available before launch, and most teams skip it because it feels like work that could happen later. It can't happen later without production failures teaching you what the test set should have covered.
Continuous retrieval monitoring after launch follows the same principle. The evaluation suite that was built before launch becomes the baseline. Every corpus update, embedding model change, or retrieval configuration change runs against that baseline before being promoted to production. A regression in context relevance or faithfulness is caught in staging. The alternative is discovering it from user complaints.
No Formal Evaluation
Manual spot-checking with developer-curated queries before launch. No retrieval quality metrics. No distinction between retrieval failures and generation failures. Quality problems discovered from user complaints and support tickets. Debugging requires re-testing queries manually to reproduce failures, with no baseline to compare against. Corpus updates and configuration changes go to production without quality checks. Retrieval gaps compound as the query distribution evolves.
Offline Evaluation Suite
A test set of 100–200 query-answer pairs drawn from realistic expected queries, including edge cases and terminology variation. Context relevance, faithfulness, and answer relevance measured before launch. Retrieval coverage measured across the full expected query distribution. Baseline scores established per metric. Every corpus update and configuration change runs against the baseline before promotion. Retrieval failures caught before users report them. Debug sessions start with a metric score, not a user complaint.
Continuous Retrieval Monitoring
Automated evaluation suite running on a sample of production queries. Context relevance and faithfulness scores tracked daily, with alerting when either metric drops below threshold. Retrieval coverage monitored as the query distribution evolves. Mean reciprocal rank tracked per corpus update. Evaluation results reported weekly alongside usage metrics. When quality changes, the metric that moved narrows the investigation to retrieval or generation within minutes. New query types discovered in production are added to the test set quarterly.
RAG retrieval evaluation readiness checklist
"A RAG system that hasn't been evaluated is a system whose quality is unknown. Unknown quality in a customer-facing system isn't acceptable risk. It's a matter of when users find the gaps, not whether they will."
What to do this week
01 Audit your current RAG test coverage this week
Pull every query that was used to test the system before launch or since. Categorize each one: was it a common-case query, an edge case, or a terminology-variation query? If more than 80% of the test queries fall in the common-case category, the test set is biased toward the happy path and the coverage picture is missing. Map the gaps against what domain experts consider the hardest question types for the system. Those gaps are where production failures are hiding. The audit takes a day, produces a clear list of test set deficiencies, and tells you exactly which queries to add before the next evaluation run.
02 Run context relevance scoring on your last 100 production queries
Pull the last 100 production queries from your system logs. For each query, retrieve the chunks the system returned and score each chunk for relevance to the query on a 0–3 scale using an LLM judge. Average the scores. If the mean context relevance is below 2.0, a significant fraction of your production queries are being answered from low-relevance context. The distribution of scores across query types will tell you which categories are underperforming. That's the retrieval tuning target. This analysis takes a day to run for the first time and directly identifies the retrieval configuration changes with the highest quality impact.
03 Measure faithfulness on a sample of recent answers
Take 30 recent query-answer pairs from production. For each answer, identify every factual claim and check whether each claim can be grounded in the retrieved context that was supplied to the model. Calculate the fraction of grounded claims. If faithfulness is below 0.85 on this sample, the system is hallucinating at a rate that should be addressed before the user base grows. Flag the specific answers with the lowest faithfulness scores for qualitative review: what type of query triggers the most hallucination? Is it multi-step queries, comparative queries, or queries where the retrieved context is partially relevant but incomplete? That pattern drives the next intervention.
04 Set one retrieval quality alert before end of week
Pick the metric that matters most for your use case, context relevance or faithfulness, and configure an alert that fires when it drops more than 10% below the current baseline. If you don't have a current baseline, run the scoring exercise from steps 02 and 03 this week to establish one, then set the alert threshold. An alert that fires before users notice a quality change is worth more than any amount of post-incident debugging. Most teams that have experienced a significant RAG quality incident wish they had set retrieval metric alerting earlier. It's a two-hour setup; the investment pays back in the first incident it catches.
Let 10decoders assess your RAG system's retrieval quality
We run two-week AI engineering assessments that build evaluation suites specific to your query distribution, measure context relevance and faithfulness baselines, and identify the retrieval configuration changes that most improve production quality.