10decoders
The single accuracy number that tells you nothing useful
The standard enterprise RAG evaluation looks like this: a team builds a holdout set of questions with known correct answers, runs the pipeline against them before shipping, and reports that the system achieves 83% accuracy. If that number stays above some threshold, the system stays in production. If it drops, the team investigates. The problem is that “accuracy” here is an end-to-end measure. It collapses three separate failure modes into a single score, and when the score drops, the number itself carries no information about which failure mode caused it.
A wrong answer has several possible origins. Retrieval may have returned the wrong documents. Retrieval may have returned the right documents but assembled so many tokens that the relevant passage got buried in the middle of the context window, where the model attends less. The generation model may have hallucinated a confident answer that is not grounded in any retrieved document. Or the holdout eval set itself may have gone stale, drifting away from the queries users are actually asking. Each of these has a different fix: index improvements for retrieval failures, token budget adjustments for context failures, faithfulness scoring for hallucinations, and fresh production sampling for eval set drift. A single accuracy number cannot point to any of them.
The five evaluation decisions below are ordered by where they catch failures in the pipeline: retrieval quality first, then generation hallucinations, then the eval set's own accuracy, then continuous production monitoring, then CI gates. Missing any one of them means some category of failure will go undetected until it becomes visible through user behavior.
End-to-end accuracy tells you something broke. Retrieval metrics, faithfulness scores, and production sampling tell you what broke and where.
The 5 evaluation decisions and what goes wrong when each is skipped
These decisions apply to any RAG pipeline where the output informs a business decision, feeds an external user, or forms part of a compliance record. The table below shows what teams typically do at each decision point, what failure that creates, and how the failure shows up when an examiner or user finally surfaces it.
| Evaluation decision | What most teams do | The failure it creates | How it surfaces | Severity |
|---|---|---|---|---|
| Measure retrieval quality separately: context precision and context recall | Retrieval is not measured at all. The pipeline is evaluated end to end, and if the final answer is correct, retrieval is assumed to have done its job | Retrieval failures are invisible. A generation model that is good at guessing can mask poor retrieval by producing plausible answers from wrong documents. When the model changes or the corpus changes, retrieval failures that were previously hidden start producing wrong answers with no warning | A corpus update adds 400 new documents to the index. Query accuracy drops 11 percentage points over three weeks. The team investigates the generation model, finds no regression, and spends two weeks before determining that the new documents are degrading retrieval precision for a specific query class. Context precision would have shown the retrieval regression the day the new documents landed | Critical |
| Score faithfulness: does every claim in the answer come from retrieved context? | Hallucination is treated as a binary pass/fail on obvious errors. No systematic score is tracked per query | Partial hallucinations are undetected. A generation model can produce a mostly correct answer with one fabricated date, one invented reference, or one plausible-but-wrong numerical value, and accuracy scoring will mark the answer as correct if the main claim is right. For regulated use cases, that one fabricated value is the one that ends up in a report | A financial services RAG system answers questions about regulatory deadlines. Accuracy on the holdout set is 87%. A compliance audit pulls 20 live answers and finds that 4 contain deadline dates not present in any retrieved document. The system was hallucinating on date-specific queries. Faithfulness scoring would have flagged queries with low grounding rates before the audit | Critical |
| Build and maintain a golden eval set refreshed from real production queries | The eval set is built once at launch from hypothetical queries the team wrote. It is never updated | Eval set drift. The queries users actually ask in production diverge from the ones the team imagined at build time. Performance on the holdout set stays stable while real-world accuracy declines, because the holdout set no longer represents the actual query distribution. The metric looks fine; the product is not | A healthcare RAG system maintains 84% holdout accuracy for six months post-launch. User satisfaction surveys show declining confidence in answers. When the team samples 50 actual production queries and runs them through the eval, accuracy is 61%. The eval set had 110 queries, all written by the engineering team, covering topics users rarely asked and missing the clinical detail-level questions the clinical staff actually needed | High |
| Sample production traffic continuously for retrieval and faithfulness monitoring | Evaluation runs before launch and optionally before major releases. Production traffic is not sampled. No ongoing monitoring operates between releases | Silent degradation. Accuracy drifts as the corpus changes, user query patterns shift, or the generation model's behavior changes with provider updates. None of these changes trigger an eval run, so none appear in any report. The team has no signal until users escalate or an audit is conducted | A vendor updates the LLM API the pipeline uses with a new model version. The new version performs better on most queries but worse on tabular data questions, which make up 23% of this pipeline's real traffic. Holdout accuracy does not capture this because the holdout set has few tabular queries. Production sampling with weekly faithfulness scoring would have surfaced the regression within days of the model update | High |
| Gate deployments on RAG quality: fail retrieval and faithfulness checks in CI before any change ships | Deployment gates check whether the application starts and passes integration tests. No RAG quality metrics are part of the CI pipeline. Quality is checked manually before major releases, if at all | Quality regressions ship undetected. Any change that touches the retrieval layer, the chunking strategy, the prompt template, or the context assembly logic can degrade RAG quality without triggering any automatic alert. Teams find out from user reports, not from their own systems | An engineer updates the chunking configuration to handle a new document format. The change improves handling of the new format but reduces context recall on the existing document types by 14%. The change passes integration tests and ships. Degradation is discovered three days later when support escalations increase. A CI gate that ran context recall on a fixed eval set would have caught the regression before the pull request merged | Moderate |
The two Critical-level gaps share a root cause: retrieval and generation are treated as a single system rather than two layers with distinct failure modes. Teams that separate the measurement of these layers can localize regressions in hours. Teams that do not can spend weeks on blind debugging, and sometimes never correctly identify the root cause at all, shipping “fixes” that change the wrong layer and have no effect on the actual problem.
Not sure where your RAG pipeline's quality gaps are?
10decoders builds RAG evaluation frameworks for enterprise pipelines: retrieval quality baselines, faithfulness monitoring, golden eval set construction, and CI quality gates. We run the first evaluation against your live pipeline and produce a diagnostic report showing where retrieval is failing, where the generation model is fabricating, and what the fix priority should be.
Book a Free AI Assessment →Why retrieval failures are harder to find than generation failures
Generation hallucinations are relatively easy to detect once you know to look for them. Run a faithfulness score on a sample of answers, check whether each claim in the answer is supported by the retrieved context, and you have a signal. Retrieval failures are harder because a good generation model can partially compensate for them. If retrieval returns the wrong document but the model has seen enough similar content in pre-training, it may produce a plausible answer anyway. That answer will score well on accuracy if the question is not too specific. The pipeline appears to be working. The failure only becomes visible when the question requires a fact that is genuinely in the corpus but was not retrieved.
Context precision measures the fraction of retrieved chunks that are actually relevant to the query. A pipeline with low context precision is retrieving a lot of noise along with the signal. The generator has to work harder to find the relevant passage, and on queries where the relevant passage is not in the top-retrieved chunks at all, the generator either hallucinates or says it cannot find the information. Context recall is the complementary measure: it asks whether all of the relevant information needed to answer the query was actually retrieved. Low recall means the pipeline is missing documents it should have found, which is a different problem from low precision and has a different fix.
Running both metrics requires a labeled dataset that maps queries to the documents that should be retrieved. Building that dataset is work, and many teams skip it because the ROI is not obvious until the first unexplained regression. The practical starting point is not a comprehensive dataset. It is 50 to 100 queries from the most common query categories, labeled with which documents should appear in the retrieved set. That is enough to detect major retrieval regressions before they reach production.
Single score, no diagnosis
The pipeline is evaluated on a fixed holdout set at launch. End-to-end accuracy is the only metric tracked. Retrieval and generation failures are indistinguishable. When accuracy drops, debugging starts from scratch and can take weeks to localize.
Layer-separated evaluation
Context precision, context recall, and faithfulness run alongside end-to-end accuracy. When accuracy drops, the team can tell within hours whether retrieval or generation is responsible, because the metrics point to different layers. Production sampling runs weekly on a 5-10% traffic slice.
Quality as infrastructure
All five metrics run in CI and block merges that cause regressions. The golden eval set gets refreshed monthly from production queries that surfaced failures, and the team that owns the pipeline reviews retrieval and faithfulness dashboards weekly rather than waiting for a user complaint to prompt an investigation.
The RAG evaluation implementation checklist
This checklist covers the minimum viable evaluation stack for an enterprise RAG pipeline that serves real users or informs regulated decisions. The items are ordered by impact: the first three are the core diagnostic layer, the next two address ongoing monitoring, and the final two connect evaluation to your deployment workflow.
A faithfulness score below 0.80 on your eval set means roughly one in five claims in your answers is not grounded in what you retrieved. That is not a hallucination rate. That is a documentation problem waiting for a compliance audit.
What to do this week
Run your last 100 production queries through RAGAS or DeepEval and look at context precision first
You do not need a labeled dataset to get started on faithfulness scoring. Pull your last 100 production queries, run them through the pipeline, and score each answer for faithfulness using RAGAS or DeepEval with your preferred LLM-as-judge. The result tells you what fraction of claims in live answers are grounded in retrieved context. If faithfulness is below 0.75, the generation layer is fabricating at a rate that warrants immediate investigation before any other optimization work. For context precision, you will need to label which documents should appear in the retrieved set for a sample of queries, but starting with 20-30 queries from your most common query category is enough to get a retrieval quality signal within a day.
Identify what percentage of your recent accuracy regressions are retrieval failures versus generation failures
Go back through the last three accuracy regressions or user-escalated failures your team investigated. For each one, determine whether the retrieved documents contained the information needed to answer the question correctly. If the right documents were not retrieved, it was a retrieval failure. If the right documents were retrieved but the generated answer was wrong or hallucinated, it was a generation failure. For most teams doing this exercise for the first time, retrieval is responsible for a larger share than expected. That finding alone is worth redirecting optimization effort: if retrieval is failing more often than generation, putting engineering time into prompt tuning will not move the number that matters.
Build a 50-query golden eval set from production logs before the end of this sprint
Query your production logs for the 50 queries that received the lowest user satisfaction ratings, the most follow-up clarification requests, or the most explicit negative feedback. These are the queries your pipeline is currently handling worst. Label each with the document or documents from your corpus that contain the correct answer. That labeled set is your first golden eval set, and it is built entirely from real failures rather than hypothetical test cases. Running context precision and faithfulness against this set will show you where the pipeline is failing most badly, in the query domain where fixing it will have the most user-visible impact.
Add a faithfulness check to your next deployment pull request template
Before the next change to your retrieval or generation configuration ships, add a step to the pull request template that requires the author to run faithfulness scoring on a 20-query sample from the golden eval set and paste the result. This does not require CI automation yet. It requires a human to run a script and record the number. That lightweight step catches regressions that would otherwise ship unnoticed, and it builds the team's familiarity with what faithfulness scoring shows before you automate it. Automation is the goal, but the fastest path to catching the next regression is adding the manual check to the process this week.
Let 10decoders evaluate your RAG pipeline before users find the failures
We run retrieval quality diagnostics, faithfulness scoring, and golden eval set construction on your live RAG pipeline and deliver a report that shows where precision and recall are breaking, what share of live answers contain ungrounded claims, and which query categories carry the most risk. Our enterprise RAG practice has built evaluation stacks for healthcare, FinTech, and document-intensive enterprise workflows.