10decoders
Why testing an LLM is different from testing software
In traditional software testing, a function either produces the correct output or it doesn't. The test oracle is a precise specification. You write a test, you run it, it passes or fails. Regression is easy to detect because the output space is finite and deterministic. LLM outputs don't work that way. A language model can produce an output that is factually correct but formatted wrong, stylistically correct but factually wrong, or correct on today's test run and wrong on tomorrow's because of a temperature setting, a prompt change, or a model version update you didn't notice. The failure modes are probabilistic rather than deterministic, and that changes the structure of every evaluation decision you make.
LLM regressions are also a different class of problem than software regressions. In a software system, a broken function usually breaks something immediately visible: a page doesn't load, a calculation returns an error. LLM regressions are often invisible at first. A model that starts returning outputs that are 15% less accurate on a specific class of inputs won't throw an exception. It will just quietly produce lower-quality outputs until someone notices the support ticket volume is up, or a compliance review flags an incorrect claim in a customer-facing document. By the time the regression is discovered, it has been running for weeks. That gap is what a pre-deployment evaluation process is designed to close.
LLMs are also unusually sensitive to changes that look innocuous. Rewriting a system prompt, upgrading to a new model version, changing a retrieval chunk size, adjusting temperature by 0.1: any of these can shift output quality in ways that a purely manual review won't reliably catch. Enterprise software typically has stable behavior between releases unless you change something that directly affects the function. Enterprise LLMs can shift in quality between identical code releases because the model provider pushed a weight update, or because the distribution of queries shifted, or because a knowledge base document was updated and a retrieval chunk changed. Evaluation is continuous by necessity, not just a gate before each deployment.
"An LLM that passes a manual demo every time can still fail 18% of real production queries. The demo selects for the cases you know about. The eval suite covers the cases you don't."
The five evaluation types that form a complete pre-deployment process
A complete LLM evaluation process covers five distinct types of assessment, each designed to catch a different category of failure. Most enterprise teams run one or two of these. Teams that cover all five before each deployment rarely find production quality surprises that weren't already in their evaluation data. The table below maps each evaluation type to what it catches, when to run it, and the cost of skipping it.
| Evaluation Type | What it measures | How to run it | When to run | Cost of skipping |
|---|---|---|---|---|
| Task accuracy evaluation | Whether the model produces correct outputs on a representative sample of production queries. Correctness is measured against a ground-truth dataset: the right answer, the right format, or a human-rated quality score depending on the task type | Run the model against 100–500 test cases that represent the actual distribution of production inputs, including the common cases, the rare cases, and the cases that have failed in the past. Score each output. Set a pass threshold before you start | Before every deployment that changes the model, prompt, or retrieval configuration. Also after any model provider update, even if your code didn't change | Critical |
| Adversarial and edge case evaluation | Whether the model behaves acceptably on inputs designed to probe failure boundaries: ambiguous queries, inputs with missing context, off-topic requests, prompt injections, cross-lingual inputs, and inputs that are structurally similar to production queries but semantically unusual | Maintain a library of hard cases curated from production failures, internal red-teaming, and systematic generation of boundary inputs. Run the model against this library before each deployment. Add new cases when production finds failures that weren't in the library | Before every deployment. This library should grow over time. A team that has been running for six months with no new hard cases is almost certainly not adding cases systematically | Critical |
| Regression evaluation | Whether the new version of the system produces better, equivalent, or worse output than the previous version on a fixed benchmark. Catches quality degradation from prompt rewrites, model version changes, and retrieval configuration changes that looked safe but shifted output distributions | Run both the old and new system configurations against an identical fixed test set. Compare the scores. Any decline beyond a predefined threshold is a regression and blocks deployment. The threshold should be set in advance, not calibrated after you see the results | Before every deployment that involves a change to the model, prompt, or retrieval system. Also useful for tracking quality trends over time independent of specific deployments | Critical |
| Latency and cost evaluation | Whether the system meets its latency and cost requirements under expected load conditions. Changes that improve output quality often increase token count or retrieval complexity in ways that break latency budgets or make the system economically unviable at production scale | Run the system against a realistic load profile: p50, p95, and p99 latency measurements at the query volume the system will handle. Measure cost per query at that volume. Set thresholds for both before running the eval. A system that is qualitatively excellent but too slow or too expensive is not production-ready | Before any deployment that changes model size, adds retrieval steps, increases context length, or changes output format in ways that might affect token count | High |
| Human review of sampled outputs | Whether outputs that pass automated evaluation actually meet the quality bar that domain experts and end users expect. Automated evals measure what they were designed to measure; human review catches quality issues that don't map cleanly to a metric, including tone problems, subtle factual errors, and outputs that are technically correct but practically useless | Sample 20–50 outputs from the automated eval run and route them to a domain expert for review. Use a structured rubric. Aim for consistency across reviewers. Human review doesn't replace automated eval; it validates that the automated eval is measuring the right things | Before any new use case launch and quarterly for established systems. Also whenever automated eval scores improve significantly but production quality complaints don't decrease | High |
Not sure where your LLM evaluation gaps are?
10decoders runs two-week AI engineering assessments that audit your current evaluation coverage, identify the specific failure modes your process is not testing for, and build the evaluation infrastructure your team can run before every deployment.
Book a Free AI Assessment →What a mature evaluation process actually looks like in practice
Most enterprise teams start in the same place: someone runs a few test queries before each release, the results look reasonable, and the deployment goes ahead. This approach catches the obvious failures and misses everything else. It is worth being specific about what "everything else" includes: adversarial inputs that break the system, edge cases that the demo never covers, regressions introduced by prompt changes that seemed minor, and latency problems that only appear under realistic load. The manual demo doesn't find these because it isn't designed to find these.
The progression toward a mature evaluation process usually happens in response to production incidents. A team discovers that their model started producing incorrect outputs on a class of inputs they never tested. They add those inputs to a test suite. The next incident reveals another gap. Over time, the test suite grows into something that actually covers the failure space rather than the success space. The problem is that this reactive approach means every gap in the evaluation process gets discovered in production. The alternative is to build the evaluation suite proactively, using the five evaluation types above as a structural template, before the first production incident tells you what you missed.
A mature process also separates evaluation from deployment approval. The evaluation results are not reviewed by the same team that built the system and wants the deployment to succeed. There's a defined pass threshold for each metric, agreed upon before the evaluation runs. Results that don't meet the threshold block the deployment, not because someone decided the numbers weren't good enough, but because the threshold was set in advance and the system didn't meet it. This separation prevents the most common failure mode in LLM quality processes: evaluations that are run but not acted on because the team is under pressure to ship.
Ad-Hoc Spot-Checking
Manual demos before each release using representative queries chosen by the team. No fixed test set, no scoring rubric, no pass threshold. Catches obvious failures reliably. Catches edge cases, adversarial inputs, and regressions unreliably. Works well enough when the system is simple and the stakes are low. Breaks down when either of those conditions changes.
Structured Evaluation Suite
A fixed test set of 100-plus cases covering task accuracy, hard cases, and regression benchmarks. Scoring rubric defined in advance. Pass threshold agreed before the eval runs. Human review of a sampled subset. Still run manually before each deployment, but with consistent methodology and documented results. This is where the 4× regression detection advantage kicks in. Most teams can reach this stage in four to six weeks.
CI/CD-Integrated Evaluation
Evaluation runs automatically on every code and prompt change, not just before releases. Results block merges that would cause regressions. Test suite continuously updated as production finds new failure modes. Latency and cost benchmarks included. Human review scheduled independently of deployment cycles. At this stage, the evaluation process generates its own data on what the system is and isn't good at, which informs future development decisions.
Before you deploy: the LLM evaluation readiness checklist
"The teams that trust their LLMs in production are the teams that earned that trust through evaluation. Not by hoping the model would behave. By testing whether it did."
What to do this week
01 Pull 30 past production quality complaints and classify each root cause
Go through support tickets, incident reports, or user feedback from the last three months and find the cases where your LLM produced output that someone complained about. For each case, write one line on what went wrong: wrong facts, wrong format, wrong tone, wrong language, prompt injection, or something the model simply didn't handle. Tally the categories. This is your evaluation gap analysis. Each category of failure that isn't currently covered by your pre-deployment evaluation is a specific gap to close. Most teams that do this exercise find two or three failure categories that they've been discovering repeatedly in production without systematically testing for.
02 Build or review your hard case library this week
If you don't have a hard case library, start one now with the inputs from the exercise above, plus any inputs that your team knows are tricky but hasn't formalized. Aim for 30–50 cases as a starting library. If you have a hard case library, check when it was last updated. A library with no new cases in the last 60 days almost certainly has gaps. Add the cases from the production complaint review and run the current model against the full library. The results will tell you whether your model handles the inputs you know are hard, which is a baseline for every future evaluation.
03 Define and document your pass threshold before the next deployment
Before the next deployment goes out, write down what score on each evaluation type constitutes a pass. This doesn't need to be a complex exercise: pick a number for task accuracy, a number for regression allowance, a latency limit at p95. Get the team to agree on the numbers in writing. Then run the evaluation and apply the threshold as written. If the numbers seem arbitrary, use the data from the production complaint review to calibrate: if your complaint rate drops when task accuracy is above a certain score, that's your threshold. The exercise of writing it down before the eval runs is more important than getting the exact number right on the first try.
04 Run a regression eval comparing your current system to last month's version
Take the fixed test set you have (or build a minimum one from production queries), run it against the current system, and run it against whatever the system looked like 30 days ago. Compare the scores. If the current version is meaningfully worse on any evaluation category, you have a regression you didn't know about. Identify what changed in the last 30 days: prompt, model version, retrieval configuration, underlying data. At least one of those changes is responsible. This one exercise will tell you more about the actual quality trajectory of your system than six months of manual spot-checking, and it takes a day to set up.
Let 10decoders build your LLM evaluation framework
We run two-week AI engineering assessments that audit your current evaluation coverage, identify the specific failure modes your process is not testing for, build the test suite infrastructure your team can run before every deployment, and define the pass thresholds and regression benchmarks that turn evaluation results into deployment decisions.
