10decoders
The Same Test, Two Different Answers, No Code Change
Picture a QA engineer re-running the same agent test suite for the third time this morning. Nothing in the codebase changed since yesterday's green build. The model version is the same, the prompt template is the same, and yet two of forty test cases that passed last night are failing now, with outputs that are plausible, coherent, and simply different from what the test expected. By the fourth rerun, both pass again. Nobody files a bug, because there's nothing clear to file.
This is not a flaky test in the traditional sense, a network timeout or a race condition that a retry papers over. It's the agent behaving exactly as designed: sampling from a distribution of reasonable outputs rather than executing a fixed set of instructions. A conventional test suite built on an exact-match assertion was written for software where the same input always produces the same output. An LLM-based agent breaks that assumption at the foundation, and most quality engineering practices haven't rebuilt their tooling to match.
The result is a pipeline producing two kinds of false signal at once. Real regressions get lost in the noise of routine variance and get rerun until they happen to pass, while genuine drift, a model update, a prompt change, a vendor's silent patch, shows up looking exactly like the flakiness everyone has already learned to ignore. Teams that don't separate the two aren't really testing their agents. They're testing their own tolerance for uncertainty.
A test suite that can't tell drift from noise will teach your team to ignore both.
Where Deterministic Testing Habits Break on Agent Output
| Deterministic-era habit | What agent output actually needs | Severity |
|---|---|---|
| Exact-match string assertion on agent output | Semantic or scored evaluation against a defined tolerance band | Critical |
| One test run per CI build | Multiple runs per case, scored and averaged, not a single pass or fail | Critical |
| Model and prompt version unpinned in test logs | Every run logs the exact model, prompt hash, and config used | High |
| Automatic retry on any failure until green | Retries logged and capped, with a root cause required past N reruns | High |
| Hand-written test cases only | A regression corpus that grows from real production failures | Moderate |
| No drift monitoring after deployment | Continuous comparison of live agent output against the test baseline | Lower |
Not sure if your AI agent test suite can tell drift from noise?
10decoders audits your CI pipeline's agent test assertions, tolerance thresholds, and version logging, then helps you build a regression corpus that catches real drift before production does.
Book a Free AI Assessment →Statistical Evaluation Is Not Optional Once Output Varies
A tolerance band is not a looser standard than a binary assertion, it's a more precise one for a system that samples instead of executes. It defines what counts as an acceptable answer for a given case: a similarity score against a reference output, a rubric an evaluator model checks against, or, in some cases, a hard constraint, "the response must include the correct account balance", that stays strictly binary even inside an otherwise probabilistic system. Not every check should turn fuzzy. Some outputs have one correct answer, and those should stay exact.
Running a test case once and calling the result a verdict throws away the information that matters most: how consistent the agent actually is. Running the same case three to five times and scoring the distribution tells you whether a failure is a one-off sampling outlier or a real shift in behavior, and it's the only reliable way to set a threshold for what should block a deployment. Teams that skip this step end up choosing between two bad defaults: treating every variance as a bug, which trains everyone to ignore the pipeline, or treating every variance as fine, which means the pipeline stops catching anything at all.
Binary Pass or Fail
Every test case gets one run and one verdict, borrowed directly from deterministic testing. Agent output that's correct but phrased differently fails the build, so the team starts rerunning failures on reflex.
Manual Tolerance, No Structure
Engineers start eyeballing failures and rerunning by hand before deciding what's really broken, but with no documented threshold, two people reviewing the same failure reach different conclusions.
Scored Evaluation, Logged Drift
Each case runs multiple times against a defined tolerance band, every run logs its model and prompt version, and production output feeds back into the regression corpus automatically.
Non-Determinism Testing Checklist
Run this against your current agent test suite before the next CI pipeline review.
Test Suite Readiness Check
The next production incident your AI agent causes was probably visible in your test logs weeks ago. You just averaged it away.
What to Do This Week
01 Audit your last 30 days of CI reruns
Pull every agent test failure from the last month and check how many were dismissed as flaky without a root-cause look. That single number tells you how much real drift your team has already been averaging away without knowing it.
02 Pick five high-risk agent workflows and define tolerance bands
Start with the workflows that touch money, compliance, or customer-facing decisions. Write down, explicitly, what counts as an acceptable answer for each one, not just what counts as an exact match against a fixed string.
03 Start logging model and prompt version on every test run
If your CI logs don't already capture this, add it before doing anything else on this list. Without it, you can't trace a score change back to a cause, and every fix downstream becomes guesswork.
04 Route one real production failure into your regression corpus this week
Pick a single confirmed production issue and turn it into a permanent test case. Repeat weekly, and your test suite starts reflecting how the agent actually fails in the field, not just how engineers imagined it might.
Let 10decoders Build a Test Suite That Can Tell Drift From Noise
We audit your CI pipeline's agent test assertions, define tolerance bands for your highest-risk workflows, and set up the version logging and regression corpus your team needs to catch real drift before it reaches production.