10decoders
A Masked Database Meets an Agent Test Run
Consider a quality team at a mid-sized insurer that has done what the policy says. The claims database copy in the test environment has names, phone numbers, and policy numbers masked, and the team passes its annual audit. Then the same team starts testing a claims-triage agent. The agent does not read columns the way a report does. It reads the adjuster notes field, which still contains a claimant's full name, a street address, and a description of an injury, because masking rules were written for structured fields.
The leak does not stop at the notes. The test harness sends each record to a hosted model, the observability platform stores the full prompt and response for ninety days, the evaluation report quotes the failing cases verbatim, and someone exports a spreadsheet of those cases to a laptop to debug them. One masked database has become five copies of sensitive text in five places, and only the first one was ever in the data inventory. Perforce's 2026 survey points at the same pattern from the other side: 57% of respondents said the volume of sensitive data in non-production grew in the past year, with faster release cycles (31%) and more use of data for decisions (30%) as the top drivers.
None of this means masking is the wrong tool. It means masking was built for a world in which test data was read by people and by deterministic code. Agents add readers that interpret free text, call tools, and write detailed records of what they saw. A quality engineering team that wants to test those agents needs a test data design that assumes the data will be read, summarized, stored, and quoted, and that decides in advance where each of those happens.
"A masking policy describes the data you meant to protect. An agent test run shows you the data you actually have."
Six Ways Test Data Leaks Once Agents Join the Test Cycle
Each row below is a pattern we look for when a quality team starts testing agents against realistic data. Severity reflects how much harm follows if the gap is found by someone other than the team.
| Test data gap | What goes wrong in practice | Severity |
|---|---|---|
| Free text left unmasked | Column rules cover names and ID fields, but notes, emails, and documents keep identifying details that the agent reads and repeats | Critical |
| Test records sent to unreviewed model endpoints | A tester points the harness at a personal API key or a public tool, and real customer text leaves the approved boundary | Critical |
| Traces and eval reports stored unmasked | Observability and evaluation tools retain full prompts, tool payloads, and failing cases with broad read access and no retention limit | High |
| Exceptions with no owner or expiry | A temporary copy of production for a bug investigation is still sitting in a shared environment a year later | High |
| Synthetic data too clean to catch real failures | Generated records are tidy and well formed, so the agent passes tests that production inputs would fail | Moderate |
| Masked values that can be linked back | Dates of birth, postcodes, and rare diagnoses or amounts remain in combination, so records can be matched to a person | Moderate |
Not sure where your test data gaps are?
10decoders reviews how test data moves through your environments, harnesses, and evaluation tools when agents are involved. You get a ranked list of exposure points and a test data model your quality, security, and data teams can sign off together.
Book a Free AI Assessment →Synthetic Data Solves Privacy and Creates a Fidelity Problem
Many teams respond by moving to synthetic data, and the survey data says the shift is under way: 51% of Perforce's 2026 respondents use it, alongside 86% using static masking and 60% using dynamic masking. Synthetic data removes the link to real people, which is its main value. It also removes the mess that makes production hard. Real claim notes contain typos, abbreviations in three languages, copied email chains, and a field that was repurposed in 2019. Generated records rarely include any of that unless someone designs it in.
Agents are sensitive to that gap in a way that most older systems are not. A classifier that handles a tidy sentence can fail on a pasted block with a signature and a disclaimer. A tool-calling agent that reads a well formed date will break on a date written as “3rd of Oct, ish.” If your test data never contains those inputs, your pass rate describes the test data and says little about production. A Tonic.ai survey of 1,000 developers and startup employees in 2022 found that 42% said their test data practices were not secure or did not scale, which fits what we see: teams get one of the two properties and not both. The sample skews toward startups, so read it as direction rather than a benchmark.
The workable answer is a tiered model. Use synthetic data for the bulk of runs, build a small library of messy cases from real incidents and rewrite them so no person is identifiable, keep masked subsets only where relational shape matters, and make every exception expire. The aim is test data that is safe to hand to an agent and rough enough to embarrass it.
Production Copies with Column Rules
Test environments are refreshed from production and structured fields are masked. Free text, attachments, traces, and evaluation outputs are outside the policy, and exceptions are granted by message.
Free Text and Tooling Brought Into Scope
The inventory covers documents, notes, harness endpoints, and trace stores. Approved model endpoints are enforced, retention is set, and exceptions have an owner and an end date.
Synthetic Core with a Real-World Edge Library
Most runs use generated data checked against production distributions. An incident-based edge-case library keeps tests honest, and re-identification checks run before any dataset is shared.
Checklist: Is Your Test Data Ready for an Agent?
Pick one agent under test and check its environment against this list. Anything you cannot show is a gap to close before the next release cycle.
"The goal is not test data that looks like production. It is test data that behaves like production and belongs to no one."
What to do this week
01 Trace one agent run end to end
Choose the agent your team is testing now and follow a single test record from the source table to every place it lands: the harness, the model endpoint, the trace store, the evaluation report, and any exported file. Write each hop in a table with the system, the owner, the retention period, and who can read it. Most teams find at least one hop nobody had on the inventory, and that finding sets your priority order for the rest of the work.
02 Scan free text for identifiers
Sample five hundred rows from each free text field in your test environment and run a pattern and entity scan for names, phone numbers, addresses, ID formats, and account numbers. Have a person review a smaller sample of the hits and the misses. The hit rate tells you whether column masking is covering the fields that agents actually read, and the miss rate tells you how much to trust the scanner.
03 Lock the model endpoints and retention
List every endpoint and API key that test harnesses use, remove the ones your security team has not reviewed, and route the rest through one configuration. Set a retention period on traces and evaluation reports that matches the data they quote, and restrict read access to the people running the tests. This is a configuration change you can finish in a day, and it closes the two highest-severity gaps in the table above.
04 Start the edge-case library
Pull ten recent production incidents that involved odd inputs, such as a pasted email chain, a malformed date, or an unusual customer history. Rewrite each as a synthetic case that keeps the shape of the problem and removes any real person, then tag it and add it to the regression suite. Ten cases will not cover production, but they will show your team the difference between a green run on clean data and a green run on realistic data.
Let 10decoders Review Your AI Test Data
We trace test data through one to three of your agent test environments, covering free text, model endpoints, trace retention, exception handling, and synthetic data fidelity. You leave with a ranked gap list and a tiered test data model your quality, security, and data teams can adopt. This is an engineering review that supports your privacy and legal teams, not legal advice.