10decoders
A Demo Read 50 Clean PDFs. Production Sends 5,000 Messy Ones.
Picture the pilot review. The team shows an agent that reads supplier invoices and posts them to the ERP. It handled the fifty sample files with near-perfect field accuracy, and the room approves the rollout. Three weeks into production the exception queue is longer than the one it replaced. A supplier sent a photographed delivery note. Another changed its invoice layout. A third attached a credit note that looks like an invoice with a minus sign.
Nothing in the model got worse. The sample was never representative. Pilot sets are usually chosen by the people building the pilot, and they pick files that look like the documents they imagine. Real inboxes hold rotated scans, stamps over totals, multi-page tables that break across pages, handwritten corrections, and languages the template never expected. Accuracy on the clean set tells you very little about accuracy on the whole stream.
The second problem sits one step later. Even when extraction is right, the output often has no safe place to go. A value read from page three of a scanned contract has no confidence score, no link back to the page, and no rule about who may see it. Teams then discover that document AI is a data readiness problem first and a model problem second, and the readiness work was not in the plan.
A document pipeline is only as good as its worst ten percent of inputs, and that is the part the demo skipped.
Six Gaps Between a Scanned File and Data an Agent Can Use
These are the gaps we check first when a document AI pilot looks good in a demo and disappoints in production. Severity reflects how much wrong data or rework the gap can push downstream before anyone notices.
| Gap | What happens | Severity |
|---|---|---|
| No labelled ground-truth set | Nobody has a set of real documents with verified answers. Accuracy claims rest on a handful of files the team picked, so regressions after a model or prompt change go unseen | Critical |
| Scoring by field, not by outcome | The report says 97% of fields are correct. One wrong total or bank account on a payment still counts as a small miss, although it is the error that costs money | Critical |
| Low-confidence output posted anyway | Uncertain values flow into the ERP or claims system with no review step, so errors surface weeks later during reconciliation or audit | High |
| Layout and template drift | A supplier updates its form or a regulator changes a template. Extraction rules quietly stop matching and accuracy drops with no alert | High |
| Permissions lost at ingestion | Documents move into a search index or agent memory without their access rules, so an agent can quote an HR file to the wrong person | Moderate |
| No path from value back to page | An extracted figure carries no reference to its source page, so reviewers and auditors must hunt through the original file to confirm it | Lower |
Not sure where your document AI gaps are?
10decoders takes one real document stream, builds a labelled test set from your own files, and measures extraction against the outcomes your team cares about. You get a ranked list of gaps and the review rules that close them.
Book a Free AI Assessment →Measure the Pipeline the Way the Business Feels It
The practical fix starts with the test set. Pull a few hundred documents from the live stream, not the ones people remember, and have someone who knows the process write down the correct answer for each. Include the ugly ones on purpose: rotated scans, faxes, photographed pages, and documents in a second language. That set becomes the standing test for every change, whether the change is a new model, a new prompt, or a new supplier template.
Then choose metrics that match the damage. Field accuracy is useful for tuning but a poor management number. Track the share of documents that went through with no human touch and no later correction, the share routed to review, and the share that were wrong and went through anyway. The last figure is the one to drive toward zero. A pipeline that routes 30% of documents to a person and gets nothing wrong is usually better than one that automates 95% and gets two percent wrong in ways nobody sees.
Routing rules matter as much as the model. Each extracted value should carry a confidence score and a pointer to its source page. Values below a threshold, totals that disagree with line items, and amounts above a set limit go to a reviewer who sees the page with the value highlighted. The reviewer's corrections should flow back into the labelled set, so the next round of testing is harder and closer to what production sends.
Works on the Sample
Accuracy is quoted from a few chosen files. Output is posted straight to the target system, and errors are found by the people downstream.
A Test Set Exists, Routing Is Basic
A labelled set covers the common document types. Low-confidence values go to review, but drift and edge cases are caught only by spot checks.
Outcome Metrics, Traceable Values
The test set is refreshed from live corrections, every value links to its page, permissions travel with the file, and the wrong-but-passed rate is reported weekly.
Checklist: Is Your Document Stream Ready for an AI Agent?
Pick one document type that an agent is expected to handle. For each item, ask whether you can show the evidence today.
Automate the documents you can prove, route the ones you cannot, and measure the difference every week.
What to do this week
01 Sample the live stream, not the pilot folder
Pull two hundred documents of one type from the last month, picked at random from the real inbox or repository. Count how many are scans, photographs, or have more than one language. Compare that mix to what your pilot used. The gap between the two numbers is the first estimate of how much your demo accuracy overstates production.
02 Write the answers down for fifty of them
Ask a process owner to record the correct values for fifty documents, including at least ten difficult ones. Run your current extraction against them and score three things: documents that passed with no errors, documents that would have been routed to review, and documents with errors that passed. Keep the file as the baseline for every future change.
03 Add one cross-check and one review rule
Choose a check that does not depend on the model, such as line items summing to the total, or a tax number matching the vendor master. Send every failure to a reviewer with the source page shown. Add a second rule that sends any amount above a set limit to a person regardless of confidence. Two simple rules usually catch the errors that cost the most.
04 Check what happens to access rules
Take one confidential document, such as an HR letter or a restricted contract, and trace it into your index or agent memory. Ask the agent a question that only an unauthorized user should not be able to answer, using a test account. If the answer comes back, fix permission handling before adding more document types.
Let 10decoders Make Your Documents Ready for AI
We take one live document stream, build a labelled test set from your files, score extraction on touchless, review, and wrong-but-passed rates, and check traceability and access rules end to end. You leave with a ranked gap list and a review design your team can put into production.