10decoders
The problem that looks like a model problem but isn't
An AI pipeline that worked last Tuesday starts producing unreliable outputs today. The first instinct is to check the model. Has something changed in the serving infrastructure? Did a prompt template update go out? Is the LLM behaving differently? The debugging process takes hours or days. Eventually, someone finds it: an upstream data team changed a field name in a source table three days ago. The old field name still exists in the schema but now returns nulls. The pipeline never flagged it, because nobody specified that field as a dependency with a non-null requirement. The model silently processed the nulls, the outputs degraded, and nobody knew until a downstream team noticed the numbers looked wrong.
This pattern is not rare. It is the most common class of AI pipeline incident in enterprise settings. The model did exactly what it was built to do. The data it received had changed in a way that invalidated the assumptions the model was built on, and there was no contract in place to catch that change at the boundary. Data contracts are how teams close that gap. They are explicit, versioned agreements between data producers and data consumers that specify what the data should look like, how fresh it should be, what quality bounds apply, and what happens when those bounds are violated. In a well-functioning AI data pipeline, a contract violation triggers an alert before bad data reaches the model, not an incident review after bad outputs reach the business.
Data contracts are not new to software engineering broadly. APIs have had contracts for decades: a defined schema, a versioning convention, a deprecation policy. The concept transfers directly to data pipelines, but most organizations have not applied it there. The data engineering team treats its output tables as internal infrastructure. The model team treats those tables as reliable inputs. The gap between those two assumptions is where AI pipeline incidents live.
“A model that performs well on clean data and poorly on real data doesn't have a model problem. It has a data agreement problem.”
The six types of data contract your AI pipeline needs
A data contract covers more than the schema. A field can have the right name and type while its value distribution has shifted enough to invalidate the model's learned assumptions. The table below maps the six contract types that matter for AI pipelines, what breaks when each is absent, and the operational priority of getting each in place.
| Contract Type | What it specifies | What breaks without it | Priority |
|---|---|---|---|
| Schema contract | Field names, data types, nullability, and the versioning policy for changes. Specifies how the producer must communicate schema changes and how much lead time is required before a breaking change takes effect | A renamed or removed field silently returns nulls. The pipeline continues running. The model processes the nulls as valid inputs. Output quality degrades without a clear trigger event or alert | Critical |
| Quality contract | Acceptable null rate per field, value distribution bounds, outlier thresholds, and referential integrity requirements. Defines what a quality violation looks like and at what severity it should trigger an alert versus halt the pipeline | Null rate creeps from 0.5% to 8% over three weeks. No alert fires because no bound was specified. The model's behavior shifts gradually, and the degradation is attributed to model drift rather than data quality drift | Critical |
| Freshness contract | Maximum acceptable data age at time of model inference, SLA on upstream data arrival, and the escalation path when the SLA is missed. Particularly important for real-time inference pipelines where stale data produces demonstrably wrong outputs | An upstream ETL job runs four hours late. The inference pipeline consumes yesterday's data without flagging it. A time-sensitive model output (fraud score, inventory recommendation) is based on data that no longer reflects current state | Critical |
| Volume contract | Expected row count range, acceptable daily variance, and alerting thresholds for significant drops or spikes. Volume anomalies often signal upstream processing failures that will not appear as schema or quality violations | An upstream source table drops 40% of its rows due to a filter logic change. The pipeline runs to completion on the reduced dataset. The model output is statistically valid on the data it received, but represents only part of the population it is supposed to cover | High |
| Lineage contract | Documentation of upstream sources, transformation logic applied to each field, and the business rules used to derive calculated fields. Maintained as versioned documentation, not tribal knowledge held by individuals | A model team inherits a pipeline from a team member who has left. The lineage is undocumented. When the model behaves unexpectedly on a specific subset of records, there is no way to trace which upstream transformation produced those records without re-engineering the pipeline from source | High |
| Semantic contract | Business definition of each field, the time period it represents, how it handles edge cases, and what it explicitly does not measure. Prevents misinterpretation of fields with identical names but different meanings across data domains | A field named "customer_status" means "subscription tier" in the billing system and "support escalation level" in the CRM. A model trained on billing data gets evaluated against CRM outputs. The apparent model underperformance is a semantic mismatch, not a modeling error | Moderate |
Not sure where your AI pipeline data contract gaps are?
10decoders runs two-week data pipeline assessments that map your current AI data flows against all six contract types, identify which gaps carry the highest incident risk, and produce a data contract implementation roadmap your engineering team can execute.
Book a Free AI Assessment →Why data contracts don't get implemented until after the first major incident
Teams know that data contracts are good engineering practice. Why they consistently deprioritize them until something goes wrong comes down to two things that tend to reinforce each other.
First, nobody owns it. The data engineering team assumes the consuming team will specify what they need. The model team assumes the data engineering team will maintain what it produces. Neither has a clear mandate to own the contract end-to-end, so it gets deferred to "we'll set it up properly after launch." After launch, data contract work competes with incident response for the same engineering time. It rarely wins that fight.
Second, the cost of skipping it is invisible until it isn't. Implementing a data contract takes real time: schema documentation, quality bound specification, monitoring setup, alerting, change notification agreements. That cost is immediate and easy to measure. The cost of not having a contract, the hours tracing a degraded model output back to a null rate that crept past 8% over three weeks, the business decisions made on stale data, the credibility lost when the AI system produces numbers that look wrong, is invisible right up until it materializes. Teams consistently underestimate that deferred cost because they haven't paid it yet. The first major incident recalibrates the calculation, but by then remediation competes with recovery for the same team's attention.
Implicit Agreements
No formal data contracts. Data engineering teams maintain source tables and pipeline logic informally. Model teams consume output tables with schema assumptions baked into code. Changes propagate without notification. Incidents are traced manually, often taking days. Contract gaps are known but deprioritized. Pipeline reliability depends on individuals, not systems.
Informal Contracts
Schema documented in wiki pages or README files. Some quality checks in pipeline code. No versioning on contracts. Change notification handled via Slack or email, inconsistently. Freshness and volume contracts absent. Lineage partially documented. Teams know who to call when something breaks but still spend significant time on manual root cause analysis. Contract gaps identified after incidents, not before.
Formal Data Contracts
All six contract types implemented and versioned. Schema, quality, freshness, and volume contracts enforced with automated monitoring. Contract violations trigger alerts before bad data reaches the model. Lineage documented and kept current. Semantic contracts reviewed by data owners and model teams jointly. Change process includes lead time requirements and consumer impact assessment. Incident MTTR reduced to hours from days.
The data contract readiness checklist
“The data contract is the agreement that makes model monitoring meaningful. Without it, you're watching metrics on outputs without knowing what you agreed the inputs should look like.”
What to do this week
01 List every data source your most important AI pipeline consumes
Pick your single highest-value AI pipeline, the one where a production incident would have the most business impact. Write down every upstream data source it consumes: source tables, API feeds, streaming topics, file drops. For each source, answer three questions: who owns it, when did we last have a formal conversation with that owner about our data requirements, and what would we do if that source changed without notice today? Most teams discover they have three to five sources where the honest answers are "unclear," "never," and "we'd find out from a production incident." That list is your data contract backlog.
02 Instrument the null rate on your top input fields before the next model retrain
Before the next model retrain or deployment, add null rate monitoring to the five input fields that have the highest weight in your model's predictions. If you don't know which fields those are, use frequency of appearance in feature importance analysis, or simply the fields your data team most often gets questions about. Log the null rate for 30 days and plot the trend. A null rate that is stable is a baseline. A null rate that is trending upward is a data contract violation waiting to be formalized. Monitoring it before the contract is written tells you what the contract bounds should be.
03 Send one email to your most critical upstream data team this week
Identify the single upstream data team whose schema changes would have the most damaging impact on your AI pipeline. Send them one email with two specific questions: what is your process for notifying downstream consumers of breaking schema changes, and what lead time can you commit to before a breaking change goes to production? You don't need a formal contract negotiation to start. The goal of this email is to surface whether a notification process exists, and if it doesn't, to begin the conversation about creating one. Most upstream teams are willing to establish this process when asked directly. They often just don't know they have downstream AI consumers who depend on schema stability.
04 Review the last three AI pipeline incidents for data contract root causes
Pull the post-mortems or incident logs from your last three AI pipeline incidents. For each one, trace the root cause and classify it against the six contract types in the table above: was it a schema issue, a quality issue, a freshness issue, a volume issue, a lineage gap, or a semantic mismatch? In our experience working with enterprise AI teams, at least two of the three incidents will trace to a contract gap rather than a model failure. That classification tells you which contract type to implement first, because it's the one that has already cost you incident time in the past 90 days.
Let 10decoders build data contracts into your AI pipeline
We run two-week data pipeline assessments that map your current AI data flows, identify contract gaps against all six contract types, set quality and freshness bounds from your historical data, establish change notification agreements with your upstream teams, and produce a data contract implementation roadmap your engineering team can execute in the next sprint.
