10decoders
A Deploy Passed Every Check, and Then All 39 Tools Stopped Working
Consider a real incident from the accessibility-agents project, tracked as issue 205. Version 7.0.2 of its MCP server started cleanly and listed all 39 of its tools. Every single call to those tools then failed. The cause was small: an SDK bump from 1.27.1 to 1.30.0 meant the server declared an output schema without returning the structured content that schema promised. The handshake passed. The tool listing passed. Any test that stopped at either of those points would have shipped it.
That case is loud, because the calls errored and someone noticed. The quieter version is worse. A server can change what a tool requires or accepts and keep every call succeeding, so nothing reports the change. The model treats a tool’s schema as instructions: which arguments to send, which are optional, which values are allowed. Remove a required entry or drop an enum value and the model’s behavior shifts, while your logs show green calls. The client receives the tool definition fresh on every listing, and the listChanged notification covers tools being added or removed, not an existing tool changing shape.
There is also no formal fix on the way. A proposal to add tool versioning to the protocol, SEP-1575, was opened on 30 September 2025 and has since been closed as having no sponsor. Until that changes, the version history of your tools lives in your own test suite or nowhere. The teams that treat tool definitions as a contract, with a stored copy and a diff, will see drift. The teams that treat them as plumbing will see it as a model that got worse for no reason.
A tool that fails loudly is a bug. A tool that changes quietly is a regression your monitoring was never built to see.
Six Kinds of Tool Change, Ranked by How Quietly They Hurt
Not every schema change deserves a failed build. The severity label below reflects how likely the change is to alter agent behavior without raising an error. Gate the top of the table and let the bottom pass.
| Tool change | What happens in production | Severity |
|---|---|---|
| A required argument is removed or made optional | The model stops sending a field the downstream system still needs. Calls may succeed with incomplete data | Critical |
| Output schema declared without matching content | The listing looks healthy while every call fails, as in issue 205. Only a real call through the real client catches it | Critical |
| An enum value is dropped or a type is widened | The model picks a value that no longer behaves as before, or a field now accepts input the handler never validated | High |
| A description is rewritten | The text the model reads as guidance changes, so tool selection and argument choices drift with no code change in your agent | High |
| Annotations change | Hints such as read-only or destructive can feed approval logic. A flipped hint changes what a human is asked to confirm | Moderate |
| A new optional parameter or a new tool appears | Additive and usually safe. Surface it in review rather than blocking the build | Lower |
Not sure where your MCP tool contract gaps are?
10decoders reviews the MCP servers your agents depend on, covering first-party and third-party ones, and checks which changes your current tests would catch. You get a gap list and a contract test plan for each server.
Book a Free AI Assessment →Four Contract Tests That Cover the Table
The first test is an end-to-end call for every tool on every build. Run a real call through the actual client and server pair and assert on the shape of the response. A handler unit test will not do this, and neither will a check that the tool list returns. This is the test that would have caught issue 205 before release, and it is the one most teams skip because it feels redundant with the smoke check.
The second is a schema snapshot with field-level assertions. Store the advertised definition of each tool and diff it against the live one on a schedule and on every deploy. Assert on the fields that carry meaning to the model: required lists, enums, types, and descriptions. A hash of the whole definition is tempting but noisy, because it flags harmless edits and teaches the team to ignore the alert. Field-level diffs tell the reviewer what changed and why it matters.
The third is a negative test. Send a deliberately incomplete call that omits a required argument and assert that the server rejects it. This catches a server or client that quietly stopped enforcing a constraint, which is the exact failure the snapshot can miss when the schema text is unchanged but the handler no longer checks it. The fourth is selective gating: fail the build on silent semantic changes such as removed requirements, dropped enum values, widened types, and rewritten descriptions, and let additive changes through as review notes. A gate that blocks everything gets overridden, and an overridden gate protects nothing.
Smoke Checks Only
Tests confirm the server starts and lists its tools. Nobody stores a copy of the definitions, and a changed tool shows up as a vague drop in agent quality.
Stored Contracts and Real Calls
Each tool has a stored definition and an end-to-end call in CI. Diffs are reviewed by a person, and third-party servers are pinned to a version you chose.
Selective Gates With Named Owners
Silent changes fail the build, additive ones open a review, and a scheduled job re-checks live servers. Every server has an owner who approves contract changes.
Checklist: Is Your MCP Tool Contract Under Test?
Run this against each server your agents call, starting with the ones you do not control.
Pin the contract, test the call, and let the model read a tool definition that you have already approved.
What to Do This Week
01 Inventory every MCP server your agents call
List each server, who runs it, which version is live, and which agents depend on it. Separate the ones your team deploys from hosted and third-party ones, because the second group can change without a release on your side. Most inventories turn up at least one server nobody on the current team set up.
02Snapshot today’s tool definitions
Call the tool listing on each server and save the full definition of every tool into version control, with the date and server version. This is your baseline. Even before you write a single test, the stored copy gives you something to diff against the next time agent behavior shifts and nobody knows why.
03 Add one real call per tool to CI
For each tool, write the smallest valid call you can, run it through your actual client against a test instance, and assert on the response shape. Add one negative case for any tool that has a required argument. Start with the five tools your agents call most often, and expand from there rather than trying to cover everything at once.
04 Write the gating rule down
Decide in advance which diffs fail the build and which open a review. A workable starting point is to fail on removed requirements, dropped enum values, widened types, and rewritten descriptions, and to review new optional parameters and new tools. Put the rule next to the baseline, name the approver, and revisit it after the first month of real diffs.
Let 10decoders Test the Contracts Your Agents Depend On
We review one agent and the MCP servers behind it: tool inventory, baseline snapshots, end-to-end and negative call tests, a selective gating rule for CI, and a schedule for re-checking third-party servers. You leave with a working contract suite and a list of the changes it would have caught.