10decoders
A Load Test Built for the Wrong Kind of Traffic
In March 2026, Datadog's analysis of enterprise LLM traces found that 2% of all LLM call spans returned an error, and rate-limit failures accounted for close to a third of them, nearly 8.4 million events in that single month across its customer base. A month earlier, in February, the error rate ran higher still: 5% of spans failed, and 60% of those failures traced back to exceeded rate limits. Capacity, not model accuracy, is now the largest single cause of AI agent failure in production.
Traditional load testing tools were built to answer a narrower question: how many requests per second can this endpoint handle before latency degrades. That question assumes each request is independent, each response is deterministic, and failure means a slow page rather than a cascading retry loop. An AI agent breaks all three assumptions at once. A single user request can fan out into a planner call, several tool calls, a summarizer call, and a verifier call, each one hitting a model endpoint with its own separate rate limit. Ten simultaneous users can turn into two to three hundred API calls a minute before anyone notices traffic has grown at all.
When a provider throttles that traffic, most agent frameworks respond by retrying the failed call automatically, often with the context from the first attempt still attached. Each retry adds tokens, each added token makes the next call slower and more expensive, and the queue behind it keeps growing while the retries keep firing. Teams that have watched this happen in production describe the same shape every time: a working system, a burst of concurrent traffic, a rate-limit wall, then a self-inflicted pileup that looks nothing like the load test that supposedly cleared the system for launch.
A load test that never gets throttled has not tested the thing that breaks in production.
Where Load Testing Assumptions Break
| Assumption in the test plan | What happens in production | Severity |
|---|---|---|
| Rate-limit response never simulated | The test never sends traffic heavy enough to get throttled, so nobody sees how the system behaves the first time a provider returns a 429 | Critical |
| Retry loops have no ceiling | A throttled call retries automatically, appending more context each time, until either the provider unblocks it or the token bill forces someone to notice | Critical |
| One user request modeled as one API call | The real request fans out into a planner call, several tool calls, a summarizer call, and a verifier call, each with its own rate limit, none of which the test accounts for | High |
| Latency measured on a single session | P95 and P99 numbers come from one simulated user at a time, not from what happens when dozens of agent sessions compete for the same provider quota | High |
| Provider quota not shared across teams in the test | Two teams calling the same model provider each pass their own load test, then exhaust a shared quota neither test accounted for | Moderate |
| Test prompts far shorter than production ones | A short test prompt uses a fraction of the tokens a real conversation does, understating both cost and how quickly the rate limit gets hit | Lower |
Not sure your AI agents can survive real production load?
10decoders builds performance test plans around your agent's actual call fan-out, injects real provider rate-limit responses instead of just slow ones, and verifies your retry budgets hold before your first real traffic burst does.
Book a Free AI Assessment →The Retry Storm Nobody Load-Tested For
A retry storm is not really about the model. It is about what the code does the moment a provider returns a 429. Left unbounded, a retry loop appends more context on every attempt, treats a temporary throttle the same as a temporary network blip, and keeps firing until either the provider unblocks it or a human notices the cost graph moving sideways. None of that shows up in a load test that only ever sends traffic the system can already handle.
Datadog's own guidance to the teams it monitors is specific about this: build queue systems, backoff logic, and fallback capacity into the runtime itself, and set a hard budget so an agent loop terminates once it has spent a maximum number of calls or tokens, rather than retrying indefinitely. That is a capacity-engineering discipline, not a prompt-engineering one, and it needs its own test plan, separate from whatever validates that the agent's answers are correct.
The Demo-Cleared Stage
The load test ran once, against a single simulated user, and the provider never throttled a single call. Nobody checked what happens at ten or a hundred concurrent users, because the demo only ever needed one.
The Reactive Stage
A production rate-limit cascade forces a retry-and-backoff fix under pressure. The fix works, but it was never load-tested either, so nobody knows its real ceiling until the next unexpected burst of traffic finds it.
The Capacity-Engineered Stage
Load tests simulate the real call fan-out, inject rate-limit responses on purpose, and enforce a hard token or call budget on every retry loop before the system ever reaches production traffic.
Is Your Load Test Testing the Agent?
A load test that never gets throttled has not tested the part of the system that fails first in production. Run your current test plan against the questions below.
AI Agent Load Testing Reality Check
Capacity engineering for agents does not end at the load test that clears a launch. The retry budget and the circuit breaker still need to be tested too, before the first real burst of traffic tests them instead.
What to Do This Week
01 Simulate a rate-limit response on purpose
Add a fault-injection step to your existing load test that returns an actual 429 from a mock provider endpoint partway through the run, at a traffic level below what you expect in production. Watch what your retry logic does with it: how many attempts it makes, how much context it appends each time, and whether it ever stops on its own. If the answer is that it keeps trying indefinitely, that is the first fix, not the load test.
02 Put a hard budget on every retry loop
Give every agent loop a maximum number of calls or tokens it can spend before it terminates and returns a clear failure instead of continuing silently. Set the number low enough to catch a runaway loop in seconds, not minutes, and log every time the budget gets hit so you can see how often it comes close during normal operation.
03 Model the real call fan-out, not the front-door request
Trace one real user request through your agent end to end and count every downstream call it triggers: planner, tool calls, summarizer, verifier, anything else in the chain. Build your load test around that full number instead of the single request the user sees, since that is the traffic your model provider experiences.
04 Share provider quota visibility across teams
If more than one team or pod calls the same model provider, put their combined expected peak load in one place both teams can see before either one runs a load test or ships a feature that increases traffic. A quota that looks fine in isolation can already be exhausted once you add up what everyone else is sending it.
Let 10decoders Load-Test Your AI Agents for the Traffic They Will See
We build performance test plans around your agent's real call fan-out, inject actual provider rate-limit responses instead of just slow ones, and verify your retry budgets and circuit breakers hold before your first real burst of production traffic does.
Book a Free AI Assessment →Talk to the 10decoders team →