10decoders
Why cheaper tokens did not lower your bill
The LLM token price drop over the past two years is real and substantial. What the price trend obscures is that agentic and RAG-based workloads consume tokens at a rate that is structurally different from a chat interface. A chat interaction passes a user message and gets a reply. An agentic task passes a system prompt, a tool manifest, conversation history, tool call results, and the current task context before generating any output. At 5 to 30 times the token consumption per task, a 10x price reduction does not produce a 10x bill reduction; it produces a cost that scales with volume in a way that most initial API budgets were not designed to accommodate.
Most unplanned LLM cost growth in production traces back to three patterns. Growing context windows are the most common: every turn in an agent loop appends to the context, and multi-turn histories that look small in development become expensive in production when users run 20-turn sessions at scale. Undifferentiated model selection is the second: every query goes to the frontier model by default because that is what was used in development, and no one has implemented a routing layer that directs simpler queries to cheaper models. Repetitive input content is the third: the same 2,000-token system prompt is sent fresh with every API call, with no caching layer to avoid paying for content the provider has already processed.
Each of these patterns is addressable, and addressing any one of them typically produces a larger cost reduction than changing providers or negotiating volume discounts. The challenge is that each fix introduces a new engineering concern: routing without quality evaluation silently degrades answers, caching introduces cache invalidation complexity, and context compression can change the semantic content the model acts on.
Every team that has cut LLM costs without an evaluation framework has also cut answer quality. They just found out about the second part six weeks later.
The 5 LLM cost optimization approaches compared: savings potential and quality risks
Each cost reduction lever operates on a different part of the token spend and carries a different quality risk. Most enterprise teams that overspend do so because they skip layers 2 and 3, which require engineering work and have measurable quality trade-offs, and go directly to layer 5, which has the highest overhead and the most ways to accidentally degrade production quality.
| Optimization approach | Cost reduction potential | Quality risk | Production failure mode | Risk level |
|---|---|---|---|---|
| No optimization: frontier model for all queries, no caching, no routing | Baseline. Costs scale linearly with query volume and context length. Agentic workflows compound costs at 5-30x chatbot rate | None: frontier model quality is unchanged. But cost predictability is low; volume spikes produce invoice surprises with no budget trigger | Monthly API bill doubles when a new agentic feature ships. Finance escalation triggers a rushed cost-cutting sprint that does not include quality evaluation and degrades production answer quality silently | Critical |
| Prompt caching for static system prompts | 75-90% reduction on cached input tokens. Applies to large system prompts, tool manifests, and RAG context that repeats across calls. Does not reduce output token cost | Low. Cached content is byte-identical to uncached content from the model's perspective. Risk: cache invalidation on prompt edits causes unexpected cost spikes | Team iterates on the system prompt in production without realizing each edit invalidates the cache and temporarily spikes input token costs; no alerting on cache hit rate means the cost spike goes unnoticed for days | High |
| Model routing: small models for simple queries, frontier for complex | 60-80% per-query cost reduction with a 70/20/10 budget-mid-frontier split (practitioner estimates). Conservative estimates at 60-70% cheap model traffic: 37-46% overall reduction | High if routing is not evaluated. Routing without a quality eval sends queries that need frontier reasoning to a budget model; answer quality degrades silently until a user escalation surfaces it | Routing classifier trained on development queries misclassifies production edge cases. Complex reasoning queries get routed to budget model; quality drops 15-25% on those cases before the team notices through user feedback rather than monitoring | High |
| Output length optimization: structured responses instead of full reasoning | 50-90% output token reduction on classification, extraction, and scoring tasks. Switching from "answer with reasoning" to "return only the category code" cut one pipeline's output tokens by 90% (practitioner report) | Moderate. Loss of reasoning output removes explainability for regulated pipelines. Downstream systems relying on reasoning text for audit trails need redesign | Output format change reduces token cost but removes the reasoning trace that downstream audit logging depended on; compliance team flags the pipeline months later when it cannot produce required decision rationale | Moderate |
| Full token economics layer: caching + routing + length optimization + monitoring | 60-85% total cost reduction possible when all layers are applied. Savings are additive but not multiplicative; gains from each layer are smaller once earlier layers are in place | Lower with proper evaluation framework. Quality regression monitoring required across all three layers; any prompt, model, or context change needs eval run before production | A prompt cache invalidation event coincides with a routing classifier update; combined with reduced output length, quality drops below threshold on a specific query class; the team has monitoring but the alert threshold was calibrated on a different query distribution | Lower |
The most common cost optimization failure in production is deploying model routing without a quality evaluation framework. Routing classifiers are trained on a sample of queries and work well on that distribution. They systematically misclassify the tail of the query distribution, which is exactly where the most consequential queries tend to live: edge cases, multi-step reasoning requirements, and domain-specific questions that look simple by surface features but require frontier model reasoning to answer correctly. Without per-route answer quality monitoring, routing degradation accumulates silently until it shows up as a product quality complaint or a support ticket escalation.
Not sure where your LLM cost overruns are coming from?
10decoders audits your AI pipeline's token consumption breakdown by component, identifies which cost reduction levers apply to your workload without quality trade-offs, and implements the caching, routing, and monitoring layer your production budget requires. We have built cost-optimized pipelines for healthcare and fintech clients processing millions of tokens per day with regulated output quality requirements.
Book a Free AI Assessment →Why model routing fails silently and how to catch it before users do
Model routing works by classifying each incoming query as "simple" or "complex" and directing it to a budget or frontier model accordingly. The classification can be as simple as a token-count threshold, or as sophisticated as a trained classifier that scores queries on several dimensions. Either way, the same failure mode appears: the classifier gets the easy cases right and fails on the tail.
The tail of an enterprise query distribution is where the most sensitive and high-stakes queries live. A healthcare documentation assistant might route 90% of queries correctly to a budget model, but misclassify the 10% that involve clinical edge cases or unusual patient presentations as "simple" because their query text is short and direct. Those are exactly the queries where frontier model reasoning matters most. The cost saving from routing 90% of queries to a budget model is real. The quality loss on the misclassified 10% is also real, and it affects the highest-risk queries in the distribution.
The fix is per-route quality monitoring rather than aggregate answer quality monitoring. Aggregate quality scores blend the well-routed majority with the misrouted minority, and the majority's high scores mask the minority's degradation. Log which model handled each query, measure answer quality per routing path, and set independent quality thresholds for each path. Any routing path where quality drops below its threshold should trigger a review of the classifier's decision boundary for that query class, not just a global retraining run on the full distribution.
Frontier model for all queries
Cost scales linearly with volume. No caching on repeated content. No routing differentiation. Agentic workloads compound cost at 5-30x chatbot rates. Cost predictability is low and invoice surprises are common.
Prompt caching + model tier split
Provider prompt caching on static system prompts cuts 75-90% of repeated input token costs. A routing layer sends simple queries to a budget model. Quality evaluation on routing paths in place before go-live.
Caching + routing + output optimization + monitoring
All three cost layers active. Output length optimized per pipeline type. Per-route quality monitoring with independent thresholds. Cache hit rate and routing distribution tracked as primary cost health metrics.
The LLM cost optimization checklist
Each item below should be evaluated before deploying any cost optimization change to a production pipeline. The single most important discipline is running a quality evaluation before and after each change, not after all changes are deployed together.
Cutting LLM costs without a quality evaluation is not optimization. It is a deferred quality incident.
What to do this week
Pull a token breakdown by component for your highest-cost pipeline
Log input and output tokens broken down by stage for one week on your production pipeline with the highest API spend. How many tokens does the system prompt account for? How long is the average conversation history on multi-turn calls? How many output tokens does each call generate? This breakdown typically reveals one component that accounts for 50% or more of total token spend. That is where optimization effort belongs, and the answer is almost always different from what the team assumed before measuring.
Enable prompt caching on your static system prompts this sprint
If your system prompt is larger than 1,000 tokens and does not change between API calls, enabling provider prompt caching is the highest return-on-effort cost reduction available. The implementation is typically a few lines of API configuration, the quality impact is zero, and the savings on a high-volume pipeline are immediate. Do this before any routing or compression work. It requires the least engineering, has no quality trade-offs, and frees up budget to invest in the evaluation infrastructure that makes routing safe to deploy.
Build a quality evaluation set for your highest-volume query type before adding any routing
If your pipeline has a routing layer, or you plan to add one, build a labeled evaluation set this week specifically for the query distribution's tail: the complex, unusual, or high-stakes queries that appear infrequently but matter most. Include 50-100 of these alongside 150-200 representative standard queries. This set is what you will use to validate that the routing classifier does not misclassify the tail before the routing goes live. Without it, routing deployment is guesswork with a quality bill that arrives weeks later.
Add token spend tracking as a primary metric in your pipeline monitoring dashboard
Most pipeline monitoring tracks latency, error rate, and answer quality. Token spend per call is rarely tracked with the same granularity or alerting threshold. Add average input tokens, average output tokens, and cache hit rate to your production dashboard as primary metrics. Set alert thresholds: any call type whose average token spend increases by more than 20% over a rolling 7-day window should generate an alert. This catches prompt changes, model version changes, and routing classifier drift before they accumulate into a budget overrun.
Let 10decoders optimize your enterprise LLM cost layer
We audit your AI pipelines' token consumption by component, identify the highest-return cost reduction levers for your workload, and implement caching, routing, and monitoring with the quality evaluation framework that ensures cost savings do not come at the expense of answer quality. Our teams have built cost-optimized AI pipelines for healthcare and fintech clients where output quality is a regulatory requirement, not just a product metric.