10decoders
Why LLM costs are harder to predict than other infrastructure costs
Traditional infrastructure costs scale with compute and storage, which are reasonably predictable from load projections. LLM inference costs scale with token volume, which is driven by prompt length, output length, context window size, and query frequency. All four of these vary in ways that are difficult to estimate before a system runs in production on real queries. A customer support use case that looked like it would average 800 input tokens per query in testing can easily average 2,400 tokens per query in production once real users include conversation history, document attachments, and follow-up clarifications the test set didn't anticipate. That's a 3× cost multiplier from query distribution alone, before any architectural decision.
The compounding problem is that LLM cost optimization decisions interact with quality. Reducing prompt length reduces cost but can reduce output quality if the removed context was load-bearing. Switching to a smaller model reduces per-token cost but may reduce quality on complex queries. Caching outputs reduces inference calls but returns stale results when the cached output is no longer accurate. Each optimization has a quality trade-off. Measure it; don't assume it away. Teams that treat cost optimization as a pure engineering problem, without an evaluation suite to measure the quality impact of each change, make optimizations that lower cost and also lower quality in ways they only discover from user complaints.
The teams that handle this well start measuring cost per query in staging, before launch. A cost budget sits alongside the quality budget. Every architectural decision gets both a cost measurement and a quality measurement. By the time the system reaches production, they know exactly which query types are expensive, which optimizations are safe, and what the cost-quality trade-off curve looks like for their specific use case. The teams that don't do this start measuring cost when the first invoice arrives and optimize under pressure, which is the worst condition for making architecture decisions that affect quality.
"Every token you send to the model costs money. Every token you could have cached, compressed, or routed to a cheaper model costs more than it should. The bill arrives whether or not you planned for it."
Five cost optimization decisions and what each one actually requires
Each of the five optimizations below reduces inference cost without requiring a quality trade-off when implemented correctly. Each one also has a common failure mode where it is implemented incorrectly and does cause a quality problem. The table maps each optimization to the cost reduction it delivers, the implementation requirement, and the risk of getting it wrong.
| Optimization | How it reduces cost | What correct implementation requires | Common failure mode | Skipping risk |
|---|---|---|---|---|
| Semantic output caching | Stores model outputs and returns cached results for semantically similar queries without making an inference call. Effective for any use case where a meaningful percentage of queries ask the same thing in slightly different words: support, internal search, policy lookups, product documentation | An embedding-based similarity search against a cache of previous query-output pairs. A similarity threshold tuned to your use case: too low returns irrelevant cached results, too high misses most cache hits. A cache invalidation strategy tied to knowledge base updates. A quality check confirming cached outputs are still accurate after knowledge base changes | Similarity threshold set too low, returning cached outputs for queries that are superficially similar but require different answers. Cached outputs served after the underlying knowledge base changes, producing answers that were correct three weeks ago but aren't now. No invalidation logic means the cache becomes a source of stale information at scale | Critical |
| Model routing and tiering | Routes queries to cheaper, smaller models when the task doesn't require frontier-model capability, and escalates to more capable models only when needed. A classification query doesn't need GPT-4 class capability. A complex multi-step reasoning task does. Routing correctly can cut per-query cost by 60–80% on the queries that go to the smaller model | A query classifier that routes with high accuracy. Evaluation suites for both the small and large model tiers to confirm quality holds at each tier. A clear definition of which query types are safe for the smaller model and which require escalation. Monitoring to detect classifier drift over time as query distributions shift | Classifier routes too many queries to the cheaper model, degrading quality on complex inputs. No evaluation of the smaller model against the full query distribution means quality problems go undetected until user complaints. Classifier accuracy degrades over time as query types evolve but the classifier doesn't get updated | Critical |
| Prompt compression | Reduces input token count by removing redundant content, compressing few-shot examples, shortening system prompt instructions, and trimming conversation history aggressively. Every token removed from the prompt is a direct cost reduction with no inference quality impact if the removed content wasn't load-bearing | Token counting on every prompt component to identify where tokens are concentrated. An evaluation run after each compression step to confirm quality doesn't degrade. Conversation history truncation logic that retains the most relevant context rather than simply cutting the oldest messages. Periodic review as prompts evolve to catch prompt bloat from iterative additions | Removing context that the model was using to produce accurate outputs, discovered only after quality drops in production. Truncating conversation history in a way that loses critical context from earlier in the conversation. Compressing few-shot examples to the point where the model loses the format or style guidance they were providing | High |
| Context window management | Controls how much document context, conversation history, and retrieved content is included in each query. Longer contexts cost more per query and don't always improve output quality. Identifying the minimum context required for acceptable quality, and setting a context length budget, reduces cost on every query without requiring any architectural change | Ablation testing to identify how much context the model actually uses versus how much is appended by default. A context budget per query type based on the ablation results. Retrieval relevance scoring to ensure retrieved chunks are actually relevant to the query before including them. Conversation compression for long sessions that retains semantic meaning at lower token count | Including the maximum context window by default on every query without testing whether the additional context improves output quality. Retrieving and including all potentially relevant document chunks rather than the most relevant ones, inflating input token count with low-value content. Carrying full conversation history indefinitely rather than compressing or summarizing older turns | High |
| Batch inference for async workloads | Groups non-real-time queries into batches and processes them at off-peak hours using batch inference APIs, which typically cost 50% less than real-time inference. Report generation, document summarization, overnight data enrichment, and any workload that doesn't require immediate response are candidates | Identifying which use cases in the system have async tolerance: they don't need a result within the user's current session. A batch scheduler with retry logic and error handling. Output storage and retrieval so batch results are accessible when the user needs them. A fallback to real-time inference when batch latency would exceed the use case's tolerance | Routing interactive workloads through batch inference, causing unacceptable latency for users who needed a real-time response. No retry logic for batch failures, producing missing outputs that require manual intervention. Using batch inference for workloads where the slight additional latency still exceeds SLA requirements and the cost saving doesn't justify the complexity | Moderate |
Not sure where your highest LLM cost exposure is?
10decoders runs two-week AI engineering assessments that instrument your current inference costs at the query level, identify the specific optimization opportunities in your architecture, and implement caching, routing, and compression strategies that reduce cost without touching quality.
Book a Free AI Assessment →How cost optimization fits into a production AI architecture
Cost optimization works best as a layer added to a working system, not as a constraint applied during initial development. A team that spends the first sprint trying to minimize token count before they know which query types are expensive and which optimizations are safe ends up optimizing the wrong things. The right sequence is to build the system, instrument cost per query, run it under realistic load to get an actual cost profile, and then apply optimizations in order of impact against a quality baseline.
The instrumentation step is where most teams are currently missing. Knowing that the system costs $0.03 per query on average is much less useful than knowing that 12% of queries cost $0.18 per query because they include full document context, and that those queries are concentrated in a specific use case where a retrieval relevance threshold could cut context length by 60% without quality impact. That level of cost visibility requires per-query logging that captures input token count, output token count, model used, cache hit or miss, and query type. It takes half a day to set up. Without it, cost optimization is guesswork.
For most enterprise systems, start with semantic caching: it has the highest impact and zero quality risk when done correctly. Model routing comes next: high impact, but it requires a classifier and an evaluation suite. Prompt compression follows, with a medium-impact payoff and a required evaluation run after each compression step. Context window management is similar in impact and requires ablation testing to do safely. Batch inference is lower priority and applies only to async workloads. Each optimization should be measured against the quality baseline before the next one is applied, so that if a quality regression appears, the cause is unambiguous.
Unmonitored Inference
No per-query cost measurement. All queries go to the same frontier model. No caching. Full context window on every call. No distinction between real-time and async workloads. The system works well and the cost is manageable at pilot volume. At production scale, the same architecture produces a bill that is 3–5 times the estimate. Optimization starts under pressure, with a live system and no cost baseline to work from.
Cost-Instrumented & Optimized
Per-query cost logging capturing input tokens, output tokens, model tier, and cache status. Semantic caching implemented for high-repetition query types. Context length budgets set per use case based on ablation results. Prompt compression applied and verified against the quality baseline. Batch inference routing for async workloads. Cost per query tracked in a dashboard with alerting on anomalous spend. The system costs 40–60% less than Stage 1 at equivalent quality.
Dynamic Model Routing
A query classifier routes each request to the appropriate model tier based on task complexity. Simple classification and extraction queries go to a small, fast, cheap model. Complex reasoning, multi-step synthesis, and edge cases escalate to a larger model. The classifier is monitored for accuracy drift. Cost and quality are tracked per model tier. Cache hit rates, average input tokens, and output tokens per use case are reported weekly. Each optimization layer is independently measurable and adjustable without touching the others.
LLM cost optimization readiness checklist
"Cost optimization done right is invisible to users. Every query returns the same quality output. The system just gets there more efficiently. The work is in knowing which queries need the full model and which ones don't."
What to do this week
01 Instrument cost per query this week, before anything else
Add input token count, output token count, model used, and use case category to your inference call logging. Pull the last 30 days of logs and calculate the distribution of cost per query: the median, the 90th percentile, and the top 10 most expensive query types. This data will determine which optimization to prioritize. If 80% of cost is concentrated in 20% of query types, the optimization target is clear. If cost is distributed evenly across all query types, a different approach is needed. Every optimization conversation is speculative until this data exists. Once it does, the highest-impact change is usually obvious within an hour of looking at the distribution.
02 Run a query repetition analysis on your highest-volume use case
Pull the last 30 days of queries from the use case with the highest query volume. Run them through a sentence embedding model and calculate pairwise similarity scores. Cluster queries that exceed a similarity threshold of 0.85. The percentage of queries that land in a cluster with an existing query is your cache hit rate estimate. If it's above 20%, semantic caching will produce meaningful cost reduction for this use case. If it's below 10%, caching won't move the number significantly and a different optimization deserves the engineering attention. This analysis takes a few hours to run and produces a clear recommendation on where caching investment is justified.
03 Run a context length ablation on your most expensive use case
Take the use case with the highest average input token count. Systematically reduce the context length in steps: first to 75% of current, then 50%, then 25%. Run each configuration against 50–100 test cases from your evaluation suite and record the quality score. Plot quality against context length. The inflection point where quality starts dropping is your context budget. Everything above that budget is tokens you're paying for without quality benefit. Most teams that run this exercise for the first time find the inflection point is significantly below what the team assumed, and the cost saving from setting a context budget is immediate and sustained.
04 Set a cost alert threshold this week
Log into your inference API provider's dashboard and configure a spend alert. Set the threshold at your estimated monthly budget plus 25%. If you don't have an estimated monthly budget, use last month's actual spend plus 25% as a starting point. An alert that fires before you exceed budget gives you time to investigate and respond. An alert that fires after you exceed budget tells you something you'll find out anyway. The alert takes ten minutes to set up. Every team that has had an unexpected LLM cost overrun wishes they had set one up earlier.
Let 10decoders reduce your LLM inference costs
We run two-week AI engineering assessments that instrument your inference costs at the query level, identify the specific optimization opportunities in your architecture, and implement caching, routing, and compression strategies that reduce cost without touching quality.