10decoders
Why LLM Observability Is Different From Application Monitoring
Standard application monitoring tracks whether a service is up, how fast it responds, and what error codes it returns. Those signals matter for LLM systems too, but they miss the failure modes that are specific to language models. A system can return HTTP 200 with 400ms latency and still be producing outputs that are wrong, unhelpful, or subtly off in ways that erode user trust over weeks. Error rate monitoring catches hard failures. It doesn't catch quality degradation, which is the more common and more expensive problem in enterprise LLM deployments.
LLM quality degradation has no equivalent in traditional software. When a function returns the wrong value, the bug is consistent and reproducible. When an LLM system starts producing lower-quality outputs, the change is gradual, probabilistic, and often tied to shifts in input distribution that the team doesn't know are happening. Query types change as more users onboard. The knowledge base gets updated and some retrieval paths break. A prompt modification gets pushed without an evaluation run. None of these changes trigger an error. All of them affect output quality. Without structured logging on what the system is receiving and producing, none of them are visible until users start complaining.
The teams that operate LLM systems well instrument them before those systems see production traffic. Input token counts, output token counts, latency per request, cost per query, and a sampled quality evaluation run on recent outputs give the team a complete picture of how the system is behaving. When something changes, the signal is in the data before it shows up in support tickets. Debugging takes hours. Without that data, debugging is a process of reproducing failures manually from user-reported symptoms, which is the slowest and most expensive way to find an LLM quality problem.
“An LLM system without observability is a system whose behavior is unknown. You find out what it's doing when users tell you, which is always too late.”
Five Observability Signals and What Each One Tells You
LLM observability requires signals at five levels: what the system is receiving, what it's producing, how long it's taking, what it's costing, and whether the outputs are any good. Each signal catches different problems. Missing any one of them leaves a blind spot that will be exploited by the first incident that lives in that category.
| Signal | What it captures | What it detects | Common gap without it | Skipping risk |
|---|---|---|---|---|
| Structured input/output logging | Full prompt text (or a hash for PII-sensitive use cases), retrieved context chunks for RAG systems, model response text, use case category, user session identifier, and timestamp for every production inference call. The complete record of what the system received and produced, queryable for debugging and pattern analysis | Prompt drift over time as prompts are iteratively modified without version control. Changes in query type distribution as the user base grows. Specific user sessions or input patterns that consistently produce poor outputs. RAG retrieval failures where the model received insufficient context for a specific query type. Reproducing a reported failure without requiring the user to reconstruct their exact input | When a quality problem is reported, the team has no record of what the system received or produced during the incident. Debugging requires manually constructing test queries that approximate what the user experienced, which rarely reproduces the exact failure. Investigation takes days. The root cause often remains uncertain | Critical |
| Latency per request with context length breakdown | End-to-end latency for each inference call, broken down by retrieval time (for RAG), model inference time, and post-processing time. Input token count and output token count per request, to separate latency increases driven by longer prompts from those driven by slower model inference or infrastructure issues | Latency regressions from prompt length increases, which are common when prompts are modified iteratively without token counting. Retrieval layer slowdowns that affect end-to-end response time before they become visible to users as timeouts. Output length increases from prompt changes that cause the model to produce longer responses, increasing both latency and cost. Infrastructure degradation that affects model inference without changing error rates | Latency increases are noticed when users report slow responses, not when the metrics change. By that point, the increase has been affecting users for days or weeks. Without the context length breakdown, the team can't tell whether the latency regression comes from the prompt, the retrieval layer, or the inference infrastructure, which means debugging starts from scratch | Critical |
| Cost per query by use case | Inference cost per request calculated from input and output token counts and the model's current pricing, attributed to the specific use case or workflow that triggered the request. Aggregated daily and weekly, with per-use-case breakdowns and trend lines. Alerting configured on cost-per-query thresholds and on aggregate daily spend | Cost increases from prompt changes that inflate token counts, which happen frequently in systems where engineers modify prompts without monitoring the cost impact. Unexpectedly expensive query types that account for a disproportionate share of total inference spend. Cost scaling patterns that make the current architecture unsustainable at projected production query volumes. Model tier routing failures where expensive frontier-model inference is used for queries a cheaper model could handle | Cost increases are discovered on monthly invoices. By then, the architecture that drives the cost is in production and harder to change. The team can't tell from an invoice which queries, use cases, or prompt changes caused the increase. Optimization requires retroactive analysis of logs that may not have been collected | High |
| Sampled quality evaluation on production outputs | Automated quality scoring on a random sample of production outputs, typically 2–5% of daily volume, using the same evaluation metrics established before launch: faithfulness, answer relevance, and context relevance for RAG systems. Scores tracked as time series, with alerting when the rolling average drops below threshold. Failed samples flagged for human review | Quality degradation that doesn't trigger errors: prompt drift, retrieval configuration changes, corpus updates that break specific query types, and model behavior changes from provider updates. Output quality shifts that would take weeks to surface in user feedback but appear in automated quality scores within 24–48 hours. Specific query categories where quality is degrading while others remain stable, narrowing the root cause investigation | Quality changes are discovered from user complaints, satisfaction score drops, or support ticket volume increases. The team has no data on when the quality started degrading, which system change caused it, or how widespread the problem is. Investigation is reactive and slow | High |
| Error rate and failure mode classification | Rate of hard failures (timeouts, API errors, content policy blocks, output parsing failures) tracked separately from soft failures (outputs that are technically valid but empty, truncated, or malformed for the use case). Each failure type classified by root cause category: model API, retrieval layer, post-processing, or application logic. Trend tracking with alerting on rate changes | Infrastructure issues at the model provider level that increase timeout rates before becoming full outages. Content policy block rate increases that indicate prompt drift into policy-sensitive territory. Output parsing failures from model responses that don't conform to expected structured output formats, which typically means a prompt change or model update changed response formatting. Retrieval failures that cause the RAG layer to pass empty context to the model | Infrastructure monitoring catches full outages. Partial failures, elevated timeout rates, and soft failure patterns are invisible without failure classification. A 3% content policy block rate increase looks like noise in aggregate error rate monitoring but signals a specific prompt problem when classified by failure type | Moderate |
Not sure what your LLM system is doing in production right now?
10decoders builds LLMOps observability stacks for enterprise teams: structured logging, quality metric sampling, cost attribution, and alerting configured before problems appear. Two-week implementation with dashboards your team can operate without us.
Book a Free AI Assessment →How Observability Fits Into the LLM System Lifecycle
Observability is most useful when it's built before the system goes live, not assembled after the first incident. A team that instruments logging, latency tracking, and cost attribution before launch has a baseline to compare against from day one. When something changes in week three, the change is visible in the data relative to the baseline. Without a baseline, every investigation starts from “something is different now” with no quantitative picture of what was normal before.
The order that works: structured logging on inputs and outputs goes in before the first user sees the system. Latency tracking with context length breakdown and cost-per-query attribution follow. Quality metric sampling needs the evaluation suite from pre-launch testing, so it requires that work to already be done. Failure mode classification needs the failure categories defined before the classification logic can be written. The full stack takes three to five days to instrument. Every day it's missing is a day the system runs without the data needed to investigate the first problem that arrives.
Post-launch, the observability stack changes how the team operates. Prompt modifications run through a cost and quality check before being pushed to production. Corpus updates run against the retrieval quality baseline. New use cases are added with their own cost attribution so the team knows immediately what they're contributing to total inference spend. The difference between a team that operates this way and one that doesn't isn't just faster debugging. It's a fundamentally different relationship with the system: the team knows what it's doing rather than finding out from users.
Infrastructure Monitoring Only
Uptime and error rate tracked at the infrastructure level. Latency monitored as a single aggregate metric with no context length breakdown. No structured logging on inputs or outputs. No cost attribution per query. No quality metric sampling. Hard failures are visible. Quality degradation, cost increases from prompt drift, and soft failure patterns are not. Incidents are discovered from user reports. Debugging starts from scratch each time with no historical data. The system's behavior is largely unknown until something breaks badly enough to generate complaints.
Structured LLM Logging
Input/output logging on every inference call, queryable by session, use case, and time range. Latency tracked with input and output token counts. Cost per query calculated and attributed by use case. Failure mode classification on hard and soft failures. Sampled quality evaluation running daily on 2–5% of production volume. Alerting on cost-per-query thresholds, quality score drops, and failure rate changes. When an incident occurs, the data to investigate it already exists. Root cause identification takes hours. Prompt changes and corpus updates have a quality and cost check before production.
Full LLMOps Observability
Complete observability dashboard showing cost, latency, quality, and failure rates by use case and time period, updated in near-real time. Anomaly detection that alerts on deviations from rolling baselines rather than static thresholds. User session traces that connect individual queries to the retrieval chunks and model outputs that produced the response. Automatic flagging of sessions with quality scores below threshold for human review. Weekly observability report surfacing cost trends, quality trends, and the specific query types or prompts contributing to each. Incident response playbooks that map alert types to investigation steps and known fixes.
LLM Observability Readiness Checklist
LLM Observability Checklist
“The teams that debug LLM incidents in hours don't have better engineers. They have better data. They built the observability before the incident. The data was waiting when the problem arrived.”
What to Do This Week
01 Audit what your current LLM logging captures
Pull a sample of 10 production requests from the past week. For each one, answer: do you have the input token count, output token count, latency breakdown by component, use case attribution, and cost? If any of these are missing for most requests, that's the gap to close first. Add structured logging that captures the minimum viable record set for every inference call. This takes one to two days to implement properly, including schema design, pipeline instrumentation, and a query interface. Once it's in place, every future investigation starts with data that already exists rather than manual reconstruction of what might have happened.
02 Calculate your current cost per query by use case
If you have input and output token counts in your logs, calculate the cost per request for the past 30 days using the model's pricing. Aggregate by use case and generate a distribution: median cost, 90th percentile, and the 10 most expensive individual requests. This calculation often produces the first clear picture of where inference spend is concentrated. In most enterprise systems, two or three use cases account for 60–70% of inference cost, and at least one of those is more expensive than it needs to be because of prompt length or context window configuration that was never revisited after the initial build. The cost distribution points directly to the optimization targets.
03 Set up quality sampling on this week's production traffic
Take the evaluation logic built during pre-launch testing and run it against a random 5% sample of this week's production inference calls. Compare the scores to the pre-launch baseline. If scores have dropped since launch, the degradation is already in the data: sort failed samples by score, look at the input distribution for the lowest-scoring requests, and identify whether the failure pattern is a specific query type, a retrieval gap, or a prompt drift issue. This is the first time most teams running this exercise have a quantitative picture of how production quality compares to the evaluation baseline. The gap between those two numbers determines the priority of the next engineering sprint.
04 Define failure categories and write the classification logic
List every way an inference call can fail in your specific system: model API timeout, content policy block, empty output, output below minimum length, output that fails JSON parsing if structured output is expected, retrieval failure returning zero chunks, retrieval failure returning irrelevant chunks. Write a classification function that assigns each failed request to one of these categories. Run it against the past 30 days of failure logs. The category distribution tells you which failure types are most common and which system layers are producing them. Classification that takes an hour to build today eliminates hours of manual triage during the first significant incident this system experiences.
Let 10decoders build your LLM observability stack
We implement LLMOps observability for enterprise teams: structured logging, quality sampling, cost attribution, failure classification, and alerting in a two-week engagement. Your team operates the stack independently from day one.