LLM observability: what to monitor when the output is text

Why the dashboards stay green

Uptime is 99.9%. The p99 latency chart is flat. The error rate is a rounding error. And the support queue is filling with people saying the assistant told them something that was not true.

Nothing in the monitoring stack is broken. It is measuring a different system than the one users are complaining about. The four golden signals — latency, traffic, errors, saturation — come from the chapter on monitoring distributed systems in Google's Site Reliability Engineering book, and they are still the right first layer. They rest on an assumption that does not hold here: that a response is either correct or an error.

An LLM feature fails in ways that return HTTP 200 inside the latency budget. It confabulates — the term NIST uses in the Generative AI Profile (NIST AI 600-1), its July 2024 companion to the AI Risk Management Framework. It returns prose where the caller expected JSON. It refuses a legitimate request, or answers one it should have refused. It gets truncated at the token limit and stops mid-sentence, with a finish reason nobody logged. Retrieval comes back empty and the model answers from parametric memory anyway. A tool call is emitted with the right name and the wrong arguments. Several of these have their own entries in the OWASP Top 10 for LLM Applications (2025) — prompt injection at LLM01, sensitive information disclosure at LLM02, excessive agency at LLM06, unbounded consumption at LLM10.

Keep the golden signals. They catch outages, and outages still happen. But they are a floor, not a ceiling, and no amount of tuning will make them see content.

What to log on every model call

Start by fixing the unit of observation. One user request might be a retrieval, three model calls, two tool executions, and a summarization pass. If your telemetry has one span for the whole thing, every question you will actually ask — which step is slow, which step is wrong, which retry succeeded — is unanswerable. Instrument the call, not the endpoint.

Do not invent field names. The OpenTelemetry GenAI semantic conventions already define them: gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, and gen_ai.response.finish_reasons. On the metrics side there is gen_ai.client.operation.duration as a latency histogram and gen_ai.client.token.usage filterable by gen_ai.token.type. Message content lives in gen_ai.system_instructions, gen_ai.input.messages, and gen_ai.output.messages — and is not captured by default, because prompts and tool arguments routinely contain sensitive data. That is an explicit opt-in, and it is a good default to leave alone until you have decided on redaction. Note also that these conventions are still under active development rather than marked stable, so pin your collector version and expect field renames rather than being surprised by one.

The conventions give you the model-facing half. The half that actually explains your failures, you have to add yourself:

• Prompt template ID and version. The single highest-value custom field. Without it, "quality dropped on Tuesday" cannot be joined to "we shipped a prompt edit on Tuesday."
• Retrieval detail — the document IDs returned, their scores, and how many came back. An empty or low-scoring retrieval that still produced a confident answer is your most common silent failure.
• Tool call outcome — the tool name, whether it actually executed, and its exit status. A tool the model described but never invoked looks identical to a successful one in most logs.
• Configuration hash — sampling parameters, max tokens, system prompt hash, retrieval index version. Anything that changes behavior belongs in one hashed field you can group by.
• The downstream human signal — did the user retry within a few seconds, edit the output, copy it, rate it, or escalate to a person. This is the closest thing to ground truth you will get for free.

One detail worth being pedantic about: log gen_ai.response.model, not just what you requested. If your code requests a floating alias, the alias resolves to a specific version, and you need to know which. Providers retire models on published schedules — Anthropic's model deprecations page defines an active / legacy / deprecated / retired lifecycle, commits to at least 60 days' notice before retiring a publicly released model, names a recommended replacement for each, and offers a Console usage export broken down by API key and model. That export is only useful if your own telemetry can tell you which of your features route to which model.

How to sample for human review

You cannot read every conversation, and you should not try. But purely random sampling spends most of a reviewer's scarce attention on the boring middle. Stratify instead, into three buckets that answer different questions.

Signal-triggered. Everything where an automatic tripwire fired: finish reason was the length limit, a tool errored, retrieval returned nothing, the output matched a refusal pattern, latency was an outlier, the user retried within a few seconds, or someone hit thumbs-down. These are for diagnosis. They tell you what breaks, not how often.

Random baseline. A plain uniform sample of production traffic. This is the only stratum that can estimate a rate, and mixing signal-triggered items into it will inflate whatever number you report. Keep the two accounted separately, always.

Novelty. Inputs unlike anything you have seen before. Embedding distance from a running centroid works; so does an unrecognized intent label. New input shapes are where the next class of failure shows up first.

All of this argues for deciding what to keep after you know how the call turned out, not before — buffer the trace, then decide. That is the tail-based sampling pattern: retain every trace with an error, plus a percentage of the clean ones.

Now the part that gets skipped. Sample size is not a formality; it determines what your review can and cannot tell you. As an illustrative calculation using the standard Wilson score interval: review 100 random calls, find 3 failures, and the true failure rate sits somewhere between roughly 1.0% and 8.5% at 95% confidence. Take the same 3% observed rate out to 500 calls and the interval tightens to about 1.8% to 4.9%. Read that again in operational terms — a 100-sample weekly review cannot distinguish a 2% failure rate from a 6% one. Two consequences follow. Stop reacting to week-over-week movement inside your interval; it is noise wearing a trend line. And if you genuinely need finer resolution, either sample considerably more or narrow the population you are measuring to something higher-frequency.

Finally: write the rubric before the review, not during it. Have two reviewers grade an overlapping subset and measure how often they agree with each other. If humans disagree about what counts as a good answer, every automated metric built on their labels inherits that disagreement — and you will spend months tuning a number that was never stable.

Catching regression without labels

Most teams do not have a labelled test set, will not get one soon, and still need to know whether today's change made things worse. Four techniques, cheapest first, and they compose.

1. A frozen replay set. Pull a few hundred real inputs from production logs, freeze them, and replay them on every change. You do not need labels for this — you need diffs. Most changes should not change most outputs. If a one-line system prompt tweak alters 60% of responses, that is a finding, whatever the quality verdict turns out to be. Diff rate is a change-detector you can build in an afternoon.

2. Proxy signals you already collect. Refusal rate, mean output length, finish-reason distribution, retrieval hit rate, tool-call success rate, user retry rate, escalation rate. None of these measures quality. All of them move when quality breaks. Alert on distribution shift rather than absolute value: a refusal rate that jumps from 2% to 9% is a page, even if nobody agreed in advance what the right number was.

3. Pairwise judging on the replay set. Show a judge model the old output and the new one for the same input, and ask which better satisfies the request. Pairwise comparison is easier and more stable than absolute scoring. In Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., NeurIPS 2023), a strong judge model reached over 80% agreement with human preferences — roughly the level at which humans agree with each other — while the same paper documented position bias, verbosity bias, and self-enhancement bias. That translates directly into three rules. Randomize presentation order and run each pair both ways. Treat any win that coincides with longer outputs as suspect, because a verbosity-biased judge will happily reward a change that only made responses wordier. And avoid using the same model family as both judge and candidate where you can. Then pin the judge model version and the judge prompt, and version them like code — a judge that silently upgrades underneath you is a ruler that changes length between measurements.

4. Measure your noise floor first. Before you attribute a diff to your change, find out how much output varies when nothing changes. Replay the same inputs twice with an identical configuration and count the differences. That number is your floor, and any regression signal smaller than it is unreadable. It is also not zero even at temperature zero: Thinking Machines Lab's Defeating Nondeterminism in LLM Inference traces the residual variation to kernels whose floating-point results depend on batch size, which means output can shift with concurrent server load rather than with anything you did.

What goes wrong

The failure modes below are the ones that actually cost teams time, in rough order of how often they show up.

• Turning on content capture before deciding on redaction. Prompts carry whatever the user typed — account numbers, health details, controlled information. The GenAI conventions default content capture to off for exactly this reason, and OWASP lists sensitive information disclosure as LLM02. Decide the redaction and retention policy first, then flip the switch.
• An eval suite where everything passes. A 100% pass rate means the suite is too easy, not that the system is good. A useful suite is mined from production failures and sits somewhere uncomfortable.
• Alerting on averages. Text-quality failures are tail events. A mean score barely moves when 4% of answers become dangerous. Alert on the rate of specific bad events, not on a composite quality score.
• Monitoring the model instead of the system. In practice, most regressions trace to retrieval, chunking, tool schemas, or a prompt template edit — not to the model weights. Instrument the pipeline end to end or you will keep blaming the wrong component.
• No path from production back into the eval set. Every confirmed failure a reviewer finds should become a permanent test case within the week. Without that loop, you re-discover the same bug every quarter.

Where to start

If you have nothing today, this order gets you the most signal for the least work. Instrument every model call with the OpenTelemetry GenAI conventions, content capture off. Add prompt template version, retrieval detail, and tool outcome as custom attributes. Wire up one downstream human signal — the retry-within-N-seconds heuristic is usually the cheapest. Freeze 200 real inputs as a replay set. Start a weekly stratified review with a written rubric and two reviewers. Only then add an LLM judge, and only after you have measured your nondeterminism floor.

None of this is exotic. It is the same discipline as any other production telemetry — instrument, sample deliberately, alert on rates, close the loop. The difference is that the correctness signal is not in the transport layer, so you have to build it.

Sources: Google — Site Reliability Engineering, Ch. 6, Monitoring Distributed Systems; OpenTelemetry — GenAI observability and the GenAI semantic conventions; Zheng et al. — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (NeurIPS 2023); Thinking Machines Lab — Defeating Nondeterminism in LLM Inference; OWASP — Top 10 for LLM Applications 2025; NIST AI 600-1 — Generative AI Profile; Anthropic — Model deprecations. Confidence-interval figures are illustrative calculations using the Wilson score interval. Analysis and framing by Precision AI Academy.

Common questions

Why don't normal monitoring metrics catch LLM failures? Latency, traffic, errors and saturation assume a response is either correct or an error. An LLM that returns a confident wrong answer returns HTTP 200 in normal latency. The request succeeded; only the content failed, and no infrastructure metric can see content.

What should you log on every LLM call? The OpenTelemetry GenAI semantic conventions define the standard fields — gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model, token usage, and finish reasons, plus duration and token-usage metrics. Content capture is opt-in and off by default. Add your own fields for prompt template version, retrieval document IDs, tool call outcomes, and the downstream user action.

How can you detect quality regression without a labelled test set? Freeze a few hundred real production inputs as a replay set and diff outputs across changes — diffs need no labels. Watch proxy signals that move when quality breaks: refusal rate, output length, finish-reason mix, retrieval hit rate, tool-call success, and user retry rate. Then add pairwise LLM judging, with order randomized to control for position bias.

How many samples do you need for human review? More than most teams use. As an illustrative calculation with the Wilson score interval, 3 failures in 100 random calls puts the true rate between roughly 1.0% and 8.5% at 95% confidence; the same 3% rate over 500 calls narrows to about 1.8% to 4.9%. A 100-sample weekly review cannot distinguish a 2% failure rate from a 6% one.