Observability: Logs, Metrics, and Traces That Earn Their Cost

Observability is the ability to answer questions about a running system that nobody thought to ask before it broke. Three data types carry almost all of the answers. Metrics are cheap numbers over time: they tell you that something is wrong and how bad it is. Traces follow one request across every service it touched: they tell you where the time or the failure came from. Logs are the detailed record of what a particular piece of code did: they tell you why. Each has a different cost curve, and most absurd observability bills come from paying log prices for questions a counter could have answered.

The plan that earns its cost fits in one sentence. Instrument once with OpenTelemetry, send everything through a Collector you control, emit one structured event per request instead of eight scattered lines, keep metric labels bounded, sample traces on the tail, and alert only on symptoms measured against a service level objective. The rest of this guide is the working version of that sentence.

This is written for engineers who already ship a service and now have to answer for it at 2 a.m. If your system is a set of containers on a cluster, pair this with the Kubernetes guide; if requests cross several services, the failure patterns in the distributed systems guide are the ones you will be diagnosing.

What Observability Actually Means

Monitoring watches failure modes you already knew about: one dashboard per known problem, one alert per known threshold. Observability is whether the telemetry you already collect can answer a question nobody wrote down in advance. The practical test is your last incident. If you answered it with data already flowing, that was observability. If you added a log line and redeployed to find out, that was monitoring.

What makes the difference is detail per event. A counter reading "requests failed: 412" cannot tell you all 412 came from one tenant on one build in one region. An event carrying tenant, route, build, region, and duration can. That is the argument for high-cardinality attributes, and also why costs get away from teams: the fields that make data useful are the fields that make it expensive.

Three checklists have survived long enough to be worth using. The four golden signals from Google's SRE book are latency, traffic, errors, and saturation. The RED method, named by Tom Wilkie, covers request-driven services with rate, errors, and duration. The USE method, from Brendan Gregg, covers resources with utilization, saturation, and errors. RED for your services, USE for the machines under them, and you have covered most of what a dashboard needs on day one.

The Three Signals and What Each One Answers

Metrics are aggregates: cheap to store, cheap to query over months, and useless for explaining a single request. Traces are per-request records of a call chain: expensive per unit, kept for days, and the only signal that shows where time went. Logs are per-event records with arbitrary detail: the most flexible and the most expensive per byte.

The value comes from correlation, not from any one signal. A metric alert fires, you jump to traces from that minute, one span is slow, and its logs carry the trace ID so you land on the five lines that matter. In practice that means one thing: every log line and every metric exemplar carries the same trace ID the span does. Skip it and you own three disconnected products.

Structured Logging: One Wide Event Per Request

Logs stop being expensive noise when two rules hold. First, every line is a JSON object with stable field names, never a sentence with values glued into it. Second, a request produces one wide event with everything you would want during an incident, instead of eight thin lines you have to reassemble by timestamp. That pattern is often called a canonical log line, and it is the single highest-return change most teams can make.

import json, logging, sys
from opentelemetry import trace

class JsonFormatter(logging.Formatter):
    def format(self, record):
        ctx = trace.get_current_span().get_span_context()
        event = {
            "ts": self.formatTime(record, "%Y-%m-%dT%H:%M:%S%z"),
            "level": record.levelname,
            "logger": record.name,
            "msg": record.getMessage(),
        }
        # This is the join key between logs and traces.
        if ctx.is_valid:
            event["trace_id"] = format(ctx.trace_id, "032x")
            event["span_id"] = format(ctx.span_id, "016x")
        if record.exc_info:
            event["error"] = self.formatException(record.exc_info)
        event.update(getattr(record, "fields", {}))
        return json.dumps(event, separators=(",", ":"))

handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JsonFormatter())
logging.basicConfig(level=logging.INFO, handlers=[handler])

log = logging.getLogger("checkout")

log.info("request", extra={"fields": {
    "route": "/orders/{id}",     # the template, never the raw path
    "method": "POST",
    "status": 201,
    "duration_ms": 84,
    "db_ms": 31,
    "cache_hit": False,
    "tenant": "acme",          # high cardinality is fine in logs
    "build": "1.8.3",
    "retries": 0,
}})

Equivalents exist everywhere: structlog in Python, pino in Node, log/slog in the Go standard library, Logback with a JSON encoder in Java. The library matters far less than the discipline of stable field names, because every saved search breaks the day someone renames duration_ms to latency.

Metrics and the Cardinality Bill

A time series is one metric name plus one exact combination of label values. Storage cost, query speed, and memory pressure all scale with how many of those combinations are active. Adding a label does not add one series, it multiplies the count by the number of distinct values that label takes.

Here is the arithmetic that surprises people. A latency histogram with ten explicit buckets emits thirteen series per label combination: one per bucket, one for the catch-all bucket, plus a sum and a count. Forty routes times four HTTP methods times six status codes is 960 combinations, so that one metric is roughly 12,480 active series. Now add a customer_id label with 5,000 values and the same metric becomes about 62 million series. Nothing warns you. The bill does, next month.

from opentelemetry import metrics

meter = metrics.get_meter("checkout")

order_duration = meter.create_histogram(
    "checkout.order.duration",
    unit="s",                      # seconds, per semantic conventions
    description="Time to accept an order",
)
orders = meter.create_counter("checkout.orders", unit="{order}")

def record_order(route, status_code, tenant_tier, seconds):
    attrs = {
        "http.route": route,                # template, ~40 values
        "http.response.status_code": status_code,
        "tenant.tier": tenant_tier,        # free / pro / gov, 3 values
    }
    order_duration.record(seconds, attrs)
    orders.add(1, attrs)

Three details save real money later. Record latency as a histogram, never an average, because an average hides exactly the tail you care about. Attach exemplars so a histogram bucket carries a trace ID and you can click from a slow bucket straight into a slow request. And pick bucket boundaries around your latency objective rather than accepting defaults, since a histogram whose buckets are all far from your threshold cannot measure compliance with it. Prometheus 3.x can also store native histograms with far higher resolution, still behind a feature flag, which is worth testing before you hand-tune bucket lists.

Traces: Where the Time Went

A span is one unit of work with a start time, a duration, attributes, and a parent. A trace is the tree of spans belonging to one request. Context propagation is what keeps the tree connected across process boundaries, and on HTTP it travels in a single W3C header: traceparent: 00-{32-hex trace id}-{16-hex span id}-01, where the final byte says whether this trace was sampled.

Automatic instrumentation already creates spans for inbound HTTP, outbound HTTP, database calls, and queue operations. What it cannot know is your business boundaries, so add spans by hand only where a name would mean something in an incident review: price_basket, authorize_payment, render_invoice. A trace with 400 spans named after functions is harder to read than one with fifteen named after decisions.

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("checkout")

def place_order(order):
    with tracer.start_as_current_span("place_order") as span:
        span.set_attribute("order.item_count", len(order.items))
        span.set_attribute("order.tenant", order.tenant)   # fine on a span
        try:
            authorize_payment(order)          # child spans attach here
        except PaymentDeclined as exc:
            span.set_status(Status(StatusCode.ERROR, "declined"))
            span.record_exception(exc)        # stack trace onto the span
            raise
        span.add_event("order.accepted", {"order.id": order.id})

The gap that trips everyone is asynchronous work. HTTP clients inject traceparent for you; queues do not. Inject the current context into message headers when you publish, and extract it before starting the span when you consume. Miss this and every background job becomes an orphan trace, the most common reason a tracing rollout feels useless after week one.

Instrumenting with OpenTelemetry

OpenTelemetry is a vendor-neutral API, a set of SDKs, and OTLP, the protocol they all speak. Its practical value is that instrumentation stops being a vendor decision. You emit OTLP, and switching backends becomes a Collector config change instead of a library migration across every service.

Start with zero-code instrumentation. In Python that is two commands and a handful of environment variables, and it will produce HTTP, database, and client spans for a typical FastAPI service without touching application code.

pip install "opentelemetry-distro[otlp]"
opentelemetry-bootstrap -a install   # installs matching instrumentation libs

export OTEL_SERVICE_NAME=checkout-api
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_RESOURCE_ATTRIBUTES=service.version=1.8.3,deployment.environment.name=prod
export OTEL_TRACES_SAMPLER=parentbased_always_on   # sample in the Collector instead
export OTEL_LOGS_EXPORTER=otlp

opentelemetry-instrument uvicorn app:app --host 0.0.0.0 --port 8080

# Node.js (CommonJS). For ESM, swap --require for --import.
npm install @opentelemetry/api @opentelemetry/auto-instrumentations-node
OTEL_SERVICE_NAME=checkout-web \
  node --require @opentelemetry/auto-instrumentations-node/register server.js

# Java: one agent flag, no code changes
java -javaagent:opentelemetry-javaagent.jar -jar app.jar

# Go has no runtime agent; wire the SDK in main() and use
# otelhttp / otelsql wrappers around your handlers and drivers.

One thing to know before you build dashboards on the output: semantic conventions define the standard attribute names, and the stable HTTP set uses http.request.method, http.response.status_code, http.route, and a server duration metric in seconds. Older instrumentation emitted http.method and milliseconds, so a mixed fleet quietly splits one dashboard into two half-empty ones. Pin instrumentation versions and roll them forward together, like any other CI-managed dependency.

The Collector: One Place to Change Your Mind

The OpenTelemetry Collector receives telemetry, transforms it, and exports it somewhere. Applications point at the Collector and know nothing about the backend. That single indirection is where redaction, batching, sampling, enrichment, and vendor routing live, and all of it changes without redeploying a service.

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  memory_limiter:              # MUST be first: sheds load instead of dying
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  resourcedetection:
    detectors: [env, system]     # adds host, cloud, and k8s attributes
  transform/redact:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          - delete_key(attributes, "http.request.header.authorization")
          - delete_key(attributes, "db.query.parameter")
  batch:                       # last before the exporters
    timeout: 5s
    send_batch_size: 8192

exporters:
  otlphttp/backend:
    endpoint: https://otlp.your-backend.example
    headers: { "x-api-key": "${env:BACKEND_KEY}" }
  prometheus:
    endpoint: 0.0.0.0:8889   # scrape target for a local Prometheus

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, transform/redact, batch]
      exporters: [otlphttp/backend]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [prometheus, otlphttp/backend]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/backend]

Most production deployments run two tiers. An agent sits next to the workload, one per host or as a sidecar, doing cheap local work: receiving, adding host attributes, batching. A gateway is a separately scaled deployment doing the expensive stateful work: tail sampling, heavy transforms, fan-out to more than one backend. Order inside a pipeline is not cosmetic. Put memory_limiter first so an overloaded Collector drops data instead of being killed, and put batch last so everything downstream ships in efficient chunks.

Sampling: Head, Tail, and What You Lose

Head sampling decides at the first span, before anything has happened, so a 1% head sample throws away 99% of your errors along with 99% of the boring successes. Tail sampling waits until the trace is complete and decides with the outcome in hand: keep every error, keep everything slow, keep a small percentage of the rest.

# --- On the gateway tier ---
processors:
  tail_sampling:
    decision_wait: 10s          # hold spans this long before deciding
    num_traces: 100000          # in-memory trace buffer; watch RAM
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 2 }

# --- On the agent tier: all spans of one trace must reach ---
# --- the SAME gateway pod, or the decision sees half a trace ---
exporters:
  loadbalancing:
    routing_key: traceID
    protocol: { otlp: { tls: { insecure: true } } }
    resolver:
      dns: { hostname: otel-gateway.observability.svc.cluster.local }

Tail sampling is not free. The gateway buffers every span of every in-flight trace for the length of decision_wait, so memory scales with span rate times that window, and a trace that takes longer than the window gets judged on what arrived in time. It also breaks naive counting: if you sample 2% of successes and 100% of errors, computing an error rate from stored spans gives a wildly wrong answer. Compute rates from metrics, which are unsampled, and use traces for explanation only.

Logs deserve the same treatment. Keep everything at WARN and above, keep INFO for requests that were sampled in or that failed, and sample the rest. The pairing to avoid is a trace you kept next to logs you dropped, which leaves you looking at a slow span with nothing inside it.

SLOs and Error Budgets

An SLI is a ratio of good events to valid events, such as non-5xx responses over all responses, or requests served under 300 ms over all requests. An SLO is a target for that ratio over a window, usually 28 or 30 days. The error budget is everything the target leaves over, and it converts an argument about reliability into arithmetic.

Two habits make objectives useful rather than decorative. Measure what a user notices, usually the success and latency of one endpoint, not the uptime of a machine. And never target 100%, because a perfect target makes every change a risk with no allowance, which in practice means nobody ships. The budget exists to be spent.

Burn rate is how fast you are spending it, expressed as a multiple of the sustainable pace. Burn rate 1 spends the whole budget exactly at the end of the window. Burn rate 14.4 spends it in about two days, which is why an hour at that rate deserves a phone call. The arithmetic is simple: burn rate times window length divided by 720 hours gives the fraction of a 30-day budget consumed. 14.4 times 1 hour over 720 is 2%. 6 times 6 hours over 720 is 5%. 3 times 24 over 720 is 10%.

Alerting People Do Not Learn to Ignore

A page must be urgent, novel, and something a human can act on right now. If it is none of those, it is a ticket or a dashboard panel. The fastest way to make an on-call rotation useless is a stream of alerts that are usually noise, because the response to a paging system that cries wolf is not investigation, it is a snooze.

Alert on symptoms, not causes. High CPU is not an incident; a checkout endpoint failing 3% of requests is. Cause-based alerts multiply as your system grows, and every one of them fires during incidents they did not cause. One symptom alert per user-facing objective covers a hundred causes, including the ones nobody predicted.

groups:
  - name: slo-checkout
    rules:
      # Failure ratio, recorded at two window lengths
      - record: checkout:errors:ratio5m
        expr: |
          sum(rate(http_server_request_duration_seconds_count{
                job="checkout",http_response_status_code=~"5.."}[5m]))
          /
          sum(rate(http_server_request_duration_seconds_count{job="checkout"}[5m]))
      - record: checkout:errors:ratio1h
        expr: |
          sum(rate(http_server_request_duration_seconds_count{
                job="checkout",http_response_status_code=~"5.."}[1h]))
          /
          sum(rate(http_server_request_duration_seconds_count{job="checkout"}[1h]))

      # 14.4x burn on BOTH windows: 2% of a 30-day budget in one hour.
      # The short window makes the alert stop quickly once it recovers.
      - alert: CheckoutBudgetBurningFast
        expr: |
          checkout:errors:ratio1h > (14.4 * 0.001)
          and
          checkout:errors:ratio5m > (14.4 * 0.001)
        for: 2m
        labels: { severity: page }
        annotations:
          summary: "Checkout is burning error budget 14.4x too fast"
          runbook: "https://runbooks.internal/checkout-slo"

That shape, popularized by the alerting chapter of Google's SRE Workbook, uses a long window to decide that something is really wrong and a short window to notice recovery, so alerts resolve on their own instead of hanging around after the fix. The usual ladder is 14.4 over an hour and 6 over six hours as pages, 3 over a day and 1 over three days as tickets. Four rules per objective replace dozens of threshold alerts.

The Failure Modes You Will Actually Hit

Each of these shows up in the first month of a rollout, and each has a signature you learn once. Check the Collector before you suspect the SDK: a debug exporter prints exactly what arrived, which settles most arguments in a minute.

Cutting the Bill Without Going Blind

Vendors bill on gigabytes ingested, unique time series, indexed fields, span counts, or a blend. The units differ; the levers are the same four.

Do the deletion at the Collector, not in application code. A filter or transform processor takes effect on a config reload, so you can try a cut, watch what breaks, and put it back in minutes. The same discipline that governs cloud spend generally applies here, with one extra rule: never cut anything an alert depends on without checking that alert first.

Observability Tools Compared, Honestly

OpenTelemetry has made the choice less permanent than it used to be. Since your applications emit OTLP either way, switching backends is a Collector change; what actually locks you in is the dashboards, alert rules, and query language your team has learned.

A defensible default for a small team: OpenTelemetry in the applications, a Collector you run, managed storage for the first year, and a written note of what moving would take. Revisit when a week of engineering time costs less than a month of the bill.

When Not to Build Any of This

Observability tooling can absorb an unlimited amount of engineering attention. The question is not whether the data would be nice to have, it is whether anyone will look at it.

The Short Version

The bottom line: pick one wire format and use it everywhere, so the backend stays a decision you can revisit. Log one wide JSON event per request with the trace ID in it. Keep metric labels to values you could list on a napkin, and push the interesting detail onto spans. Sample traces after you know how they ended. Write two or three objectives that describe what users actually feel, and let burn-rate alerts be the only thing that wakes anyone up. Then, every month, delete something nobody read. Observability that earns its cost is mostly a habit of subtraction.

Frequently Asked Questions

What is the difference between monitoring and observability?

Monitoring watches failure modes you already knew about: a dashboard per known problem, an alert per known threshold. Observability is whether your existing telemetry can answer a question nobody wrote down in advance, such as why checkout is slow only for one tenant on one build. The practical test is your last incident. If you answered it with data already flowing, that was observability. If you added a log line and redeployed to find out, that was monitoring. In practice the difference comes down to whether your events carry enough detail, such as tenant, route, build, and region, to slice a problem you did not predict.

Do I really need all three of logs, metrics, and traces?

Not on day one, and not in equal measure. A small service gets most of the value from structured logs plus a handful of metrics. Metrics tell you something is wrong and how bad. Logs tell you what happened inside one request. Traces earn their keep once a request crosses three or more services and the question shifts from what broke to where the time went. The mistake is buying all three at full fidelity before you know which questions you keep asking, and paying log prices for questions a counter could answer. The microservices guide covers the point where that shift usually happens.

What is a good sampling rate for traces?

There is no single number, because the rate is not the decision that matters. Keep every trace containing an error and every trace slower than your latency target, then keep a small percentage of the rest as a baseline, commonly one to five percent. That is tail sampling, decided in the Collector once the spans have arrived, and it is why fixed head sampling disappoints: a 1% head sample discards 99% of your errors too. Tail sampling costs memory, and it requires every span of one trace to reach the same Collector instance, which is what the load-balancing exporter is for.

How do I reduce my observability bill without going blind?

Four levers, in order of payoff. Cardinality first: find metric labels with unbounded values, such as user ID, request ID, or raw URL path, and remove them or move that detail onto spans. Retention second: keep raw logs hot for one or two weeks, then archive to object storage instead of paying indexed prices for a year. Sampling third: drop repetitive successful traces and DEBUG logs while keeping every error. The audit last: for every metric and log stream, name the dashboard or alert that reads it, and drop what nobody can name. Do all of it at the Collector so no application has to be redeployed.

Reference documentation: OpenTelemetry Docs, Semantic Conventions, Prometheus Docs, Google SRE Workbook: Alerting on SLOs