Skip to content

Observability with OpenTelemetry — Traces, Metrics & Logs

Last reviewed: 2026-09-17

Purpose: A vendor-neutral guide to instrumenting applications and building an observability pipeline with OpenTelemetry — the data model, the SDK and Collector, backend choices, sampling, and production practice.

Contents

Overview

Observability is the ability to answer arbitrary questions about a running system from the data it emits — without shipping new code to find out. Teams typically answer those questions with three signals:

Signal Answers Typical cost driver
Traces Where did this request spend its time? Which service failed? Span volume, retention
Metrics How much / how many / how fast, over time? Active series (cardinality)
Logs What exactly happened in this code path? Ingest volume, retention

Before OpenTelemetry, every backend shipped its own agent, its own SDK and its own wire format. Moving to a different vendor meant re-instrumenting every service — so instrumentation was effectively a permanent lock-in.

OpenTelemetry (OTel) removes that coupling. It is a CNCF project that standardises:

  • the API your code calls (a thin, vendor-neutral interface),
  • the SDK that implements it,
  • the wire protocol (OTLP) used to ship telemetry,
  • the Collector that receives, processes and routes it,
  • semantic conventions — a shared vocabulary of attribute names.

Instrument once against OTel, and the backend (Prometheus, Grafana, Jaeger, Tempo, Loki, Elastic, or a SaaS vendor) becomes a deployment choice rather than a rewrite.

[!NOTE] OTel is not a storage, query or dashboard product. It is the instrumentation and transport layer. You still need a backend to store and visualise the data — and a sampling strategy to keep the bill sane.

The OpenTelemetry Data Model

Traces and spans

A trace is the end-to-end journey of one request. It is a tree of spans; each span represents one unit of work with:

Field Meaning
trace_id Shared by every span in the request
span_id / parent_span_id Position in the tree
name Operation, e.g. GET /orders/{id}
kind server, client, producer, consumer, internal
start_time / end_time Duration, and where the time went
status OK, ERROR (plus an error message)
attributes Key/value detail (HTTP method, DB statement, peer service)
events Timestamped annotations inside the span
links Causality across traces — e.g. a batch consumer back to its producers

Span context propagates between services over the wire (W3C traceparent / tracestate headers are the default). Break propagation — a queue consumer that drops headers, a proxy that strips them — and your traces fragment.

Metrics

Metrics follow the metric / instrument / measurement model:

Instrument Behaviour Use for
Counter Monotonic sum Requests, errors, bytes
UpDownCounter Sum that can decrease Queue depth, in-flight requests
Histogram Distribution in buckets Latency percentiles, payload sizes
Gauge Point-in-time value CPU %, temperature, pool size

Attributes turn a metric into multiple time series — powerful, and the single most common way to blow up a backend. See Production Practice.

Logs

OTel defines a log record model (timestamp, severity, body, attributes, trace context). The highest-value part is correlation: when a log record carries trace_id/span_id, a log viewer can jump straight from a stack trace to the trace and back.

Beyond the three pillars

  • Baggage — arbitrary key/values propagated with the context across services (tenant id, feature flag). Small and bounded only; it travels on every hop.
  • Resource — describes what emitted the data (service, host, container, cloud region). Set once, applies to every signal.
  • Profiles — continuous CPU/heap profiling as a fourth signal; the specification is newer and less uniformly supported.

Core Concepts

Concept What it is
API Interfaces your code compiles against: Tracer, Meter, Logger. No-op by default, so a library can ship OTel support at zero cost when unused
SDK The implementation: span processors, exporters, samplers, resource detection, metric readers
Resource Immutable attributes describing the emitter — service.name, service.version, deployment.environment, host.name, k8s.pod.name
Exporter Ships data out of the process: OTLP (gRPC or HTTP), Prometheus scrape endpoint, console, or a vendor exporter
Processor Operates on spans/metrics/logs before export: batching, memory limiting, attribute filtering, tail sampling
Context propagation Carries the active span and baggage across function, thread, process and network boundaries
Instrumentation Code that produces telemetry: zero-code/auto (library hooks) or manual (explicit spans, metrics, attributes)
Collector Standalone process/service that receives OTLP, enriches, filters, samples and routes it to one or many backends
OTLP The OpenTelemetry Protocol — the standard OTLP/gRPC (:4317) and OTLP/HTTP (:4318) endpoints
Semantic conventions Standard attribute names (http.response.status_code, db.system, messaging.destination.name) so dashboards and alerts are portable

How the Pieces Fit Together

The canonical pipeline:

  Application                    Collector (agent)            Collector (gateway)        Backends
┌───────────────┐             ┌────────────────────┐        ┌────────────────────┐    ┌──────────────┐
│ OTel SDK      │  OTLP/gRPC  │ receivers          │  OTLP  │ tail sampling      │    │ Prometheus   │
│ auto + manual │ ──────────► │ processors         │ ─────► │ attribute filter   │──► │ Tempo/Jaeger │
│ instrumentation│   :4317    │ exporters (batch)  │        │ fan-out routing     │    │ Loki         │
└───────────────┘             └────────────────────┘        └────────────────────┘    └──────────────┘

Two components, two different jobs:

  • SDK (in-process) — what your code knows: span boundaries, business attributes, metric values.
  • Collector (out-of-process) — what your code should not know: which backend, how much to sample, what to redact, how to retry.

Keeping backend routing in the Collector is what makes the pipeline reconfigurable without redeploying services.

Instrumentation

Zero-code / automatic

The fastest path. Point the runtime at the OTel agent and the SDK instruments the frameworks it knows (HTTP servers and clients, database drivers, messaging libraries).

# Python — e.g. Flask/Django/FastAPI
export OTEL_SERVICE_NAME=checkout-api
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_TRACES_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=otlp
export OTEL_RESOURCE_ATTRIBUTES="service.version=1.4.2,deployment.environment=prod"
opentelemetry-instrument python app.py

# Node.js — load the SDK before the app
export NODE_OPTIONS="--require ./otel.js"
node server.js

# Java — the JVM agent, no code change at all
export JAVA_TOOL_OPTIONS="-javaagent:/opt/opentelemetry-javaagent.jar"

Auto-instrumentation gets you spans, RED metrics and context propagation for free. It cannot know your domain, so the highest-value step is always layering manual attributes on top.

Manual — traces

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def process_order(order_id: str, tenant: str) -> None:
    with tracer.start_as_current_span("order.process") as span:
        # Domain attributes make traces queryable: "all slow spans for tenant X"
        span.set_attribute("order.id", order_id)
        span.set_attribute("tenant.id", tenant)
        try:
            charge(order_id)
            span.set_status(trace.StatusCode.OK)
        except PaymentError as exc:
            span.record_exception(exc)            # keeps the stack trace on the span
            span.set_status(trace.StatusCode.ERROR, "payment failed")
            raise

Manual — metrics

from opentelemetry import metrics

meter = metrics.get_meter(__name__)

orders = meter.create_counter(
    "orders.processed", unit="{order}", description="Orders processed"
)
latency = meter.create_histogram(
    "orders.processing.duration", unit="ms", description="Order processing latency"
)

def on_order(tenant: str, duration_ms: float) -> None:
    # Low-cardinality attributes only — never order_id, user_id or a URL path
    attrs = {"tenant": tenant, "outcome": "success"}
    orders.add(1, attrs)
    latency.record(duration_ms, attrs)

Correlating logs

import logging
from opentelemetry import trace

log = logging.getLogger("checkout")

def pay(order_id: str) -> None:
    ctx = trace.get_current_span().get_span_context()
    log.info(
        "payment authorised",
        extra={"trace_id": format(ctx.trace_id, "032x"),
               "span_id": format(ctx.span_id, "016x"),
               "order_id": order_id},
    )

Once logs carry trace_id, a trace → logs jump (and the reverse) turns an hour of grepping into one click.

The Collector

The Collector is configured with five kinds of component:

Component Role Examples
Receivers Get data in otlp (gRPC + HTTP), prometheus, filelog, jaeger, zipkin
Processors Transform / protect batch, memory_limiter, attributes, filter, resource, tail_sampling, transform
Exporters Send data out otlp, otlphttp, prometheus, prometheusremotewrite, loki, debug
Connectors Join two pipelines spanmetrics (traces → RED metrics), routing, forward
Extensions Process-level services health_check, pprof, zpages, file_storage

Pipelines wire them together per signal:

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  memory_limiter:              # MUST be first — protects the Collector itself
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20
  resource:
    attributes:
      - key: deployment.environment
        value: prod
        action: upsert
  attributes/scrub:
    actions:
      - key: http.request.header.authorization
        action: delete
  batch:                       # MUST be last before exporters
    send_batch_size: 8192
    timeout: 5s

connectors:
  spanmetrics:                 # derive request-rate/error/duration metrics from traces
    dimensions:
      - name: http.response.status_code
      - name: http.route

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls: { insecure: false }
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write
  debug:
    verbosity: basic

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource, attributes/scrub, batch]
      exporters: [otlp/tempo, spanmetrics]
    metrics:
      receivers: [otlp, spanmetrics]
      processors: [memory_limiter, resource, batch]
      exporters: [prometheusremotewrite]
  telemetry:
    logs: { level: info }

Deployment patterns

Pattern Shape When
No Collector SDK → backend directly Single small app, backend accepts OTLP
Agent One Collector per host / node / sidecar Enrich with host metadata, buffer during backend outages
Gateway Central Collector deployment behind a load balancer Tail sampling, redaction, fan-out, credential isolation
Both Agent → gateway → backends The common production shape: local batching, central policy

The two-processor ordering rule above is not cosmetic: without memory_limiter first, a traffic spike can OOM the Collector; without batch last, you pay one network round-trip per span.

Sampling

You cannot afford to keep every trace. Sampling decides what survives — and where you decide matters:

Strategy Where Behaviour
Head sampling SDK or Collector, on span start Cheap, no buffering, but decides before it knows if the request fails
Tail sampling Collector, after the trace completes Can keep all errors and all slow traces — needs a gateway that sees the whole trace, plus memory

Sensible defaults:

  • Head-sample to a few percent for baseline traffic.
  • Tail-sample in the gateway: errors → 100%, latency > p99 → 100%, everything else → 1–5%.
  • Always keep a rate floor (e.g. 10 traces/s per service) so low-traffic services stay observable.
  • Sample consistently across services for a given trace, or you get trees with missing branches.
  • Metrics are not sampled — that is exactly why they are the right signal for SLIs and alerting.

Backends, Storage and Dashboards

A typical open-source split, and the reason OTel's neutrality matters:

Backend Signal Notes
Prometheus Metrics Scrape or remote-write; the de-facto query language (PromQL) and alerting rule format
Grafana All Single UI over Prometheus, Tempo, Loki, Elasticsearch and SQL sources
Tempo / Jaeger Traces Jaeger offers a UI and collector; Tempo is object-storage-native and cheap at scale
Loki / Elasticsearch Logs Label-indexed vs full-text; pick by query pattern and budget
Alertmanager Alert routing Deduplication, grouping, silences, inhibition on top of Prometheus rules

Pragmatic guidance:

  • Metrics → Prometheus. Keep retention short (15–30 days) and downsample long-term.
  • Traces → object storage. Trace volume dwarfs metric volume; a local SSD-backed store gets expensive fast.
  • Logs → wherever you can afford. Logs are the biggest bill; filter and drop before storage, not after.
  • One Grafana. Correlating a latency spike → traces → logs in one UI is the entire payoff of doing this.

For service-level objectives, define SLIs on metrics (availability = non-5xx / total; latency = percentile of a request-duration histogram) and alert on multi-window burn rate rather than raw thresholds — it catches real degradation without paging on a single blip.

Production Practice

Resource attributes are non-negotiable. At minimum set service.name, service.version and deployment.environment. Without them, data from staging and prod blend together and no dashboard can be scoped.

Control cardinality deliberately. Active series is the metric cost unit. Design your attribute sets:

Keep as an attribute Do not use as an attribute
http.route (/orders/{id}) http.url / raw path with IDs
http.response.status_code full user-agent strings
tenant.tier (free, pro, ent) tenant.id, user.id
db.system, db.operation raw SQL statements

Never put secrets or PII in telemetry. Redact at the Collector (attributes processor, delete authorization, cookie, and anything matching a token pattern) and again at the backend where possible.

Keep telemetry off the request path. Exporters are asynchronous and batched; a synchronous exporter that blocks on a slow backend will take your service down with it.

Set resource limits. Give the Collector explicit CPU and memory limits, keep memory_limiter first in every pipeline, and watch otelcol_processor_dropped_spans / ..._refused_spans.

Version and review your telemetry like code. Collector config in Git, semantic-convention updates reviewed, and alerts owned by the service team.

Troubleshooting

Symptom Likely cause Check
No spans at all Exporter never flushed; wrong endpoint; TLS mismatch Enable the debug exporter, test the OTLP port, verify scheme (grpc vs http)
Spans arrive but traces are fragmented Context propagation broken — queue, proxy or async boundary drops headers Confirm traceparent survives each hop; check the messaging/client library is instrumented
Volume far higher than expected Head sampling disabled, or the same trace exported by agent and gateway Set a sampler; check for double pipelines
Backend rejects metrics Cardinality explosion Top series by count in Prometheus; drop high-cardinality attributes at the Collector
Collector OOMs on spikes Missing or misordered memory_limiter Move it to position 1; lower limit_percentage; scale the gateway horizontally
Timings look impossible (negative durations) Clock skew between hosts Run NTP/chrony everywhere; prefer a single clock source in containers
Logs have no trace_id Logging bridge not installed Add the OTel logging instrumentation or inject IDs manually as shown above

Migration Notes

  • Instrument with the OTel API, not a vendor SDK. Pointing a vendor exporter at the backend later is a config-only change; swapping a hard-coded vendor SDK is a code change across every service.
  • Run new and old side by side. The Collector fans out, so you can dual-write to the legacy agent and the new backend, compare for a week, then cut over.
  • Legacy agents (vendor APM agents, Prometheus client libraries used as an SDK, Jaeger clients) still work but keep you on their wire format and lifecycle — treat them as a migration source, not a target.
  • Bridge the gaps. The ecosystem has receivers for Jaeger/Zipkin and exporters for Prometheus and most commercial vendors, so partial migration is normal and safe.

Quick Reference

Key environment variables recognised by every OTel SDK:

Variable Purpose
OTEL_SERVICE_NAME Shortcut for the service.name resource attribute
OTEL_RESOURCE_ATTRIBUTES Additional resource attributes, comma-separated k=v
OTEL_EXPORTER_OTLP_ENDPOINT Collector or backend OTLP endpoint
OTEL_EXPORTER_OTLP_PROTOCOL grpc or http/protobuf
OTEL_TRACES_SAMPLER always_on, always_off, traceidratio, parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG Ratio for the ratio-based samplers (e.g. 0.05)
OTEL_METRICS_EXPORTER otlp, prometheus, none
OTEL_LOGS_EXPORTER otlp, none
OTEL_PROPAGATORS tracecontext, baggage, b3, jaeger
OTEL_SDK_DISABLED true kills all telemetry without a redeploy needed elsewhere

Default ports: 4317 OTLP/gRPC, 4318 OTLP/HTTP, 9464 Prometheus scrape endpoint from an SDK, 8888 Collector's own internal metrics, 13133 Collector health check.

Further Reading