Go Observability — Metrics, Traces, and Correlation

Production Go services need at least two always-on signals to be debuggable: metrics (aggregated measurements for alerting and SLOs) and traces (per-request flow showing where time went). The third deliverable is correlation: a P99 metric spike must lead, in one click, to the trace that caused it.

Logging is not in this skill. Structured logging with log/slog, log levels, and zap/logrus/zerolog migration belong to the go-logging skill. This skill only references logs in the context of correlating them with traces.

Core Rules

1. A feature is not done until it is observable. New code MUST export at least: one Counter for operations, one Counter for errors, one Histogram for latency.
2. Histograms, not Summaries, for latency. Summaries cannot be aggregated across instances; Histograms support histogram_quantile() server-side.
3. Label cardinality is bounded. Never put unbounded values (user IDs, full URLs, request IDs) in Prometheus labels. Use route patterns, status classes, method.
4. Context flows everywhere. A function that does I/O takes ctx context.Context as its first argument. No context = no trace propagation = no correlation.
5. Record errors on the span. When a span ends in failure, call span.RecordError(err) and span.SetStatus(codes.Error, ...). A green span hides a real failure.
6. Correlate or it didn't happen. Inject trace_id into logs, attach exemplars to histograms. Otherwise the three signals are three different products.

Signal Decision

Pick the signal that matches the question. Do not log what should be a metric.

Read references/metrics.md for Counter/Gauge/Histogram patterns, naming, and PromQL-as-comments. Read references/tracing.md for TracerProvider setup, span attributes, and otelhttp middleware.

Metrics — the 60-Second Setup

import "github.com/prometheus/client_golang/prometheus"

// rate(http_requests_total{code=~"5.."}[5m]) / rate(http_requests_total[5m])
var httpRequests = prometheus.NewCounterVec(
    prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total HTTP requests by method, route, status class.",
    },
    []string{"method", "route", "code"}, // ALL bounded
)

// histogram_quantile(0.99, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))
var httpLatency = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "HTTP request latency in seconds.",
        Buckets: prometheus.DefBuckets,
    },
    []string{"method", "route"},
)

The comment above each metric is the PromQL it is designed to answer. This makes the metric discoverable and grep-able from a dashboard.

Traces — the 60-Second Setup

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/codes"
)

func (s *OrderService) Create(ctx context.Context, in CreateOrderInput) (*Order, error) {
    ctx, span := otel.Tracer("order-service").Start(ctx, "OrderService.Create")
    defer span.End()
    span.SetAttributes(attribute.String("order.user_id", in.UserID))

    order, err := s.repo.Insert(ctx, in)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "insert failed")
        return nil, fmt.Errorf("creating order: %w", err)
    }
    return order, nil
}

Every service method, every DB query, every external HTTP call gets a span. Context must flow into s.repo.Insert(ctx, ...) so the DB span is a child of the service span.

Correlation

Metrics → Traces with Exemplars

An exemplar attaches a single trace_id to a histogram observation. In Grafana, click the dot on a P99 spike and you land on the trace.

obs := httpLatency.WithLabelValues(r.Method, routePattern)
sc := trace.SpanContextFromContext(ctx)
if eo, ok := obs.(prometheus.ExemplarObserver); ok && sc.IsValid() {
    eo.ObserveWithExemplar(elapsed.Seconds(),
        prometheus.Labels{"trace_id": sc.TraceID().String()})
} else {
    obs.Observe(elapsed.Seconds())
}

Logs → Traces

Use the otelslog bridge (see go-logging skill) so every slog.InfoContext(ctx, ...) call automatically emits trace_id and span_id. You can then grep logs by trace_id when starting from a trace, or jump from a log line to the trace.

Read references/correlation.md for exemplar wiring details, request-id propagation, and the end-to-end "metric spike → trace → log" workflow.

Context Propagation

// Bad — breaks trace propagation; the DB call starts a new root trace.
result, err := db.Query("SELECT ...")

// Good — the DB span is a child of the HTTP request span.
result, err := db.QueryContext(ctx, "SELECT ...")

Every I/O boundary takes ctx: HTTP client, database, gRPC client, message queue. Use otelhttp.NewTransport to instrument outbound HTTP and otelhttp.NewHandler for inbound.

Anti-Patterns

Verification Checklist

Before marking the change done:

• [ ] Each new code path emits at least one counter (operations) and one histogram (latency)
• [ ] All metric labels are bounded (method, route pattern, status class — not IDs)
• [ ] Each PromQL query the metric is meant to answer appears as a comment above the metric declaration
• [ ] Every I/O method takes ctx first; downstream calls pass it through
• [ ] Every service method, DB call, and outbound HTTP call has a span
• [ ] Failure paths call span.RecordError(err) and span.SetStatus(codes.Error, ...)
• [ ] At least one histogram uses ObserveWithExemplar for trace correlation
• [ ] Logs and traces are correlated (see go-logging skill: otelslog bridge)

References

• references/metrics.md — Counter/Gauge/Histogram, naming, cardinality, PromQL examples
• references/tracing.md — TracerProvider, spans, attributes, otelhttp, sampling
• references/correlation.md — Exemplars, trace_id in logs, end-to-end workflow
• references/anti-patterns.md — Detailed walkthrough of each anti-pattern with code