Observability and Logging

The operability layer every long-running service needs: how it logs, how it reports health, and how its metrics and traces show a failing dependency. The cross-cutting principles hub is the coding-standards skill, this skill is the detailed playbook for the logging and observability rules it points to.

Baseline: OpenTelemetry is the reference instrumentation API for traces, metrics, and logs. Prometheus exposition is the reference metrics format. Both are language-neutral, so pick the current stable SDK for your stack.

When to activate

• Adding or reviewing logging in a service.
• Instrumenting a service with health checks, metrics, or tracing.
• Defining SLOs, alerts, or dashboards.
• Writing the startup readiness banner.

When not to activate

• Building the CI/CD pipeline or the release gate that ships the service, use deployment-patterns.
• Writing the Kubernetes liveness, readiness, and startup probe manifests, use deployment-patterns.
• Designing timeouts, capped retries, circuit breakers, and graceful shutdown, use backend-patterns.
• Writing the Dockerfile or Compose file the service runs in, use docker-patterns.
• Chasing a specific slow endpoint or query with a profiler, use performance-optimization.
• Deciding how exceptions are raised, typed, and chained, use coding-standards.
• Writing shell or PowerShell logging helpers, use bash or powershell.

Log through one abstraction

Business code names a logger interface, never a concrete backend. Swapping the backend must not touch a call site.

Pass:

private static final Logger log = LoggerFactory.getLogger(OrderService.class);

Fail:

private static final org.apache.logging.log4j.core.Logger log = new Log4jContextFactory().getContext().getLogger("x");

Use levels that mean something

Informational for major steps, warning for recoverable trouble, error with the full stack trace for a real failure. An expected, handled condition is not an error. Begin each line with a service or component tag and carry a correlation or trace identifier so one request can be followed across services.

Pass:

log.warn("[orders] payment declined, retrying, orderId={} attempt={}", orderId, attempt);

Fail:

log.error("payment declined");

Never log a secret

Mask credentials, tokens, decrypted payloads, and personal data everywhere they could reach a log or a crash dump.

Pass:

log.info("[auth] token issued, subject={} tokenSuffix=***{}", subject, tokenSuffix);

Fail:

log.info("[auth] token issued: {}", accessToken);

Keep logging out of control flow

Logging is a side effect. Do not branch on a log call, and do not build an expensive message the logger may discard.

Pass:

if (log.isDebugEnabled()) {
    log.debug("[cart] snapshot={}", renderExpensiveSnapshot(cart));
}

Fail:

log.debug("[cart] snapshot=" + renderExpensiveSnapshot(cart));

The error-handling rules (specific exception types, chaining to preserve the cause, central handling, structured error shape) live in the coding-standards hub. This skill covers how those errors are logged, not how they are raised.

Expose two separate health checks

Liveness reports only that the process is alive and never touches a dependency. Readiness verifies the database, cache, and external dependencies and reports unhealthy when one is degraded. Keep both outside the versioned API path, so a version bump cannot move the probe URL. The generic paths are /health and /ready, defined by api-design. Spring Boot Actuator serves the same two checks under its own paths, shown below. This skill owns what each endpoint reports. The probe manifests that point Kubernetes at them belong to deployment-patterns.

Pass:

GET /actuator/health/liveness    200 while the process runs, touches no dependency
GET /actuator/health/readiness   200 when the database, cache, and downstream services answer, 503 otherwise

Fail:

GET /api/v1/health/full   one versioned endpoint that checks every dependency and answers both probes

Emit metrics that drive action

Request rate, error rate, latency percentiles, and saturation of the resources that actually constrain the service. Propagate a trace context across every hop. Define SLOs in terms a user would recognise, a fast response or a successful request, and alert on the SLO burn rate rather than every transient blip.

Pass:

sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m])) / sum(rate(http_server_requests_seconds_count[5m]))

Fail:

process_cpu_seconds_total

Emit one startup readiness block

Every long-running service prints one canonical multi-line INFO entry the moment it starts accepting traffic, in a single log call with a leading newline. Per-line emission lets other threads interleave and rips the block apart. The full convention, the ANSI Shadow banner, the section order, the probe timeouts, and the per-stack hooks live in the reference below.

Pass:

log.info("\n{}", buildStartupLog());

Fail:

buildStartupLog().lines().forEach(log::info);

Reference files

Related skills

• coding-standards for the error-handling and design principles this skill logs against.
• backend-patterns for timeouts, capped retries, circuit breakers, and graceful shutdown.
• deployment-patterns for the probe manifests, rollout gating, and the pipeline that ships the service.
• docker-patterns for the container the service logs from.
• performance-optimization for turning a latency metric into a fix.
• springboot-patterns, python-patterns, golang-patterns, node-backend-patterns for the per-stack hook.

Checklist

• [ ] Business code logs through one abstraction, no concrete backend at a call site.
• [ ] Every log line carries a component tag and a correlation or trace identifier.
• [ ] No secret, credential, decrypted payload, or personal field reaches a log.
• [ ] Expensive message construction is guarded or lazy.
• [ ] Liveness and readiness are separate endpoints outside the versioned API path.
• [ ] Rate, error, latency, and saturation metrics are exported and an SLO burn-rate alert exists.
• [ ] The startup readiness block is emitted in one log call with a leading newline.