Workload Security & Production Hardening

Deep-dive companion to the workload skill, which owns workload types, the spec shape, and the readiness-vs-liveness model. Everything below is production-hardening detail for an existing workload.

Where settings live. Health probes go inline in containers[] via create_workload / update_workload. Every other block here — sidecar.envoy, loadBalancer, securityOptions, rolloutOptions — is set by its own configure_workload_* tool, a set-or-clear PATCH on that one field (remove: true clears it). Tool availability: these tools live in the full toolset profile; if one isn't advertised, reconnect with ?toolsets=full or use the CLI. Reads/deletes work on any profile (list_resources / get_resource / delete_resource).

Health Probes

Define readinessProbe (gate traffic) and livenessProbe (restart on failure) as distinct probes in the container spec.

Defaults by workload type:

• Serverless — a TCP readiness probe on the listening port is injected by default (plus default startup and liveness probes). Adequate, but an httpGet against a real endpoint catches more failure modes (DB unreachable, dependency timeout, deadlock).
• Standard / Stateful — no probes by default; add them explicitly for any production workload.
• Cron — probes are stripped (ignored).

No HTTP healthcheck? Use tcpSocket on the listening port as a baseline — don't run a long-lived workload probe-less.

Probe schema

Each probe takes exactly one of exec / grpc / tcpSocket / httpGet, plus these timing fields:

httpGet omitting port defaults to the first container port; httpGet.scheme defaults to HTTP. Keep liveness looser than readiness (e.g. periodSeconds: 30) — restarts are expensive.

containers:
  - name: api
    image: //image/api:v1.0
    ports: [{ number: 8080, protocol: http }]
    readinessProbe:
      httpGet: { path: /healthz/ready, port: 8080 }
      initialDelaySeconds: 5
      failureThreshold: 3
    livenessProbe:
      httpGet: { path: /healthz/live, port: 8080 }
      initialDelaySeconds: 30
      periodSeconds: 30

JWT Authentication

JWTs are validated at the Envoy sidecar before requests reach the workload, via the jwt_authn HTTP filter under spec.sidecar.envoy. Set it per-workload with configure_workload_sidecar, or org-wide-per-GVC by putting the same sidecar.envoy on the GVC (update_gvc) — it then applies to every workload in that GVC.

Must-know rules (the filter is strictly validated, not passthrough):

• The filter name, typed_config."@type", and priority (0-100) must be exact — copy them from the example.
• Each provider needs a matching clusters[] entry (STRICT_DNS + TLS transport socket) so Envoy can fetch the JWKS over HTTPS. remote_jwks.http_uri.cluster must equal that cluster's name.
• rules are first-match-wins. A rule with no requires lets matching paths through without a token — use it to exempt health/metrics endpoints.
• claim_to_headers forwards JWT claims into request headers, so the workload gets identity context without re-parsing the token.
• Provider/cluster names starting with cpln_ are UI-managed and restricted (no async_fetch / retry_policy, and cache_duration must equal http_uri.timeout). For hand-authored configs use a non-`cpln_` name for full Envoy flexibility.

spec:
  sidecar:
    envoy:
      clusters:
        - name: auth0
          type: STRICT_DNS
          load_assignment:
            cluster_name: auth0
            endpoints:
              - lb_endpoints:
                  - endpoint:
                      address:
                        socket_address: { address: YOUR_TENANT.auth0.com, port_value: 443 }
          transport_socket:
            name: envoy.transport_sockets.tls
      http:
        - name: envoy.filters.http.jwt_authn
          priority: 50
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication
            providers:
              auth0:
                issuer: https://YOUR_TENANT.auth0.com/
                audiences: [https://api.example.com]
                remote_jwks:
                  http_uri: { uri: https://YOUR_TENANT.auth0.com/.well-known/jwks.json, cluster: auth0, timeout: 5s }
                  cache_duration: 300s
                claim_to_headers:
                  - { header_name: X-User-Sub, claim_name: sub }
            rules:
              - match: { prefix: /healthz }          # public, no token required
              - match: { prefix: / }
                requires: { provider_name: auth0 }    # everything else needs a valid JWT

For any OIDC provider (Auth0, Firebase, Cognito, Okta), set issuer to its issuer URL, remote_jwks.http_uri.uri to its JWKS endpoint (usually /.well-known/jwks.json), and audiences to your client ID or API identifier.

Security Options

spec.securityOptions (set with configure_workload_security):

Neither has a schema default — unset means the image's user and root/GID 0 for volumes. Set runAsUser to a non-root UID for defense in depth; set filesystemGroupId so containers can share access to mounted volumes (e.g. volume sets). Not valid for `type: vm` (the guest OS owns its own security context).

Trap — never `runAsUser: 1337`. That is the mesh proxy's UID. A container running as 1337 is excluded from the Envoy redirect, so it bypasses the mesh entirely — losing mTLS and firewall enforcement and getting unfiltered egress.

Workload permissions

Beyond standard view / edit (implies view) / create / delete / manage, the workload kind adds three non-obvious permissions: connect (interactive shell into a replica) and configureLoadBalancer (toggle direct/dedicated LBs) — neither implied by `edit` — plus exec, which implies exec.runCronWorkload + exec.stopReplica. Full policy and principal setup: access-control skill.

Direct Load Balancers

spec.loadBalancer.direct exposes workload ports through a per-location cloud LB. Unlike the shared LB, it supports custom TCP/UDP ports and needs no domain registration — use it for non-HTTP protocols or custom ports. You are responsible for your own TLS certificates. Set with configure_workload_load_balancer.

• enabled (bool, required) — when false, the LB is stopped and accrues no charges.
• ports[] — each entry: externalPort (22-32768, required), protocol (TCP/UDP, required), containerPort (80-65535, excludes reserved ports), scheme (http/tcp/https/ws/wss, optional, sets the UI link scheme; default https).
• ipSet (optional) — link to an IP set for reserved static IPs.

Reserved container ports (rejected): 8012, 8022, 9090, 9091, 15000, 15001, 15006, 15020, 15021, 15090, 41000.

spec:
  loadBalancer:
    direct:
      enabled: true
      ports:
        - { externalPort: 5432, protocol: TCP, scheme: tcp, containerPort: 5432 }

Each direct-LB location also gets a public Geo DNS address with latency-based routing, usable as a CNAME target for custom domains.

Geo location headers

spec.loadBalancer.geoLocation adds MaxMind GeoLite2 data to inbound HTTP requests under the header names you set in headers.{asn,city,country,region} (each max 128 chars; enabled defaults false). When enabled, at least one header is required, names must be unique, and existing same-named headers are replaced. HTTP-exposed workloads only. Pair with header-based firewall rules for geo-filtering (firewall-networking skill).

Replica-direct endpoints

spec.loadBalancer.replicaDirect: true gives each replica its own stable hostname (default false, stateful workloads only):

• External: WORKLOAD-GVCALIAS-INDEX.LOCATION.controlplane.us
• Internal: replica-INDEX.WORKLOAD.LOCATION.GVC.cpln.local:PORT

Graceful Termination

spec.rolloutOptions (set with configure_workload_rollout) governs how replicas are removed during scaling, version updates, Capacity AI rollouts, and maintenance.

terminationGracePeriodSeconds is the total budget for a replica to shut down before SIGKILL; raise it for long-running requests. Termination sequence:

1. The load balancer removes the replica from the pool (up to ~10s) so new requests route elsewhere.
2. The Control Plane sidecar drains in parallel: it holds for grace - 10 seconds (default 80), then waits for in-flight connections to complete before shutting down.
3. Containers run their preStop hook. With no custom hook, a default sh -c "sleep N" runs where N = half the grace period (45s at the 90s default), giving the LB time to stop routing.
4. After preStop, the container receives SIGINT and has the remaining grace to exit, then SIGKILL. Handle the termination signal to drain cleanly.

*Trap — immediate SIGKILL of all containers if `sleep`/`sh` is missing in any container (common with distroless/minimal images) or a custom `preStop` errors in any* container. A custom preStop must include a delay or connection check, and must be tested — an error there force-kills the whole replica.

Verify

• After any change, poll mcp__cpln__list_deployments until each location reports ready — it surfaces probe failures per location.
• mcp__cpln__get_workload_events gives the probe/liveness failure reason and message; mcp__cpln__get_workload_logs shows app-side errors.
• For JWT, send a request with and without a valid token (expect 401 without) and confirm the claim_to_headers header arrives at the workload.

Troubleshooting

Quick reference

MCP tools

All configure_workload_* tools are full-profile, set-or-clear PATCH (remove: true clears).

CLI (fallback)

Use the CLI when the MCP server is unavailable or unauthenticated, or in CI/CD (service-account CPLN_TOKEN).

cpln workload get WORKLOAD --gvc GVC -o yaml-slim > workload.yaml
# edit, then apply (CI/CD: add --ready to block until deployed)
cpln apply -f workload.yaml --gvc GVC

Related skills

• workload — primary skill (types, defaults, spec shape, tool division); start here.
• firewall-networking — CIDR / header rules, geo-filtering, LB types.
• access-control — policies, principals, the full permission model.
• autoscaling-capacity — scaling and Capacity AI settings.
• stateful-storage — volume sets (pair with filesystemGroupId).

Documentation

• Workload Security
• JWT Auth
• Termination
• Load Balancing