Workload Troubleshooting

The symptom-first companion to the workload skill (which owns workload types, the spec shape, and the create/update tools). Given an unhealthy workload, map what you observe to its platform-specific root cause and a fix the schema will actually accept. Diagnosis is read-only and MCP-first; most failures trace to a Control Plane rule a generic engineer would not guess — deny-by-default firewalls, the secret identity+policy chain, blocked ports, the sleep-binary shutdown rule. The single most common is OOMKilled. Deep remediation for each area lives in the domain skill named in that section; this skill is the diagnostic map.

Step 1 — Gather state (read-only)

CLI fallback (MCP unavailable, interactive shell, or CI/CD):

cpln workload get WORKLOAD --gvc GVC -o json
cpln workload eventlog WORKLOAD --gvc GVC -o json
cpln logs '{gvc="GVC", workload="WORKLOAD"}' --limit 50          # |= "error" filters; container="_accesslog" for HTTP codes
cpln workload connect WORKLOAD --gvc GVC --location LOCATION      # interactive shell

Failure catalog

Out of memory (OOMKilled) — the most common issue

Symptoms: container restarts repeatedly, events show OOMKilled, crashes under load.

memory is a hard cap — exceed it (app + runtime + GC + buffers) and the kernel kills the container. Usual culprits: Java without -Xmx, Node without --max-old-space-size, Python loading large datasets. Each container, sidecars included, has its own limit. With Capacity AI on, spiky workloads can be downsized too aggressively — set minMemory as a floor (Capacity AI never downscales CPU below 25 millicores).

Fix: check real usage with query_metrics, then raise memory — but memory (MiB) must stay within 8× CPU (millicores), so raise cpu alongside it or the update is rejected (the default cpu: 50m caps memory at 400Mi). See Apply fixes within the schema's limits.

Image pull failures

Symptoms: events show ImagePullBackOff / ErrImagePull, deployment stuck.

• Reference format — //image/NAME:TAG (org registry), bare NAME:TAG (Docker Hub, no docker.io/), full URL (other registries).
• Platform — images must be linux/amd64 for managed locations (BYOK allows more).
• Pull secret — a private external registry needs a pull secret in the GVC's pullSecretLinks; only docker, ecr, gcp secret types work, and org //image/ images need none.

Create with create_secret_docker, attach with update_gvc. Deep setup: image skill.

Secret access failures

Symptoms: env vars empty, logs show missing config, secret-access errors in events.

A workload reaches a secret only with all three in place: an identity linked to it (spec.identityLink), a policy granting that identity `reveal` on the secret, and a correct reference. Fastest fix: workload_reveal_secret builds the whole chain in one call. Reference format by type:

The manual chain: access-control and setup-secret skills.

Port mismatch — healthy but 502/503

• The spec port must match what the process listens on — confirm with netstat -tln via workload_exec.
• On serverless, the runtime injects PORT and rejects a PORT env var that doesn't equal the exposed port.
• Type rules: serverless exposes exactly one port (zero is rejected; TCP needs a dedicated LB — see Dedicated load balancer & domain); standard and stateful expose zero or more; cron serves no endpoint.
• Blocked ports (cannot bind, invalid for TCP probes): 8012, 8022, 9090, 9091, 15000, 15001, 15006, 15020, 15021, 15090, 41000.

Firewall blocking traffic

Symptoms: unreachable externally, can't reach external APIs, or can't talk to other workloads.

Deny-by-default: external inbound disabled, external outbound disabled, internal none. Fix via update_workload:

• Inbound — external.inboundAllowCIDR (e.g. 0.0.0.0/0, or specific CIDRs).
• Outbound — external.outboundAllowCIDR, or outboundAllowHostname (hostname rules reach only ports 80, 443, 445).
• Internal — internal.inboundAllowType: same-gvc / same-org / workload-list (default none blocks all workload-to-workload traffic).

Full model: firewall-networking skill.

Health-check failures

Symptoms: events show probe failures, replicas unready, restarts.

Default probes: serverless gets readiness + liveness TCP on the container port; standard, stateful, and cron have none (cron strips them). Common fixes: raise initialDelaySeconds (0-600; default 10 readiness / 60 liveness) for slow starts; raise periodSeconds (1-600, default 10) or timeoutSeconds (1-600, default 1) for over-aggressive probes; ensure an HTTP path returns 200-399. Readiness failure removes the replica from the pool and pauses rollout; liveness failure restarts it. An autoscaled workload with no real readiness probe gets traffic before it's ready, causing 502s on scale-up — add an httpGet readiness probe (it needs a port; defaults to the container's). Probe tuning: workload-security skill.

Resource limits & Capacity AI

Symptoms: won't schedule, throttled, or Capacity AI not adjusting.

• CPU/memory must fit org quota; maxScale × per-replica resources is enforced at scheduling.
• Capacity AI does not apply with CPU-utilization or multi-metric autoscaling, or on stateful workloads — use minCpu / minMemory instead (those persist; capacityAI is stripped on stateful).
• Stateful sizing — minCpu/cpu at most 4000m apart (ratio ≤ 4:1); minMemory/memory at most 4096Mi apart (ratio ≤ 4:1).
• Ephemeral storage — 1GB per CPU core (minimum 1GB); exceeding it replaces the replica.

Deep model: autoscaling-capacity skill.

Container won't start (restrictions)

• UID 1337 is the mesh proxy's UID — running as it excludes the container from the sidecar, disabling mesh communication and mTLS. Override runAsUser to another UID in 1-65534 (0/root is rejected).
• Reserved container names — not istio-proxy / queue-proxy / istio-validation (or other reserved names); cannot start with cpln- or debugger-.
• Reserved env vars — names starting CPLN_, plus K_SERVICE / K_CONFIGURATION / K_REVISION, are rejected; each value caps at 4096 characters.
• Suspended — spec.defaultOptions.suspend: true stops the workload (min/max scale 0). Clear it, or cpln workload start WORKLOAD --gvc GVC.

Autoscaling misconfiguration

Symptoms: won't scale, 502s on scale-up, scale-to-zero not working, or an invalid-strategy error on create.

Strategies: concurrency, cpu, memory, rps, latency, keda, disabled. Per type: serverless has no latency or multi-metric; standard has no concurrency; stateful has no concurrency; cron has no autoscaling at all (the block is removed). Scale-to-zero: serverless with rps or concurrency; standard and stateful only with metric: keda (otherwise the update is rejected); cron cannot. For 502s on scale-up, fix the readiness probe (above). Details: autoscaling-capacity skill.

Termination / graceful shutdown

Symptoms: requests fail during deploys or scale-down, 502/503 on rollout, containers killed abruptly.

• Missing `sleep` — if sleep is absent from any container, all containers get SIGKILL immediately (no drain). Many distroless/minimal images lack it; confirm with which sleep. Fix: include sleep or add a custom preStop.
• preStop error — a failing custom preStop in any container SIGKILLs all of them.
• Ignores SIGTERM — after the preStop (default sleep 45) the container gets the termination signal, then SIGKILL once terminationGracePeriodSeconds (default 90; max 900 without the cpln/relaxGracePeriodMax tag) elapses.

Sequence and rollout options: workload-security skill.

Volume mount failures

Symptoms: can't read mounted files, permission denied, empty cloud volume.

• Secret volumes need the identity + reveal policy chain (as Secret access failures).
• Cloud volumes (S3, GCS, Azure Blob/Files) need an identity, a cloud-access policy, and outbound firewall to the provider hosts (*.amazonaws.com, *.googleapis.com, *.blob.core.windows.net / *.file.core.windows.net plus *.azure.com); auth is identity-only — embedded keys do not work. Read-only except Azure Files.
• Reserved mount paths: /dev, /dev/log, /tmp, /var, /var/log. Max 15 volumes; no path may be a parent of another.
• filesystemGroupId defaults to 0 (root) when unset — set it (1-65534) for a non-root app.

Volume sets: stateful-storage skill.

Service-to-service failures

Symptoms: a workload can't reach another internally (connection refused or timeout).

• Target's internal firewall must allow the caller — same-gvc / same-org / workload-list (default none); listing a workload needs view on it.
• Endpoint — http://WORKLOAD.GVC.cpln.local:PORT (use http://; the sidecar adds mTLS). An omitted port defaults to the target's first container port; only listed ports are reachable.
• Cross-GVC — the target must allow same-org or list the caller; traffic may span locations and incur egress charges.

Dedicated load balancer & domain

Symptoms: unreachable after enabling a dedicated LB, TCP broken, wrong Host header.

• TCP needs a dedicated LB on the GVC plus a custom Domain with a TCP port — not on default endpoints (HTTP/HTTP2/gRPC only).
• Enabling/disabling a dedicated LB causes brief DNS-propagation downtime and per-location charges.
• Serverless Host header — a custom domain delivers the canonical endpoint as Host (the custom domain moves to X-Forwarded-Host); standard and stateful workloads get the custom domain as Host.
• Protocol compatibility — the domain port protocol must match the container's: HTTP2 fronts HTTP2 or gRPC, HTTP fronts HTTP.

Routing, TLS, and LBs: domain and ipset-load-balancing skills.

Apply fixes within the schema's limits

A fix the Joi schema rejects at update_workload time is worse than none. Before applying a resource or option change, confirm it stays within these (the validator names the violated rule):

• memory (MiB) at most 8× cpu (millicores); CPU ≥ 25m; memory ≥ 32MiB; minCpu/minMemory never above cpu/memory.
• runAsUser / filesystemGroupId: 1-65534 (0/root rejected).
• terminationGracePeriodSeconds: ≤ 900 (higher only with the cpln/relaxGracePeriodMax tag).
• capacityAI with metric: cpu is rejected; capacityAI is stripped on stateful, cron, and vm.
• Standard/stateful minScale: 0 requires metric: keda; cron and vm cannot scale to zero.
• A metric outside the workload type's allow-list is rejected.

Prefer update_workload (PATCH — only the fields you set change). For full manifest control, author against get_resource_schema then cpln apply -f workload.yaml --gvc GVC.

Verify

After applying, poll mcp__cpln__list_deployments until every location reports ready, confirm the original symptom cleared (events/logs), and report the canonical endpoint list_deployments returns — never a constructed URL. For a public workload, confirm it actually responds, not just that it is ready.

Troubleshooting

Quick reference

MCP tools

CLI (fallback)

Use when MCP is unavailable, for an interactive shell, or in CI/CD (service-account CPLN_TOKEN).

cpln workload get WORKLOAD --gvc GVC -o json
cpln workload eventlog WORKLOAD --gvc GVC -o json
cpln workload connect WORKLOAD --gvc GVC --location LOCATION
cpln apply -f workload.yaml --gvc GVC

Related skills

• workload — primary skill (types, spec shape, tool division); start here.
• workload-security — probe tuning, termination, securityOptions, direct LBs.
• firewall-networking — inbound / outbound / internal rules.
• autoscaling-capacity — scaling strategies and Capacity AI.
• access-control / setup-secret — the identity + reveal chain.
• stateful-storage — volume sets; domain / ipset-load-balancing — routing and LBs; image — registries and pull secrets.

Documentation

• Workload Types
• Containers
• Capacity AI
• Firewall
• Termination