troubleshoot-zymtrace-backend — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited troubleshoot-zymtrace-backend (Agent Skill) and scored it 96/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 1 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 1 flagged
The text {match} tells the agent to skip the normal "ask the user first" gate. Used adversarially it removes the human-in-the-loop check before destructive or sensitive actions, turning a normally-gated agent into a fire-and-forget executor.
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
Helps the user diagnose problems on a deployed zymtrace install — the most common one is "no data is showing up in the UI."
This skill spans both backend (ingest, ClickHouse, MinIO) and profiler-side (agent reporting, CUDA injection) checks because "no data" usually requires walking both. Pure profiler-agent issues (CrashLoopBackOff, ImagePullBackOff, OOMKilled, NVML, license rejected on agent side) are handled by troubleshoot-zymtrace-profiler — route there if the user's symptom is agent-specific.
Open warmly. People reaching for a troubleshoot skill are usually frustrated.
👋 Sorry zymtrace isn't behaving — let's find it. Tell me what you're seeing (e.g. "the UI loads but no profiles", "ingest pods crash-looping", "got a license error") and I'll walk the diagnosis with you.
>
Stuck or need a human? - Community Slack: <https://join.slack.com/t/zymtrace/shared_invite/zt-3fdidjufl-q~NHxDzQlzal2B9mujfaoQ> - Email: <[email protected]>
>
Once data is flowing again, connect the zymtrace MCP to your agent (Claude Code, Codex, or Cursor) — see configure-zymtrace-mcp — and analyze GPU + CPU flamegraphs in natural language. Docs: <https://docs.zymtrace.com/mcp>If the user already named a specific symptom, skip the prompt and jump to the matching section below.
##### Claude runs
helm version --short && kubectl version --client
kubectl cluster-info | head -2
helm list -A | grep -i zymtraceResolve two sets of (namespace, release) — backend AND profiler — since "no data" can be either side:
| Variable | Resolve by | ||
|---|---|---|---|
| Backend release + namespace | `helm list -A \ | grep -iE 'backend.*zymtrace\ | zymtrace.*backend'` |
| Profiler release + namespace | `helm list -A \ | grep -iE 'profiler.*zymtrace\ | zymtrace.*profiler'` |
global.namePrefix (drives resource names) | `helm get values <REL> -n <NS> \ | awk '/^\s*namePrefix:/ {print $2}' (defaults to zymtrace`) |
If the profiler release isn't installed at all, that explains "no data". Route to install-zymtrace-profiler.Ask which symptom matches. Don't guess — the diagnostics differ.
| Symptom | Section |
|---|---|
| No data in the UI | § No data coming through — most common, full walkthrough below |
| License invalid / expired in ingest logs | § License errors |
| Ingest pods CrashLoopBackOff | § Ingest crash loop |
| Query is slow | § Slow queries |
| Disk filling rapidly | § Storage growth |
If the user's symptom isn't listed → ask them to describe what they see (UI behavior, recent logs), and pick the closest section above.
The end-to-end profile path is: workload → profiler agent → backend gateway (gRPC) → ingest → ClickHouse → UI. Any link broken = no data. Walk all four steps; don't stop at step 1.
##### Claude runs
bash ${CLAUDE_PLUGIN_ROOT}/skills/troubleshoot-zymtrace-backend/scripts/diagnose-no-data.sh <backend-NS> <backend-REL> <profiler-NS> <profiler-REL>The script automates all four checks below. The manual walkthrough:
# DaemonSet healthy?
kubectl get ds -n <profiler-NS> <PREFIX>-profiler
# Pick a pod, tail the logs
POD=$(kubectl get pods -n <profiler-NS> -l app.kubernetes.io/component=profiler -o name | head -1)
kubectl logs -n <profiler-NS> "$POD" --tail=100Look for:
Your license is valid until ... — agent reached the backend, license accepted.streaming connection established — agent is sending profiles.buffers_processed or bytes_sent counters incrementing — actual data leaving the agent.If you see connection refused, dns lookup failed, or no such host → the --collection-agent target is wrong. Fix the values file (profiler.args[0]) and re-apply with helm upgrade --install ... --reset-then-reuse-values --atomic.
If you see no positive signals but no errors → the agent might be running on nodes that have no workloads to profile. Confirm there are processes on the node (kubectl get pods --all-namespaces -o wide | grep $(node-name)).
This is the silent failure mode for GPU profiling: the agent is happily running, but no workload has actually loaded the CUDA profiler library, so no GPU profiles flow. The agent logs the interception event when a workload picks up the library:
kubectl logs -n <profiler-NS> "$POD" --tail=500 | grep -i 'intercepted.*implant'Expected line (one per profiled GPU process):
level=info msg="Intercepted zymtrace implant at /proc/576236/root//var/lib/zymtrace/profiler/libzymtracecudaprofiler.so"If this line is absent, no workload is being GPU-profiled. Common causes:
CUDA_INJECTION64_PATH env var set./var/lib/zymtrace/profiler host path mounted in.ls /var/lib/zymtrace/profiler on the node — should contain libzymtracecudaprofiler.so).CPU profiling does not need this check — there is no injection, the eBPF profiler attaches to processes directly.
# Pods OK?
kubectl get pods -n <backend-NS> -l app.kubernetes.io/component=ingest
# Recent logs
kubectl logs -n <backend-NS> deployment/<PREFIX>-ingest --tail=100 --all-containers=trueLook for:
Your license is valid until ... — backend accepted the license.received profile / processed batch lines (or quiet logs with no errors).Red flags:
clickhouse: connection refused / clickhouse: dial tcp → ClickHouse pod down or unreachable (Step 4 will tell you which).forbidden, unauthorized → license or service-token issue (jump to § License errors).disk full / out of space → jump to Step 4.# Pod running?
kubectl get pods -n <backend-NS> -l app.kubernetes.io/component=clickhouse
# PVC fill level
kubectl get pvc -n <backend-NS> | grep -i clickhouse
kubectl exec -n <backend-NS> <PREFIX>-clickhouse-0 -- df -h /var/lib/clickhouse
# Recent CH logs
kubectl logs -n <backend-NS> <PREFIX>-clickhouse-0 --tail=50If df -h shows the data volume >85% → ClickHouse stops accepting writes. Free space by lowering global.dataRetentionDays and waiting for the next compaction, or expand the PVC. The latter only works on storage classes with allowVolumeExpansion: true:
kubectl get sc <storage-class> -o yaml | grep allowVolumeExpansion
# If true:
kubectl edit pvc data-<PREFIX>-clickhouse-0 -n <backend-NS>
# bump spec.resources.requests.storageIf ClickHouse is healthy but ingest still can't reach it → check the use_existing config in the values file (host/port/credentials). For mode: create, the chart wires this automatically.
Data should appear in the UI within ~30 seconds of fixing the broken link. Refresh the browser. If you went through all four steps and still see no data, capture the four kubectl logs outputs (steps 1–4) and email <[email protected]> with the chart version (helm list -A | grep zymtrace).
Symptoms in ingest or profiler logs: license expired, license invalid, forbidden, unauthorized.
##### Claude runs
# Backend ingest
kubectl logs -n <backend-NS> deployment/<PREFIX>-ingest --tail=50 | grep -iE 'license|forbidden|unauthorized'
# Profiler
POD=$(kubectl get pods -n <profiler-NS> -l app.kubernetes.io/component=profiler -o name | head -1)
kubectl logs -n <profiler-NS> "$POD" --tail=50 | grep -iE 'license|forbidden|unauthorized'Fix paths:
global.licenseKey (or the referenced secret), helm upgrade --install ... --reset-then-reuse-values --atomic.global.licenseKey AND global.licenseKeySecretName both set means the secret wins. Decide which path you want.auth.serviceToken.enabled: true on backend, profiler missing --auth-token) → set profiler.args to include --auth-token "$ZYMTRACE_AUTH_TOKEN" or disable service-token auth in dev.##### Claude runs
kubectl describe pod -n <backend-NS> -l app.kubernetes.io/component=ingest | tail -50
kubectl logs -n <backend-NS> deployment/<PREFIX>-ingest --previous --tail=100Most common causes:
kubectl get secret -n <backend-NS> to verify.services.ingest.resources.limits.memory.8123/8443. See install skill's pitfalls.##### Claude runs
kubectl top pods -n <backend-NS> 2>/dev/null | grep -iE 'web|clickhouse'
kubectl logs -n <backend-NS> <PREFIX>-clickhouse-0 --tail=100 | grep -iE 'query|exception'Common causes:
clickhouse.create.resources or shorten global.dataRetentionDays.web — enable services.common.hpa in the values file.If ClickHouse / MinIO PVCs are filling faster than expected:
kubectl exec -n <backend-NS> <PREFIX>-clickhouse-0 -- du -sh /var/lib/clickhouse/*
kubectl exec -n <backend-NS> <PREFIX>-minio-0 -- du -sh /dataKnobs:
global.dataRetentionDays — lower it (default 30, can go down to 7 in dev). Existing data ages out at the next compaction.-samples-per-second (default 19) — lower it on agents to reduce ingestion volume.profiler.nodeSelector to scope which nodes report).Exit when the user confirms data is flowing again or escalates to support. Always recap which step fixed it, so future-them can pattern-match.
kubectl edit mid-diagnosis without confirming with the user — leaks the value into shell history.kubectl delete pvc, kubectl delete namespace, or helm uninstall without explicit user confirmation. Backups don't come from this skill.--reset-then-reuse-values — see shared/conventions.md.~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.