install-zymtrace-profiler — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited install-zymtrace-profiler (Agent Skill) and scored it 92/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 2 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 2 flagged
The text {match} tells the agent to skip the normal "ask the user first" gate. Used adversarially it removes the human-in-the-loop check before destructive or sensitive actions, turning a normally-gated agent into a fire-and-forget executor.
The text {match} tells the agent to skip the normal "ask the user first" gate. Used adversarially it removes the human-in-the-loop check before destructive or sensitive actions, turning a normally-gated agent into a fire-and-forget executor.
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
Helps the user install the zymtrace profiler agent — the low-overhead eBPF + CUDA profiler that runs on every node and ships profiles to the backend gateway.
The backend must be installed first (install-zymtrace-backend skill). Profiler agents have nowhere to send profiles otherwise.Deep details (NVML library paths, PC sampling, env var reference, air-gapped image mirroring, framework metrics) live in ${CLAUDE_PLUGIN_ROOT}/skills/install-zymtrace-profiler/reference.md.
Open with a short welcome before any commands or questions:
👋 Thanks for installing the zymtrace profiler! It's a low-overhead agent (<1% CPU, ~256 MB RAM) that ships profiles to your backend.
>
Stuck? Reach out: - Community Slack: <https://join.slack.com/t/zymtrace/shared_invite/zt-3fdidjufl-q~NHxDzQlzal2B9mujfaoQ> - Email: <[email protected]>
>
Tip — analyze GPU and CPU flamegraphs via MCP: once profiles start flowing, connect the zymtrace MCP to your agent (Claude Code, Codex, or Cursor) — see configure-zymtrace-mcp — and analyze GPU + CPU flamegraphs in natural language. Docs: <https://docs.zymtrace.com/mcp>>
Here's the plan: 1. Verify your tools and locate the backend gateway. 2. Decide CPU-only vs CUDA (GPU) profiling. 3. Install the DaemonSet (or Docker / binary). 4. Verify the agent is reporting.
>
Ready when you are.
If the user has already specified Helm / GPU / target, skip the roadmap and dive in.
values.yaml: <https://raw.githubusercontent.com/zystem-io/zymtrace-charts/main/charts/profiler/values.yaml>helm search can't reach the repo (just added / stale cache / air-gapped).##### Claude runs
helm version --short && kubectl version --client
kubectl cluster-info | head -2
helm list -A | grep -i zymtrace # locate the backend releaseIf helm/kubectl are missing → point to install docs; do not install them. If no backend release is found anywhere → STOP and route to install-zymtrace-backend first; the profiler has nowhere to send data without it.
Recommend defaults `zymtrace` / `profiler` for namespace + release name. Full policy: shared/conventions.md.| Variable | Resolve by | ||
|---|---|---|---|
| Backend release & its namespace | `helm list -A \ | grep -i 'backend.*zymtrace'` | |
| Backend gateway service FQDN | <PREFIX>-gateway.<backend-NS>.svc.cluster.local:80 (in-cluster) or external ingress host | ||
| GPU nodes present? | `kubectl get nodes -l nvidia.com/gpu=true 2>/dev/null \ | wc -l` | |
| NVIDIA device plugin / GPU operator? | `kubectl get pods -A \ | grep -E 'nvidia-device-plugin\ | gpu-operator'` |
| CUDA runtime ≥ 12.x? (required for GPU profiling) | `kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.labels.nvidia\.com/cuda\.runtime-version\.major}{"\n"}{end}' \ | sort -u — must be 12 or higher. If labels are missing, kubectl exec into an existing GPU pod and run nvidia-smi, or ask the user to run nvidia-smi` on a GPU node themselves and report back (the skill usually can't SSH into the node). | |
| Existing profiler release? | `helm list -A \ | grep -i 'profiler.*zymtrace'` | |
| Customer-provided values file? | Ask — same rule as backend skills. Filename respected. |
Things you must ask:
Blockers (stop, surface):
Recommendations (note, proceed):
| Method | When |
|---|---|
| Helm DaemonSet (recommended) | Kubernetes — gives you the full chart with GPU support and framework metrics. |
| kubectl manifest | Kubernetes without Helm. CPU-only is fine; GPU profiling has fewer config knobs. |
| Docker | Single host (non-k8s) — same VM the workload runs on. |
| Binary + systemd | Bare-metal Linux (Slurm nodes, on-prem GPU servers). |
| Profiling | Template | What it gives you |
|---|---|---|
| CPU only | values/helm-cpu.yaml | eBPF unwinder for native, Python, Go, Java, Node. No CUDA libraries. |
| GPU (CUDA) | values/helm-gpu.yaml | Above + CUDA kernel profiling + GPU metrics + vLLM/SGLang/Triton metrics. Same template works for MIG slices and dense multi-GPU nodes. |
GPU profiling requires CUDA 12.x or higher. Older CUDA runtimes are not supported. If the customer's nodes are on CUDA 11.x, GPU profiling is a blocker — recommend CPU-only profiling until they upgrade the CUDA toolkit / driver. Architecture: AMD64/x86_64 and ARM64 are both supported.
If GPU nodes exist on the cluster (nvidia.com/gpu=true label) and CUDA ≥ 12.x, default-recommend GPU profiling. The agent works on CPU nodes too — cudaProfiler.enabled: true is a no-op when there's no NVIDIA driver.
Where does the agent send profiles? The format is always `<host-or-ip>:<port>` — never a URL with `https://` in front. TLS is controlled by the -disable-tls flag, not by the URL scheme.
| Setup | Set -collection-agent to |
|---|---|
| Same cluster as backend (default in templates) | <PREFIX>-gateway.<backend-NS>.svc.cluster.local:80 + -disable-tls |
| Different cluster, backend has TLS ingress | <gateway-host>:443 — remove -disable-tls |
| Different cluster, NodePort | <any-node-ip>:<nodeport> + -disable-tls |
Resolve <PREFIX> and <backend-NS> from the backend release: helm list -A | grep -i 'backend.*zymtrace'.
If mentioned, mirror ghcr.io/zystem-io/zymtrace-pub-profiler:<VERSION> into the customer's registry and set:
global:
imageRegistry: "<your-registry>"
registry:
requirePullSecret: true # only if registry needs authFull procedure: reference.md § Air-gapped install.
##### Claude runs
helm repo add zymtrace https://helm.zystem.io # idempotent if already added
helm repo update zymtrace
helm search repo zymtrace/profiler --versions | head -5Version to install. helm search (above) is the source of truth for available chart versions — install the newest it lists unless the user pins one. If it returns nothing (repo just added and the cache is stale, or air-gapped), fall back to the changelog at <https://docs.zymtrace.com/changelog> — the top entry is the latest release. Keep the profiler on the same version as the backend. Tell the user which version you're installing before Step 3.
Copy the matching template from values/ to zymtrace-profiler-values.yaml in the user's working directory. Don't ask the customer if they already have one — customers typically don't ship with a profiler values file (the backend often does, the profiler rarely does). Only respect a different filename if they explicitly volunteer that they have one (per shared/conventions.md).
Pick the template that fits and edit:
profiler.args[0] -collection-agent=... to the backend gateway endpoint.-disable-tls if pointing at an HTTPS endpoint.nodeSelector: nvidia.com/gpu: "true" matches your cluster's GPU label.Print the exact command + resolved values (release name, namespace, target backend gateway, GPU yes/no, image tag if pinned). Wait for explicit confirmation.
##### Claude runs
helm upgrade --install <REL> zymtrace/profiler \
--namespace <NS> --create-namespace \
-f <values-file> \
--reset-then-reuse-values \
--atomic --debugDefault <REL> = profiler, <NS> = zymtrace (same namespace as backend works fine — no conflict).
ERROR: daemonset.apps/zymtrace-profiler is not ready: pods are not ready → expected for ~30s while pods start. Wait. If it persists past 2 min, kubectl describe ds -n <NS> <PREFIX>-profiler.
ERROR: ImagePullBackOff → registry / version mismatch. Verify image tag at <https://github.com/orgs/zystem-io/packages>.
ERROR: pod CrashLoopBackOff with permission denied on /sys/kernel/debug → kernel doesn't support eBPF or the security context is being stripped. kubectl describe pod to confirm, and check securityContext.capabilities.add: [SYS_ADMIN] is honored.
##### Claude runs
bash ${CLAUDE_PLUGIN_ROOT}/skills/install-zymtrace-profiler/scripts/verify-profiler.sh <NS> <REL>Checks DaemonSet readiness, license validity in logs, GPU library extraction (if cudaProfiler.enabled), and connection-to-gateway evidence (no connection refused or dns lookup failed in last 50 log lines).
##### Claude runs
helm get values <REL> -n <NS> > <values-file>Recommend the customer commit the file: git add <values-file> && git commit -m "zymtrace: profiler install for <NS>/<REL>".
If GPU profiling was enabled, the agent is running but no workload is being profiled yet — workloads need to set CUDA_INJECTION64_PATH. The pattern is one env var; the variation is where you set it (Slurm prolog, k8s pod spec, Docker -e, ~/.bashrc).
Point the user at reference.md § Profiling real workloads — it covers:
hostIPC, --shm-size, VLLM_ATTENTION_BACKEND), with concrete Docker and Kubernetes manifests.Authoritative docs: <https://docs.zymtrace.com/install/profiler/cuda-gpu-profiler> and <https://docs.zymtrace.com/install/profiler/gpu-profiler-quick-start>.
CPU-only installs are done at this point — profiles for every process on every node start flowing within ~30 seconds.
For kubectl manifest, Docker, or binary+systemd installs, see reference.md § Other install methods. The same pre-flight rules apply; only Step 4 (install) differs.
Exit when ALL of the following are true (substitute <NS> / <REL> / <PREFIX>):
helm status <REL> -n <NS> reports STATUS: deployed.<PREFIX>-profiler has DESIRED == READY (kubectl get ds -n <NS>).Your license is valid until … OR streaming connection established (means the agent reached the backend).connection refused, dns lookup failed, or forbidden in the last 50 log lines of any pod.ls -la /var/lib/zymtrace/profiler on a GPU node shows libzymtracecudaprofiler.so (the agent extracted it for workload mounts).If any box fails, route by symptom:
troubleshoot-zymtrace-profiler.troubleshoot-zymtrace-backend, which walks the full backend ↔ profiler path.Both have first-pass diagnostic scripts to run before going manual.
zymtrace.example.com:443 — NOT https://zymtrace.example.com:443 or https://zymtrace.example.com/. Adding a scheme prefix makes the agent fail to parse the value and retry forever. TLS is controlled separately by presence/absence of the -disable-tls flag, not by the URL scheme.<backend-NS>, or pointed at a non-existent service) → agent retries forever, no profiles appear. Test with kubectl run -it --rm dns-test --image=busybox -- nslookup <PREFIX>-gateway.<backend-NS>.svc.cluster.local.nodeSelector or scope to GPU nodes.kubectl get nodes --show-labels to confirm.--nvml-path=<path> once the path is known (saves startup scan).helm upgrade / helm upgrade --install for this chart without --reset-then-reuse-values. See shared/conventions.md.-project= / ZYMTRACE_PROJECT in profiler args, env vars, or values templates. Always use the default project — the agent creates it automatically.privileged: true on the k8s pod, or sudo when running a bare binary. That's a workload-side change that needs explicit user confirmation. Recommend it for dev/staging deep-dives or on-demand production debugging. See reference.md § PC sampling.~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.