install-zymtrace-profiler — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited install-zymtrace-profiler (Agent Skill) and scored it 92/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 2 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 2 flagged
The text {match} tells the agent to skip the normal "ask the user first" gate. Used adversarially it removes the human-in-the-loop check before destructive or sensitive actions, turning a normally-gated agent into a fire-and-forget executor.
The text {match} tells the agent to skip the normal "ask the user first" gate. Used adversarially it removes the human-in-the-loop check before destructive or sensitive actions, turning a normally-gated agent into a fire-and-forget executor.
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
Helps the user install the zymtrace profiler agent — the low-overhead eBPF + CUDA profiler that runs on every node and ships profiles to the backend gateway.
The backend must be installed first (install-zymtrace-backend skill). Profiler agents have nowhere to send profiles otherwise.Deep details (NVML library paths, PC sampling, env var reference, air-gapped image mirroring, framework metrics) live in ${CLAUDE_PLUGIN_ROOT}/skills/install-zymtrace-profiler/reference.md.
Open with a short welcome before any commands or questions:
👋 Thanks for installing the zymtrace profiler! It's a low-overhead agent (<1% CPU, ~256 MB RAM) that ships profiles to your backend.
>
Stuck? Reach out: - Community Slack: <https://join.slack.com/t/zymtrace/shared_invite/zt-3fdidjufl-q~NHxDzQlzal2B9mujfaoQ> - Email: <[email protected]>
>
Tip — analyze GPU and CPU flamegraphs via MCP: once profiles start flowing, connect the zymtrace MCP to your agent (Claude Code, Codex, or Cursor) — see configure-zymtrace-mcp — and analyze GPU + CPU flamegraphs in natural language. Docs: <https://docs.zymtrace.com/mcp>>
Here's the plan: 1. Verify your tools and locate the backend gateway. 2. Decide CPU-only vs CUDA (GPU) profiling. 3. Install the DaemonSet (or Docker / binary). 4. Verify the agent is reporting.
>
Ready when you are.
If the user has already specified Helm / GPU / target, skip the roadmap and dive in.
values.yaml: <https://raw.githubusercontent.com/zystem-io/zymtrace-charts/main/charts/profiler/values.yaml>##### Claude runs
helm version --short && kubectl version --client
kubectl cluster-info | head -2
helm list -A | grep -i zymtrace # locate the backend releaseIf helm/kubectl are missing → point to install docs; do not install them. If no backend release is found anywhere → STOP and route to install-zymtrace-backend first; the profiler has nowhere to send data without it.
Recommend defaults `zymtrace` / `profiler` for namespace + release name. Full policy: shared/conventions.md.| Variable | Resolve by | ||
|---|---|---|---|
| Backend release & its namespace | `helm list -A \ | grep -i 'backend.*zymtrace'` | |
| Backend gateway service FQDN | <PREFIX>-gateway.<backend-NS>.svc.cluster.local:80 (in-cluster) or external ingress host | ||
| GPU nodes present? | `kubectl get nodes -l nvidia.com/gpu=true 2>/dev/null \ | wc -l` | |
| NVIDIA device plugin / GPU operator? | `kubectl get pods -A \ | grep -E 'nvidia-device-plugin\ | gpu-operator'` |
| CUDA runtime ≥ 12.x? (required for GPU profiling) | `kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.labels.nvidia\.com/cuda\.runtime-version\.major}{"\n"}{end}' \ | sort -u — must be 12 or higher. If labels are missing, kubectl exec into an existing GPU pod and run nvidia-smi, or ask the user to run nvidia-smi` on a GPU node themselves and report back (the skill usually can't SSH into the node). | |
| Existing profiler release? | `helm list -A \ | grep -i 'profiler.*zymtrace'` | |
| Customer-provided values file? | Ask — same rule as backend skills. Filename respected. |
Things you must ask:
Blockers (stop, surface):
Recommendations (note, proceed):
| Method | When |
|---|---|
| Helm DaemonSet (recommended) | Kubernetes — gives you the full chart with GPU support and framework metrics. |
| kubectl manifest | Kubernetes without Helm. CPU-only is fine; GPU profiling has fewer config knobs. |
| Docker | Single host (non-k8s) — same VM the workload runs on. |
| Binary + systemd | Bare-metal Linux (Slurm nodes, on-prem GPU servers). |
| Profiling | Template | What it gives you |
|---|---|---|
| CPU only | values/helm-cpu.yaml | eBPF unwinder for native, Python, Go, Java, Node. No CUDA libraries. |
| GPU (CUDA) | values/helm-gpu.yaml | Above + CUDA kernel profiling + GPU metrics + vLLM/SGLang/Triton metrics. Same template works for MIG slices and dense multi-GPU nodes. |
GPU profiling requires CUDA 12.x or higher. Older CUDA runtimes are not supported. If the customer's nodes are on CUDA 11.x, GPU profiling is a blocker — recommend CPU-only profiling until they upgrade the CUDA toolkit / driver. Architecture: AMD64/x86_64 and ARM64 are both supported.
If GPU nodes exist on the cluster (nvidia.com/gpu=true label) and CUDA ≥ 12.x, default-recommend GPU profiling. The agent works on CPU nodes too — cudaProfiler.enabled: true is a no-op when there's no NVIDIA driver.
Where does the agent send profiles? The format is always `<host-or-ip>:<port>` — never a URL with `https://` in front. TLS is controlled by the -disable-tls flag, not by the URL scheme.
| Setup | Set -collection-agent to |
|---|---|
| Same cluster as backend (default in templates) | <PREFIX>-gateway.<backend-NS>.svc.cluster.local:80 + -disable-tls |
| Different cluster, backend has TLS ingress | <gateway-host>:443 — remove -disable-tls |
| Different cluster, NodePort | <any-node-ip>:<nodeport> + -disable-tls |
Resolve <PREFIX> and <backend-NS> from the backend release: helm list -A | grep -i 'backend.*zymtrace'.
If mentioned, mirror ghcr.io/zystem-io/zymtrace-pub-profiler:<VERSION> into the customer's registry and set:
global:
imageRegistry: "<your-registry>"
registry:
requirePullSecret: true # only if registry needs authFull procedure: reference.md § Air-gapped install.
##### Claude runs
helm repo add zymtrace https://helm.zystem.io # idempotent if already added
helm repo update zymtrace
helm search repo zymtrace/profiler --versions | head -5Copy the matching template from values/ to zymtrace-profiler-values.yaml in the user's working directory. Don't ask the customer if they already have one — customers typically don't ship with a profiler values file (the backend often does, the profiler rarely does). Only respect a different filename if they explicitly volunteer that they have one (per shared/conventions.md).
Pick the template that fits and edit:
profiler.args[0] -collection-agent=... to the backend gateway endpoint.-disable-tls if pointing at an HTTPS endpoint.nodeSelector: nvidia.com/gpu: "true" matches your cluster's GPU label.Print the exact command + resolved values (release name, namespace, target backend gateway, GPU yes/no, image tag if pinned). Wait for explicit confirmation.
##### Claude runs
helm upgrade --install <REL> zymtrace/profiler \
--namespace <NS> --create-namespace \
-f <values-file> \
--reset-then-reuse-values \
--atomic --debugDefault <REL> = profiler, <NS> = zymtrace (same namespace as backend works fine — no conflict).
ERROR: daemonset.apps/zymtrace-profiler is not ready: pods are not ready → expected for ~30s while pods start. Wait. If it persists past 2 min, kubectl describe ds -n <NS> <PREFIX>-profiler.
ERROR: ImagePullBackOff → registry / version mismatch. Verify image tag at <https://github.com/orgs/zystem-io/packages>.
ERROR: pod CrashLoopBackOff with permission denied on /sys/kernel/debug → kernel doesn't support eBPF or the security context is being stripped. kubectl describe pod to confirm, and check securityContext.capabilities.add: [SYS_ADMIN] is honored.
##### Claude runs
bash ${CLAUDE_PLUGIN_ROOT}/skills/install-zymtrace-profiler/scripts/verify-profiler.sh <NS> <REL>Checks DaemonSet readiness, license validity in logs, GPU library extraction (if cudaProfiler.enabled), and connection-to-gateway evidence (no connection refused or dns lookup failed in last 50 log lines).
##### Claude runs
helm get values <REL> -n <NS> > <values-file>Recommend the customer commit the file: git add <values-file> && git commit -m "zymtrace: profiler install for <NS>/<REL>".
If GPU profiling was enabled, the agent is running but no workload is being profiled yet — workloads need to set CUDA_INJECTION64_PATH. The pattern is one env var; the variation is where you set it (Slurm prolog, k8s pod spec, Docker -e, ~/.bashrc).
Point the user at reference.md § Profiling real workloads — it covers:
hostIPC, --shm-size, VLLM_ATTENTION_BACKEND), with concrete Docker and Kubernetes manifests.Authoritative docs: <https://docs.zymtrace.com/install/profiler/cuda-gpu-profiler> and <https://docs.zymtrace.com/install/profiler/gpu-profiler-quick-start>.
CPU-only installs are done at this point — profiles for every process on every node start flowing within ~30 seconds.
For kubectl manifest, Docker, or binary+systemd installs, see reference.md § Other install methods. The same pre-flight rules apply; only Step 4 (install) differs.
Exit when ALL of the following are true (substitute <NS> / <REL> / <PREFIX>):
helm status <REL> -n <NS> reports STATUS: deployed.<PREFIX>-profiler has DESIRED == READY (kubectl get ds -n <NS>).Your license is valid until … OR streaming connection established (means the agent reached the backend).connection refused, dns lookup failed, or forbidden in the last 50 log lines of any pod.ls -la /var/lib/zymtrace/profiler on a GPU node shows libzymtracecudaprofiler.so (the agent extracted it for workload mounts).If any box fails, route by symptom:
troubleshoot-zymtrace-profiler.troubleshoot-zymtrace-backend, which walks the full backend ↔ profiler path.Both have first-pass diagnostic scripts to run before going manual.
zymtrace.example.com:443 — NOT https://zymtrace.example.com:443 or https://zymtrace.example.com/. Adding a scheme prefix makes the agent fail to parse the value and retry forever. TLS is controlled separately by presence/absence of the -disable-tls flag, not by the URL scheme.<backend-NS>, or pointed at a non-existent service) → agent retries forever, no profiles appear. Test with kubectl run -it --rm dns-test --image=busybox -- nslookup <PREFIX>-gateway.<backend-NS>.svc.cluster.local.nodeSelector or scope to GPU nodes.kubectl get nodes --show-labels to confirm.--nvml-path=<path> once the path is known (saves startup scan).helm upgrade / helm upgrade --install for this chart without --reset-then-reuse-values. See shared/conventions.md.-project= / ZYMTRACE_PROJECT in profiler args, env vars, or values templates. Always use the default project — the agent creates it automatically.privileged: true on the k8s pod, or sudo when running a bare binary. That's a workload-side change that needs explicit user confirmation. Recommend it for dev/staging deep-dives or on-demand production debugging. See reference.md § PC sampling.~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.