bench — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited bench (Agent Skill) and scored it 96/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 1 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 1 flagged
The text {match} tells the agent to skip the normal "ask the user first" gate. Used adversarially it removes the human-in-the-loop check before destructive or sensitive actions, turning a normally-gated agent into a fire-and-forget executor.
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
Analyze per-turn NDJSON logs emitted by a running clawback proxy.
node benchmark/bin/analyze.js --out runs/report-$(date +%s) <turn-log-path>...Inputs may be individual .ndjson files or directories containing them. Each record's arm field determines whether it counts as treatment, passthrough, or treatment-ping (keep-alive overhead).
Turn-logs are produced by clawback when started with --turn-log <path>:
clawback --turn-log ./runs/turns.ndjson ... # treatment
clawback --turn-log ./runs/turns.ndjson --passthrough ... # baselineBoth arms write to the same file — arm label is embedded per record.
Written to the --out directory:
report.md — leads with the **billable input tokens reclaimed vspassthrough** headline (per-turn rate + bootstrap CI; full-rate quota clawback keeps off your bill — no pricing). The $/turn cost detail is a demoted appendix below it. Then two diagnostic sections:
stable system prefix, per knobProfile. 1 = one logical context maps to one Anthropic cache key (ideal); >1 (flagged ⚠️) means the same context was split across keys, each cold-starting Anthropic's cache — strip-ephemeral collapses this toward 1. This is the headline finding on hot loops where passthrough fragments but the stack does not.
treatment-ping during the gap, plus mean pings/turn. High coverage on a >5-min gap bucket alongside a high hit rate is keep-alive keeping the cache warm (the 15-min warmth test). Renders only when the log carries ping records.
report.csv — turn-level rows for downstream plotting. IncludespingsSincePrevTurn and msSinceLastPing per turn (ping coverage), alongside gapMs/gapBucket and the priced usd_estimate.
summary.json — machine-readable aggregates; the top-level tokensblock is the headline (baseline vs treatment billable totals + mean per-turn, reclaimedPerTurn + CI, pctLessPerTurn, reclaimedTotalIsProjected). The prefixFragmentation array carries the per-prefix key counts, and each arm stratum carries meanPingsSincePrevTurn + pingCoverageShare.
manifest.json — input file list, pricing hash, clawback version,wall-clock coverage per arm.
All gap, ping-coverage, and fragmentation metrics are derived analyzer-side from the existing turn-log fields (ts, sessionKey, systemStableKey, and the treatment-ping records) — no proxy change is needed to collect them.
Bootstrap CIs use a seeded PRNG; rerunning on the same inputs yields byte-identical reports (ignoring generatedAt timestamps).
per arm, ideally across multiple days).
report.md that's committed alongside the study.pricing-hash field in manifest.json to detect drift).
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.