ab — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited ab (Agent Skill) and scored it 96/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 1 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 1 flagged
The text {match} tells the agent to skip the normal "ask the user first" gate. Used adversarially it removes the human-in-the-loop check before destructive or sensitive actions, turning a normally-gated agent into a fire-and-forget executor.
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
Run .skills/ab/scripts/ab_block.sh from the project root. It is the single command that ties the validation harness together: capture (separately, once) → this (drive both arms + analyze + plot). It runs one arm at a time:
--state file** + a shared --turn-log (via .skills/monitor's run_monitor.sh --detach);
--turns N of load through it (--driver pty = real claude via.skills/drive; --driver replay = HTTP fixture via .skills/replay);
.skills/bench) and render charts(.skills/plot) into the --out directory.
A knob A/B is only honest if it measures what production does, and production forwards no salt. The harness used to prepend a per-arm --cache-salt block to system to force each arm cold — but that salt forced whole-body re-serialization and zeroed within-arm cache warming (salted arms measured ~0% warm vs a pristine A0 at ~98%), measuring a virtual system instead of the proxy. It is gone. Arms are isolated three production-faithful ways instead:
RESUMES a prior arm's persisted SESSION KEYs.
forwarded bytes (strip-ephemeral, 1h-TTL, mobile) already gets a distinct ANTHROPIC KEY, so it cannot read a prior arm's warm entries.
same real-turn bytes as passthrough, so per-arm --state does not change the content-addressed ANTHROPIC KEY. Two production-faithful guards, neither of which deletes a turn: (a) the analyzer reports cold-starts in their own "first" gap bucket, so a carry-over-warmed open is never pooled into the warm-loop buckets it's compared within; and (b) --cooldown-sec spaces serial arms past Anthropic's 5-min cache TTL so each arm's opening turn is a genuine cold-start (counted, exactly as a production user pays it after an idle), not a fake-warm read of the prior arm's cache — set ~330 for a byte-identical-arm headline. The analyzer's --warmup discard (drop each arm's first N turns) stays an opt-in diagnostic (default 0), not the standing guard: dropping opening turns also deletes legitimate production cold-start cost, so the default counts them.
This is driver-independent — the proxy forwards the arm's bytes the same way regardless of how the turn was produced.
The proxy labels every turn passthrough or treatment; this script writes one turn-log per arm and labels it into the analyzer as knobProfile <arm>. The analyzer pools all passthrough turns into the per-bucket baseline, then compares each treatment knobProfile against that baseline. So:
baseline every treatment arm is measured against.
--arms "A0 A2 A5"): each is labeled separately and compared against the shared A0 baseline in the same run.
# short L0 no-regression smoke, baseline vs recommended stack, on Haiku:
.skills/ab/scripts/ab_block.sh --profile L0 --turns 8 --driver pty \
--model claude-haiku-4-5-20251001
# headline L2 (5-30 min gaps) block, >=200 turns/arm for a reportable %:
.skills/ab/scripts/ab_block.sh --profile L2 --turns 200 --driver pty \
--model claude-haiku-4-5-20251001 --out runs/L2-headline
# clean HTTP replay arm (needs a CLAWBACK_OAUTH_TOKEN bearer + a captured
# fixture; never an API key):
CLAWBACK_OAUTH_TOKEN=… .skills/ab/scripts/ab_block.sh --profile L2 --turns 200 \
--driver replay --fixture benchmark/fixtures/ccode.json
# keep-alive warmth test: does A1 stay warm across a 15-min idle on the
# native 5m TTL?
CLAWBACK_OAUTH_TOKEN=… .skills/ab/scripts/ab_block.sh --driver replay \
--arms "A1 A0" --gap-sec 900 --turns 5 --out runs/keepalive-15mA0 passthrough baseline (forces all knobs off)A1 keep-alive onlyA2 1h-TTL only (nested-cache-control rewrite on — the moat)A3 1h-TTL + keep-aliveA4 strip-ephemeral onlyA5 recommended stack (keep-alive + 1h-TTL + strip-ephemeral)Each arm's knobs are set explicitly on the proxy command, not inherited from DEFAULTS, so the arm is the arm regardless of the operator's global CLAWBACK.md.
--profile L0|L1|L2|L3|L4 — inter-turn gap regime (default L0).--gap-sec N — fixed inter-turn gap in seconds; overrides --profile'sjittered range (forwarded to both drivers). For a precise idle, e.g. the 15-min keep-alive warmth test above.
--turns N — turns per arm (default 8; analyzer marks savingsinsufficient below 30/arm; target ≥200 for a reportable %).
--arms "A0 A5" — space-separated arm list (default "A0 A5").--driver pty|replay — load driver (default pty).--model ID — pin claude's model for the pty driver (cost control).--fixture PATH — replay fixture (default benchmark/fixtures/ccode.json).--prompts PATH — pty prompt file (default benchmark/prompts/coding.txt).--settle-sec N — pty quiescence threshold (default 8).--host, --port — proxy bind (default 127.0.0.1:8787).--out DIR — output dir (default runs/ab-<timestamp>).--no-plot — skip chart rendering.--driver pty runs the real claude binary once per arm against your own Anthropic limits. Keep --turns small and pin --model (e.g. claude-haiku-4-5-20251001) for plumbing / no-regression runs; reserve a larger, default-model run for the headline numbers. --driver replay also spends tokens but forwards a CLAWBACK_OAUTH_TOKEN bearer (never an API key) and is cheaper to script.
In the --out directory:
turns.<arm>.ndjson — one turn-log per arm (labeled into the analyzer asknobProfile <arm>; the passthrough arm pools into the baseline).
proxy.<arm>.log — each arm's proxy stdout+stderr.state.<arm>.json — each arm's own proxy state file, so a restart neverresumes the previous arm's persisted SESSION KEYs (the clawback-side carry-over guard described above).
report.md, report.csv, summary.json, manifest.json — the analyzeroutputs (.skills/bench).
charts/ — the four §9 marketing charts (.skills/plot).Read the headline with sed -n '1,40p' <out>/report.md.
Capture a fixture once with .skills/capture. This skill is the runner; .skills/{drive,replay} are the load arms it calls, .skills/bench is the analyzer, .skills/plot is the chart renderer. To run an arm by hand instead, see those skills.
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.