Skill Improver

/skill-improver <mode> <target> [--opts]. <target> is a skill name, a SKILL.md path, --all, or a glob ('vllm-*'). No mode → Auto. No target (except ages, batch) → ask.

Flags are mode-specific: --iterations N, --probe-budget N, --runs-per-query N, --missed "<query>" (trigger; repeatable — seeds a user-reported miss as a should-trigger query), --against <git-ref> (outcome).

Auto

1. Read the evidence, no edits: the Freshened: line in

references/sources.md; one gh release list --repo <o>/<r> --limit 5 per upstream whose version the skill states, compared against the version in the skill's text (never assume "no new release"); git log --format='%ad %s' --date=short -- <skill-dir> | head -20; references/improvement-backlog.md §Open (has a named blocker arrived?); disable-model-invocation and references/trigger-evals.json; how fast the subject moves.

1. Decide each step:

1. Print the plan — one line per step: run or skip, with the evidence. Run in

table order. Nothing due is a valid result: say so and stop.

1. Report what ran, what changed, the commits, what was skipped and why.

Fix

The cheapest change that measurably improves answers: find what the skill lacks, from the answers it produced. Prefer it over improve for a skill with eval cases.

1. Run outcome-eval.py <skill-dir> (no --against). It ends by listing the

weak cases — the skill loses to the without arm or scores under 0.7 — with each failed assertion and how many runs failed it.

1. For each weak case, read the failed answers in the printed current.json

(grader evidence) and name the cause: fact missing from the skill; fact only in sources.md or buried; skill text wrong; assertion stale; judge noise (the answer was right). Cite the moment: case, run, and the quoted answer line where it goes wrong. A failure in 1 of 3 runs is noise until that moment shows the mechanism; no cited moment, no fix.

1. One cause per commit. Verify every fact against the primary source; put a

known failure and its fix where the agent reads it; correct an assertion only with source evidence, in its own commit that cites the source. The report lists every changed assertion, before and after — a score that rose because a check got easier is not a better answer.

1. Re-run just those cases: --case '03-*,06-*' (comma-separated globs). Keep a fix when the case rises,

stays above the without arm, and the re-run answers no longer show the cited moment; revert otherwise. A rise with the moment still present is noise.

1. Done = every weak case fixed, shown to be noise, or recorded in the backlog

with its blocker; committed; references/lessons.md read first and given its end-of-pass entry (or none). The step-1 run is the benchmark (--write-benchmark); re-run in full only when several cases changed.

Improve

Score, apply ONE change, re-score cold, keep +3 or more; keep +1/+2 only when a second cold score confirms it or the change also simplifies; keep Δ0 only when it fixes a verifiable defect or simplifies; revert the rest. Stop at 90+ with no dimension below 7, when the ceiling is mapped (improve-loop.md Phase 5), or after 10 iterations. Re-scoring an unchanged skill moves the total 2–3 points (up to 6), so +2 is noise. Phases, decision rules and stop conditions: references/improve-loop.md.

• One change per iteration. State it in 10 words with one verb; an "and"

means two iterations. A move that starts rewording prose is two changes. A change plus the pointer and source row it needs is one change.

• Two lenses, never mixed in one hypothesis. INV = how the agent works

(procedure, tool use, done conditions, scripts). DOM = what a practitioner knows (facts, failure mechanisms, decision rules). improve-loop.md Phase 2.

• The backlog records blockers only. Every Open entry in

<skill>/references/improvement-backlog.md names the absent thing (a ruling, a credential, an unreleased version, a measurement nobody can run now) and passes the absence checks in references/backlog-format.md. Effort is not a blocker: do it before the pass ends.

• **Read references/lessons.md in Phase 0; write at most one entry at the

end.** Each kept change gets an attribution verdict in the landing commit (improve-loop.md Phase 7).

• A pass ends with work, not a report. Done = every keep applied, blind

score at baseline (on a snapshot, with the baseline commit recorded) and at stop, the A/B comparator verdict, an outcome run when the skill has eval cases, committed with the backlog in the same commit, resolved items deleted from Open. Zero discards = stopped early.

Rules for every mode

• Run without pausing between iterations; print status lines.
• State the spend before any fan-out of subagents or claude -p probes:

calls, model, rough dollars from scripts/model-rates.json. Pin a cheap model on mechanical work (graders, row probes).

• Git is the state. Commit each kept change; git diff before reverting.

Never rm: revert with git, put temp dirs and snapshots under a fresh mktemp -d and leave them.

• Classify overlap before deleting. Only DUPLICATE is deletable;

INTENTIONAL_DETAIL (summary in SKILL.md, detail in references/) and RELATED_BUT_DISTINCT stay. scripts/dedup-fleet.py produces the table; references/improvement-patterns.md §Pattern 6.1 reads it.

• Preserve the author's domain knowledge. Change how the skill teaches, not

what it teaches.

• Write for the agent; history goes to `sources.md`. SKILL.md and task

references state the current rule and the action. Where a fact came from, why it changed and when it was checked go in references/sources.md; SKILL.md carries one pointer to it. Keep a version or issue number in agent text only where the agent acts on it ("on 0.7.0 every command fails — upgrade"). A known failure is agent text, never history: the symptom, the affected versions and the fix stay in SKILL.md/references even when the citation lives in sources.md — an agent never loads sources.md mid-task. Existing history is not a defect by itself; move it when it is in the diff.

• A failed measurement is NO SCORE, never 0. A timed-out probe, a dead

scorer, an errored eval run: exclude it from the denominator and say what is missing. An incomplete run is never compared with a complete one; a pass that could not measure its mode's evidence is stopped early.

• The skill outranks training data. Never change an external claim

(version, date, flag, model name, SHA) from memory — verify online and cite, or drop the change. Wanting to lower a version or move a date back is the stale-prior alarm: check first, expect the skill to be right. A subagent's "wrong version" finding is applied only after the primary source confirms it. Open a new citation and confirm its title, date and the attributed figure before writing it down.

Fleet checks

Run each over the whole tree, not only the skill being edited. Each exits non-zero on findings and has --selfcheck. Read the non-findings column before acting; detail in references/fleet-checks.md.

Blind validation

Scorer — the blind-scorer agent, spawned with a two-line path tail. Omit model and effort (the definition pins them). Run at baseline (on a snapshot, in the background) and at stop. Print the bias table: every dimension where self and blind differ by 2+.

A/B comparator — decides the pass. Absolute scores cannot show whether a pass helped. scripts/ab-setup.py <baseline-ref> <skill-dir> extracts baseline and final into a mktemp -d without evals/ or improvement-backlog.md, with one timestamp on every file; label them DIR A / DIR B by coin flip; spawn three skill-comparator agents from outside the repo (inside, they see recent commit subjects). Majority gives IMPROVED / NO CHANGE / REGRESSED; a tie is NO CHANGE; REGRESSED outranks any score gain — revert the responsible iteration. Commands and leakage classes: references/blind-validation.md.

Outcome

Rubric and comparator judge the skill's text. outcome measures whether the skill makes the model's answers better:

python3 ${CLAUDE_SKILL_DIR}/scripts/outcome-eval.py <skill-dir> [--against <git-ref>] [--write-benchmark]

It converts <skill>/evals/evals.json into claude plugin eval cases (one grader per assertion), runs each with and without the skill, and with --against the other version too. Prompts force invocation, so the number is about content; trigger mode measures triggering. Defaults: 3 runs, Sonnet, Sonnet judge, \$20 cap.

• Price it first: \$0.10–0.30 per with-skill run and \$0.05–0.20 per without

(larger skills cost more), times cases × runs × arms; --against adds an arm. The run ends by printing what it actually used: tokens per model (grader calls excluded) and, on a subscription, the plan's 5-hour and weekly usage before and after — read from the runs' own logs, whole percents.

• Read failures before concluding. Three runs and a 2-of-3 judge vote are

noisy: a per-case difference of one run, or a total gap under ~0.15, is noise until the failed answers are read and found worse.

• A skill that does not beat the without arm on a case carries nothing the

model lacks there — a deletion candidate, confirmed with floor.

• --write-benchmark writes evals/benchmark.plugin-eval.json, which

eval-evidence.py reads for the Dim 10 gate.

• Fewer than 8 cases cannot resolve a delta: grow them with

scripts/grow-evals.py first.

Score

1. Read the target and references/quality-rubric.md.
2. Run python3 ${CLAUDE_SKILL_DIR}/scripts/eval-evidence.py <skill-dir>; take

Dim 10's cap from it, never judge the delta by eye.

1. Score all 10 dimensions with the rubric's template; print the table; name the

lowest dimension and the single highest-impact fix.

1. Dim 9 capped by sources.md staleness → recommend freshen. Fewer than 8

eval cases → recommend scripts/grow-evals.py.

1. Stop. Do not start the loop unless asked.

Freshen

Verify every sources.md row — delegate rows to cheap subagents in one background wave (web-searcher for web/gh rows, Explore for local clones). Apply one verified finding at a time. End by writing the single Freshened: <date> header stamp; an unreachable row gets an inline exception note. A partial pass keeps the old stamp. Workflow F0–F6, probe templates and classification: references/freshen-patterns.md.

Trigger

Metric: trigger rate on an eval set of should- and should-not-trigger queries, 60/40 train/test, 7 runs per query, test scores blinded, description ≤1024 characters. Probe: scripts/probe-trigger.py. Eval set: <skill>/references/trigger-evals.json, [{"query", "should_trigger", "source", "bucket"}], buckets explicit / implicit / contextual / negative. Workflow T0–T7: references/trigger-patterns.md.

Floor

python3 ${CLAUDE_SKILL_DIR}/scripts/knowledge-floor.py --skill <name> [--extract] (about \$6 for 20 claims on haiku, sonnet and opus); fleet: scripts/floor-fleet.py --root <dir>. Read-only. Classify the skill first: on an encoded-preference skill a high floor is expected, not a delete list. KNOWS is a candidate, never a licence to cut; CONFLICTS never means the skill is wrong. references/floor-patterns.md.

Ages

Run scripts/staleness-report.py [<glob>|<root>], print its table verbatim, then one sentence naming the stalest bucket and the next freshen target. No probes, scoring or edits. cases marked ! = fewer than 8 eval cases; open = backlog Open items — report it when the fleet total moved.

Batch

scripts/scan-skills.sh lists targets. Baseline-score each, run worst first, cap 5 iterations per skill. The batch is done when every listed skill has a summary row (skipped, crashed and capped included). With the Workflow tool opted in, use scripts/batch-workflow.js (args: ["keda", "helm", ...]); no agent there does git — commit per skill after review. Size fan-outs to 20 concurrent subagents and the workflow size guideline.

Files

Why a rule exists and when it changed: references/sources.md.