Skill Eval

Overview

This skill owns the repository's skill-eval test operation. It is the eval-focused alternative to skill-creator: it drives skills/skill-eval/scripts/eval_runner.py, keeps the executor and grader as separate agents, surfaces run time and token usage, and verifies results honestly before they are reported.

Eval execution is always structurally separated. A single agent must never both produce an answer and grade it. The runner is the authoritative eval mechanism: do not hand-run prompts or hand-record results, and never estimate or hand-type metrics. These rules are enforced in the runner's code; this skill is the always-on instruction to route eval work through that runner so the enforcement actually applies.

When to Use

• Running, grading, aggregating, or reporting a skill eval suite under evals/.
• Verifying a with_skill vs without_skill pass-rate result before reporting

it as a real delta.

• Deciding eval workspace placement, model passthrough, run bounds, or metric

display for an eval run.

• Reading execution time or token usage from benchmark.md or the run summary.

When Not to Use

• Creating or editing the eval suite JSON schema or the assertion model.
• General skill creation, description/trigger optimization, or eval-viewer work —

those remain with skill-creator.

• Quality decisions about what to change from results — that is skill-quality's

role. This skill owns the execution and verification of a run, not the contract-change call.

Executor and Grader Stay Separate

This is a hard rule grounded in a real prior incident where execution and grading collapsed into one agent and the run scored its own output.

• The executor prompt carries the task only and no assertions. The grader prompt

carries the recorded executor output plus the assertions and must return a structured verdict. The runner derives pass/fail from the grader subprocess, never from text the executor wrote about its own output, and the grader grades the whole recorded output, not a sub-artifact.

• There is no "grade inline" shortcut. Never instruct an agent to execute and

grade in the same turn, even to save a subprocess, and never substitute a single-agent path that produces and scores an answer together. Executor and grader run as separate subprocesses with separate, clean invocations.

• The grader runs with a clean environment that strips CLAUDECODE, so a nested

grader does not inherit executor state.

• Driving eval work through skills/skill-eval/scripts/eval_runner.py is what makes this

separation hold. Eval work performed outside the runner does not get the code-enforced separation, so route eval runs through the runner.

Evaluation Workspace

• Keep eval definitions under evals/<skill-name>/.
• Store generated eval run outputs under evals/<skill-name>/workspace/<agent>/.
• Do not create generated eval workspaces next to skill packages under skills/.
• Do not commit generated eval workspaces unless the user explicitly asks for

them; they are local artifacts covered by .gitignore.

Shared Eval CLI

• Use python3 skills/skill-eval/scripts/eval_runner.py for repo-level skill eval runs. It has

three commands: validate, run, and report.

• The runner drives execution itself. run executes the bounded matrix end to

end: for each eval x config x run it spawns a fresh executor subprocess with the prompt only, then a fresh grader subprocess with a clean environment and only the executor output (plus any plan artifact the executor wrote to the designated path) and the assertions, then aggregates a with_skill vs without_skill raw pass-rate comparison. No agent hand-runs prompts or hand-records results.

• The provider selector is a registry. --agent selects a registered provider;

claude and codex are built in, and another agent is added as an adapter. The core path (execute, grade, compare, aggregate, report) is provider-neutral and must work on Codex; Claude-only precision such as opt-in metric capture is additive, and other providers skip it.

• An optional --model is passed through to the selected provider CLI verbatim

(whatever model name that CLI accepts); it is validated before any subprocess launches, applied to both the executor and grader, and recorded in the iteration manifest and benchmark. Absence means the provider's default model, never an injected or guessed model id.

• All input validation runs before any subprocess launches: suite shape, the

authoritative with_skill skill source, provider availability, and run bounds. Invalid input exits non-zero with zero subprocess launches. An empty suite is not an error: it exits 0 with an explicit empty result and zero subprocess launches.

• Total work is bounded. --runs is capped at 1..5 (default 1), --timeout

bounds each subprocess (default 600s), and --concurrency caps concurrent provider subprocesses (1..16, default 4). A timed-out or failed executor is recorded as a failed run, not a pass, and the grader is skipped for it; there are no retries.

• Metrics are never hand-typed or estimated. No flag injects a token or duration

value. When a provider exposes machine-readable usage (for example claude -p --output-format json), the runner captures it into metrics.json with its source; when a provider does not, absence is recorded as absence, never a placeholder number.

• with_skill runs must use the authoritative skills/<skill-name>/SKILL.md

source package. The runner resolves --skill-path from the repo root and, for every provider, rejects .agents/skills snapshots, .claude/skills links, files not named SKILL.md, and paths outside skills/<skill-name>/. The executor prompt instructs reading that source directly and not substituting a host skill tool, snapshot, link, or cached copy.

• Provider subprocesses run from a per-run sandbox copy of the current repository

tree outside the source checkout, not from the source checkout or a nested directory inside it. The runner excludes host-local and generated state such as .git, .agents, .claude, .codex, evals/*/workspace/, node_modules/, and __pycache__/, initializes a throwaway git repository when git is available, remaps the with_skill skill path to the sandbox copy, sets provider cwd/PWD to the sandbox, and records sandbox details in run.json. Executor or grader edits, installs, and commits must stay inside that sandbox; sandbox git initialization failure is recorded, never worked around by running in the real repository. Sandbox isolation prevents new writes from contaminating the source checkout, but it does not prove the source fixtures were clean before copy; the runner records declared fixture-root dirtiness before and after execution as a sanity-check anomaly.

• Executor and grader stay separate. The executor prompt carries the task only

and no assertions; the grader prompt carries the recorded output plus the assertions and must return a structured verdict. The runner derives pass/fail from the grader subprocess, never from text the executor wrote about its own output. The grader grades the whole recorded output, not a sub-artifact.

• The executor prompt names one designated artifact path inside the sandbox,

config-symmetrically for both with_skill and without_skill, so a skill whose deliverable is a written file (an implementation plan, spec, or other primary Markdown artifact) is not scored only on its concise chat summary. After execution the runner copies any file written to that path into outputs/plan.md under the run dir and folds its contents into the grader's recorded output under a delimited Written Plan Artifact section, capped and with truncation recorded, never silently dropped. Runs whose executor writes no such file (the deliverable is the chat reply) keep the grader prompt unchanged and record written_artifact.captured = false. The designated path does not instruct the executor how to structure the artifact, so it adds no target-behavior leakage.

• The grader returns a structured, schema-constrained verdict. Verdicts are keyed

by the assertion's 1-based id ({"verdicts": [{"id", "passed", "evidence"}]}), not by an echoed assertion string, so a grader cannot break grading by re-numbering or paraphrasing the assertion text. The runner requests provider-native structured output where the CLI supports it (codex --output-schema <file>, claude --json-schema <schema>) and carries the same contract in the grader prompt for providers that do not; the legacy text-keyed {"expectations": [{"text", ...}]} shape is still accepted. A grader output the runner cannot parse into a verdict list is recorded as grader_unparseable with pass_rate absent and excluded from the comparison, never scored as a real 0%.

• Standard command sequence:

python3 skills/skill-eval/scripts/eval_runner.py validate evals/vibe-planning/evals.json
python3 skills/skill-eval/scripts/eval_runner.py run evals/vibe-planning/evals.json --agent codex --config with_skill,without_skill --runs 1
python3 skills/skill-eval/scripts/eval_runner.py report evals/vibe-planning/workspace/codex/iteration-1

• run writes iteration_manifest.json and, for each run, prompt.md,

grader_prompt.md, outputs/, grading.json, metrics.json, and run.json under evals/<skill-name>/workspace/<agent>/iteration-N/, plus benchmark.json and benchmark.md at the iteration root. run.json records the external sandbox repo path for audit. benchmark.json/benchmark.md carry per-eval and overall raw pass rate, the with_skill/without_skill comparison, the execution-metrics summary, and a sanity_checks section flagging infrastructure failures, scored-0% cells, candidate-below-baseline cells, and dirty declared fixture roots for review.

• report <iteration-dir> re-renders benchmark.md from benchmark.json. It

does not start a server, open a browser, bind a port, write a PID file, or leave a background process.

• grading.json includes every assertion (common_assertions then per-eval

expectations) exactly once, in order, each with text, passed, and evidence. An assertion the grader omits is recorded as failed.

• Generated eval workspaces are local .gitignore artifacts; do not commit them

unless the user explicitly asks.

Execution Metrics (executor-only)

The run stdout summary and benchmark.md show per-config execution time and token usage for at least the claude provider.

• The displayed values are the executor subprocess metrics, labeled

executor-only so grader scoring cost is excluded and the values are not read as total run cost. The executor is the subprocess that runs the skill (with_skill vs without_skill), so the executor-only metrics are the skill's own performance signal; the with_skill vs without_skill delta is the meaningful reading.

• Aggregation is computed from the existing per-run metrics, so report

re-renders older benchmark.json files that predate the metric rows. A per-config mean is shown with ± stddev only when more than one run captured a numeric value for that metric, so a single captured value or the --runs 1 default never produces a misleading spread.

• Uncaptured or partial provider metrics are shown as absent with a reason, never

a placeholder. This includes a claude run whose output was not a JSON envelope, individual missing sub-fields on an otherwise captured run, and providers such as codex whose metric capture is not enabled. Never read an absent metric as 0.

Result Verification and Reporting

• An agent that supervises an eval run must verify the result before reporting

it. A run that finished without a crash is not the same as a clean result; do not present a with_skill/without_skill delta as normal completion until the verification below passes.

• After every run, read the Sanity checks status (printed to stdout and

written to benchmark.md) and the error_run_count. Treat any of the following as a stop-and-verify condition, not a pass: a REVIEW REQUIRED sanity status, error_run_count > 0, any grader_unparseable/grader_failed/executor_failed/timeout status, any scored-0% cell, any candidate-below-baseline cell, or any dirty source-fixture signal before or after execution.

• For each flagged cell, open the recorded outputs/output.txt and

outputs/grader_output.txt and determine whether the cause is the executor output, the grader verdict, or the runner before attributing it to the skill. A grader-side or runner-side failure must not be reported as a skill score. Fix the cause and re-run, or report the cell as an excluded infrastructure failure with the reason; never silently fold it into the headline number.

• Always report a summary, not just the headline delta. The summary states: agent

and model, configs and runs, scored versus excluded run counts, overall with_skill/without_skill pass rate and delta, and the sanity-check status with any flagged cells (or an explicit "no anomalies"). If any cell was excluded or re-graded, say so and give the corrected reading.

• Do not claim an improvement, regression, or delta as proven from a run that has

flagged anomalies or excluded cells until they are explained or the run is repeated cleanly.

Local Snapshots and Release

• skill-creator exists only as a managed snapshot under .agents/skills/.

Read it for reference, but do not edit, copy, or commit it. Make eval and skill edits against tracked skills/<skill-name>/ packages.

• Do not bump any skill version or assign a release version while running evals.

Record notable changes under ## [Unreleased] in CHANGELOG.md until the user instructs a release.