skills/eval_verdict_write/SKILL.md

Status: live Type: advisor-return skill Inputs: a populated manifest plus the artifacts it references Output: a verdict file at <workdir>/<task>_EVAL_VERDICT_v1.md Rule reference: see docs/FLAT_BOOLEAN_RULE.md §7 (closing-state semantics) and §8 (verdict file shape).

Purpose

Read a manifest the executor has populated with EVIDENCE. For each row, declare PASS, FAIL, or SKIP. Write a verdict file that downstream consumers can grep for OVERALL=PASS or OVERALL=FAIL.

Input contract

1. manifest_path: path to a populated eval manifest. Every row must be

in a final state: [x] with non-empty EVIDENCE, [SKIP] with non-empty NOTE, or [ ] with non-empty NOTE describing the blocker.

1. artifacts_root: directory containing the artifacts the manifest's

verification fields reference. May be unreadable (advisor sees only the manifest); the skill handles read-only mode.

1. mode: one of verify (re-run each verification target where

possible) or trust (accept EVIDENCE as recorded).

Output contract

A markdown file at <workdir>/<task_name>_EVAL_VERDICT_v1.md with:

• Top header naming the manifest path it judges.
• Mode line declaring verify or trust.
• Counts: PASS_COUNT, FAIL_COUNT, SKIP_COUNT, OPEN_COUNT, OVERALL.
• One line per C-NNN row in the manifest.

The line format:

C-NNN: PASS | FAIL | SKIP | OPEN -- <one-line basis>

OVERALL is PASS only when:

• FAIL_COUNT == 0
• OPEN_COUNT == 0
• Every [x] row has non-empty EVIDENCE
• Every [SKIP] row has non-empty NOTE

Any other state yields OVERALL=FAIL with the failing condition cited at the top of the file.

Procedure

1. Read the manifest. Parse each row into `(state, id, phase, subject,

verification, expected, evidence, note)`.

1. Run the integrity gates first. If C-FMT-ROW-SHAPE or

C-FMT-NO-STACK fail on the manifest as it stands, the verdict is OVERALL=FAIL with reason "manifest violates its own format rules"; the advisor stops here.

1. For each row, validate state.

• [x] row with empty EVIDENCE -> FAIL with reason "checked but no

evidence recorded".

• [SKIP] row with empty NOTE -> FAIL with reason "skipped without

justification".

• [ ] row with empty NOTE -> OPEN (not FAIL; the row is incomplete

but the executor may not have intended to close it).

• [ ] row with non-empty NOTE describing a blocker -> SKIP with the

blocker as basis.

1. In verify mode, re-run each row's verification target.

• File-existence rows: stat the path.
• Grep rows: re-run the grep with the same pattern and scope.
• Line-existence rows: grep the named log for the named pattern.
• Compare result to <expected>. Match -> PASS. Mismatch -> FAIL with

the diff captured.

1. In trust mode, accept EVIDENCE without re-running. Useful when the

advisor session lacks access to the artifacts. The verdict file's mode line declares trust; downstream consumers know this is a read-only verdict.

1. Compute counts and OVERALL.

1. Write the verdict file. Use the same convention as the manifest:

one line per row, no narrative, no reasoning beyond the one-line basis.

Failure modes

Related

• skills/eval_criteria_create/SKILL.md produced the input.
• templates/verdict_template.md is the empty skeleton this skill writes.
• tools/lint_no_stack.sh runs the integrity gates this skill verifies.
• docs/FLAT_BOOLEAN_RULE.md §7 defines the closing-state semantics.
• docs/FLAT_BOOLEAN_RULE.md §8 defines the verdict file shape.