Evaluation

Concept of the skill

Evaluation is a gate, not a compliment: compare the requested outcome, produced artifact, evidence trail, and meaningful failure modes before deciding whether the work is done.

Coverage

This skill covers evidence-based evaluation of a completed artifact or task result:

• Skeptical critic stance: assume the work may be incomplete until evidence says otherwise.
• Goal alignment: compare the artifact to the original user request, acceptance criteria, and any stated constraints.
• Evidence inventory: name the checks, files, sources, tests, screenshots, traces, or manual inspections that support the verdict.
• 1-5 scoring: assign a numeric score with a one-sentence justification and a visible score ceiling when evidence is missing.
• Evaluation-revision loop: record required revisions, apply them, re-run focused verification, and rescore.
• Finding discipline: preserve every material gap with severity, evidence, required action, and status.
• Boundary routing: separate holistic evaluation from diff review, test planning, eval-suite design, debugging, and methodology design.
• Honest completion reporting: distinguish passed checks, unverified claims, accepted risks, deferred work, and blockers.

Philosophy of the skill

Agents are biased toward marking their own work done once the visible output looks plausible. That optimism is useful for momentum and dangerous at the finish line. Evaluation exists to slow the final decision just enough to ask: did this artifact actually satisfy the request, and can we prove it?

Modern agent-evaluation practice points in the same direction. OpenAI Evals treats evals as runnable checks for LLM systems. Anthropic frames an eval around tasks, trials, graders, traces, outcomes, and an evaluation harness, and recommends eval-driven development for agent capabilities. Google ADK separates final-response quality from trajectory and tool-use evaluation. NIST's AI Resource Center places testing, evaluation, verification, and validation inside broader AI risk-management practice. The shared lesson is simple: quality claims need observable criteria, not confidence.

This skill is the lightweight human-readable gate for a concrete deliverable. It does not replace automated eval suites or tests; it decides whether the available evidence is enough to accept the work, revise it, or block it.

Evaluation Inputs

Gather the smallest sufficient evidence packet before scoring.

If one of these inputs does not exist, say so and apply the score ceiling in the next section.

Score Rubric

Use the full 1-5 range. A score is a decision aid, not decoration.

Score Ceilings

Apply ceilings mechanically. Do not negotiate them upward with confidence language.

Evaluation-Revision Loop

1. Identify the artifact and the requested outcome in one sentence.
2. Inventory evidence: changed files, relevant sources, commands, checks, screenshots, traces, or manual inspections.
3. Load the domain skills or standards that govern the artifact type.
4. Score each dimension: request fit, evidence sufficiency, artifact quality, risk handling, and reporting completeness.
5. List every material finding with severity, evidence, required action, and status.
6. Revise the artifact for every required action inside the current scope.
7. Re-run focused verification that matches the revised surface.
8. Rescore and produce a final accept, accept-with-risk, revise, or block verdict.

Do not mark the task done between steps 5 and 7. Findings are work, not commentary.

Finding Format

Use this shape when the evaluation finds a gap:

Preserve all findings. Prioritization may order them; it must not hide them.

Dimension Checklist

Boundary Decisions

Source Notes

• OpenAI Evals grounds the idea that LLM-system quality can be measured through reusable evaluation tasks and benchmark registries.
• Anthropic's agent-eval guidance grounds tasks, trials, graders, traces, outcomes, harnesses, and eval-driven development.
• Anthropic's evaluation-docs guidance supports task-specific criteria, edge cases, automation where possible, and calibrated rubric grading.
• Google ADK grounds the distinction between evaluating final responses and evaluating trajectory or tool use.
• NIST AIRC grounds evaluation inside testing, evaluation, verification, and validation for AI risk management.

Verification

After applying this skill, verify:

• [ ] The original request and acceptance criteria were explicitly checked.
• [ ] The evaluated artifact and all changed or relevant files were named.
• [ ] Verification evidence was listed with commands, outputs, source reads, traces, or inspected artifacts.
• [ ] Relevant domain skills or standards were applied, or their absence was called out.
• [ ] A 1-5 score was assigned with score ceilings respected.
• [ ] Every material finding has severity, surface, evidence, required action, and status.
• [ ] Residual risks and unrun checks are stated instead of hidden.
• [ ] Any required revision was followed by focused re-verification and rescoring.

Do NOT Use When

Skill Graph context

<!-- skill-graph-context:start (generated — do not edit by hand) -->

Classification

• Subject: ai-engineering
• Public: true
• Domain: engineering/evaluation
• Scope: Evidence-based evaluation of completed agent tasks, implementations, documents, skill upgrades, and other deliverables against the original request, acceptance criteria, verification evidence, quality rubric, and residual risks before calling the work done. Portable across software, documentation, and agent-output review when the evaluator can inspect the request, artifact, and evidence. Excludes designing eval datasets or graders (eval-driven-development), line-by-line diff review (code-review), choosing test levels (testing-strategy), designing the process before work starts (methodology), and root-cause investigation of known failures (debugging).

When to use

• score this completed agent task against the original request, changed files, tests, and remaining risks before we call it done
• act as a skeptical critic and tell me whether this deliverable is actually complete or only technically working
• run a completion gate on this implementation: what is the 1-5 score, what evidence supports it, and what must be revised?
• evaluate whether this documentation update satisfies the acceptance criteria and preserves all required information
• after the focused checks pass, decide whether the skill upgrade can be marked complete without inflating eval claims
• review the final artifact and produce a scorecard with blockers, required revisions, residual risks, and verification evidence
• Triggers: evaluation-skill, critic-loop-skill, quality-gate-skill

Not for

• design a new eval dataset, grader, and hard negatives for this router
• review this pull request line by line for bugs and security issues
• choose unit versus integration versus end-to-end tests for this feature
• design the whole implementation methodology and quality gate sequence before work starts
• debug why the existing run failed in production
• author a new SKILL.md from scratch

Related skills

• Verify with: epistemic-grounding, code-review, best-practice, testing-strategy
• Related: methodology, debugging, best-practice, testing-strategy, code-review, epistemic-grounding, eval-driven-development

Concept

• Mental model: Evaluation is a gate, not a compliment: compare the requested outcome, produced artifact, evidence trail, and meaningful failure modes before deciding whether the work is done.
• Purpose: Prevent optimistic completion claims by making scores, evidence gaps, required revisions, and residual risks explicit before acceptance.
• Boundary: This skill applies after a concrete artifact or task result exists. It does not design eval suites, review a diff line by line, choose test levels, debug a failure, or design the process before implementation.
• Analogy: Evaluation is an inspection gate at the end of a production line: passing one measurement is not enough if the product misses the order, lacks traceability, or carries a known risk.
• Common misconception: Tests passing, lint passing, or a plausible final answer is not the same as completion. Completion requires fit to request, sufficient evidence, handled risks, and honest reporting.

Grounding

• Mode: universal
• Truth sources: https://github.com/openai/evals, https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents, https://docs.anthropic.com/en/docs/test-and-evaluate/develop-tests, https://adk.dev/evaluate/, https://airc.nist.gov/

Keywords

• evaluation, quality gate, deliverable review, completion review, evidence-based score, skeptical critic, score 1-5, evaluation revision loop, acceptance criteria review, done gate

<!-- skill-graph-context:end -->