name: aoa-eval-design
description: Design a bounded local eval suite, prompt set, report, or validator plan after selection finds no adequate existing surface, using deterministic checks first and trace or rubric review only when needed. Use when a local eval needs positive, negative, collision, or regression cases around a clear behavior or invariant. Do not use when an existing eval should simply run, when only a local intake packet is needed, or when central aoa-evals proof doctrine is the target.
license: Apache-2.0
compatibility: Designed for Codex or similar coding agents with repository file access and an interactive shell. Network access is optional and only needed when repository validation or referenced workflows require it.
metadata:
aoa_scope: core
aoa_status: scaffold
aoa_invocation_mode: explicit-preferred
aoa_source_skill_path: skills/core/engineering/aoa-eval-design/SKILL.md
aoa_source_repo: 8Dionysus/aoa-skills
aoa_technique_dependencies: AOA-T-0015,AOA-T-0017,AOA-T-0007
aoa_portable_profile: codex-facing-wave-3
aoa-eval-design
Intent
Use this skill to design a bounded local eval suite, prompt set, report, or validator plan after existing surfaces have been inspected and found insufficient.
Trigger boundary
Use this skill when:
- a local eval needs design from a clear failure mode or missing invariant
- deterministic tests, validators, or scripts should be planned before rubric or
transcript graders
- positive, negative, collision, or regression cases need to be shaped for a
local repo
- the output is a draft suite or report, not central proof acceptance
Do not use this skill when:
- an existing eval should simply run now
- the right next artifact is only an intake pressure packet
- the invariant is unknown and discovery is still needed
- the user asks for central
aoa-evals proof doctrine changes
- selected failure mode, target behavior, invariant, or trigger boundary
- local repo tests, scripts, validators, fixtures, generated surfaces, and
evals/ port
- evidence refs and rejected existing evals
- central
aoa-evals standards when relevant
Outputs
- draft suite structure, report plan, fixture set, or validator design
- positive, negative, and regression cases
- deterministic checks first, rubric or trace review only when needed
- proof limit and owner handoff
Procedure
- restate the behavior or invariant to evaluate
- choose the lowest-level deterministic checks that can constrain it
- add positive and negative examples; add collision cases for trigger or router
behavior
- add trace or rubric review only where deterministic checks cannot cover the
behavior
- place the draft in the local eval port or report surface named by the repo
- keep central acceptance language out of the draft
- hand off to
aoa-eval-apply when a runnable check exists
Contracts
- design starts after selection fails, not before
- local design remains local until central review accepts it
- deterministic checks are preferred for coding-agent behavior
- transcript and rubric grading are supporting layers when objective checks are
insufficient
Risks and anti-patterns
- designing a suite around vague success criteria
- using only positive examples
- creating rubric-only evals where a script or test can check the behavior
- making one local report sound like system-wide proof
Verification
- confirm the target invariant or behavior is explicit
- confirm positive, negative, and regression cases exist or are deliberately
deferred
- confirm deterministic checks were considered first
- confirm local owner path and proof limit
- confirm next runnable command or review route
Technique traceability
Manifest-backed techniques:
- AOA-T-0015 from
8Dionysus/aoa-techniques at 1a7d146957108ecefc24219c7d56357c5a4a2c2c using path techniques/proof/skill-support/contract-test-design/TECHNIQUE.md and sections: Intent, Inputs, Outputs, Core procedure, Contracts, Risks, Validation - AOA-T-0017 from
8Dionysus/aoa-techniques at 1a7d146957108ecefc24219c7d56357c5a4a2c2c using path techniques/proof/skill-support/property-invariants/TECHNIQUE.md and sections: Intent, Inputs, Outputs, Core procedure, Contracts, Risks, Validation - AOA-T-0007 from
8Dionysus/aoa-techniques at 1a7d146957108ecefc24219c7d56357c5a4a2c2c using path techniques/proof/evaluation-chain/signal-first-gate-promotion/TECHNIQUE.md and sections: Intent, Inputs, Outputs, Core procedure, Contracts, Risks, Validation
Adaptation points
- local fixture format
- local suite/report naming
- central review handoff