Write Judge Prompt

Create .kensa/judges/*.yaml files with structured evaluation criteria.

Lifecycle

Setup → Design → ► Calibrate → Execute → Diagnose → Iterate

This skill is conditional, only needed when inline criteria: in scenario YAML produce inconsistent or wrong judge verdicts. You should already have scenarios in .kensa/scenarios/. After writing judge prompts, wire them into scenario files and run kensa eval to test.

When to use this

Use inline criteria: in scenario YAML for objective, simple evaluations:

• "The response mentions Tokyo"
• "The agent called the correct tool"

Use structured judge prompts (this skill) when:

• The pass/fail boundary is subjective and needs explicit definitions
• The judge is producing inconsistent or wrong verdicts
• You need few-shot examples to calibrate the judge
• Multiple scenarios share the same evaluation standard

If you don't have scenarios yet, use generate-scenarios first.

The four components

Every structured judge prompt has four parts. When writing the YAML, load ${CLAUDE_SKILL_DIR}/references/judge-prompt-schema.md for the full schema.

1. Criterion

One thing being evaluated. Not two, not three. One.

criterion: Email tone matches the client persona

Not: "Email is good and well-formatted and uses correct tone"

2. Pass definition

Explicitly define what constitutes a pass. Be concrete, no wiggle room.

pass: |
  Language appropriate for the client type:
  - Luxury: formal, exclusive features, premium positioning
  - First-time: warm, educational, avoids jargon
  - Investor: data-driven, ROI-focused, concise

3. Fail definition

Explicitly define what constitutes a fail. Mirror the pass definition.

fail: |
  Tone mismatched to persona:
  - Casual slang for luxury clients
  - Heavy financial jargon for first-time buyers
  - Overly emotional language for investors

4. Few-shot examples

2-4 examples with output, label, and critique. Include at least one clear pass, one clear fail, and one borderline case. Borderline examples teach the judge nuance.

examples:
  - output: "Dear Mr. Harrington, I am pleased to present an exclusive listing..."
    label: pass
    critique: "Formal salutation, luxury positioning language throughout."
  - output: "Hey! Check out this awesome place, it's got a pool and stuff!"
    label: fail
    critique: "Casual slang inappropriate for luxury buyer."
  - output: "Hi Sarah, I found a property that might be a great fit..."
    label: pass
    critique: "Warm greeting, relatable terms. Matches first-time buyer tone."

Critiques must be detailed, not terse. They set the bar for the judge's own reasoning.

Write the file

Save to .kensa/judges/{name}.yaml:

mkdir -p .kensa/judges

Use a descriptive slug: tone-match, sql-accuracy, safety-check.

Wire it to the scenario

In the scenario YAML, replace criteria: with judge::

# Before
criteria: The email should match the client persona tone

# After
judge: tone-match   # loads .kensa/judges/tone-match.yaml

criteria and judge are mutually exclusive. Setting both raises a validation error.

Test the judge

Run the scenario and inspect the judge's reasoning:

kensa eval -s <scenario_id>
kensa report --format json

Read the judge_result.reasoning field. If the judge misses failures or flags passing outputs, adjust:

• Sharpen pass/fail definitions: make boundaries more explicit
• Add or swap examples: especially borderline cases
• Split the criterion: if one judge evaluates multiple things, decompose

Repeat until the judge's verdicts match your expectations. Then run kensa eval for the full suite.

Code-based checks first

Exhaust deterministic checks before reaching for an LLM judge. Many criteria that seem subjective reduce to code:

• Format validation → output_matches with regex
• Required content → output_contains
• Tool usage → tools_called, tools_not_called, tool_order
• Resource limits → max_cost, max_turns, max_duration

Reserve judge prompts for criteria that genuinely require interpretation: tone, faithfulness, relevance, completeness.

Anti-patterns

• Multi-criterion judges. One file, one criterion. If you need to check tone AND accuracy, write two judge files.
• Vague definitions like "the response is helpful." Define what helpful means for this specific application.
• No examples. Without examples, the judge doesn't know what counts as failure in your application.
• Likert scales. Binary pass/fail only. Likert scales can't be calibrated and produce unactionable scores.
• Judging without fixing obvious problems first. If the prompt never asked for the behavior, add the instruction before building a judge for it.