Audit Eval

Set up and assess an agent codebase for kensa.

Lifecycle

► Setup → Design → Execute → Diagnose → Iterate

This is the entry point, both for first-time setup and for returning users who need to figure out where they are. After this skill, the next step is generate-scenarios (if no scenarios exist) or kensa eval (if scenarios are ready).

Check kensa is installed

python ${CLAUDE_SKILL_DIR}/scripts/check_library.py

If missing, ask the user before installing:

"kensa is not installed. Add it as a project dependency?"

If approved:

python ${CLAUDE_SKILL_DIR}/scripts/check_library.py --install

Determine current state

Use Glob to check what exists:

1. Glob(".kensa/"): does the directory exist at all?
2. Glob(".kensa/scenarios/*.yaml"): scenario files?
3. Glob(".kensa/agents/*"): agent entry points?
4. Glob(".kensa/traces/*.jsonl"): trace files?
5. Glob(".kensa/results/*.json"): result files from previous runs?
6. Glob(".kensa/judges/*.yaml"): judge prompt files?

Recommend a single next action based on findings:

• No `.kensa/` directory: Fresh start. Run kensa init to scaffold (bare; no example). Then scan the codebase (below). Prefer the trace-first path: kensa capture -i "<example input>" -- <agent command> to record one real run, then hand off to generate-scenarios (it will use the capture). If capture is not feasible, hand off without traces.
• `.kensa/` exists but empty (no scenarios, no traces): Bare scaffold. Scan the codebase, then offer kensa capture before handing off to generate-scenarios.
• Has `.kensa/` with only example scenario: Init was run with --example. Scan the codebase, then hand off to generate-scenarios to replace the example.
• No scenarios, has traces: Traces exist from kensa capture. Run kensa analyze to surface patterns, then hand off to generate-scenarios with the analysis as context.
• Has scenarios, no traces: Scenarios are ready but never run. Recommend: kensa eval.
• Has scenarios, has traces and results: Full loop is active. Check both data stores:

1. Run kensa report --format json and count scenarios with status: "fail", "error", or "uncertain". This catches check failures and judge rejections that leave no trace anomaly.
2. Run kensa analyze to check for trace-level anomalies (cost/latency outliers, errors, looping).

If either surface has failures, recommend diagnose-errors. If both are clean, recommend kensa eval.

• Has scenarios, has traces but no results: Traces exist but judge hasn't run. Recommend: kensa judge.

State the recommendation clearly. Mention alternatives only if genuinely relevant.

Scan codebase

Identify these five things. Use Agent with subagent_type: "Explore" for thorough scanning on larger codebases, or Read and Grep directly on small ones.

1. Entry point: how to run the agent
2. LLM SDK: anthropic, openai, langchain, etc.
3. Tools: functions the agent can call
4. Behaviors: from system prompts, docstrings, README
5. Env vars: required configuration

Record as working memory:

Entry point: src/agent.py
Run command: [uv, run, python, src/agent.py]
SDK: anthropic
Tools: [get_weather, search_web, calculate]
Behaviors: Answers user questions using web search and weather data
Env vars: ANTHROPIC_API_KEY

Assess readiness

Tell the user what you found honestly:

"I scanned your codebase. Here's what I can infer and what I can't. Scenario quality will be limited by [X]. To improve: [concrete action]."

Verify instrumentation

Instrumentation is automatic. The runner injects sitecustomize.py via PYTHONPATH before the agent runs. No code changes needed in the agent.

Ensure the matching SDK extra is installed:

Verify with:

kensa doctor

If kensa doctor passes, instrumentation is ready. If it flags issues, fix them before proceeding.

Next step

Hand off to generate-scenarios with the codebase scan context (entry point, tools, behaviors, env vars). The user's next action is to design test scenarios.

Gotchas

• instrument() is idempotent. Agents that still have from kensa import instrument; instrument() work fine (no duplicate spans).
• instrument() warns when no SDK instrumentors are installed. Install the matching extra.
• .env is auto-loaded by the runner (walks up from cwd). Never read or print .env yourself: it holds secrets. Use kensa doctor to verify which env vars are set.
• Judge model resolution: KENSA_JUDGE_MODEL env var > ANTHROPIC_API_KEY (claude-sonnet-4-6) > OPENAI_API_KEY (gpt-5.4-mini) > error.
• For non-Python commands or python -S/-I, the escape hatch is from kensa import instrument; instrument() in the agent code.