eval-coach — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited eval-coach (Plugin) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
An Agent Skill for designing comprehensive AI evaluation strategies using Evaluation-Driven Development (EDD).
Created by Bayram Annakov while building Onsa.ai - AI agents for B2B sales prospecting.
If you find this useful, say hi on LinkedIn!
Eval Coach guides you through a structured 5-step framework for evaluating LLM applications:
Copy this skill to your project's skills directory:
git clone https://github.com/BayramAnnakov/eval-coach.git ~/.claude/skills/eval-coachOr add to your project:
mkdir -p skills
git clone https://github.com/BayramAnnakov/eval-coach.git skills/eval-coachInvoke the skill by name or with trigger keywords:
/eval-coachOr just mention evaluation-related topics:
A practical distribution for evaluation methods:
| Tier | Method | Cost | Percentage |
|---|---|---|---|
| 1 | Automated (schema, keywords, latency) | $0.00/run | 50% |
| 2 | LLM-as-Judge (quality, relevance) | $0.01-0.05/run | 40% |
| 3 | Human Review | $5-50/run | 10% |
The most dangerous failures are the ones your tests don't catch. The input_data_consistency_evaluator was born from a real production issue:
We ran an agent with --target john_smith --company CompanyA, but John actually works at CompanyB. The agent silently reconciled the contradiction by writing "engaged with CompanyA via partnerships" instead of flagging the mismatch. All 9 evaluators passed, but the output was misleading.Always include adversarial test cases that probe for silent failures.
This skill follows the Agent Skills open standard (SKILL.md format), supported by:
For the templates:
pip install langsmith langchain-google-genaiSet the judge model via environment variable:
export GOOGLE_API_KEY="your-api-key"
export JUDGE_MODEL="gemini-3-flash-preview" # Default (fast, cost-effective)
export JUDGE_MODEL="gemini-3-pro-preview" # For higher quality judgingMIT License - see LICENSE
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.