SkillBench Lite

A tiny eval harness that tests whether agent skill files actually improve task outcomes.

Most skill repos tell you what a skill is supposed to do. SkillBench Lite asks a sharper question: did the skill change the result enough to justify using it?

Why This Exists

Agent skills are spreading fast across Claude Code, Codex, Cursor, Gemini, Copilot, and local agent tools. The missing layer is lightweight proof. A skill can sound excellent and still add no measurable value, add too much token overhead, or quietly regress an answer.

SkillBench Lite gives skill authors a small, local scorecard:

• Run the same task with and without a skill.
• Check concrete assertions.
• Estimate token overhead.
• Write a Markdown report that maintainers can review.

Quick Start

git clone https://github.com/redkoai/skillbench-lite.git
cd skillbench-lite
npm test
npm run demo

The demo writes:

reports/demo-report.md

Example Output

Config

Create skillbench.config.json:

{
  "command": "node examples/demo-agent.js",
  "skills": [
    {
      "id": "precise-debugging",
      "path": "skills/precise-debugging.md"
    }
  ],
  "tasks": [
    {
      "id": "timezone-bug",
      "prompt": "Explain why the date test fails only outside UTC.",
      "assertions": [
        { "contains": "timezone normalization" },
        { "contains": "Date.UTC" },
        { "exitCode": 0 }
      ]
    }
  ]
}

Run it:

node ./bin/skillbench.js run --config skillbench.config.json

Your command receives these environment variables:

• SKILLBENCH_MODE: baseline or skill
• SKILLBENCH_TASK_ID
• SKILLBENCH_TASK
• SKILLBENCH_TASK_JSON
• SKILLBENCH_SKILL_ID
• SKILLBENCH_SKILL_PATH
• SKILLBENCH_FIXTURE_DIR

Assertions

Supported assertions:

{ "contains": "required text" }
{ "notContains": "forbidden text" }
{ "matches": "regular expression" }
{ "exitCode": 0 }

Good Uses

• Test whether a coding-agent skill improves bug-fix quality.
• Compare two versions of the same skill.
• Catch skill regressions before publishing.
• Produce honest README proof instead of vague benchmark claims.

Non-Goals

• This is not a model leaderboard.
• This does not claim your skill is universally good.
• This does not replace human review.

Architecture

See docs/architecture.md.

License

MIT