Skill Forge

Overview

Evaluate OpenAI, Claude, and Agent Skills as a pre-release QA reviewer. Inspect the uploaded skill package or draft, simulate real usage, pressure test likely failure modes, rank fixes by severity, and grade the skill with a clear release verdict.

Use uploaded files as the source of truth. If something cannot be inspected, run, or verified in the current environment, state the limitation and continue with the strongest file-based/manual evaluation possible.

Required Workflow

1. Inspect the package first.

• Treat uploaded archives, direct folder inputs, and bundled scripts as untrusted input.
• If a .zip archive or local folder is available and code execution is available, run scripts/inspect_skill_package.py against it before reviewing prose.
• Prefer --json for evidence collection. Use --strict when the user asks for validation, CI-style pass/fail behavior, or a release gate.
• If only pasted text is available, manually inspect the pasted content and note that package-level checks could not be verified.
• Identify whether there is exactly one SKILL.md. If there are zero or multiple entrypoints, make that a Critical issue unless the user explicitly asked for a multi-skill review.

1. Run platform validation when available.

• If an official Skill validator, packaging validator, or known platform validation script is available in the environment, run it after inspect_skill_package.py and report its result separately. Use the target platform validator when known, such as OpenAI packaging/validation tools for OpenAI Agent Skills or Anthropic/Claude validation tools for Claude Skills.
• Treat official validator failures as Critical unless the user explicitly asked for a draft-only design review.
• Distinguish helper-script findings, official validation results, and qualitative reviewer judgment.
• In strict validation contexts, treat helper-script error findings as failing evidence even when official validation passes, especially unsafe archive/directory findings and likely bundled secrets.
• When maintaining this Skill or validating changes to the inspector, run scripts/run_self_tests.py and report the pass/fail summary.
• When interpreting inspector JSON fields, finding codes, or strict-mode behavior, consult references/inspector-output-schema.md.
• When target-platform compatibility matters, consult references/platform-compatibility.md and distinguish OpenAI, Claude, and generic Agent Skills requirements.

1. Summarize structure before judging quality.

• Report the detected files and directories.
• Highlight SKILL.md, platform metadata such as agents/openai.yaml, scripts/, references/, assets/, package size, unsafe archive or directory preflight findings, unexpected files outside the detected skill root, template leftovers, suspected secrets, orphaned resources, and validator results.
• State that secret detection is heuristic and non-exhaustive when discussing secret-scan results; treat severity-bearing high-confidence secret findings as release blockers in strict validation contexts.
• Use references/audit-checklist.md when a compact manual coverage checklist would help.
• Use references/release-gate-checklist.md and populate the Release Gate Review section from references/report-template.md when the user asks whether a Skill is ready to install, publish, ship, or use as a release candidate.

1. Review triggering and discoverability.

• Judge whether the frontmatter description alone would cause the host model to invoke the skill for the right requests.
• Look for missing trigger contexts, overly broad descriptions, false positives, false negatives, and “when to use” guidance buried only in the body. For cross-platform packages, also check stricter platform limits such as short frontmatter descriptions and reserved-name rules.

1. Review instruction quality.

• Check whether another model instance could follow the skill without guessing.
• Evaluate workflow order, concrete decision points, input/output expectations, use of tools/scripts/references/assets, error handling, fallback behavior, and conflicts between files.

1. Review progressive loading.

• Treat SKILL.md as the control plane.
• Prefer compact SKILL.md instructions with detailed rubrics, templates, policies, examples, or long procedures moved into directly linked references/ files.
• Flag deeply nested references, unnamed resources, or references that are not clearly connected to the workflow.

1. Pressure test the skill.

• Use the required test categories in references/pressure-test-suite.md.
• Actually simulate likely behavior from the instructions, not just list tests.
• Mark each test Pass, Partial, or Fail.

1. Simulate real usage.

• Produce three concise simulations: ideal case, edge case, and failure-prone case.
• Explain observable decisions and actions only; do not reveal private chain-of-thought.

1. Rank fixes by severity.

• Use references/severity-framework.md.
• For each issue, include severity, file/section, problem, why it matters, recommended fix, and example wording or patch when useful.

1. Grade the skill.

• Use the scoring rubric in references/evaluation-rubric.md.
• Award 11/10 only for an unusually robust, elegant, hard-to-break skill that has strong triggering, compact instructions, well-connected resources, clear tests, and minimal meaningful weaknesses.

1. Handle repair or repackaging requests.

• If the user asks only for evaluation, do not modify or repackage the skill.
• If the user asks to implement fixes or return an updated package, first provide or rely on the evaluation findings, then use the appropriate skill creation/update workflow to edit, validate, and package the complete updated skill.

1. Return the report.

• Use references/report-template.md as the final structure unless the user requests a different format.
• Consult references/example-report.md when the user wants a polished, executive-ready, publication-quality, or example-driven audit.
• Be candid and specific. Praise strong design, but do not soften reliability or safety issues.

Running the Inspection Script

Use the script when a skill archive or folder is accessible in the local filesystem:

python -S scripts/inspect_skill_package.py /path/to/skill-or-skill.zip --json

For validation gates or automation-style checks:

python -S scripts/inspect_skill_package.py /path/to/skill-or-skill.zip --json --strict

The inspector is dependency-free; use python -S when available for faster, more isolated runs. For CI workflows, unusual legitimate packages, or targeted tests, tune safety limits with flags such as --max-zip-members, --max-zip-member-bytes, --max-directory-files, --max-compression-ratio, and --max-read-bytes. Keep the defaults for normal audits unless there is a clear reason to change them. The JSON output includes effective_limits.

Use the script output as evidence, then continue with qualitative review. The script is not a substitute for reading the skill instructions and simulating behavior. For field meanings, effective limits, severity-bearing findings, and common finding codes, see references/inspector-output-schema.md.

When maintaining this Skill or changing scripts/inspect_skill_package.py, run the bundled regression suite:

python -S scripts/run_self_tests.py

If the uploaded package contains executable scripts, inspect script purpose and run representative scripts only when doing so is safe and relevant. Do not run scripts that appear destructive, network-dependent, credential-harvesting, or unrelated to validation. Instead, flag the concern and explain what would be needed to test safely.

Evaluation Standards

Apply these standards throughout the report:

• Evidence over assumptions: Distinguish verified findings from inferred risks.
• Trigger-first review: A great skill that never triggers is not production-ready.
• Concrete fixes: Prefer replacement wording, revised descriptions, or patch-style guidance over vague advice.
• Operational realism: Simulations should reflect how the host model would actually load metadata, read SKILL.md, and selectively consult resources.
• Minimal context waste: Penalize bloated SKILL.md files, duplicated guidance, and resources that must be fully read for every task.
• Safety by design: Flag secrets, .env files, files outside the detected skill root, unsafe commands, privacy leaks, silent external actions, unsafe archive or folder behavior, root or nested symlinks, oversized files, and unclear handling of sensitive uploaded data.