evaluate-it-ticket-resolution — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited evaluate-it-ticket-resolution (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
| Metric | Target | How to Measure |
|---|---|---|
| Classification accuracy | ≥ 92% | Labeled test set comparison |
| Priority assignment accuracy | ≥ 95% | P1-P4 label match |
| Auto-resolution rate | ≥ 60% | Tickets resolved without human |
| SLA compliance | ≥ 95% | Tickets resolved within SLA window |
| PII detection recall | ≥ 99% | Known PII test patterns |
| First-response time | < 30 seconds | Time from creation to AI response |
| KB retrieval relevance | ≥ 0.8 | Groundedness of KB-sourced answers |
| False escalation rate | < 10% | Tickets unnecessarily sent to humans |
Create labeled test cases in evaluation/test-set.jsonl:
{"id": "t001", "ticket": "Cannot connect to VPN from home", "category": "Network", "priority": "P3", "resolution": "Reset VPN credentials in portal", "has_pii": false}
{"id": "t002", "ticket": "Laptop won't boot, employee ID 12345", "category": "Hardware", "priority": "P2", "resolution": "Submit hardware replacement request", "has_pii": true}
{"id": "t003", "ticket": "Production DB unreachable, 500 errors on main app", "category": "Software", "priority": "P1", "resolution": "escalate_to_human", "has_pii": false}Minimum: 50 test cases spanning all categories and priorities.
python evaluation/eval.py --test-set evaluation/test-set.jsonl --metrics classificationpython evaluation/eval.py --test-set evaluation/test-set.jsonl --metrics resolutionpython evaluation/eval.py --test-set evaluation/test-set.jsonl --metrics piipython evaluation/eval.py --all --output evaluation/report.json --ci-gate| Result | Action |
|---|---|
| All PASS | Deploy to production |
| Classification < 85% | Expand training examples for failing categories |
| Resolution < 50% | Improve KB content, add more runbooks |
| PII recall < 95% | Blockers — fix PII patterns before deploying |
| SLA compliance < 90% | Tune escalation timing, reduce resolution latency |
| Symptom | Root Cause | Fix |
|---|---|---|
| Software vs Hardware confusion | Overlapping keywords ("screen") | Add disambiguation examples |
| P1 classified as P3 | Missing urgency keywords | Add "production", "outage", "down" to P1 patterns |
| Auto-resolution gives wrong answer | KB retrieval returns irrelevant doc | Improve chunking, add semantic reranking |
| PII missed in attachments | OCR not enabled | Enable document intelligence for attachments |
| False escalation spike | Confidence threshold too high | Lower from 0.9 to 0.85 |
# Add quality gates to CI pipeline
- name: Classification Gate
run: python evaluation/eval.py --metrics classification --ci-gate --threshold 0.92
- name: PII Detection Gate
run: python evaluation/eval.py --metrics pii --ci-gate --threshold 0.99
- name: Resolution Gate
run: python evaluation/eval.py --metrics resolution --ci-gate --threshold 0.60~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.