tune-it-ticket-resolution — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited tune-it-ticket-resolution (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
| Parameter | Default | Range | Impact |
|---|---|---|---|
| Temperature | 0.0 | 0.0-0.1 | Must be 0 for deterministic classification |
| Category count | 6 | 5-15 | More categories = lower per-category accuracy |
| Confidence threshold | 0.85 | 0.7-0.95 | Lower = more auto-routes, higher = more human |
| Few-shot examples | 3 per category | 2-5 | More examples = better accuracy, more tokens |
| System prompt length | 500 tokens | 300-800 | Too short = misclassification, too long = slow |
Diagnostic: Run python evaluation/eval.py --metrics classification to get per-category F1 scores.
Optimization steps:
| Parameter | Default | Recommendation | Why |
|---|---|---|---|
| Auto-resolve threshold | 0.85 | 0.80-0.90 | Balance auto-resolution rate vs accuracy |
| Max auto-attempts | 2 | 1-3 | Retry with rephrased query before escalating |
| Escalation trigger | Confidence < threshold | Also: sentiment negative | Frustrated users get human faster |
| P1 routing | Always human + AI | Never bypass human for P1 | P1 = production impact |
| P4 auto-close | After 48h resolved | After 24h with user confirmation | Faster ticket lifecycle |
| Parameter | Default | Range | Impact |
|---|---|---|---|
| Search type | hybrid | keyword/vector/hybrid | Hybrid best for IT vocabulary |
| top_k | 5 | 3-10 | More = better recall, slower, higher cost |
| Relevance threshold | 0.75 | 0.6-0.9 | Below threshold = "I don't know" |
| Chunk size | 512 tokens | 256-1024 | Smaller = precise, larger = more context |
| Reranking | Enabled | On/Off | Enabled significantly improves relevance |
Diagnostic: Run python evaluation/eval.py --metrics resolution to measure KB relevance.
| Task | Model | Cost | Latency | Accuracy |
|---|---|---|---|---|
| Classification | gpt-4o-mini | $0.15/1M | < 500ms | 92%+ |
| Resolution (simple) | gpt-4o-mini | $0.15/1M | < 800ms | 85%+ |
| Resolution (complex) | gpt-4o | $2.50/1M | < 1.5s | 95%+ |
| PII detection | Content Safety API | Fixed | < 200ms | 99%+ |
Cost optimization strategy:
| Priority | Default SLA | Tune When | Adjustment |
|---|---|---|---|
| P1 | 1 hour | Resolution time consistently > 45min | Auto-escalate at 30min |
| P2 | 4 hours | AI resolves 80%+ of P2 | Extend auto-resolve window to 2h |
| P3 | 8 hours | Low auto-resolution rate | Add more KB content for P3 categories |
| P4 | 24 hours | Tickets pile up | Enable auto-close after AI resolution |
After tuning, generate a report comparing before/after:
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.