evaluate-anomaly-detection — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited evaluate-anomaly-detection (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
| Metric | Target | How to Measure |
|---|---|---|
| Precision | ≥ 85% | True anomalies / all alerts |
| Recall | ≥ 90% | Detected anomalies / all real anomalies |
| F1 Score | ≥ 0.87 | Harmonic mean of precision and recall |
| False positive rate | < 15% | False alerts / total alerts |
| Detection latency | < 5 min | Time from anomaly start to alert |
| Root cause accuracy | ≥ 70% | LLM suggestion matches actual cause |
| Alert fatigue score | < 3 alerts/day avg | Alert volume tracking |
| Mean time to resolution | Decreased vs baseline | Before/after comparison |
{"timestamp": "2026-04-01T10:00:00Z", "metric": "cpu_percent", "value": 95, "is_anomaly": true, "root_cause": "deployment_spike"}
{"timestamp": "2026-04-01T10:05:00Z", "metric": "cpu_percent", "value": 92, "is_anomaly": true, "root_cause": "deployment_spike"}
{"timestamp": "2026-04-01T10:10:00Z", "metric": "cpu_percent", "value": 45, "is_anomaly": false, "root_cause": null}
{"timestamp": "2026-04-01T10:15:00Z", "metric": "error_rate", "value": 0.15, "is_anomaly": true, "root_cause": "upstream_dependency"}Minimum: 200 labeled data points across normal + anomalous periods.
python evaluation/eval.py --test-set evaluation/test-set.jsonl --metrics detection_accuracypython evaluation/eval.py --test-set evaluation/test-set.jsonl --metrics root_causepython evaluation/eval.py --all --output evaluation/detection-report.json --ci-gate| Result | Action |
|---|---|
| All PASS | Deploy detection pipeline to production |
| Recall < 80% | Lower detection threshold, add more detection methods |
| False positive > 25% | Raise threshold, add suppression rules |
| Detection latency > 10 min | Reduce time window, optimize pipeline |
| Root cause < 50% accurate | Enrich context in LLM prompt |
| Symptom | Root Cause | Fix |
|---|---|---|
| Missing real anomalies | Threshold too high (>3σ) | Lower to 2σ, add multiple detection methods |
| Alert storm | Auto-scaling triggers false alerts | Exclude scaling events, add cooldown window |
| Seasonal false positives | Not accounting for time-of-day | Use seasonal decomposition baseline |
| Multivariate anomaly missed | Only checking individual metrics | Enable multivariate detection |
| Root cause always generic | Insufficient context in prompt | Add deployment history, dependency map |
# Add detection quality gates to pipeline
- name: Detection Precision Gate
run: python evaluation/eval.py --metrics detection_accuracy --ci-gate --min-precision 0.85
- name: False Positive Gate
run: python evaluation/eval.py --metrics detection_accuracy --ci-gate --max-fp-rate 0.15
- name: Root Cause Accuracy Gate
run: python evaluation/eval.py --metrics root_cause --ci-gate --min-accuracy 0.70~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.