fai-tune-17-ai-observability — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited fai-tune-17-ai-observability (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
solution-plays/17-ai-observability/config/ ├── tracing.json ├── metrics.json ├── alerts.json └── retention.json
// config/metrics.json
{
"sampling_rate": 1.0,
"capture_prompt": true,
"capture_completion": true,
"redact_pii": true,
"slo": {
"latency_p95_ms": 1800,
"error_rate_max": 0.02,
"groundedness_min": 0.80
}
}| Parameter | Range | Default | Guidance |
|---|---|---|---|
sampling_rate | 0.01-1.0 | 1.0 | Reduce in prod for cost if volume is high. |
latency_p95_ms | 200-5000 | 1800 | Set per endpoint class. |
error_rate_max | 0.001-0.10 | 0.02 | Alert above this threshold. |
groundedness_min | 0.60-0.95 | 0.80 | Raise for regulated workloads. |
{
"evaluation": {
"enabled": true,
"dataset": "evaluation/test-cases.jsonl",
"sample_size": 200,
"gates": {
"quality_min": 0.80,
"safety_min": 0.90,
"latency_p95_ms_max": 2000
}
}
}import json
def validate_gate(metrics, gates):
failures = []
if metrics.get("quality", 0) < gates["quality_min"]:
failures.append("quality")
if metrics.get("safety", 0) < gates["safety_min"]:
failures.append("safety")
if metrics.get("latency_p95_ms", 999999) > gates["latency_p95_ms_max"]:
failures.append("latency")
if failures:
raise SystemExit(f"Gate failed: {', '.join(failures)}")
print("PASS: all gates met"){
"cost_controls": {
"daily_budget_usd": 500,
"monthly_budget_usd": 10000,
"alert_thresholds": [50, 75, 90],
"throttle_on_budget_breach": true
}
}| Check | Expected | Command |
|---|---|---|
| PII redaction | true | jq '.redact_pii' config/metrics.json |
| P95 latency objective | <= 5000 | jq '.slo.latency_p95_ms' config/metrics.json |
| Error rate objective | <= 0.10 | jq '.slo.error_rate_max' config/metrics.json |
| Issue | Cause | Fix |
|---|---|---|
| No traces in dashboard | Sampling disabled or exporter misconfigured | Set sampling_rate > 0 and verify exporter endpoint. |
| Alert fatigue | Threshold too strict | Tune error_rate_max and add debounce windows. |
| Compliance risk in logs | PII redaction disabled | Set redact_pii true and re-run log scan. |
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.