fai-tune-18-prompt-management — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited fai-tune-18-prompt-management (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
solution-plays/18-prompt-management/config/ ├── prompts.json ├── experiments.json ├── safety.json └── rollout.json
// config/experiments.json
{
"active_prompt": "v12",
"ab_test": {
"enabled": true,
"variant_a": "v12",
"variant_b": "v13",
"traffic_split": [50, 50],
"min_sample": 200
},
"promotion_rule": {
"metric": "task_success",
"delta_min": 0.03
}
}| Parameter | Range | Default | Guidance |
|---|---|---|---|
traffic_split | sum=100 | [50,50] | Use 90/10 for safer rollout. |
min_sample | 50-5000 | 200 | Increase for low-variance confidence. |
delta_min | 0.01-0.20 | 0.03 | Required improvement to promote variant. |
rollback_window_h | 1-168 | 24 | Auto-revert if quality drops. |
{
"evaluation": {
"enabled": true,
"dataset": "evaluation/test-cases.jsonl",
"sample_size": 200,
"gates": {
"quality_min": 0.80,
"safety_min": 0.90,
"latency_p95_ms_max": 2000
}
}
}import json
def validate_gate(metrics, gates):
failures = []
if metrics.get("quality", 0) < gates["quality_min"]:
failures.append("quality")
if metrics.get("safety", 0) < gates["safety_min"]:
failures.append("safety")
if metrics.get("latency_p95_ms", 999999) > gates["latency_p95_ms_max"]:
failures.append("latency")
if failures:
raise SystemExit(f"Gate failed: {', '.join(failures)}")
print("PASS: all gates met"){
"cost_controls": {
"daily_budget_usd": 500,
"monthly_budget_usd": 10000,
"alert_thresholds": [50, 75, 90],
"throttle_on_budget_breach": true
}
}| Check | Expected | Command | |
|---|---|---|---|
| A/B enabled | true or false by plan | jq '.ab_test.enabled' config/experiments.json | |
| Traffic split sum | 100 | `jq '.ab_test.traffic_split | add' config/experiments.json` |
| Promotion delta | >= 0.01 | jq '.promotion_rule.delta_min' config/experiments.json |
| Issue | Cause | Fix |
|---|---|---|
| No significant winner | Sample size too small | Increase min_sample and run longer. |
| Quality regression after promotion | Gate too weak | Raise delta_min and add groundedness gate. |
| Inconsistent outputs | Prompt templates not pinned | Pin version and lock template variables. |
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.