evaluate-cost-optimized-ai-gateway — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited evaluate-cost-optimized-ai-gateway (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
| Metric | Target | How to Measure |
|---|---|---|
| Cache hit rate | ≥ 30% | Redis hit/miss ratio |
| Cost savings vs direct | ≥ 25% | Compare APIM cost to direct OpenAI cost |
| Gateway overhead (latency) | < 50ms added | Side-by-side timing |
| Budget enforcement accuracy | 100% | Test at budget boundary |
| Routing accuracy | 100% | Verify correct model per request type |
| Failover time | < 3 seconds | Disable primary, measure switchover |
| Rate limit accuracy | 100% | Test at rate boundary |
| Per-tenant cost tracking | ≤ 5% error | Compare tracked vs actual |
python evaluation/eval.py --metrics cache_performance --duration 24h| Scenario | Direct OpenAI | Through Gateway | Savings |
|---|---|---|---|
| 10K requests/day, 30% cache | $50/day | $35/day | 30% |
| 10K requests/day, 50% cache | $50/day | $25/day | 50% |
| 10K requests/day, no cache | $50/day | $50 + APIM overhead | -5% |
Break-even: Gateway must save ≥ APIM cost ($0.003/API call) + Redis cost.
# Send requests up to budget limit
python evaluation/test_budget.py --tier free --token-limit 100000
# Verify: requests succeed until 100K, then get 429
python evaluation/test_budget.py --tier standard --token-limit 1000000# Disable primary region
az openai deployment delete --name gpt-4o --resource-group $PRIMARY_RG
# Verify: requests automatically route to secondary
python evaluation/test_failover.py --expect-region westus3
# Re-enable primaryMeasure: failover time, error rate during switchover, auto-recovery.
python evaluation/benchmark.py --direct $OPENAI_ENDPOINT --gateway $APIM_URL --requests 100python evaluation/eval.py --full-report --output evaluation/finops-report.json| Result | Action |
|---|---|
| Cache hit ≥30%, savings ≥25% | Deploy to production |
| Cache hit <15% | Tune similarity threshold, expand cache scope |
| Gateway overhead >100ms | Optimize Redis connection, simplify policies |
| Budget enforcement fails | Fix rate-limit-by-key policy |
| Failover >10s | Add health probes, reduce retry interval |
| Symptom | Root Cause | Fix |
|---|---|---|
| Low cache hits | Threshold too strict (0.99) | Lower to 0.95 |
| High cost despite caching | Cache TTL too short | Increase TTL for stable content |
| Budget exceeded | Race condition in counter | Use Redis atomic increment |
| Wrong region served | Weight misconfigured | Verify backend weights sum to 100 |
| APIM 500 errors | Backend timeout | Increase timeout, add circuit breaker |
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.