evaluate-agentic-rag — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited evaluate-agentic-rag (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
| Metric | Target | How to Measure |
|---|---|---|
| Groundedness | ≥ 0.90 | Azure AI Evaluation SDK |
| Relevance | ≥ 0.85 | Retrieved docs relevance to query |
| Source selection accuracy | ≥ 90% | Correct source chosen for query type |
| Avg retrieval hops | < 2.0 | Average iterations per query |
| Iteration reduction | ≥ 30% vs max | Queries resolved in 1 hop vs needing max |
| Citation accuracy | ≥ 95% | Citations link to correct source doc |
| Cache hit rate | ≥ 40% | Cached responses / total queries |
| Cost per query | < $0.05 | Tokens across all iterations |
| Self-eval accuracy | ≥ 85% | Agent correctly knows when to stop |
{"query": "What's our refund policy?", "expected_source": "knowledge-base", "expected_hops": 1}
{"query": "Latest Azure pricing changes", "expected_source": "web-search", "expected_hops": 1}
{"query": "Order status for customer 12345", "expected_source": "product-db", "expected_hops": 1}
{"query": "Compare our warranty vs competitor X", "expected_source": "knowledge-base+web-search", "expected_hops": 2}Minimum: 50 test queries across single-source, multi-source, and iterative scenarios.
python evaluation/eval.py --test-set evaluation/test-set.jsonl --metrics source_selectionRun same test set through both Play 01 (fixed RAG) and Play 21 (agentic RAG):
| Metric | Standard RAG | Agentic RAG | Delta |
|---|---|---|---|
| Groundedness | Measure | Measure | Improvement |
| Multi-source coverage | N/A (single) | Measure | New capability |
| Cost per query | Measure | Measure | Cost overhead |
| Complex query accuracy | Measure | Measure | Key differentiator |
python evaluation/eval.py --all --output evaluation/report.json --ci-gate| Result | Action |
|---|---|
| All PASS | Deploy to production |
| Groundedness < 0.85 | Improve retrieval prompts, add more sources |
| Source selection < 80% | Refine tool descriptions, add routing examples |
| Avg hops > 3 | Raise self-eval threshold, improve first-hop quality |
| Cache hit < 20% | Lower similarity threshold, increase TTL |
| Symptom | Root Cause | Fix |
|---|---|---|
| Agent ignores available sources | Poor tool descriptions | Rewrite tool descriptions with clear use cases |
| Always uses max iterations | Self-eval threshold too high (0.99) | Lower to 0.85 |
| Bad citations | Source metadata not returned from tools | Include doc title + URL in tool response |
| Cache returns wrong answer | Similarity threshold too low | Raise from 0.85 to 0.92 |
| High cost on simple queries | No routing shortcut for FAQ | Add direct-answer path for high-confidence KB matches |
| Multi-source synthesis poor | No merge strategy | Add explicit "synthesize from multiple sources" in system prompt |
- name: Agentic RAG Quality Gate
run: python evaluation/eval.py --all --ci-gate
- name: Source Selection Gate
run: python evaluation/eval.py --metrics source_selection --ci-gate --threshold 0.90
- name: Cost Gate
run: python evaluation/eval.py --metrics cost_per_query --ci-gate --max-cost 0.05~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.