evaluate-ai-search-portal — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited evaluate-ai-search-portal (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
| Metric | Target | How to Measure |
|---|---|---|
| NDCG@10 | ≥ 0.7 | Normalized Discounted Cumulative Gain on top 10 |
| MRR | ≥ 0.6 | Mean Reciprocal Rank of first relevant result |
| Zero-result rate | < 5% | Queries returning 0 results |
| Query latency (p50) | < 200ms | 50th percentile response time |
| Query latency (p95) | < 500ms | 95th percentile response time |
| Autocomplete latency | < 100ms | Suggestion response time |
| Click-through rate | ≥ 40% | Users clicking a result |
| Semantic ranking lift | ≥ 15% | NDCG improvement vs keyword-only |
Create judged query-document pairs:
{"query": "azure container deployment", "relevant_docs": ["doc-123", "doc-456"], "relevance_levels": [3, 2]}
{"query": "kubernetes scaling policies", "relevant_docs": ["doc-789", "doc-012"], "relevance_levels": [3, 1]}
{"query": "pricing calculator", "relevant_docs": ["doc-345"], "relevance_levels": [3]}Minimum: 50 judged queries across different categories and intents.
python evaluation/eval.py --test-set evaluation/test-set.jsonl --metrics relevancepython evaluation/eval.py --test-set evaluation/test-set.jsonl --metrics latencypython evaluation/eval.py --test-set evaluation/test-set.jsonl --metrics coveragepython evaluation/eval.py --all --output evaluation/report.json --ci-gate| Result | Action |
|---|---|
| All PASS | Deploy to production |
| NDCG < 0.6 | Tune scoring profiles, add boost functions |
| Zero-result > 10% | Expand content, add synonym maps |
| Latency p95 > 1s | Check index size, optimize query, add replicas |
| Semantic lift < 10% | Verify semantic config, check content quality |
| Symptom | Root Cause | Fix |
|---|---|---|
| Relevant docs ranked low | No scoring profile | Add boost for title matches |
| Vector search dominates | Hybrid weights wrong | Balance vector:keyword to 0.5:0.5 |
| Old content ranked high | No date boost | Add freshness boost in scoring |
| Autocomplete irrelevant | Wrong suggester fields | Use title + category only |
| Slow under load | Single replica | Add replicas for read throughput |
| Facet counts inaccurate | Index not refreshed | Check indexer schedule |
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.