rag-design-doc-f76f49 — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited rag-design-doc-f76f49 (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
Most RAG systems fail not at generation but at retrieval — the model answers confidently from the wrong chunks. This skill forces the decisions that actually determine quality (chunking, retrieval, reranking, grounding) and pairs each with how you'll evaluate it, so "it hallucinates sometimes" becomes a diagnosable, fixable pipeline.
Ask for these only if they aren't already provided:
1. Goal & non-goals — what questions it answers well, and what it explicitly won't do.
2. Ingestion & chunking
3. Embeddings & index — embedding model + dimension, vector store, and the index/filter strategy (incl. metadata filters and per-tenant isolation).
4. Retrieval — top-k, hybrid (dense + keyword/BM25) vs. pure vector, metadata pre-filtering, and query transformation (rewriting, decomposition, HyDE) if used.
5. Reranking — whether a cross-encoder/reranker narrows the candidate set before generation, and the final context budget.
6. Generation — the prompt template, how retrieved context is formatted, the instruction to answer only from context and say "I don't know" otherwise, and how citations are produced and verified.
7. Evaluation — retrieval metrics (recall@k, MRR) separately from answer quality (faithfulness/groundedness, correctness). Pair with an ai-eval-plan.
8. Failure modes & mitigations — a table: symptom → likely stage → fix.
| Symptom | Likely cause (stage) | Mitigation |
|---|---|---|
| Confident but wrong | retrieval missed the chunk | hybrid search, better chunking, rerank |
| Right doc, wrong detail | chunk too large/small | tune size+overlap, structure-aware split |
| Ignores retrieved context | prompt/format | stronger grounding instruction, fewer/cleaner chunks |
| Stale answers | index freshness | incremental re-index, timestamp filter |
Retrieval-Augmented Generation practice — hybrid retrieval, reranking, grounded generation, and faithfulness evaluation.
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.