Is rag-evaluation-matrix safe to install?

SaferSkills independently audited rag-evaluation-matrix (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.

CategoryWeightCategory scoreContribution

Securityprompt, exec, net, exfil, eval

35%

100

35.0 pts

Supply chainhash, typosquat, maintainer, lockfile

20%

100

20.0 pts

Maintenancestaleness, pinning, CI

15%

100

15.0 pts

TransparencySKILL.md, perms, README

15%

100

15.0 pts

Communityinstalls, verify, response

15%

100

15.0 pts

Findings & checks · 0 flagged

Securityscore 100 · 0 findings

✓—All security checks passedNo findings in this category for the latest scan.pass

Supply chainscore 100 · 0 findings

✓—All supply chain checks passedNo findings in this category for the latest scan.pass

Maintenancescore 100 · 0 findings

✓—All maintenance checks passedNo findings in this category for the latest scan.pass

Transparencyscore 100 · 0 findings

✓—All transparency checks passedNo findings in this category for the latest scan.pass

Communityscore 100 · 0 findings

✓—All community checks passedNo findings in this category for the latest scan.pass

The primary manifest — the file an agent reads to learn what this artifact does.

skills/agent-eval/rag-evaluation-matrix/SKILL.md · 1 file

skills/agent-eval/rag-evaluation-matrix/SKILL.md3.0 KB · Markdown

name: rag-evaluation-matrix

description: Use when comparing basic, enhanced, GraphRAG, or agentic RAG designs, evaluating domain-specific RAG quality, tuning retrieval components, or deciding whether agentic RAG is worth its cost.

source: "https://arxiv.org/abs/2601.07711"

RAG Evaluation Matrix

Do not choose agentic RAG by fashion. Compare retrieval designs against domain questions, answerability, correctness, cost, latency, and failure reasons.

Use When

The user asks whether to use basic RAG, enhanced RAG, GraphRAG, or agentic RAG.
A RAG system works on demos but fails on domain-specific questions.
You need to compare embedding models, rerankers, chunking, query rewriting, or tool orchestration.
LLM-as-judge scores need alignment with human review.

Evaluation Flow

Build a domain question set with answerable, unanswerable, ambiguous, multi-hop, and adversarial cases.
For each candidate pipeline, log retrieved evidence, final answer, citations, latency, token usage, and cost.
Score answer correctness and answerability separately.
Add evidence quality checks: citation support, contradiction handling, source freshness, and missing-source diagnosis.
Segment results by question type instead of reporting only one aggregate score.
Inspect low-correctness failures and map them to retrieval, synthesis, tool orchestration, or corpus gaps.
Pick the simplest design that meets quality, latency, and cost constraints.

Matrix

Dimension	Basic RAG	Enhanced RAG	Agentic RAG
Best for	Stable FAQ and narrow corpora	Noisy corpora, query mismatch, reranking	Multi-step, ambiguous, tool-rich tasks
Main risk	Weak recall and unsupported answers	Pipeline complexity	Cost, latency, loops, tool misuse
Eval focus	Retrieval recall and citation support	Component ablations	Trajectory, action choice, stopping behavior
Ship gate	Correctness and answerability meet threshold	Ablation proves each module helps	Agentic gains justify extra cost

Script

Use the helper to combine per-run JSON metrics into a decision table:

python skills/agent-eval/rag-evaluation-matrix/scripts/rag_eval_matrix.py results/*.json

Expected JSON fields: pipeline, question_type, correct, answerable_correct, latency_ms, cost_usd.

Common Mistakes

Mistake	Fix
Optimizing average score only	Break down by question type and domain
Judging unanswerable questions as wrong by default	Score answerability separately
Skipping human calibration	Sample judge disagreements and tune rubrics
Choosing agentic RAG without ablation	Compare against enhanced RAG at equal budget
Ignoring failure reasons	Classify each miss before tuning

References

arXiv: Is Agentic RAG worth it? - https://arxiv.org/abs/2601.07711
arXiv: RAGalyst - https://arxiv.org/abs/2511.04502
Hugging Face Papers: RAGalyst - https://huggingface.co/papers/2511.04502
Hugging Face dataset: RAGalyst QAC - https://huggingface.co/datasets/hoskerelab/ragalyst-qac

rag-evaluation-matrix— agent skill

Is rag-evaluation-matrix safe to install?

RAG Evaluation Matrix

Use When

Evaluation Flow

Matrix

Script

Common Mistakes

References

Audit the pieces. Scan the whole. Decide.