evaluate-multi-agent-service — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited evaluate-multi-agent-service (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
| Metric | Target | How to Measure |
|---|---|---|
| Task completion rate | ≥ 90% | End-to-end task success tracking |
| Handoff success rate | ≥ 98% | Agent-to-agent message delivery |
| End-to-end latency | < 30 seconds | Full task timing |
| Loop detection accuracy | 100% | Intentional loop test scenarios |
| Cost per task | < $0.50 | Token usage across all agents |
| Agent utilization | > 70% | Active time vs idle time |
| Conflict resolution rate | ≥ 95% | Conflicting output reconciliation |
| Human escalation rate | < 10% | Tasks requiring human intervention |
Create multi-agent test cases in evaluation/test-set.jsonl:
{"id": "ma001", "task": "Research and summarize Azure pricing changes", "expected_agents": ["researcher", "writer"], "expected_output_contains": "pricing", "max_latency_s": 30}
{"id": "ma002", "task": "Analyze sales data and create chart recommendations", "expected_agents": ["analyst", "writer"], "expected_output_contains": "chart", "max_latency_s": 25}
{"id": "ma003", "task": "Review code PR and suggest improvements", "expected_agents": ["researcher", "analyst", "validator"], "expected_output_contains": "suggestion", "max_latency_s": 30}Minimum: 20 test scenarios covering different agent combinations.
python evaluation/eval.py --test-set evaluation/test-set.jsonl --metrics task_completionpython evaluation/eval.py --test-set evaluation/test-set.jsonl --metrics handoffpython evaluation/eval.py --test-set evaluation/test-set.jsonl --metrics safetypython evaluation/eval.py --all --output evaluation/report.json --ci-gate| Result | Action |
|---|---|
| All PASS | Deploy to production |
| Task completion < 85% | Improve supervisor routing prompt |
| Handoff failures > 5% | Check message queue, increase timeouts |
| Loop detection gaps | Fix max_handoffs configuration |
| Cost per task > $1.00 | Switch workers to gpt-4o-mini, add caching |
| Latency > 45s | Enable parallel subtasks, reduce agent chain |
| Symptom | Root Cause | Fix |
|---|---|---|
| Supervisor delegates everything | Too many agent options | Reduce topology, clarify agent roles |
| Worker returns partial results | Timeout too short | Increase per-agent timeout |
| Conflicting agent outputs | No validator in chain | Add validator as final aggregation step |
| Same subtask sent twice | No dedup in supervisor | Add task ID tracking in shared state |
| Agent stuck waiting | Deadlock in handoff chain | Implement timeout + supervisor retry |
| High cost on simple tasks | Using gpt-4o for all agents | Route simple subtasks to gpt-4o-mini |
- name: Multi-Agent Quality Gate
run: python evaluation/eval.py --all --ci-gate~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.