evaluate-multi-agent-swarm — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited evaluate-multi-agent-swarm (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
| Metric | Target | How to Measure |
|---|---|---|
| Consensus agreement rate | ≥ 80% | Votes agreeing / total votes |
| Task quality (winner) | ≥ 4.0/5.0 | Human eval of consensus output |
| Bidding accuracy | ≥ 85% | Best agent wins bid for task type |
| Parallel speedup | ≥ 2x vs sequential | Swarm time / single-agent time |
| Vote convergence rounds | ≤ 2 | Rounds to reach quorum |
| Cost per swarm task | < $0.15 | Tokens across all agents × rounds |
| Agent utilization | ≥ 60% | Active time / total time |
| Self-healing success | 100% | Task completes when agent fails |
| Deduplication rate | < 5% | Duplicate work across agents |
{"task": "Research and summarize cloud pricing trends", "type": "research", "expected_agents": 3, "expected_quality": 4}
{"task": "Analyze dataset and identify outliers", "type": "analysis", "expected_agents": 2, "expected_quality": 4}
{"task": "Write technical blog post on AI governance", "type": "writing", "expected_agents": 2, "expected_quality": 4}Minimum: 30 tasks spanning different complexity levels and agent roles.
python evaluation/eval.py --test-set evaluation/test-set.jsonl --metrics consensus| Metric | Single Agent | Swarm (3 agents) | Delta |
|---|---|---|---|
| Quality | Measure | Measure | Swarm advantage |
| Latency | Measure | Measure | Parallel speedup |
| Cost | Measure | Measure | Cost overhead |
| Reliability | Measure | Measure | Fault tolerance |
python evaluation/eval.py --all --output evaluation/report.json --ci-gate| Result | Action |
|---|---|
| All PASS | Deploy swarm to production |
| Consensus < 70% | Improve agent prompts, add validation round |
| Bidding wrong agent | Refine bid scoring, add task type classifier |
| Cost > $0.30/task | Reduce swarm size, use gpt-4o-mini for more agents |
| No parallel speedup | Check Service Bus throughput, reduce bidding timeout |
| Symptom | Root Cause | Fix |
|---|---|---|
| Always same output | One agent dominates votes | Add bid diversity bonus |
| Low consensus rate | Agents interpret task differently | Standardize task prompts |
| Swarm slower than single agent | Bidding overhead | Reduce bid timeout, fewer agents |
| Duplicate work | No dedup check | Add result hashing before voting |
| Cost runaway | Too many voting rounds | Set max_rounds=2, lower quorum |
| Agent never wins bids | Low confidence score | Retrain or replace underperforming agent |
- name: Swarm Consensus Gate
run: python evaluation/eval.py --metrics consensus --ci-gate --threshold 0.80
- name: Swarm Cost Gate
run: python evaluation/eval.py --metrics cost --ci-gate --max-cost 0.15
- name: Self-Healing Test
run: python evaluation/test_self_healing.py --kill-agent researcher --verify-completion~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.