evaluate-copilot-studio-bot — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited evaluate-copilot-studio-bot (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
| Metric | Target | Source |
|---|---|---|
| Topic trigger accuracy | ≥ 90% | Test utterance matching |
| Resolution rate | ≥ 65% | Conversations resolved without escalation |
| CSAT score | ≥ 4.0/5.0 | Post-conversation survey |
| Avg conversation turns | < 5 | Turn count per resolved conversation |
| Fallback rate | < 20% | Conversations hitting no-topic-match |
| Knowledge answer relevance | ≥ 0.8 | Generative answer grounding score |
| Escalation rate | < 25% | Human handoff count |
| Abandonment rate | < 15% | Conversations user left mid-flow |
Create a test matrix covering all topics:
{"topic": "password_reset", "utterance": "I forgot my password", "expected_topic": "PasswordReset"}
{"topic": "password_reset", "utterance": "Can't log in to my account", "expected_topic": "PasswordReset"}
{"topic": "leave_request", "utterance": "I want to take PTO next week", "expected_topic": "LeaveRequest"}
{"topic": "fallback", "utterance": "What's the weather like?", "expected_topic": "Fallback"}Minimum: 10 test utterances per topic, 5 known fallback scenarios.
Review Copilot Studio Analytics dashboard:
| Result | Action |
|---|---|
| All PASS | Publish to production channel |
| Trigger accuracy < 85% | Add more trigger phrases, deduplicate |
| Resolution < 55% | Simplify conversation flows, improve actions |
| Fallback > 30% | Add more topics or expand knowledge sources |
| CSAT < 3.5 | Review conversation transcripts for UX issues |
| Symptom | Root Cause | Fix |
|---|---|---|
| Wrong topic fires | Overlapping trigger phrases | Unique triggers per topic |
| Bot loops in topic | Missing exit condition | Add "anything else?" + exit node |
| Knowledge hallucination | No grounding instruction | Configure "answer only from content" |
| Always falls back | Too few trigger phrases | Add 8-10 diverse phrases per topic |
| Auth flow breaks | Token expiry not handled | Add token refresh logic |
| Action fails silently | Power Automate error not surfaced | Add error handling in flow |
Before publishing to production channel:
1. Export bot solution from dev environment
2. Run test utterance suite against staging bot
3. Verify all quality gates pass
4. Import solution to production environment
5. Publish to production channels
6. Monitor analytics for 24h post-publish
7. Roll back if any metric drops > 10%~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.