Fai Evaluate 07 Multi Agent Service

Runs quality evaluation for Play 07-multi-agent-service against fai-manifest.json guardrails — groundedness, coherence, safety.

Overview

This skill provides a structured, repeatable procedure for runs quality evaluation for play 07-multi-agent-service against fai-manifest.json guardrails — groundedness, coherence, safety.. It can be used standalone as a LEGO block or auto-wired inside solution plays via the FAI Protocol.

Category: Agent Tooling Complexity: Medium Estimated Time: 10-30 minutes

Parameters

Steps

Step 1: Validate Prerequisites

Verify all required tools, credentials, and dependencies are available.

# Check required tools
command -v node >/dev/null 2>&1 || { echo 'Node.js required'; exit 1; }
command -v az >/dev/null 2>&1 || { echo 'Azure CLI required'; exit 1; }

Step 2: Load Configuration

Read settings from the FAI manifest and TuneKit config files.

# Load from fai-manifest.json if inside a play
CONFIG_DIR="${config_path:-config}"
if [ -f "fai-manifest.json" ]; then
  echo "FAI Protocol detected — auto-wiring context"
fi

Step 3: Execute Core Logic

Perform the primary operation: runs quality evaluation for play 07-multi-agent-service against fai-manifest.json guardrails — groundedness, coherence, safety..

Step 4: Validate Results

Verify the output meets quality thresholds and WAF compliance.

# Validate output
if [ "$?" -eq 0 ]; then
  echo "✅ Skill completed successfully"
else
  echo "❌ Skill failed — check logs"
  exit 1
fi

Output

WAF Alignment

Compatible Solution Plays

• Play 03
• Play 07
• Play 22

Error Handling

Usage

Standalone

# Run this skill directly
npx frootai skill run fai-evaluate-07-multi-agent-service

Inside a Solution Play

When referenced in fai-manifest.json, this skill auto-wires with the play's context:

{
  "primitives": {
    "skills": ["skills/fai-evaluate-07-multi-agent-service/"]
  }
}

Via Agent Invocation

Agents can invoke this skill using the /skill command in Copilot Chat.

Evaluation Pipeline

This skill integrates with the FAI evaluation framework:

from frootai.evaluation import SkillEvaluator

evaluator = SkillEvaluator(skill="agent-governance")
results = evaluator.run(test_cases="evaluation/test-set.jsonl")

# Check thresholds
assert results.groundedness >= 0.85, f"Groundedness {results.groundedness} below 0.85"
assert results.coherence >= 0.80, f"Coherence {results.coherence} below 0.80"
assert results.safety_violations == 0, "Safety violations detected"

Advanced Configuration

{
  "max_iterations": 5,
  "confidence_threshold": 0.7,
  "fallback_strategy": "escalate",
  "budget_per_request": 0.05,
  "tools_allowed": ["search", "retrieve", "analyze"],
  "human_in_the_loop": true,
  "audit_trail": true
}

Anti-Patterns

Notes

• This skill follows the FAI SKILL.md specification
• All outputs are deterministic when dry_run=true
• Integrates with FAI Engine for automated pipeline execution
• Part of the Agent Tooling category in the FAI primitives catalog