Fai Evaluation Framework

Sets up an AI evaluation framework with metrics, test sets, and CI/CD integration.

Overview

This skill provides a structured, repeatable procedure for sets up an ai evaluation framework with metrics, test sets, and ci/cd integration.. It can be used standalone as a LEGO block or auto-wired inside solution plays via the FAI Protocol.

Category: Evaluation Complexity: Medium Estimated Time: 10-30 minutes

Parameters

Steps

Step 1: Validate Prerequisites

Verify all required tools, credentials, and dependencies are available.

# Check required tools
command -v node >/dev/null 2>&1 || { echo 'Node.js required'; exit 1; }
command -v az >/dev/null 2>&1 || { echo 'Azure CLI required'; exit 1; }

Step 2: Load Configuration

Read settings from the FAI manifest and TuneKit config files.

# Load from fai-manifest.json if inside a play
CONFIG_DIR="${config_path:-config}"
if [ -f "fai-manifest.json" ]; then
  echo "FAI Protocol detected — auto-wiring context"
fi

Step 3: Execute Core Logic

Perform the primary operation: sets up an ai evaluation framework with metrics, test sets, and ci/cd integration..

Step 4: Validate Results

Verify the output meets quality thresholds and WAF compliance.

# Validate output
if [ "$?" -eq 0 ]; then
  echo "✅ Skill completed successfully"
else
  echo "❌ Skill failed — check logs"
  exit 1
fi

Output

WAF Alignment

Compatible Solution Plays

• Play 03
• Play 60

Error Handling

Usage

Standalone

# Run this skill directly
npx frootai skill run fai-evaluation-framework

Inside a Solution Play

When referenced in fai-manifest.json, this skill auto-wires with the play's context:

{
  "primitives": {
    "skills": ["skills/fai-evaluation-framework/"]
  }
}

Via Agent Invocation

Agents can invoke this skill using the /skill command in Copilot Chat.

Metrics Reference

Test Set Format

{"question": "What is RAG?", "context": "RAG combines...", "expected": "Retrieval-Augmented Generation..."}
{"question": "How does chunking work?", "context": "Documents are split...", "expected": "Chunking divides..."}

CI/CD Integration

# .github/workflows/eval.yml
- name: Run FAI Evaluation
  run: |
    python evaluation/eval.py --test-set evaluation/test-set.jsonl
    python evaluation/check-thresholds.py --groundedness 0.85 --coherence 0.80

Regression Tracking

Track evaluation scores over time to detect quality regressions:

# Compare with baseline
python evaluation/regression.py --baseline results/baseline.json --current results/latest.json

Notes

• This skill follows the FAI SKILL.md specification
• All outputs are deterministic when dry_run=true
• Integrates with FAI Engine for automated pipeline execution
• Part of the Evaluation category in the FAI primitives catalog