Evaluate AI Observability

When to Use

• Audit telemetry coverage (are all AI pipeline stages instrumented?)
• Verify alert rules cover critical failure scenarios
• Check for PII leakage in logs and traces
• Validate dashboard accuracy against actual metrics
• Assess data retention compliance

Observability Metrics & Targets

Step 1: Audit Telemetry Coverage

// Check which pipeline stages have traces (last 24h)
traces
| where timestamp > ago(24h)
| distinct operation_Name
| sort by operation_Name

Expected stages: retrieval, generation, safety-check, response, caching. Missing stage = blind spot in observability.

Step 2: Audit Alert Coverage

Every ❌ is a gap — create the alert.

Step 3: PII Scan

// Search for potential PII in traces (last 7 days)
traces
| where timestamp > ago(7d)
| where message matches regex @"\b\d{3}-\d{2}-\d{4}\b"  // SSN pattern
    or message matches regex @"\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b"  // Email
    or message matches regex @"\b\d{16}\b"  // Credit card
| project timestamp, operation_Name, message
| take 100

• Zero PII allowed in production logs
• If found: add telemetry processor to scrub before export
• Check both structured and unstructured log fields

Step 4: Dashboard Accuracy

• Compare dashboard numbers to raw KQL queries
• Verify aggregation logic (sum vs avg vs percentile)
• Check time range alignment (UTC vs local)
• Test dashboard with no data (shows "no data" not errors)

Step 5: Data Retention Compliance

Step 6: Generate Report

python evaluation/eval.py --all --output evaluation/observability-report.json

Observability Gate Decision

Common Issues

Evaluation Cadence

• Pre-deployment: Full observability audit checklist
• Weekly: Review alert fire rate, false positive ratio
• Monthly: PII scan, retention compliance check
• Quarterly: Full telemetry coverage audit
• On pipeline change: Verify new stages are instrumented