deploy-ai-observability — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited deploy-ai-observability (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
az account showaz bicep versionaz bicep lint -f infra/main.bicep
az deployment group create -g $RG -f infra/main.bicep -p infra/parameters.jsonResources:
# OpenTelemetry + Azure Monitor exporter
from azure.monitor.opentelemetry import configure_azure_monitor
configure_azure_monitor(connection_string=os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"])
# Custom AI metrics
from opentelemetry import metrics
meter = metrics.get_meter("ai-workload")
token_counter = meter.create_counter("ai.tokens.total", description="Total tokens consumed")
latency_hist = meter.create_histogram("ai.latency.ms", description="AI call latency")
quality_gauge = meter.create_up_down_counter("ai.quality.score", description="Quality score")| Metric | Type | What to Track |
|---|---|---|
ai.tokens.input | Counter | Input tokens per request |
ai.tokens.output | Counter | Output tokens per request |
ai.tokens.cost | Counter | Dollar cost per request |
ai.latency.ttft | Histogram | Time to first token |
ai.latency.total | Histogram | Total request latency |
ai.quality.groundedness | Gauge | Groundedness score |
ai.quality.relevance | Gauge | Relevance score |
ai.safety.blocked | Counter | Content safety blocks |
ai.cache.hit | Counter | Semantic cache hits |
ai.error.rate | Counter | Failed AI calls |
// Token usage by model (last 24h)
customMetrics
| where name startswith "ai.tokens"
| summarize total=sum(value) by model=tostring(customDimensions.model), bin(timestamp, 1h)
| render timechart
// Latency percentiles
customMetrics
| where name == "ai.latency.total"
| summarize p50=percentile(value, 50), p95=percentile(value, 95), p99=percentile(value, 99)
by bin(timestamp, 1h)
| render timechart
// Cost per tenant
customMetrics
| where name == "ai.tokens.cost"
| summarize cost=sum(value) by tenant=tostring(customDimensions.tenant)
| top 10 by cost desc| Alert | Condition | Severity | Action |
|---|---|---|---|
| High latency | p95 > 5s for 5 min | Warning (Sev 2) | Email + Teams |
| Error spike | Error rate > 5% for 10 min | Critical (Sev 1) | PagerDuty |
| Cost spike | Daily cost > 2× baseline | Warning (Sev 2) | |
| Quality drop | Groundedness < 0.7 for 1 hour | Critical (Sev 1) | PagerDuty |
| Safety block spike | Blocks > 10% for 15 min | Warning (Sev 2) | Teams |
| Token budget exceeded | Monthly tokens > budget | Info (Sev 3) |
# Trace AI pipeline stages
with tracer.start_as_current_span("ai-pipeline") as span:
span.set_attribute("ai.model", "gpt-4o")
span.set_attribute("ai.play", "01-enterprise-rag")
with tracer.start_as_current_span("retrieval"):
# Search + retrieve context
with tracer.start_as_current_span("generation"):
# LLM call
with tracer.start_as_current_span("safety-check"):
# Content safety validation| Issue | Cause | Fix |
|---|---|---|
| No telemetry appearing | Missing connection string | Set APPLICATIONINSIGHTS_CONNECTION_STRING env var |
| Custom metrics not showing | Wrong meter name | Verify meter name matches KQL query |
| Alerts not firing | Threshold too high | Lower threshold, verify action group |
| Traces incomplete | Missing span context propagation | Use W3C trace context headers |
| PII in logs | Unfiltered user input | Add telemetry processor to scrub PII |
| High Log Analytics cost | Verbose log level | Set to Warning in production |
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.