scan-health: Service Health Monitor

The "eyes" of the harness. Checks configured HTTP endpoints for availability and response time, compares against baseline metrics, and generates alerts when thresholds are breached.

• Checks configured HTTP endpoints for availability and response time
• Compares against baseline metrics stored in service_health.json
• Generates alerts when thresholds are breached
• READ-ONLY: no code changes, no PRs — writes only to agent/state/service_health.json

Safety Rules

1. HALT file — Check config.safety.halt_file before any work. If present, print reason and stop.
2. Read-only — Only writes to agent/state/service_health.json. No external API modifications.
3. Graceful degradation — Record failures without crashing. Missing config → skip with a message.

Step 0: Safety

Check the HALT file at config.safety.halt_file. If it exists, print: HALT: <reason>. scan-health aborted. and exit.

Check config.health_endpoints. If missing or empty: No health endpoints configured. Skipping. — exit 0.

Validate the list before proceeding:

• Deduplicate — remove duplicate URLs (keep first occurrence).
• Reject malformed URLs — entries that do not start with http:// or https:// are skipped with a warning: [WARN] Skipping malformed URL: <url>.
• Cap at 20 endpoints — if more than 20 remain after dedup, check only the first 20 and warn: [WARN] Endpoint list truncated to 20 (had <N>).

Step 0.5: Lens Load (Adaptive Context)

Read agent/state/service_health/lens.json. If missing or unparseable, proceed with base behavior only — this step is purely additive.

If lens exists:

• focus_items (confidence >= 0.6): Prioritize these endpoints/metrics first.

Allocate extra time and retries to high-priority items.

• learned_thresholds (confidence >= 0.6): Override default thresholds. For

example, if the lens says endpoint X has a learned threshold of 800ms instead of the default 1500ms, use 800ms for anomaly detection on that endpoint.

• discovered_signals (actionable=true): Include these as additional diagnostic

checks beyond the base behavior. For example, if the lens says "deploy failures correlate with health degradation", check recent deploy status first.

If lens is corrupt (invalid JSON, missing schema_version): Log: "[WARN] Lens file corrupt, using base behavior." Continue normally. Do NOT crash.

Step 1: Baseline Load

Read agent/state/service_health.json. If missing, create with initial structure:

{
  "schema_version": "1.0.0",
  "last_run": null,
  "run_count": 0,
  "status": "unknown",
  "alerts": [],
  "baseline": { "endpoints": [] }
}

Extract previous per-endpoint avg_response_ms, last_status, consecutive_failures for use in anomaly detection.

Step 2: Endpoint Checks

For each URL in config.health_endpoints:

curl -s -o /dev/null -w "%{http_code} %{time_total}" --max-time <timeout_seconds> <url>

Record: url, status_code, response_time_ms, timestamp. Timeout: config.health_check_timeout_seconds (default 10, valid range 2-30).

Method support: if the endpoint object includes a method field (e.g., "method": "POST"), pass it to curl via -X {method}. Default: GET.

curl -s -o /dev/null -w "%{http_code} %{time_total}" -X {method} --max-time <timeout_seconds> <url>

Status code note: curl's %{http_code} returns a three-digit string. On connection refused, DNS failure, or timeout it returns 000 (not 0). Treat both `0` and `000` as network failure — always compare numerically (e.g., parseInt(code) === 0) rather than by string equality.

Response time: curl's %{time_total} is in seconds (float). Convert to milliseconds: response_time_ms = time_total * 1000.

Retry once on network failure (status 0 / 000 / connection refused). Skip retry on timeout (saves waiting another full timeout period).

If curl unavailable, fall back to wget --server-response --timeout=<timeout_seconds>. If both unavailable, record status_code: 0, error: "no_http_client".

Step 3: Anomaly Detection

Expected status override: if the endpoint object includes expected_status (e.g., 301), treat that status code as healthy (equivalent to 2xx). This allows monitoring redirect-based or custom health endpoints. When expected_status is set, suppress the corresponding alert type (e.g., endpoint_redirect for 3xx). If the response does NOT match expected_status, generate an unexpected_status alert with severity warning.

Baseline comparison on failure — when an endpoint is DOWN (HTTP 000, connection refused, or timeout with no response), include baseline context in the alert detail so the user understands what changed:

Previous baseline: {avg_response_ms}ms / 200 OK (from {samples} samples)
Current: CONNECTION REFUSED
Change: endpoint_down (was healthy)

If no baseline exists for the endpoint (first run), omit the "Previous baseline" line and note Change: endpoint_down (no prior baseline).

Alert shape: { "severity": "warning", "type": "...", "endpoint": "...", "detail": "..." }

Increment consecutive_failures per endpoint on 5xx or status 0/000 (timeout/connection refused). Reset to 0 on any 2xx response. Do NOT increment on 3xx or 403 (endpoint is reachable).

Step 4: State Update

Write to agent/state/service_health.json:

{
  "schema_version": "1.0.0",
  "last_run": "<ISO 8601>",
  "run_count": "<N>",
  "status": "healthy | degraded | critical",
  "consecutive_failures": 0,
  "alerts": [{ "severity": "warning", "type": "slow_response", "endpoint": "...", "detail": "..." }],
  "baseline": {
    "endpoints": [{ "url": "...", "avg_response_ms": 130, "last_status": 200, "consecutive_failures": 0 }]
  }
}

Top-level consecutive_failures = the MAX of the per-endpoint consecutive_failures values, written every run. This is the variable evolve's 4-B chain trigger evaluates (consecutive_failures >= 3) — trigger conditions read top-level state fields, so the nested per-endpoint counters alone would leave the trigger undecidable. Also: when config.health_endpoints is empty, write status: "no_endpoints" (readers must treat it as neither healthy nor critical — informational only).

Status: healthy = all 2xx, no warning/critical alerts (info alerts are OK). degraded = any warning alert. critical = any critical alert. Update avg_response_ms using an exponential moving average: new_avg = old_avg * 0.8 + current_response_ms * 0.2. On the first run for an endpoint (no prior baseline), set avg_response_ms = current_response_ms directly.

EMA on failure: when status_code is 0/000 (connection refused, DNS failure, timeout), do NOT update avg_response_ms — the response time is meaningless (either ~0ms for connection refused or the full timeout duration). Keep the previous avg_response_ms intact so it remains a reliable baseline of healthy performance.

Note: This skill does NOT write to agent/state/service_health/lens.json. Lens updates (learning thresholds, promoting signals, adjusting focus) happen exclusively in evolve's Reflect phase (Step 5-E).

Step 5: Report

scan-health — <ISO timestamp>
Overall status: healthy | degraded | critical

| Endpoint                 | Status | Response (ms) | Baseline (ms) | Alert          |
|--------------------------|--------|---------------|---------------|----------------|
| https://example.com/     |  200   |     142       |     130       | —              |
| https://example.com/api  |  503   |     —         |     120       | endpoint_down  |

If alerts exist: Alerts: N warning, N critical. Consider running /dev-cycle for investigation. If consecutive_failures >= 3, note that the dev-cycle chain trigger condition is met.

Graceful Degradation