Findings & checks · 1 flagged
MEDIUMInstruction telling the agent not to ask for approvalSS-SKILL-INJECT-DONT-ASK-01 · Prompt injection · CLAUDE.md
The text {match} tells the agent to skip the normal "ask the user first" gate. Used adversarially it removes the human-in-the-loop check before destructive or sensitive actions, turning a normally-gated agent into a fire-and-forget executor.
- Delete blanket "don't ask / no need to confirm" directives from the skill.
- If the skill is a genuine autonomous job, restrict the opt-out to a named non-destructive action rather than all actions.
Trace & refs
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
azure-compliance-mcp
Honest Azure compliance for LLM agents — ask infra-health questions in plain English and get answers that distinguish a realfailfrom anot_evaluable, and never fake a `pass`.
An MCP server, built with [FastMCP](https://gofastmcp.com) 3.x, that exposes read-only Azure resource-compliance data so an agent can answer questions like "which storage accounts allow TLS 1.0?" or "which VMs have no disk encryption?" — each finding tagged with where the verdict came from and whether it could be evaluated at all.
Demo
Why honest compliance?
Most tooling collapses "compliant", "non-compliant", and "couldn't tell" into a single pass/fail bit — so a control it can't actually measure silently shows green. This server refuses to guess:
- `fail` — positive evidence a protection is absent (e.g.
encryptionAtHost = false). Actionable; shows up in the default view. - `not_evaluable` — the signal genuinely isn't in the data (e.g. no guest-config policy assigned). Surfaced explicitly, never as a pass.
- `source` — every finding records the signal that decided its verdict:
arg|azure_policy|graph|defender.
Quickstart
Requires uv and Python 3.12+. Runs out of the box — mock mode needs zero Azure setup.
uv sync # install dependencies
uv run python scripts/demo.py # fastest look — runs the tools against mock data
uv run server.py # mock data, stdio transport — works immediatelyPoint it at your own tenant when you're ready:
uv run server.py --mode live # real Azure Resource Graph (e.g. after `az login`)
uv run server.py --transport http # remote: Streamable HTTP
uv run fastmcp dev inspector server.py # explore the tools interactivelyLive mode authenticates with DefaultAzureCredential against your own tenant and is strictly read-only.
Architecture
Tools depend only on a Provider protocol, never on a concrete data source — so the same tool code runs against synthetic mock data or live Azure Resource Graph (ARG), selected at startup with --mode.
flowchart LR
Agent["LLM agent / MCP client"] -->|stdio · HTTP| Server["FastMCP server\n(5 read-only tools)"]
Server --> Protocol["Provider protocol\n(read-only)"]
Protocol -->|--mode mock| Mock["MockProvider\nseeded ARG-shaped data"]
Protocol -->|--mode live| Live["LiveProvider\nDefaultAzureCredential"]
Live -->|injection-safe KQL| ARG[("Azure Resource Graph\nresources · policyresources ·\npatchassessmentresources · authorizationresources")]
Live -->|getByIds| Graph[("Microsoft Graph\nprincipal existence")]Controls (check_compliance evaluates five):
| control | source | signal |
|---|---|---|
required_tags | arg | resources.tags (env / owner / costCenter) |
tls_min_1_2 | arg | resources.properties.minimumTlsVersion |
public_network_access | arg | resources.properties.publicNetworkAccess |
disk_encryption | arg | securityProfile.encryptionAtHost — host-level only; OFF ⇒ fail, ADE not assessed |
guest_config_extension | azure_policy | policyresources guest-config states; no data ⇒ not_evaluable |
Tools (5 of a 5–6 budget):
| tool | source | honest behavior |
|---|---|---|
check_compliance | arg / azure_policy | the five controls above; unmeasurable signal ⇒ not_evaluable, never a fake pass |
query_resources | arg | read-only projection with filters pushed into KQL |
get_patch_status | arg | Update Manager assessments from patchassessmentresources; deallocated VM or no assessment data ⇒ not_evaluable with pendingUpdateCount: null (unknown ≠ 0), never a fake "current" |
find_orphaned_rbac | graph | assignments from ARG authorizationresources, principal existence resolved via Microsoft Graph; orphaned only on a positive directory not-found — unresolvable (no Directory.Read.All) ⇒ not_evaluable, never orphaned on a guess |
Transports: stdio (default, local) and Streamable HTTP (remote, intended behind OAuth 2.1). In stdio mode stdout is reserved for the protocol — logging goes to stderr only.
See SPEC.md for the full tool contracts.
Design decisions & tradeoffs
- Mock provider is the default. The repo runs with zero Azure account, credentials, or network — so reviewers, CI, and new contributors get meaningful output immediately. The mock dataset is shaped exactly like ARG rows, so one mapping serves both modes.
- Read-only by construction. No tool can create, update, or delete a resource. The live path only ever issues ARG queries; there is no mutation surface to misuse.
- Control-based, not Azure-Policy-based. Compliance is evaluated from resource configuration (the ARG row), so it works even when no Azure Policy is assigned — except where the signal genuinely lives elsewhere (guest-config), which routes to
policyresourcesand is honest about it. - The `not_evaluable` + `source` honesty model. A third status plus a provenance field means the agent can tell "this is broken" from "I can't see this from here" — the core reason to trust the output.
- KQL pushdown for scale. Live filters (
query_resources) are pushed into the ARG query (where+take) rather than fetched-then-filtered, so large tenants stay cheap. An opt-in contract test asserts the pushdown matches the reference Python filter, so both modes stay provably consistent. - Injection-safe escaping. ARG has no bind parameters, so every user-supplied value is encoded as an escaped KQL string literal — never concatenated raw. (ARG is read-only regardless, but the discipline is enforced and unit-tested.)
Status & roadmap
v0.3.0 — check_compliance, query_resources, get_patch_status, and find_orphaned_rbac shipped (mock + live), honest not_evaluable/source model across all of them, KQL pushdown, CI on every PR.
Next:
- [ ]
summarize_health— rolled-up infra-health summary - [ ] evals — task-level evaluation of agent answers over the mock dataset
Development
uv run pytest # tests (live tests are opt-in: RUN_LIVE_TESTS=1)
uv run ruff check . # lint
uv run ruff format --check . # formatEvals (opt-in)
Tests prove the tools behave; the eval suite (SPEC §5) checks the property the server exists for: given a natural-language question, does an agent call the right tool, with the right arguments, and answer correctly — including saying "that can't be determined" when the honest answer is not_evaluable.
# needs ANTHROPIC_API_KEY (env or the gitignored .env); calls the API — costs money
uv run --group evals python evals/run.py # full suite (~20 cases)
uv run --group evals python evals/run.py --cases rbac # filter by case id
uv run --group evals python evals/run.py --agent-model <id> # evaluate another agent model
# alternate provider for cost-free smoke runs (needs OPENROUTER_API_KEY);
# a non-Anthropic judge is OFF-PIN vs SPEC §5.4 — the report flags it
uv run --group evals python evals/run.py --provider openrouter \
--agent-model <openrouter-id> --judge-model <openrouter-id>Runs against mock mode only (deterministic, no Azure credentials). Results land in evals/results/<timestamp>-<model>.json plus a rendered summary; they're gitignored except deliberately committed baseline-* files. Structured assertions are exact-match; open-ended prose is scored by a pinned LLM judge whose ~85–92% human-agreement caveat is printed in every report — judge scores are signal, not ground truth. Deliberately not in CI (SPEC §5.6).
Security
- All Azure tools are read-only — nothing in this server modifies Azure resources.
- Secrets, tenant IDs, and
.envare gitignored and must never be committed; live mode uses only your localDefaultAzureCredential. - In stdio mode, logging goes to stderr only (stdout is reserved for the protocol).