sota-llm-engineering — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited sota-llm-engineering (Agent Skill) and scored it 96/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 1 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 1 flagged
The text {match} tells the agent to skip the normal "ask the user first" gate. Used adversarially it removes the human-in-the-loop check before destructive or sensitive actions, turning a normally-gated agent into a fire-and-forget executor.
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
One skill, two modes. The rules/ files define the mid-2026 baseline for engineering LLM-powered software that is measured, grounded, bounded, and observable. In BUILD mode you write LLM features that conform to the rules by default. In AUDIT mode you hunt for violations and report them as severity-rated findings. The rules are the single source of truth for both.
Framing: an LLM call is a non-deterministic, expensive, latency-heavy RPC to a dependency that changes underneath you. Everything here follows from that — you don't ship logic you can't measure (evals), you don't trust output you didn't validate (structured output, grounding), you don't run loops you can't stop (budgets), and you don't deploy what you can't trace (observability).
Scope boundary: this skill owns build quality. Prompt injection, the lethal trifecta, tool-call authorization, and model-output-as-untrusted-data live in sota-code-security rules/08; agent sandboxing/isolation lives in sota-sandboxing rules/05; PII/regulatory handling lives in sota-privacy-compliance. Reference them; don't re-derive them.
Freshness rule: model names, prices, context limits, and spec revisions in these files were verified June 2026 and rot fast. When a decision hinges on a specific model/price/limit, re-verify against provider docs (or the project's claude-api-style reference skill) before encoding it. Write code version-agnostically: model IDs and parameters in config, never inline.
produce a 20+ case golden set with graded criteria (rules/01). A feature without an eval is a prototype; treat requests to "just add an LLM call" as a request for an eval + a call.
(code-orchestrated, deterministic) → agent. Escalate only when the eval shows the simpler tier failing (rules/04 has the decision gates).
bounded repair (rules/02); hybrid retrieval with a golden retrieval set when RAG is needed (rules/03); stop conditions + token/cost budgets on every loop (rules/04); retries with jittered backoff, model IDs in config, per-call tracing (rules/05).
can't be written: typed prompt templates instead of f-string soup, versioned prompts in repo, a single gateway module for all model calls that enforces tracing/budgets/retries centrally.
a judge, unbounded agent demanded), implement the rules file's documented mitigation and leave a LLMENG: comment stating the residual risk.
end-of-file checklists against your own diff before declaring done.
Process:
messages.create,chat.completions, generateContent, invoke, raw https://api.), prompt templates, retrieval pipelines, agent loops, eval suites (or their absence), and the config/env that selects models.
systemic finding), 04 (unbounded agents), 05 (cost/reliability), 02 (prompt/output hygiene), 03 (RAG), 06 (data lifecycle).
regression gate in CI, no max_tokens/iteration cap, no timeout, no fallback, no trace of prompt/completion/tokens, no schema on extraction, no retrieval metrics, judge never validated against humans.
you think (run the template), confirm the parser actually handles refusals /truncation/stop_reason, confirm the "eval" actually gates merges rather than decorating a dashboard.
shipped, runaway bill, silent regression on model bump). If unverified, say so and rate conservatively.
| Severity | Criteria | Examples |
|---|---|---|
| Critical | User-facing wrongness or unbounded damage with no detection/limit: consequential output (money, health, legal, code-exec) shipped with no eval and no human gate; agent loop with no iteration/cost cap; auto-upgraded model with no regression gate; prompt/completion logs leaking secrets or PII wholesale | while True: tool loop, no budget; prod extraction parsed with regex from free text feeding payments; model="latest" with zero evals |
| High | Silent quality/cost failure likely in normal operation: no eval suite on a shipped LLM feature; parse-and-pray JSON (no schema/validation); RAG with no retrieval eval or no grounding; judge-as-metric never validated; no retry/backoff/fallback on a user-facing path; cache-busting prompt structure tripling spend | json.loads(response.text) with no repair/reject path; timestamp interpolated at top of system prompt; recall@k never measured |
| Medium | Degraded quality/efficiency or weakened safeguards: golden set <20 cases or stale; eval not in CI; no pairwise/error analysis discipline; missing stop_reason handling; unpinned embedding model version; few-shot examples contradicting instructions; no streaming on long user-facing outputs | Eval exists but only run manually; chunking by fixed chars across tables; temperature/params copy-pasted without rationale |
| Low | Hygiene/debt: prompts unversioned outside repo; no token-count telemetry; missing batch API on offline workloads; verbose few-shot inflating cost; TODO repair loops | Prompt edited in a dashboard, not git; offline summarizer paying real-time prices |
Adjust one band up/down for context: consequential domain (↑), internal prototype behind a flag (↓), volume/blast radius (↑ at scale), compensating human review (↓).
[SEVERITY] <title>
File: <path>:<line>
Rule: rules/<NN> §<section>
Failure scenario: <what concretely goes wrong, for whom, at what scale/cost>
Evidence: <code/config/trace anchoring the claim>
Fix: <specific change, referencing the rules/ pattern>Order Critical→Low; lead with an executive summary (counts, worst finding, systemic themes — "no evals anywhere" is one systemic finding, not twenty).
| File | Topics | Read this when... |
|---|---|---|
| rules/01-evals.md | Eval-first development, golden sets, assertion/rubric/pairwise evals, LLM-as-judge + judge validation, offline vs online, CI regression gates, production sampling into eval sets, error analysis, metric pitfalls (judge bias, contamination) | ...building ANY LLM feature (start here), changing a prompt/model, reviewing whether quality claims are real, setting up CI for an LLM repo |
| rules/02-prompt-context-engineering.md | System prompt structure, instruction placement, few-shot selection, context budget management, caching-aware prefix design, prompt versioning/testing, injection-safe template interpolation, structured output (schema/tool-based) + validation/repair, prompt rot | ...writing or editing prompts/templates, extraction or classification features, diagnosing cost spikes or cache misses, anything that parses model output |
| rules/03-rag-retrieval.md | RAG vs long context vs fine-tuning, chunking strategies, embedding selection & versioning, hybrid retrieval (dense+lexical+rerank), query transformation, retrieval evals (recall@k, golden retrieval sets), grounding & citation, freshness/invalidation, agentic retrieval | ...building search/Q&A over documents, choosing a vector DB/embedding model, "the answers are wrong/stale", evaluating an existing RAG pipeline |
| rules/04-agents-tools.md | Workflow vs agent decision, tool design (descriptions, scope, idempotency, model-actionable errors), context management across turns (compaction, memory), MCP integration & spec status, multi-agent costs, human-in-the-loop gates, stopping conditions & budgets | ...anything with tool calls or loops, designing tool schemas, MCP servers/clients, multi-agent proposals, runaway-cost or stuck-agent debugging |
| rules/05-production-engineering.md | Model selection & routing, capability tiers, fallbacks, provider abstraction caution, latency (streaming, parallel calls, caching), cost engineering (budgets, batch APIs, right-sizing), 429/529 + jittered backoff, LLM observability (trace every call), graceful degradation, versioning & rollout (shadow, A/B, pinning vs auto-upgrade) | ...shipping to production, picking models, cost/latency complaints, retry/fallback design, rollout of a new model or prompt, building dashboards |
| rules/06-data-lifecycle.md | Fine-tune vs prompt vs RAG decision, dataset curation hygiene, feedback loops (thumbs → eval sets), embedding/index migration, PII in prompts/logs, provider data-retention settings | ...someone proposes fine-tuning, designing feedback capture, migrating embedding models, setting up logging/retention for an LLM app |
Violations are findings regardless of context; in BUILD mode they are never acceptable shortcuts:
with graded criteria, runnable by one command, gating CI. (rules/01)
"It looked better on three examples" is not evidence. (rules/01)
measured, bias checked) before its scores are trusted. (rules/01)
tool call), validated, with a bounded repair-or-reject path — never regex/parse-and-pray on free text. (rules/02)
delimited and labeled as untrusted; never f-string-concatenated into the instruction stream. (rules/02 + sota-code-security rules/08)
tokens/cost, wall-clock timeout — and human gates on consequential actions. (rules/04)
(golden retrieval set, recall@k), answers cite sources, and "not in the corpus" is an honored outcome. (rules/03)
counts, latency, model ID, cache hits, cost — linked to eval/run IDs. (rules/05)
(shadow or A/B with eval gates) — never silent auto-upgrade. (rules/05)
honoring retry-after, idempotency on retried writes, a degradation path when the provider is down. (rules/05)
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.