sota-detection-engineering — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited sota-detection-engineering (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
Assume prevention fails. This skill builds and audits the layer that notices: detective controls, the SOC that triages them, the hunts that find what alerts miss, and the IR process that contains what hunts surface. One question defines success:
When a real adversary acts inside your environment, does a high-fidelity signal fire, reach a human (or automation) with the context to act, and drive a bounded response — fast enough to matter?
Detection is engineering, not art. Detections are code: version-controlled, peer-reviewed, CI-tested, ATT&CK-mapped, FP-budgeted, and retired when stale. The dominant failure mode is not missing rules — it is alert fatigue: noise that buries the one true positive. Optimize signal-to-noise relentlessly.
Ownership boundary. sota-observability owns the telemetry pipeline (logs, metrics, traces, SLOs, log shipping, retention plumbing). This skill owns turning that telemetry into security detections, the SOC workflow, hunting, and IR. sota-threat-modeling owns design-time threat enumeration (STRIDE/ ATT&CK/ATLAS catalogs); this skill owns catching those threats at runtime. If you find yourself designing the logging schema, that's observability rules/01; if you find yourself enumerating threats on a DFD, that's threat-modeling.
Run the detection lifecycle as a loop, not a one-shot. Hypothesis → build → test → deploy → tune → retire. Workflow:
abuse case, the adversary behavior, and the telemetry that would witness it. Use the ADS framework (Palantir): goal, categorization (ATT&CK), strategy abstract, technical context, blind spots/assumptions, false positives, validation, priority. Write this before the rule.
collect. Map the hypothesis to a concrete data source (EDR, cloud audit, K8s audit, network/flow, identity, app). If it's missing, the deliverable is a logging gap, not a rule. See rules/02.
TTP/behavioral logic over brittle hashes/IPs/domains. IOC matches are cheap and disposable; TTP detections cost the adversary real money to evade.
agnostic, compiled to your SIEM), YARA-X for file/memory/malware, Suricata for network, Falco/Tetragon for eBPF runtime/container/K8s, SIEM-native (KQL/SPL/EQL) for correlation the portable formats can't express.
allowlist context, require corroboration for noisy signals, set a severity honestly. Every detection ships with a runbook (link to observability rules/04 alerting plumbing) and an owner.
(endpoint), Stratus Red Team (cloud), Caldera (campaigns). Confirm the detection fires on the real technique and stays quiet on benign baselines. No detection merges without a passing test. See rules/06.
Navigator. Coverage heatmaps reveal blind spots — feed them back to step 1.
delete detections nobody trusts. A muted alert is worse than none.
For hunting and deception, see rules/05; for IR, see rules/06.
Assess an existing detection/SOC/IR posture adversarially. Read rules/06 (IR & validation) and rules/04 (SOC/triage) first. Sample real detections, real alerts, and real incidents — do not trust a coverage dashboard or a wiki runbook that has never fired. The cardinal test: pick three ATT&CK techniques relevant to the environment and prove, end to end, that each would be caught.
Severity:
| Severity | Meaning | Examples |
|---|---|---|
| Critical | Blind to a primary attack path, or IR cannot execute | No log source for the crown-jewel system; no EDR/cloud-audit/K8s-audit collection; no IR plan or no one on call; detections exist but nothing routes alerts to a human |
| High | Major coverage gap or SOC dysfunction | Alert fatigue (analysts mute/ignore); detections never tested against the technique; IOC-only coverage of behaviors that need TTP logic; runbooks absent or stale; no ATT&CK coverage map; retention too short for IR |
| Medium | Degraded fidelity or process gaps | Detections with no owner/ADS doc; suppressions with no expiry; no deduplication/correlation; severity inflation; no purple-team/regression testing; TI not operationalized into detections |
| Low | Hygiene | Detections not in version control; inconsistent naming; no FP metrics; Navigator layer stale; no blameless PIR template |
| Info | Observation / hardening opportunity | Deception not deployed where it'd be high-value; coverage maturity below target; SOAR automation candidates |
Finding format (one per finding):
file:line | rule | severity | effort (trivial/small/medium/large) | fixExample:
detections/aws/iam.yml:14 | ioc-only-detection-of-ttp-behavior | High | medium |
GuardDuty-finding-name match is brittle; rewrite as CloudTrail behavioral
Sigma rule on CreateAccessKey+AttachUserPolicy by non-admin principal,
test with Stratus Red Team aws.persistence.iam-create-admin-access-key.Conclude with the verdict: for the top 3 techniques in scope, is detection PRESENT / PARTIAL / ABSENT end-to-end (signal → alert → human → response), and the shortest path to closing the worst gap.
| File | Read this when... |
|---|---|
rules/01-detection-engineering-discipline.md | Running the detection lifecycle, writing ADS docs, doing detection-as-code (CI/peer review/regression), mapping coverage to ATT&CK + Navigator, applying the Pyramid of Pain, picking maturity targets and metrics (coverage/precision/MTTD) |
rules/02-telemetry-siem-data-layer.md | Deciding what to collect (the #1 gap), choosing SIEM/data-lake, normalizing with OCSF/ECS, sizing retention for IR/hunting, controlling volume/cost, assessing data quality |
rules/03-rule-languages-engines.md | Choosing and writing detections in Sigma, YARA/YARA-X, Suricata, Falco, Tetragon, or SIEM-native (KQL/SPL/EQL); rule quality, specificity, FP-resistance, performance; good/bad examples |
rules/04-alerting-triage-soc-soar.md | Fighting alert fatigue, tuning/suppression with expiry, severity assignment, enrichment, dedup/correlation, runbooks, SOAR + auto-containment guardrails, case management, FP lifecycle, SOC metrics |
rules/05-hunting-intel-deception.md | Hypothesis-driven hunting + the hunt loop, IOC vs IOA/TTP hunting, threat-intel lifecycle + TIP, STIX 2.1/TAXII 2.1, diamond model/kill chain, deception (honeypots/honeytokens/canaries) |
rules/06-incident-response-validation.md | Running IR (NIST SP 800-61r3 / CSF 2.0, PICERL), playbooks, severity classification, containment/eradication/recovery, forensic readiness + chain of custody, blameless PIR, tabletops; validating detections via Atomic Red Team/Caldera/Stratus, purple teaming, regression testing |
rules. Audit log-source coverage against your attack paths before writing a single detection.
ATT&CK-mapped, with an owner and an ADS doc. A detection that isn't tested isn't a detection — it's a hope.
/ Stratus / Caldera proves it fires; a benign baseline proves it stays quiet. No merge without both.
domains. IOCs are a supplement and an enrichment, never the strategy.
failure. Tune aggressively, suppress with expiry, and treat a chronically ignored alert as a Critical defect.
human without next steps. Wire alerting plumbing via sota-observability rules/04; you own the security content.
nobody updates is theater. Coverage drives the next hypothesis.
with asset/identity/TI context so triage is seconds, not minutes.
and authority-to-contain current, and run blameless post-incident reviews that feed new detections.
or honeypot has ~zero false positives. Deploy canaries in the paths attackers must traverse (see sota-secrets-management rules/04 honeytokens).
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.