sota-data-engineering — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited sota-data-engineering (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
Expert rules for analytical data systems: pipelines, streaming, warehousing, lakehouse storage, data quality, and operations. OLTP schema/query/index craft is owned by sota-databases — reference it, do not duplicate it. Outbox and event-driven service patterns live in sota-architecture; backpressure mechanics in sota-async-concurrency; PII handling in sota-privacy-compliance (or sota-code-security if absent).
Two modes. Pick by intent, then load only the rules/ files the task needs.
Use when designing or implementing pipelines, models, streaming jobs, or storage layouts.
rules/01-architecture-and-modeling.mdbefore choosing tools. Most "big data" is small data; DuckDB/Polars on one node before a distributed engine. ELT into a warehouse/lakehouse is the default shape, not a decision to revisit per pipeline.
safe to rerun for any interval at any time. No blind appends. Design the write strategy (partition overwrite / merge key / insert-overwrite) before the transform logic (rules/02-pipelines-and-orchestration.md).
writing before building it (rules/03-streaming-and-cdc.md).
freshness, volume, uniqueness, and not-null checks tiered block/warn (rules/04-data-quality-and-contracts.md).
clustering/sort, file sizing, and the maintenance job are part of the table's definition (rules/05-storage-and-performance.md).
risky changes, freshness alerting, a runbook entry (rules/06-operations-and-governance.md).
Use when reviewing an existing pipeline repo, dbt project, streaming topology, or warehouse.
Procedure:
other), storage/table formats, streaming components, quality tooling, environments. Read the actual DAGs/models — never audit from README claims.
write by reading the write statement, not by inferring from naming.
Severity conventions:
that double-count on retry, reruns that duplicate or lose data, CDC deletes not applied, PII landing in unprotected zones, prod credentials in dev.
business-critical pipelines, unbounded retries, full-table rescans of large sources each run, no backfill path, schema changes that break consumers.
SELECT *staging, small-files accumulation, no documentation/lineage, warn-tier checks failing for weeks.
suboptimal compression.
Finding format:
[SEVERITY] <one-line title>
Where: <file:line / model / DAG / topic>
Evidence: <the actual code/config/SQL that proves it>
Impact: <what goes wrong, when>
Fix: <concrete change, smallest safe diff>| File | Read this when... |
|---|---|
rules/01-architecture-and-modeling.md | Choosing engines/architecture (warehouse vs lakehouse vs DuckDB), designing layers (staging/core/mart), dimensional modeling, SCDs, One Big Table, semantic layers, evaluating a data-mesh pitch. |
rules/02-pipelines-and-orchestration.md | Writing or reviewing any batch pipeline: idempotency, incremental loads, watermarks, late data, backfills, Airflow/orchestrator DAG design, dbt project discipline, scheduling strategy. |
rules/03-streaming-and-cdc.md | Anything Kafka/Flink/CDC: deciding streaming vs micro-batch, partition keys, consumer groups, offsets, exactly-once claims, schema registry, Debezium, tombstones, DLQs, windowing. |
rules/04-data-quality-and-contracts.md | Defining data contracts, adding expectation tests, tiering checks block vs warn, drift/anomaly detection, lineage, responding to a data incident, testing transforms in CI. |
rules/05-storage-and-performance.md | Creating tables, Parquet tuning, partitioning vs clustering, small-files/compaction, Iceberg/Delta features and maintenance, compression, reducing scan cost / warehouse spend. |
rules/06-operations-and-governance.md | Environments and deployment of pipeline changes, write-audit-publish, blue-green tables, access control, GDPR deletion in lakehouses, pipeline observability, runbooks. |
produces the same result. Overwrite partitions or MERGE on keys; a blind INSERT INTO ... SELECT in a scheduled job is a CRITICAL finding.
freshness alerts routed to an owner. A pipeline that fails quietly is worse than one that doesn't exist — people keep trusting its output.
DuckDB/Polars beats a cluster on cost, speed, and ops. Spark/distributed engines need a stated reason (data volume, existing platform, ML scale).
Process data by the orchestrator-supplied logical interval; reruns and backfills must produce identical results regardless of wall-clock time.
freely; never rename, retype, or drop in place. Producers that break consumers without a contract bump are HIGH findings.
for uniqueness/null/contract violations on critical models; warn (alert, continue) for distribution drift. Everything-blocks and nothing-blocks are both failure modes.
needs sub-minute latency. Hourly micro-batch covers most "real-time" requests at a tenth of the operational cost.
Kafka; your sink makes it true via idempotent writes (merge keys, deterministic IDs). Audit the sink, not the producer config.
rename, and type columns. Upstream schema drift must break loudly in your staging layer, not silently in a dashboard.
cleanup, and retention jobs are part of creating an Iceberg/Delta table. A lakehouse without maintenance jobs is a slow-motion outage.
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.