spark-and-distributed-processing — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited spark-and-distributed-processing (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
Use this skill for batch workloads that exceed single-node processing and need distributed execution discipline. It helps agents reason about Spark, managed Spark services such as Glue and EMR, partitioning, joins, storage layout, and failure-safe processing.
Spark jobsSpark, Glue, EMR, or smaller enginesIceberg, Delta, or HudiDo not use this for lightweight local transforms that fit comfortably in a single process.
Check:
Spark: direct control and broad ecosystem supportGlue: managed AWS-native Spark executionEMR: broader cluster control for Spark and related enginesLambda, serverless Glue, hard timeout ceilings): load spark-serverless-reliability-and-state-managementAccount for:
Validate input assumptions before expensive execution and verify output contracts before publish.
Historical batch recomputation should define overwrite, merge, or append semantics explicitly.
| Rationalization | Reality |
|---|---|
| "Spark will handle optimization for us." | Engine optimizations help, but poor partitioning, skew, and write strategy still create failures or huge cost. |
| "We can just scale the cluster." | Scaling often masks bad physical design and can still fail on skew or bad shuffles. |
| "Managed Spark means we do not need runtime design." | Glue and EMR still require deliberate partitioning, retries, and storage strategy. |
Spark is chosen without a scale or latency reason~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.