Data Engineering Agent Skills

Data Engineering Agent Skills

-purple.svg)](#end-to-end-examples)

🌐 [Project Website](https://vaquarkhan.github.io/data-engineering-agent-skills/) | 📦 [VS Code Marketplace](https://marketplace.visualstudio.com/items?itemName=ViquarKhan.data-engineering-agent-skills) | 🔌 [JetBrains Marketplace](https://plugins.jetbrains.com/plugin/32163-data-engineering-agent-skills) | ⬇️ [GitHub Releases](https://github.com/vaquarkhan/data-engineering-agent-skills/releases/latest)

Production-grade data engineering skills for AI agents.

The open skill registry and execution toolkit for data engineering agents.

This repository packages repeatable workflows, quality gates, hooks, installer surfaces, and example packs (runnable scaffolds plus architecture blueprints) so agents can build data systems with the same discipline used by strong data engineering teams.

The goal is not to give agents generic prompts. The goal is to give them operating procedures for defining, planning, implementing, validating, replaying, and shipping reliable data products.

Agent Skills Registry Compatibility

This repository is structured to work with open Agent Skills registries:

• every capability lives in a directory containing a SKILL.md
• every SKILL.md starts with YAML frontmatter including at least name and description
• descriptions are written for progressive disclosure so agents can decide when to load the full skill
• supporting materials can live in references/, templates/, examples/, hooks/, and scripts/

Real setup paths:

git clone https://github.com/vaquarkhan/data-engineering-agent-skills.git
scripts/install.sh --tool all --target /path/to/project

Windows-friendly file install:

pwsh scripts/install.ps1 --tool all --target C:\path\to\project

If your agent or editor supports importing a GitHub-hosted skill registry directly, use that tool's documented repository-install command against this repository URL rather than the example syntax from another project.

For Python-based proof assets and validators, install local dependencies with:

pip install -r requirements.txt

Contract validation quickstart

The dataset contract validator checks sample data against templates/dataset-contract.yaml-style contracts. It requires a contract file and a data source (--data or --duckdb); validating a template alone will error by design.

python scripts/validate_dataset_contract.py --help

python scripts/validate_dataset_contract.py \
  --contract examples/aws-s3-glue-athena-iceberg/contracts/customers-contract.yaml \
  --data examples/aws-s3-glue-athena-iceberg/data/customers.jsonl \
  --reference-time 2026-05-02T00:00:00Z

For dbt + DuckDB local proof:

python scripts/validate_dataset_contract.py \
  --contract examples/dbt-warehouse-marts/contracts/fct_daily_revenue-contract.yaml \
  --duckdb examples/dbt-warehouse-marts/build/dbt_warehouse_marts.duckdb \
  --query "select * from fct_daily_revenue order by order_date"

Bootstrap install shortcuts:

./bootstrap.sh /path/to/project auto

pwsh .\bootstrap.ps1 C:\path\to\project auto

Quick Start

Start Here

1. load skills/using-data-engineering-agent-skills/SKILL.md
2. pick the closest platform preset from presets/
3. choose the safest next command from the lifecycle below
4. use one starter pack, template, or example to reduce guessing

Install Surfaces

#### 📦 Plugin And Release Downloads

⬇️ [Install from VS Code Marketplace](https://marketplace.visualstudio.com/items?itemName=ViquarKhan.data-engineering-agent-skills) — search "Data Engineering Agent Skills" in VS Code Extensions panel

>

⬇️ [Install from JetBrains Marketplace](https://plugins.jetbrains.com/plugin/32163-data-engineering-agent-skills) — IntelliJ, PyCharm, DataGrip, WebStorm, GoLand

>

⬇️ [Download VS Code Extension (.vsix)](https://github.com/vaquarkhan/data-engineering-agent-skills/releases/latest) — manual install for VS Code, Cursor, Windsurf, VSCodium

>

⬇️ [Download JetBrains Plugin (.zip)](https://github.com/vaquarkhan/data-engineering-agent-skills/releases/latest) — alternative manual download

How to install the `.vsix`:

1. Download the .vsix file from the latest release
2. Open VS Code / Cursor / Windsurf / Kiro
3. Press Ctrl+Shift+P (or Cmd+Shift+P on Mac) to open the Command Palette
4. Type Extensions: Install from VSIX... and select it
5. Browse to and select the downloaded .vsix file
6. Reload the editor when prompted
7. Use the Command Palette again — search for Data Engineering Skills: to see all available commands

How to install the JetBrains `.zip`:

1. Download the .zip file from the latest release
2. Open IntelliJ / PyCharm / DataGrip
3. Go to Settings → Plugins → ⚙️ gear icon → Install Plugin from Disk...
4. Select the downloaded .zip file
5. Restart the IDE when prompted

Additional plugin resources:

• plugin publishing and marketplace workflow setup: docs/plugin-publishing.md
• marketplace publish workflows exist, but this repository should be treated as GitHub Releases-first unless a marketplace listing is explicitly published
• Claude plugin bundle: use .claude-plugin/, .claude/commands/, and CLAUDE.md

#### Install By Tool

#### One-Line Script Install

scripts/install.sh --tool claude --target /path/to/project
scripts/install.sh --tool kiro --target /path/to/project
scripts/install.sh --tool windsurf --target /path/to/project
scripts/install.sh --tool opencode --target /path/to/project
scripts/install.sh --tool all --target /path/to/project
scripts/install.sh --tool auto --target /path/to/project
scripts/install.sh --tool cursor,claude --target /path/to/project
scripts/install.sh --tool all --target /path/to/project --symlink

Windows-friendly install path:

pwsh scripts/install.ps1 --tool all --target C:\path\to\project

Evaluation runner:

python evals/run.py

#### Tool Setup Guides

• docs/getting-started.md
• docs/cursor-setup.md
• docs/kiro-setup.md
• docs/claude-setup.md
• docs/copilot-setup.md
• docs/windsurf-setup.md
• docs/opencode-setup.md
• docs/codex-setup.md
• docs/jetbrains-setup.md
• docs/plugin-publishing.md

#### Tutorials

• tutorials/README.md
• tutorials/using-data-engineering-agent-skills.md
• tutorials/cloud-data-engineering-architecture-patterns.md
• tutorials/pipeline-orchestration-patterns.md
• tutorials/streaming-architecture-patterns.md
• tutorials/data-resiliency-testing-patterns.md
• tutorials/etl-elt-modernization-and-cutover.md
• tutorials/regulated-data-and-compliance-workflows.md
• tutorials/installing-vscode-and-jetbrains-plugins.md

#### Case Studies

• case-studies/README.md
• case-studies/incident-bad-publish-recovery.md
• case-studies/replay-safe-backfill-after-corruption.md
• case-studies/regulated-data-release-gate.md

Core Principles

• Spec before pipeline code
• Contract-first source and output design
• Idempotent, replayable, backfill-safe execution
• Data quality checks before publish
• Lineage, ownership, and governance by default
• Clear evidence for every change

Feature Highlights

• Spec-first lifecycle with /spec, /plan, /build, /validate, /review, /backfill, and /ship
• 73 workflow skills covering ingestion, transformation, orchestration, streaming, lakehouse, warehousing, governance, quality tooling, legacy modernization, release, incident recovery, and platform operating concerns, plus 1 compatibility alias entry skill
• focused resilience testing coverage for failure drills, replay safety, restart behavior, backlog catch-up, publish protection, and disaster recovery readiness
• 14 platform presets spanning AWS, Azure, GCP, Databricks, Snowflake, Alibaba Cloud, Informatica, Talend, and Apache-first stacks
• Multi-agent packaging for Cursor, Claude, Copilot, Gemini, Codex, Kiro, OpenCode, Windsurf, AGENTS.md, and CLAUDE.md consumers
• Install surfaces for VS Code family editors, JetBrains IDEs, setup guides, one-line install scripts, and plugin release artifacts
• Plugin delivery and discovery workflows for VS Code and JetBrains, including release downloads, install smoke tests, markdown linting, and optional marketplace publish automation
• 5 runnable example scaffolds with contract validation, rollback demonstrations, and smoke-test proof paths, plus 9 architecture blueprint examples (spec/plan/tasks only)
• Agent benchmark pack with quantified with-skills coverage improvement (23→67 on the included task set)
• Structured operational templates for dataset contracts, compliance controls, backfills, schema changes, release gates, and incident response

Feature Coverage

Lifecycle Commands

These commands are the clearest way to understand the repo. They can be mapped to slash commands, prompts, hooks, or local workflows.

Choose Your Path

• New pipeline -> use /spec
• dbt project -> use starter-packs/warehouse-analytics-starter.yaml
• Streaming system -> use starter-packs/streaming-reliability-starter.yaml
• Production reliability and failure recovery -> use starter-packs/production-reliability-starter.yaml and tutorials/production-reliability-and-mcp-observability.md
• Serverless Spark or MSK hardening -> use examples/aws-serverless-spark-msk-reliability/
• Streaming architecture review -> use references/streaming-architecture-patterns.md
• Cloud architecture review -> use references/cloud-data-engineering-architecture-patterns.md
• Pipeline orchestration review -> use references/pipeline-orchestration-patterns.md
• Validation and testcase review -> use starter-packs/validation-security-review-starter.yaml
• Resiliency testing -> use starter-packs/resiliency-testing-starter.yaml
• Python data pipeline work -> use python-data-engineering-and-pipeline-packaging
• Scala Spark or JVM jobs -> use scala-data-engineering-on-jvm-runtimes
• Java connectors or data services -> use java-data-engineering-and-integration-services
• MySQL versus NoSQL choice -> use operational-datastore-selection-relational-and-nosql
• Anti-pattern review -> use references/data-engineering-anti-patterns.md
• ETL or ELT modernization -> use etl-elt-and-modernization-strategy
• Partner file or SFTP ingestion -> use file-and-partner-feed-ingestion
• Glue Data Catalog or Lake Formation governance -> use glue-data-catalog-and-lake-formation-governance
• Unity Catalog governance -> use unity-catalog-and-lakehouse-governance
• Azure governance or Purview -> use microsoft-purview-and-azure-data-governance
• Dataplex or BigQuery governance -> use dataplex-and-bigquery-governance
• Snowflake-native pipelines -> use snowflake-native-pipelines-and-governance
• BigQuery or Dataform platform work -> use bigquery-and-dataform-platform-engineering
• Data-quality tool strategy -> use data-quality-platforms-and-rule-management
• Incident recovery -> use incident-triage-and-pipeline-recovery
• Disaster recovery planning -> use data-platform-disaster-recovery-and-business-continuity
• Platform operating model or golden paths -> use data-platform-operating-model-and-service-ownership
• Schema migration -> use schema-evolution-and-contract-migrations
• Cutover or backfill -> use safe-backfill-and-replay-orchestration and data-migration-and-platform-cutover
• CI/CD and progressive release -> use starter-packs/data-platform-cicd-release-starter.yaml
• Regional compliance or sovereignty -> use regional-data-compliance-and-sovereignty
• ESG reporting -> use starter-packs/regional-compliance-and-esg-reporting-starter.yaml
• Regulated data (PII, PCI, HIPAA) -> use data-security-compliance-and-regulated-data
• Test data or synthetic fixtures -> use test-data-preparation-and-synthetic-data
• Lower-environment masked refresh -> use lower-environment-data-masking-and-obfuscation
• Informatica or Talend estate -> use enterprise-etl-and-data-integration-modernization
• Mainframe modernization -> use mainframe-modernization-and-data-offload

Hooks

The hooks/ directory adds a lightweight operating layer:

• session-start.sh
• contract-check-pre.sh
• pipeline-review-pre.sh
• incident-mode.sh
• backfill-guard.sh
• schema-change-guard.sh
• cost-check.sh
• release-guard.sh

Use these hooks when you want the repository to behave more like a workflow system than a static library.

Initial Skill Pack

This repository includes 73 workflow skills plus 1 compatibility alias (using-data-agent-skills redirects to using-data-engineering-agent-skills). The alias exists for older installs; do not count it as a separate workflow.

This repository now includes a broader production-grade skill pack:

• using-data-engineering-agent-skills (canonical start-here skill)
• using-data-agent-skills (compatibility alias — redirect only)
• data-specification
• pipeline-planning-and-task-breakdown
• data-quality-and-contract-testing
• data-resiliency-testing-and-failure-injection
• orchestration-and-backfills
• lineage-pii-and-governance
• data-security-compliance-and-regulated-data
• regional-data-compliance-and-sovereignty
• esg-and-sustainability-regulatory-reporting
• lower-environment-data-masking-and-obfuscation
• spark-and-distributed-processing
• airflow-and-workflow-orchestration
• streaming-and-messaging-systems
• file-and-partner-feed-ingestion
• lakehouse-table-format-engineering
• data-lake-and-zone-architecture
• warehouse-and-schema-design
• python-data-engineering-and-pipeline-packaging
• scala-data-engineering-on-jvm-runtimes
• java-data-engineering-and-integration-services
• operational-datastore-selection-relational-and-nosql
• data-mesh-and-domain-oriented-design
• delta-lake-and-medallion-architecture
• dbt-and-analytics-engineering
• cdc-and-incremental-loading
• schema-evolution-and-contract-migrations
• warehouse-performance-and-cost-optimization
• data-platform-disaster-recovery-and-business-continuity
• data-observability-and-sla-management
• incident-triage-and-pipeline-recovery
• data-platform-operating-model-and-service-ownership
• terraform-and-data-platform-infrastructure
• snowflake-native-pipelines-and-governance
• bigquery-and-dataform-platform-engineering
• semantic-layer-and-metric-governance
• notebook-to-production-hardening
• data-sharing-and-publishing-contracts
• data-catalog-and-discovery
• glue-data-catalog-and-lake-formation-governance
• unity-catalog-and-lakehouse-governance
• microsoft-purview-and-azure-data-governance
• dataplex-and-bigquery-governance
• privacy-retention-and-right-to-delete
• etl-elt-and-modernization-strategy
• mainframe-modernization-and-data-offload
• test-data-preparation-and-synthetic-data
• reverse-etl-and-operational-data-serving
• feature-store-and-ml-data-pipelines
• data-migration-and-platform-cutover
• api-and-saas-ingestion-patterns
• source-reliability-and-extraction-resilience
• enterprise-etl-and-data-integration-modernization
• data-reconciliation-and-financial-controls
• data-platform-ci-cd-and-release-management
• master-data-and-entity-resolution
• debezium-and-kafka-connect-cdc
• apache-beam-unified-batch-and-stream
• apache-hudi-lakehouse
• trino-presto-federated-query
• openmetadata-datahub-and-openlineage
• great-expectations-deequ-and-cuallee
• data-quality-platforms-and-rule-management
• lakefs-and-data-versioning
• clickhouse-real-time-analytics
• superset-and-metrics-serving
• avro-protobuf-json-schema-registry
• duckdb-local-analytics-and-dev
• data-contract-testing-with-schema-registry
• safe-backfill-and-replay-orchestration
• spark-serverless-reliability-and-state-management
• kafka-resilience-and-schema-evolution
• mcp-data-observability-integration

Future skills can extend this pack for:

• additional platform-specific delivery patterns for tools such as EMR, Flink, Kafka, Kinesis, Iceberg, SSIS, DataStage, and Matillion
• enterprise governance overlays and automated validation hooks

Platform Presets

The core skills stay vendor-neutral. Platform-specific guidance lives in presets so agents can adapt the same workflow to the stack a team actually uses.

Current presets:

• aws-data-engineering
• azure-data-engineering
• gcp-data-engineering
• databricks-lakehouse-engineering
• alibaba-cloud-data-engineering
• snowflake-modern-data-platform
• informatica-data-integration
• talend-data-integration
• multi-cloud-hybrid-data-engineering
• apache-spark-engineering
• apache-flink-stream-processing
• apache-airflow-orchestration
• apache-kafka-streaming
• apache-iceberg-lakehouse

Project Structure

data-engineering-agent-skills/
+-- skills/           # Workflow-driven skill definitions
+-- presets/          # Platform and stack-specific operating profiles
+-- references/       # Reusable checklists and guidance
+-- templates/        # Spec, plan, and task templates
+-- registry/         # Machine-readable asset registry and indexes
+-- hooks/            # Session-start and pre-flight automation hooks
+-- docs/             # Setup, onboarding, and format guidance
+-- vscode-extension/ # VS Code installer extension
+-- jetbrains-plugin/ # JetBrains installer plugin
+-- mcp/              # MCP configuration templates
+-- starter-packs/    # Opinionated bundles by use case
+-- agents/           # Specialist personas and review roles
+-- .claude-plugin/   # Claude plugin manifests and install bundles
+-- .kiro/            # Kiro steering adapter files
+-- .opencode/        # OpenCode integration surface
+-- CLAUDE.md         # Claude-style repository entry point
`-- examples/         # Example projects and generated artifacts

Setup Guides

Agent-specific setup guides are available in docs/:

• docs/getting-started.md
• docs/cursor-setup.md
• docs/kiro-setup.md
• docs/claude-setup.md
• docs/copilot-setup.md
• docs/windsurf-setup.md
• docs/opencode-setup.md
• docs/generic-markdown-rules-setup.md
• docs/codex-setup.md
• docs/jetbrains-setup.md

Tutorials

Longer-form walkthroughs live in tutorials/:

• tutorials/README.md
• tutorials/using-data-engineering-agent-skills.md
• tutorials/cloud-data-engineering-architecture-patterns.md
• tutorials/pipeline-orchestration-patterns.md
• tutorials/streaming-architecture-patterns.md
• tutorials/data-resiliency-testing-patterns.md
• tutorials/platform-native-governance-patterns.md
• tutorials/partner-feed-ingestion-and-replay-safe-file-processing.md
• tutorials/disaster-recovery-and-restore-drills.md
• tutorials/mainframe-modernization-and-cutover.md
• tutorials/data-quality-tools-and-rule-operating-model.md
• tutorials/etl-elt-modernization-and-cutover.md
• tutorials/regulated-data-and-compliance-workflows.md
• tutorials/installing-vscode-and-jetbrains-plugins.md

VS Code Extension

The first working VS Code extension scaffold lives in vscode-extension/.

Installable release assets are published from the repository release page as .vsix downloads.

Marketplace publishing is supported through .github/workflows/publish-plugins.yml for VS Code Marketplace and Open VSX when publish secrets are configured.

Start here for the VS Code family:

• vscode-extension/README.md
• tutorials/installing-vscode-and-jetbrains-plugins.md
• docs/plugin-publishing.md
• latest release downloads

It provides command-palette installers for:

• the full toolkit
• the core pack
• agent adapters
• starter packs
• MCP templates
• runnable examples

It is intended for the VS Code family:

• VS Code
• Cursor
• Windsurf
• VSCodium

JetBrains Plugin

[Install from JetBrains Marketplace](https://plugins.jetbrains.com/plugin/32163-data-engineering-agent-skills) — search "Data Engineering Agent Skills" in your IDE's plugin settings.

The plugin source lives in jetbrains-plugin/.

Start here for JetBrains IDEs:

• JetBrains Marketplace listing
• jetbrains-plugin/README.md
• docs/jetbrains-setup.md
• tutorials/installing-vscode-and-jetbrains-plugins.md
• docs/plugin-publishing.md
• latest release downloads

It targets IntelliJ-platform IDEs such as:

• IntelliJ IDEA
• PyCharm
• WebStorm
• DataGrip
• GoLand
• PhpStorm

Multi-Agent Packaging

Native adapters are included for multiple agent ecosystems:

• .cursor/rules/
• .claude/commands/
• .claude-plugin/
• .gemini/commands/
• .kiro/steering/
• .github/copilot-instructions.md
• AGENTS.md
• CLAUDE.md
• .opencode/
• .windsurfrules.example

Direct setup links for common agent surfaces:

• Claude -> docs/claude-setup.md
• Cursor -> docs/cursor-setup.md
• Copilot -> docs/copilot-setup.md
• Kiro -> docs/kiro-setup.md
• OpenCode -> docs/opencode-setup.md
• Windsurf -> docs/windsurf-setup.md
• Codex -> docs/codex-setup.md
• generic AGENTS.md setup -> docs/getting-started.md

Install Scripts

Installation helpers are available in scripts/:

• scripts/install.sh --tool cursor|claude|copilot|gemini|kiro|windsurf|opencode|generic|all --target <path>
• scripts/install.ps1 --tool cursor|claude|copilot|gemini|kiro|windsurf|opencode|generic|all --target <path>
• scripts/install.sh --tool codex --target <path>
• scripts/install_toolkit.py
• scripts/hook_runner.py
• scripts/validate-skills.py
• scripts/validate-assets.py
• scripts/validate_dataset_contract.py
• scripts/check-links.py
• scripts/test-plugin-installation.py
• requirements.txt
• requirements-proof.txt
• scripts/sync-rules.sh
• scripts/sync-rules.ps1

Workflow Automation

Repository automation includes:

• .github/workflows/validate-and-package.yml
• .github/workflows/test-plugin-installation.yml
• .github/workflows/proof-assets.yml
• .github/workflows/agent-benchmarks.yml
• .github/workflows/release-artifacts.yml
• .github/workflows/publish-plugins.yml
• .github/workflows/markdown-lint.yml

Machine-Readable Templates

Starter contract and response templates are available in templates/:

• source-contract.yaml
• dataset-contract.yaml
• metric-contract.yaml
• data-compliance-controls.yaml
• backfill-plan.yaml
• schema-change-plan.yaml
• release-gate-evidence.yaml
• incident-runbook.md

Asset Registry

Machine-readable discovery lives in:

• registry/assets.json

Use it to index templates, starter packs, examples, MCP templates, agent adapters, and installer bundle contents without scraping the Markdown docs.

For human-readable selection of checklists and decision guides, start with:

• references/README.md

MCP Config Templates

Template MCP configs are available in mcp/ for:

• GitHub
• Postgres
• Snowflake
• BigQuery
• Databricks
• dbt Cloud
• Airflow
• Kafka
• Spark
• Terraform
• Slack and Jira incident flows

Each template is a starting point only. Use mcp/README.md for environment-variable requirements and service-specific setup notes before connecting a live system.

End-To-End Examples

The repository includes 14 example packs in examples/:

See examples/README.md for the full selector table.

Runnable scaffolds (5)

• aws-s3-glue-athena-iceberg
• databricks-delta-medallion
• dbt-warehouse-marts
• kafka-flink-streaming
• aws-serverless-spark-msk-reliability

Run make smoke-test inside each directory (or follow the README commands on Windows).

Architecture blueprints (9)

• api-saas-to-warehouse-ingestion
• gcp-pubsub-dataflow-bigquery
• snowflake-dbt-reverse-etl
• privacy-retention-deletion-workflow
• feature-store-online-offline-parity
• multi-cloud-warehouse-cutover
• data-platform-cicd-progressive-release
• esg-regulatory-reporting-foundation
• validation-and-security-review-foundation

Use blueprints for agent planning and /spec → /plan workflows. Use runnable scaffolds when you need executable proof.

Starter Packs

Use the starter packs in starter-packs/ to adopt the repository by problem area:

• aws-lakehouse-starter.yaml
• databricks-medallion-starter.yaml
• warehouse-analytics-starter.yaml
• streaming-reliability-starter.yaml
• privacy-governance-starter.yaml
• regulated-data-compliance-starter.yaml
• data-platform-cicd-release-starter.yaml
• validation-security-review-starter.yaml
• regional-compliance-and-esg-reporting-starter.yaml
• test-data-lower-environments-starter.yaml
• enterprise-etl-modernization-starter.yaml
• resiliency-testing-starter.yaml
• production-reliability-starter.yaml

Skills Catalog

See skills-index.md for a grouped catalog by lifecycle and platform.

Agent Personas

Pre-configured specialist personas give targeted review perspectives for data engineering work:

Use personas mainly for /review, /validate, and /ship when you want one focused lens on the current change.

Reference Guides

Use references/ for high-signal checklists and workflow support material.

How This Toolkit Works

The repository is intentionally layered so agents do not confuse workflow, perspective, and entry point:

Recommended composition:

/spec -> /plan -> /build -> /validate -> /review -> /ship

Parallel review is useful only when perspectives are genuinely independent:

/ship
  +-- analytics-engineer-reviewer
  +-- data-platform-reliability-reviewer
  `-- data-security-and-compliance-auditor

How Skills Work

Each skill lives in its own directory and has a SKILL.md entry point.

skills/
  skill-name/
    SKILL.md
    supporting-file.md

Each SKILL.md should include:

• an overview of what the skill does
• clear signals for when to use it
• a step-by-step workflow
• common rationalizations and rebuttals
• red flags that indicate the skill is being ignored
• a verification checklist with evidence requirements

Key design choices:

• Workflow, not prose. Skills are meant to guide agent behavior, not just describe concepts.
• Verification is required. Validation evidence, reconciliation, or rollout proof should exist before publish.
• Progressive disclosure. Start with one entry skill, then load the minimum useful preset, references, and examples.
• Operational realism. Replay, lineage, access, retention, and rollback are part of delivery, not afterthoughts.

Why Data Engineering Agent Skills?

AI agents often default to the shortest path, which is risky in data systems. That usually means:

• skipping specification and contract definition
• treating a successful run as proof of correctness
• ignoring replay, backfill, and consumer impact
• leaving lineage, access, retention, and ownership implicit

This project is opinionated about avoiding those failure modes:

• A pipeline is not done because it runs once.
• A model is not done because the SQL compiles.
• A table is not production-ready without ownership, contracts, checks, and recovery paths.
• An agent should not guess unclear data requirements when the specification can force clarity up front.

Release Notes

• CHANGELOG.md
• docs/release-process.md
• docs/tag-taxonomy.md
• docs/walkthroughs.md
• docs/ide-support-matrix.md

Roadmap

Phase 1

• publish the core skill pack
• add references and templates
• define contribution rules

Phase 2

• add warehouse, transformation, ingestion, and streaming skills
• add domain presets for common stacks
• add specialist agent personas

Phase 3

• add examples, automation hooks, and validation helpers
• add governance and enterprise preset overlays

Contributing

Contributions should be:

• specific
• verifiable
• grounded in real data engineering practice
• compact enough for agents to follow consistently

See CONTRIBUTING.md, docs/skill-anatomy.md, docs/preset-anatomy.md, skills-index.md, and CHANGELOG.md.

Inspired by addyosmani/agent-skills.