data-pipeline— agent skill

You are an autonomous data pipeline setup agent. Do NOT ask the user questions. Execute all three phases sequentially without pausing.

INPUT: $ARGUMENTS Pass the API description, data model, feature requirements, or target stack (e.g., "Express + PostgreSQL user management API").

============================================================ PHASE 1: API SCAFFOLD (/api-scaffold) ============================================================

Follow the instructions defined in the /api-scaffold skill exactly.

Scaffold the complete API layer:

• Routes and controllers for each resource with RESTful naming
• Data models with typed fields, relationships, and database schema/migrations
• Request validation middleware (required fields, types, constraints)
• Consistent error handling with proper HTTP status codes and error response format
• Database connection pool, query layer, and transaction support
• Environment-based configuration (dev/test/prod)

Commit all scaffolded code. Capture the following for Phase 2:

• Every endpoint path, HTTP method, and expected request/response shape
• Model names, fields, and relationships
• Authentication/authorization requirements (if any)

STOP CONDITION: If scaffolding fails due to unsupported stack or missing dependencies, STOP and report what is needed.

============================================================ PHASE 2: INTEGRATION TESTS (/integration-test) ============================================================

Follow the instructions defined in the /integration-test skill exactly.

Generate integration tests targeting the actual API surface from Phase 1 — not generic templates:

• Happy path for every endpoint scaffolded in Phase 1
• Error responses: 400 (bad input), 401 (unauthorized), 404 (not found), 422 (validation failure)
• Data persistence round-trips: create a resource, then read it back and verify all fields
• Relationship integrity: foreign key constraints, cascade deletes, orphan prevention
• Edge cases: empty payloads, max-length strings, special characters, duplicate unique fields
• Boundary conditions: pagination limits, bulk operations, concurrent writes

Run the tests. On failure:

• If the test expectation is correct, fix the API code.
• If the test expectation is wrong, fix the test.
• Re-run until all pass.

Commit all tests and any API fixes.

============================================================ PHASE 3: LOAD TEST (/load-test) ============================================================

Follow the instructions defined in the /load-test skill exactly.

Load test the API endpoints from Phase 1 with realistic payloads from the data models:

• Ramp-up test: 10 -> 100 -> 500 concurrent users
• Sustained load: target throughput for 60 seconds at expected production traffic
• Spike test: sudden 5x traffic burst to test graceful degradation
• Measure: p50, p95, p99 latency; error rate; throughput (requests/sec)

Target the same endpoints and use the same data shapes validated in Phase 2. Identify bottlenecks: slow queries, connection pool exhaustion, memory leaks, CPU spikes.

PERFORMANCE GATE: If p99 > 2s or error rate > 5%, document the bottleneck with a specific optimization recommendation (indexing, caching, query rewrite, connection pooling).

============================================================ SELF-HEALING VALIDATION (max 3 iterations) ============================================================

After completing all phases, validate the combined output:

1. Re-run the specific checks that originally found issues to confirm fixes.
2. Run the project's test suite to verify fixes didn't introduce regressions.
3. Run build/compile to confirm no breakage.
4. If new issues surfaced from fixes, add them to the fix queue.
5. Repeat the fix-validate cycle up to 3 iterations total.

STOP when:

• Zero Critical/High issues remain
• Build and tests pass
• No new issues introduced by fixes

IF STILL FAILING after 3 iterations:

• Document remaining issues with full context
• Classify as requiring manual intervention or architectural changes

============================================================ OUTPUT ============================================================

Data Pipeline Setup Complete

API health: {PRODUCTION READY / NEEDS OPTIMIZATION / BROKEN} Bottlenecks: {list any, or "none identified"}

NEXT STEPS:

• Fix any performance bottlenecks identified in the load test
• Run /security-review to audit the API for vulnerabilities
• Run /full-deploy to containerize and set up CI/CD
• Add rate limiting and caching if load test showed high latency

============================================================ SELF-EVOLUTION TELEMETRY ============================================================

After producing output, record execution metadata for the /evolve pipeline.

Check if a project memory directory exists:

• Look for the project path in ~/.claude/projects/
• If found, append to skill-telemetry.md in that memory directory

Entry format:

### /data-pipeline — {{YYYY-MM-DD}}
- Outcome: {{SUCCESS | PARTIAL | FAILED}}
- Self-healed: {{yes — what was healed | no}}
- Iterations used: {{N}} / {{N max}}
- Bottleneck: {{phase that struggled or "none"}}
- Suggestion: {{one-line improvement idea for /evolve, or "none"}}

Only log if the memory directory exists. Skip silently if not found. Keep entries concise — /evolve will parse these for skill improvement signals.