/ingest-source

The fat-skill orchestrator for the three-layer ingest pipeline. See docs/proposals/three-layer-ingest.md for the architecture.

Parameters (per the Arquitetura.pdf skill-as-method-call pattern)

The same procedure handles every source. Different parameter → different universe ingested (MCP Registry servers, Printing Press bundles, ClawHub skills, …). One skill, many capabilities. Per the framework: "Same skill. Same seven steps. Same markdown file."

When to invoke

• The user asks to ingest from a specific source (/ingest-source mcp-registry).
• A scheduled cron fires per a source's refresh_cadence field.
• The user adds a new source record and wants to seed it.
• A previous ingest run left records in data/capabilities-staging/<source>/

that need to finish their journey.

The seven steps

Step 1 — Confirm the source is registered

Read data/sources/<SOURCE>.yml. Verify:

• adapter_status: shipped (or spike).
• An adapter is registered: auxiliar ingest list-sources shows ✓.

If the source isn't registered, do not improvise. Stop and ask the user to either build the adapter or correct the slug. The skill is not a code-writer; it is an orchestrator.

Step 2 — Pull thin records into staging

auxiliar ingest pull $SOURCE ${LIMIT:+--limit $LIMIT}

Outputs at data/capabilities-staging/$SOURCE/<slug>.yml. Every record arrives with experimental: true, cost.type: unknown, and only the fields the upstream registry could supply (id, name, description, repository, mcp_registry_id, etc.).

Sanity-check the count against the prior run if there is one — a 10x drop or 10x spike usually means upstream changed shape; surface that to the user before continuing.

Step 3 — Run content-key dedup

auxiliar ingest dedup $SOURCE --report data/capabilities-staging/$SOURCE/_dedup-report.json

The CLI applies the source's dedup_strategy (default skip-on-hand-curated):

• kept — no existing capability matches by content keys. Proceed to enrichment.
• dropped — hand-curated record matches by content keys. Already

discarded by the CLI; no further action.

• conflict — overwrite-with-flag policy hit a match; the CLI left

the staging file in place. Human review required — surface the conflict list to the user and pause.

Step 4 — Enrich each kept record

For every staging YAML that survived dedup:

invoke /enrich-capability with RECORD=data/capabilities-staging/$SOURCE/<slug>.yml

The enrichment skill writes back jtbd_tags, aliases_search, categories, choose_if, avoid_if, and (if it can confirm) updates cost.type from unknown to a known value. Do not run more than 5 concurrent enrichments per source — the upstream README/repo fetches hit rate limits.

If /enrich-capability returns "INSUFFICIENT_DATA" for a record, leave it staged with experimental: true and a # enrichment-failed: 2026-... comment line at the top of the YAML. The next refresh cron will retry.

Step 5 — Curator-fleet smoke test

For each enriched record, invoke the appropriate curator smoke:

• element_type: mcp + has remotes[] → HTTP transport: call the

initialize MCP method, expect a 200 with a valid response.

• element_type: mcp + has packages[] → stdio transport: install via

the declared package manager, spawn, check initialize.

• element_type: skill from pp-library → run the printed skill's

smoke-test against a canonical input where available.

• Anything else → mark smoke-skipped: true in a transient sidecar.

The smoke result feeds the promotion decision. Smoke is mandatory — records that don't pass stay experimental: true and remain in staging. Per the business model: "add to the list only when we can credibly deliver."

Step 6 — Promote survivors

Build the promote allowlist as: enriched and smoke-passed. Flip experimental: false in each YAML, then:

auxiliar ingest promote $SOURCE --only <slug1> --only <slug2> ...

The CLI refuses to overwrite hand-curated records (provenance: hand-curated) — that's the dedup gate's last line of defense.

Step 7 — Report

Summarize for the user:

• Pulled: N
• Dropped (dedup): N
• Enriched: N
• Smoke passed: N
• Promoted: N
• Still staged (experimental=true, awaiting next cycle): N

Plus any conflicts that need human review.

Cron hook

This skill is the natural unit for a scheduled run:

# Pseudocode for /schedule wiring
on cron(weekly):
  /ingest-source mcp-registry
  /ingest-source printing-press

on cron(daily, for high-velocity sources):
  /ingest-source npm-mcp

Cadence is read from data/sources/<SOURCE>.yml → refresh_cadence. Operators wire the cron via /schedule once the policy is validated on the first manual runs.

What this skill does NOT do

• Write code. Adapter implementation is out of scope. If the

adapter is missing for a source, stop and surface the gap.

• Decide dedup policy. That's in the source YAML. The skill applies

what's declared.

• Add new sources. Adding a source is a separate workflow that

requires writing the adapter, the source YAML, and (sometimes) a schema-enum expansion.

• Bypass curator smoke. A record that won't pass smoke stays

experimental. We rank capabilities; we don't fake them.

Reading list

• docs/proposals/three-layer-ingest.md — the architecture
• docs/sources/inventory.md — universe enumeration + per-source verification
• data/sources/ — per-source config records
• src/auxiliar/cli/ingest.py — the CLI surface this skill orchestrates
• memos/Arquitetura.pdf — the thin-harness-fat-skills frame