Web Scraping (Firecrawl-first)

Goal

Extract LLM-ready web content without hacking around: clean markdown, structured JSON, anti-bot and JS-rendering handled. Firecrawl is the reference wrapper; fallback to Playwright or curl + html2text if unavailable.

When to trigger this skill

• "scrape this page / this site"
• "extract data from ..."
• "crawl site X"
• "fetch all articles from ..."
• "search the web and extract the content"
• "parse this dynamic page" (site with JS-rendering)
• "bypass the paywall / anti-bot" (legitimate use only)

When NOT to use this skill

• Quick web search without structured extraction -> WebSearch is enough
• A single static URL, simple page -> WebFetch is enough
• Visual test / browser interaction -> agent-browser, or ask the user to run /qa:qa-chrome (the qa-chrome skill is manual-only)
• Form / login automation -> agent-browser or Playwright directly

Prerequisites

Option 1: Firecrawl cloud (recommended)

export FIRECRAWL_API_KEY="fc-xxx"      # https://firecrawl.dev
npm install -g firecrawl-cli           # the CLI (`firecrawl`); the SDKs are firecrawl / firecrawl-py

Option 2: Firecrawl self-hosted

Docker compose available on github.com/mendableai/firecrawl. Useful if data is sensitive or budget is limited.

Option 3: Fallback without Firecrawl

If Firecrawl is missing, degrade gracefully:

IMPORTANT: always announce when degrading. The user must know if the content is partial (JS not rendered).

The 5 Firecrawl operations (CLI firecrawl-cli v1.25)

1. Scrape (one URL)

firecrawl scrape https://example.com/article \
  --format markdown,links \
  --only-main-content

Output: clean markdown (navigation / footers stripped), list of links, OG metadata.

2. Crawl (whole site)

firecrawl crawl https://docs.example.com \
  --limit 100 \
  --include-paths /docs \
  --exclude-paths /docs/legacy \
  --scrape-options '{"formats":["markdown"]}' \
  --wait -o crawl.json

Output: one JSON file holding every page (markdown in each entry). Ask for confirmation before crawl > 50 pages (API costs + time).

3. Map (URL discovery)

firecrawl map https://example.com --search "pricing"

Output: list of relevant URLs. Useful BEFORE a crawl to target the right sections.

4. Agent (structured data via LLM)

firecrawl agent "Extract the plans with name, price and features" \
  --urls https://example.com/pricing \
  --schema-file plans.schema.json \
  --max-credits 50 --wait

Output: JSON conforming to the schema. Saves hours of fragile CSS selectors. An agent job takes minutes and spends credits: always set --max-credits. (The CLI's former extract command is gone; agent replaced it.)

5. Search (search + extract in one pass)

firecrawl search "best pve proxmox backup strategies" \
  --limit 10 \
  --scrape --scrape-formats markdown

Output: top N results with extracted content. Replaces WebSearch + N WebFetch.

Recommended workflow

1. IDENTIFY the need
   - 1 page               -> scrape
   - N known pages        -> scrape in a loop with `xargs -P 4`
   - Whole site           -> map (recon) -> targeted crawl
   - Structured data      -> agent with a schema
   - Search + extract     -> search

2. ESTIMATE costs
   - Firecrawl cloud: credits per page scraped
   - Ask for confirmation if > 50 pages or > 10 MB expected

3. RUN with limits on the first attempt
   - --limit 5 to test
   - Inspect the output
   - Re-run at full volume if OK

4. SAVE the result
   - `./scraped/<date>/<domain>.md` by convention
   - Commit if data is reusable (mind copyright)

5. CHECK legality / ethics
   - Respect robots.txt unless explicitly authorized
   - No personal data without consent (GDPR)
   - No commercial paywall bypass

Concrete examples

Extract a lib's docs for RAG

firecrawl crawl https://docs.terraform.io/language \
  --limit 200 --scrape-options '{"formats":["markdown"]}' \
  --wait -o rag-corpus/terraform.json

Compare pricings of 5 competitors

firecrawl agent "Extract the pricing plans of each site" \
  --urls url1,url2,url3,url4,url5 \
  --schema-file pricing.schema.json \
  --max-credits 100 --wait -o pricing-compared.json

Monitor a changelog

firecrawl scrape https://example.com/changelog --format markdown -o new-changelog.md
if ! diff -q new-changelog.md last-changelog.md >/dev/null 2>&1; then
  diff last-changelog.md new-changelog.md   # what changed
  mv new-changelog.md last-changelog.md
fi

Red Flags — STOP immediately

Integration with the rest of the foundation

Anti-patterns

• NEVER scrape without checking robots.txt AND Terms of Service
• NEVER commit scraped data without checking the rights
• NEVER launch a crawl > 50 pages without user confirmation
• NEVER use Firecrawl to replace WebSearch for a simple factual question (needlessly expensive)
• NEVER bruteforce a site in massive parallel (max 4 workers by default)

Absolute rules

IMPORTANT: Always announce when degrading to a fallback (Playwright / curl) — the content may be partial.

IMPORTANT: Ask for confirmation before any crawl exceeding 50 pages or a site outside the user's control.

YOU MUST respect robots.txt and the target site's ToS.

YOU MUST save outputs in ./scraped/<date>/ with timestamp for traceability.

NEVER bypass an anti-bot system without documented legitimate justification.

See also

Firecrawl publishes its own skills in firecrawl/cli (13 skills, CLI v1.25.3). Not recommended next to this skill (checked 2026-10-04):

• firecrawl-search claims "web research" and firecrawl-scrape "read a known webpage" — the same claim as the firecrawl router: installed, they route ordinary lookups to the paid API.
• firecrawl-search sends search feedback after every search, including text derived from the query (opt-out: FIRECRAWL_NO_SEARCH_FEEDBACK=1, separate from FIRECRAWL_NO_TELEMETRY=1).
• They contradict this skill's own rules: output to .firecrawl/ instead of ./scraped/<date>/, agent examples without --max-credits, pricing "not an extra confirmation gate".

This skill's decision tree, free fallbacks, cost gates and legal rules only hold if it is the one that runs.