create-scraper— agent skill

Your goal is to explore a company's careers page and write a permanent, reliable Scraper subclass that other code can use going forward. The argument is a careers page URL, e.g. /create-scraper https://explore.jobs.netflix.net/careers.

Step 1 — Identify the company name

Derive the class name and filename from the company's domain or brand name. E.g.:

• explore.jobs.netflix.net → NetflixScraper → src/scraping/NetflixScraper.ts
• jobs.stripe.com → StripeScraper → src/scraping/StripeScraper.ts

Step 2 — Navigate and capture network traffic

The goal is to find a JSON API the page calls internally — these are far more reliable than DOM scraping.

1. Navigate to the URL with browser_navigate
2. Immediately inject a network monitor via browser_evaluate:

window.__reqs = [];
const _f = window.fetch.bind(window);
window.fetch = function(input, init) {
  const url = typeof input === 'string' ? input : input instanceof URL ? input.href : input.url;
  window.__reqs.push({ url, method: (init?.method || 'GET').toUpperCase() });
  return _f(input, init);
};
const _open = XMLHttpRequest.prototype.open;
XMLHttpRequest.prototype.open = function(method, url) {
  window.__reqs.push({ url: String(url), method: method.toUpperCase() });
  return _open.apply(this, arguments);
};

1. Take a browser_snapshot to understand the page structure
2. Interact with the page to trigger data loads: scroll to the bottom, wait a moment, then retrieve what was captured:

window.__reqs

1. Look for requests that look like job listing APIs — JSON endpoints with paths like /api/jobs, /search, /v1/positions, /careers/api, etc. Filter out analytics, fonts, images, and tracking pixels.

Step 3 — Probe the API (if found)

If you found a promising API endpoint:

1. Use browser_evaluate to call it directly and inspect the response shape:

const res = await fetch('ENDPOINT_URL');
const data = await res.json();
JSON.stringify(data, null, 2).slice(0, 3000); // preview

1. Understand the response structure: where are jobs listed? What fields are available (title, location, id, url)?
2. Identify pagination: look for total, offset, limit, page, cursor, or similar fields. Test by modifying query params.
3. Find the job detail endpoint: try appending a job ID to the base URL, or look for a detail URL in the job object.

If the site uses Greenhouse or Lever, stop — those are already handled by GreenhouseScraper and LeverScraper. Report this to the user instead of writing a new scraper.

Step 4 — Explore pagination

Click through to page 2 (or trigger a "Load more"), confirm the data pattern holds, and understand the full pagination mechanism you'll need to implement.

Step 5 — Inspect a job detail page

Navigate to 1–2 individual job listings:

1. Either call the detail API endpoint directly (if found)
2. Or click a job link and browser_snapshot the detail page

Understand what data is available: full description, requirements, salary, etc. This is what getJobContent will return.

Step 6 — Write the Scraper

Now write src/scraping/{Name}Scraper.ts. The approach depends on what you found above.

6a — API-based scraper (preferred)

If you found a reliable JSON API:

import { cachedFetch } from '../fetch.ts';
import { Scraper, type ListedJob } from './Scraper.ts';

export class {Name}Scraper extends Scraper {
  async getJobsList(testing = false): Promise<ListedJob[]> {
    // Fetch all pages via the API and return the full list
    // if `testing` is true, only fetch a single job (if possible) to speed up tests
  }

  async getJobContent(id: string): Promise<string> {
    // Call the job detail API endpoint and return JSON.stringify(data)
  }
}

6b — DOM-based scraper (fallback)

If no usable API exists, scrape the DOM with Playwright. Ask the user to confirm before adding the dependency (yarn add playwright), then write the scraper using the selectors and patterns you identified while exploring.

Use the accessibility snapshot you took in Step 2 to identify the most stable selectors — prefer aria-label, role, and data-* attributes over class names, which tend to change. If a job row has a consistent structure in the snapshot (e.g. a link with a heading inside it), reflect that in your selectors.

import { chromium } from 'playwright';
import { Scraper, type ListedJob } from './Scraper.ts';

export class {Name}Scraper extends Scraper {
  private async withPage<T>(fn: (page: import('playwright').Page) => Promise<T>): Promise<T> {
    const browser = await chromium.launch({ headless: true });
    try {
      return await fn(await browser.newPage());
    } finally {
      await browser.close();
    }
  }

  async getJobsList(testing = false): Promise<ListedJob[]> {
    return this.withPage(async (page) => {
      // Navigate, handle pagination, and extract jobs from the DOM
      // Return ListedJob[] = { title: string; location: string; id: string }
      // Use the job detail page URL (or a stable path segment) as `id`
    });
  }

  async getJobContent(id: string): Promise<string> {
    return this.withPage(async (page) => {
      // Navigate to the job detail page using `id`
      // Extract and return the meaningful text content of the posting
    });
  }
}

Pagination in DOM scrapers: look for a "Next" button or "Load more" and loop until it's gone or disabled. Confirm the selector works by watching the snapshot change between pages during exploration.

Shared conventions

• Prefer `cachedFetch` over raw `fetch` for any HTTP calls — it's a drop-in and caches responses to disk
• id in ListedJob must be stable and usable in getJobContent to retrieve details
• getJobContent should return a rich string — either JSON.stringify(apiResponse) or the full text of the job posting — since it's passed to the Analyzer to assess fit
• Handle pagination fully in getJobsList — return all jobs, not just page 1
• The location field can be a comma-joined string if multiple locations exist
• When you're done, update company-listings.ts accordingly

Step 7 — Verify

Run yarn typecheck to confirm the file compiles cleanly. Fix any type errors before reporting done.

Add the new company to scrapers.test.ts then run TEST_COMPANIES=[COMPANY_SLUG] yarn test ./src/scraping/scrapers.test.ts