Job Scraper + Gatekeeper Pipeline

Automated job scraping, deduplication, and gatekeeper scoring for ML/AI job hunting.

When to Use

• Cron job that scrapes AI company career pages on Greenhouse, AshbyHQ, and Lever
• Runs on a schedule (e.g. every 15m, 6h, or daily)
• Appends new jobs to the queue file
• Scores pending jobs against profile keywords

Configuration

All PII, paths, and scoring keywords live in profile.yaml at the repo root. Copy profile.yaml.example to profile.yaml and edit. Keys this skill reads:

• paths.queue — queue JSON location (default ~/hermes-merchant/state/queue/jobs.json)
• paths.scraper_log — log file
• scoring.required_keywords, scoring.adjacent_roles, scoring.reject_keywords, scoring.ai_companies, scoring.tiers

Minimal loader (Python):

import yaml, os
from pathlib import Path

cfg = yaml.safe_load(Path("profile.yaml").read_text())
queue_path = Path(os.path.expanduser(cfg["paths"]["queue"]))
queue_path.parent.mkdir(parents=True, exist_ok=True)

Scraping Strategy

Sources — What Works / What Doesn't

ATS Platform Taxonomy

Three ATS platforms dominate AI company career pages. Each wants a different scraping approach:

Known Greenhouse: Anthropic, Together AI, Cerebras. Known AshbyHQ: Cohere, Baseten, LangChain, Modal, Pinecone. Known Lever: Mistral AI.

Greenhouse Job Extraction

Many AI companies host on Greenhouse at https://job-boards.greenhouse.io/{company}.

JavaScript snippet to extract ALL job links from a Greenhouse page (run via browser_console):

"use strict"; (() => {
  const links = document.querySelectorAll('a[href*="/jobs/"]');
  const seen = new Set();
  return Array.from(links)
    .filter(l => l.href && !seen.has(l.href) && seen.add(l.href))
    .map(l => l.href + '\t' + l.textContent.trim())
    .join('\n');
})()

Returns a newline-separated list of URL\tJob Title pairs. Greenhouse pages load all jobs on one page — no pagination.

web_extract is unreliable for job scraping

• web_extract frequently fails with 409 BILLING_ERROR on job-heavy targets (Naukri, Indeed, Anthropic, Cohere, Mistral, Together AI, Groq, LangChain, Baseten).
• web_search is the working fallback — it returns titles, descriptions, and listing URLs from search-engine indexes. Use it to discover individual job-listing URLs.
• Parallel `web_search` calls hit 409 — always run sequentially.
• Fallback chain when `web_extract` fails:

1. web_search first (sequentially) to discover URLs.
2. If web_extract fails (409/504), use search-result snippets as job data — they contain title, company, and location.
3. For career pages, use browser_navigate directly to the ATS URL.

Indeed / Naukri

• Indeed: blocked by Cloudflare. site:indeed.com web_search queries sometimes return snippet data with enough info (title, company, location) to score.
• Naukri: some LLM/GenAI index pages (e.g. naukri.com/llm-engineer-jobs-16) extract successfully via web_extract; search pages don't.

Gatekeeper Scoring

Load thresholds and keyword lists from profile.yaml:

tiers = cfg["scoring"]["tiers"]
required = cfg["scoring"]["required_keywords"]
adjacent = cfg["scoring"]["adjacent_roles"]
reject   = cfg["scoring"]["reject_keywords"]
ai_cos   = set(cfg["scoring"]["ai_companies"])

Tier Thresholds

• Tier 1: score >= tiers.tier_1 (default 75) → tailor resume, apply immediately.
• Tier 2: tiers.tier_2 <= score < tiers.tier_1 → generic or light tailoring.
• Tier 3: below tiers.tier_2 → skip unless specifically interested.

Scoring Logic

def score_job(title, location="", source="", cfg=None):
    text = f"{title} {location}".lower()
    score = 0
    matched = []

    core_roles = [
        "machine learning engineer", "ml engineer", "ai engineer",
        "generative ai engineer", "rag engineer", "llm engineer",
    ]
    for role in core_roles:
        if role in text:
            score += 40
            matched.append(role)

    for kw in cfg["scoring"]["required_keywords"]:
        if kw.lower() in text:
            score += 10
            matched.append(kw)

    for adj in cfg["scoring"]["adjacent_roles"]:
        if adj.lower() in text:
            score += 10

    high_value = [
        "llm", "rag", "generative ai", "genai", "agentic", "nlp",
        "deep learning", "fine-tuning", "rlhf", "multimodal",
        "foundation model", "transformer", "post-training",
        "model serving", "vector db", "embeddings",
    ]
    for hw in high_value:
        if hw in text:
            score += 5

    company_key = source.lower().strip()
    if company_key in {c.lower() for c in cfg["scoring"]["ai_companies"]}:
        has_ai  = any(kw.lower() in text for kw in cfg["scoring"]["required_keywords"])
        has_adj = any(adj.lower() in text for adj in cfg["scoring"]["adjacent_roles"])
        if not has_ai and has_adj:
            score += 15
        elif has_ai or has_adj:
            score += 5

    for rk in cfg["scoring"]["reject_keywords"]:
        if rk.lower() in text:
            score -= 15

    score = max(0, min(100, score))
    t = cfg["scoring"]["tiers"]
    tier = "tier-1" if score >= t["tier_1"] else ("tier-2" if score >= t["tier_2"] else "tier-3")
    return score, tier, matched

AI companies use generic SWE titles

Cohere, Baseten, Modal, and friends use generic titles like "Member of Technical Staff", "Forward Deployed Engineer", or "Agentic Platform". These score low without the AI-company context bonus because the title doesn't contain explicit ML keywords.

Roles like "Forward Deployed Engineer, Agentic Platform" (Cohere) score 0 by title alone. Include agentic, post-training, inference, forward deployed, prompt specialist, applied researcher, research engineer, ml researcher, ai researcher, reinforcement learning, rlhf, and agent in either adjacent_roles or required_keywords to catch them.

Reject keyword false positives

"Research Engineer" contains "engineer" but is NOT a reject. Reject logic should only trigger on exact matches like "java developer" or "react developer" — not any string containing "developer".

Job Data Shape

{
  "url": "https://jobs.ashbyhq.com/cohere/...",
  "source": "cohere",
  "title": "Forward Deployed Engineer, Agentic Platform",
  "company": "Cohere",
  "location": "Toronto; New York",
  "experience": "Not specified",
  "score": 35,
  "tier": "tier-2",
  "matched_keywords": [],
  "scraped_at": "2026-04-19T17:35:45+00:00",
  "status": "pending"
}

Status Values

• pending — in queue, not yet applied.
• applied — successfully applied via web form.
• applied_email — applied via email.
• failed — application attempt failed; include failure_reason and manual_apply_url.
• blocked — known blocker (visa, experience mismatch, etc.).

Deduplication

Before appending, compute existing_urls = {j["url"] for j in queue} and skip URLs already present.

Known Gotchas

• f-string + dict access: f"score={j['score']}" with nested dict access confuses older Python parsers. Use string concatenation or .format().
• Tuple unpacking: if score_job returns (score, tier, matched), unpack all three at every call site.
• Parallel web calls: web_search / web_extract in parallel reliably hits 409 BILLING_ERROR. Run sequentially.
• LangChain Ashby: the jobs board is iframed on the careers page. Go direct to https://jobs.ashbyhq.com/langchain — don't try to scrape the embed.
• Mistral Lever: times out in browser without residential proxies on some networks. Mark blocked and move on.