Jobseek Error Review

Use this skill for the daily crawler error review. Operate from the repo state and these instructions; do not rely on prior conversation context.

Run recurring reviews through the Hetzner Codex runner. An operator may invoke the same skill from the repository root for a bounded manual recovery. The Hetzner runner uses an isolated worktree and a root-collected redacted evidence bundle, which keeps routine output isolated from active development work while avoiding direct Docker or deploy-env access from the Codex process.

The legacy Claude slash command at .claude/commands/jobseek-error-review.md remains a compatibility fallback. Keep behavior equivalent when using that fallback, but treat this skill as the Codex-first runbook.

Do not spawn subagents by default. They cost extra tokens and are only useful when the current run has large independent evidence sets and the user explicitly asks for them. If subagents are used, keep them read-only and make the main agent responsible for dedupe, classification, redaction, and all GitHub write decisions. Use the project custom agent jobseek-error-review-researcher (GPT-6.1 Sol, high reasoning) for each bounded evidence set.

Mission

Scan the last 24 hours of errors on the crawler Hetzner box, classify each error class against prior reviews and prior GitHub issues, write a dated Markdown report, and file or update GitHub issues only for classes that meet the filing criteria.

Model issue quality after prior examples #2622, #2621, #2470, and #2431. Never mutate remote host state. GitHub writes are allowed only when the class is novel, regressing, spiking, or an incident after dedupe.

Target

Use the root-collected runner bundle by default. A local manual fallback may resolve the production crawler host from approved operator configuration, but must not copy host addresses into reports, traces, prompts, or GitHub content. The Compose file reference is /home/deploy/docker-compose.yml.

Long-running services. Collect logs with explicit --since and --until:

• deploy-worker-1-1 - HTTP worker
• deploy-worker-2-1 - HTTP worker
• deploy-worker-3-1 - HTTP worker
• deploy-browser-1-1 - Playwright/Lightpanda worker
• deploy-exporter-1 - Postgres to Supabase and Typesense CDC
• deploy-drain-1 - R2 description uploader
• deploy-redis-1 - local queue; only OOM and level=error lines
• deploy-alloy-1 - log/metric shipper; only Alloy's own level=error

Ephemeral one-shots are exited containers created inside the review window whose image matches ghcr.io/*/jobseek-crawler:*. Enumerate them with:

docker ps -a --filter status=exited --format '{{.Names}} {{.Image}} {{.CreatedAt}} {{.Status}}'

These include crawler refresh-typesense, backfill-typesense, sync, notify-indexnow, and reconcile runs. Grab logs with docker logs <id> before exited container logs age out. Ignore interactive debug containers such as tesla-debug, stupefied_hofstadter, and goofy_haibt.

Inputs

1. Use a window of the last 24 hours ending at UTC now, rounded down to the

minute. Use explicit --since and --until values so log collection exactly matches the report header. When the Hetzner runner provides a redacted evidence bundle, read manifest.json, host/docker-inspect-state.txt, and host/docker-cgroup-memory.json before classifying host incidents. Also read host/docker-lifecycle.jsonl, the durable allowlisted stream of create/start/restart/die/oom/kill/stop/destroy events. These files record container identity, exit code or signal, event time, restart count, and cgroup-v2 memory.events counters without exposing arbitrary Docker labels. Read host/maintenance-correlation.json before classifying any synchronized service stop. It contains deterministic, command-free correlation of validated maintenance operation/issue/revision/budget labels. Read host/cross-store-reconciliation.log when a reconciliation one-shot exits nonzero; it retains the unit's bounded, redacted application journal after Compose removes the short-lived container. Always read host/redis-capacity.prom and host/redis-capacity-observer.log. The first distinguishes Redis maxmemory pressure from its larger container cgroup; the second exposes exact-window capacity scan failures and cooldown behavior. Treat manifest.json redis_capacity.complete=false as an explicit evidence gap, not as a healthy capacity signal. Always read the disk_capacity manifest entry and all six bounded disk artifacts: host/docker-system-df.txt, host/docker-container-sizes.txt, host/docker-images.txt, host/disk-attribution.txt, and host/docker-gc-timer.txt and host/docker-gc.log. Treat disk_capacity.complete=false as an explicit attribution gap. When / or Docker is at least 85% full, do not close or downgrade the incident from a percentage alone: identify the dominant allowlisted root/image family and check the exact-window GC journal, or state precisely which incomplete artifact prevents attribution. Treat ActiveState=active with SubState=running as a healthy in-flight timer and inspect its target service separately. A quiescent timer must be active (waiting) with a finite next trigger; treat any other state, or a waiting timer with a missing/infinite next trigger, as a failed recurrence guard even when disk usage is currently below the incident threshold.

1. Read every .md report under

~/dev/claude/review-jobseek-errors/ before classifying. The directory name is legacy; keep using it for cross-run continuity unless a migration has already been completed.

1. Load prior GitHub issues for dedupe and pattern matching:

   gh issue list --label daily-error-review --state all --limit 200 \
     --json number,title,state,createdAt,body

1. Load recently merged PRs for regression correlation:

   gh pr list --state merged --limit 40 --json number,title,mergedAt

Host Signals

Collect host signals before detailed log review. These commands are allowed without sudo as deploy:

df -h /
df -h /var/lib/docker
free -h
uptime
docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}'
docker inspect --format '{{.Name}} Image={{.Config.Image}} Created={{.Created}} StartedAt={{.State.StartedAt}} OOMKilled={{.State.OOMKilled}} Status={{.State.Status}} RestartCount={{.RestartCount}} FinishedAt={{.State.FinishedAt}} ExitCode={{.State.ExitCode}} Error={{json .State.Error}}' $(docker ps -aq)
dmesg -T 2>/dev/null | tail -n 500
journalctl -k --since "24 hours ago" --no-pager 2>/dev/null | tail -500

Use journalctl -k as a fallback if dmesg is restricted. If both kernel log commands are restricted, note the gap under ## Host and do not treat that gap alone as an incident.

Docker's OOMKilled state is historical and can remain true while a container keeps running. RestartCount is also scoped to one container object. Never subtract either value across different pseudonymous container generations. Treat an OOM or restart as new inside the report window only when one of these proves it:

• The same pseudonymous container_generation exists in both evidence

snapshots and its cgroup-v2 memory.events.oom_kill or Docker RestartCount increased.

• A container created inside the window already has a positive counter.
• An exited container finished inside the window with OOMKilled=true, or

kernel logs timestamp the OOM inside the window.

• The durable lifecycle stream timestamps the same container's die, oom,

kill, or subsequent start event inside the window.

Use lifecycle sequences to classify the mechanism, not to guess a caller:

• oom or inspected oom_killed=true followed by die is a cgroup/kernel OOM.
• kill records a Docker API signal; report the signal and say the caller is

unknown unless deployment or host evidence identifies it.

• die without a preceding kill/oom is a native process exit; use its

exit code and adjacent application logs.

• stop/die/destroy followed by a new container generation's create/start

is a replacement or deployment, while die/start on the same generation with an incremented restart count is an automatic restart.

Before treating a Docker API stop or replacement as spontaneous instability, apply the maintenance correlation contract:

• Attribute it only when host/maintenance-correlation.json contains exactly

one validated overlapping operation. Never infer authorization from names, timestamps alone, or an unlabelled one-off.

• Report an attributed window under ## Maintenance windows with operation,

tracking issue, reviewed revision, duration, affected services, measured downtime, one-off results, and restoration health.

• Do not file/reopen a spontaneous-instability issue for a clean authorized

window. Forced termination, OOM/native exit, nonzero one-off, budget overrun, and failed restoration remain actionable; update the maintenance tracking issue or file a deduplicated maintenance-outcome issue.

• Missing, partial, invalid, or conflicting provenance remains unknown.

OOM/native exits remain incidents regardless of any adjacent maintenance.

If the lifecycle journal has connect_error/stream_closed records or does not cover the full window, report the evidence gap explicitly. Do not infer a clean window from a missing or unavailable journal.

A changed pseudonymous container generation normally means a deployment or replacement, not a restart-count delta. A sticky OOMKilled=true without a same-generation counter delta is historical context, not a fresh daily incident.

Flag an incident if any of these are true:

• / usage is at least 85%.
• /var/lib/docker usage is at least 85% when it is a separate mount.
• A generation-safe cgroup or exited-container signal proves an OOM inside

the window.

• A same-container RestartCount increment proves a restart inside the

window.

• An unattributed lifecycle die/start sequence proves a restart inside the

window even when yesterday's snapshot is unavailable. An attributed sequence follows the maintenance-outcome rules above.

• 15-minute load average is greater than CPU count x 2.
• Swap usage is greater than 50% of swap total.
• Kernel logs show Out of memory: Killed process, I/O error,

EXT4-fs error, Call Trace:, or systemd OOM-killer entries.

• Any expected long-running service is missing from docker ps.

Log Collection

For each long-running service:

docker logs --since "<ISO>" --until "<ISO>" <container> 2>&1

Parse structlog JSON where possible. Fall back to line matching only for non-JSON lines. Extract level, event, service_name, exception class, and a stable stack or message fingerprint.

For exited ephemeral containers inside the window:

docker logs <id> 2>&1 | tail -n 1000

Attribute those errors to a synthetic service derived from the command, such as refresh-typesense or notify-indexnow.

Classification

Group errors by (service, exception class, stable message stem).

For a proxy-enabled board where one origin returns a typed bot/origin block on every configured slot, do not repeatedly probe the target and do not attempt to solve or bypass a CAPTCHA. Confirm the failure once with the production worker's real headful entrypoint, wait for the full origin quarantine cooldown, and confirm once more. Then use the sanitized proxy-audit capacity fields and the guarded proxy-replace-webshare-pool dry-run/apply workflow documented in apps/crawler/AGENTS.md. Replacement is an explicit operator mutation and must fit within included plan capacity; it is never automatic.

Update only the production GitHub environment's runtime proxy secret; keep no repository-level shadow. Pipe the JSON into gh secret set WEBSHARE_PROXY_URLS --env production without --body (--body - stores a literal dash). Start a new manual Deploy Crawler (Hetzner) dispatch on main; reruns retain the original run's secret snapshot. The guarded path attests the current exact default-branch SHA, validates the settings inside the built image before SSH, and derives its rollback parent. Verify only the deployed pool cardinality before the bounded origin canary; never print the endpoints.

Classify Njoyn inventory reconciliation independently from proxy transport. njoyn.snapshot.reconciled is a successful two-pass result inside the code-owned churn budget and is not an error class. first_page_state_changed, pagination_state_mismatch, empty_page_shape_mismatch, page_count_mismatch, page_row_count_mismatch, page_transition_did_not_converge, pass_*_exceeded, and reconciliation_*_exceeded are inventory_unstable failures and must stay fail-closed. Do not prescribe proxy replacement for those failures unless the same evidence window separately proves a typed all-slot origin block.

Treat batch.scrape.required_field_missing as extraction-contract evidence, not transport evidence. Group it by board and scraper. For Njoyn, inspect one bounded rendered sample for a valid schema.org JobPosting; when present, classify the cluster as scraper/config drift and validate rendered JSON-LD. Never prescribe proxy replacement without separate typed all-slot origin-block evidence.

jsonld.render.origin_block_retry is typed transport evidence from the in-context rendered-response guard; for Njoyn it includes the bounded HTTP-200 Invalid request XWP... shell. Treat jsonld.render.direct_fallback as successful recovery, not extraction drift. It is valid only with explicit board opt-in after a typed block; proxy pool unavailability alone must stay fail-closed. Group without response bodies or opaque challenge URLs.

• known: appears in any prior daily report within 14 days. Count it but do

not file.

• novel: absent from every prior report.
• regression: a known class that was absent for at least 3 consecutive days

and is back today.

• spike: a known class whose 24-hour count is at least 3x its 7-day median.

Require at least 3 prior days with non-zero signal.

• incident: host-signal trigger, unexpected zero logs from a long-running

service for more than 1 hour, data loss or corruption signal, repeated OOM kills, or exporter/drain stall with queue depth climbing and no progress.

Incomplete evidence narrows confidence but does not make GitHub filing impossible. If a bundle covers less than the full 24-hour window, classify full-window trends as unclassified — partial window, but still file or update an issue when the available evidence shows a concrete, redacted, deduped error class that would independently meet a filing criterion inside the observed window. In that case, state the observed window and evidence gap in the report and issue body, avoid unsupported 24-hour trend claims, and use severity based only on the verified blast radius.

Report

Always write a report, even on a healthy day.

Path:

~/dev/claude/review-jobseek-errors/YYYY-MM-DD.md

If today's file already exists, append a ## Rerun HH:MM UTC section instead of overwriting it.

Report schema:

# Daily error review - YYYY-MM-DD
Window: YYYY-MM-DD HH:MM UTC -> YYYY-MM-DD HH:MM UTC

## Host
| metric | value |
|---|---|

## Totals
| service | info | warning | error |
|---|---:|---:|---:|

## Error classes (24h)
| class | service | count | status |
|---|---|---:|---|

## Maintenance windows
| operation / issue | revision | duration | affected services | outcome |
|---|---|---:|---|---|

## Details

## Filed issues

## Health

Under ## Details, include every novel, regression, spike, or incident class:

• Error class and service.
• 24-hour count plus first-seen and last-seen UTC.
• One redacted sample log line.
• Brief root-cause hypothesis with 2 or 3 candidates. Use file paths only.
• Recent PR cross-references when correlation is plausible.

Under ## Maintenance windows, include every deterministic correlation row, including clean authorized operations. Report forced termination, OOM/native exit, failed/nonzero one-offs, budget overruns, measured service downtime, and failed restoration explicitly. Use none when no maintenance window is present.

Under ## Filed issues, list issue URLs filed or updated in this run, or write none.

GitHub Issues

File or update issues for novel, regression, spike, or incident classes. Partial-window evidence is allowed when the observed error class is concrete and issue-worthy on its own; caveat the reduced window in the signal section rather than suppressing the issue categorically.

Deduplicate first:

gh issue list --label daily-error-review --state all \
  --search "<error class stem or keyword>"

Match by error class and service, not exact title.

• If a matching open issue exists, add a comment with today's window, count,

first-seen, and last-seen. Do not open a duplicate.

• If a matching closed issue from the last 30 days exists, reopen it with a

comment explaining the recurrence.

• Otherwise open a new issue.

New issue template:

## Summary

One paragraph with service, what broke, whether it is visible or silent, the
explicit 24-hour window, and suspected PR reference if any.

## Signal

5-day trend table (-4, -3, -2, -1, today) of count or batched log-line count.
Explain the unit.

## Sample log line

{"level":"error","event":"redacted sample"}

## Root cause hypothesis

1. Candidate with file-path reference only and fix direction.
2. Candidate with file-path reference only and fix direction.

## Impact

User-facing consequence such as stale data, disabled boards, or silent
failure. If invisible beyond `level=error`, say so.

Title shape:

[daily-error-review] <service>.<error_stem> (<trend>; <blast>)

For critical incidents, prefix the title with [critical].

Labels are required:

• daily-error-review
• Exactly one of error-review:critical, error-review:high,

error-review:medium, or error-review:low

Severity mapping:

• error-review:critical: active incident or data-loss risk.
• error-review:high: regression or full-service blast radius.
• error-review:medium: spike or partial-service regression.
• error-review:low: novel but contained.

Redaction

The repository is public. Redact before any GitHub write:

• Full URLs with query strings: keep host and path shape only.
• IPs, port numbers, and hostnames beyond jseek.co or public CDNs.
• UUIDs, emails, user IDs, and company internal IDs.
• Raw Docker/container IDs and cloud resource IDs. Use the bundle's

pseudonymous container_generation only in the private report when needed.

• JWTs, cookies, API keys, and Authorization headers.
• SQL literal values; keep statement skeleton only.
• Request or response bodies beyond the exception message itself.
• Absolute paths containing /home or /root.
• Stack traces with parameter values. Keep exception class and innermost frame

as filename:function only.

Local reports under ~/dev/claude/review-jobseek-errors/ are private. Raw details belong there, not in GitHub issues.

Guardrails

Allowed on the remote box:

• docker logs
• docker ps
• docker inspect
• docker stats --no-stream
• df
• free
• uptime
• dmesg
• journalctl -k when accessible
• Reading files under /home/deploy/
• Reading and writing temporary files under /tmp/ that you created this run

Forbidden on the remote box:

• Restarts, stops, docker exec, docker kill, and any docker compose

subcommand.

• systemctl.
• File edits outside /tmp temporary files you created this run.
• Deletions outside /tmp temporary files you created this run.
• Any sudo.
• Any outbound HTTP to third-party hosts.

Writing scripts to /tmp on the remote box is allowed only when the script is idempotent and read-only. Remove scripts after use.

If a command's effect is not completely known to be read-only, stop, write the intent to today's report, classify the gap as an incident, and do not run it.

Escalation

For any incident:

1. File or update an issue labelled daily-error-review and

error-review:critical.

1. Put [critical] at the start of a new critical issue title.
2. Append a line to

~/dev/claude/review-jobseek-errors/ALERTS.md:

   YYYY-MM-DD HH:MM UTC  <one-sentence>  <issue URL>

1. Do not attempt to fix production during this routine.

Verification Rollout

Use this rollout when changing the routine or moving it to a new automation surface:

1. Read-only dry run: collect host signals, logs, prior reports, and GitHub

issue state; write the report; list would-file or would-update issues without creating, reopening, or commenting on GitHub issues.

1. Production pilot: run against the real host and real GitHub issue state,

but file only when the criteria are unambiguous and the evidence is already redacted.

1. Require two clean runs before normal filing behavior. A clean run means the

report is written, known issues are deduped, no forbidden command is used, and any GitHub write is justified by the classification rules.

1. After edits, scan docs and agent instructions for stale primary-runtime

wording such as Claude-only phrasing, direct API billing, or obsolete command names.