jobd

jobd is a job broker with an MCP server that lets AI agents submit, route and babysit jobs on a few machines of your own. Like task-spooler or pueue, but across all your machines on Tailscale or a LAN, routing each job to a worker with the VRAM free, with no cloud, Kubernetes or Slurm.

<!-- mcp-name: io.github.musharna/jobd -->

<p align="center"> <img src="https://raw.githubusercontent.com/musharna/jobd/main/docs/assets/demo.svg" alt="jobd in action: job fleet status shows four workers and their versions; a GPU job routes to the worker with enough free VRAM and streams back; a stdin batch submits two jobs at once; job logs -f follows one to completion" width="100%"> </p>

# from any machine on your tailnet:
job submit --project myproj --gpu --vram-required 16 --wait -- python train.py
# → routed to whichever worker has ≥16 GB VRAM free, streamed back to your terminal

VRAM routing tracks one GPU per host: the worker reports free memory on GPU index 0 only.

Why it exists

• VRAM-fit routing. Each job goes to a worker whose live free VRAM, RAM, CPUs, capability tags and arch/OS fit it, instead of you guessing which box is free.
• Preemption with a checkpoint window. A higher-priority job can preempt a running one: SIGTERM, a grace window for the workload to save into its JOBD_CHECKPOINT_DIR, then SIGKILL. A preempted job is not re-run; to resume, submit a new job pointed at the checkpoint (docs/preemption.md).
• Survives sessions. Jobs live in the broker, not your shell. Submit, close the laptop, check back tomorrow.
• Agent-native. An MCP server lets an AI agent submit, monitor and babysit jobs as tool calls (benchmark).
• Yours. One broker process on a machine you own. No accounts, no telemetry, no per-GPU-hour billing. Only the optional self-update scripts go online (runbook).

Why not just use…?

Install

pip install jobd               # broker + CLI
pip install "jobd[mcp]"        # adds the MCP server
pip install "jobd[worker]"     # adds the worker daemon (jobd-worker)

Requires Python ≥ 3.11. scripts/install-worker.sh sets up a worker under ~/jobd-worker with its own venv and a generated config.

Quickstart (single host)

# 1. start the broker (binds 127.0.0.1:8765 by default)
JOBD_ALLOW_NO_AUTH=1 jobd          # no-auth is fine for a loopback-only broker

# 2. in another shell, install + start a worker pointed at it
pip install "jobd[worker]"
JOBD_URL=http://127.0.0.1:8765 JOBD_WORKER_HOST=local jobd-worker

# 3. submit a job and wait for it
job submit --project demo --wait -- echo hello
job list
job logs <id>

For a multi-host deployment (Docker broker, systemd workers, Tailscale binding, shared auth token), see docs/security.md and the templates in docker-compose.yml and scripts/. Adding a worker to a running fleet is one command:

job fleet add user@newbox      # ssh in, install pinned to the broker's version,
                               # wire systemd units + the self-update timer,
                               # verify it registers. `job fleet status` shows drift.

Day-2 operations (health, draining a worker, upgrades, token rotation, backups) are in docs/runbook.md.

How it works

<p align="center"> <img src="https://raw.githubusercontent.com/musharna/jobd/main/docs/assets/architecture.svg" alt="Architecture: job CLI, jobd-mcp MCP tools, and HTTP/SSE clients talk to the jobd broker (FastAPI — queue, VRAM matcher, priorities, SQLite) over one tailnet; the broker dispatches via long-poll claims and heartbeats to workers A (24 GB GPU), B (8 GB GPU), and C (CPU-only)" width="640"> </p>

The broker (FastAPI + SQLite) holds the queue, runs the matcher and resolves per-project priorities. Workers, one per host, poll it, so nothing connects in to a worker: each heartbeats its live capacity, claims jobs it can run, streams logs back and honors preemption. The argv you submit runs as-is (shell=False); job submit --stdin and the MCP jobd_submit tool are the exceptions and run a command string as bash -c <string>. Design notes: docs/ARCHITECTURE.md.

CLI

job submit -p PROJ [options] -- CMD...       # queue a job
job list [--state STATE] [--project P]       # queue + recent jobs
job status ID | A<id> [--watch]              # one job, or an array's aggregate
job logs ID [-n BYTES]                       # tail captured output
job wait ID                                  # block until terminal
job cancel ID  /  job preempt ID             # stop a job
job adopt --pid N -p PROJ [--gpu]            # track a process started outside jobd (Linux)
job workers                                  # fleet snapshot + health
job projects list | set NAME PRI | nudge NAME DELTA
job audit [--project P] [--since 24h]        # event history

Job arrays and sweeps, --explain dry runs, job adopt, and a pueue/task-spooler command map: docs/cli.md.

MCP / agent integration

jobd-mcp exposes the queue as nine tools: jobd_submit, jobd_status, jobd_logs, jobd_list, jobd_cancel, jobd_preempt, jobd_events, jobd_workers, jobd_worker_delete. docs/agent-cookbook.md walks through babysitting, preemption, sweeps, and asking why a job won't schedule.

For Claude Code:

claude mcp add jobd \
  --env JOBD_URL=http://127.0.0.1:8765 \
  --env JOBD_API_TOKEN=<your-token> \
  -- jobd-mcp

Any other MCP client:

{
  "mcpServers": {
    "jobd": {
      "command": "jobd-mcp",
      "env": {
        "JOBD_URL": "http://127.0.0.1:8765",
        "JOBD_API_TOKEN": "<your-token>"
      }
    }
  }
}

JOBD_API_TOKEN must match the broker's token, or every call returns 401. examples/claude-code-hooks/ has optional hooks that nudge (or hard-block) an agent toward submitting heavy commands through jobd.

What the benchmark shows

Claude with the server was compared with Claude using jobd's own job CLI, on 38 questions about a live broker, 3 runs each. With Claude Haiku 5.5 the server got all 38 right in every run and the CLI 29: a pre-registered win (Holm-adjusted McNemar p = 0.008). With Sonnet 5.5 it was 36 against 30 and with Opus 5.5 38 against 37, neither separated. On all three models the server's answers were cheaper and faster; with Sonnet, 0.20 times the cost and 0.17 times the time.

<p align="center"> <img src="https://raw.githubusercontent.com/musharna/jobd/main/benchmarks/agent_qa/rerun_2026-10-18/agent_qa.png" alt="Bar chart of right and wrong answers to 38 questions about a live jobd broker, 3 runs per question for each Claude setup. Constant guess with no model: 3 right, 35 wrong. Best constant fit to the answer key: 9 right, 29 wrong. Claude with the job CLI: 113 right, 1 wrong. Claude with jobd-mcp: 114 of 114 right." width="780"> </p>

The chart is the Opus 5.5 run (2026-10-18). There the server's median answer cost 0.20 times as much (95% CI 0.15 to 0.28) and took 0.16 times as long (0.13 to 0.20), and it was cheaper and faster on all 38 questions. These were secondary endpoints, measured on an unreleased commit that adds a finished_at window to jobd_list. On an earlier, easier set both setups got 123 of 123.

We wrote the questions and ran the benchmark ourselves; an outside re-run is welcome. Method, controls and limitations: benchmarks/agent_qa; the Haiku and Sonnet runs: 2026-10-20 re-run.

Supported platforms

CI runs on Ubuntu with no GPU runner (VRAM paths are tested against mocks), and without a systemd user instance the worker still runs jobs but drops its memory caps, process reaping and preemption guarantees.

Configuration

• Config files (projects.yaml, profiles.yaml, classifier.yaml) are all optional and read from JOBD_CONFIG_DIR; see docs/projects-yaml.md.
• Environment variables: every JOBD_* variable is in docs/configuration.md.
• Concurrency (several jobs per worker): docs/concurrency.md. Retention of logs and job records: docs/runbook.md.

Security

Run the broker on loopback or a Tailscale tailnet, never a public interface. Three stacked controls:

1. Bind: JOBD_HOST is 127.0.0.1 by default; set it to a Tailscale address (100.64.0.0/10), never 0.0.0.0. The broker does not enforce this.
2. Source IP: requests from outside loopback and 100.64.0.0/10 get 403 (JOBD_DISABLE_TAILNET_ACL=1 turns this off).
3. Token: every host sets JOBD_API_TOKEN; JOBD_ALLOW_NO_AUTH=1 is for a loopback-only broker, and on any other bind exposes unauthenticated code execution to your tailnet.

/livez, /readyz and /metrics are exempt from both the source-IP check and the token; /metrics shows every worker's hostname and version, so don't port-forward the broker. Threat model and token rotation: docs/security.md.

License

MIT; see LICENSE.