ML research workflow

projtool runs ML experiments across a local development machine and remote AutoDL containers. Three asset categories — code, outputs, reports — have different version-control needs and different storage backends. Your job is to keep them coherent without making the user do bookkeeping.

The model in two paragraphs

projtool v4 uses a container root. Code lives in code/<worktree>/ inside that container, with the main git worktree at code/main/ and experiment worktrees at code/<experiment_id>/. Container-root data/ and output/ are local runtime directories; each worktree gets a data symlink so user code can keep reading ./data. Outputs are indexed by run_id and never committed.

Reports — markdown analyses of one or more runs — live in an independent docs repo at container-root docs/, not inside any code worktree. Inside the docs repo, reports are organized by code-repo branch namespace — exp/lora reports live at docs/exp/lora/<date>-<slug>.md, cross-branch summaries at docs/summary/<date>-<slug>.md. Reports cite run_ids and inline headline metrics so they remain readable even if outputs are deleted later, plus they record git.code_repo_url and git.commit_url for cross-repo provenance.

Mutagen handles cross-machine syncing of code (forward, one-way-replica, per-worktree) and outputs (reverse, one-way-safe, single global session). The docs repo does NOT participate in mutagen sync — it is local-only plus its own GitHub origin. AutoDL instances are managed via the algorithm in src/projtool/autodl/ — you only need the MCP tools.

Tool inventory

Always read the project's .proj-tool/project.toml before invoking any tool that needs a path; never hardcode paths in your tool calls. v4 path fields are dir names (code_dir, data_dir, docs_dir, output_dir) plus remote paths (outputs_remote, data_remote, remote_workspace). Removed v3 fields such as docs_repo_path and outputs_local should not be used for new projects.

Hot paths

Starting a new experiment

1. new_experiment with a topic slug → creates exp/<topic> branch,

code/<topic> worktree, and the docs repo's `<branch_full>/` placeholder directory + commit. If a bound AutoDL instance already exists for this experiment, the mutagen code-sync session is created immediately; otherwise it is deferred to start_training.

1. The user (or you) edits in the new worktree.
2. Commit. (start_training's pre-hook will refuse a dirty tree.)
3. instance_up for the experiment.
4. start_training with the chosen config.

Running multiple instances on one code branch

Use slot when the same experiment branch needs multiple GPU instances. Do not create empty branches only to borrow more machines.

Example: one exp/distill branch with three distillation configs:

instance_up(experiment_id="distill", slot="w2")
start_training(experiment_id="distill", slot="w2", ...)
instance_down(experiment_id="distill", slot="w2", mode="release")

Identity rules:

• experiment_id is normalized to canonical exp-<slug>.
• The default slot is main; its storage key is just exp-<slug>.
• Non-default slots use exp-<slug>#<slot>.
• Pass the same (experiment_id, slot) pair through instance_up,

start_training, get_run_status, finalize_run, and instance_down.

Polling a run

get_run_status returns:

• manifest snapshot (status, started/finished, latest metric snapshot)
• tmux session liveness
• last log line tail (capped)

Don't poll faster than every ~30s; it costs an SSH round-trip.

Finishing a run

start_training launches the tmux command and writes the initial manifest, but it does not inject an automatic finalize hook. A training wrapper/script may call finalize_run(run_status="completed") on clean exit; otherwise you must call finalize_run manually after inspecting get_run_status, logs, and metrics. If get_run_status reports run_status="orphaned", tmux is gone while the manifest still says running; choose completed or failed based on the outputs. See references/dirty-worktree-handling.md for edge cases.

Writing a report

1. Decide scope:

• Single branch (default) — the report analyzes runs from one

exp/<topic> branch. The report file goes to <container>/docs/<branch_full>/<date>-<slug>.md.

• Cross-branch summary — the report compares runs from multiple

branches (e.g., lora vs qformer), or is a periodic milestone spanning the project. Goes to <container>/docs/summary/<date>-<slug>.md.

1. Call start_report(experiment_id, run_ids, report_type, scope). The tool

loads each run's manifest, picks the docs-repo target directory (mkdir -p so nested branch names like exp/lora create both segments), prefills references and metrics_inlined skeletons, and fills the cross-repo fields (git.branch, git.commit, git.code_repo_url resolved from the code repo's git remote, git.commit_url constructed).

1. Edit the body. Keep metrics_inlined consistent with cited numbers.

1. Call commit_report(report_path, slug=..., commit_message=...). The tool:

• Validates each cited code commit is already pushed to the code repo's

origin (cross-repo invariant). On failure, raises CrossRepoReferenceError with the explicit instruction to cd <code_repo> && git push origin <branch> first.

• In the docs repo (NOT the code repo!): git add the specific

.md and (if externalized) CSV paths, then commits and pushes to the docs repo's current branch (resolved via git symbolic-ref --short HEAD — never hardcoded as main).

See references/report-frontmatter.md for the schema and references/manifest-schema.md for the manifest fields you can cite.

Cross-worktree access: from any code-repo worktree, you can ls <container>/docs/exp/lora/ to read reports written from another branch — the docs repo is outside all worktrees, so the git-worktree isolation barrier doesn't apply. v2's list_my_reports.sh helper is gone.

Image build

When the user asks to build a project image:

1. image_build_start — opens a dedicated builder instance and pushes

setup.sh (you draft it from the user's requirements + base image). For AutoDL PyTorch base images, put export PATH="/root/miniconda3/bin:$PATH" immediately after set -euo pipefail; non-interactive SSH shells may not have conda's Python on PATH.

1. remote_exec "bash setup.sh" — blocks until done; exit_code is in

the result (non-zero means setup.sh failed, NOT a tool error). Full output is in log_path; stderr_tail has the inline error context.

1. On exit code 0: image_save (non-blocking — returns pending_image_uuid

immediately) → poll image_save_status until image_status='ready' → image_build_finalize. The tool writes .proj-tool/project.toml's [image].project_image_uuid (and related fields) on finalize. Future instance_up calls use that project image by default.

1. On non-zero exit: load the `image-debug` skill and follow its

diagnostic procedure. Don't free-style; common errors have known patterns in image-debug/references/.

Quick diagnostics / step-0 sanity check

When you need to run a short script on the remote — sanity-check that the warmstart loads, read step-0 metrics, eval an existing checkpoint, etc. — use remote_exec, not start_training. Three reasons:

1. start_training registers a run in state.json's run_index, writes a

manifest stub, and expects the run to converge to completed / failed / killed via finalize_run. A 30-second diagnostic pollutes that audit trail.

1. The training-time hooks (check-ssh-train.sh) refuse a dirty worktree

and require the commit to be pushed; you usually don't want either constraint for a one-shot diagnostic.

1. Claude Code's auto-mode classifier sometimes infers "user said no full

experiment" and blocks start_training even when you only meant a one-step run; remote_exec is unaffected.

Pattern:

remote_exec(ssh_alias=<exp_alias>, command="cd /workspace && python diag.py")

The script should live at a path outside <remote_workspace> (e.g. /tmp/diag.py, pushed with write_remote_file, or composed inline via bash -c '...'). Anything inside <remote_workspace> that doesn't exist on alpha will be deleted by the next mutagen flush — see the <remote_workspace> invariant below.

Diagnosing sync trouble

When the user says "sync seems stuck" / "mutagen 卡住":

1. diagnose — runs all health checks; returns a structured report.
2. Read the report sections; for each red flag, consult

references/mutagen-troubleshooting.md.

1. Apply fixes via the named MCP tools (sync_resume, refresh_ssh).
2. If diagnose reports the AutoDL instance has gone away while a binding

still exists → the binding is stale. instance_up will repair it (creates a fresh instance under the same experiment_id).

Note: the docs repo is not part of mutagen. If the docs repo seems out of sync between local and GitHub, that's a plain git pull/git push issue in <container>/docs/, not a projtool diagnostic concern.

Reconciling binding identity after upgrade

When an existing project was used before binding normalization, state may contain split keys such as distill-w2 and exp-distill-w2. The automatic state migration adds missing slot fields but intentionally does not rename or merge live bindings.

Use this sequence after upgrading projtool if diagnose reports split, duplicate, or legacy bindings:

1. diagnose — identify suspicious keys and stale bindings.
2. reconcile_bindings(dry_run=True) — ask projtool for proposed merges.
3. Review the proposals. Use the dry-run output as the source of truth for

decisions; if both sides look live, choose the winner explicitly.

1. reconcile_bindings(dry_run=False, decisions=[...]) — apply the reviewed

mapping.

1. diagnose again — confirm the split keys and stale aliases are gone.

Do not edit state.json directly unless projtool itself is unusable and the user explicitly accepts the risk.

Upgrading after pip install -U projtool

User: "projtool 升级了" / "check if templates have updates".

1. retrofit_check_upgrade — returns per-file diffs.
2. Walk the user through each diff: keep theirs / take mine / merge by hand.
3. Don't silently overwrite user-modified files. The marker block

(>>> projtool >>> … <<< projtool <<<) is the safe scope; outside it is user-owned.

1. v3-specific (upgrading a project from template_version 0.3.0 to 0.4.0):

the upgrade is a major bump.

• [paths] schema changed: reports field removed, docs_repo_path and

docs_repo_url added. Tool will offer to set defaults.

• claude/hooks/check-reports-on-exp.sh — tool will offer to delete it.
• The legacy reports/ directory in the code repo is not auto-migrated.

Tool will detect it and emit a warning suggesting manual migration to ~/proj-docs/<project>/<branch_full>/.... v0.4+ may add a migrate_reports_to_docs_repo helper.

Strict invariants

• `<remote_workspace>` is a mutagen mirror — never write to it manually.

Mutagen is the sole source of truth for files under /workspace/. Manual writes are unstable in two directions:

• If alpha (the local worktree) has the file, the next flush overwrites

whatever you placed on the remote.

• If alpha doesn't have the file (you scp'd a snippet from a

different worktree, dropped an ad-hoc script the local repo lacks, etc.), the next flush deletes it as out-of-tree drift — usually within seconds.

For files that must not propagate back to alpha — diagnostic scripts, cross-worktree temp blobs, throwaway data fixtures — use /tmp/ or any other path outside <remote_workspace>. To inspect a synced file, use read_remote_file. To fix a synced file, edit locally and let the next flush carry the change.

• `git` does not work inside `<remote_workspace>`. mutagen replicates

the worktree's files but its .git pointer is the alpha-side absolute path (e.g. /home/<user>/.../code/<exp>/.git/worktrees/<exp>), which doesn't exist on the remote. git log / git status / git diff invoked inside /workspace will fail with fatal: not a git repository. To check which commit is on the remote, either run git log locally in the alpha worktree, read the latest run's manifest.yaml (git.commit), or grep file contents for the change you're tracking.

• `OUTPUT_ROOT` must be outside the worktree (typically

/root/autodl-fs/runs/...). Otherwise outputs get sucked into the code forward sync and you'll see a sync storm.

• Reports are always in the docs repo, never in the code repo. If you

find yourself about to write a reports/ file inside any code-repo worktree, stop and use start_report.

• Docs repo is never in mutagen sync. Its v4 path (<container>/docs/)

must not appear in .mutagen.yml.

• Instance-down success is not the MCP envelope. instance_down returning

a successful tool result only means the request was handled. Inspect acted: true means AutoDL was actually asked to power off or release; false means no platform action happened. If acted=false, read reason and did_you_mean before telling the user an instance was closed.

• Cross-repo invariant: cited code commits must be pushed to the code

repo's origin before commit_report will push the docs repo. The hook check-ssh-train.sh already enforces "training-time push" for the manifest's commit; commit_report enforces this again at report time in case you're writing a report retroactively.

• Manifest commits should be real. start_training enforces clean +

pushed by default; only set allow_dirty=true or require_pushed=false when the user explicitly accepts the provenance warning.

• Don't ask the user about paths and conventions. Read

.proj-tool/project.toml. The user already configured this once during retrofit.

Reference modules

Read these when their topic is in play:

• references/manifest-schema.md — full manifest.yaml fields, when each is

filled, how to handle missing fields.

• references/report-frontmatter.md — report frontmatter schema (with

git.code_repo_url and git.commit_url cross-repo fields), single vs comparison vs ablation, the >8-runs externalization rule, file path conventions in the docs repo.

• references/dirty-worktree-handling.md — what to do when finalize_run

failed, when a manifest is stuck on running, when the user really needs to override the cleanliness check.

• references/mutagen-troubleshooting.md — diagnose output → fix mapping;

agent version mismatches; one-way-safe conflict resolution. (The docs repo isn't relevant here — it's not in mutagen.)

Version log

• 0.5.9 — Updated the default remote output guidance to

/root/autodl-fs/runs/... so run outputs survive AutoDL instance release; /root/autodl-tmp/... is now described as explicit scratch storage.

• 0.5.8 — Added "Quick diagnostics / step-0 sanity check" hot path so

short remote scripts route through remote_exec instead of polluting run_index via start_training; rewrote the <remote_workspace> invariant to be bidirectional (mutagen also deletes manual writes that aren't on alpha) with explicit /tmp/ guidance for cross-worktree temp files; added a new invariant noting that git is unusable inside <remote_workspace> because mutagen doesn't replicate the alpha-side .git pointer.

• 0.5.7 — Updated docs for v4 container layout; removed false claim that

start_training injects a finalize hook; documented get_run_status orphaned state and project-image default for instance_up.

• 0.5.6 — Image-build guidance now exports AutoDL's Miniconda path before

running Python in setup.sh; manifest reference documents git.is_unpushed for start_training(require_pushed=false).

• 0.5.5 — M5 codex integrative-review fix (cdcbc02): image_save_status

return field renamed status → image_status to avoid colliding with the MCP envelope's reserved status key; updated 2 wording occurrences in the Image build section so the polling instruction matches the new field name.

• 0.5.4 — M5 C4: updated Image build section to reflect non-blocking

image_save + image_save_status poll flow; added image_save_status to tool inventory table; clarified that remote_exec returns exit_code (non-zero is a setup.sh failure, not a tool error) with full output at log_path.

• 0.5.3 — corrected CSV columns in report-frontmatter.md to match

build_runs_csv_content output (run_id, commit_at_run_time, started_at, status, <metrics>); removed unimplemented --allow-unpushed-code flag reference.