image-debug

projtool builds project images by spinning up a one-shot AutoDL "builder" instance, running the project's setup.sh inside it, and (on success) saving the result as a custom image. This skill activates when something goes wrong during setup.sh. Most failures fall into a small number of categories, listed below — read the relevant references/ file before proposing a fix.

Workflow

1. Don't tear down on first failure. The builder instance is alive and

has all the partial state. Closing it loses minutes of debug context.

1. Read the tail of stderr. mcp__projtool__remote_exec returns

stderr_tail (and stdout_tail) inline in the response; the full output is written to the file at log_path in the result. The error message is almost always in the last 30 lines of stderr_tail.

1. Match the error to a category (below). Read the matching reference.
2. Propose a fix. Either patch setup.sh locally + push it to the

instance via write_remote_file, or run the fix command directly via remote_exec to validate before bake-in. For non-trivial changes, patching setup.sh is preferred — it makes the next build idempotent. For AutoDL PyTorch base images, make the first line after set -euo pipefail export Miniconda onto PATH: export PATH="/root/miniconda3/bin:$PATH". Some non-interactive SSH shells do not source bashrc, so plain python / python3 may otherwise be missing even though the image has conda.

1. Re-run the failed step, not the whole setup.sh. Most steps are

idempotent (apt-get, pip install). Only re-run the full script when confident the earlier steps will no-op cleanly.

1. On success, confirm the build passes end-to-end before saving.

Then: image_save (non-blocking — returns pending_image_uuid immediately) → poll image_save_status until image_status='ready' → image_build_finalize. Saving an image with hidden partial state (e.g., wrong CUDA toolkit) creates a long-tail bug.

1. On give-up, image_build_abandon releases the builder. Don't leave

builders running — they bill at GPU rates.

Failure categories

Disk full

Builder images sometimes have small system disks (typically 30–50GB). pip caches and apt downloads chew it fast.

df -h /                  # see what's full
apt-get clean            # frees /var/cache/apt
pip cache purge          # frees ~/.cache/pip
rm -rf /tmp/*            # if anything is staged there

If /root/autodl-tmp/ is mounted with extra space, redirect caches there before reinstalling:

export PIP_CACHE_DIR=/root/autodl-tmp/pip-cache
mkdir -p $PIP_CACHE_DIR

If the system disk is genuinely too small, ask the user to switch to a builder image with a larger root volume and start over.

Network unreachable

AutoDL containers ship with regional mirrors configured. The package manager often points to mirrors.aliyun.com or similar. Failures are typically transient — retry once. If persistent:

# Quick test
curl -fsSI https://mirrors.aliyun.com/ -o /dev/null && echo OK || echo BAD

# Fall back to pypi.org / archive.ubuntu.com
sed -i '[email protected]/pypi/[email protected]/simple@g' /etc/pip.conf
sed -i '[email protected]/[email protected]/ubuntu@g' /etc/apt/sources.list

Don't bake the fallback into the saved image — it'll be slow for the next user. Patch setup.sh to detect and switch instead, or just re-run.

Strict invariants

• Every failure must result in a patched `setup.sh` (or an explicit note

that no patch is needed because it was transient). Otherwise the next build will hit the same problem.

• **The setup.sh patches go into the local copy, not just the remote

one.** setup.sh is a project file (committed to the repo). After a fix, edit the local file, then write_remote_file to push it before re-run.

• Don't `image_save` until you're confident. A saved image with quiet

problems (e.g., missing CUDA libs that don't surface until first training) wastes much more time than careful verification.

• Builder instance must be released. On giving up, image_build_abandon.

On success: image_save (non-blocking) → poll image_save_status until image_status='ready' → image_build_finalize (which also releases the builder).