██████╗ ██████╗ ███╗   ███╗███████╗██╗   ██╗██████╗ ██╗██╗      ██████╗ ████████╗
██╔════╝██╔═══██╗████╗ ████║██╔════╝╚██╗ ██╔╝██╔══██╗██║██║     ██╔═══██╗╚══██╔══╝
██║     ██║   ██║██╔████╔██║█████╗   ╚████╔╝ ██████╔╝██║██║     ██║   ██║   ██║
██║     ██║   ██║██║╚██╔╝██║██╔══╝    ╚██╔╝  ██╔═══╝ ██║██║     ██║   ██║   ██║
╚██████╗╚██████╔╝██║ ╚═╝ ██║██║        ██║   ██║     ██║███████╗╚██████╔╝   ██║
 ╚═════╝ ╚═════╝ ╚═╝     ╚═╝╚═╝        ╚═╝   ╚═╝     ╚═╝╚══════╝ ╚═════╝    ╚═╝

ComfyPilot v1.8.1

ComfyPilot is an MCP server for ComfyUI. It gives an AI agent a clean tool surface for workflow building, queueing, progress monitoring, image retrieval, snapshots, and VRAM safety.

Documentation

• Production manual: docs/MANUAL.md
• Release notes: CHANGELOG.md

What This Is

• A practical bridge between AI agents and ComfyUI.
• An 88-tool MCP surface for workflow building, model management, image upload/download, history, monitoring, snapshots, technique memory, VRAM safety, output routing, blueprints, Mermaid viz, PNG ingest, n-dim parameter sweeps, partner-API discovery, model-hub search, comfy-cli lifecycle (install ComfyUI, launch/stop server, install nodes, download models), and diagnostics (workflow schema extraction, log fetching, trust checks, hardware verdicts, timeout suggestions, auto-fix workflow deps, upload-and-inject convenience).
• Live model-folder discovery: diffusion_models/, text_encoders/, clip_vision/, style_models/, gligen/ (and every other folder the connected ComfyUI actually exposes) are searched by default - no more checkpoint-centric blind spots.
• A workflow-oriented loop built for iteration, not one-shot guessing.
• A small technique library for saving and replaying working patterns.

Core Thinking Model (How To Think With This MCP)

Use this loop for every non-trivial task:

1. Check system first - Read GPU state before loading models. Start with comfy_get_system_stats, comfy_check_vram.

1. Check memory - Before building from scratch, use comfy_search_techniques to check if a similar workflow already exists in the library.

1. Build in small steps - Use comfy_build_workflow for any intent (txt2img, img2img, upscale, inpaint, controlnet, txt2video, img2video, image2_3d, txt2music, super_resolution, interpolate_frames, segment, train_lora, txt2audio). The family is auto-detected from the installed checkpoint for family-routed intents; family-agnostic intents (super_resolution, interpolate_frames, segment, train_lora, txt2audio) dispatch by intent name regardless of checkpoint. For common recipes, start from comfy_insert_blueprint(name="flux2-txt2img") or any of the 13 bundled blueprints. Validate with comfy_validate_workflow before queueing (6-pass: schema, catalog, graph, anti-cycle, environment, execution-risk).

1. Monitor and retrieve - Queue with comfy_queue_prompt, watch with comfy_watch_progress, retrieve with comfy_get_output_image (returns image content blocks directly in chat).

1. Snapshot before changes - Always comfy_snapshot_workflow before modifying a working workflow. Use comfy_diff_snapshots and comfy_restore_snapshot for undo.

1. Route outputs - Send generated images to disk, TouchDesigner, or Blender with comfy_send_to_disk, comfy_send_to_td, comfy_send_to_blender.

Tool Map (88 Tools)

1) System + GPU

Use for connection health, GPU diagnostics, and VRAM management.

• comfy_get_system_stats, comfy_get_gpu_info, comfy_get_features
• comfy_list_extensions, comfy_restart, comfy_free_vram

2) Models

Use for discovering and managing every model file ComfyUI exposes. Folder taxonomy is discovered live from /models so modern families whose weights live under diffusion_models/, text_encoders/, clip_vision/, style_models/, or gligen/ are covered by default.

• comfy_list_models - Paginated listing inside a single folder (folder, limit, offset).
• comfy_get_model_info - Normalized node schema for a model-loader node class.
• comfy_list_model_folders - Live folder list from GET /models, with fallback to a 16-folder static default when the endpoint is unreachable. Reports source: live | fallback.
• comfy_search_models - Searches every discovered folder by default. Pass folders=["checkpoints", "loras"] to narrow. Empty query returns a full inventory.
• comfy_refresh_models - Re-fetches the cache across every folder; returns per-folder counts + a total_models number.

3) Workflow Execution

Use for queueing, cancelling, and managing prompt execution.

• comfy_queue_prompt, comfy_get_queue, comfy_cancel_run
• comfy_interrupt, comfy_clear_queue
• comfy_validate_workflow, comfy_export_workflow, comfy_import_workflow

4) Nodes + Schema

Use for exploring ComfyUI's node catalog and understanding widget schemas.

• comfy_list_node_types, comfy_get_node_info, comfy_search_nodes
• comfy_get_categories, comfy_get_embeddings, comfy_inspect_widget

5) Images + Visual Output

Use for retrieving generated images (returned as image content blocks in chat).

• comfy_get_output_image, comfy_upload_image, comfy_list_output_images
• comfy_download_batch, comfy_get_image_url

6) History

Use for inspecting past generations and their outputs.

• comfy_get_history, comfy_get_run_result, comfy_delete_history
• comfy_clear_history, comfy_search_history

7) Monitoring + Progress

Use for tracking active jobs and observing workflow dynamics.

• comfy_watch_progress, comfy_subscribe, comfy_unsubscribe
• comfy_get_events, comfy_describe_dynamics, comfy_get_status

8) Workflow Snapshots

Use for undo/restore and workflow version tracking.

• comfy_snapshot_workflow, comfy_list_snapshots, comfy_diff_snapshots
• comfy_restore_snapshot, comfy_delete_snapshot, comfy_auto_snapshot

9) Technique Memory

Use for learning, saving, and replaying reusable workflow patterns.

• comfy_save_technique - Save a workflow as a reusable technique with tags and metadata.
• comfy_search_techniques - Search the library by text query and/or tags.
• comfy_list_techniques - List all saved techniques with metadata.
• comfy_replay_technique - Load a saved technique's workflow for immediate use.
• comfy_favorite_technique - Mark techniques as favorites and rate them (0-5).

Technique storage lives at ~/.comfypilot/techniques/ as individual JSON files.

10) Safety + VRAM Guard

Use for guardrails, pre-flight checks, and emergency control.

• comfy_check_vram - Check GPU VRAM usage with status levels (ok/warn/critical).
• comfy_set_limits - Configure safety thresholds (warn %, block %, max queue size).
• comfy_detect_instability - Check for near-OOM conditions and stuck jobs.
• comfy_validate_before_queue - Pre-flight check: VRAM headroom + queue capacity.
• comfy_emergency_stop - Interrupt current job, clear queue, free all VRAM.

11) Workflow Builder

Use for family-aware template construction and node editing.

• comfy_build_workflow - Family-aware: detects the checkpoint family (SD 1.5, SDXL, SD 3.5, Flux 2, Qwen-Image, Wan 2.2, LTX-2, HunyuanVideo, Hunyuan3D, ACE-Step) and dispatches to the right graph topology. Intents: txt2img, img2img, upscale, inpaint, controlnet, txt2video, img2video, image2_3d, txt2music.
• comfy_add_node - Add a node to a workflow-in-progress.
• comfy_connect_nodes - Wire node outputs to inputs.
• comfy_set_widget_value - Set widget values on nodes.
• comfy_apply_template - Alias for comfy_build_workflow.

12) Output Routing

Use for agent-orchestrated cross-app delivery of generated images.

• comfy_send_to_disk - Save output image to local filesystem.
• comfy_send_to_td - Route output to TouchDesigner project directory.
• comfy_send_to_blender - Route output to Blender project directory.
• comfy_list_destinations - List configured output destinations.

13) Subgraph Blueprints

Use for assembling workflows from named reusable macros.

• comfy_list_blueprints - List all available blueprints (user + bundled).
• comfy_insert_blueprint - Materialize a blueprint into a workflow dict with optional per-node input overrides.
• comfy_publish_subgraph - Save a set of nodes as a reusable named blueprint.

Blueprint storage: user-published at ~/.comfypilot/blueprints/ (override via COMFY_BLUEPRINT_DIR); bundled examples ship in blueprints/ inside the repo.

14) Visualization + Ingest + Sweep

Use for rendering workflows, re-importing PNG outputs, and parameter sweeps.

• comfy_visualize_workflow - Render an API-format workflow as a Mermaid flowchart TD source.
• comfy_import_workflow_from_png - Extract the embedded workflow JSON from a ComfyUI-saved PNG (prefers 'prompt' API-format chunk, falls back to 'workflow' UI chunk).
• comfy_sweep - Enqueue N copies of a workflow with a single parameter varying across values; returns prompt_ids + grid layout hint.
• comfy_sweep_grid - N-dimensional Cartesian-product sweep over multiple node_id.param axes. Capped at 64 combinations by default (override with max_combinations).

15) Model Hub Search

Use for discovering models to download from public hubs.

• comfy_search_hub - Search HuggingFace or CivitAI for models matching a query. Returns normalized hits (id, name, url, downloads, tags). Use source="huggingface" or source="civitai".

16) Partner APIs

Use for surfacing which vendor-hosted partner / API custom nodes are installed.

• comfy_list_partner_apis - Intersects ComfyUI's installed-extensions list with a curated catalog (Veo, Kling, ByteDance Seedance/Seedream, GPT-Image, Topaz, Tripo3D, Rodin, Recraft, Ideogram, NanoBanana, Sonilo, ElevenLabs, etc.) and reports which are live on this server with vendor / category / homepage metadata.

17) Lifecycle (v1.8.0)

Use for full setup-to-server flow when ComfyUI isn't running yet. All five tools shell out to the official comfy-cli — install it via pipx install comfy-cli or uvx --from comfy-cli comfy ....

• comfy_launch_server - Start ComfyUI (comfy launch --background --port N --listen H).
• comfy_stop_server - Stop the running server (comfy stop).
• comfy_install_node - Install a custom-node package by slug via Comfy Manager.
• comfy_list_installed_nodes - List currently-installed custom nodes.
• comfy_download_model - Download a model by URL into a chosen models/<folder> (supports CivitAI token).

18) Diagnostics (v1.8.0)

Use for workflow introspection, hardware verdicts, traceback extraction, and trust checks.

• comfy_extract_schema - Workflow-level controllable param + dependency summary (parameter list, model deps, embedding refs, output nodes, with summary_only mode).
• comfy_fetch_logs - Extract just the error traceback + completed-nodes list from /history/{prompt_id} — handles both status.exec_info.errors and status.messages shapes.
• comfy_inspect_workflow - Trust check: classify every class_type as stock / custom and surface warnings before queueing untrusted workflows.
• comfy_recommend_runtime - Hardware verdict (ok / marginal / cloud) + the right comfy-cli install flag for this hardware.
• comfy_suggest_timeout - Per-workflow HTTP timeout suggestion based on long-running output classes (default 300s, up to 3600s for training).

19) Convenience (v1.8.0)

Use to compress multi-step flows that an agent would otherwise script manually.

• comfy_install_workflow_deps - Auto-install every missing custom-node package the workflow references (wraps comfy node install-deps --workflow).
• comfy_run_with_inputs - Upload local images + inject as workflow inputs + queue — collapses the three-step img2img / inpaint flow into one call.
• comfy_randomize_seeds - Replace seed=-1 sentinels with cryptographic-grade random uint32 values; force=True randomises every seed widget.

MCP Resources

Six fixed resources:

• comfy://system/info - System stats, GPU info, ComfyUI version
• comfy://server/capabilities - Detected server profile, version, frontend version, auth method, WebSocket availability, OpenAPI version, cache provider
• comfy://nodes/catalog - Node catalog preview (first 100 names)
• comfy://models/{folder} - Model listing by folder (checkpoints, loras, vae, diffusion_models, text_encoders, etc.)
• comfy://embeddings - Available embeddings
• comfy://api/openapi - ComfyUI's OpenAPI 3.1 spec (v0.20.0+). Returns {"error": ...} on older builds.

Plus four resource templates (parameterized URIs):

• comfy://nodes/catalog/{page} - Paginated node catalog, 100 nodes per page ({page} = 0, 1, 2, ...)
• comfy://nodes/by-category/{category} - Node class_types whose category starts with the given prefix (e.g. sampling, loaders/video)
• comfy://templates/catalog - Workflow templates advertised by ComfyUI core + custom nodes (via /workflow_templates)
• comfy://docs/{node_class} - Embedded node documentation (v0.3.68+ docs endpoint, with object_info description fallback)

How To Use It (Practical Workflow)

1. Connect MCP client to ComfyPilot.
2. Check system state and VRAM headroom.
3. Build a workflow (template or custom API JSON).
4. Validate, then queue the prompt.
5. Watch progress until complete.
6. Retrieve the output image (displayed inline in chat).
7. Route to disk/TD/Blender if needed. Snapshot at stable milestones.

What It Is Good At

• Building and iterating on ComfyUI workflows through conversation.
• Family-aware generation across 11 model families (SD 1.5, SDXL, SD 3.5, Flux 2, Qwen-Image, Wan 2.2, LTX-2, HunyuanVideo, Hunyuan3D, ACE-Step, Ernie Image) covering images, video, 3D meshes, and music. Plus 5 family-agnostic intents (super-resolution via SUPIR, frame interpolation via RIFE/FILM, segmentation via SAM 3.1, native LoRA training, text-to-audio via Stable Audio).
• Starting from bundled blueprints for every family (13 ship in-repo) and customizing from there.
• 6-pass workflow validation (schema + catalog + graph + anti-cycle + environment + execution-risk) catching broken links, cycles, missing models, deprecated models, oversized latents before you hit ComfyUI.
• Monitoring GPU resources and preventing OOM situations via VRAMGuard.
• Returning generated images directly in the chat (image content blocks).
• Cross-app output routing to TouchDesigner and Blender with atomic writes + sidecar manifests (prompt_id, seeds, model refs, dimensions).
• Snapshot/restore for non-destructive workflow experimentation, now optionally persistent across restarts.
• Learning and replaying reusable workflow patterns via the technique library.
• Parameter sweeps (queue N variations of a single widget value).
• Importing a workflow back from a saved PNG's embedded metadata.
• Rendering any workflow as Mermaid for quick visual inspection.
• Searching HuggingFace and CivitAI directly for models.

What It Is Not Good At

• Replacing artistic direction by itself.
• Streaming real-time video output (snapshots and polls, not live frames).
• "One shot perfect generation" without iterative refinement.
• Running without a working ComfyUI instance somewhere (cloud or local). The lifecycle tools (comfy_launch_server, comfy_install_node, comfy_install_workflow_deps, comfy_download_model) can bring a local install from zero to ready, but they need comfy-cli on PATH — install via pipx install comfy-cli or uvx --from comfy-cli comfy ... first.

Model Families

11 first-class model families plus 5 family-agnostic intent-overrides. Each family has a builder template (or several) that emits a workflow matched to the right graph topology. The family is detected from the checkpoint filename; you can override by passing checkpoint= in params. Intent-overrides skip family detection entirely and dispatch by intent name.

Intent-overrides (family-agnostic; dispatched by intent name regardless of detected checkpoint family):

Transports

• stdio (default) - Best for local clients like Claude Code, Claude Desktop, Cursor.
• streamable-http - Best for hosted / remote deployments. Enable with:

uv run comfypilot --transport streamable-http --host 0.0.0.0 --port 8765

Support Matrix

Quick Setup

Option 1: MCPB (one-click, Claude Desktop)

Easiest for Claude Desktop users:

1. Download comfypilot-v1.8.1.mcpb from the latest release.
2. Double-click the file (or drag it into Claude Desktop's Settings > Extensions panel).
3. Claude Desktop renders a form for COMFY_URL, COMFY_API_KEY, and the other config fields — fill in what you need.
4. Click Install. Restart is handled automatically.

MCPB uses the type: "uv" server mode, so Python dependencies (mcp, httpx, pydantic, websockets) are resolved by uv at first run. You need uv installed on the host — install uv.

The bundle ships manifest.json + pyproject.toml + the full src/comfy_mcp/ package + the 13 bundled blueprints. Manifest: manifest.json.

Option 2: Local development runtime

git clone https://github.com/dreamrec/ComfyPilot.git
cd ComfyPilot
uv sync
uv run comfypilot

Option 3: Claude Code plugin

claude plugin add /path/to/ComfyPilot

Local override

If your ComfyUI instance lives somewhere other than 127.0.0.1:8188 (Tailscale, remote LAN, custom port), copy .mcp.local.json.example to .mcp.local.json and edit it. The .mcp.local.json file is gitignored so per-machine URLs and secrets never get committed.

MCP Bundle (Standardized)

ComfyPilot ships a standard MCP bundle in-repo:

• mcp/manifest.json
• mcp/profiles/claude-desktop.json, cursor.json, generic.json

The bundled profile JSON files pin uv run with --directory; replace /path/to/ComfyPilot with your checkout path when copying them into a client config. If you're launching from inside this repo, .mcp.json already uses ${CLAUDE_PLUGIN_ROOT}.

Manual client configuration example (Claude Desktop):

{
  "mcpServers": {
    "comfypilot": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/ComfyPilot", "comfypilot"],
      "env": {
        "COMFY_URL": "http://127.0.0.1:8188",
        "COMFY_API_KEY": ""
      }
    }
  }
}

Environment Variables

Test Suite

Run the test suite:

uv run pytest -v

Run a specific category:

uv run pytest tests/test_builder.py -v

For a quick smoke run:

uv run pytest -q

Reliability Habit

Treat this as mandatory for every meaningful task: before generations check VRAM, before workflow changes snapshot, after builds validate, after errors check comfy_detect_instability.

License

MIT

dreamrec // ComfyPilot // live laugh diffuse