ComfyUI Workflow Helper Skill

Expert guidance for ComfyUI workflow creation, node configuration, and optimization.

Capabilities

This skill provides expert assistance with:

1. Workflow Setup & Management

• Creating custom workflows from scratch
• Importing and modifying existing workflows
• Workflow organization and structure
• Node connection best practices

1. Node Configuration

• Model loaders (Checkpoints, LoRAs, VAEs)
• Samplers and schedulers
• Conditioning nodes (prompts)
• Image processing nodes
• ControlNet integration

1. Model Management

• Installing models (checkpoints, VAEs, text encoders, diffusion models)
• Model organization in directories
• Model format compatibility
• OmniGen2 setup

1. Performance Optimization

• VRAM management strategies
• Batch processing optimization
• Workflow efficiency improvements
• Node caching strategies

1. Troubleshooting

• Model loading errors
• Node connection issues
• OOM (Out of Memory) errors
• Workflow execution failures

Key Knowledge Base

ComfyUI Directory Structure

/srv/comfyui/
├── models/
│   ├── checkpoints/          # SD models (.safetensors, .ckpt)
│   ├── vae/                  # VAE models
│   ├── loras/                # LoRA models
│   ├── text_encoders/        # Text encoder models
│   ├── diffusion_models/     # Diffusion models
│   ├── controlnet/           # ControlNet models
│   ├── upscale_models/       # Upscaler models
│   └── embeddings/           # Textual inversion embeddings
├── input/                    # Input images
├── output/                   # Generated images
└── custom_nodes/             # Custom node extensions

Essential Nodes

Loading Nodes:

• Load Checkpoint - Load SD models
• Load VAE - Load VAE models
• Load LoRA - Load LoRA models

Conditioning Nodes:

• CLIP Text Encode (Prompt) - Positive and negative prompts
• Conditioning Combine - Merge multiple conditioning

Sampling Nodes:

• KSampler - Main sampler node
• KSampler Advanced - Advanced sampling control

Image Nodes:

• VAE Decode - Convert latent to image
• VAE Encode - Convert image to latent
• Save Image - Save output

Model Nodes:

• Model Sampling Discrete - Sampling configuration
• Model Sampling SD3 - SD3-specific sampling

OmniGen2 Setup

Required Models:

1. VAE: ae.safetensors → /srv/comfyui/models/vae/
2. Text Encoder: qwen_2.5_vl_fp16.safetensors → /srv/comfyui/models/text_encoders/
3. Diffusion Model: omnigen2_fp16.safetensors → /srv/comfyui/models/diffusion_models/

Download Script:

# Use the provided download script
/root/homelab/scripts/download-comfyui-omnigen2-models.sh

# Or manual download:
mkdir -p /srv/comfyui/models/{vae,text_encoders,diffusion_models}
wget -O /srv/comfyui/models/vae/ae.safetensors \
  "https://huggingface.co/Comfy-Org/Lumina_Image_2.0_Repackaged/resolve/main/split_files/vae/ae.safetensors"
# ... (similar for other models)

Common Workflows

Basic txt2img Workflow

Nodes Required:

1. Load Checkpoint
2. CLIP Text Encode (Prompt) x2 (positive & negative)
3. Empty Latent Image
4. KSampler
5. VAE Decode
6. Save Image

Connection Flow:

Load Checkpoint → CLIP (positive/negative) → KSampler
Empty Latent → KSampler → VAE Decode → Save Image
Load Checkpoint → VAE Decode

img2img Workflow

Additional Nodes:

1. Load Image
2. VAE Encode

Connection Flow:

Load Image → VAE Encode → KSampler
(Rest similar to txt2img)

ControlNet Workflow

Additional Nodes:

1. Load ControlNet Model
2. Apply ControlNet
3. Preprocessor nodes (depends on ControlNet type)

Usage Examples

Example 1: Setting Up OmniGen2

User: "Help me set up OmniGen2 in ComfyUI"

Assistant Response: I'll guide you through setting up OmniGen2 in ComfyUI.

Step 1: Download Models

Run the download script:

bash /root/homelab/scripts/download-comfyui-omnigen2-models.sh

Or download manually:

• VAE: ae.safetensors → /srv/comfyui/models/vae/
• Text Encoder: qwen_2.5_vl_fp16.safetensors → /srv/comfyui/models/text_encoders/
• Diffusion Model: omnigen2_fp16.safetensors → /srv/comfyui/models/diffusion_models/

Step 2: Restart ComfyUI

# If using Docker:
docker restart comfyui-container

# If using systemd:
sudo systemctl restart comfyui

# If running manually:
# Stop and restart the ComfyUI process

Step 3: Verify in WebUI

1. Open ComfyUI web interface
2. Check that models appear in dropdowns:

• VAE loader should show "ae.safetensors"
• Text encoder nodes should show "qwen_2.5_vl_fp16.safetensors"
• Diffusion model loader should show "omnigen2_fp16.safetensors"

Step 4: Create Workflow

Use OmniGen2-specific nodes:

• Load the OmniGen2 diffusion model
• Load Qwen text encoder
• Load Lumina VAE
• Configure sampler for OmniGen2

Example 2: Creating a Basic Workflow

User: "How do I create a simple txt2img workflow in ComfyUI?"

Assistant Response: I'll help you create a basic txt2img workflow.

Nodes to Add:

1. Right-click → Add Node → loaders → Load Checkpoint

• Select your SD model

1. Right-click → Add Node → conditioning → CLIP Text Encode (Prompt) × 2

• One for positive prompt
• One for negative prompt

1. Right-click → Add Node → latent → Empty Latent Image

• Set width: 512
• Set height: 512
• Batch size: 1

1. Right-click → Add Node → sampling → KSampler

• Seed: -1 (random)
• Steps: 20-40
• CFG: 4.5 (for photorealism)
• Sampler: dpmpp_2m_karras
• Scheduler: karras

1. Right-click → Add Node → latent → VAE Decode

1. Right-click → Add Node → image → Save Image

Connections:

Load Checkpoint [MODEL] → KSampler [model]
Load Checkpoint [CLIP] → CLIP Text Encode (pos) [clip]
Load Checkpoint [CLIP] → CLIP Text Encode (neg) [clip]
Load Checkpoint [VAE] → VAE Decode [vae]

CLIP Text Encode (pos) [CONDITIONING] → KSampler [positive]
CLIP Text Encode (neg) [CONDITIONING] → KSampler [negative]

Empty Latent Image [LATENT] → KSampler [latent_image]

KSampler [LATENT] → VAE Decode [samples]
VAE Decode [IMAGE] → Save Image [images]

Usage:

1. Enter your prompt in positive CLIP Text Encode
2. Enter negative prompt in negative CLIP Text Encode
3. Click "Queue Prompt" to generate

Example 3: Troubleshooting OOM Errors

User: "ComfyUI keeps crashing with out of memory errors. How do I fix this?"

Assistant Response: OOM errors in ComfyUI can be resolved with these strategies:

1. Reduce Resolution

• Change Empty Latent Image dimensions:
• From 768×768 → 512×512
• From 1024×1024 → 768×768
• Lower resolution = less VRAM usage

2. Reduce Batch Size

• In Empty Latent Image node:
• Set batch_size to 1
• Generate images one at a time

3. Use VRAM-Efficient Models

• Use pruned models (smaller file size)
• Use fp16 models instead of fp32
• Example: v1-5-pruned-emaonly.safetensors vs full versions

4. ComfyUI Launch Args Add these flags when starting ComfyUI:

--lowvram          # Aggressive VRAM optimization
--medvram          # Moderate VRAM optimization
--normalvram       # Normal VRAM usage
--cpu              # Use CPU instead of GPU (slow but no VRAM)

5. Clear Cache

• Restart ComfyUI between generations
• Clear browser cache
• Delete temporary files

6. Check Node Efficiency

• Remove unnecessary nodes
• Avoid multiple simultaneous samplers
• Use single VAE decode instead of multiple

For 2GB VRAM:

• Stick to 512×512 resolution
• Use --lowvram flag
• Generate single images (batch=1)
• Use pruned/optimized models

Best Practices

1. Workflow Organization

• Group related nodes together
• Use reroute nodes for clean connections
• Add note nodes to document workflow sections
• Save workflows with descriptive names

1. Node Connections

• Always connect correct output to correct input types
• Use color coding: MODEL, CLIP, VAE, CONDITIONING, LATENT, IMAGE
• Double-check all connections before queuing

1. Model Management

• Keep models organized in proper subdirectories
• Use descriptive filenames
• Delete unused models to save space
• Verify checksums after downloading

1. Performance

• Start with lower resolutions for testing
• Use batch generation only when VRAM allows
• Cache models by reusing same checkpoint across generations
• Close other GPU applications

1. Troubleshooting

• Check ComfyUI console for error messages
• Verify model files are in correct directories
• Ensure model formats are compatible
• Test with simple workflow first

Quick Reference

KSampler Settings for Photorealism

seed: -1 (random)
steps: 20-40
cfg: 4.5 (photorealism breakthrough)
sampler_name: dpmpp_2m_karras
scheduler: karras
denoise: 1.0

Resolution Limits by VRAM

Common Node Shortcuts

• Ctrl + Enter: Queue Prompt
• Ctrl + Shift + Enter: Queue Prompt (front of queue)
• Double Click: Add node (search)
• Ctrl + D: Duplicate selected nodes
• Delete: Remove selected nodes

Advanced Techniques

Batch Processing

Use batch nodes to generate multiple variations:

Batch Size: 4 in Empty Latent Image
→ Generates 4 images per queue

Seed Control

For consistent results:

Fixed Seed: Use specific number (e.g., 12345)
Random Seed: Use -1
Seed Increment: Batch Size controls seed increment

LoRA Stacking

Apply multiple LoRAs:

Load Checkpoint → Load LoRA (1) → Load LoRA (2) → KSampler
Set strength: 0.5-1.0 per LoRA

Additional Resources

• ComfyUI GitHub: https://github.com/comfyanonymous/ComfyUI
• Custom Nodes Registry: https://github.com/ltdrdata/ComfyUI-Manager
• OmniGen2 Documentation: Hugging Face model pages
• Download Script: /root/homelab/scripts/download-comfyui-omnigen2-models.sh

When to Invoke This Skill

Invoke this skill when users ask about:

• ComfyUI workflow creation
• Node configuration and connections
• Model installation for ComfyUI
• OmniGen2 setup
• Workflow optimization
• VRAM management in ComfyUI
• Troubleshooting ComfyUI errors
• Sampler and scheduler settings
• ControlNet integration
• Batch processing in ComfyUI