AutoForge — Autonomous Optimization Framework

Stop reflecting. Start converging. Every iteration is measured, logged, and validated — not vibed.

AutoForge replaces ad-hoc "improve this" prompts with a rigorous optimization loop: define evals, run iterations, track pass rates in TSV, report live to your channel, and stop only when math says you're done. Multi-model cross-validation prevents the "same model grades its own homework" blind spot.

Four modes. One convergence standard.

AutoForge — Top-Agent Architecture

Overview

Agent (you)
├── State: results.tsv, current target file state, iteration counter
├── Iteration 1: evaluate → improve → write TSV → report
├── Iteration 2: evaluate → improve → write TSV → report
├── ...
└── Finish: report.sh --final → configured channel

Sub-Agent = You

"Sub-Agent" is a conceptual role, not a separate process. You (the top-agent) execute each iteration yourself: simulate/execute → evaluate → write TSV → call report.sh. The templates below describe what you do PER ITERATION — not what you send to another agent.

For code mode, run tests using the exec tool.

Multi-Model Setup (recommended for Deep Audits)

For complex audits, you can split two roles across different models:

Flow: Optimizer and Validator alternate. Optimizer iterations have status improved/retained/discard. Validator iterations confirm or refute the pass rate. Spawn validators as sub-agents with sessions_spawn and explicit model.

When to use Multi-Model: Deep Audits (>5 iterations expected), complex ground truth, or when a single model is blind to its own errors.

When Single-Model suffices: Simple CLI audits, prompt optimization, code with clear tests.

Configuration

AutoForge uses environment variables for reporting. All are optional — without them, output goes to stdout.

Hard Invariants

These rules apply always, regardless of mode:

1. TSV is mandatory. Every iteration writes exactly one row to results/[target]-results.tsv.
2. Reporting is mandatory. Call report.sh immediately after every TSV row.
3. --dry-run never overwrites the target. Only TSV, *-proposed.md, and reports are written.
4. Mode isolation is strict. Only execute steps for the assigned mode.
5. Iteration 1 = Baseline. Evaluate the original version unchanged, status baseline.

Modes — Read ONLY Your Mode!

You are assigned ONE mode. Ignore all sections for other modes.

Your mode is in the task prompt. Everything else is irrelevant to you.

TSV Format (same for ALL modes)

Header (once at loop start):

printf '%s\t%s\t%s\t%s\t%s\n' "iteration" "prompt_version_summary" "pass_rate" "change_description" "status" > results/[target]-results.tsv

Row per iteration:

printf '%s\t%s\t%s\t%s\t%s\n' "1" "Baseline" "58%" "Original version" "baseline" >> results/[target]-results.tsv

Use `printf` not `echo -e`! echo -e interprets backslashes in field values. printf '%s' outputs strings literally.

5 columns, TAB-separated, EXACTLY this order:

Escaping rules:

• Tabs in text fields → replace with spaces
• Newlines in text fields → replace with |
• Empty fields → use hyphen - (never leave empty)
• `$` and backticks → use printf '%s' or escape with \$ (prevents unintended variable interpolation)
• Unicode/Emoji allowed, count as 1 character (not bytes)

Status rules (based on pass-rate comparison):

• baseline — Mandatory for Iteration 1. Evaluate original version only.
• improved — Pass rate higher than previous best → new version becomes current state
• retained — Pass rate equal or marginally better → predecessor remains
• discard — Pass rate lower → change discarded, revert to best state

Reporting (same for ALL modes)

After EVERY TSV row (including baseline):

bash scripts/report.sh results/[target]-results.tsv "[Skill Name]"

After loop ends, additionally with --final:

bash scripts/report.sh results/[target]-results.tsv "[Skill Name]" --final

The report script reads AF_CHANNEL, AF_CHAT_ID, and AF_TOPIC_ID from environment. Without them, it prints to stdout with ANSI colors.

Stop Conditions (for ALL modes)

Priority — first matching condition wins, top to bottom:

1. 🛑 Minimum iterations — If specified in task (e.g. "min 5"), this count MUST be reached. No other condition can stop before.
2. 🛑 Max 30 iterations — Hard safety net, stop immediately.
3. ❌ 3× `discard` in a row → structural problem, stop + analyze.
4. ✅ 3× 100% pass rate (after minimum) → confirmed perfect, done.
5. ➡️ 5× `retained` in a row → converged, done.

Counting rules:

• 3× 100% = three iterations with pass_rate == 100%, not necessarily consecutive.
• 5× retained and 3× discard = consecutive (in a row).
• baseline counts toward no series.
• improved interrupts retained and discard series.

At 100% in early iterations: Keep going! Test harder edge cases. Only 3× 100% after the minimum confirms true perfection.

Recognizing Validator Noise

In multi-model setups, the Validator can produce false positives — fails that aren't real issues:

• Config path vs tool name confusion (e.g. agents.list[] ≠ agents_list tool)
• Inverted checks ("no X" → Validator looks for X as required)
• Normal English as forbidden reference (e.g. "runtime outcome" ≠ runtime: "acp")
• Overcounting (thread commands counted as subagent commands)

Rule: If after all real fixes >3 discards come in a row and the fail justifications don't hold up under scrutiny → declare convergence, don't validate endlessly.

Execution Modes

mode: prompt

Only read if your task contains `mode: prompt`!

Per Iteration: What you do

1. Read current prompt/skill
2. Mentally simulate 5 different realistic scenarios
3. Evaluate each scenario against all evals (Yes=1, No=0)
4. Pass rate = (Sum Yes) / (Eval count × 5 scenarios) × 100
5. Compare with best previous pass rate → determine status
6. On improved: propose minimal, surgical improvement
7. Write TSV row + call report.sh
8. Check stop conditions

At the End

Best version → results/[target]-proposed.md + report.sh --final

mode: code

Only read if your task contains `mode: code`!

Per Iteration: What you do

1. Create sandbox: SCRATCH=$(mktemp -d) && cd $SCRATCH
2. Write current code to sandbox
3. Execute test command (with timeout 60s)
4. Measure: exit_code, stdout, stderr, runtime
5. Evaluate against evals → calculate pass rate
6. On improved: minimal code improvement + verify again
7. Write TSV row + call report.sh
8. Check stop conditions

Code Eval Types

At the End

Best code → results/[target]-proposed.[ext] + report.sh --final

mode: audit

Only read if your task contains `mode: audit`!

⚠️ DO NOT write your own code. Only test CLI commands of the target tool (--help + read-only).

Two Variants

Simple Audit (CLI skill, clear commands):

• 2 iterations: Baseline → Proposed Fix
• For tools with clear --help output and simple command structure

Deep Audit (complex docs, many checks):

• Iterative loop like prompt/code, same stop conditions
• For extensive documentation with many checkpoints (e.g. config keys, tool policy, parameter lists)
• Recommended: Multi-Model setup (Opus Optimizer + external Validator)

Simple Audit Flow

1. Write TSV header
2. Iteration 1 (Baseline): Test every documented command → pass rate → TSV + report
3. Iteration 2 (Proposed Fix): Write improved SKILL.md → expected pass rate → TSV + report
4. Improved SKILL.md → results/[target]-proposed.md
5. Detail results → results/[target]-audit-details.md (NOT in TSV!)
6. report.sh --final

Deep Audit Flow

1. Write TSV header
2. Iteration 1 (Baseline): Extract ground truth from source, define all checks, evaluate baseline
3. Iterations 2+: Optimizer fixes issues → Validator checks → TSV + report per iteration
4. Loop runs until stop conditions trigger (3× 100%, 5× retained, 3× discard)
5. Final version → results/[target]-proposed.md or results/[target]-v1.md
6. report.sh --final

Fixed Evals (audit)

1. Completeness — Does SKILL.md cover ≥80% of real commands/config?
2. Correctness — Are ≥90% of documented commands/params syntactically correct?
3. No stale references — Does everything documented actually exist?
4. No missing core features — Are all important features covered?
5. Workflow quality — Does quick-start actually work?

mode: project

Only read if your task contains `mode: project`!

⚠️ This mode operates on an ENTIRE repository/directory, not a single file. Cross-file consistency is the core feature — this is NOT "audit on many files."

Three Phases

Project mode runs through three sequential phases. Phases 1 and 2 happen once (in Iteration 1 = Baseline). Phase 3 is the iterative fix loop.

Phase 1: Scan & Plan

1. Analyze the repo directory:

   # Discover structure
   tree -L 3 --dirsfirst [target_dir]
   ls -la [target_dir]

1. Identify relevant files and classify by priority:

1. Build the File-Map — a mental inventory of what exists and what's missing.
2. Compose eval set: Merge user-provided evals with auto-detected evals (see Default Evals below).

Phase 2: Cross-File Analysis

Run consistency checks across files. Each check = one eval point:

Result of Phase 2: A complete eval checklist with per-file and cross-file checks, each scored Yes/No.

Phase 3: Iterative Fix Loop

Same loop logic as prompt/code/audit — TSV, report.sh, stop conditions. Key differences:

• Multiple files can be changed per iteration
• Pass rate = aggregated over ALL evals (file-specific + cross-file)
• Fixes are minimal and surgical — don't refactor blindly, only fix what improves pass rate
• `change_description` includes which files were touched: "Fix Dockerfile + CI workflow sync"

Per Iteration: What you do

1. Evaluate current repo state against all evals (file-specific + cross-file)
2. Calculate pass rate: (passing evals / total evals) × 100
3. Compare with best previous pass rate → determine status
4. On improved: apply minimal, surgical fixes to the fewest files necessary
5. Verify the fix didn't break other evals (re-run affected checks)
6. Write TSV row + call report.sh
7. Check stop conditions

Dry-Run vs Live

Default Evals (auto-applied unless overridden)

These evals are automatically used when the user doesn't provide custom evals. The agent detects which are applicable based on what exists in the repo:

Eval Scoring

Pass Rate = (Passing Evals / Total Applicable Evals) × 100

Evals that don't apply (e.g. "Dockerfile functional?" when no Dockerfile exists) are excluded from the total, not counted as passes.

At the End

• --dry-run: All proposed changes → results/[target]-proposed/ directory
• --live: Changes already applied, backups in results/backups/
• report.sh --final
• Optionally: results/[target]-project-details.md with per-file findings (NOT in TSV!)

Directory Structure

autoforge/
├── SKILL.md                    ← This file
├── results/
│   ├── [target]-results.tsv    ← TSV logs
│   ├── [target]-proposed.md    ← Proposed improvement (prompt/audit)
│   ├── [target]-proposed/      ← Proposed repo changes (project mode)
│   │   ├── README.md
│   │   ├── Dockerfile
│   │   └── ...
│   ├── [target]-v1.md          ← Deep audit final version
│   ├── [target]-audit-details.md ← Audit details (audit mode only)
│   ├── [target]-project-details.md ← Project details (project mode only)
│   └── backups/                ← Auto-backups (--live)
│       ├── [file].bak          ← Single file backups (prompt/code/audit)
│       └── [target]-backup/    ← Full directory backup (project mode)
├── scripts/
│   ├── report.sh               ← Channel reporting
│   └── visualize.py            ← PNG chart (optional)
├── references/
│   ├── eval-examples.md        ← Pre-built evals
│   └── ml-mode.md              ← ML training guide
└── examples/
    ├── demo-results.tsv        ← Demo data
    └── example-config.json     ← Example configuration

Examples (task descriptions, NOT CLI commands)

AutoForge is not a CLI tool — it's a skill prompt for the agent:

# Optimize a prompt
"Start autoforge mode: prompt for the coding-agent skill.
 Evals: PTY correct? Workspace protected? Clearly structured?"

# Audit a CLI skill (simple)
"Start autoforge mode: audit for notebooklm-py."

# Deep audit with multi-model
"Start autoforge mode: audit (deep) for subagents docs.
 Optimizer: Opus, Validator: GPT-5
 Extract ground truth from source, validate iteratively."

# Optimize code
"Start autoforge mode: code for backup.sh.
 File: ./backup.sh
 Test: bash backup.sh personal --dry-run
 Evals: exit_code==0, backup file created, < 10s runtime"

# Optimize a whole repository
"Start autoforge mode: project for ./my-app
 Evals: Tests green? CI correct? No hardcoded secrets? README accurate?"

# Project mode with custom focus
"Start autoforge mode: project for /path/to/api-server
 Focus: Docker + CI pipeline consistency
 Evals: docker build succeeds, CI workflow references correct paths,
        .env.example covers all env vars used in code"

# Project mode dry-run (default)
"Start autoforge mode: project for ./my-tool --dry-run
 Use default evals. Show me what needs fixing."

Eval Examples → Mode Mapping

references/eval-examples.md provides ready-to-use Yes/No evals grouped by category. Here's how they map to AutoForge modes:

Pick evals from the matching category and paste them into your task prompt as the eval set.

Tips

• Always start with --dry-run
• prompt = think, code = execute, audit = test CLI, project = optimize repo
• Simple Audit for clear CLI skills, Deep Audit for complex docs
• Project mode scans the whole repo — cross-file consistency is the killer feature
• Multi-Model for Deep Audits: different models cover different blind spots
• At >3 discards after all fixes: check for validator noise, declare convergence if justified
• TSV + report.sh are NOT optional — they are the user interface
• For ML training: see references/ml-mode.md