Autoresearch

An autonomous agent that finds improvements through measured experiments or deep research. Based on Karpathy's autoresearch pattern: separate what the human controls (strategy) from what the agent controls (execution), then let the agent iterate indefinitely with objective verification.

Choosing a Mode

When no mode is specified, infer from context: metric or benchmark mentioned → Optimize. Question or topic exploration → Research. Wants something "better" without a defined measure → Improve.

Mode 1: Optimize (Experiment Loop)

The core Karpathy pattern. A hill-climbing ratchet where only measurable improvements accumulate.

Step 1: Configure the Experiment

Before looping, establish four components. Ask the user to confirm if anything is ambiguous — but if the project structure makes the answers obvious, just proceed.

Read references/experiment-loop.md for auto-detection heuristics when the user doesn't specify these explicitly.

When to suggest classical tools instead: For pure numeric parameter sweeps (no code logic — YAML thresholds, hyperparameters), mention that Optuna or BOHB may converge faster. Autoresearch's edge is mutating arbitrary code and algorithms. Don't gate on this; just note it so the user can choose.

Step 2: Establish Baseline

1. Create a git branch: autoresearch/<descriptive-tag> from current HEAD
2. Read all mutable surface files to build full context
3. Run the verifier once unmodified to get the baseline metric
4. Record in results.tsv (see "Results Ledger" below for the canonical schema):

   commit	metric	delta	status	duration_s	description
   <hash>	<value>	0	baseline	<s>	Initial measurement

Step 3: The Loop

Run this loop autonomously without pausing for confirmation. The user may be asleep, at lunch, or doing other work — they will interrupt when they want it to stop. Note that Bash is pre-approved only for git * commands — the Step 2 baseline run doubles as the permission warm-up for the verifier command, so it gets approved while the user is still present, not mid-loop while they are away.

LOOP:
  1. HYPOTHESIZE: Read results.tsv, recent verifier output (errors, warnings,
     timing breakdowns — not just the scalar), and the mutable surface. Form
     one specific hypothesis with expected impact and rationale.

  2. MUTATE: Apply exactly ONE atomic change. Small reversible edit over large
     rewrite. Never bundle. Don't retry discarded ideas without a meaningfully
     different approach. ANNOTATE non-obvious values inline per "Provenance
     Comments" below.

  3. COMMIT: `git add <mutable files> && git commit -m "experiment: <description>"`

  4. RUN: Execute the verifier. Capture ALL output; retain ~200 lines for the
     next HYPOTHESIZE (warnings, profiling, timing are signal).
     - Trivial bug (typo, import): fix and retry once, else log "crash".
     - Duration >2x baseline: kill, log "timeout".

  5. MEASURE: Extract the metric from the output.

  6. DECIDE:
     - IMPROVED: Keep the commit as new baseline. Log "kept".
       **Anomaly check:** If delta >3x rolling average of kept deltas AND
       follows 3+ consecutive discards, flag: `⚠ ANOMALY: delta=X is Nx rolling
       avg after plateau — inspect for reward hacking.` Pause one iteration to
       reflect. Do NOT auto-discard — could be a breakthrough — but be suspicious.
     - EQUAL: Keep ONLY if simpler (fewer lines, simpler logic). Log
       "kept-simpler" or "discarded-no-gain".
     - REGRESSED: `git revert HEAD --no-edit` (preserves history). Log "discarded".

  7. LOG: Append to results.tsv (commit, metric, delta, status, duration_s, description).

  8. STATUS: Print `[iteration N] metric=X delta=Y status=Z`

  9. REFLECT (every 5): Re-read results.tsv. Categorize experiments (hyperparameter,
     algorithmic, structural, config). If last 5 are same category, force a
     different category next. Print `[reflect] N kept from <cat>, pivoting to <new>`.

  10. GOTO 1

Stopping Conditions

At 5 consecutive discards (plateau — likely a local maximum), do NOT stop yet: apply the escape strategies in references/experiment-loop.md §"Local Maxima" and pivot to a different hypothesis category.

Stop the loop when ANY of these are true:

• Ceiling mapped: 8+ consecutive discards spanning at least 3 different hypothesis

categories. This is not a failure — it means the optimization space has been explored and the system is near its ceiling. Report it as a positive finding: ✓ Optimization ceiling mapped at <metric>=<value>. Tried <N> experiments across <categories>. The system is near-optimal for the current architecture/approach. Further gains likely require a fundamentally different strategy.

• Target reached: The user specified a target metric and the loop reaches it
• User interrupt: The user sends any message
• Iteration cap: 20 iterations by default (user can override with --max N)

When stopping, print a summary table of all experiments and the cumulative improvement.

The Simplicity Criterion

Prefer deletions. A change that removes code for equal-or-better metric is always worth keeping; a small gain that adds ugly complexity is not. The git history should read as a clean sequence of wins, not a pile of hacks.

Mode 2: Research (Deep Multi-Agent Research)

Recursive depth+breadth research with parallel agents. Produces a comprehensive, source-grounded report.

Step 1: Decompose the Question

Break the user's question into 3-6 independent research angles. Use the STORM multi-perspective pattern — don't just split by subtopic, split by viewpoint:

• What would a practitioner want to know?
• What would a skeptic question?
• What does the academic literature say?
• What are the competing approaches?
• What are the failure modes and edge cases?

Step 2: Dispatch Parallel Research Agents

For each angle, spawn a subagent using the Research Agent Prompt Template in references/deep-research.md. Each agent returns structured LEARNINGS, CONTRADICTIONS, FOLLOW_UPS, SOURCES, and a CONFIDENCE rating.

Step 3: Synthesize and Recurse

After all agents return:

1. Merge learnings — deduplicate, resolve contradictions, note confidence levels
2. Identify gaps — what follow-up questions are most important?
3. Recurse if needed — for the top 2-3 follow-up questions, dispatch another round

of agents. Reduce breadth by half each level. Default depth: 2 levels. Configurable with --depth N and --breadth N.

1. Synthesize — produce a structured report with: Executive Summary, Key Findings

(by theme, not by source), Competing Perspectives, Gaps/Uncertainties, and Sources. Read references/deep-research.md for report templates, agent prompt templates, and synthesis patterns.

1. Save — write the final report to results/<topic>-research-<date>.md. This

file serves as the provenance record. Code changes informed by this research should reference it in comments (see "Provenance Comments" in Mode 1).

Depth Control

The user can specify: /autoresearch research --depth deep "topic"

Mode 3: Improve (Research-then-Optimize)

For when the user wants something better but doesn't yet know what "better" looks like. This mode runs Research first to discover best practices, then Optimize to apply them.

Phase 1: Research

Identify what the user wants to improve (code, config, prompt, workflow), then run Mode 2 targeting: best practices for this type of artifact, common performance pitfalls, what the state of the art looks like, and specific techniques that have worked for others. Present the findings to the user as a brief summary (not the full report) and propose a metric + verifier grounded in them.

If the metric is subjective (quality scores, "is it better?", LLM-as-judge), recommend converting to 3-5 binary pass/fail assertions instead. Binary evals (e.g., "Does the output contain X?", "Is the response under N tokens?", "Does it compile?") resist drift and enable truly autonomous operation. Fuzzy 1-5 rubrics cause the agent to score itself leniently over time. A test either passes or doesn't.

Phase 2: Optimize

Present the proposed experiment configuration to the user — truth layer, mutable surface, verifier command, metric + direction, and the top 5 hypotheses ranked by expected impact from the research — then let them confirm or override and enter the Mode 1 loop. Order hypotheses research-informed first, speculative later. When keeping changes informed by the research phase, include provenance comments that reference the research file (e.g., See results/<topic>-research-<date>.md).

The research phase turns blind exploration into targeted experimentation.

Operational Details

Git as State Machine

Always work on branch autoresearch/<tag>, never on main/master. Never force push. The branch tip is always the best-known version — commit on keep, git revert HEAD --no-edit on discard. If not in a git repo, keep a copy of the last-known-good version of the mutable surface and restore it on discard instead.

Results Ledger

Track all experiments in results.tsv (append-only) at the project root:

commit	metric	delta	status	duration_s	description
abc1234	0.9979	0.0000	baseline	301	Initial measurement
def5678	0.9952	-0.0027	kept	298	Increased depth from 8 to 12

Read this before each hypothesis to avoid repeating failed ideas.

Provenance Comments

Leave inline comments on non-obvious experimentally-derived values so future readers don't have to reconstruct the reasoning from git blame or chat history. Include: the autoresearch: prefix, before→after metric, why it works, and a pointer to results.tsv or the research report. Skip obvious defaults and self-explanatory diffs.

# autoresearch: batch_size=384 outperformed 128/256/512 (throughput 1.8x baseline).
# Fits in L2 cache on target hardware. See results.tsv for full sweep.
BATCH_SIZE = 384

When Mode 2/3 research informed a choice, reference the report file instead.

End-of-Session Summary Comment

When the optimize loop stops, add a block comment at the top of the primary mutable file: session branch/date, metric baseline→final, iteration count (kept/discarded), key changes that moved the needle, and a pointer to results.tsv. Append below any previous session comments — don't replace them.

Crash Handling

Don't get stuck — if an experiment fails, extract signal and move on:

• Trivial bug (typo, import): fix and retry once, then discard
• Runtime crash: apply obvious fix or log as "crash" and move on
• Timeout (>2x baseline): kill, discard, log as "timeout"
• Flaky results: run verifier twice and average; note variance >5%

Blind Validation (Subjective Metrics)

Skip for objective metrics (latency, bytes, pass rate) — the number is the number.

For subjective metrics (LLM-as-judge, rubric scores, design ratings), the agent that proposed a change is biased toward keeping it. Counter by spawning a blind evaluator subagent — once on a baseline snapshot (background), once on the final version — and comparing Self / Agent / Gap per component. A gap ≥2 flags that component for the next hypothesis; the blind score surfaces bias, it never overrides the self-score.

See references/experiment-loop.md (Blind Validation Protocol) for when to spawn, the agent prompt template, and the comparison-table format.

Additional Resources

References

• references/experiment-loop.md — Auto-detection heuristics, advanced loop

mechanics, timeout policies, common pitfalls, and the Blind Validation Protocol (agent prompt template + comparison-table format for subjective metrics)

• references/deep-research.md — Full research agent prompt templates, structured

extraction schemas, synthesis patterns, and source quality assessment

• references/domain-templates.md — Pre-built experiment configurations for web

perf, ML training, prompt optimization, test coverage, bundle size, API latency

• references/ecosystem.md — Prior art: canonical repos, tree search / evolutionary

/ meta-agent alternatives, Claude Code implementations, reward hacking case studies

• references/sources.md — Dated per-URL index backing ecosystem.md; freshen passes

stamp Last verified: fields here

• references/improvement-backlog.md — Ceiling findings carried across skill-improver

passes; not needed at invocation time

Example Reports

• results/autoresearch-evolution-research-2026-04-06.md — Mode 2 output: how the

autoresearch ecosystem has evolved since Karpathy's original release