---
name: autoresearch
description: Autonomous experiment loop that tries ideas, measures results, keeps what works, and discards what doesn't. Use when the user asks to optimize a metric, run an experiment loop, improve performance iteratively, or automate benchmarking.
allowed-tools: Bash(git:*), Bash(docker:*), Bash(modal:*), Bash(runpodctl:*)
---

# Autoresearch — Autonomous Optimization Loop

Run an iterative optimize-measure-commit loop against a user-chosen metric. The loop edits code, runs a benchmark, keeps commits that improve the metric, and reverts the rest.

> **Upstream note.** Feynman ships `pi-autoresearch` with the `init_experiment` / `run_experiment` / `log_experiment` tools. This skill documents the same loop as a generic procedure so it works in Claude Code and OpenCode without those tools. When `pi-autoresearch` is present, delegate to it; otherwise implement the loop with git + bash.

## Step 1 — Gather requirements

If `autoresearch.md` and `autoresearch.jsonl` already exist in the workspace, ask the user whether to **resume** or **start fresh**. If `CHANGELOG.md` exists, read the most recent relevant entries before resuming.

Otherwise, collect from the user before doing anything else:

- **What to optimize** — test speed, bundle size, training loss, build time, etc.
- **Benchmark command** — the exact shell command that produces the metric
- **Metric** — name, unit, and direction (lower-is-better or higher-is-better)
- **Files in scope** — which files the loop is allowed to modify
- **Max iterations** — default 20

## Step 2 — Pick an environment

Ask the user where to run iterations:

- **Local** — current working directory
- **New git branch** — create a branch so `main` stays clean
- **Virtual environment** — isolated venv/conda first
- **Docker** — run iterations inside a container (see `docker` skill)
- **Modal** — serverless GPU; stateless burst (see `modal-compute` skill)
- **RunPod** — persistent GPU pod with SSH (see `runpod-compute` skill)

Do not proceed without a clear answer.

## Step 3 — Confirm

Before starting the loop, present the full plan:

```
Optimization target: <metric> (<direction>)
Benchmark command:   <command>
Files in scope:      <files>
Environment:         <chosen environment>
Max iterations:      <N>
```

Wait for explicit approval. No silent starts.

## Step 4 — Run the loop

Initialize session files:

- `autoresearch.md` — human-readable running log (one section per iteration)
- `autoresearch.sh` — the benchmark command, committed so it's reproducible
- `autoresearch.jsonl` — one JSON record per iteration: `{iter, diff_ref, metric_value, kept, duration_s, notes}`

Run the **baseline** once and record the metric. This is iteration 0.

Then loop until `max_iterations` or user interruption:

1. **Propose a change** — pick one hypothesis (fewer deps, tighter loop, different algo, different config) based on what you've learned from prior iterations. State it in one sentence before editing.
2. **Edit** the files in scope.
3. **Commit** with a descriptive message (`autoresearch: iter N — <hypothesis>`).
4. **Run the benchmark** (`bash autoresearch.sh`), capture output and wall-clock time.
5. **Decide**:
   - If the metric improved (per direction): **keep** the commit.
   - Otherwise: `git revert` the commit or `git reset --hard HEAD~1` if still uncommitted-on-branch.
6. **Log** the iteration to `autoresearch.md` + `autoresearch.jsonl`.
7. After meaningful milestones, append a concise entry to `CHANGELOG.md` summarizing what changed, the metric movement, and the next hypothesis.

## Step 5 — When to stop

- `max_iterations` reached
- The metric plateaus for 3+ iterations
- The user interrupts
- You run out of clearly-motivated hypotheses (don't flail)

## Step 6 — Final report

When the loop stops, write a short summary: starting metric, ending metric, which hypotheses helped, which didn't, and what the next direction would be. Save to `outputs/<slug>-autoresearch-summary.md`.

## Subcommands (Feynman parity)

- `autoresearch <text>` — start or resume the loop
- `autoresearch off` — stop the loop, keep data
- `autoresearch clear` — delete all state and start fresh

## Key invariants

- **No silent rewrites.** Every iteration's metric movement must be traceable to a commit.
- **No invented results.** If the benchmark fails, log the failure as iteration data; don't pretend it succeeded.
- **No config drift.** The benchmark command must be stable across iterations — if it needs to change, that's a new session.
