---
name: replication
description: Plan or execute a replication of a paper, claim, or benchmark. Use when the user asks to replicate results, reproduce an experiment, verify a claim empirically, or build a replication package.
---

# Replication

Plan — and optionally execute — a replication of a paper, claim, or benchmark. Always confirm the execution environment with the user before running any code.

Derive a slug from the paper or claim (lowercase, hyphens, ≤5 words).

## Subagent mapping

See `../_platform-mapping.md`. Use `researcher` to extract implementation details from the paper and repo.

## Workflow

### 1. Extract

Dispatch a `researcher` subagent (or read directly for small papers) to pull implementation details from the target paper and any linked code:

- Algorithm / architecture specifics
- Hyperparameters, config defaults, random seeds
- Dataset — name, source, preprocessing
- Training regime — epochs, batch size, optimizer
- Metrics — exact definitions, evaluation splits
- Hardware — what they used, what you'll need

If `CHANGELOG.md` exists in the workspace, read the most recent relevant entries before planning or resuming.

### 2. Plan

Write `outputs/.plans/<slug>.md` with three explicit columns:

- **Verified** — details you confirmed from paper + code
- **Inferred** — details the paper/code implied but didn't state directly
- **Missing** — details the paper/code doesn't specify

Also include:

- **Check oracles** — the specific measurements that will decide whether the replication succeeded (e.g. "top-1 accuracy within ±0.5% of reported 78.2%")
- **Scope cut** — what's in (core claim) and what's out (ablations, follow-on experiments)

### 3. Environment — ask before running

Before executing anything, ask the user where to run:

- **Local** — current working directory
- **Virtual environment** — isolated venv/conda
- **Docker** — isolated container (see `docker` skill)
- **Modal** — serverless GPU (see `modal-compute` skill). Best for burst jobs without persistent state. Requires `modal` CLI.
- **RunPod** — persistent GPU pod with SSH (see `runpod-compute` skill). Best for long-running experiments. Requires `runpodctl` and `RUNPOD_API_KEY`.
- **Plan only** — produce the replication plan without executing

Do not install packages, run training, or execute experiments without an explicit answer.

### 4. Execute (if a runtime was chosen)

Implement and run the replication steps in the chosen environment. Save:

- **Scripts** — checked-in `.py` / `.sh` files in `experiments/<slug>/`
- **Configs** — exact configs used, checked in
- **Raw outputs** — logs, metrics, predictions, checkpoints (or at least checksums) in a reproducible layout
- **Results summary** — `outputs/<slug>-results.md` comparing your numbers to the paper's

Do not call the outcome "replicated" unless the planned oracles actually passed. If they didn't, write up what you observed and what diverged.

### 5. Log

For multi-step or resumable replication work, append concise entries to `CHANGELOG.md` after:

- Meaningful progress
- Failed attempts
- Major verification outcomes
- Before stopping for the session

Each entry: active objective, what changed, what was checked, next step.

### 6. Report

Save the final replication write-up to `outputs/<slug>-replication.md`:

```markdown
# Replication: <paper title>

**Paper:** <link> (<version>)
**Claim replicated:** <specific claim>
**Date:** YYYY-MM-DD
**Environment:** <chosen runtime>

## Oracles
- <oracle 1>: TARGET <x> — OBSERVED <y> — PASS / FAIL
- <oracle 2>: ...

## What matched
- <list>

## What diverged
- <list, with severity>

## Plausible causes
- <what could explain divergences>

## Reproducibility grade
- FULL | PARTIAL | FAILED | BLOCKED

## Sources
- <paper URL>
- <repo URL at commit used>
```

End with a `Sources` section containing paper and repository URLs pinned to the commit used for replication.

## Invariants

- **Confirm runtime before executing.** No silent installs or training runs.
- **Don't claim replication without oracle checks.**  "Numbers look close" isn't a check; "top-1 within ±0.5%" is.
- **Log failures.** A failed replication with a written-up reason is more valuable than a hand-wavy "seems to work".
