---
name: paper-code-audit
description: Compare a paper's claims against its public codebase. Use when the user asks to audit a paper, check code-claim consistency, verify reproducibility of a specific paper, or find mismatches between a paper and its implementation.
---

# Paper-Code Audit

Compare a paper's claimed methods, defaults, metrics, and data handling against the actual code. Surface mismatches, omissions, and reproduction risks.

Derive a slug from the audit target (lowercase, hyphens, ≤5 words).

## Subagent mapping

See `../_platform-mapping.md`. Dispatch `researcher` for evidence gathering, `verifier` for citation/URL verification.

## Workflow

### 1. Plan

Write `outputs/.plans/<slug>.md` with:

- **Paper** — title, arXiv id or DOI, and the specific version being audited
- **Repo** — canonical URL (be explicit about fork vs upstream)
- **Claims to check** — numbered list of specific claims (e.g. "claim 3: final layer uses GELU activation")
- **Verification approach** — per-claim, how will you check it (grep source, run a specific script, diff configs)

Summarize the plan briefly, continue immediately unless the user asked for plan review.

### 2. Gather evidence

**Non-trivial audits:** dispatch a `researcher` subagent to pull implementation details from paper sections and linked code. Use the `alpha-research` skill's `alpha code` command (or equivalent repo browsing) to read source files.

**Small audits (single claim, ≤3 files):** the lead agent gathers directly.

For each claim, record:

- The paper section or figure where the claim is made
- The code location (file:line or function name) where it should be implemented
- What the code actually does

### 3. Compare

Organize findings under these buckets:

| Bucket | Meaning |
|---|---|
| **MATCH** | Code matches the claim faithfully |
| **MISMATCH** | Code contradicts the paper |
| **OMITTED** | Claim is in paper but code doesn't implement it |
| **UNDOCUMENTED** | Code does something material that isn't in the paper |
| **AMBIGUOUS** | Paper's description is too vague to verify against code |
| **MISSING CODE** | The referenced module/experiment is not in the public repo |

### 4. Cite

For non-trivial audits, dispatch `verifier` against the draft to verify every URL (paper links, repo links, commit hashes) and add inline citations where missing.

### 5. Deliver

Save exactly one audit artifact to `outputs/<slug>-audit.md`:

```markdown
# Audit: <paper title>

**Paper:** <link> (<version/date>)
**Repo:** <link> (<commit hash used for audit>)
**Date:** YYYY-MM-DD

## Summary
- Claims checked: <N>
- MATCH: <n> | MISMATCH: <n> | OMITTED: <n> | UNDOCUMENTED: <n> | AMBIGUOUS: <n> | MISSING CODE: <n>

## Findings

### <claim 1>
- **Paper says:** <quote or summary> (<section>)
- **Code does:** <what you found> (<file:line>)
- **Verdict:** MATCH / MISMATCH / OMITTED / …
- **Impact on reproducibility:** <brief>

### <claim 2>
...

## Reproduction risks
- <risks ordered by severity>

## Sources
- <paper URL>
- <repo URL at audit commit>
```

End with a `Sources` section containing paper and repository URLs pinned to the version audited (commit hash, not `main`).

## What NOT to do

- Don't run the code unless the user explicitly asked for an execution audit. Reading is often enough to find the mismatch.
- Don't generalize from `src/models/transformer.py` to "the method" without checking the experiment scripts actually call it.
- Don't grade papers. The audit reports what is and isn't in the code; it doesn't pass judgement on the research.
