---
<<<<<<< HEAD
phase: 14-osint_ci_cd_logs_web_archives_frontend_leaks
plan: 02
type: execute
wave: 1
depends_on: []
files_modified:
  - pkg/recon/sources/wayback.go
  - pkg/recon/sources/wayback_test.go
  - pkg/recon/sources/commoncrawl.go
  - pkg/recon/sources/commoncrawl_test.go
autonomous: true
requirements:
  - RECON-ARCH-01
  - RECON-ARCH-02

must_haves:
  truths:
    - "Wayback Machine CDX API queries find historical snapshots containing provider keywords"
    - "CommonCrawl index search finds pages matching provider keywords and scans WARC content"
  artifacts:
    - path: "pkg/recon/sources/wayback.go"
      provides: "WaybackSource implementing ReconSource"
      contains: "func (s *WaybackSource) Sweep"
    - path: "pkg/recon/sources/commoncrawl.go"
      provides: "CommonCrawlSource implementing ReconSource"
      contains: "func (s *CommonCrawlSource) Sweep"
  key_links:
    - from: "pkg/recon/sources/wayback.go"
      to: "pkg/recon/source.go"
      via: "implements ReconSource interface"
      pattern: "var _ recon\\.ReconSource"
    - from: "pkg/recon/sources/commoncrawl.go"
      to: "pkg/recon/source.go"
      via: "implements ReconSource interface"
      pattern: "var _ recon\\.ReconSource"
---

<objective>
Implement two web archive scanning sources: WaybackSource (Wayback Machine CDX API) and CommonCrawlSource (CommonCrawl index API). Both search historical web snapshots for leaked API keys.

Purpose: Web archives preserve historical versions of pages that may have since been scrubbed. Keys accidentally exposed in config files, JavaScript, or API documentation may persist in archive snapshots even after removal from the live site.

Output: 2 source files + 2 test files in pkg/recon/sources/
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@pkg/recon/source.go
@pkg/recon/sources/httpclient.go
@pkg/recon/sources/queries.go
@pkg/recon/sources/npm.go
@pkg/recon/sources/npm_test.go

<interfaces>
From pkg/recon/source.go:
```go
type Finding = engine.Finding
type ReconSource interface {
    Name() string
    RateLimit() rate.Limit
    Burst() int
    RespectsRobots() bool
    Enabled(cfg Config) bool
    Sweep(ctx context.Context, query string, out chan<- Finding) error
}
```

From pkg/recon/sources/httpclient.go:
```go
type Client struct { HTTP *http.Client; MaxRetries int; UserAgent string }
func NewClient() *Client
func (c *Client) Do(ctx context.Context, req *http.Request) (*http.Response, error)
```

From pkg/recon/sources/queries.go:
```go
func BuildQueries(reg *providers.Registry, source string) []string
```
</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Implement WaybackSource with tests</name>
  <files>pkg/recon/sources/wayback.go, pkg/recon/sources/wayback_test.go</files>
  <action>
Create WaybackSource (RECON-ARCH-01):
- Struct fields: BaseURL string, Registry *providers.Registry, Limiters *recon.LimiterRegistry, Client *Client
- Name() returns "wayback"
- RateLimit: rate.Every(5*time.Second), Burst: 1 (Wayback CDX API is rate-sensitive)
- RespectsRobots: true (web archive, respect their robots.txt)
- Enabled: always true (credentialless, public CDX API)
- Sweep: For each query from BuildQueries(registry, "wayback"):
  1. Query CDX API: GET http://web.archive.org/cdx/search/cdx?url=*.{domain}/*&output=json&fl=timestamp,original,statuscode&filter=statuscode:200&limit=10&matchType=domain where domain is derived from the query keyword (e.g., "api.openai.com" for OpenAI keywords). For generic keywords like "sk-proj-", use the CDX full-text search approach: GET http://web.archive.org/cdx/search/cdx?url=*&output=json&fl=timestamp,original&limit=10 with the keyword in the URL pattern.
  2. For each CDX result, the snapshot URL is: https://web.archive.org/web/{timestamp}/{original_url}
  3. Emit findings with Source set to the snapshot URL and SourceType "recon:wayback"
  4. Do NOT fetch the actual archived page content (that would be too slow and bandwidth-heavy). Instead, emit the CDX match as a lead for further investigation.
- BaseURL defaults to "http://web.archive.org" if empty (allows test injection).
- Compile-time interface check: var _ recon.ReconSource = (*WaybackSource)(nil)

Test: httptest.NewServer returning CDX JSON fixture (array-of-arrays format: [["timestamp","original","statuscode"],["20240101120000","https://example.com/config.js","200"]]). Verify Sweep emits findings with correct snapshot URLs. Test context cancellation. Test empty CDX response produces no findings.
  </action>
  <verify>
    <automated>cd /home/salva/Documents/apikey && go test ./pkg/recon/sources/ -run "TestWayback" -count=1 -v</automated>
  </verify>
  <done>WaybackSource implements ReconSource, queries CDX API via mock, emits findings with archive snapshot URLs, all tests pass</done>
</task>

<task type="auto">
  <name>Task 2: Implement CommonCrawlSource with tests</name>
  <files>pkg/recon/sources/commoncrawl.go, pkg/recon/sources/commoncrawl_test.go</files>
  <action>
Create CommonCrawlSource (RECON-ARCH-02):
- Struct fields: BaseURL string, Registry *providers.Registry, Limiters *recon.LimiterRegistry, Client *Client
- Name() returns "commoncrawl"
- RateLimit: rate.Every(5*time.Second), Burst: 1 (CommonCrawl index is rate-sensitive)
- RespectsRobots: false (API-based index query, not scraping)
- Enabled: always true (credentialless, public index API)
- Sweep: For each query from BuildQueries(registry, "commoncrawl"):
  1. Query CommonCrawl Index API: GET https://index.commoncrawl.org/CC-MAIN-2024-10-index?url=*.{domain}/*&output=json&limit=10 where CC-MAIN-2024-10 is the latest available index (hardcode a recent crawl ID; can be updated later). For keyword-based queries, use the URL pattern matching.
  2. CommonCrawl index returns NDJSON (one JSON object per line), each with fields: url, timestamp, filename, offset, length.
  3. Emit findings with Source set to the matched URL and SourceType "recon:commoncrawl". Include the WARC filename in the finding metadata for follow-up retrieval.
  4. Do NOT fetch actual WARC records (too large). Emit index matches as leads.
- BaseURL defaults to "https://index.commoncrawl.org" if empty.
- Use a CrawlID field (default "CC-MAIN-2024-10") to allow specifying which crawl index to search.
- Compile-time interface check: var _ recon.ReconSource = (*CommonCrawlSource)(nil)

Test: httptest.NewServer returning NDJSON fixture (one JSON object per line with url, timestamp, filename fields). Verify Sweep emits findings. Test empty response. Test context cancellation. Test malformed NDJSON lines are skipped gracefully.
  </action>
  <verify>
    <automated>cd /home/salva/Documents/apikey && go test ./pkg/recon/sources/ -run "TestCommonCrawl" -count=1 -v</automated>
  </verify>
  <done>CommonCrawlSource implements ReconSource, queries index API via mock, emits findings from NDJSON results, all tests pass</done>
</task>

</tasks>

<verification>
cd /home/salva/Documents/apikey && go test ./pkg/recon/sources/ -run "TestWayback|TestCommonCrawl" -count=1 -v
cd /home/salva/Documents/apikey && go vet ./pkg/recon/sources/
</verification>

<success_criteria>
- 2 new source files compile and implement ReconSource (var _ check)
- 2 test files pass with httptest mocks
- Both sources use BuildQueries + Client + LimiterRegistry pattern
- Both are credentialless (always enabled)
- WaybackSource constructs proper CDX snapshot URLs
- CommonCrawlSource parses NDJSON line-by-line
</success_criteria>

<output>
After completion, create `.planning/phases/14-osint_ci_cd_logs_web_archives_frontend_leaks/14-02-SUMMARY.md`
</output>
=======
phase: "14"
plan: "02"
type: feature
autonomous: true
wave: 1
depends_on: []
requirements: [RECON-ARCH-01, RECON-ARCH-02]
---

# Plan 14-02: Wayback Machine + CommonCrawl Sources

## Objective
Implement WaybackMachineSource and CommonCrawlSource as ReconSource modules for searching historical web snapshots for leaked API keys.

## Context
- @pkg/recon/source.go — ReconSource interface
- @pkg/recon/sources/httpclient.go — shared retry Client
- @pkg/recon/sources/register.go — RegisterAll wiring
- @pkg/recon/sources/queries.go — BuildQueries helper

## Tasks

### Task 1: Implement WaybackMachineSource and CommonCrawlSource
type="auto"

Implement two new ReconSource modules:

1. **WaybackMachineSource** (`pkg/recon/sources/wayback.go`):
   - Queries the Wayback Machine CDX API (`web.archive.org/cdx/search/cdx`) for historical snapshots
   - Uses provider keywords to search for pages containing API key patterns
   - Credentialless, always Enabled
   - Rate limit: 1 req/5s (conservative for public API)
   - RespectsRobots: true (web archive, HTML scraper)
   - Emits Finding per snapshot URL with SourceType=recon:wayback

2. **CommonCrawlSource** (`pkg/recon/sources/commoncrawl.go`):
   - Queries CommonCrawl Index API (`index.commoncrawl.org`) for matching pages
   - Uses provider keywords to search the CC index
   - Credentialless, always Enabled
   - Rate limit: 1 req/5s (conservative for public API)
   - RespectsRobots: true
   - Emits Finding per indexed URL with SourceType=recon:commoncrawl

3. **Tests** for both sources using httptest stubs following the established pattern.

4. **Wire into RegisterAll** and update register_test.go to expect 42 sources.

Done criteria:
- Both sources implement recon.ReconSource
- Tests pass with httptest stubs
- RegisterAll includes both sources
- `go test ./pkg/recon/sources/...` passes

## Verification
```bash
go test ./pkg/recon/sources/... -run "Wayback|CommonCrawl|RegisterAll" -v
```

## Success Criteria
- WaybackMachineSource queries CDX API and emits findings
- CommonCrawlSource queries CC Index API and emits findings
- Both wired into RegisterAll (42 total sources)
- All tests pass
>>>>>>> worktree-agent-a1113d5a
