---
phase: 14-osint_ci_cd_logs_web_archives_frontend_leaks
plan: "02"
subsystem: recon
tags: [wayback-machine, commoncrawl, web-archives, cdx-api, osint]

requires:
  - phase: 09-osint-infrastructure
    provides: ReconSource interface, LimiterRegistry, shared Client
  - phase: 10-osint-code-hosting
    provides: BuildQueries helper, RegisterAll pattern
provides:
  - WaybackMachineSource querying Wayback CDX API for historical snapshots
  - CommonCrawlSource querying CC Index API for crawled pages
  - RegisterAll extended to 42 sources
affects: [14-frontend-leaks, 14-ci-cd-logs]

tech-stack:
  added: []
  patterns: [CDX text parsing, NDJSON streaming decode]

key-files:
  created:
    - pkg/recon/sources/wayback.go
    - pkg/recon/sources/wayback_test.go
    - pkg/recon/sources/commoncrawl.go
    - pkg/recon/sources/commoncrawl_test.go
  modified:
    - pkg/recon/sources/register.go
    - pkg/recon/sources/register_test.go
    - pkg/recon/sources/integration_test.go

key-decisions:
  - "CDX API text output with fl=timestamp,original for minimal bandwidth"
  - "CommonCrawl NDJSON streaming decode for memory-efficient parsing"
  - "Both sources rate-limited at 1 req/5s (conservative for public APIs)"
  - "RespectsRobots=true for both (HTML/archive scraping context)"

patterns-established:
  - "Web archive sources: credentialless, always-enabled, conservative rate limits"

requirements-completed: [RECON-ARCH-01, RECON-ARCH-02]

duration: 3min
completed: 2026-04-06
---

# Phase 14 Plan 02: Wayback Machine + CommonCrawl Sources Summary

**WaybackMachineSource and CommonCrawlSource scanning historical web snapshots via CDX and CC Index APIs for leaked API keys**

## Performance

- **Duration:** 3 min
- **Started:** 2026-04-06T10:13:36Z
- **Completed:** 2026-04-06T10:16:23Z
- **Tasks:** 1
- **Files modified:** 7

## Accomplishments
- WaybackMachineSource queries CDX Server API with keyword-based search, emits findings with full snapshot URLs
- CommonCrawlSource queries CC Index API with NDJSON streaming decode, emits findings with original crawled URLs
- Both sources wired into RegisterAll (42 total sources, up from 40)
- Full httptest-based test coverage: sweep, URL format, enabled, name/rate, ctx cancellation, nil registry

## Task Commits

Each task was committed atomically:

1. **Task 1: Implement WaybackMachineSource and CommonCrawlSource** - `c533245` (feat)

## Files Created/Modified
- `pkg/recon/sources/wayback.go` - WaybackMachineSource querying CDX API for historical snapshots
- `pkg/recon/sources/wayback_test.go` - Tests for wayback source (6 tests)
- `pkg/recon/sources/commoncrawl.go` - CommonCrawlSource querying CC Index API for crawled pages
- `pkg/recon/sources/commoncrawl_test.go` - Tests for commoncrawl source (6 tests)
- `pkg/recon/sources/register.go` - Extended RegisterAll to 42 sources with Phase 14 web archives
- `pkg/recon/sources/register_test.go` - Updated expected source list to 42
- `pkg/recon/sources/integration_test.go` - Updated integration test to include Phase 14 sources

## Decisions Made
- CDX API queried with `output=text&fl=timestamp,original` for minimal bandwidth and simple parsing
- CommonCrawl uses NDJSON streaming (one JSON object per line) for memory-efficient parsing
- Both sources use 1 req/5s rate limit (conservative for public unauthenticated APIs)
- RespectsRobots=true for both sources since they operate in web archive/HTML scraping context
- Default CC index name set to CC-MAIN-2024-10 (overridable via IndexName field)

## Deviations from Plan

### Auto-fixed Issues

**1. [Rule 3 - Blocking] Fixed integration test source count**
- **Found during:** Task 1
- **Issue:** Integration test TestRegisterAll_Phase12 hardcoded 40 source count
- **Fix:** Updated to 42 and added Phase 14 source registrations to the integration test
- **Files modified:** pkg/recon/sources/integration_test.go
- **Verification:** All tests pass
- **Committed in:** c533245

---

**Total deviations:** 1 auto-fixed (1 blocking)
**Impact on plan:** Necessary fix to keep integration test passing with new sources.

## Issues Encountered
None

## User Setup Required
None - both sources are credentialless and require no external service configuration.

## Next Phase Readiness
- RegisterAll at 42 sources, ready for Phase 14 CI/CD log sources and frontend leak sources
- Web archive pattern established for any future archive-based sources
