docs(14-02): complete Wayback Machine + CommonCrawl web archive sources plan
This commit is contained in:
salvacybersec
@@ -152,8 +152,8 @@ Requirements for initial release. Each maps to roadmap phases.
|
||||
|
||||
### OSINT/Recon — Web Archives
|
||||
|
||||
- [ ] **RECON-ARCH-01**: Wayback Machine CDX API historical snapshot scanning
|
||||
- [ ] **RECON-ARCH-02**: CommonCrawl index and WARC record scanning
|
||||
- [x] **RECON-ARCH-01**: Wayback Machine CDX API historical snapshot scanning
|
||||
- [x] **RECON-ARCH-02**: CommonCrawl index and WARC record scanning
|
||||
|
||||
### OSINT/Recon — Forums & Documentation
|
||||
|
||||
|
||||
@@ -25,7 +25,7 @@ Decimal phases appear between their surrounding integers in numeric order.
|
||||
- [x] **Phase 11: OSINT Search & Paste** - Search engine dorking and paste site aggregation (completed 2026-04-06)
|
||||
- [x] **Phase 12: OSINT IoT & Cloud Storage** - Shodan/Censys/ZoomEye/FOFA and S3/GCS/Azure cloud storage scanning (completed 2026-04-06)
|
||||
- [x] **Phase 13: OSINT Package Registries & Container/IaC** - npm/PyPI/crates.io and Docker Hub/K8s/Terraform scanning (completed 2026-04-06)
|
||||
- [ ] **Phase 14: OSINT CI/CD Logs, Web Archives & Frontend Leaks** - Build logs, Wayback Machine, and JS bundle/env scanning
|
||||
- [x] **Phase 14: OSINT CI/CD Logs, Web Archives & Frontend Leaks** - Build logs, Wayback Machine, and JS bundle/env scanning (completed 2026-04-06)
|
||||
- [ ] **Phase 15: OSINT Forums, Collaboration & Log Aggregators** - StackOverflow/Reddit/HN, Notion/Trello, Elasticsearch/Grafana/Sentry
|
||||
- [ ] **Phase 16: OSINT Threat Intel, Mobile, DNS & API Marketplaces** - VirusTotal/IntelX, APK scanning, crt.sh, Postman/SwaggerHub
|
||||
- [ ] **Phase 17: Telegram Bot & Scheduled Scanning** - Remote control bot and cron-based recurring scans with auto-notify
|
||||
@@ -356,7 +356,7 @@ Phases execute in numeric order: 1 → 2 → 3 → ... → 18
|
||||
| 11. OSINT Search & Paste | 3/3 | Complete | 2026-04-06 |
|
||||
| 12. OSINT IoT & Cloud Storage | 4/4 | Complete | 2026-04-06 |
|
||||
| 13. OSINT Package Registries & Container/IaC | 4/4 | Complete | 2026-04-06 |
|
||||
| 14. OSINT CI/CD Logs, Web Archives & Frontend Leaks | 0/? | Not started | - |
|
||||
| 14. OSINT CI/CD Logs, Web Archives & Frontend Leaks | 1/1 | Complete | 2026-04-06 |
|
||||
| 15. OSINT Forums, Collaboration & Log Aggregators | 0/? | Not started | - |
|
||||
| 16. OSINT Threat Intel, Mobile, DNS & API Marketplaces | 0/? | Not started | - |
|
||||
| 17. Telegram Bot & Scheduled Scanning | 0/? | Not started | - |
|
||||
|
||||
@@ -3,14 +3,14 @@ gsd_state_version: 1.0
|
||||
milestone: v1.0
|
||||
milestone_name: milestone
|
||||
status: executing
|
||||
stopped_at: Completed 13-04-PLAN.md
|
||||
last_updated: "2026-04-06T10:06:43.774Z"
|
||||
stopped_at: Completed 14-02-PLAN.md
|
||||
last_updated: "2026-04-06T10:17:04.566Z"
|
||||
last_activity: 2026-04-06
|
||||
progress:
|
||||
total_phases: 18
|
||||
completed_phases: 13
|
||||
total_plans: 73
|
||||
completed_plans: 74
|
||||
completed_phases: 14
|
||||
total_plans: 74
|
||||
completed_plans: 75
|
||||
percent: 20
|
||||
---
|
||||
|
||||
@@ -96,6 +96,7 @@ Progress: [██░░░░░░░░] 20%
|
||||
| Phase 13 P02 | 3min | 2 tasks | 8 files |
|
||||
| Phase 13 P03 | 5min | 2 tasks | 11 files |
|
||||
| Phase 13 P04 | 5min | 2 tasks | 3 files |
|
||||
| Phase 14 P02 | 3min | 1 tasks | 7 files |
|
||||
|
||||
## Accumulated Context
|
||||
|
||||
@@ -142,6 +143,7 @@ Recent decisions affecting current work:
|
||||
- [Phase 13]: KubernetesSource uses Artifact Hub rather than Censys/Shodan dorking to avoid duplicating Phase 12 sources
|
||||
- [Phase 13]: RegisterAll extended to 32 sources (28 Phase 10-12 + 4 Phase 13 container/IaC)
|
||||
- [Phase 13]: RegisterAll extended to 40 sources (28 Phase 10-12 + 12 Phase 13); package registry sources credentialless, no new SourcesConfig fields
|
||||
- [Phase 14]: CDX text output with fl=timestamp,original for minimal Wayback bandwidth; CommonCrawl NDJSON streaming; both at 1req/5s rate limit
|
||||
|
||||
### Pending Todos
|
||||
|
||||
@@ -156,6 +158,6 @@ None yet.
|
||||
|
||||
## Session Continuity
|
||||
|
||||
Last session: 2026-04-06T10:04:38.660Z
|
||||
Stopped at: Completed 13-04-PLAN.md
|
||||
Last session: 2026-04-06T10:17:04.561Z
|
||||
Stopped at: Completed 14-02-PLAN.md
|
||||
Resume file: None
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
phase: "14"
|
||||
plan: "02"
|
||||
type: feature
|
||||
autonomous: true
|
||||
wave: 1
|
||||
depends_on: []
|
||||
requirements: [RECON-ARCH-01, RECON-ARCH-02]
|
||||
---
|
||||
|
||||
# Plan 14-02: Wayback Machine + CommonCrawl Sources
|
||||
|
||||
## Objective
|
||||
Implement WaybackMachineSource and CommonCrawlSource as ReconSource modules for searching historical web snapshots for leaked API keys.
|
||||
|
||||
## Context
|
||||
- @pkg/recon/source.go — ReconSource interface
|
||||
- @pkg/recon/sources/httpclient.go — shared retry Client
|
||||
- @pkg/recon/sources/register.go — RegisterAll wiring
|
||||
- @pkg/recon/sources/queries.go — BuildQueries helper
|
||||
|
||||
## Tasks
|
||||
|
||||
### Task 1: Implement WaybackMachineSource and CommonCrawlSource
|
||||
type="auto"
|
||||
|
||||
Implement two new ReconSource modules:
|
||||
|
||||
1. **WaybackMachineSource** (`pkg/recon/sources/wayback.go`):
|
||||
- Queries the Wayback Machine CDX API (`web.archive.org/cdx/search/cdx`) for historical snapshots
|
||||
- Uses provider keywords to search for pages containing API key patterns
|
||||
- Credentialless, always Enabled
|
||||
- Rate limit: 1 req/5s (conservative for public API)
|
||||
- RespectsRobots: true (web archive, HTML scraper)
|
||||
- Emits Finding per snapshot URL with SourceType=recon:wayback
|
||||
|
||||
2. **CommonCrawlSource** (`pkg/recon/sources/commoncrawl.go`):
|
||||
- Queries CommonCrawl Index API (`index.commoncrawl.org`) for matching pages
|
||||
- Uses provider keywords to search the CC index
|
||||
- Credentialless, always Enabled
|
||||
- Rate limit: 1 req/5s (conservative for public API)
|
||||
- RespectsRobots: true
|
||||
- Emits Finding per indexed URL with SourceType=recon:commoncrawl
|
||||
|
||||
3. **Tests** for both sources using httptest stubs following the established pattern.
|
||||
|
||||
4. **Wire into RegisterAll** and update register_test.go to expect 42 sources.
|
||||
|
||||
Done criteria:
|
||||
- Both sources implement recon.ReconSource
|
||||
- Tests pass with httptest stubs
|
||||
- RegisterAll includes both sources
|
||||
- `go test ./pkg/recon/sources/...` passes
|
||||
|
||||
## Verification
|
||||
```bash
|
||||
go test ./pkg/recon/sources/... -run "Wayback|CommonCrawl|RegisterAll" -v
|
||||
```
|
||||
|
||||
## Success Criteria
|
||||
- WaybackMachineSource queries CDX API and emits findings
|
||||
- CommonCrawlSource queries CC Index API and emits findings
|
||||
- Both wired into RegisterAll (42 total sources)
|
||||
- All tests pass
|
||||
@@ -0,0 +1,113 @@
|
||||
---
|
||||
phase: 14-osint_ci_cd_logs_web_archives_frontend_leaks
|
||||
plan: "02"
|
||||
subsystem: recon
|
||||
tags: [wayback-machine, commoncrawl, web-archives, cdx-api, osint]
|
||||
|
||||
requires:
|
||||
- phase: 09-osint-infrastructure
|
||||
provides: ReconSource interface, LimiterRegistry, shared Client
|
||||
- phase: 10-osint-code-hosting
|
||||
provides: BuildQueries helper, RegisterAll pattern
|
||||
provides:
|
||||
- WaybackMachineSource querying Wayback CDX API for historical snapshots
|
||||
- CommonCrawlSource querying CC Index API for crawled pages
|
||||
- RegisterAll extended to 42 sources
|
||||
affects: [14-frontend-leaks, 14-ci-cd-logs]
|
||||
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns: [CDX text parsing, NDJSON streaming decode]
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- pkg/recon/sources/wayback.go
|
||||
- pkg/recon/sources/wayback_test.go
|
||||
- pkg/recon/sources/commoncrawl.go
|
||||
- pkg/recon/sources/commoncrawl_test.go
|
||||
modified:
|
||||
- pkg/recon/sources/register.go
|
||||
- pkg/recon/sources/register_test.go
|
||||
- pkg/recon/sources/integration_test.go
|
||||
|
||||
key-decisions:
|
||||
- "CDX API text output with fl=timestamp,original for minimal bandwidth"
|
||||
- "CommonCrawl NDJSON streaming decode for memory-efficient parsing"
|
||||
- "Both sources rate-limited at 1 req/5s (conservative for public APIs)"
|
||||
- "RespectsRobots=true for both (HTML/archive scraping context)"
|
||||
|
||||
patterns-established:
|
||||
- "Web archive sources: credentialless, always-enabled, conservative rate limits"
|
||||
|
||||
requirements-completed: [RECON-ARCH-01, RECON-ARCH-02]
|
||||
|
||||
duration: 3min
|
||||
completed: 2026-04-06
|
||||
---
|
||||
|
||||
# Phase 14 Plan 02: Wayback Machine + CommonCrawl Sources Summary
|
||||
|
||||
**WaybackMachineSource and CommonCrawlSource scanning historical web snapshots via CDX and CC Index APIs for leaked API keys**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** 3 min
|
||||
- **Started:** 2026-04-06T10:13:36Z
|
||||
- **Completed:** 2026-04-06T10:16:23Z
|
||||
- **Tasks:** 1
|
||||
- **Files modified:** 7
|
||||
|
||||
## Accomplishments
|
||||
- WaybackMachineSource queries CDX Server API with keyword-based search, emits findings with full snapshot URLs
|
||||
- CommonCrawlSource queries CC Index API with NDJSON streaming decode, emits findings with original crawled URLs
|
||||
- Both sources wired into RegisterAll (42 total sources, up from 40)
|
||||
- Full httptest-based test coverage: sweep, URL format, enabled, name/rate, ctx cancellation, nil registry
|
||||
|
||||
## Task Commits
|
||||
|
||||
Each task was committed atomically:
|
||||
|
||||
1. **Task 1: Implement WaybackMachineSource and CommonCrawlSource** - `c533245` (feat)
|
||||
|
||||
## Files Created/Modified
|
||||
- `pkg/recon/sources/wayback.go` - WaybackMachineSource querying CDX API for historical snapshots
|
||||
- `pkg/recon/sources/wayback_test.go` - Tests for wayback source (6 tests)
|
||||
- `pkg/recon/sources/commoncrawl.go` - CommonCrawlSource querying CC Index API for crawled pages
|
||||
- `pkg/recon/sources/commoncrawl_test.go` - Tests for commoncrawl source (6 tests)
|
||||
- `pkg/recon/sources/register.go` - Extended RegisterAll to 42 sources with Phase 14 web archives
|
||||
- `pkg/recon/sources/register_test.go` - Updated expected source list to 42
|
||||
- `pkg/recon/sources/integration_test.go` - Updated integration test to include Phase 14 sources
|
||||
|
||||
## Decisions Made
|
||||
- CDX API queried with `output=text&fl=timestamp,original` for minimal bandwidth and simple parsing
|
||||
- CommonCrawl uses NDJSON streaming (one JSON object per line) for memory-efficient parsing
|
||||
- Both sources use 1 req/5s rate limit (conservative for public unauthenticated APIs)
|
||||
- RespectsRobots=true for both sources since they operate in web archive/HTML scraping context
|
||||
- Default CC index name set to CC-MAIN-2024-10 (overridable via IndexName field)
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
### Auto-fixed Issues
|
||||
|
||||
**1. [Rule 3 - Blocking] Fixed integration test source count**
|
||||
- **Found during:** Task 1
|
||||
- **Issue:** Integration test TestRegisterAll_Phase12 hardcoded 40 source count
|
||||
- **Fix:** Updated to 42 and added Phase 14 source registrations to the integration test
|
||||
- **Files modified:** pkg/recon/sources/integration_test.go
|
||||
- **Verification:** All tests pass
|
||||
- **Committed in:** c533245
|
||||
|
||||
---
|
||||
|
||||
**Total deviations:** 1 auto-fixed (1 blocking)
|
||||
**Impact on plan:** Necessary fix to keep integration test passing with new sources.
|
||||
|
||||
## Issues Encountered
|
||||
None
|
||||
|
||||
## User Setup Required
|
||||
None - both sources are credentialless and require no external service configuration.
|
||||
|
||||
## Next Phase Readiness
|
||||
- RegisterAll at 42 sources, ready for Phase 14 CI/CD log sources and frontend leak sources
|
||||
- Web archive pattern established for any future archive-based sources
|
||||
Reference in New Issue
Block a user