diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index a1c8a93..ff5647d 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -121,7 +121,7 @@ Requirements for initial release. Each maps to roadmap phases. ### OSINT/Recon — Paste Sites -- [ ] **RECON-PASTE-01**: Multi-paste aggregator (Pastebin, dpaste, paste.ee, rentry, hastebin, ix.io, etc.) +- [x] **RECON-PASTE-01**: Multi-paste aggregator (Pastebin, dpaste, paste.ee, rentry, hastebin, ix.io, etc.) ### OSINT/Recon — Package Registries @@ -302,7 +302,7 @@ Requirements for initial release. Each maps to roadmap phases. | RECON-CODE-01, RECON-CODE-02, RECON-CODE-03, RECON-CODE-04, RECON-CODE-05 | Phase 10 | Pending | | RECON-CODE-06, RECON-CODE-07, RECON-CODE-08, RECON-CODE-09, RECON-CODE-10 | Phase 10 | Pending | | RECON-DORK-01, RECON-DORK-02, RECON-DORK-03 | Phase 11 | Pending | -| RECON-PASTE-01 | Phase 11 | Pending | +| RECON-PASTE-01 | Phase 11 | Complete | | RECON-IOT-01, RECON-IOT-02, RECON-IOT-03, RECON-IOT-04, RECON-IOT-05, RECON-IOT-06 | Phase 12 | Pending | | RECON-CLOUD-01, RECON-CLOUD-02, RECON-CLOUD-03, RECON-CLOUD-04 | Phase 12 | Pending | | RECON-PKG-01, RECON-PKG-02, RECON-PKG-03 | Phase 13 | Pending | diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index e99c3bd..f91fffa 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -22,7 +22,7 @@ Decimal phases appear between their surrounding integers in numeric order. - [ ] **Phase 8: Dork Engine** - YAML-based dork definitions with 150+ built-in dorks and management commands - [ ] **Phase 9: OSINT Infrastructure** - Per-source rate limiter architecture and recon engine framework before any sources - [x] **Phase 10: OSINT Code Hosting** - GitHub, GitLab, Bitbucket, HuggingFace and 6 more code hosting sources (completed 2026-04-05) -- [ ] **Phase 11: OSINT Search & Paste** - Search engine dorking and paste site aggregation +- [x] **Phase 11: OSINT Search & Paste** - Search engine dorking and paste site aggregation (completed 2026-04-06) - [ ] **Phase 12: OSINT IoT & Cloud Storage** - Shodan/Censys/ZoomEye/FOFA and S3/GCS/Azure cloud storage scanning - [ ] **Phase 13: OSINT Package Registries & Container/IaC** - npm/PyPI/crates.io and Docker Hub/K8s/Terraform scanning - [ ] **Phase 14: OSINT CI/CD Logs, Web Archives & Frontend Leaks** - Build logs, Wayback Machine, and JS bundle/env scanning @@ -238,9 +238,9 @@ Plans: **Plans**: 3 plans Plans: -- [ ] 11-01-PLAN.md — GoogleDorkSource + BingDorkSource + DuckDuckGoSource + YandexSource + BraveSource (RECON-DORK-01, RECON-DORK-02, RECON-DORK-03) -- [ ] 11-02-PLAN.md — PastebinSource + GistPasteSource + PasteSitesSource multi-paste aggregator (RECON-PASTE-01) -- [ ] 11-03-PLAN.md — RegisterAll wiring + cmd/recon.go credentials + integration test (all Phase 11 reqs) +- [x] 11-01-PLAN.md — GoogleDorkSource + BingDorkSource + DuckDuckGoSource + YandexSource + BraveSource (RECON-DORK-01, RECON-DORK-02, RECON-DORK-03) +- [x] 11-02-PLAN.md — PastebinSource + GistPasteSource + PasteSitesSource multi-paste aggregator (RECON-PASTE-01) +- [x] 11-03-PLAN.md — RegisterAll wiring + cmd/recon.go credentials + integration test (all Phase 11 reqs) ### Phase 12: OSINT IoT & Cloud Storage **Goal**: Users can discover exposed LLM endpoints via IoT scanners (Shodan, Censys, ZoomEye, FOFA, Netlas, BinaryEdge) and scan publicly accessible cloud storage buckets (S3, GCS, Azure Blob, MinIO, GrayHatWarfare) for leaked keys @@ -342,7 +342,7 @@ Phases execute in numeric order: 1 → 2 → 3 → ... → 18 | 8. Dork Engine | 0/? | Not started | - | | 9. OSINT Infrastructure | 2/6 | In Progress| | | 10. OSINT Code Hosting | 9/9 | Complete | 2026-04-06 | -| 11. OSINT Search & Paste | 0/3 | Planning complete | - | +| 11. OSINT Search & Paste | 3/3 | Complete | 2026-04-06 | | 12. OSINT IoT & Cloud Storage | 0/? | Not started | - | | 13. OSINT Package Registries & Container/IaC | 0/? | Not started | - | | 14. OSINT CI/CD Logs, Web Archives & Frontend Leaks | 0/? | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index bf2e45e..7f8f6f5 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -2,15 +2,15 @@ gsd_state_version: 1.0 milestone: v1.0 milestone_name: milestone -status: executing -stopped_at: Completed 11-01-PLAN.md -last_updated: "2026-04-06T08:55:35.271Z" +status: completed +stopped_at: Completed 11-03-PLAN.md +last_updated: "2026-04-06T09:07:51.984Z" last_activity: 2026-04-06 progress: total_phases: 18 - completed_phases: 9 - total_plans: 57 - completed_plans: 64 + completed_phases: 10 + total_plans: 60 + completed_plans: 66 percent: 20 --- @@ -21,13 +21,13 @@ progress: See: .planning/PROJECT.md (updated 2026-04-04) **Core value:** Detect leaked LLM API keys across more providers and more internet sources than any other tool, with active verification to confirm keys are real and alive. -**Current focus:** Phase 10 — osint-code-hosting +**Current focus:** Phase 11 — osint-search-paste (complete) ## Current Position Phase: 11 -Plan: Not started -Status: Ready to execute +Plan: 3 of 3 complete +Status: Phase 11 complete Last activity: 2026-04-06 Progress: [██░░░░░░░░] 20% @@ -89,6 +89,7 @@ Progress: [██░░░░░░░░] 20% | Phase 10-osint-code-hosting P02 | 5min | 1 tasks | 2 files | | Phase 10-osint-code-hosting P07 | 6 | 2 tasks | 6 files | | Phase 10 P09 | 12min | 2 tasks | 5 files | +| Phase 11 P03 | 6min | 2 tasks | 4 files | | Phase 11 P01 | 3min | 2 tasks | 11 files | ## Accumulated Context @@ -127,6 +128,8 @@ Recent decisions affecting current work: - [Phase 10-osint-code-hosting]: github/gist use 'kw' in:file; all other sources use bare keyword - [Phase 10-osint-code-hosting]: GitHubSource reuses shared sources.Client + LimiterRegistry; builds queries from providers.Registry via BuildQueries; missing token disables (not errors) - [Phase 10]: RegisterAll registers all ten Phase 10 sources unconditionally; missing credentials flip Enabled()==false rather than hiding sources from the CLI catalog +- [Phase 11]: RegisterAll extended to 18 sources (10 Phase 10 + 8 Phase 11); paste sources use BaseURL prefix in integration test to avoid /search path collision +- [Phase 11]: Integration test uses injected test platforms for PasteSites (same pattern as SandboxesSource) - [Phase 11]: All five search sources use dork query format to focus on paste/code hosting leak sites ### Pending Todos @@ -142,6 +145,6 @@ None yet. ## Session Continuity -Last session: 2026-04-06T08:55:35.267Z -Stopped at: Completed 11-01-PLAN.md +Last session: 2026-04-06T09:07:51.980Z +Stopped at: Completed 11-03-PLAN.md Resume file: None diff --git a/.planning/phases/11-osint_search_paste/11-03-SUMMARY.md b/.planning/phases/11-osint_search_paste/11-03-SUMMARY.md new file mode 100644 index 0000000..991eac4 --- /dev/null +++ b/.planning/phases/11-osint_search_paste/11-03-SUMMARY.md @@ -0,0 +1,99 @@ +--- +phase: 11-osint-search-paste +plan: 03 +subsystem: recon +tags: [register-all, wiring, integration-test, credentials, search-engines, paste-sites] + +requires: + - phase: 11-osint-search-paste + provides: GoogleDorkSource, BingDorkSource, DuckDuckGoSource, YandexSource, BraveSource (Plan 01) + - phase: 11-osint-search-paste + provides: PastebinSource, GistPasteSource, PasteSitesSource (Plan 02) + - phase: 10-osint-code-hosting + provides: RegisterAll, SourcesConfig, buildReconEngine, 10 Phase 10 sources + +provides: + - RegisterAll extended to wire all 18 sources (Phase 10 + Phase 11) + - SourcesConfig with Google/Bing/Yandex/Brave credential fields + - cmd/recon.go credential wiring from env vars and viper config + - Integration test proving SweepAll across all 18 sources + +affects: [12-osint-iot-cloud-storage, recon-registration, recon-engine] + +tech-stack: + added: [] + patterns: [per-source BaseURL prefix in integration tests to avoid path collisions] + +key-files: + created: [] + modified: + - pkg/recon/sources/register.go + - pkg/recon/sources/register_test.go + - pkg/recon/sources/integration_test.go + - cmd/recon.go + +key-decisions: + - "Paste sources use BaseURL prefix (/pb/, /gp/) in integration test to avoid /search path collision with Replit/CodeSandbox" + - "PasteSites uses injected test platform in integration test, same pattern as SandboxesSource" + +patterns-established: + - "Integration test BaseURL prefix pattern for sources sharing HTTP paths" + +requirements-completed: [RECON-DORK-01, RECON-DORK-02, RECON-DORK-03, RECON-PASTE-01] + +duration: 6min +completed: 2026-04-06 +--- + +# Phase 11 Plan 03: RegisterAll Wiring + Integration Test Summary + +**RegisterAll extended to 18 sources with search engine credential wiring and full SweepAll integration test** + +## Performance + +- **Duration:** 6 min +- **Started:** 2026-04-06T09:00:51Z +- **Completed:** 2026-04-06T09:06:34Z +- **Tasks:** 2 +- **Files modified:** 4 + +## Accomplishments +- Extended SourcesConfig with 6 new credential fields (GoogleAPIKey, GoogleCX, BingAPIKey, YandexUser, YandexAPIKey, BraveAPIKey) +- RegisterAll now registers all 18 sources (10 Phase 10 + 8 Phase 11) unconditionally +- cmd/recon.go reads search engine API keys from env vars with viper config fallback +- Integration test proves SweepAll emits findings from all 18 SourceTypes via httptest fixtures + +## Task Commits + +Each task was committed atomically: + +1. **Task 1: Extend SourcesConfig + RegisterAll + cmd/recon.go credential wiring** - `3250408` (feat) +2. **Task 2: Integration test -- SweepAll across all 18 sources** - `bebc3e7` (test) + +## Files Created/Modified +- `pkg/recon/sources/register.go` - Extended SourcesConfig and RegisterAll with Phase 11 sources +- `pkg/recon/sources/register_test.go` - Guardrail tests updated to assert 18 sources +- `pkg/recon/sources/integration_test.go` - SweepAll integration test covering all 18 sources +- `cmd/recon.go` - Credential wiring for Google/Bing/Yandex/Brave API keys + +## Decisions Made +- Paste sources use BaseURL prefix in integration test to avoid /search path collision with existing Replit/CodeSandbox handlers +- PasteSites uses injected test platform (same pattern as SandboxesSource) rather than default production platforms + +## Deviations from Plan + +None - plan executed exactly as written. + +## Issues Encountered +None + +## User Setup Required +None - no external service configuration required. + +## Next Phase Readiness +- Phase 11 complete: all 18 OSINT sources (10 code-hosting + 5 search engine + 3 paste site) wired and tested +- Ready for Phase 12 (IoT/cloud storage sources) which will extend RegisterAll further + +--- +*Phase: 11-osint-search-paste* +*Completed: 2026-04-06*