diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 61093d6..dd6db5d 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -96,9 +96,9 @@ Requirements for initial release. Each maps to roadmap phases. - [x] **RECON-IOT-01**: Shodan API search and dorking - [x] **RECON-IOT-02**: Censys API search - [x] **RECON-IOT-03**: ZoomEye API search -- [ ] **RECON-IOT-04**: FOFA API search -- [ ] **RECON-IOT-05**: Netlas API search -- [ ] **RECON-IOT-06**: BinaryEdge API search +- [x] **RECON-IOT-04**: FOFA API search +- [x] **RECON-IOT-05**: Netlas API search +- [x] **RECON-IOT-06**: BinaryEdge API search ### OSINT/Recon — Code Hosting & Snippets diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index e8b6eb2..aaed487 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -23,7 +23,7 @@ Decimal phases appear between their surrounding integers in numeric order. - [ ] **Phase 9: OSINT Infrastructure** - Per-source rate limiter architecture and recon engine framework before any sources - [x] **Phase 10: OSINT Code Hosting** - GitHub, GitLab, Bitbucket, HuggingFace and 6 more code hosting sources (completed 2026-04-05) - [x] **Phase 11: OSINT Search & Paste** - Search engine dorking and paste site aggregation (completed 2026-04-06) -- [ ] **Phase 12: OSINT IoT & Cloud Storage** - Shodan/Censys/ZoomEye/FOFA and S3/GCS/Azure cloud storage scanning +- [x] **Phase 12: OSINT IoT & Cloud Storage** - Shodan/Censys/ZoomEye/FOFA and S3/GCS/Azure cloud storage scanning (completed 2026-04-06) - [ ] **Phase 13: OSINT Package Registries & Container/IaC** - npm/PyPI/crates.io and Docker Hub/K8s/Terraform scanning - [ ] **Phase 14: OSINT CI/CD Logs, Web Archives & Frontend Leaks** - Build logs, Wayback Machine, and JS bundle/env scanning - [ ] **Phase 15: OSINT Forums, Collaboration & Log Aggregators** - StackOverflow/Reddit/HN, Notion/Trello, Elasticsearch/Grafana/Sentry @@ -256,9 +256,9 @@ Plans: Plans: - [x] 12-01-PLAN.md — ShodanSource + CensysSource + ZoomEyeSource (RECON-IOT-01, RECON-IOT-02, RECON-IOT-03) -- [ ] 12-02-PLAN.md — FOFASource + NetlasSource + BinaryEdgeSource (RECON-IOT-04, RECON-IOT-05, RECON-IOT-06) +- [x] 12-02-PLAN.md — FOFASource + NetlasSource + BinaryEdgeSource (RECON-IOT-04, RECON-IOT-05, RECON-IOT-06) - [x] 12-03-PLAN.md — S3Scanner + GCSScanner + AzureBlobScanner + DOSpacesScanner (RECON-CLOUD-01, RECON-CLOUD-02, RECON-CLOUD-03, RECON-CLOUD-04) -- [ ] 12-04-PLAN.md — RegisterAll wiring + cmd/recon.go credentials + integration test (all Phase 12 reqs) +- [x] 12-04-PLAN.md — RegisterAll wiring + cmd/recon.go credentials + integration test (all Phase 12 reqs) ### Phase 13: OSINT Package Registries & Container/IaC **Goal**: Users can scan npm, PyPI, and 6 other package registries for packages containing leaked keys, and scan Docker Hub image layers, Kubernetes configs, Terraform state files, Helm charts, and Ansible Galaxy for secrets in infrastructure code @@ -349,7 +349,7 @@ Phases execute in numeric order: 1 → 2 → 3 → ... → 18 | 9. OSINT Infrastructure | 2/6 | In Progress| | | 10. OSINT Code Hosting | 9/9 | Complete | 2026-04-06 | | 11. OSINT Search & Paste | 3/3 | Complete | 2026-04-06 | -| 12. OSINT IoT & Cloud Storage | 1/4 | In Progress| | +| 12. OSINT IoT & Cloud Storage | 4/4 | Complete | 2026-04-06 | | 13. OSINT Package Registries & Container/IaC | 0/? | Not started | - | | 14. OSINT CI/CD Logs, Web Archives & Frontend Leaks | 0/? | Not started | - | | 15. OSINT Forums, Collaboration & Log Aggregators | 0/? | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index 3be47e6..d9a193e 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -3,8 +3,8 @@ gsd_state_version: 1.0 milestone: v1.0 milestone_name: milestone status: completed -stopped_at: Completed 12-01-PLAN.md -last_updated: "2026-04-06T09:25:12.004Z" +stopped_at: Completed 12-04-PLAN.md +last_updated: "2026-04-06T09:42:09.000Z" last_activity: 2026-04-06 progress: total_phases: 18 @@ -21,13 +21,13 @@ progress: See: .planning/PROJECT.md (updated 2026-04-04) **Core value:** Detect leaked LLM API keys across more providers and more internet sources than any other tool, with active verification to confirm keys are real and alive. -**Current focus:** Phase 11 — osint-search-paste (complete) +**Current focus:** Phase 12 — osint_iot_cloud_storage (in progress) ## Current Position Phase: 12 -Plan: Not started -Status: Phase 11 complete +Plan: 4 of 4 +Status: Plan 04 complete Last activity: 2026-04-06 Progress: [██░░░░░░░░] 20% @@ -92,6 +92,7 @@ Progress: [██░░░░░░░░] 20% | Phase 11 P03 | 6min | 2 tasks | 4 files | | Phase 11 P01 | 3min | 2 tasks | 11 files | | Phase 12 P01 | 3min | 2 tasks | 6 files | +| Phase 12 P04 | 14min | 2 tasks | 4 files | ## Accumulated Context @@ -133,6 +134,7 @@ Recent decisions affecting current work: - [Phase 11]: Integration test uses injected test platforms for PasteSites (same pattern as SandboxesSource) - [Phase 11]: All five search sources use dork query format to focus on paste/code hosting leak sites - [Phase 12]: Shodan/Censys/ZoomEye use bare keyword queries; Censys POST+BasicAuth, Shodan key param, ZoomEye API-KEY header +- [Phase 12]: RegisterAll extended to 28 sources (18 Phase 10-11 + 10 Phase 12); cloud scanners credentialless, IoT scanners credential-gated ### Pending Todos @@ -147,6 +149,6 @@ None yet. ## Session Continuity -Last session: 2026-04-06T09:25:12.000Z -Stopped at: Completed 12-01-PLAN.md +Last session: 2026-04-06T09:42:09.000Z +Stopped at: Completed 12-04-PLAN.md Resume file: None diff --git a/.planning/phases/12-osint_iot_cloud_storage/12-04-SUMMARY.md b/.planning/phases/12-osint_iot_cloud_storage/12-04-SUMMARY.md new file mode 100644 index 0000000..fd63e83 --- /dev/null +++ b/.planning/phases/12-osint_iot_cloud_storage/12-04-SUMMARY.md @@ -0,0 +1,117 @@ +--- +phase: 12-osint_iot_cloud_storage +plan: 04 +subsystem: recon +tags: [shodan, censys, zoomeye, fofa, netlas, binaryedge, s3, gcs, azureblob, spaces, registerall, integration-test] + +requires: + - phase: 12-01 + provides: Shodan, Censys, ZoomEye source implementations + - phase: 12-02 + provides: FOFA, Netlas, BinaryEdge source implementations + - phase: 12-03 + provides: S3, GCS, AzureBlob, DOSpaces scanner implementations +provides: + - RegisterAll wiring for all 28 sources (Phase 10-11-12) + - cmd/recon.go credential lookup for 6 IoT scanner APIs + - Integration test covering all 28 sources end-to-end +affects: [phase-13, phase-14, phase-15, phase-16] + +tech-stack: + added: [] + patterns: [per-phase RegisterAll extension, env+viper credential precedence chain] + +key-files: + created: [] + modified: + - pkg/recon/sources/register.go + - cmd/recon.go + - pkg/recon/sources/integration_test.go + - pkg/recon/sources/register_test.go + +key-decisions: + - "Cloud storage sources registered as credentialless (Enabled()==true always); IoT sources require API keys" + - "Integration test uses separate cloud storage handlers per format (S3 XML, GCS JSON, Azure EnumerationResults XML)" + +patterns-established: + - "Phase source wiring: extend SourcesConfig + RegisterAll + cmd/recon.go buildReconEngine + integration test in lockstep" + +requirements-completed: [RECON-IOT-01, RECON-IOT-02, RECON-IOT-03, RECON-IOT-04, RECON-IOT-05, RECON-IOT-06, RECON-CLOUD-01, RECON-CLOUD-02, RECON-CLOUD-03, RECON-CLOUD-04] + +duration: 14min +completed: 2026-04-06 +--- + +# Phase 12 Plan 04: RegisterAll Wiring + Integration Test Summary + +**Wire all 10 Phase 12 IoT/cloud sources into RegisterAll with env/viper credentials and 28-source integration test** + +## Performance + +- **Duration:** 14 min +- **Started:** 2026-04-06T09:28:20Z +- **Completed:** 2026-04-06T09:42:09Z +- **Tasks:** 2 +- **Files modified:** 4 + +## Accomplishments +- Extended SourcesConfig with 8 credential fields for 6 IoT scanner APIs (Shodan, Censys, ZoomEye, FOFA, Netlas, BinaryEdge) +- Registered all 10 Phase 12 sources in RegisterAll (6 IoT + 4 cloud storage), bringing total to 28 +- Wired env var + viper config credential lookup in cmd/recon.go for all Phase 12 sources +- Integration test verifies all 28 sources produce findings through multiplexed httptest server + +## Task Commits + +Each task was committed atomically: + +1. **Task 1: Extend SourcesConfig, RegisterAll, and cmd/recon.go** - `8704316` (feat) +2. **Task 2: Integration test for all 28 registered sources** - `f0f2219` (test) + +## Files Created/Modified +- `pkg/recon/sources/register.go` - Added Phase 12 credential fields + source registrations (28 total) +- `cmd/recon.go` - Added env/viper credential wiring for 8 IoT scanner fields +- `pkg/recon/sources/integration_test.go` - Extended with Phase 12 IoT + cloud storage fixtures and assertions +- `pkg/recon/sources/register_test.go` - Updated expected source count from 18 to 28 + +## Decisions Made +- Cloud storage sources (S3, GCS, AzureBlob, DOSpaces) are credentialless and always enabled +- IoT sources require API keys and report Enabled()==false when credentials are empty +- Integration test uses format-specific handlers: S3/DOSpaces share S3 XML handler, GCS gets JSON handler, AzureBlob gets EnumerationResults XML handler + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Bug] Updated existing register_test.go expected source count** +- **Found during:** Task 2 (integration test) +- **Issue:** TestRegisterAll_WiresAllEighteenSources and TestRegisterAll_MissingCredsStillRegistered expected 18 sources, now 28 +- **Fix:** Updated expected count to 28 and added all Phase 12 source names to expected list +- **Files modified:** pkg/recon/sources/register_test.go +- **Verification:** All RegisterAll tests pass +- **Committed in:** f0f2219 (Task 2 commit) + +**2. [Rule 3 - Blocking] Merged main branch to get Phase 12 source files** +- **Found during:** Task 1 (build verification) +- **Issue:** Worktree branch did not have Phase 12-01/12-02 source files (shodan.go, censys.go, etc.) +- **Fix:** Merged main branch into worktree (fast-forward) +- **Verification:** go build ./cmd/... succeeds + +--- + +**Total deviations:** 2 auto-fixed (1 bug, 1 blocking) +**Impact on plan:** Both fixes necessary for correctness. No scope creep. + +## Issues Encountered +None beyond the deviations listed above. + +## User Setup Required +None - no external service configuration required. + +## Next Phase Readiness +- All 28 OSINT sources are wired and discoverable via `keyhunter recon list` +- Phase 13+ sources can follow the same pattern: add fields to SourcesConfig, register in RegisterAll, wire credentials in cmd/recon.go +- Integration test template established for validating all sources end-to-end + +--- +*Phase: 12-osint_iot_cloud_storage* +*Completed: 2026-04-06*