salvacybersec/keyhunter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(12-04): complete RegisterAll wiring + integration test plan

Browse Source 
- SUMMARY.md with 28-source RegisterAll wiring and integration test
- STATE.md, ROADMAP.md, REQUIREMENTS.md updated
This commit is contained in:
salvacybersec
2026-04-06 12:43:34 +03:00
parent f0f22191ef
commit a2347f150a
4 changed files with 133 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
.planning/REQUIREMENTS.md
View File

@@ -96,9 +96,9 @@ Requirements for initial release. Each maps to roadmap phases.
- [x] **RECON-IOT-01**: Shodan API search and dorking
- [x] **RECON-IOT-02**: Censys API search
- [x] **RECON-IOT-03**: ZoomEye API search
- [ ] **RECON-IOT-04**: FOFA API search
- [ ] **RECON-IOT-05**: Netlas API search
- [ ] **RECON-IOT-06**: BinaryEdge API search
- [x] **RECON-IOT-04**: FOFA API search
- [x] **RECON-IOT-05**: Netlas API search
- [x] **RECON-IOT-06**: BinaryEdge API search
### OSINT/Recon — Code Hosting & Snippets

8
.planning/ROADMAP.md
View File

@@ -23,7 +23,7 @@ Decimal phases appear between their surrounding integers in numeric order.
- [ ] **Phase 9: OSINT Infrastructure** - Per-source rate limiter architecture and recon engine framework before any sources
- [x] **Phase 10: OSINT Code Hosting** - GitHub, GitLab, Bitbucket, HuggingFace and 6 more code hosting sources (completed 2026-04-05)
- [x] **Phase 11: OSINT Search & Paste** - Search engine dorking and paste site aggregation (completed 2026-04-06)
- [ ] **Phase 12: OSINT IoT & Cloud Storage** - Shodan/Censys/ZoomEye/FOFA and S3/GCS/Azure cloud storage scanning
- [x] **Phase 12: OSINT IoT & Cloud Storage** - Shodan/Censys/ZoomEye/FOFA and S3/GCS/Azure cloud storage scanning (completed 2026-04-06)
- [ ] **Phase 13: OSINT Package Registries & Container/IaC** - npm/PyPI/crates.io and Docker Hub/K8s/Terraform scanning
- [ ] **Phase 14: OSINT CI/CD Logs, Web Archives & Frontend Leaks** - Build logs, Wayback Machine, and JS bundle/env scanning
- [ ] **Phase 15: OSINT Forums, Collaboration & Log Aggregators** - StackOverflow/Reddit/HN, Notion/Trello, Elasticsearch/Grafana/Sentry
@@ -256,9 +256,9 @@ Plans:
Plans:
- [x] 12-01-PLAN.md — ShodanSource + CensysSource + ZoomEyeSource (RECON-IOT-01, RECON-IOT-02, RECON-IOT-03)
- [ ] 12-02-PLAN.md — FOFASource + NetlasSource + BinaryEdgeSource (RECON-IOT-04, RECON-IOT-05, RECON-IOT-06)
- [x] 12-02-PLAN.md — FOFASource + NetlasSource + BinaryEdgeSource (RECON-IOT-04, RECON-IOT-05, RECON-IOT-06)
- [x] 12-03-PLAN.md — S3Scanner + GCSScanner + AzureBlobScanner + DOSpacesScanner (RECON-CLOUD-01, RECON-CLOUD-02, RECON-CLOUD-03, RECON-CLOUD-04)
- [ ] 12-04-PLAN.md — RegisterAll wiring + cmd/recon.go credentials + integration test (all Phase 12 reqs)
- [x] 12-04-PLAN.md — RegisterAll wiring + cmd/recon.go credentials + integration test (all Phase 12 reqs)
### Phase 13: OSINT Package Registries & Container/IaC
**Goal**: Users can scan npm, PyPI, and 6 other package registries for packages containing leaked keys, and scan Docker Hub image layers, Kubernetes configs, Terraform state files, Helm charts, and Ansible Galaxy for secrets in infrastructure code
@@ -349,7 +349,7 @@ Phases execute in numeric order: 1 → 2 → 3 → ... → 18
| 9. OSINT Infrastructure | 2/6 | In Progress| |
| 10. OSINT Code Hosting | 9/9 | Complete | 2026-04-06 |
| 11. OSINT Search & Paste | 3/3 | Complete | 2026-04-06 |
| 12. OSINT IoT & Cloud Storage | 1/4 | In Progress| |
| 12. OSINT IoT & Cloud Storage | 4/4 | Complete | 2026-04-06 |
| 13. OSINT Package Registries & Container/IaC | 0/? | Not started | - |
| 14. OSINT CI/CD Logs, Web Archives & Frontend Leaks | 0/? | Not started | - |
| 15. OSINT Forums, Collaboration & Log Aggregators | 0/? | Not started | - |

16
.planning/STATE.md
View File

@@ -3,8 +3,8 @@ gsd_state_version: 1.0
milestone: v1.0
milestone_name: milestone
status: completed
stopped_at: Completed 12-01-PLAN.md
last_updated: "2026-04-06T09:25:12.004Z"
stopped_at: Completed 12-04-PLAN.md
last_updated: "2026-04-06T09:42:09.000Z"
last_activity: 2026-04-06
progress:
total_phases: 18
@@ -21,13 +21,13 @@ progress:
See: .planning/PROJECT.md (updated 2026-04-04)
**Core value:** Detect leaked LLM API keys across more providers and more internet sources than any other tool, with active verification to confirm keys are real and alive.
**Current focus:** Phase 11 — osint-search-paste (complete)
**Current focus:** Phase 12 — osint_iot_cloud_storage (in progress)
## Current Position
Phase: 12
Plan: Not started
Status: Phase 11 complete
Plan: 4 of 4
Status: Plan 04 complete
Last activity: 2026-04-06
Progress: [██░░░░░░░░] 20%
@@ -92,6 +92,7 @@ Progress: [██░░░░░░░░] 20%
| Phase 11 P03 | 6min | 2 tasks | 4 files |
| Phase 11 P01 | 3min | 2 tasks | 11 files |
| Phase 12 P01 | 3min | 2 tasks | 6 files |
| Phase 12 P04 | 14min | 2 tasks | 4 files |
## Accumulated Context
@@ -133,6 +134,7 @@ Recent decisions affecting current work:
- [Phase 11]: Integration test uses injected test platforms for PasteSites (same pattern as SandboxesSource)
- [Phase 11]: All five search sources use dork query format to focus on paste/code hosting leak sites
- [Phase 12]: Shodan/Censys/ZoomEye use bare keyword queries; Censys POST+BasicAuth, Shodan key param, ZoomEye API-KEY header
- [Phase 12]: RegisterAll extended to 28 sources (18 Phase 10-11 + 10 Phase 12); cloud scanners credentialless, IoT scanners credential-gated
### Pending Todos
@@ -147,6 +149,6 @@ None yet.
## Session Continuity
Last session: 2026-04-06T09:25:12.000Z
Stopped at: Completed 12-01-PLAN.md
Last session: 2026-04-06T09:42:09.000Z
Stopped at: Completed 12-04-PLAN.md
Resume file: None

117
.planning/phases/12-osint_iot_cloud_storage/12-04-SUMMARY.md Normal file
View File

@@ -0,0 +1,117 @@
---
phase: 12-osint_iot_cloud_storage
plan: 04
subsystem: recon
tags: [shodan, censys, zoomeye, fofa, netlas, binaryedge, s3, gcs, azureblob, spaces, registerall, integration-test]
requires:
- phase: 12-01
provides: Shodan, Censys, ZoomEye source implementations
- phase: 12-02
provides: FOFA, Netlas, BinaryEdge source implementations
- phase: 12-03
provides: S3, GCS, AzureBlob, DOSpaces scanner implementations
provides:
- RegisterAll wiring for all 28 sources (Phase 10-11-12)
- cmd/recon.go credential lookup for 6 IoT scanner APIs
- Integration test covering all 28 sources end-to-end
affects: [phase-13, phase-14, phase-15, phase-16]
tech-stack:
added: []
patterns: [per-phase RegisterAll extension, env+viper credential precedence chain]
key-files:
created: []
modified:
- pkg/recon/sources/register.go
- cmd/recon.go
- pkg/recon/sources/integration_test.go
- pkg/recon/sources/register_test.go
key-decisions:
- "Cloud storage sources registered as credentialless (Enabled()==true always); IoT sources require API keys"
- "Integration test uses separate cloud storage handlers per format (S3 XML, GCS JSON, Azure EnumerationResults XML)"
patterns-established:
- "Phase source wiring: extend SourcesConfig + RegisterAll + cmd/recon.go buildReconEngine + integration test in lockstep"
requirements-completed: [RECON-IOT-01, RECON-IOT-02, RECON-IOT-03, RECON-IOT-04, RECON-IOT-05, RECON-IOT-06, RECON-CLOUD-01, RECON-CLOUD-02, RECON-CLOUD-03, RECON-CLOUD-04]
duration: 14min
completed: 2026-04-06
---
# Phase 12 Plan 04: RegisterAll Wiring + Integration Test Summary
**Wire all 10 Phase 12 IoT/cloud sources into RegisterAll with env/viper credentials and 28-source integration test**
## Performance
- **Duration:** 14 min
- **Started:** 2026-04-06T09:28:20Z
- **Completed:** 2026-04-06T09:42:09Z
- **Tasks:** 2
- **Files modified:** 4
## Accomplishments
- Extended SourcesConfig with 8 credential fields for 6 IoT scanner APIs (Shodan, Censys, ZoomEye, FOFA, Netlas, BinaryEdge)
- Registered all 10 Phase 12 sources in RegisterAll (6 IoT + 4 cloud storage), bringing total to 28
- Wired env var + viper config credential lookup in cmd/recon.go for all Phase 12 sources
- Integration test verifies all 28 sources produce findings through multiplexed httptest server
## Task Commits
Each task was committed atomically:
1. **Task 1: Extend SourcesConfig, RegisterAll, and cmd/recon.go** - `8704316` (feat)
2. **Task 2: Integration test for all 28 registered sources** - `f0f2219` (test)
## Files Created/Modified
- `pkg/recon/sources/register.go` - Added Phase 12 credential fields + source registrations (28 total)
- `cmd/recon.go` - Added env/viper credential wiring for 8 IoT scanner fields
- `pkg/recon/sources/integration_test.go` - Extended with Phase 12 IoT + cloud storage fixtures and assertions
- `pkg/recon/sources/register_test.go` - Updated expected source count from 18 to 28
## Decisions Made
- Cloud storage sources (S3, GCS, AzureBlob, DOSpaces) are credentialless and always enabled
- IoT sources require API keys and report Enabled()==false when credentials are empty
- Integration test uses format-specific handlers: S3/DOSpaces share S3 XML handler, GCS gets JSON handler, AzureBlob gets EnumerationResults XML handler
## Deviations from Plan
### Auto-fixed Issues
**1. [Rule 1 - Bug] Updated existing register_test.go expected source count**
- **Found during:** Task 2 (integration test)
- **Issue:** TestRegisterAll_WiresAllEighteenSources and TestRegisterAll_MissingCredsStillRegistered expected 18 sources, now 28
- **Fix:** Updated expected count to 28 and added all Phase 12 source names to expected list
- **Files modified:** pkg/recon/sources/register_test.go
- **Verification:** All RegisterAll tests pass
- **Committed in:** f0f2219 (Task 2 commit)
**2. [Rule 3 - Blocking] Merged main branch to get Phase 12 source files**
- **Found during:** Task 1 (build verification)
- **Issue:** Worktree branch did not have Phase 12-01/12-02 source files (shodan.go, censys.go, etc.)
- **Fix:** Merged main branch into worktree (fast-forward)
- **Verification:** go build ./cmd/... succeeds
---
**Total deviations:** 2 auto-fixed (1 bug, 1 blocking)
**Impact on plan:** Both fixes necessary for correctness. No scope creep.
## Issues Encountered
None beyond the deviations listed above.
## User Setup Required
None - no external service configuration required.
## Next Phase Readiness
- All 28 OSINT sources are wired and discoverable via `keyhunter recon list`
- Phase 13+ sources can follow the same pattern: add fields to SourcesConfig, register in RegisterAll, wire credentials in cmd/recon.go
- Integration test template established for validating all sources end-to-end
---
*Phase: 12-osint_iot_cloud_storage*
*Completed: 2026-04-06*