docs(13): create phase plan — 4 plans for package registries + container/IaC sources

2026-04-06 12:50:38 +03:00
parent a5253cf9dd
commit 877ae8c743
5 changed files with 917 additions and 1 deletions
--- a/.planning/phases/13-osint_package_registries_container_iac/13-04-PLAN.md
+++ b/.planning/phases/13-osint_package_registries_container_iac/13-04-PLAN.md
@@ -0,0 +1,237 @@
+---
+phase: 13-osint_package_registries_container_iac
+plan: 04
+type: execute
+wave: 2
+depends_on:
+  - "13-01"
+  - "13-02"
+  - "13-03"
+files_modified:
+  - pkg/recon/sources/register.go
+  - pkg/recon/sources/register_test.go
+  - pkg/recon/sources/integration_test.go
+  - cmd/recon.go
+autonomous: true
+requirements:
+  - RECON-PKG-01
+  - RECON-PKG-02
+  - RECON-PKG-03
+  - RECON-INFRA-01
+  - RECON-INFRA-02
+  - RECON-INFRA-03
+  - RECON-INFRA-04
+
+must_haves:
+  truths:
+    - "RegisterAll registers all 12 new Phase 13 sources (40 total) on the engine"
+    - "All 40 sources appear in engine.List() sorted alphabetically"
+    - "Integration test runs SweepAll across all 40 sources with httptest fixtures and gets at least one finding per SourceType"
+    - "cmd/recon.go wires any new SourcesConfig fields needed for Phase 13 sources"
+  artifacts:
+    - path: "pkg/recon/sources/register.go"
+      provides: "Updated RegisterAll with 12 new Phase 13 source registrations"
+      contains: "NpmSource"
+    - path: "pkg/recon/sources/register_test.go"
+      provides: "Updated test asserting 40 sources registered"
+      contains: "40"
+    - path: "pkg/recon/sources/integration_test.go"
+      provides: "Updated integration test with httptest mux handlers for all 12 new sources"
+      contains: "recon:npm"
+  key_links:
+    - from: "pkg/recon/sources/register.go"
+      to: "pkg/recon/sources/npm.go"
+      via: "engine.Register call"
+      pattern: "NpmSource"
+    - from: "pkg/recon/sources/register.go"
+      to: "pkg/recon/sources/dockerhub.go"
+      via: "engine.Register call"
+      pattern: "DockerHubSource"
+    - from: "pkg/recon/sources/integration_test.go"
+      to: "all 12 new sources"
+      via: "httptest mux handlers"
+      pattern: "recon:(npm|pypi|crates|rubygems|maven|nuget|goproxy|packagist|dockerhub|k8s|terraform|helm)"
+---
+
+<objective>
+Wire all 12 Phase 13 sources into RegisterAll, update register_test.go to assert 40 total sources, and extend the integration test with httptest handlers for all new sources.
+
+Purpose: Connects the individually-implemented sources into the recon engine so `keyhunter recon` discovers and runs them. Integration test proves end-to-end SweepAll works across all 40 sources.
+Output: Updated register.go, register_test.go, integration_test.go, cmd/recon.go
+</objective>
+
+<execution_context>
+@$HOME/.claude/get-shit-done/workflows/execute-plan.md
+@$HOME/.claude/get-shit-done/templates/summary.md
+</execution_context>
+
+<context>
+@.planning/PROJECT.md
+@.planning/ROADMAP.md
+@.planning/STATE.md
+@pkg/recon/sources/register.go
+@pkg/recon/sources/register_test.go
+@pkg/recon/sources/integration_test.go
+@cmd/recon.go
+
+<!-- Depends on Plans 13-01, 13-02, 13-03 outputs -->
+@.planning/phases/13-osint_package_registries_container_iac/13-01-SUMMARY.md
+@.planning/phases/13-osint_package_registries_container_iac/13-02-SUMMARY.md
+@.planning/phases/13-osint_package_registries_container_iac/13-03-SUMMARY.md
+
+<interfaces>
+From pkg/recon/sources/register.go (current):
+```go
+type SourcesConfig struct {
+    GitHubToken string
+    // ... existing fields ...
+    Registry *providers.Registry
+    Limiters *recon.LimiterRegistry
+}
+
+func RegisterAll(engine *recon.Engine, cfg SourcesConfig) { ... }
+```
+
+From pkg/recon/engine.go:
+```go
+func (e *Engine) Register(src ReconSource)
+func (e *Engine) List() []string  // sorted source names
+```
+
+New sources created by Plans 13-01..03 (all credentialless, struct-literal style):
+- NpmSource{BaseURL, Registry, Limiters, Client}
+- PyPISource{BaseURL, Registry, Limiters, Client}
+- CratesIOSource{BaseURL, Registry, Limiters, Client}
+- RubyGemsSource{BaseURL, Registry, Limiters, Client}
+- MavenSource{BaseURL, Registry, Limiters, Client}
+- NuGetSource{BaseURL, Registry, Limiters, Client}
+- GoProxySource{BaseURL, Registry, Limiters, Client}
+- PackagistSource{BaseURL, Registry, Limiters, Client}
+- DockerHubSource{BaseURL, Registry, Limiters, Client}
+- KubernetesSource{BaseURL, Registry, Limiters, Client}
+- TerraformSource{BaseURL, Registry, Limiters, Client}
+- HelmSource{BaseURL, Registry, Limiters, Client}
+</interfaces>
+</context>
+
+<tasks>
+
+<task type="auto">
+  <name>Task 1: Wire Phase 13 sources into RegisterAll and update register_test</name>
+  <files>pkg/recon/sources/register.go, pkg/recon/sources/register_test.go</files>
+  <action>
+**register.go updates:**
+1. Add a `// Phase 13: Package registry sources (credentialless).` comment block after the Phase 12 cloud storage block
+2. Register all 8 package registry sources as struct literals (no New* constructors needed since they're credentialless):
+   ```go
+   engine.Register(&NpmSource{Registry: reg, Limiters: lim})
+   engine.Register(&PyPISource{Registry: reg, Limiters: lim})
+   engine.Register(&CratesIOSource{Registry: reg, Limiters: lim})
+   engine.Register(&RubyGemsSource{Registry: reg, Limiters: lim})
+   engine.Register(&MavenSource{Registry: reg, Limiters: lim})
+   engine.Register(&NuGetSource{Registry: reg, Limiters: lim})
+   engine.Register(&GoProxySource{Registry: reg, Limiters: lim})
+   engine.Register(&PackagistSource{Registry: reg, Limiters: lim})
+   ```
+3. Add a `// Phase 13: Container & IaC sources (credentialless).` comment block
+4. Register all 4 infra sources:
+   ```go
+   engine.Register(&DockerHubSource{Registry: reg, Limiters: lim})
+   engine.Register(&KubernetesSource{Registry: reg, Limiters: lim})
+   engine.Register(&TerraformSource{Registry: reg, Limiters: lim})
+   engine.Register(&HelmSource{Registry: reg, Limiters: lim})
+   ```
+5. Update the RegisterAll doc comment: change "28 sources total" to "40 sources total" and mention Phase 13
+6. No new SourcesConfig fields needed — all Phase 13 sources are credentialless
+
+**register_test.go updates:**
+1. Rename `TestRegisterAll_WiresAllTwentyEightSources` to `TestRegisterAll_WiresAllFortySources`
+2. Update `want` slice to include all 12 new names in alphabetical order: "crates", "dockerhub", "goproxy", "helm", "k8s", "maven", "npm", "nuget", "packagist", "pypi", "rubygems", "terraform" merged into existing list
+3. Update `TestRegisterAll_MissingCredsStillRegistered` count from 28 to 40
+4. The full sorted list should be: azureblob, binaryedge, bing, bitbucket, brave, censys, codeberg, codesandbox, crates, dockerhub, dospaces, duckduckgo, fofa, gcs, gist, gistpaste, github, gitlab, google, goproxy, helm, huggingface, k8s, kaggle, maven, netlas, npm, nuget, packagist, pastebin, pastesites, pypi, replit, rubygems, s3, sandboxes, shodan, spaces, terraform, yandex, zoomeye
+   Wait — that's 41. Let me recount existing: azureblob, binaryedge, bing, bitbucket, brave, censys, codeberg, codesandbox, duckduckgo, fofa, gcs, gist, gistpaste, github, gitlab, google, huggingface, kaggle, netlas, pastebin, pastesites, replit, s3, sandboxes, shodan, spaces, yandex, zoomeye = 28.
+   Add 12 new: crates, dockerhub, goproxy, helm, k8s, maven, npm, nuget, packagist, pypi, rubygems, terraform = 12.
+   But wait — check if dospaces is already in the list. Looking at register.go: DOSpacesScanner is registered. Check its Name(). Need to verify.
+   Read the current want list from register_test.go to be precise. It has 28 entries already listed. Add the 12 new ones merged alphabetically. Total = 40.
+  </action>
+  <verify>
+    <automated>cd /home/salva/Documents/apikey && go test ./pkg/recon/sources/ -run "TestRegisterAll" -v -count=1</automated>
+  </verify>
+  <done>RegisterAll registers all 40 sources. TestRegisterAll_WiresAllFortySources passes with complete sorted name list. Missing creds test asserts 40.</done>
+</task>
+
+<task type="auto">
+  <name>Task 2: Extend integration test with Phase 13 httptest handlers</name>
+  <files>pkg/recon/sources/integration_test.go, cmd/recon.go</files>
+  <action>
+**integration_test.go updates:**
+1. Add httptest mux handlers for all 12 new sources. Each handler serves canned JSON/HTML fixture matching the API format that source expects:
+
+   **npm** — `mux.HandleFunc("/npm/-/v1/search", ...)` returning `{"objects": [{"package": {"name": "leak-pkg", "links": {"npm": "https://npmjs.com/package/leak-pkg"}}}]}`
+
+   **pypi** — `mux.HandleFunc("/pypi/search/", ...)` returning HTML with `<a href="/project/leaked-pkg/">` links
+
+   **crates** — `mux.HandleFunc("/crates/api/v1/crates", ...)` returning `{"crates": [{"name": "leaked-crate"}]}`
+
+   **rubygems** — `mux.HandleFunc("/rubygems/api/v1/search.json", ...)` returning `[{"name": "leaked-gem", "project_uri": "https://rubygems.org/gems/leaked-gem"}]`
+
+   **maven** — `mux.HandleFunc("/maven/solrsearch/select", ...)` returning `{"response": {"docs": [{"g": "com.leak", "a": "sdk", "latestVersion": "1.0"}]}}`
+
+   **nuget** — `mux.HandleFunc("/nuget/query", ...)` returning `{"data": [{"id": "LeakedPkg", "version": "1.0"}]}`
+
+   **goproxy** — `mux.HandleFunc("/goproxy/search", ...)` returning HTML with `<a href="/github.com/leak/module">` links
+
+   **packagist** — `mux.HandleFunc("/packagist/search.json", ...)` returning `{"results": [{"name": "vendor/leaked", "url": "https://packagist.org/packages/vendor/leaked"}]}`
+
+   **dockerhub** — `mux.HandleFunc("/dockerhub/v2/search/repositories/", ...)` returning `{"results": [{"repo_name": "user/leaked-image"}]}`
+
+   **k8s** — `mux.HandleFunc("/k8s/api/v1/packages/search", ...)` returning `{"packages": [{"name": "leaked-operator", "repository": {"name": "bitnami", "kind": 6}}]}`
+
+   **terraform** — `mux.HandleFunc("/terraform/v1/modules", ...)` returning `{"modules": [{"namespace": "hashicorp", "name": "leaked", "provider": "aws"}]}`
+
+   **helm** — `mux.HandleFunc("/helm/api/v1/packages/search", ...)` returning `{"packages": [{"name": "leaked-chart", "repository": {"name": "bitnami", "kind": 0}}]}`
+
+   NOTE: The mux path prefixes (e.g., `/npm/`, `/pypi/`) are conventions to route in a single httptest server. Each source constructor in the test sets BaseURL to `srv.URL + "/npm"`, `srv.URL + "/pypi"`, etc.
+
+2. Register each new source with BaseURL pointing at `srv.URL + "/{prefix}"`:
+   ```go
+   engine.Register(&NpmSource{BaseURL: srv.URL + "/npm", Registry: reg, Limiters: lim, Client: NewClient()})
+   // ... same for all 12
+   ```
+
+3. Update the expected SourceType set to include all 12 new types: "recon:npm", "recon:pypi", "recon:crates", "recon:rubygems", "recon:maven", "recon:nuget", "recon:goproxy", "recon:packagist", "recon:dockerhub", "recon:k8s", "recon:terraform", "recon:helm"
+
+4. Update the test name/comment from "28 sources" to "40 sources"
+
+**cmd/recon.go updates:**
+- No new SourcesConfig fields needed since all Phase 13 sources are credentialless
+- Verify the existing cmd/recon.go RegisterAll call passes through correctly — no changes expected but confirm no compilation errors
+  </action>
+  <verify>
+    <automated>cd /home/salva/Documents/apikey && go test ./pkg/recon/sources/ -run "TestIntegration_AllSources" -v -count=1 -timeout=60s</automated>
+  </verify>
+  <done>Integration test passes with all 40 sources producing at least one finding each via httptest. Full package compiles clean.</done>
+</task>
+
+</tasks>
+
+<verification>
+Full test suite passes:
+```bash
+go test ./pkg/recon/sources/ -v -count=1 -timeout=120s
+go vet ./pkg/recon/sources/
+go build ./cmd/...
+```
+</verification>
+
+<success_criteria>
+- RegisterAll registers 40 sources (28 existing + 12 new)
+- register_test.go asserts exact 40-name sorted list
+- Integration test exercises all 40 sources via httptest
+- cmd/recon.go compiles with updated register.go
+- `go test ./pkg/recon/sources/ -count=1` all green
+</success_criteria>
+
+<output>
+After completion, create `.planning/phases/13-osint_package_registries_container_iac/13-04-SUMMARY.md`
+</output>