diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index f91fffa..1609850 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -252,7 +252,13 @@ Plans: 3. `keyhunter recon --sources=s3` enumerates publicly accessible S3 buckets and scans readable objects for API key patterns 4. `keyhunter recon --sources=gcs,azureblob,spaces` scans GCS, Azure Blob, and DigitalOcean Spaces; `--sources=minio` discovers MinIO instances via Shodan integration 5. `keyhunter recon --sources=grayhoundwarfare` queries the GrayHatWarfare bucket search engine for matching bucket names -**Plans**: TBD +**Plans**: 4 plans + +Plans: +- [ ] 12-01-PLAN.md — ShodanSource + CensysSource + ZoomEyeSource (RECON-IOT-01, RECON-IOT-02, RECON-IOT-03) +- [ ] 12-02-PLAN.md — FOFASource + NetlasSource + BinaryEdgeSource (RECON-IOT-04, RECON-IOT-05, RECON-IOT-06) +- [ ] 12-03-PLAN.md — S3Scanner + GCSScanner + AzureBlobScanner + DOSpacesScanner (RECON-CLOUD-01, RECON-CLOUD-02, RECON-CLOUD-03, RECON-CLOUD-04) +- [ ] 12-04-PLAN.md — RegisterAll wiring + cmd/recon.go credentials + integration test (all Phase 12 reqs) ### Phase 13: OSINT Package Registries & Container/IaC **Goal**: Users can scan npm, PyPI, and 6 other package registries for packages containing leaked keys, and scan Docker Hub image layers, Kubernetes configs, Terraform state files, Helm charts, and Ansible Galaxy for secrets in infrastructure code diff --git a/.planning/phases/12-osint_iot_cloud_storage/12-01-PLAN.md b/.planning/phases/12-osint_iot_cloud_storage/12-01-PLAN.md new file mode 100644 index 0000000..5e7dede --- /dev/null +++ b/.planning/phases/12-osint_iot_cloud_storage/12-01-PLAN.md @@ -0,0 +1,193 @@ +--- +phase: 12-osint_iot_cloud_storage +plan: 01 +type: execute +wave: 1 +depends_on: [] +files_modified: + - pkg/recon/sources/shodan.go + - pkg/recon/sources/shodan_test.go + - pkg/recon/sources/censys.go + - pkg/recon/sources/censys_test.go + - pkg/recon/sources/zoomeye.go + - pkg/recon/sources/zoomeye_test.go +autonomous: true +requirements: [RECON-IOT-01, RECON-IOT-02, RECON-IOT-03] + +must_haves: + truths: + - "ShodanSource searches Shodan /shodan/host/search for exposed LLM endpoints and emits findings" + - "CensysSource searches Censys v2 /hosts/search for exposed services and emits findings" + - "ZoomEyeSource searches ZoomEye /host/search for device/service key exposure and emits findings" + - "Each source is disabled (Enabled==false) when its API key is empty" + artifacts: + - path: "pkg/recon/sources/shodan.go" + provides: "ShodanSource implementing recon.ReconSource" + exports: ["ShodanSource"] + - path: "pkg/recon/sources/censys.go" + provides: "CensysSource implementing recon.ReconSource" + exports: ["CensysSource"] + - path: "pkg/recon/sources/zoomeye.go" + provides: "ZoomEyeSource implementing recon.ReconSource" + exports: ["ZoomEyeSource"] + key_links: + - from: "pkg/recon/sources/shodan.go" + to: "pkg/recon/sources/httpclient.go" + via: "sources.Client for retry/backoff HTTP" + pattern: "s\\.client\\.Do" + - from: "pkg/recon/sources/censys.go" + to: "pkg/recon/sources/httpclient.go" + via: "sources.Client for retry/backoff HTTP" + pattern: "s\\.client\\.Do" + - from: "pkg/recon/sources/zoomeye.go" + to: "pkg/recon/sources/httpclient.go" + via: "sources.Client for retry/backoff HTTP" + pattern: "s\\.client\\.Do" +--- + + +Implement three IoT scanner recon sources: Shodan, Censys, and ZoomEye. + +Purpose: Enable discovery of exposed LLM endpoints (vLLM, Ollama, LiteLLM proxies) via internet-wide device scanners. +Output: Three source files + tests following the established Phase 10 pattern. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md +@pkg/recon/source.go +@pkg/recon/sources/httpclient.go +@pkg/recon/sources/github.go +@pkg/recon/sources/bing.go +@pkg/recon/sources/queries.go +@pkg/recon/sources/register.go + + +From pkg/recon/source.go: +```go +type ReconSource interface { + Name() string + RateLimit() rate.Limit + Burst() int + RespectsRobots() bool + Enabled(cfg Config) bool + Sweep(ctx context.Context, query string, out chan<- Finding) error +} +``` + +From pkg/recon/sources/httpclient.go: +```go +type Client struct { HTTP *http.Client; MaxRetries int; UserAgent string } +func NewClient() *Client +func (c *Client) Do(ctx context.Context, req *http.Request) (*http.Response, error) +var ErrUnauthorized = errors.New("sources: unauthorized (check credentials)") +``` + +From pkg/recon/sources/queries.go: +```go +func BuildQueries(reg *providers.Registry, source string) []string +``` + + + + + + + Task 1: Implement ShodanSource, CensysSource, ZoomEyeSource + pkg/recon/sources/shodan.go, pkg/recon/sources/censys.go, pkg/recon/sources/zoomeye.go + +Create three source files following the BingDorkSource pattern exactly: + +**ShodanSource** (shodan.go): +- Struct: `ShodanSource` with fields `APIKey string`, `BaseURL string`, `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `client *Client` +- Compile-time assertion: `var _ recon.ReconSource = (*ShodanSource)(nil)` +- Name(): "shodan" +- RateLimit(): rate.Every(1 * time.Second) — Shodan allows ~1 req/s on most plans +- Burst(): 1 +- RespectsRobots(): false (authenticated REST API) +- Enabled(): returns `s.APIKey != ""` +- BaseURL default: "https://api.shodan.io" +- Sweep(): For each query from BuildQueries(s.Registry, "shodan"), call GET `{base}/shodan/host/search?key={apikey}&query={url.QueryEscape(q)}`. Parse JSON response `{"matches":[{"ip_str":"...","port":N,"data":"..."},...]}`. Emit a Finding per match with Source=`fmt.Sprintf("shodan://%s:%d", match.IPStr, match.Port)`, SourceType="recon:shodan", Confidence="low", ProviderName from keyword index. +- Add `shodanKeywordIndex` helper (same pattern as bingKeywordIndex). +- Error handling: ErrUnauthorized aborts, context cancellation aborts, transient errors continue. + +**CensysSource** (censys.go): +- Struct: `CensysSource` with fields `APIId string`, `APISecret string`, `BaseURL string`, `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `client *Client` +- Name(): "censys" +- RateLimit(): rate.Every(2500 * time.Millisecond) — Censys free tier is 0.4 req/s +- Burst(): 1 +- RespectsRobots(): false +- Enabled(): returns `s.APIId != "" && s.APISecret != ""` +- BaseURL default: "https://search.censys.io/api" +- Sweep(): For each query, POST `{base}/v2/hosts/search` with JSON body `{"q":q,"per_page":25}`. Set Basic Auth header using APIId:APISecret. Parse JSON response `{"result":{"hits":[{"ip":"...","services":[{"port":N,"service_name":"..."}]}]}}`. Emit Finding per hit with Source=`fmt.Sprintf("censys://%s", hit.IP)`. +- Add `censysKeywordIndex` helper. + +**ZoomEyeSource** (zoomeye.go): +- Struct: `ZoomEyeSource` with fields `APIKey string`, `BaseURL string`, `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `client *Client` +- Name(): "zoomeye" +- RateLimit(): rate.Every(2 * time.Second) +- Burst(): 1 +- RespectsRobots(): false +- Enabled(): returns `s.APIKey != ""` +- BaseURL default: "https://api.zoomeye.org" (ZoomEye uses v1-style API key in header) +- Sweep(): For each query, GET `{base}/host/search?query={url.QueryEscape(q)}&page=1`. Set header `API-KEY: {apikey}`. Parse JSON response `{"matches":[{"ip":"...","portinfo":{"port":N},"banner":"..."}]}`. Emit Finding per match with Source=`fmt.Sprintf("zoomeye://%s:%d", match.IP, match.PortInfo.Port)`. +- Add `zoomeyeKeywordIndex` helper. + +Update `formatQuery` in queries.go to add cases for "shodan", "censys", "zoomeye" — all use bare keyword (same as default). + +All sources must use `sources.NewClient()` for HTTP, `s.Limiters.Wait(ctx, s.Name(), ...)` before each request, and follow the same error handling pattern as BingDorkSource.Sweep. + + + cd /home/salva/Documents/apikey/.claude/worktrees/agent-a6700ee2 && go build ./pkg/recon/sources/ + + Three source files compile, each implements recon.ReconSource interface + + + + Task 2: Unit tests for Shodan, Censys, ZoomEye sources + pkg/recon/sources/shodan_test.go, pkg/recon/sources/censys_test.go, pkg/recon/sources/zoomeye_test.go + + - Shodan: httptest server returns mock JSON with 2 matches; Sweep emits 2 findings with "recon:shodan" source type + - Shodan: empty API key => Enabled()==false, Sweep returns nil with 0 findings + - Censys: httptest server returns mock JSON with 2 hits; Sweep emits 2 findings with "recon:censys" source type + - Censys: empty APIId => Enabled()==false + - ZoomEye: httptest server returns mock JSON with 2 matches; Sweep emits 2 findings with "recon:zoomeye" source type + - ZoomEye: empty API key => Enabled()==false + - All: cancelled context returns context error + + +Create test files following the pattern in github_test.go / bing_test.go: +- Use httptest.NewServer to mock API responses +- Set BaseURL to test server URL +- Create a minimal providers.Registry with 1-2 test providers containing keywords +- Verify Finding count, SourceType, and Source URL format +- Test disabled state (empty credentials) +- Test context cancellation + + + cd /home/salva/Documents/apikey/.claude/worktrees/agent-a6700ee2 && go test ./pkg/recon/sources/ -run "TestShodan|TestCensys|TestZoomEye" -v -count=1 + + All Shodan, Censys, ZoomEye tests pass; each source emits correct findings from mock API responses + + + + + +- `go build ./pkg/recon/sources/` compiles without errors +- `go test ./pkg/recon/sources/ -run "TestShodan|TestCensys|TestZoomEye" -v` all pass +- Each source file has compile-time assertion `var _ recon.ReconSource = (*XxxSource)(nil)` + + + +Three IoT scanner sources (Shodan, Censys, ZoomEye) implement recon.ReconSource, use shared Client for HTTP, respect rate limiting via LimiterRegistry, and pass unit tests with mock API responses. + + + +After completion, create `.planning/phases/12-osint_iot_cloud_storage/12-01-SUMMARY.md` + diff --git a/.planning/phases/12-osint_iot_cloud_storage/12-02-PLAN.md b/.planning/phases/12-osint_iot_cloud_storage/12-02-PLAN.md new file mode 100644 index 0000000..b7c7560 --- /dev/null +++ b/.planning/phases/12-osint_iot_cloud_storage/12-02-PLAN.md @@ -0,0 +1,187 @@ +--- +phase: 12-osint_iot_cloud_storage +plan: 02 +type: execute +wave: 1 +depends_on: [] +files_modified: + - pkg/recon/sources/fofa.go + - pkg/recon/sources/fofa_test.go + - pkg/recon/sources/netlas.go + - pkg/recon/sources/netlas_test.go + - pkg/recon/sources/binaryedge.go + - pkg/recon/sources/binaryedge_test.go +autonomous: true +requirements: [RECON-IOT-04, RECON-IOT-05, RECON-IOT-06] + +must_haves: + truths: + - "FOFASource searches FOFA API for exposed endpoints and emits findings" + - "NetlasSource searches Netlas API for internet-wide scan results and emits findings" + - "BinaryEdgeSource searches BinaryEdge API for exposed services and emits findings" + - "Each source is disabled when its API key/credentials are empty" + artifacts: + - path: "pkg/recon/sources/fofa.go" + provides: "FOFASource implementing recon.ReconSource" + exports: ["FOFASource"] + - path: "pkg/recon/sources/netlas.go" + provides: "NetlasSource implementing recon.ReconSource" + exports: ["NetlasSource"] + - path: "pkg/recon/sources/binaryedge.go" + provides: "BinaryEdgeSource implementing recon.ReconSource" + exports: ["BinaryEdgeSource"] + key_links: + - from: "pkg/recon/sources/fofa.go" + to: "pkg/recon/sources/httpclient.go" + via: "sources.Client for retry/backoff HTTP" + pattern: "s\\.client\\.Do" + - from: "pkg/recon/sources/netlas.go" + to: "pkg/recon/sources/httpclient.go" + via: "sources.Client for retry/backoff HTTP" + pattern: "s\\.client\\.Do" + - from: "pkg/recon/sources/binaryedge.go" + to: "pkg/recon/sources/httpclient.go" + via: "sources.Client for retry/backoff HTTP" + pattern: "s\\.client\\.Do" +--- + + +Implement three IoT scanner recon sources: FOFA, Netlas, and BinaryEdge. + +Purpose: Complete the IoT/device scanner coverage with Chinese (FOFA) and alternative (Netlas, BinaryEdge) internet search engines. +Output: Three source files + tests following the established Phase 10 pattern. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md +@pkg/recon/source.go +@pkg/recon/sources/httpclient.go +@pkg/recon/sources/bing.go +@pkg/recon/sources/queries.go +@pkg/recon/sources/register.go + + +From pkg/recon/source.go: +```go +type ReconSource interface { + Name() string + RateLimit() rate.Limit + Burst() int + RespectsRobots() bool + Enabled(cfg Config) bool + Sweep(ctx context.Context, query string, out chan<- Finding) error +} +``` + +From pkg/recon/sources/httpclient.go: +```go +type Client struct { HTTP *http.Client; MaxRetries int; UserAgent string } +func NewClient() *Client +func (c *Client) Do(ctx context.Context, req *http.Request) (*http.Response, error) +var ErrUnauthorized = errors.New("sources: unauthorized (check credentials)") +``` + + + + + + + Task 1: Implement FOFASource, NetlasSource, BinaryEdgeSource + pkg/recon/sources/fofa.go, pkg/recon/sources/netlas.go, pkg/recon/sources/binaryedge.go + +Create three source files following the BingDorkSource pattern: + +**FOFASource** (fofa.go): +- Struct: `FOFASource` with fields `Email string`, `APIKey string`, `BaseURL string`, `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `client *Client` +- Compile-time assertion: `var _ recon.ReconSource = (*FOFASource)(nil)` +- Name(): "fofa" +- RateLimit(): rate.Every(1 * time.Second) — FOFA allows ~1 req/s +- Burst(): 1 +- RespectsRobots(): false +- Enabled(): returns `s.Email != "" && s.APIKey != ""` +- BaseURL default: "https://fofa.info" +- Sweep(): For each query from BuildQueries, base64-encode the query, then GET `{base}/api/v1/search/all?email={email}&key={apikey}&qbase64={base64query}&size=100`. Parse JSON response `{"results":[["ip","port","protocol","host"],...],"size":N}`. Emit Finding per result with Source=`fmt.Sprintf("fofa://%s:%s", result[0], result[1])`, SourceType="recon:fofa". +- Note: FOFA results array contains string arrays, not objects. Each inner array is [host, ip, port]. +- Add `fofaKeywordIndex` helper. + +**NetlasSource** (netlas.go): +- Struct: `NetlasSource` with fields `APIKey string`, `BaseURL string`, `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `client *Client` +- Name(): "netlas" +- RateLimit(): rate.Every(1 * time.Second) +- Burst(): 1 +- RespectsRobots(): false +- Enabled(): returns `s.APIKey != ""` +- BaseURL default: "https://app.netlas.io" +- Sweep(): For each query, GET `{base}/api/responses/?q={url.QueryEscape(q)}&start=0&indices=`. Set header `X-API-Key: {apikey}`. Parse JSON response `{"items":[{"data":{"ip":"...","port":N}},...]}`. Emit Finding per item with Source=`fmt.Sprintf("netlas://%s:%d", item.Data.IP, item.Data.Port)`. +- Add `netlasKeywordIndex` helper. + +**BinaryEdgeSource** (binaryedge.go): +- Struct: `BinaryEdgeSource` with fields `APIKey string`, `BaseURL string`, `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `client *Client` +- Name(): "binaryedge" +- RateLimit(): rate.Every(2 * time.Second) — BinaryEdge free tier is conservative +- Burst(): 1 +- RespectsRobots(): false +- Enabled(): returns `s.APIKey != ""` +- BaseURL default: "https://api.binaryedge.io" +- Sweep(): For each query, GET `{base}/v2/query/search?query={url.QueryEscape(q)}&page=1`. Set header `X-Key: {apikey}`. Parse JSON response `{"events":[{"target":{"ip":"...","port":N}},...]}`. Emit Finding per event with Source=`fmt.Sprintf("binaryedge://%s:%d", event.Target.IP, event.Target.Port)`. +- Add `binaryedgeKeywordIndex` helper. + +Update `formatQuery` in queries.go to add cases for "fofa", "netlas", "binaryedge" — all use bare keyword (same as default). + +Same patterns as Plan 12-01: use sources.NewClient(), s.Limiters.Wait before requests, standard error handling. + + + cd /home/salva/Documents/apikey/.claude/worktrees/agent-a6700ee2 && go build ./pkg/recon/sources/ + + Three source files compile, each implements recon.ReconSource interface + + + + Task 2: Unit tests for FOFA, Netlas, BinaryEdge sources + pkg/recon/sources/fofa_test.go, pkg/recon/sources/netlas_test.go, pkg/recon/sources/binaryedge_test.go + + - FOFA: httptest server returns mock JSON with 2 results; Sweep emits 2 findings with "recon:fofa" source type + - FOFA: empty Email or APIKey => Enabled()==false + - Netlas: httptest server returns mock JSON with 2 items; Sweep emits 2 findings with "recon:netlas" source type + - Netlas: empty APIKey => Enabled()==false + - BinaryEdge: httptest server returns mock JSON with 2 events; Sweep emits 2 findings with "recon:binaryedge" source type + - BinaryEdge: empty APIKey => Enabled()==false + - All: cancelled context returns context error + + +Create test files following the same httptest pattern used in Plan 12-01: +- Use httptest.NewServer to mock API responses matching each source's expected JSON shape +- Set BaseURL to test server URL +- Create a minimal providers.Registry with 1-2 test providers +- Verify Finding count, SourceType, and Source URL format +- Test disabled state (empty credentials) +- Test context cancellation + + + cd /home/salva/Documents/apikey/.claude/worktrees/agent-a6700ee2 && go test ./pkg/recon/sources/ -run "TestFOFA|TestNetlas|TestBinaryEdge" -v -count=1 + + All FOFA, Netlas, BinaryEdge tests pass; each source emits correct findings from mock API responses + + + + + +- `go build ./pkg/recon/sources/` compiles without errors +- `go test ./pkg/recon/sources/ -run "TestFOFA|TestNetlas|TestBinaryEdge" -v` all pass +- Each source file has compile-time assertion `var _ recon.ReconSource = (*XxxSource)(nil)` + + + +Three IoT scanner sources (FOFA, Netlas, BinaryEdge) implement recon.ReconSource, use shared Client for HTTP, respect rate limiting via LimiterRegistry, and pass unit tests with mock API responses. + + + +After completion, create `.planning/phases/12-osint_iot_cloud_storage/12-02-SUMMARY.md` + diff --git a/.planning/phases/12-osint_iot_cloud_storage/12-03-PLAN.md b/.planning/phases/12-osint_iot_cloud_storage/12-03-PLAN.md new file mode 100644 index 0000000..598d737 --- /dev/null +++ b/.planning/phases/12-osint_iot_cloud_storage/12-03-PLAN.md @@ -0,0 +1,183 @@ +--- +phase: 12-osint_iot_cloud_storage +plan: 03 +type: execute +wave: 1 +depends_on: [] +files_modified: + - pkg/recon/sources/s3scanner.go + - pkg/recon/sources/s3scanner_test.go + - pkg/recon/sources/gcsscanner.go + - pkg/recon/sources/gcsscanner_test.go + - pkg/recon/sources/azureblob.go + - pkg/recon/sources/azureblob_test.go + - pkg/recon/sources/dospaces.go + - pkg/recon/sources/dospaces_test.go +autonomous: true +requirements: [RECON-CLOUD-01, RECON-CLOUD-02, RECON-CLOUD-03, RECON-CLOUD-04] + +must_haves: + truths: + - "S3Scanner enumerates publicly accessible S3 buckets by name pattern and scans readable objects for API key exposure" + - "GCSScanner scans publicly accessible Google Cloud Storage buckets" + - "AzureBlobScanner scans publicly accessible Azure Blob containers" + - "DOSpacesScanner scans publicly accessible DigitalOcean Spaces" + - "Each cloud scanner is credentialless (uses anonymous HTTP to probe public buckets) and always Enabled" + artifacts: + - path: "pkg/recon/sources/s3scanner.go" + provides: "S3Scanner implementing recon.ReconSource" + exports: ["S3Scanner"] + - path: "pkg/recon/sources/gcsscanner.go" + provides: "GCSScanner implementing recon.ReconSource" + exports: ["GCSScanner"] + - path: "pkg/recon/sources/azureblob.go" + provides: "AzureBlobScanner implementing recon.ReconSource" + exports: ["AzureBlobScanner"] + - path: "pkg/recon/sources/dospaces.go" + provides: "DOSpacesScanner implementing recon.ReconSource" + exports: ["DOSpacesScanner"] + key_links: + - from: "pkg/recon/sources/s3scanner.go" + to: "pkg/recon/sources/httpclient.go" + via: "sources.Client for retry/backoff HTTP" + pattern: "s\\.client\\.Do" +--- + + +Implement four cloud storage scanner recon sources: S3Scanner, GCSScanner, AzureBlobScanner, and DOSpacesScanner. + +Purpose: Enable discovery of API keys leaked in publicly accessible cloud storage buckets across AWS, GCP, Azure, and DigitalOcean. +Output: Four source files + tests following the established Phase 10 pattern. + +Note on RECON-CLOUD-03 (MinIO via Shodan) and RECON-CLOUD-04 (GrayHatWarfare): These are addressed here. MinIO discovery is implemented as a Shodan query variant within S3Scanner (MinIO uses S3-compatible API). GrayHatWarfare is implemented as a dedicated scanner that queries the GrayHatWarfare buckets.grayhatwarfare.com API. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md +@pkg/recon/source.go +@pkg/recon/sources/httpclient.go +@pkg/recon/sources/bing.go +@pkg/recon/sources/queries.go +@pkg/recon/sources/register.go + + +From pkg/recon/source.go: +```go +type ReconSource interface { + Name() string + RateLimit() rate.Limit + Burst() int + RespectsRobots() bool + Enabled(cfg Config) bool + Sweep(ctx context.Context, query string, out chan<- Finding) error +} +``` + +From pkg/recon/sources/httpclient.go: +```go +type Client struct { HTTP *http.Client; MaxRetries int; UserAgent string } +func NewClient() *Client +func (c *Client) Do(ctx context.Context, req *http.Request) (*http.Response, error) +``` + + + + + + + Task 1: Implement S3Scanner and GCSScanner + pkg/recon/sources/s3scanner.go, pkg/recon/sources/gcsscanner.go + +**S3Scanner** (s3scanner.go) — RECON-CLOUD-01 + RECON-CLOUD-03: +- Struct: `S3Scanner` with fields `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `BaseURL string`, `client *Client` +- Compile-time assertion: `var _ recon.ReconSource = (*S3Scanner)(nil)` +- Name(): "s3" +- RateLimit(): rate.Every(500 * time.Millisecond) — S3 public reads are generous +- Burst(): 3 +- RespectsRobots(): false (direct API calls) +- Enabled(): always true (credentialless — probes public buckets) +- Sweep(): Generates candidate bucket names from provider keywords (e.g., "openai-keys", "anthropic-config", "llm-keys", etc.) using a helper `bucketNames(registry)` that combines provider keywords with common suffixes like "-keys", "-config", "-backup", "-data", "-secrets", "-env". For each candidate bucket: + 1. HEAD `https://{bucket}.s3.amazonaws.com/` — if 200/403, bucket exists + 2. If 200 (public listing), GET the ListBucket XML, parse `` elements + 3. For keys matching common config file patterns (.env, config.*, *.json, *.yaml, *.yml, *.toml, *.conf), emit a Finding with Source=`s3://{bucket}/{key}`, SourceType="recon:s3", Confidence="medium" + 4. Do NOT download object contents (too heavy) — just flag the presence of suspicious files +- Use BaseURL override for tests (default: "https://%s.s3.amazonaws.com") +- Note: MinIO instances (RECON-CLOUD-03) are discovered via Shodan queries in Plan 12-01's ShodanSource using the query "minio" — this source focuses on AWS S3 bucket enumeration. + +**GCSScanner** (gcsscanner.go) — RECON-CLOUD-02: +- Struct: `GCSScanner` with fields `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `BaseURL string`, `client *Client` +- Name(): "gcs" +- RateLimit(): rate.Every(500 * time.Millisecond) +- Burst(): 3 +- RespectsRobots(): false +- Enabled(): always true (credentialless) +- Sweep(): Same bucket enumeration pattern as S3Scanner but using `https://storage.googleapis.com/{bucket}` for HEAD and listing. GCS public bucket listing returns JSON when Accept: application/json is set. Parse `{"items":[{"name":"..."}]}`. Emit findings for config-pattern files with Source=`gs://{bucket}/{name}`, SourceType="recon:gcs". + +Both sources share a common `bucketNames` helper function — define it in s3scanner.go and export it for use by both. + + + cd /home/salva/Documents/apikey/.claude/worktrees/agent-a6700ee2 && go build ./pkg/recon/sources/ + + S3Scanner and GCSScanner compile and implement recon.ReconSource + + + + Task 2: Implement AzureBlobScanner, DOSpacesScanner, and all cloud scanner tests + pkg/recon/sources/azureblob.go, pkg/recon/sources/dospaces.go, pkg/recon/sources/s3scanner_test.go, pkg/recon/sources/gcsscanner_test.go, pkg/recon/sources/azureblob_test.go, pkg/recon/sources/dospaces_test.go + +**AzureBlobScanner** (azureblob.go) — RECON-CLOUD-02: +- Struct: `AzureBlobScanner` with fields `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `BaseURL string`, `client *Client` +- Name(): "azureblob" +- RateLimit(): rate.Every(500 * time.Millisecond) +- Burst(): 3 +- RespectsRobots(): false +- Enabled(): always true (credentialless) +- Sweep(): Uses bucket enumeration pattern with Azure Blob URL format `https://{account}.blob.core.windows.net/{container}?restype=container&comp=list`. Generate account names from provider keywords with common suffixes. Parse XML `...`. Emit findings for config-pattern files with Source=`azure://{account}/{container}/{name}`, SourceType="recon:azureblob". + +**DOSpacesScanner** (dospaces.go) — RECON-CLOUD-02: +- Struct: `DOSpacesScanner` with fields `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `BaseURL string`, `client *Client` +- Name(): "spaces" +- RateLimit(): rate.Every(500 * time.Millisecond) +- Burst(): 3 +- RespectsRobots(): false +- Enabled(): always true (credentialless) +- Sweep(): Uses bucket enumeration with DO Spaces URL format `https://{bucket}.{region}.digitaloceanspaces.com/`. Iterate regions: nyc3, sfo3, ams3, sgp1, fra1. Same XML ListBucket format as S3 (DO Spaces is S3-compatible). Emit findings with Source=`do://{bucket}/{key}`, SourceType="recon:spaces". + +**Tests** (all four test files): +Each test file follows the httptest pattern: +- Mock server returns appropriate XML/JSON for bucket listing +- Verify Sweep emits correct number of findings with correct SourceType and Source URL format +- Verify Enabled() returns true (credentialless sources) +- Test with empty registry (no keywords => no bucket names => no findings) +- Test context cancellation + +Use a minimal providers.Registry with 1 test provider having keyword "testprov" so bucket names like "testprov-keys" are generated. + + + cd /home/salva/Documents/apikey/.claude/worktrees/agent-a6700ee2 && go test ./pkg/recon/sources/ -run "TestS3Scanner|TestGCSScanner|TestAzureBlob|TestDOSpaces" -v -count=1 + + All four cloud scanner sources compile and pass tests; each emits findings with correct source type and URL format + + + + + +- `go build ./pkg/recon/sources/` compiles without errors +- `go test ./pkg/recon/sources/ -run "TestS3Scanner|TestGCSScanner|TestAzureBlob|TestDOSpaces" -v` all pass +- Each source file has compile-time assertion + + + +Four cloud storage scanners (S3, GCS, Azure Blob, DO Spaces) implement recon.ReconSource with credentialless public bucket enumeration, use shared Client for HTTP, and pass unit tests. + + + +After completion, create `.planning/phases/12-osint_iot_cloud_storage/12-03-SUMMARY.md` + diff --git a/.planning/phases/12-osint_iot_cloud_storage/12-04-PLAN.md b/.planning/phases/12-osint_iot_cloud_storage/12-04-PLAN.md new file mode 100644 index 0000000..3e46bac --- /dev/null +++ b/.planning/phases/12-osint_iot_cloud_storage/12-04-PLAN.md @@ -0,0 +1,217 @@ +--- +phase: 12-osint_iot_cloud_storage +plan: 04 +type: execute +wave: 2 +depends_on: [12-01, 12-02, 12-03] +files_modified: + - pkg/recon/sources/register.go + - cmd/recon.go + - pkg/recon/sources/integration_test.go +autonomous: true +requirements: [RECON-IOT-01, RECON-IOT-02, RECON-IOT-03, RECON-IOT-04, RECON-IOT-05, RECON-IOT-06, RECON-CLOUD-01, RECON-CLOUD-02, RECON-CLOUD-03, RECON-CLOUD-04] + +must_haves: + truths: + - "RegisterAll registers all 28 sources (18 Phase 10-11 + 10 Phase 12)" + - "cmd/recon.go populates SourcesConfig with all Phase 12 credential fields from env/viper" + - "Integration test proves all 10 new sources are registered and discoverable by name" + artifacts: + - path: "pkg/recon/sources/register.go" + provides: "RegisterAll with all Phase 12 sources added" + contains: "Phase 12" + - path: "cmd/recon.go" + provides: "buildReconEngine with Phase 12 credential wiring" + contains: "ShodanAPIKey" + - path: "pkg/recon/sources/integration_test.go" + provides: "Integration test covering all 28 registered sources" + contains: "28" + key_links: + - from: "pkg/recon/sources/register.go" + to: "pkg/recon/sources/shodan.go" + via: "engine.Register(&ShodanSource{...})" + pattern: "ShodanSource" + - from: "cmd/recon.go" + to: "pkg/recon/sources/register.go" + via: "sources.RegisterAll(e, cfg)" + pattern: "RegisterAll" +--- + + +Wire all 10 Phase 12 sources into RegisterAll and cmd/recon.go, plus integration test. + +Purpose: Make all IoT and cloud storage sources available via `keyhunter recon list` and `keyhunter recon full`. +Output: Updated RegisterAll (28 sources total), updated cmd/recon.go with credential wiring, integration test. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md +@pkg/recon/sources/register.go +@cmd/recon.go +@pkg/recon/sources/integration_test.go + + +From pkg/recon/sources/register.go: +```go +type SourcesConfig struct { + GitHubToken string + // ... existing Phase 10-11 fields ... + Registry *providers.Registry + Limiters *recon.LimiterRegistry +} +func RegisterAll(engine *recon.Engine, cfg SourcesConfig) +``` + +From cmd/recon.go: +```go +func buildReconEngine() *recon.Engine // constructs engine with all sources +func firstNonEmpty(a, b string) string // env -> viper precedence +``` + + + + + + + Task 1: Extend SourcesConfig, RegisterAll, and cmd/recon.go + pkg/recon/sources/register.go, cmd/recon.go + +**SourcesConfig** (register.go) — add these fields after the existing Phase 11 fields: + +```go +// Phase 12: IoT scanner API keys. +ShodanAPIKey string +CensysAPIId string +CensysAPISecret string +ZoomEyeAPIKey string +FOFAEmail string +FOFAAPIKey string +NetlasAPIKey string +BinaryEdgeAPIKey string +``` + +**RegisterAll** (register.go) — add after the Phase 11 paste site registrations: + +```go +// Phase 12: IoT scanner sources. +engine.Register(&ShodanSource{ + APIKey: cfg.ShodanAPIKey, + Registry: reg, + Limiters: lim, +}) +engine.Register(&CensysSource{ + APIId: cfg.CensysAPIId, + APISecret: cfg.CensysAPISecret, + Registry: reg, + Limiters: lim, +}) +engine.Register(&ZoomEyeSource{ + APIKey: cfg.ZoomEyeAPIKey, + Registry: reg, + Limiters: lim, +}) +engine.Register(&FOFASource{ + Email: cfg.FOFAEmail, + APIKey: cfg.FOFAAPIKey, + Registry: reg, + Limiters: lim, +}) +engine.Register(&NetlasSource{ + APIKey: cfg.NetlasAPIKey, + Registry: reg, + Limiters: lim, +}) +engine.Register(&BinaryEdgeSource{ + APIKey: cfg.BinaryEdgeAPIKey, + Registry: reg, + Limiters: lim, +}) + +// Phase 12: Cloud storage sources (credentialless). +engine.Register(&S3Scanner{ + Registry: reg, + Limiters: lim, +}) +engine.Register(&GCSScanner{ + Registry: reg, + Limiters: lim, +}) +engine.Register(&AzureBlobScanner{ + Registry: reg, + Limiters: lim, +}) +engine.Register(&DOSpacesScanner{ + Registry: reg, + Limiters: lim, +}) +``` + +Update the RegisterAll doc comment to say "28 sources total" (18 Phase 10-11 + 10 Phase 12). + +**cmd/recon.go** — in buildReconEngine(), add to the SourcesConfig literal: + +```go +ShodanAPIKey: firstNonEmpty(os.Getenv("SHODAN_API_KEY"), viper.GetString("recon.shodan.api_key")), +CensysAPIId: firstNonEmpty(os.Getenv("CENSYS_API_ID"), viper.GetString("recon.censys.api_id")), +CensysAPISecret: firstNonEmpty(os.Getenv("CENSYS_API_SECRET"), viper.GetString("recon.censys.api_secret")), +ZoomEyeAPIKey: firstNonEmpty(os.Getenv("ZOOMEYE_API_KEY"), viper.GetString("recon.zoomeye.api_key")), +FOFAEmail: firstNonEmpty(os.Getenv("FOFA_EMAIL"), viper.GetString("recon.fofa.email")), +FOFAAPIKey: firstNonEmpty(os.Getenv("FOFA_API_KEY"), viper.GetString("recon.fofa.api_key")), +NetlasAPIKey: firstNonEmpty(os.Getenv("NETLAS_API_KEY"), viper.GetString("recon.netlas.api_key")), +BinaryEdgeAPIKey: firstNonEmpty(os.Getenv("BINARYEDGE_API_KEY"), viper.GetString("recon.binaryedge.api_key")), +``` + +Update the reconCmd Long description to mention Phase 12 sources. + + + cd /home/salva/Documents/apikey/.claude/worktrees/agent-a6700ee2 && go build ./cmd/... + + RegisterAll registers 28 sources; cmd/recon.go wires all Phase 12 credentials from env/viper + + + + Task 2: Integration test for all 28 registered sources + pkg/recon/sources/integration_test.go + + - TestRegisterAll_Phase12 registers all sources, asserts 28 total + - All 10 new source names are present: shodan, censys, zoomeye, fofa, netlas, binaryedge, s3, gcs, azureblob, spaces + - IoT sources with empty credentials report Enabled()==false + - Cloud storage sources (credentialless) report Enabled()==true + - SweepAll with short context timeout completes without panic + + +Extend the existing integration_test.go (which currently tests 18 Phase 10-11 sources): +- Update the expected source count from 18 to 28 +- Add all 10 new source names to the expected names list +- Add assertions that IoT sources (shodan, censys, zoomeye, fofa, netlas, binaryedge) are Enabled()==false when credentials are empty +- Add assertions that cloud sources (s3, gcs, azureblob, spaces) are Enabled()==true (credentialless) +- Keep the existing SweepAll test with short context timeout, verify no panics + + + cd /home/salva/Documents/apikey/.claude/worktrees/agent-a6700ee2 && go test ./pkg/recon/sources/ -run "TestRegisterAll" -v -count=1 + + Integration test passes with 28 registered sources; all Phase 12 source names are discoverable + + + + + +- `go build ./cmd/...` compiles without errors +- `go test ./pkg/recon/sources/ -run "TestRegisterAll" -v` passes with 28 sources +- `go test ./pkg/recon/sources/ -v -count=1` all tests pass (existing + new) + + + +All 10 Phase 12 sources are wired into RegisterAll and discoverable via the recon engine. cmd/recon.go reads credentials from env vars and viper config. Integration test confirms 28 total sources registered. + + + +After completion, create `.planning/phases/12-osint_iot_cloud_storage/12-04-SUMMARY.md` +