--- phase: 12-osint_iot_cloud_storage plan: 02 type: execute wave: 1 depends_on: [] files_modified: - pkg/recon/sources/fofa.go - pkg/recon/sources/fofa_test.go - pkg/recon/sources/netlas.go - pkg/recon/sources/netlas_test.go - pkg/recon/sources/binaryedge.go - pkg/recon/sources/binaryedge_test.go autonomous: true requirements: [RECON-IOT-04, RECON-IOT-05, RECON-IOT-06] must_haves: truths: - "FOFASource searches FOFA API for exposed endpoints and emits findings" - "NetlasSource searches Netlas API for internet-wide scan results and emits findings" - "BinaryEdgeSource searches BinaryEdge API for exposed services and emits findings" - "Each source is disabled when its API key/credentials are empty" artifacts: - path: "pkg/recon/sources/fofa.go" provides: "FOFASource implementing recon.ReconSource" exports: ["FOFASource"] - path: "pkg/recon/sources/netlas.go" provides: "NetlasSource implementing recon.ReconSource" exports: ["NetlasSource"] - path: "pkg/recon/sources/binaryedge.go" provides: "BinaryEdgeSource implementing recon.ReconSource" exports: ["BinaryEdgeSource"] key_links: - from: "pkg/recon/sources/fofa.go" to: "pkg/recon/sources/httpclient.go" via: "sources.Client for retry/backoff HTTP" pattern: "s\\.client\\.Do" - from: "pkg/recon/sources/netlas.go" to: "pkg/recon/sources/httpclient.go" via: "sources.Client for retry/backoff HTTP" pattern: "s\\.client\\.Do" - from: "pkg/recon/sources/binaryedge.go" to: "pkg/recon/sources/httpclient.go" via: "sources.Client for retry/backoff HTTP" pattern: "s\\.client\\.Do" --- Implement three IoT scanner recon sources: FOFA, Netlas, and BinaryEdge. Purpose: Complete the IoT/device scanner coverage with Chinese (FOFA) and alternative (Netlas, BinaryEdge) internet search engines. Output: Three source files + tests following the established Phase 10 pattern. @$HOME/.claude/get-shit-done/workflows/execute-plan.md @$HOME/.claude/get-shit-done/templates/summary.md @.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @pkg/recon/source.go @pkg/recon/sources/httpclient.go @pkg/recon/sources/bing.go @pkg/recon/sources/queries.go @pkg/recon/sources/register.go From pkg/recon/source.go: ```go type ReconSource interface { Name() string RateLimit() rate.Limit Burst() int RespectsRobots() bool Enabled(cfg Config) bool Sweep(ctx context.Context, query string, out chan<- Finding) error } ``` From pkg/recon/sources/httpclient.go: ```go type Client struct { HTTP *http.Client; MaxRetries int; UserAgent string } func NewClient() *Client func (c *Client) Do(ctx context.Context, req *http.Request) (*http.Response, error) var ErrUnauthorized = errors.New("sources: unauthorized (check credentials)") ``` Task 1: Implement FOFASource, NetlasSource, BinaryEdgeSource pkg/recon/sources/fofa.go, pkg/recon/sources/netlas.go, pkg/recon/sources/binaryedge.go Create three source files following the BingDorkSource pattern: **FOFASource** (fofa.go): - Struct: `FOFASource` with fields `Email string`, `APIKey string`, `BaseURL string`, `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `client *Client` - Compile-time assertion: `var _ recon.ReconSource = (*FOFASource)(nil)` - Name(): "fofa" - RateLimit(): rate.Every(1 * time.Second) — FOFA allows ~1 req/s - Burst(): 1 - RespectsRobots(): false - Enabled(): returns `s.Email != "" && s.APIKey != ""` - BaseURL default: "https://fofa.info" - Sweep(): For each query from BuildQueries, base64-encode the query, then GET `{base}/api/v1/search/all?email={email}&key={apikey}&qbase64={base64query}&size=100`. Parse JSON response `{"results":[["ip","port","protocol","host"],...],"size":N}`. Emit Finding per result with Source=`fmt.Sprintf("fofa://%s:%s", result[0], result[1])`, SourceType="recon:fofa". - Note: FOFA results array contains string arrays, not objects. Each inner array is [host, ip, port]. - Add `fofaKeywordIndex` helper. **NetlasSource** (netlas.go): - Struct: `NetlasSource` with fields `APIKey string`, `BaseURL string`, `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `client *Client` - Name(): "netlas" - RateLimit(): rate.Every(1 * time.Second) - Burst(): 1 - RespectsRobots(): false - Enabled(): returns `s.APIKey != ""` - BaseURL default: "https://app.netlas.io" - Sweep(): For each query, GET `{base}/api/responses/?q={url.QueryEscape(q)}&start=0&indices=`. Set header `X-API-Key: {apikey}`. Parse JSON response `{"items":[{"data":{"ip":"...","port":N}},...]}`. Emit Finding per item with Source=`fmt.Sprintf("netlas://%s:%d", item.Data.IP, item.Data.Port)`. - Add `netlasKeywordIndex` helper. **BinaryEdgeSource** (binaryedge.go): - Struct: `BinaryEdgeSource` with fields `APIKey string`, `BaseURL string`, `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `client *Client` - Name(): "binaryedge" - RateLimit(): rate.Every(2 * time.Second) — BinaryEdge free tier is conservative - Burst(): 1 - RespectsRobots(): false - Enabled(): returns `s.APIKey != ""` - BaseURL default: "https://api.binaryedge.io" - Sweep(): For each query, GET `{base}/v2/query/search?query={url.QueryEscape(q)}&page=1`. Set header `X-Key: {apikey}`. Parse JSON response `{"events":[{"target":{"ip":"...","port":N}},...]}`. Emit Finding per event with Source=`fmt.Sprintf("binaryedge://%s:%d", event.Target.IP, event.Target.Port)`. - Add `binaryedgeKeywordIndex` helper. Update `formatQuery` in queries.go to add cases for "fofa", "netlas", "binaryedge" — all use bare keyword (same as default). Same patterns as Plan 12-01: use sources.NewClient(), s.Limiters.Wait before requests, standard error handling. cd /home/salva/Documents/apikey/.claude/worktrees/agent-a6700ee2 && go build ./pkg/recon/sources/ Three source files compile, each implements recon.ReconSource interface Task 2: Unit tests for FOFA, Netlas, BinaryEdge sources pkg/recon/sources/fofa_test.go, pkg/recon/sources/netlas_test.go, pkg/recon/sources/binaryedge_test.go - FOFA: httptest server returns mock JSON with 2 results; Sweep emits 2 findings with "recon:fofa" source type - FOFA: empty Email or APIKey => Enabled()==false - Netlas: httptest server returns mock JSON with 2 items; Sweep emits 2 findings with "recon:netlas" source type - Netlas: empty APIKey => Enabled()==false - BinaryEdge: httptest server returns mock JSON with 2 events; Sweep emits 2 findings with "recon:binaryedge" source type - BinaryEdge: empty APIKey => Enabled()==false - All: cancelled context returns context error Create test files following the same httptest pattern used in Plan 12-01: - Use httptest.NewServer to mock API responses matching each source's expected JSON shape - Set BaseURL to test server URL - Create a minimal providers.Registry with 1-2 test providers - Verify Finding count, SourceType, and Source URL format - Test disabled state (empty credentials) - Test context cancellation cd /home/salva/Documents/apikey/.claude/worktrees/agent-a6700ee2 && go test ./pkg/recon/sources/ -run "TestFOFA|TestNetlas|TestBinaryEdge" -v -count=1 All FOFA, Netlas, BinaryEdge tests pass; each source emits correct findings from mock API responses - `go build ./pkg/recon/sources/` compiles without errors - `go test ./pkg/recon/sources/ -run "TestFOFA|TestNetlas|TestBinaryEdge" -v` all pass - Each source file has compile-time assertion `var _ recon.ReconSource = (*XxxSource)(nil)` Three IoT scanner sources (FOFA, Netlas, BinaryEdge) implement recon.ReconSource, use shared Client for HTTP, respect rate limiting via LimiterRegistry, and pass unit tests with mock API responses. After completion, create `.planning/phases/12-osint_iot_cloud_storage/12-02-SUMMARY.md`