---
phase: 12-osint_iot_cloud_storage
plan: 02
type: execute
wave: 1
depends_on: []
files_modified:
- pkg/recon/sources/fofa.go
- pkg/recon/sources/fofa_test.go
- pkg/recon/sources/netlas.go
- pkg/recon/sources/netlas_test.go
- pkg/recon/sources/binaryedge.go
- pkg/recon/sources/binaryedge_test.go
autonomous: true
requirements: [RECON-IOT-04, RECON-IOT-05, RECON-IOT-06]
must_haves:
truths:
- "FOFASource searches FOFA API for exposed endpoints and emits findings"
- "NetlasSource searches Netlas API for internet-wide scan results and emits findings"
- "BinaryEdgeSource searches BinaryEdge API for exposed services and emits findings"
- "Each source is disabled when its API key/credentials are empty"
artifacts:
- path: "pkg/recon/sources/fofa.go"
provides: "FOFASource implementing recon.ReconSource"
exports: ["FOFASource"]
- path: "pkg/recon/sources/netlas.go"
provides: "NetlasSource implementing recon.ReconSource"
exports: ["NetlasSource"]
- path: "pkg/recon/sources/binaryedge.go"
provides: "BinaryEdgeSource implementing recon.ReconSource"
exports: ["BinaryEdgeSource"]
key_links:
- from: "pkg/recon/sources/fofa.go"
to: "pkg/recon/sources/httpclient.go"
via: "sources.Client for retry/backoff HTTP"
pattern: "s\\.client\\.Do"
- from: "pkg/recon/sources/netlas.go"
to: "pkg/recon/sources/httpclient.go"
via: "sources.Client for retry/backoff HTTP"
pattern: "s\\.client\\.Do"
- from: "pkg/recon/sources/binaryedge.go"
to: "pkg/recon/sources/httpclient.go"
via: "sources.Client for retry/backoff HTTP"
pattern: "s\\.client\\.Do"
---
Implement three IoT scanner recon sources: FOFA, Netlas, and BinaryEdge.
Purpose: Complete the IoT/device scanner coverage with Chinese (FOFA) and alternative (Netlas, BinaryEdge) internet search engines.
Output: Three source files + tests following the established Phase 10 pattern.
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@pkg/recon/source.go
@pkg/recon/sources/httpclient.go
@pkg/recon/sources/bing.go
@pkg/recon/sources/queries.go
@pkg/recon/sources/register.go
From pkg/recon/source.go:
```go
type ReconSource interface {
Name() string
RateLimit() rate.Limit
Burst() int
RespectsRobots() bool
Enabled(cfg Config) bool
Sweep(ctx context.Context, query string, out chan<- Finding) error
}
```
From pkg/recon/sources/httpclient.go:
```go
type Client struct { HTTP *http.Client; MaxRetries int; UserAgent string }
func NewClient() *Client
func (c *Client) Do(ctx context.Context, req *http.Request) (*http.Response, error)
var ErrUnauthorized = errors.New("sources: unauthorized (check credentials)")
```
Task 1: Implement FOFASource, NetlasSource, BinaryEdgeSource
pkg/recon/sources/fofa.go, pkg/recon/sources/netlas.go, pkg/recon/sources/binaryedge.go
Create three source files following the BingDorkSource pattern:
**FOFASource** (fofa.go):
- Struct: `FOFASource` with fields `Email string`, `APIKey string`, `BaseURL string`, `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `client *Client`
- Compile-time assertion: `var _ recon.ReconSource = (*FOFASource)(nil)`
- Name(): "fofa"
- RateLimit(): rate.Every(1 * time.Second) — FOFA allows ~1 req/s
- Burst(): 1
- RespectsRobots(): false
- Enabled(): returns `s.Email != "" && s.APIKey != ""`
- BaseURL default: "https://fofa.info"
- Sweep(): For each query from BuildQueries, base64-encode the query, then GET `{base}/api/v1/search/all?email={email}&key={apikey}&qbase64={base64query}&size=100`. Parse JSON response `{"results":[["ip","port","protocol","host"],...],"size":N}`. Emit Finding per result with Source=`fmt.Sprintf("fofa://%s:%s", result[0], result[1])`, SourceType="recon:fofa".
- Note: FOFA results array contains string arrays, not objects. Each inner array is [host, ip, port].
- Add `fofaKeywordIndex` helper.
**NetlasSource** (netlas.go):
- Struct: `NetlasSource` with fields `APIKey string`, `BaseURL string`, `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `client *Client`
- Name(): "netlas"
- RateLimit(): rate.Every(1 * time.Second)
- Burst(): 1
- RespectsRobots(): false
- Enabled(): returns `s.APIKey != ""`
- BaseURL default: "https://app.netlas.io"
- Sweep(): For each query, GET `{base}/api/responses/?q={url.QueryEscape(q)}&start=0&indices=`. Set header `X-API-Key: {apikey}`. Parse JSON response `{"items":[{"data":{"ip":"...","port":N}},...]}`. Emit Finding per item with Source=`fmt.Sprintf("netlas://%s:%d", item.Data.IP, item.Data.Port)`.
- Add `netlasKeywordIndex` helper.
**BinaryEdgeSource** (binaryedge.go):
- Struct: `BinaryEdgeSource` with fields `APIKey string`, `BaseURL string`, `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `client *Client`
- Name(): "binaryedge"
- RateLimit(): rate.Every(2 * time.Second) — BinaryEdge free tier is conservative
- Burst(): 1
- RespectsRobots(): false
- Enabled(): returns `s.APIKey != ""`
- BaseURL default: "https://api.binaryedge.io"
- Sweep(): For each query, GET `{base}/v2/query/search?query={url.QueryEscape(q)}&page=1`. Set header `X-Key: {apikey}`. Parse JSON response `{"events":[{"target":{"ip":"...","port":N}},...]}`. Emit Finding per event with Source=`fmt.Sprintf("binaryedge://%s:%d", event.Target.IP, event.Target.Port)`.
- Add `binaryedgeKeywordIndex` helper.
Update `formatQuery` in queries.go to add cases for "fofa", "netlas", "binaryedge" — all use bare keyword (same as default).
Same patterns as Plan 12-01: use sources.NewClient(), s.Limiters.Wait before requests, standard error handling.
cd /home/salva/Documents/apikey/.claude/worktrees/agent-a6700ee2 && go build ./pkg/recon/sources/
Three source files compile, each implements recon.ReconSource interface
Task 2: Unit tests for FOFA, Netlas, BinaryEdge sources
pkg/recon/sources/fofa_test.go, pkg/recon/sources/netlas_test.go, pkg/recon/sources/binaryedge_test.go
- FOFA: httptest server returns mock JSON with 2 results; Sweep emits 2 findings with "recon:fofa" source type
- FOFA: empty Email or APIKey => Enabled()==false
- Netlas: httptest server returns mock JSON with 2 items; Sweep emits 2 findings with "recon:netlas" source type
- Netlas: empty APIKey => Enabled()==false
- BinaryEdge: httptest server returns mock JSON with 2 events; Sweep emits 2 findings with "recon:binaryedge" source type
- BinaryEdge: empty APIKey => Enabled()==false
- All: cancelled context returns context error
Create test files following the same httptest pattern used in Plan 12-01:
- Use httptest.NewServer to mock API responses matching each source's expected JSON shape
- Set BaseURL to test server URL
- Create a minimal providers.Registry with 1-2 test providers
- Verify Finding count, SourceType, and Source URL format
- Test disabled state (empty credentials)
- Test context cancellation
cd /home/salva/Documents/apikey/.claude/worktrees/agent-a6700ee2 && go test ./pkg/recon/sources/ -run "TestFOFA|TestNetlas|TestBinaryEdge" -v -count=1
All FOFA, Netlas, BinaryEdge tests pass; each source emits correct findings from mock API responses
- `go build ./pkg/recon/sources/` compiles without errors
- `go test ./pkg/recon/sources/ -run "TestFOFA|TestNetlas|TestBinaryEdge" -v` all pass
- Each source file has compile-time assertion `var _ recon.ReconSource = (*XxxSource)(nil)`
Three IoT scanner sources (FOFA, Netlas, BinaryEdge) implement recon.ReconSource, use shared Client for HTTP, respect rate limiting via LimiterRegistry, and pass unit tests with mock API responses.