docs(12): create phase plan — IoT scanners + cloud storage sources
This commit is contained in:
salvacybersec
187
.planning/phases/12-osint_iot_cloud_storage/12-02-PLAN.md
Normal file
187
.planning/phases/12-osint_iot_cloud_storage/12-02-PLAN.md
Normal file
@@ -0,0 +1,187 @@
|
||||
---
|
||||
phase: 12-osint_iot_cloud_storage
|
||||
plan: 02
|
||||
type: execute
|
||||
wave: 1
|
||||
depends_on: []
|
||||
files_modified:
|
||||
- pkg/recon/sources/fofa.go
|
||||
- pkg/recon/sources/fofa_test.go
|
||||
- pkg/recon/sources/netlas.go
|
||||
- pkg/recon/sources/netlas_test.go
|
||||
- pkg/recon/sources/binaryedge.go
|
||||
- pkg/recon/sources/binaryedge_test.go
|
||||
autonomous: true
|
||||
requirements: [RECON-IOT-04, RECON-IOT-05, RECON-IOT-06]
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "FOFASource searches FOFA API for exposed endpoints and emits findings"
|
||||
- "NetlasSource searches Netlas API for internet-wide scan results and emits findings"
|
||||
- "BinaryEdgeSource searches BinaryEdge API for exposed services and emits findings"
|
||||
- "Each source is disabled when its API key/credentials are empty"
|
||||
artifacts:
|
||||
- path: "pkg/recon/sources/fofa.go"
|
||||
provides: "FOFASource implementing recon.ReconSource"
|
||||
exports: ["FOFASource"]
|
||||
- path: "pkg/recon/sources/netlas.go"
|
||||
provides: "NetlasSource implementing recon.ReconSource"
|
||||
exports: ["NetlasSource"]
|
||||
- path: "pkg/recon/sources/binaryedge.go"
|
||||
provides: "BinaryEdgeSource implementing recon.ReconSource"
|
||||
exports: ["BinaryEdgeSource"]
|
||||
key_links:
|
||||
- from: "pkg/recon/sources/fofa.go"
|
||||
to: "pkg/recon/sources/httpclient.go"
|
||||
via: "sources.Client for retry/backoff HTTP"
|
||||
pattern: "s\\.client\\.Do"
|
||||
- from: "pkg/recon/sources/netlas.go"
|
||||
to: "pkg/recon/sources/httpclient.go"
|
||||
via: "sources.Client for retry/backoff HTTP"
|
||||
pattern: "s\\.client\\.Do"
|
||||
- from: "pkg/recon/sources/binaryedge.go"
|
||||
to: "pkg/recon/sources/httpclient.go"
|
||||
via: "sources.Client for retry/backoff HTTP"
|
||||
pattern: "s\\.client\\.Do"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Implement three IoT scanner recon sources: FOFA, Netlas, and BinaryEdge.
|
||||
|
||||
Purpose: Complete the IoT/device scanner coverage with Chinese (FOFA) and alternative (Netlas, BinaryEdge) internet search engines.
|
||||
Output: Three source files + tests following the established Phase 10 pattern.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@pkg/recon/source.go
|
||||
@pkg/recon/sources/httpclient.go
|
||||
@pkg/recon/sources/bing.go
|
||||
@pkg/recon/sources/queries.go
|
||||
@pkg/recon/sources/register.go
|
||||
|
||||
<interfaces>
|
||||
From pkg/recon/source.go:
|
||||
```go
|
||||
type ReconSource interface {
|
||||
Name() string
|
||||
RateLimit() rate.Limit
|
||||
Burst() int
|
||||
RespectsRobots() bool
|
||||
Enabled(cfg Config) bool
|
||||
Sweep(ctx context.Context, query string, out chan<- Finding) error
|
||||
}
|
||||
```
|
||||
|
||||
From pkg/recon/sources/httpclient.go:
|
||||
```go
|
||||
type Client struct { HTTP *http.Client; MaxRetries int; UserAgent string }
|
||||
func NewClient() *Client
|
||||
func (c *Client) Do(ctx context.Context, req *http.Request) (*http.Response, error)
|
||||
var ErrUnauthorized = errors.New("sources: unauthorized (check credentials)")
|
||||
```
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: Implement FOFASource, NetlasSource, BinaryEdgeSource</name>
|
||||
<files>pkg/recon/sources/fofa.go, pkg/recon/sources/netlas.go, pkg/recon/sources/binaryedge.go</files>
|
||||
<action>
|
||||
Create three source files following the BingDorkSource pattern:
|
||||
|
||||
**FOFASource** (fofa.go):
|
||||
- Struct: `FOFASource` with fields `Email string`, `APIKey string`, `BaseURL string`, `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `client *Client`
|
||||
- Compile-time assertion: `var _ recon.ReconSource = (*FOFASource)(nil)`
|
||||
- Name(): "fofa"
|
||||
- RateLimit(): rate.Every(1 * time.Second) — FOFA allows ~1 req/s
|
||||
- Burst(): 1
|
||||
- RespectsRobots(): false
|
||||
- Enabled(): returns `s.Email != "" && s.APIKey != ""`
|
||||
- BaseURL default: "https://fofa.info"
|
||||
- Sweep(): For each query from BuildQueries, base64-encode the query, then GET `{base}/api/v1/search/all?email={email}&key={apikey}&qbase64={base64query}&size=100`. Parse JSON response `{"results":[["ip","port","protocol","host"],...],"size":N}`. Emit Finding per result with Source=`fmt.Sprintf("fofa://%s:%s", result[0], result[1])`, SourceType="recon:fofa".
|
||||
- Note: FOFA results array contains string arrays, not objects. Each inner array is [host, ip, port].
|
||||
- Add `fofaKeywordIndex` helper.
|
||||
|
||||
**NetlasSource** (netlas.go):
|
||||
- Struct: `NetlasSource` with fields `APIKey string`, `BaseURL string`, `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `client *Client`
|
||||
- Name(): "netlas"
|
||||
- RateLimit(): rate.Every(1 * time.Second)
|
||||
- Burst(): 1
|
||||
- RespectsRobots(): false
|
||||
- Enabled(): returns `s.APIKey != ""`
|
||||
- BaseURL default: "https://app.netlas.io"
|
||||
- Sweep(): For each query, GET `{base}/api/responses/?q={url.QueryEscape(q)}&start=0&indices=`. Set header `X-API-Key: {apikey}`. Parse JSON response `{"items":[{"data":{"ip":"...","port":N}},...]}`. Emit Finding per item with Source=`fmt.Sprintf("netlas://%s:%d", item.Data.IP, item.Data.Port)`.
|
||||
- Add `netlasKeywordIndex` helper.
|
||||
|
||||
**BinaryEdgeSource** (binaryedge.go):
|
||||
- Struct: `BinaryEdgeSource` with fields `APIKey string`, `BaseURL string`, `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `client *Client`
|
||||
- Name(): "binaryedge"
|
||||
- RateLimit(): rate.Every(2 * time.Second) — BinaryEdge free tier is conservative
|
||||
- Burst(): 1
|
||||
- RespectsRobots(): false
|
||||
- Enabled(): returns `s.APIKey != ""`
|
||||
- BaseURL default: "https://api.binaryedge.io"
|
||||
- Sweep(): For each query, GET `{base}/v2/query/search?query={url.QueryEscape(q)}&page=1`. Set header `X-Key: {apikey}`. Parse JSON response `{"events":[{"target":{"ip":"...","port":N}},...]}`. Emit Finding per event with Source=`fmt.Sprintf("binaryedge://%s:%d", event.Target.IP, event.Target.Port)`.
|
||||
- Add `binaryedgeKeywordIndex` helper.
|
||||
|
||||
Update `formatQuery` in queries.go to add cases for "fofa", "netlas", "binaryedge" — all use bare keyword (same as default).
|
||||
|
||||
Same patterns as Plan 12-01: use sources.NewClient(), s.Limiters.Wait before requests, standard error handling.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd /home/salva/Documents/apikey/.claude/worktrees/agent-a6700ee2 && go build ./pkg/recon/sources/</automated>
|
||||
</verify>
|
||||
<done>Three source files compile, each implements recon.ReconSource interface</done>
|
||||
</task>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 2: Unit tests for FOFA, Netlas, BinaryEdge sources</name>
|
||||
<files>pkg/recon/sources/fofa_test.go, pkg/recon/sources/netlas_test.go, pkg/recon/sources/binaryedge_test.go</files>
|
||||
<behavior>
|
||||
- FOFA: httptest server returns mock JSON with 2 results; Sweep emits 2 findings with "recon:fofa" source type
|
||||
- FOFA: empty Email or APIKey => Enabled()==false
|
||||
- Netlas: httptest server returns mock JSON with 2 items; Sweep emits 2 findings with "recon:netlas" source type
|
||||
- Netlas: empty APIKey => Enabled()==false
|
||||
- BinaryEdge: httptest server returns mock JSON with 2 events; Sweep emits 2 findings with "recon:binaryedge" source type
|
||||
- BinaryEdge: empty APIKey => Enabled()==false
|
||||
- All: cancelled context returns context error
|
||||
</behavior>
|
||||
<action>
|
||||
Create test files following the same httptest pattern used in Plan 12-01:
|
||||
- Use httptest.NewServer to mock API responses matching each source's expected JSON shape
|
||||
- Set BaseURL to test server URL
|
||||
- Create a minimal providers.Registry with 1-2 test providers
|
||||
- Verify Finding count, SourceType, and Source URL format
|
||||
- Test disabled state (empty credentials)
|
||||
- Test context cancellation
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd /home/salva/Documents/apikey/.claude/worktrees/agent-a6700ee2 && go test ./pkg/recon/sources/ -run "TestFOFA|TestNetlas|TestBinaryEdge" -v -count=1</automated>
|
||||
</verify>
|
||||
<done>All FOFA, Netlas, BinaryEdge tests pass; each source emits correct findings from mock API responses</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
- `go build ./pkg/recon/sources/` compiles without errors
|
||||
- `go test ./pkg/recon/sources/ -run "TestFOFA|TestNetlas|TestBinaryEdge" -v` all pass
|
||||
- Each source file has compile-time assertion `var _ recon.ReconSource = (*XxxSource)(nil)`
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
Three IoT scanner sources (FOFA, Netlas, BinaryEdge) implement recon.ReconSource, use shared Client for HTTP, respect rate limiting via LimiterRegistry, and pass unit tests with mock API responses.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
After completion, create `.planning/phases/12-osint_iot_cloud_storage/12-02-SUMMARY.md`
|
||||
</output>
|
||||
Reference in New Issue
Block a user