docs(12): create phase plan — IoT scanners + cloud storage sources
This commit is contained in:
salvacybersec
193
.planning/phases/12-osint_iot_cloud_storage/12-01-PLAN.md
Normal file
193
.planning/phases/12-osint_iot_cloud_storage/12-01-PLAN.md
Normal file
@@ -0,0 +1,193 @@
|
||||
---
|
||||
phase: 12-osint_iot_cloud_storage
|
||||
plan: 01
|
||||
type: execute
|
||||
wave: 1
|
||||
depends_on: []
|
||||
files_modified:
|
||||
- pkg/recon/sources/shodan.go
|
||||
- pkg/recon/sources/shodan_test.go
|
||||
- pkg/recon/sources/censys.go
|
||||
- pkg/recon/sources/censys_test.go
|
||||
- pkg/recon/sources/zoomeye.go
|
||||
- pkg/recon/sources/zoomeye_test.go
|
||||
autonomous: true
|
||||
requirements: [RECON-IOT-01, RECON-IOT-02, RECON-IOT-03]
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "ShodanSource searches Shodan /shodan/host/search for exposed LLM endpoints and emits findings"
|
||||
- "CensysSource searches Censys v2 /hosts/search for exposed services and emits findings"
|
||||
- "ZoomEyeSource searches ZoomEye /host/search for device/service key exposure and emits findings"
|
||||
- "Each source is disabled (Enabled==false) when its API key is empty"
|
||||
artifacts:
|
||||
- path: "pkg/recon/sources/shodan.go"
|
||||
provides: "ShodanSource implementing recon.ReconSource"
|
||||
exports: ["ShodanSource"]
|
||||
- path: "pkg/recon/sources/censys.go"
|
||||
provides: "CensysSource implementing recon.ReconSource"
|
||||
exports: ["CensysSource"]
|
||||
- path: "pkg/recon/sources/zoomeye.go"
|
||||
provides: "ZoomEyeSource implementing recon.ReconSource"
|
||||
exports: ["ZoomEyeSource"]
|
||||
key_links:
|
||||
- from: "pkg/recon/sources/shodan.go"
|
||||
to: "pkg/recon/sources/httpclient.go"
|
||||
via: "sources.Client for retry/backoff HTTP"
|
||||
pattern: "s\\.client\\.Do"
|
||||
- from: "pkg/recon/sources/censys.go"
|
||||
to: "pkg/recon/sources/httpclient.go"
|
||||
via: "sources.Client for retry/backoff HTTP"
|
||||
pattern: "s\\.client\\.Do"
|
||||
- from: "pkg/recon/sources/zoomeye.go"
|
||||
to: "pkg/recon/sources/httpclient.go"
|
||||
via: "sources.Client for retry/backoff HTTP"
|
||||
pattern: "s\\.client\\.Do"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Implement three IoT scanner recon sources: Shodan, Censys, and ZoomEye.
|
||||
|
||||
Purpose: Enable discovery of exposed LLM endpoints (vLLM, Ollama, LiteLLM proxies) via internet-wide device scanners.
|
||||
Output: Three source files + tests following the established Phase 10 pattern.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@pkg/recon/source.go
|
||||
@pkg/recon/sources/httpclient.go
|
||||
@pkg/recon/sources/github.go
|
||||
@pkg/recon/sources/bing.go
|
||||
@pkg/recon/sources/queries.go
|
||||
@pkg/recon/sources/register.go
|
||||
|
||||
<interfaces>
|
||||
From pkg/recon/source.go:
|
||||
```go
|
||||
type ReconSource interface {
|
||||
Name() string
|
||||
RateLimit() rate.Limit
|
||||
Burst() int
|
||||
RespectsRobots() bool
|
||||
Enabled(cfg Config) bool
|
||||
Sweep(ctx context.Context, query string, out chan<- Finding) error
|
||||
}
|
||||
```
|
||||
|
||||
From pkg/recon/sources/httpclient.go:
|
||||
```go
|
||||
type Client struct { HTTP *http.Client; MaxRetries int; UserAgent string }
|
||||
func NewClient() *Client
|
||||
func (c *Client) Do(ctx context.Context, req *http.Request) (*http.Response, error)
|
||||
var ErrUnauthorized = errors.New("sources: unauthorized (check credentials)")
|
||||
```
|
||||
|
||||
From pkg/recon/sources/queries.go:
|
||||
```go
|
||||
func BuildQueries(reg *providers.Registry, source string) []string
|
||||
```
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: Implement ShodanSource, CensysSource, ZoomEyeSource</name>
|
||||
<files>pkg/recon/sources/shodan.go, pkg/recon/sources/censys.go, pkg/recon/sources/zoomeye.go</files>
|
||||
<action>
|
||||
Create three source files following the BingDorkSource pattern exactly:
|
||||
|
||||
**ShodanSource** (shodan.go):
|
||||
- Struct: `ShodanSource` with fields `APIKey string`, `BaseURL string`, `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `client *Client`
|
||||
- Compile-time assertion: `var _ recon.ReconSource = (*ShodanSource)(nil)`
|
||||
- Name(): "shodan"
|
||||
- RateLimit(): rate.Every(1 * time.Second) — Shodan allows ~1 req/s on most plans
|
||||
- Burst(): 1
|
||||
- RespectsRobots(): false (authenticated REST API)
|
||||
- Enabled(): returns `s.APIKey != ""`
|
||||
- BaseURL default: "https://api.shodan.io"
|
||||
- Sweep(): For each query from BuildQueries(s.Registry, "shodan"), call GET `{base}/shodan/host/search?key={apikey}&query={url.QueryEscape(q)}`. Parse JSON response `{"matches":[{"ip_str":"...","port":N,"data":"..."},...]}`. Emit a Finding per match with Source=`fmt.Sprintf("shodan://%s:%d", match.IPStr, match.Port)`, SourceType="recon:shodan", Confidence="low", ProviderName from keyword index.
|
||||
- Add `shodanKeywordIndex` helper (same pattern as bingKeywordIndex).
|
||||
- Error handling: ErrUnauthorized aborts, context cancellation aborts, transient errors continue.
|
||||
|
||||
**CensysSource** (censys.go):
|
||||
- Struct: `CensysSource` with fields `APIId string`, `APISecret string`, `BaseURL string`, `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `client *Client`
|
||||
- Name(): "censys"
|
||||
- RateLimit(): rate.Every(2500 * time.Millisecond) — Censys free tier is 0.4 req/s
|
||||
- Burst(): 1
|
||||
- RespectsRobots(): false
|
||||
- Enabled(): returns `s.APIId != "" && s.APISecret != ""`
|
||||
- BaseURL default: "https://search.censys.io/api"
|
||||
- Sweep(): For each query, POST `{base}/v2/hosts/search` with JSON body `{"q":q,"per_page":25}`. Set Basic Auth header using APIId:APISecret. Parse JSON response `{"result":{"hits":[{"ip":"...","services":[{"port":N,"service_name":"..."}]}]}}`. Emit Finding per hit with Source=`fmt.Sprintf("censys://%s", hit.IP)`.
|
||||
- Add `censysKeywordIndex` helper.
|
||||
|
||||
**ZoomEyeSource** (zoomeye.go):
|
||||
- Struct: `ZoomEyeSource` with fields `APIKey string`, `BaseURL string`, `Registry *providers.Registry`, `Limiters *recon.LimiterRegistry`, `client *Client`
|
||||
- Name(): "zoomeye"
|
||||
- RateLimit(): rate.Every(2 * time.Second)
|
||||
- Burst(): 1
|
||||
- RespectsRobots(): false
|
||||
- Enabled(): returns `s.APIKey != ""`
|
||||
- BaseURL default: "https://api.zoomeye.org" (ZoomEye uses v1-style API key in header)
|
||||
- Sweep(): For each query, GET `{base}/host/search?query={url.QueryEscape(q)}&page=1`. Set header `API-KEY: {apikey}`. Parse JSON response `{"matches":[{"ip":"...","portinfo":{"port":N},"banner":"..."}]}`. Emit Finding per match with Source=`fmt.Sprintf("zoomeye://%s:%d", match.IP, match.PortInfo.Port)`.
|
||||
- Add `zoomeyeKeywordIndex` helper.
|
||||
|
||||
Update `formatQuery` in queries.go to add cases for "shodan", "censys", "zoomeye" — all use bare keyword (same as default).
|
||||
|
||||
All sources must use `sources.NewClient()` for HTTP, `s.Limiters.Wait(ctx, s.Name(), ...)` before each request, and follow the same error handling pattern as BingDorkSource.Sweep.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd /home/salva/Documents/apikey/.claude/worktrees/agent-a6700ee2 && go build ./pkg/recon/sources/</automated>
|
||||
</verify>
|
||||
<done>Three source files compile, each implements recon.ReconSource interface</done>
|
||||
</task>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 2: Unit tests for Shodan, Censys, ZoomEye sources</name>
|
||||
<files>pkg/recon/sources/shodan_test.go, pkg/recon/sources/censys_test.go, pkg/recon/sources/zoomeye_test.go</files>
|
||||
<behavior>
|
||||
- Shodan: httptest server returns mock JSON with 2 matches; Sweep emits 2 findings with "recon:shodan" source type
|
||||
- Shodan: empty API key => Enabled()==false, Sweep returns nil with 0 findings
|
||||
- Censys: httptest server returns mock JSON with 2 hits; Sweep emits 2 findings with "recon:censys" source type
|
||||
- Censys: empty APIId => Enabled()==false
|
||||
- ZoomEye: httptest server returns mock JSON with 2 matches; Sweep emits 2 findings with "recon:zoomeye" source type
|
||||
- ZoomEye: empty API key => Enabled()==false
|
||||
- All: cancelled context returns context error
|
||||
</behavior>
|
||||
<action>
|
||||
Create test files following the pattern in github_test.go / bing_test.go:
|
||||
- Use httptest.NewServer to mock API responses
|
||||
- Set BaseURL to test server URL
|
||||
- Create a minimal providers.Registry with 1-2 test providers containing keywords
|
||||
- Verify Finding count, SourceType, and Source URL format
|
||||
- Test disabled state (empty credentials)
|
||||
- Test context cancellation
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd /home/salva/Documents/apikey/.claude/worktrees/agent-a6700ee2 && go test ./pkg/recon/sources/ -run "TestShodan|TestCensys|TestZoomEye" -v -count=1</automated>
|
||||
</verify>
|
||||
<done>All Shodan, Censys, ZoomEye tests pass; each source emits correct findings from mock API responses</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
- `go build ./pkg/recon/sources/` compiles without errors
|
||||
- `go test ./pkg/recon/sources/ -run "TestShodan|TestCensys|TestZoomEye" -v` all pass
|
||||
- Each source file has compile-time assertion `var _ recon.ReconSource = (*XxxSource)(nil)`
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
Three IoT scanner sources (Shodan, Censys, ZoomEye) implement recon.ReconSource, use shared Client for HTTP, respect rate limiting via LimiterRegistry, and pass unit tests with mock API responses.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
After completion, create `.planning/phases/12-osint_iot_cloud_storage/12-01-SUMMARY.md`
|
||||
</output>
|
||||
Reference in New Issue
Block a user