docs(11): create phase plan — 3 plans for search engine dorking + paste sites
This commit is contained in:
salvacybersec
221
.planning/phases/11-osint_search_paste/11-03-PLAN.md
Normal file
221
.planning/phases/11-osint_search_paste/11-03-PLAN.md
Normal file
@@ -0,0 +1,221 @@
|
||||
---
|
||||
phase: 11-osint-search-paste
|
||||
plan: 03
|
||||
type: execute
|
||||
wave: 2
|
||||
depends_on: ["11-01", "11-02"]
|
||||
files_modified:
|
||||
- pkg/recon/sources/register.go
|
||||
- pkg/recon/sources/register_test.go
|
||||
- pkg/recon/sources/integration_test.go
|
||||
- cmd/recon.go
|
||||
autonomous: true
|
||||
requirements: [RECON-DORK-01, RECON-DORK-02, RECON-DORK-03, RECON-PASTE-01]
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "RegisterAll wires all 8 new Phase 11 sources onto the recon engine alongside the 10 Phase 10 sources"
|
||||
- "cmd/recon.go reads Google/Bing/Yandex/Brave API keys from env vars and viper config"
|
||||
- "keyhunter recon list shows all 18 sources (10 Phase 10 + 8 Phase 11)"
|
||||
- "Integration test with httptest fixtures proves SweepAll emits findings from all 18 source types"
|
||||
- "Sources with missing credentials are registered but Enabled()==false"
|
||||
artifacts:
|
||||
- path: "pkg/recon/sources/register.go"
|
||||
provides: "RegisterAll extended with Phase 11 sources"
|
||||
contains: "GoogleDorkSource"
|
||||
- path: "pkg/recon/sources/register_test.go"
|
||||
provides: "Guardrail test asserting 18 sources registered"
|
||||
contains: "18"
|
||||
- path: "pkg/recon/sources/integration_test.go"
|
||||
provides: "SweepAll integration test covering all 18 sources"
|
||||
contains: "recon:google"
|
||||
- path: "cmd/recon.go"
|
||||
provides: "Credential wiring for search engine API keys"
|
||||
contains: "GoogleAPIKey"
|
||||
key_links:
|
||||
- from: "pkg/recon/sources/register.go"
|
||||
to: "pkg/recon/sources/google.go"
|
||||
via: "RegisterAll calls engine.Register(GoogleDorkSource)"
|
||||
pattern: "GoogleDorkSource"
|
||||
- from: "cmd/recon.go"
|
||||
to: "pkg/recon/sources/register.go"
|
||||
via: "SourcesConfig credential fields"
|
||||
pattern: "GoogleAPIKey|GoogleCX|BingAPIKey|YandexUser|YandexAPIKey|BraveAPIKey"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Wire all 8 Phase 11 sources into RegisterAll, extend SourcesConfig with search engine credentials, update cmd/recon.go for env/viper credential lookup, and create the integration test proving all 18 sources work end-to-end via SweepAll.
|
||||
|
||||
Purpose: Complete Phase 11 by connecting all new sources to the engine and proving the full 18-source sweep works.
|
||||
Output: Updated register.go, register_test.go, integration_test.go, cmd/recon.go.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@pkg/recon/sources/register.go
|
||||
@pkg/recon/sources/register_test.go
|
||||
@pkg/recon/sources/integration_test.go
|
||||
@cmd/recon.go
|
||||
|
||||
<interfaces>
|
||||
From pkg/recon/sources/register.go (current):
|
||||
```go
|
||||
type SourcesConfig struct {
|
||||
GitHubToken string
|
||||
GitLabToken string
|
||||
BitbucketToken string
|
||||
BitbucketWorkspace string
|
||||
CodebergToken string
|
||||
HuggingFaceToken string
|
||||
KaggleUser string
|
||||
KaggleKey string
|
||||
Registry *providers.Registry
|
||||
Limiters *recon.LimiterRegistry
|
||||
}
|
||||
func RegisterAll(engine *recon.Engine, cfg SourcesConfig)
|
||||
```
|
||||
|
||||
From cmd/recon.go (current):
|
||||
```go
|
||||
func buildReconEngine() *recon.Engine // constructs SourcesConfig, calls RegisterAll
|
||||
func firstNonEmpty(a, b string) string
|
||||
```
|
||||
|
||||
New sources from Plan 11-01 (to be registered):
|
||||
```go
|
||||
type GoogleDorkSource struct { APIKey, CX, BaseURL string; Registry; Limiters; client }
|
||||
type BingDorkSource struct { APIKey, BaseURL string; Registry; Limiters; client }
|
||||
type DuckDuckGoSource struct { BaseURL string; Registry; Limiters; Client }
|
||||
type YandexSource struct { User, APIKey, BaseURL string; Registry; Limiters; client }
|
||||
type BraveSource struct { APIKey, BaseURL string; Registry; Limiters; client }
|
||||
```
|
||||
|
||||
New sources from Plan 11-02 (to be registered):
|
||||
```go
|
||||
type PastebinSource struct { BaseURL string; Registry; Limiters; Client }
|
||||
type GistPasteSource struct { BaseURL string; Registry; Limiters; Client }
|
||||
type PasteSitesSource struct { Platforms; BaseURL string; Registry; Limiters; Client }
|
||||
```
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 1: Extend SourcesConfig + RegisterAll + cmd/recon.go credential wiring</name>
|
||||
<files>pkg/recon/sources/register.go, pkg/recon/sources/register_test.go, cmd/recon.go</files>
|
||||
<behavior>
|
||||
- SourcesConfig gains 6 new fields: GoogleAPIKey, GoogleCX, BingAPIKey, YandexUser, YandexAPIKey, BraveAPIKey
|
||||
- RegisterAll registers 18 sources total (10 Phase 10 + 8 Phase 11)
|
||||
- RegisterAll with nil engine is still a no-op
|
||||
- TestRegisterAll_WiresAllEighteenSources asserts eng.List() contains all 18 names sorted
|
||||
- TestRegisterAll_MissingCredsStillRegistered asserts 18 sources with empty config
|
||||
- buildReconEngine reads: GOOGLE_API_KEY / recon.google.api_key, GOOGLE_CX / recon.google.cx, BING_API_KEY / recon.bing.api_key, YANDEX_USER / recon.yandex.user, YANDEX_API_KEY / recon.yandex.api_key, BRAVE_API_KEY / recon.brave.api_key
|
||||
- reconCmd Long description updated to mention Phase 11 sources
|
||||
</behavior>
|
||||
<action>
|
||||
Update `pkg/recon/sources/register.go`:
|
||||
- Add to SourcesConfig: GoogleAPIKey, GoogleCX, BingAPIKey, YandexUser, YandexAPIKey, BraveAPIKey (all string)
|
||||
- Add Phase 11 registrations to RegisterAll after the Phase 10 block:
|
||||
```
|
||||
// Phase 11: Search engine dorking sources.
|
||||
engine.Register(&GoogleDorkSource{APIKey: cfg.GoogleAPIKey, CX: cfg.GoogleCX, Registry: reg, Limiters: lim})
|
||||
engine.Register(&BingDorkSource{APIKey: cfg.BingAPIKey, Registry: reg, Limiters: lim})
|
||||
engine.Register(&DuckDuckGoSource{Registry: reg, Limiters: lim})
|
||||
engine.Register(&YandexSource{User: cfg.YandexUser, APIKey: cfg.YandexAPIKey, Registry: reg, Limiters: lim})
|
||||
engine.Register(&BraveSource{APIKey: cfg.BraveAPIKey, Registry: reg, Limiters: lim})
|
||||
|
||||
// Phase 11: Paste site sources.
|
||||
engine.Register(&PastebinSource{Registry: reg, Limiters: lim})
|
||||
engine.Register(&GistPasteSource{Registry: reg, Limiters: lim})
|
||||
engine.Register(&PasteSitesSource{Registry: reg, Limiters: lim})
|
||||
```
|
||||
- Update doc comment on RegisterAll to say "Phase 10 + Phase 11" and total "18 sources"
|
||||
|
||||
Update `pkg/recon/sources/register_test.go`:
|
||||
- TestRegisterAll_WiresAllEighteenSources: want list = sorted 18 names: ["bing", "bitbucket", "brave", "codeberg", "codesandbox", "duckduckgo", "gist", "gistpaste", "github", "gitlab", "google", "huggingface", "kaggle", "pastebin", "pastesites", "replit", "sandboxes", "yandex"]
|
||||
- TestRegisterAll_MissingCredsStillRegistered: assert n == 18
|
||||
|
||||
Update `cmd/recon.go`:
|
||||
- Add to SourcesConfig construction in buildReconEngine():
|
||||
GoogleAPIKey: firstNonEmpty(os.Getenv("GOOGLE_API_KEY"), viper.GetString("recon.google.api_key")),
|
||||
GoogleCX: firstNonEmpty(os.Getenv("GOOGLE_CX"), viper.GetString("recon.google.cx")),
|
||||
BingAPIKey: firstNonEmpty(os.Getenv("BING_API_KEY"), viper.GetString("recon.bing.api_key")),
|
||||
YandexUser: firstNonEmpty(os.Getenv("YANDEX_USER"), viper.GetString("recon.yandex.user")),
|
||||
YandexAPIKey: firstNonEmpty(os.Getenv("YANDEX_API_KEY"), viper.GetString("recon.yandex.api_key")),
|
||||
BraveAPIKey: firstNonEmpty(os.Getenv("BRAVE_API_KEY"), viper.GetString("recon.brave.api_key")),
|
||||
- Update reconCmd.Long to list Phase 11 sources
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd /home/salva/Documents/apikey && go test ./pkg/recon/sources/ -run "TestRegisterAll" -v -count=1 && go build ./cmd/...</automated>
|
||||
</verify>
|
||||
<done>RegisterAll registers 18 sources. cmd/recon.go compiles with credential wiring. Guardrail tests pass.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 2: Integration test -- SweepAll across all 18 sources</name>
|
||||
<files>pkg/recon/sources/integration_test.go</files>
|
||||
<behavior>
|
||||
- TestIntegration_AllSources_SweepAll registers all 18 sources with BaseURL overrides pointing at an httptest mux
|
||||
- SweepAll returns findings from all 18 SourceType values
|
||||
- Each SourceType (recon:github, recon:gitlab, ..., recon:google, recon:bing, recon:duckduckgo, recon:yandex, recon:brave, recon:pastebin, recon:gistpaste, recon:pastesites) has at least 1 finding
|
||||
</behavior>
|
||||
<action>
|
||||
Update `pkg/recon/sources/integration_test.go`:
|
||||
- Extend the existing httptest mux with handlers for the 8 new sources:
|
||||
|
||||
Google Custom Search: mux.HandleFunc("/customsearch/v1", ...) serves JSON `{"items":[{"link":"https://pastebin.com/abc123","title":"leak","snippet":"sk-proj-xxx"}]}`
|
||||
|
||||
Bing Web Search: mux.HandleFunc("/v7.0/search", ...) serves JSON `{"webPages":{"value":[{"url":"https://example.com/leak","name":"leak"}]}}`
|
||||
|
||||
DuckDuckGo HTML: mux.HandleFunc("/html/", ...) serves HTML with `<a class="result__a" href="https://example.com/ddg-leak">result</a>`
|
||||
|
||||
Yandex XML: mux.HandleFunc("/search/xml", ...) serves XML `<yandexsearch><response><results><grouping><group><doc><url>https://example.com/yandex-leak</url></doc></group></grouping></results></response></yandexsearch>`
|
||||
|
||||
Brave Search: mux.HandleFunc("/res/v1/web/search", ...) serves JSON `{"web":{"results":[{"url":"https://example.com/brave-leak","title":"leak"}]}}`
|
||||
|
||||
Pastebin search + raw: mux.HandleFunc("/pastebin-search", ...) serves HTML with paste links; mux.HandleFunc("/pastebin-raw/", ...) serves raw content with "sk-proj-ABC"
|
||||
|
||||
GistPaste search + raw: mux.HandleFunc("/gistpaste-search", ...) serves HTML with gist links; mux.HandleFunc("/gistpaste-raw/", ...) serves raw content with keyword
|
||||
|
||||
PasteSites: mux.HandleFunc("/pastesites-search", ...) + mux.HandleFunc("/pastesites-raw/", ...) similar pattern
|
||||
|
||||
Register all 18 sources on the engine with BaseURL=srv.URL, appropriate credentials for API sources (fake tokens). Then call eng.SweepAll and assert byType map has all 18 SourceType keys.
|
||||
|
||||
Update wantTypes to include: "recon:google", "recon:bing", "recon:duckduckgo", "recon:yandex", "recon:brave", "recon:pastebin", "recon:gistpaste", "recon:pastesites"
|
||||
|
||||
Keep the existing 10 Phase 10 source fixtures and registrations intact.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd /home/salva/Documents/apikey && go test ./pkg/recon/sources/ -run "TestIntegration_AllSources" -v -count=1 -timeout=60s</automated>
|
||||
</verify>
|
||||
<done>Integration test proves SweepAll emits findings from all 18 sources. Full Phase 11 wiring confirmed end-to-end.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
Full Phase 11 verification:
|
||||
```bash
|
||||
cd /home/salva/Documents/apikey && go test ./pkg/recon/sources/ -v -count=1 -timeout=120s && go build ./cmd/...
|
||||
```
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- RegisterAll registers 18 sources (10 Phase 10 + 8 Phase 11)
|
||||
- cmd/recon.go compiles with all credential wiring
|
||||
- Integration test passes with all 18 SourceTypes emitting findings
|
||||
- `go build ./cmd/...` succeeds
|
||||
- Guardrail test asserts exact 18-source name list
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
After completion, create `.planning/phases/11-osint_search_paste/11-03-SUMMARY.md`
|
||||
</output>
|
||||
Reference in New Issue
Block a user