docs(16): create phase plan

This commit is contained in:
salvacybersec
2026-04-06 16:42:15 +03:00
parent 5216b39826
commit 8bcd9ebc18
5 changed files with 688 additions and 1 deletions

View File

@@ -321,7 +321,13 @@ Plans:
2. `keyhunter recon --sources=apk --target=com.example.app` downloads, decompiles (via apktool/jadx), and scans APK content for API keys 2. `keyhunter recon --sources=apk --target=com.example.app` downloads, decompiles (via apktool/jadx), and scans APK content for API keys
3. `keyhunter recon --sources=crtsh --target=example.com` discovers subdomains via Certificate Transparency logs and probes each for `.env`, `/api/config`, and `/actuator/env` endpoints 3. `keyhunter recon --sources=crtsh --target=example.com` discovers subdomains via Certificate Transparency logs and probes each for `.env`, `/api/config`, and `/actuator/env` endpoints
4. `keyhunter recon --sources=postman,swaggerhub` scans public Postman collections and SwaggerHub API definitions for hardcoded keys in request examples 4. `keyhunter recon --sources=postman,swaggerhub` scans public Postman collections and SwaggerHub API definitions for hardcoded keys in request examples
**Plans**: TBD **Plans**: 4 plans
Plans:
- [ ] 16-01-PLAN.md — VirusTotal, IntelligenceX, URLhaus threat intelligence sources (RECON-INTEL-01, RECON-INTEL-02, RECON-INTEL-03)
- [ ] 16-02-PLAN.md — APKMirror, crt.sh, SecurityTrails mobile and DNS sources (RECON-MOBILE-01, RECON-DNS-01, RECON-DNS-02)
- [ ] 16-03-PLAN.md — Postman, SwaggerHub, RapidAPI marketplace sources (RECON-API-01, RECON-API-02)
- [ ] 16-04-PLAN.md — RegisterAll wiring + cmd/recon.go credentials + integration test (all Phase 16 reqs)
### Phase 17: Telegram Bot & Scheduled Scanning ### Phase 17: Telegram Bot & Scheduled Scanning
**Goal**: Users can control KeyHunter remotely via a Telegram bot with scan, verify, recon, status, and subscription commands, and set up cron-based recurring scans that auto-notify on new findings **Goal**: Users can control KeyHunter remotely via a Telegram bot with scan, verify, recon, status, and subscription commands, and set up cron-based recurring scans that auto-notify on new findings

View File

@@ -0,0 +1,168 @@
---
phase: 16-osint-threat-intel-mobile-dns-api-marketplaces
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- pkg/recon/sources/virustotal.go
- pkg/recon/sources/virustotal_test.go
- pkg/recon/sources/intelligencex.go
- pkg/recon/sources/intelligencex_test.go
- pkg/recon/sources/urlhaus.go
- pkg/recon/sources/urlhaus_test.go
autonomous: true
requirements: [RECON-INTEL-01, RECON-INTEL-02, RECON-INTEL-03]
must_haves:
truths:
- "VirusTotal source searches VT API for files/URLs containing provider keywords"
- "IntelligenceX source searches IX archive for leaked credentials"
- "URLhaus source searches abuse.ch URLhaus API for malicious URLs containing keys"
artifacts:
- path: "pkg/recon/sources/virustotal.go"
provides: "VirusTotalSource implementing recon.ReconSource"
contains: "func (s *VirusTotalSource) Sweep"
- path: "pkg/recon/sources/intelligencex.go"
provides: "IntelligenceXSource implementing recon.ReconSource"
contains: "func (s *IntelligenceXSource) Sweep"
- path: "pkg/recon/sources/urlhaus.go"
provides: "URLhausSource implementing recon.ReconSource"
contains: "func (s *URLhausSource) Sweep"
key_links:
- from: "pkg/recon/sources/virustotal.go"
to: "pkg/recon/sources/queries.go"
via: "BuildQueries call"
pattern: "BuildQueries\\(s\\.Registry"
- from: "pkg/recon/sources/intelligencex.go"
to: "pkg/recon/sources/queries.go"
via: "BuildQueries call"
pattern: "BuildQueries\\(s\\.Registry"
- from: "pkg/recon/sources/urlhaus.go"
to: "pkg/recon/sources/queries.go"
via: "BuildQueries call"
pattern: "BuildQueries\\(s\\.Registry"
---
<objective>
Implement three threat intelligence ReconSource modules: VirusTotal, IntelligenceX, and URLhaus.
Purpose: Detect API keys appearing in threat intelligence feeds — malware samples (VT), breach archives (IX), and malicious URL databases (URLhaus).
Output: Three source files + tests in pkg/recon/sources/
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@pkg/recon/source.go
@pkg/recon/sources/httpclient.go
@pkg/recon/sources/queries.go
@pkg/recon/sources/sentry.go
@pkg/recon/sources/sentry_test.go
</context>
<interfaces>
<!-- Established patterns from the codebase that executors must follow -->
From pkg/recon/source.go:
```go
type ReconSource interface {
Name() string
RateLimit() rate.Limit
Burst() int
RespectsRobots() bool
Enabled(cfg Config) bool
Sweep(ctx context.Context, query string, out chan<- Finding) error
}
```
From pkg/recon/sources/httpclient.go:
```go
func NewClient() *Client // 30s timeout, 2 retries
func (c *Client) Do(ctx context.Context, req *http.Request) (*http.Response, error)
```
From pkg/recon/sources/queries.go:
```go
func BuildQueries(reg *providers.Registry, source string) []string
```
From pkg/recon/sources/travisci.go:
```go
var ciLogKeyPattern = regexp.MustCompile(`(?i)(api[_-]?key|secret[_-]?key|token|password|credential|auth[_-]?token)['":\s]*[=:]\s*['"]?([a-zA-Z0-9_\-]{16,})['"]?`)
```
</interfaces>
<tasks>
<task type="auto">
<name>Task 1: VirusTotal and IntelligenceX sources</name>
<files>pkg/recon/sources/virustotal.go, pkg/recon/sources/virustotal_test.go, pkg/recon/sources/intelligencex.go, pkg/recon/sources/intelligencex_test.go</files>
<action>
Create VirusTotalSource in virustotal.go following the exact SentrySource pattern:
- Struct: VirusTotalSource with APIKey, BaseURL, Registry (*providers.Registry), Limiters (*recon.LimiterRegistry), Client (*Client) fields.
- Name() returns "virustotal". RateLimit() returns rate.Every(15*time.Second) (VT free tier: 4 req/min). Burst() returns 2. RespectsRobots() returns false. Enabled() returns s.APIKey != "".
- Compile-time interface check: `var _ recon.ReconSource = (*VirusTotalSource)(nil)`
- Sweep(): Default BaseURL to "https://www.virustotal.com/api/v3". Use BuildQueries(s.Registry, "virustotal") to get keyword list. For each query, call GET `{base}/intelligence/search?query={url-encoded query}&limit=10` with header `x-apikey: {APIKey}`. Parse JSON response `{"data":[{"id":"...","attributes":{"meaningful_name":"...","tags":[...],...}}]}`. For each result, stringify the attributes JSON and check with ciLogKeyPattern.MatchString(). Emit Finding with SourceType "recon:virustotal", Source as VT permalink `https://www.virustotal.com/gui/file/{id}`.
- Rate-limit via s.Limiters.Wait(ctx, s.Name(), ...) before each HTTP call, same as SentrySource pattern.
Create IntelligenceXSource in intelligencex.go:
- Struct: IntelligenceXSource with APIKey, BaseURL, Registry, Limiters, Client fields.
- Name() returns "intelligencex". RateLimit() returns rate.Every(5*time.Second). Burst() returns 3. RespectsRobots() false. Enabled() returns s.APIKey != "".
- Sweep(): Default BaseURL to "https://2.intelx.io". Use BuildQueries. For each query: POST `{base}/intelligent/search` with JSON body `{"term":"{query}","maxresults":10,"media":0,"timeout":5}` and header `x-key: {APIKey}`. Parse response `{"id":"search-id","status":0}`. Then GET `{base}/intelligent/search/result?id={search-id}&limit=10` with same x-key header. Parse `{"records":[{"systemid":"...","name":"...","storageid":"...","bucket":"..."}]}`. For each record, fetch content via GET `{base}/file/read?type=0&storageid={storageid}&bucket={bucket}` — read up to 512KB, check with ciLogKeyPattern. Emit Finding with SourceType "recon:intelligencex".
Tests: Follow sentry_test.go pattern exactly. Use httptest.NewServer with mux routing. Test Name(), Enabled() (true with key, false without), Sweep with mock responses returning key-like content, and Sweep with empty results.
</action>
<verify>
<automated>cd /home/salva/Documents/apikey && go test ./pkg/recon/sources/ -run "TestVirusTotal|TestIntelligenceX" -count=1 -v</automated>
</verify>
<done>VirusTotalSource and IntelligenceXSource implement ReconSource, tests pass with httptest mocks proving Sweep emits findings for key-containing responses and zero findings for clean responses</done>
</task>
<task type="auto">
<name>Task 2: URLhaus source</name>
<files>pkg/recon/sources/urlhaus.go, pkg/recon/sources/urlhaus_test.go</files>
<action>
Create URLhausSource in urlhaus.go:
- Struct: URLhausSource with BaseURL, Registry (*providers.Registry), Limiters (*recon.LimiterRegistry), Client (*Client) fields. No API key needed — URLhaus API is free/unauthenticated.
- Name() returns "urlhaus". RateLimit() returns rate.Every(3*time.Second). Burst() returns 2. RespectsRobots() false. Enabled() always returns true (credentialless).
- Sweep(): Default BaseURL to "https://urlhaus-api.abuse.ch/v1". Use BuildQueries(s.Registry, "urlhaus"). For each query: POST `{base}/tag/{url-encoded query}/` (URLhaus tag lookup). If that returns empty or error, fallback to POST `{base}/payload/` with form body `md5_hash=&sha256_hash=&tag={query}`. Parse JSON response `{"query_status":"ok","urls":[{"url":"...","url_status":"...","tags":[...],"reporter":"..."}]}`. For each URL entry, stringify the URL record and check with ciLogKeyPattern. Emit Finding with SourceType "recon:urlhaus", Source as the url field.
- Note: URLhaus uses POST with form-encoded body for most endpoints. Set Content-Type to "application/x-www-form-urlencoded".
Tests: httptest mock. Test Name(), Enabled() (always true), Sweep happy path, Sweep empty results.
</action>
<verify>
<automated>cd /home/salva/Documents/apikey && go test ./pkg/recon/sources/ -run "TestURLhaus" -count=1 -v</automated>
</verify>
<done>URLhausSource implements ReconSource, tests pass confirming credentialless Sweep emits findings for key-containing URL records</done>
</task>
</tasks>
<verification>
All three threat intel sources compile and pass unit tests:
```bash
cd /home/salva/Documents/apikey && go test ./pkg/recon/sources/ -run "TestVirusTotal|TestIntelligenceX|TestURLhaus" -count=1 -v
go vet ./pkg/recon/sources/
```
</verification>
<success_criteria>
- virustotal.go, intelligencex.go, urlhaus.go each implement recon.ReconSource
- VirusTotal and IntelligenceX are credential-gated (Enabled returns false without API key)
- URLhaus is credentialless (Enabled always true)
- All tests pass with httptest mocks
- ciLogKeyPattern used for content matching (no custom regex)
</success_criteria>
<output>
After completion, create `.planning/phases/16-osint_threat_intel_mobile_dns_api_marketplaces/16-01-SUMMARY.md`
</output>

View File

@@ -0,0 +1,159 @@
---
phase: 16-osint-threat-intel-mobile-dns-api-marketplaces
plan: 02
type: execute
wave: 1
depends_on: []
files_modified:
- pkg/recon/sources/apkmirror.go
- pkg/recon/sources/apkmirror_test.go
- pkg/recon/sources/crtsh.go
- pkg/recon/sources/crtsh_test.go
- pkg/recon/sources/securitytrails.go
- pkg/recon/sources/securitytrails_test.go
autonomous: true
requirements: [RECON-MOBILE-01, RECON-DNS-01, RECON-DNS-02]
must_haves:
truths:
- "APKMirror source searches for APK metadata containing provider keywords"
- "crt.sh source discovers subdomains via CT logs and probes config endpoints for keys"
- "SecurityTrails source searches DNS/subdomain data for key exposure indicators"
artifacts:
- path: "pkg/recon/sources/apkmirror.go"
provides: "APKMirrorSource implementing recon.ReconSource"
contains: "func (s *APKMirrorSource) Sweep"
- path: "pkg/recon/sources/crtsh.go"
provides: "CrtShSource implementing recon.ReconSource"
contains: "func (s *CrtShSource) Sweep"
- path: "pkg/recon/sources/securitytrails.go"
provides: "SecurityTrailsSource implementing recon.ReconSource"
contains: "func (s *SecurityTrailsSource) Sweep"
key_links:
- from: "pkg/recon/sources/crtsh.go"
to: "pkg/recon/sources/httpclient.go"
via: "Client.Do for endpoint probing"
pattern: "client\\.Do\\(ctx"
- from: "pkg/recon/sources/securitytrails.go"
to: "pkg/recon/sources/queries.go"
via: "BuildQueries call"
pattern: "BuildQueries\\(s\\.Registry"
---
<objective>
Implement APKMirror (mobile), crt.sh (CT log DNS), and SecurityTrails (DNS intel) ReconSource modules.
Purpose: Detect API keys in mobile app metadata, discover subdomains via certificate transparency and probe their config endpoints, and search DNS intelligence for key exposure.
Output: Three source files + tests in pkg/recon/sources/
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@pkg/recon/source.go
@pkg/recon/sources/httpclient.go
@pkg/recon/sources/queries.go
@pkg/recon/sources/sentry.go
@pkg/recon/sources/sentry_test.go
</context>
<interfaces>
From pkg/recon/source.go:
```go
type ReconSource interface {
Name() string
RateLimit() rate.Limit
Burst() int
RespectsRobots() bool
Enabled(cfg Config) bool
Sweep(ctx context.Context, query string, out chan<- Finding) error
}
```
From pkg/recon/sources/httpclient.go:
```go
func NewClient() *Client
func (c *Client) Do(ctx context.Context, req *http.Request) (*http.Response, error)
```
From pkg/recon/sources/queries.go:
```go
func BuildQueries(reg *providers.Registry, source string) []string
```
</interfaces>
<tasks>
<task type="auto">
<name>Task 1: APKMirror and crt.sh sources</name>
<files>pkg/recon/sources/apkmirror.go, pkg/recon/sources/apkmirror_test.go, pkg/recon/sources/crtsh.go, pkg/recon/sources/crtsh_test.go</files>
<action>
Create APKMirrorSource in apkmirror.go:
- Struct: APKMirrorSource with BaseURL, Registry (*providers.Registry), Limiters (*recon.LimiterRegistry), Client (*Client) fields. Credentialless.
- Name() returns "apkmirror". RateLimit() returns rate.Every(5*time.Second). Burst() returns 2. RespectsRobots() returns true (scraping). Enabled() always true.
- Sweep(): Default BaseURL to "https://www.apkmirror.com". Use BuildQueries(s.Registry, "apkmirror"). For each query: GET `{base}/?post_type=app_release&searchtype=apk&s={url-encoded query}`. Parse HTML response — search for APK listing entries. Since we cannot decompile APKs in a network sweep, focus on metadata: search page HTML content for ciLogKeyPattern matches in APK descriptions, changelogs, and file listings. Emit Finding with SourceType "recon:apkmirror", Source as the page URL.
- Note: This is a metadata/description scanner, not a full APK decompiler. The decompile capability (apktool/jadx) is noted in RECON-MOBILE-01 but that requires local binary dependencies — the ReconSource focuses on web-searchable APK metadata for keys in descriptions and changelogs.
Create CrtShSource in crtsh.go:
- Struct: CrtShSource with BaseURL, Registry (*providers.Registry), Limiters (*recon.LimiterRegistry), Client (*Client) fields. Credentialless.
- Name() returns "crtsh". RateLimit() returns rate.Every(3*time.Second). Burst() returns 3. RespectsRobots() false (API). Enabled() always true.
- Sweep(): Default BaseURL to "https://crt.sh". The query parameter is used as the target domain. If query is empty, use BuildQueries but for crt.sh the query should be a domain — if query looks like a keyword rather than a domain, skip (return nil). GET `{base}/?q=%25.{domain}&output=json` to find subdomains. Parse JSON array `[{"name_value":"sub.example.com","common_name":"..."}]`. Deduplicate name_value entries. For each unique subdomain (limit 20), probe three config endpoints: `https://{subdomain}/.env`, `https://{subdomain}/api/config`, `https://{subdomain}/actuator/env`. Use a short 5s timeout per probe. For each successful response (200 OK), check body with ciLogKeyPattern. Emit Finding with SourceType "recon:crtsh", Source as the probed URL.
- Important: The probe HTTP client should be separate from the crt.sh API client — create a short-timeout *http.Client{Timeout: 5*time.Second} for probing. Do NOT use the retry Client for probes (probes should fail fast, not retry).
Tests: httptest for both. APKMirror: mock returns HTML with key-like content in description. CrtSh: mock returns JSON subdomain list, mock probe endpoints return .env-like content with key patterns.
</action>
<verify>
<automated>cd /home/salva/Documents/apikey && go test ./pkg/recon/sources/ -run "TestAPKMirror|TestCrtSh" -count=1 -v</automated>
</verify>
<done>APKMirrorSource scans APK metadata pages, CrtShSource discovers subdomains and probes config endpoints, both emit findings on ciLogKeyPattern match</done>
</task>
<task type="auto">
<name>Task 2: SecurityTrails source</name>
<files>pkg/recon/sources/securitytrails.go, pkg/recon/sources/securitytrails_test.go</files>
<action>
Create SecurityTrailsSource in securitytrails.go:
- Struct: SecurityTrailsSource with APIKey, BaseURL, Registry (*providers.Registry), Limiters (*recon.LimiterRegistry), Client (*Client) fields.
- Name() returns "securitytrails". RateLimit() returns rate.Every(2*time.Second). Burst() returns 5. RespectsRobots() false. Enabled() returns s.APIKey != "".
- Sweep(): Default BaseURL to "https://api.securitytrails.com/v1". The query parameter is used as the target domain. If empty, return nil. Two-phase approach:
1. Subdomain enumeration: GET `{base}/domain/{domain}/subdomains?children_only=false` with header `APIKEY: {APIKey}`. Parse `{"subdomains":["www","api","staging",...]}`. Build full FQDNs by appending `.{domain}`.
2. For each subdomain (limit 20), probe same three config endpoints as CrtShSource: `/.env`, `/api/config`, `/actuator/env`. Use short-timeout probe client (5s, no retries). Check responses with ciLogKeyPattern. Emit Finding with SourceType "recon:securitytrails".
- Also: GET `{base}/domain/{domain}` for the domain's DNS history. Parse response and check the full JSON body with ciLogKeyPattern (DNS TXT records sometimes contain API keys).
Tests: httptest mock. Test Enabled() with/without API key. Sweep with mock subdomain list and probe endpoints.
</action>
<verify>
<automated>cd /home/salva/Documents/apikey && go test ./pkg/recon/sources/ -run "TestSecurityTrails" -count=1 -v</automated>
</verify>
<done>SecurityTrailsSource discovers subdomains via API, probes config endpoints, and scans DNS records for key patterns; credential-gated via API key</done>
</task>
</tasks>
<verification>
All three sources compile and pass tests:
```bash
cd /home/salva/Documents/apikey && go test ./pkg/recon/sources/ -run "TestAPKMirror|TestCrtSh|TestSecurityTrails" -count=1 -v
go vet ./pkg/recon/sources/
```
</verification>
<success_criteria>
- apkmirror.go, crtsh.go, securitytrails.go each implement recon.ReconSource
- APKMirror and crt.sh are credentialless (Enabled always true)
- SecurityTrails is credential-gated
- crt.sh and SecurityTrails both probe /.env, /api/config, /actuator/env on discovered subdomains
- All tests pass with httptest mocks
</success_criteria>
<output>
After completion, create `.planning/phases/16-osint_threat_intel_mobile_dns_api_marketplaces/16-02-SUMMARY.md`
</output>

View File

@@ -0,0 +1,155 @@
---
phase: 16-osint-threat-intel-mobile-dns-api-marketplaces
plan: 03
type: execute
wave: 1
depends_on: []
files_modified:
- pkg/recon/sources/postman.go
- pkg/recon/sources/postman_test.go
- pkg/recon/sources/swaggerhub.go
- pkg/recon/sources/swaggerhub_test.go
- pkg/recon/sources/rapidapi.go
- pkg/recon/sources/rapidapi_test.go
autonomous: true
requirements: [RECON-API-01, RECON-API-02]
must_haves:
truths:
- "Postman source searches public collections/workspaces for hardcoded API keys"
- "SwaggerHub source searches published API definitions for embedded keys in examples"
- "RapidAPI source searches public API listings for exposed credentials"
artifacts:
- path: "pkg/recon/sources/postman.go"
provides: "PostmanSource implementing recon.ReconSource"
contains: "func (s *PostmanSource) Sweep"
- path: "pkg/recon/sources/swaggerhub.go"
provides: "SwaggerHubSource implementing recon.ReconSource"
contains: "func (s *SwaggerHubSource) Sweep"
- path: "pkg/recon/sources/rapidapi.go"
provides: "RapidAPISource implementing recon.ReconSource"
contains: "func (s *RapidAPISource) Sweep"
key_links:
- from: "pkg/recon/sources/postman.go"
to: "pkg/recon/sources/queries.go"
via: "BuildQueries call"
pattern: "BuildQueries\\(s\\.Registry"
- from: "pkg/recon/sources/swaggerhub.go"
to: "pkg/recon/sources/queries.go"
via: "BuildQueries call"
pattern: "BuildQueries\\(s\\.Registry"
---
<objective>
Implement Postman, SwaggerHub, and RapidAPI ReconSource modules for API marketplace scanning.
Purpose: Detect API keys hardcoded in public Postman collections, SwaggerHub API definitions, and RapidAPI listings where developers accidentally include real credentials in request examples.
Output: Three source files + tests in pkg/recon/sources/
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@pkg/recon/source.go
@pkg/recon/sources/httpclient.go
@pkg/recon/sources/queries.go
@pkg/recon/sources/sentry.go
@pkg/recon/sources/sentry_test.go
</context>
<interfaces>
From pkg/recon/source.go:
```go
type ReconSource interface {
Name() string
RateLimit() rate.Limit
Burst() int
RespectsRobots() bool
Enabled(cfg Config) bool
Sweep(ctx context.Context, query string, out chan<- Finding) error
}
```
From pkg/recon/sources/httpclient.go:
```go
func NewClient() *Client
func (c *Client) Do(ctx context.Context, req *http.Request) (*http.Response, error)
```
From pkg/recon/sources/queries.go:
```go
func BuildQueries(reg *providers.Registry, source string) []string
```
</interfaces>
<tasks>
<task type="auto">
<name>Task 1: Postman and SwaggerHub sources</name>
<files>pkg/recon/sources/postman.go, pkg/recon/sources/postman_test.go, pkg/recon/sources/swaggerhub.go, pkg/recon/sources/swaggerhub_test.go</files>
<action>
Create PostmanSource in postman.go:
- Struct: PostmanSource with BaseURL, Registry (*providers.Registry), Limiters (*recon.LimiterRegistry), Client (*Client) fields. Credentialless — Postman public API search does not require authentication.
- Name() returns "postman". RateLimit() returns rate.Every(3*time.Second). Burst() returns 3. RespectsRobots() false. Enabled() always true.
- Sweep(): Default BaseURL to "https://www.postman.com/_api". Use BuildQueries(s.Registry, "postman"). For each query: GET `{base}/ws/proxy?request=%2Fsearch%2Fall%3Fquerytext%3D{url-encoded query}%26size%3D10%26type%3Dall` (Postman's internal search proxy). Parse JSON response containing search results with collection/workspace metadata. For each result, fetch the collection detail: GET `{base}/collection/{collection-id}` or use the direct URL from search. Stringify the collection JSON and check with ciLogKeyPattern. Emit Finding with SourceType "recon:postman", Source as `https://www.postman.com/collection/{id}`.
- Alternative simpler approach: Use Postman's public network search at `https://www.postman.com/_api/ws/proxy` with the search endpoint. The response contains snippets — check snippets directly with ciLogKeyPattern without fetching full collections (faster, fewer requests).
Create SwaggerHubSource in swaggerhub.go:
- Struct: SwaggerHubSource with BaseURL, Registry (*providers.Registry), Limiters (*recon.LimiterRegistry), Client (*Client) fields. Credentialless.
- Name() returns "swaggerhub". RateLimit() returns rate.Every(3*time.Second). Burst() returns 3. RespectsRobots() false. Enabled() always true.
- Sweep(): Default BaseURL to "https://app.swaggerhub.com/apiproxy/specs". Use BuildQueries(s.Registry, "swaggerhub"). For each query: GET `{base}?specType=ANY&visibility=PUBLIC&query={url-encoded query}&limit=10&page=1`. Parse JSON `{"apis":[{"name":"...","url":"...","description":"...","properties":[{"type":"Swagger","url":"..."}]}]}`. For each API result, fetch the spec URL to get the full OpenAPI/Swagger JSON. Check the spec content with ciLogKeyPattern (keys often appear in example values, server URLs, and security scheme defaults). Emit Finding with SourceType "recon:swaggerhub", Source as the SwaggerHub URL.
Tests: httptest mocks for both. Postman: mock search returns results with key-like content in snippets. SwaggerHub: mock returns API list, spec fetch returns OpenAPI JSON with embedded key pattern.
</action>
<verify>
<automated>cd /home/salva/Documents/apikey && go test ./pkg/recon/sources/ -run "TestPostman|TestSwaggerHub" -count=1 -v</automated>
</verify>
<done>PostmanSource searches public collections, SwaggerHubSource searches published API specs, both emit findings on ciLogKeyPattern match in response content</done>
</task>
<task type="auto">
<name>Task 2: RapidAPI source</name>
<files>pkg/recon/sources/rapidapi.go, pkg/recon/sources/rapidapi_test.go</files>
<action>
Create RapidAPISource in rapidapi.go:
- Struct: RapidAPISource with BaseURL, Registry (*providers.Registry), Limiters (*recon.LimiterRegistry), Client (*Client) fields. Credentialless.
- Name() returns "rapidapi". RateLimit() returns rate.Every(3*time.Second). Burst() returns 3. RespectsRobots() false. Enabled() always true.
- Sweep(): Default BaseURL to "https://rapidapi.com". Use BuildQueries(s.Registry, "rapidapi"). For each query: GET `{base}/search/{url-encoded query}?sortBy=ByRelevance&page=1` — RapidAPI's search page. Parse the HTML response body or use the internal JSON API if available. Check content with ciLogKeyPattern. Focus on API listings that include code snippets and example requests where developers may have pasted real API keys. Emit Finding with SourceType "recon:rapidapi", Source as the API listing URL.
- Simpler approach: Since RapidAPI's internal search API may not be stable, treat this as a scraping source. GET the search page, read up to 512KB of HTML, and scan with ciLogKeyPattern. This catches keys in code examples, API descriptions, and documentation snippets visible on the public page.
Tests: httptest mock. Test Name(), Enabled() (always true), Sweep with mock HTML containing key patterns, Sweep with clean HTML returning zero findings.
</action>
<verify>
<automated>cd /home/salva/Documents/apikey && go test ./pkg/recon/sources/ -run "TestRapidAPI" -count=1 -v</automated>
</verify>
<done>RapidAPISource searches public API listings for key patterns, credentialless, tests pass with httptest mocks</done>
</task>
</tasks>
<verification>
All three API marketplace sources compile and pass tests:
```bash
cd /home/salva/Documents/apikey && go test ./pkg/recon/sources/ -run "TestPostman|TestSwaggerHub|TestRapidAPI" -count=1 -v
go vet ./pkg/recon/sources/
```
</verification>
<success_criteria>
- postman.go, swaggerhub.go, rapidapi.go each implement recon.ReconSource
- All three are credentialless (Enabled always true)
- All use BuildQueries + ciLogKeyPattern (consistent with other sources)
- Tests pass with httptest mocks
</success_criteria>
<output>
After completion, create `.planning/phases/16-osint_threat_intel_mobile_dns_api_marketplaces/16-03-SUMMARY.md`
</output>

View File

@@ -0,0 +1,199 @@
---
phase: 16-osint-threat-intel-mobile-dns-api-marketplaces
plan: 04
type: execute
wave: 2
depends_on: [16-01, 16-02, 16-03]
files_modified:
- pkg/recon/sources/register.go
- pkg/recon/sources/register_test.go
- cmd/recon.go
autonomous: true
requirements: [RECON-INTEL-01, RECON-INTEL-02, RECON-INTEL-03, RECON-MOBILE-01, RECON-DNS-01, RECON-DNS-02, RECON-API-01, RECON-API-02]
must_haves:
truths:
- "RegisterAll registers all 9 Phase 16 sources (76 total)"
- "cmd/recon.go populates SourcesConfig with VT, IX, SecurityTrails credentials from env/viper"
- "Integration test proves all 76 sources are registered and the 9 new ones are present"
artifacts:
- path: "pkg/recon/sources/register.go"
provides: "RegisterAll with 76 sources (67 + 9 Phase 16)"
contains: "VirusTotalSource"
- path: "cmd/recon.go"
provides: "buildReconEngine with Phase 16 credential wiring"
contains: "VirusTotalAPIKey"
key_links:
- from: "pkg/recon/sources/register.go"
to: "pkg/recon/sources/virustotal.go"
via: "engine.Register(&VirusTotalSource{...})"
pattern: "VirusTotalSource"
- from: "cmd/recon.go"
to: "pkg/recon/sources/register.go"
via: "sources.RegisterAll(e, cfg)"
pattern: "sources\\.RegisterAll"
---
<objective>
Wire all 9 Phase 16 sources into RegisterAll and cmd/recon.go, bringing total from 67 to 76 sources. Add integration test validating the complete source catalog.
Purpose: Complete the last OSINT phase by connecting all new sources to the engine so `keyhunter recon list` shows 76 sources and `keyhunter recon full` sweeps them all.
Output: Updated register.go, register_test.go, cmd/recon.go
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@pkg/recon/sources/register.go
@cmd/recon.go
</context>
<interfaces>
From pkg/recon/sources/register.go (current):
```go
type SourcesConfig struct {
// ... existing fields through CircleCIToken ...
Registry *providers.Registry
Limiters *recon.LimiterRegistry
}
func RegisterAll(engine *recon.Engine, cfg SourcesConfig) { ... } // 67 sources
```
From cmd/recon.go (current):
```go
func buildReconEngine() *recon.Engine {
cfg := sources.SourcesConfig{
// ... existing credential bindings ...
}
sources.RegisterAll(e, cfg)
}
```
</interfaces>
<tasks>
<task type="auto">
<name>Task 1: Extend SourcesConfig, RegisterAll, and cmd/recon.go</name>
<files>pkg/recon/sources/register.go, cmd/recon.go</files>
<action>
Add new fields to SourcesConfig in register.go:
```go
// Phase 16: Threat intel, DNS, and API marketplace tokens.
VirusTotalAPIKey string
IntelligenceXAPIKey string
SecurityTrailsAPIKey string
```
Add Phase 16 registrations to RegisterAll, after the Phase 15 block:
```go
// Phase 16: Threat intelligence sources.
engine.Register(&VirusTotalSource{
APIKey: cfg.VirusTotalAPIKey,
Registry: reg,
Limiters: lim,
})
engine.Register(&IntelligenceXSource{
APIKey: cfg.IntelligenceXAPIKey,
Registry: reg,
Limiters: lim,
})
engine.Register(&URLhausSource{
Registry: reg,
Limiters: lim,
})
// Phase 16: Mobile and DNS sources.
engine.Register(&APKMirrorSource{
Registry: reg,
Limiters: lim,
})
engine.Register(&CrtShSource{
Registry: reg,
Limiters: lim,
})
engine.Register(&SecurityTrailsSource{
APIKey: cfg.SecurityTrailsAPIKey,
Registry: reg,
Limiters: lim,
})
// Phase 16: API marketplace sources (credentialless).
engine.Register(&PostmanSource{
Registry: reg,
Limiters: lim,
})
engine.Register(&SwaggerHubSource{
Registry: reg,
Limiters: lim,
})
engine.Register(&RapidAPISource{
Registry: reg,
Limiters: lim,
})
```
Update RegisterAll doc comment to say "76 sources total" and mention Phase 16.
In cmd/recon.go buildReconEngine(), add the three credential fields to the SourcesConfig literal:
```go
VirusTotalAPIKey: firstNonEmpty(os.Getenv("VIRUSTOTAL_API_KEY"), viper.GetString("recon.virustotal.api_key")),
IntelligenceXAPIKey: firstNonEmpty(os.Getenv("INTELLIGENCEX_API_KEY"), viper.GetString("recon.intelligencex.api_key")),
SecurityTrailsAPIKey: firstNonEmpty(os.Getenv("SECURITYTRAILS_API_KEY"), viper.GetString("recon.securitytrails.api_key")),
```
</action>
<verify>
<automated>cd /home/salva/Documents/apikey && go build ./cmd/... && go vet ./pkg/recon/sources/ ./cmd/...</automated>
</verify>
<done>RegisterAll registers 76 sources, cmd/recon.go wires VT/IX/SecurityTrails credentials from env/viper, project compiles cleanly</done>
</task>
<task type="auto">
<name>Task 2: Integration test for 76-source catalog</name>
<files>pkg/recon/sources/register_test.go</files>
<action>
Update or create register_test.go with an integration test that validates:
1. TestRegisterAll_SourceCount: Create a SourcesConfig with a test Registry (providers.NewRegistryFromProviders with one dummy provider) and a LimiterRegistry. Call RegisterAll on a fresh engine. Assert engine.List() returns exactly 76 names. If count differs, print the actual list for debugging.
2. TestRegisterAll_Phase16Sources: Assert the following 9 names are present in engine.List(): "virustotal", "intelligencex", "urlhaus", "apkmirror", "crtsh", "securitytrails", "postman", "swaggerhub", "rapidapi".
3. TestRegisterAll_CredentialGating: Register with empty SourcesConfig (no API keys). For each source via engine.Get(), call Enabled(recon.Config{}). Assert:
- virustotal, intelligencex, securitytrails: Enabled == false (credential-gated)
- urlhaus, apkmirror, crtsh, postman, swaggerhub, rapidapi: Enabled == true (credentialless)
Follow the existing test pattern from prior phases. Use testify/assert if already used in the file, otherwise use stdlib testing.
</action>
<verify>
<automated>cd /home/salva/Documents/apikey && go test ./pkg/recon/sources/ -run "TestRegisterAll" -count=1 -v</automated>
</verify>
<done>Integration test confirms 76 registered sources, all 9 Phase 16 sources present, credential gating correct for VT/IX/SecurityTrails vs credentialless sources</done>
</task>
</tasks>
<verification>
Full build and test:
```bash
cd /home/salva/Documents/apikey && go build ./cmd/... && go test ./pkg/recon/sources/ -run "TestRegisterAll" -count=1 -v
```
</verification>
<success_criteria>
- RegisterAll registers 76 sources (67 existing + 9 new)
- cmd/recon.go reads VIRUSTOTAL_API_KEY, INTELLIGENCEX_API_KEY, SECURITYTRAILS_API_KEY from env/viper
- Integration test passes confirming source count, names, and credential gating
- `go build ./cmd/...` succeeds with no errors
</success_criteria>
<output>
After completion, create `.planning/phases/16-osint_threat_intel_mobile_dns_api_marketplaces/16-04-SUMMARY.md`
</output>