187 lines
7.3 KiB
Markdown
187 lines
7.3 KiB
Markdown
---
|
|
phase: 09-osint-infrastructure
|
|
plan: 03
|
|
type: execute
|
|
wave: 1
|
|
depends_on: []
|
|
files_modified:
|
|
- pkg/recon/stealth.go
|
|
- pkg/recon/stealth_test.go
|
|
- pkg/recon/dedup.go
|
|
- pkg/recon/dedup_test.go
|
|
autonomous: true
|
|
requirements: [RECON-INFRA-06]
|
|
must_haves:
|
|
truths:
|
|
- "Stealth mode exposes a UA pool of 10 realistic browser user-agents (Chrome/Firefox/Safari across Linux/macOS/Windows)"
|
|
- "RandomUserAgent returns a UA from the pool, distributed across calls"
|
|
- "Dedup drops duplicate Findings keyed by SHA256(provider + masked_key + source)"
|
|
- "Dedup preserves first-seen order and metadata"
|
|
artifacts:
|
|
- path: "pkg/recon/stealth.go"
|
|
provides: "UA pool, RandomUserAgent, StealthHeaders helper"
|
|
contains: "var userAgents"
|
|
- path: "pkg/recon/dedup.go"
|
|
provides: "Dedup([]Finding) []Finding keyed by sha256(provider|masked|source)"
|
|
contains: "func Dedup"
|
|
key_links:
|
|
- from: "pkg/recon/dedup.go"
|
|
to: "crypto/sha256"
|
|
via: "finding hash key"
|
|
pattern: "sha256\\.Sum256"
|
|
---
|
|
|
|
<objective>
|
|
Implement stealth mode helpers (UA rotation) and cross-source deduplication. Both are small, self-contained, and unblock the parallel sweep orchestrator from producing noisy duplicate findings.
|
|
|
|
Purpose: Satisfies RECON-INFRA-06 (stealth UA rotation) and provides the dedup primitive that SweepAll callers use to satisfy RECON-INFRA-08's "deduplicates findings before persisting" criterion.
|
|
Output: pkg/recon/stealth.go, pkg/recon/dedup.go, and their tests
|
|
</objective>
|
|
|
|
<execution_context>
|
|
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
|
@$HOME/.claude/get-shit-done/templates/summary.md
|
|
</execution_context>
|
|
|
|
<context>
|
|
@.planning/phases/09-osint-infrastructure/09-CONTEXT.md
|
|
@pkg/engine/finding.go
|
|
</context>
|
|
|
|
<tasks>
|
|
|
|
<task type="auto" tdd="true">
|
|
<name>Task 1: Stealth UA pool + RandomUserAgent</name>
|
|
<files>pkg/recon/stealth.go, pkg/recon/stealth_test.go</files>
|
|
<behavior>
|
|
- userAgents is an unexported slice of exactly 10 realistic UA strings covering Chrome/Firefox/Safari on Linux/macOS/Windows
|
|
- RandomUserAgent() returns a random entry from the pool
|
|
- StealthHeaders() returns map[string]string{"User-Agent": RandomUserAgent(), "Accept-Language": "en-US,en;q=0.9"}
|
|
- Tests: TestUAPoolSize (== 10), TestRandomUserAgentInPool (returned value is in pool), TestStealthHeadersHasUA
|
|
</behavior>
|
|
<action>
|
|
Create pkg/recon/stealth.go with a package-level `userAgents` slice of 10 realistic UAs. Include at least:
|
|
- Chrome 120 Windows
|
|
- Chrome 120 macOS
|
|
- Chrome 120 Linux
|
|
- Firefox 121 Windows
|
|
- Firefox 121 macOS
|
|
- Firefox 121 Linux
|
|
- Safari 17 macOS
|
|
- Safari 17 iOS
|
|
- Edge 120 Windows
|
|
- Chrome Android
|
|
|
|
```go
|
|
package recon
|
|
|
|
import "math/rand"
|
|
|
|
var userAgents = []string{
|
|
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
|
|
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
|
|
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
|
|
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0",
|
|
"Mozilla/5.0 (Macintosh; Intel Mac OS X 14.2; rv:121.0) Gecko/20100101 Firefox/121.0",
|
|
"Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0",
|
|
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15",
|
|
"Mozilla/5.0 (iPhone; CPU iPhone OS 17_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Mobile/15E148 Safari/604.1",
|
|
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.2210.61",
|
|
"Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Mobile Safari/537.36",
|
|
}
|
|
|
|
// RandomUserAgent returns a random browser user-agent from the pool.
|
|
// Used when Config.Stealth is true.
|
|
func RandomUserAgent() string {
|
|
return userAgents[rand.Intn(len(userAgents))]
|
|
}
|
|
|
|
// StealthHeaders returns a minimal headers map with rotated UA and Accept-Language.
|
|
func StealthHeaders() map[string]string {
|
|
return map[string]string{
|
|
"User-Agent": RandomUserAgent(),
|
|
"Accept-Language": "en-US,en;q=0.9",
|
|
}
|
|
}
|
|
```
|
|
|
|
Create pkg/recon/stealth_test.go with the three tests. TestRandomUserAgentInPool should loop 100 times and assert each result is present in the `userAgents` slice.
|
|
</action>
|
|
<verify>
|
|
<automated>cd /home/salva/Documents/apikey && go test ./pkg/recon/ -run 'TestUAPool|TestRandomUserAgent|TestStealthHeaders' -count=1</automated>
|
|
</verify>
|
|
<done>Tests pass. Pool has exactly 10 UAs. Random selection always within pool.</done>
|
|
</task>
|
|
|
|
<task type="auto" tdd="true">
|
|
<name>Task 2: Cross-source finding dedup</name>
|
|
<files>pkg/recon/dedup.go, pkg/recon/dedup_test.go</files>
|
|
<behavior>
|
|
- Dedup(in []Finding) []Finding drops duplicates keyed by sha256(ProviderName + "|" + KeyMasked + "|" + Source)
|
|
- First-seen wins: returned slice preserves the first occurrence's metadata (SourceType, DetectedAt, etc.)
|
|
- Order is preserved from the input (stable dedup)
|
|
- Nil/empty input returns nil
|
|
- Tests:
|
|
- TestDedupEmpty: Dedup(nil) == nil
|
|
- TestDedupNoDuplicates: 3 distinct findings -> 3 returned
|
|
- TestDedupAllDuplicates: 3 identical findings -> 1 returned
|
|
- TestDedupPreservesFirstSeen: two findings with same key, different DetectedAt — the first-seen timestamp wins
|
|
- TestDedupDifferentSource: same provider/masked, different Source URLs -> both kept
|
|
</behavior>
|
|
<action>
|
|
Create pkg/recon/dedup.go:
|
|
|
|
```go
|
|
package recon
|
|
|
|
import (
|
|
"crypto/sha256"
|
|
"encoding/hex"
|
|
)
|
|
|
|
// Dedup removes duplicate findings using SHA256(provider|masked|source) as key.
|
|
// Stable: preserves input order and first-seen metadata.
|
|
func Dedup(in []Finding) []Finding {
|
|
if len(in) == 0 {
|
|
return nil
|
|
}
|
|
seen := make(map[string]struct{}, len(in))
|
|
out := make([]Finding, 0, len(in))
|
|
for _, f := range in {
|
|
h := sha256.Sum256([]byte(f.ProviderName + "|" + f.KeyMasked + "|" + f.Source))
|
|
k := hex.EncodeToString(h[:])
|
|
if _, dup := seen[k]; dup {
|
|
continue
|
|
}
|
|
seen[k] = struct{}{}
|
|
out = append(out, f)
|
|
}
|
|
return out
|
|
}
|
|
```
|
|
|
|
Create pkg/recon/dedup_test.go with the five tests. Use testify require.
|
|
</action>
|
|
<verify>
|
|
<automated>cd /home/salva/Documents/apikey && go test ./pkg/recon/ -run TestDedup -count=1</automated>
|
|
</verify>
|
|
<done>All dedup tests pass. First-seen wins. Different Source URLs are kept separate.</done>
|
|
</task>
|
|
|
|
</tasks>
|
|
|
|
<verification>
|
|
- `go test ./pkg/recon/ -run 'TestUAPool|TestRandom|TestStealth|TestDedup' -count=1` passes
|
|
- `go vet ./pkg/recon/...` clean
|
|
</verification>
|
|
|
|
<success_criteria>
|
|
- Stealth UA pool (10 entries) exported via RandomUserAgent/StealthHeaders
|
|
- Dedup primitive removes duplicates stably by sha256(provider|masked|source)
|
|
</success_criteria>
|
|
|
|
<output>
|
|
After completion, create `.planning/phases/09-osint-infrastructure/09-03-SUMMARY.md`
|
|
</output>
|
|
</content>
|