305 lines
10 KiB
Markdown
305 lines
10 KiB
Markdown
---
|
|
phase: 09-osint-infrastructure
|
|
plan: 01
|
|
type: execute
|
|
wave: 1
|
|
depends_on: []
|
|
files_modified:
|
|
- pkg/recon/source.go
|
|
- pkg/recon/engine.go
|
|
- pkg/recon/example.go
|
|
- pkg/recon/engine_test.go
|
|
autonomous: true
|
|
requirements: [RECON-INFRA-08]
|
|
must_haves:
|
|
truths:
|
|
- "pkg/recon package compiles with a ReconSource interface"
|
|
- "Engine.Register adds a source; Engine.List returns registered names"
|
|
- "Engine.SweepAll fans out to all enabled sources via ants pool and returns aggregated Findings"
|
|
- "ExampleSource implements ReconSource end-to-end and emits a deterministic fake Finding"
|
|
artifacts:
|
|
- path: "pkg/recon/source.go"
|
|
provides: "ReconSource interface + Finding type alias + Config struct"
|
|
contains: "type ReconSource interface"
|
|
- path: "pkg/recon/engine.go"
|
|
provides: "Engine with Register, List, SweepAll (parallel fanout via ants)"
|
|
contains: "func (e *Engine) SweepAll"
|
|
- path: "pkg/recon/example.go"
|
|
provides: "ExampleSource stub that emits hardcoded findings"
|
|
contains: "type ExampleSource"
|
|
- path: "pkg/recon/engine_test.go"
|
|
provides: "Tests for Register/List/SweepAll with ExampleSource"
|
|
contains: "func TestSweepAll"
|
|
key_links:
|
|
- from: "pkg/recon/engine.go"
|
|
to: "github.com/panjf2000/ants/v2"
|
|
via: "parallel source fanout"
|
|
pattern: "ants\\.NewPool"
|
|
- from: "pkg/recon/engine.go"
|
|
to: "pkg/engine.Finding"
|
|
via: "aliased as recon.Finding for SourceType=\"recon:*\""
|
|
pattern: "engine\\.Finding"
|
|
---
|
|
|
|
<objective>
|
|
Create the pkg/recon/ package foundation: ReconSource interface, Engine orchestrator with parallel fanout via ants pool, and an ExampleSource stub that proves the pipeline end-to-end. This is the contract that all later sources (Phases 10-16) will implement.
|
|
|
|
Purpose: Establish the interface + engine skeleton so subsequent Wave 1 plans (limiter, stealth, robots) can land in parallel without conflict, and Wave 2 can wire the CLI.
|
|
Output: pkg/recon/source.go, pkg/recon/engine.go, pkg/recon/example.go, pkg/recon/engine_test.go
|
|
</objective>
|
|
|
|
<execution_context>
|
|
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
|
@$HOME/.claude/get-shit-done/templates/summary.md
|
|
</execution_context>
|
|
|
|
<context>
|
|
@.planning/PROJECT.md
|
|
@.planning/ROADMAP.md
|
|
@.planning/STATE.md
|
|
@.planning/phases/09-osint-infrastructure/09-CONTEXT.md
|
|
@pkg/engine/engine.go
|
|
@pkg/engine/finding.go
|
|
|
|
<interfaces>
|
|
<!-- Key types the executor needs. -->
|
|
|
|
From pkg/engine/finding.go:
|
|
```go
|
|
type Finding struct {
|
|
ProviderName string
|
|
KeyValue string
|
|
KeyMasked string
|
|
Confidence string
|
|
Source string
|
|
SourceType string // existing: "file","git","stdin","url","clipboard". New: "recon:<name>"
|
|
LineNumber int
|
|
Offset int64
|
|
DetectedAt time.Time
|
|
Verified bool
|
|
VerifyStatus string
|
|
VerifyHTTPCode int
|
|
VerifyMetadata map[string]string
|
|
VerifyError string
|
|
}
|
|
```
|
|
|
|
ants pool pattern (pkg/engine/engine.go): `ants.NewPool(workers)`, `pool.Submit(func(){...})`, `pool.Release()`, coordinated via `sync.WaitGroup`.
|
|
</interfaces>
|
|
</context>
|
|
|
|
<tasks>
|
|
|
|
<task type="auto" tdd="true">
|
|
<name>Task 1: Define ReconSource interface and Config</name>
|
|
<files>pkg/recon/source.go</files>
|
|
<behavior>
|
|
- ReconSource interface has methods: Name() string, RateLimit() rate.Limit, Burst() int, RespectsRobots() bool, Enabled(cfg Config) bool, Sweep(ctx, query, out chan<- Finding) error
|
|
- Finding is a type alias for pkg/engine.Finding so downstream code reuses the existing storage path
|
|
- Config struct carries Stealth bool, RespectRobots bool, EnabledSources []string, Query string
|
|
</behavior>
|
|
<action>
|
|
Create pkg/recon/source.go with package recon. Import golang.org/x/time/rate, context, and pkg/engine.
|
|
|
|
```go
|
|
package recon
|
|
|
|
import (
|
|
"context"
|
|
"golang.org/x/time/rate"
|
|
"github.com/salvacybersec/keyhunter/pkg/engine"
|
|
)
|
|
|
|
// Finding is the recon package's alias for the canonical engine.Finding.
|
|
// Recon sources set SourceType = "recon:<source-name>".
|
|
type Finding = engine.Finding
|
|
|
|
// Config controls a recon sweep.
|
|
type Config struct {
|
|
Stealth bool
|
|
RespectRobots bool
|
|
EnabledSources []string // empty = all
|
|
Query string
|
|
}
|
|
|
|
// ReconSource is implemented by every OSINT source module (Phases 10-16).
|
|
// Each source owns its own rate.Limiter constructed from RateLimit()/Burst().
|
|
type ReconSource interface {
|
|
Name() string
|
|
RateLimit() rate.Limit
|
|
Burst() int
|
|
RespectsRobots() bool
|
|
Enabled(cfg Config) bool
|
|
Sweep(ctx context.Context, query string, out chan<- Finding) error
|
|
}
|
|
```
|
|
|
|
Per Config decisions in 09-CONTEXT.md. No external deps beyond golang.org/x/time/rate (already in go.mod) and pkg/engine.
|
|
</action>
|
|
<verify>
|
|
<automated>cd /home/salva/Documents/apikey && go build ./pkg/recon/...</automated>
|
|
</verify>
|
|
<done>pkg/recon/source.go compiles; ReconSource interface exported; Finding aliased to engine.Finding.</done>
|
|
</task>
|
|
|
|
<task type="auto" tdd="true">
|
|
<name>Task 2: Engine with Register/List/SweepAll + ExampleSource + tests</name>
|
|
<files>pkg/recon/engine.go, pkg/recon/example.go, pkg/recon/engine_test.go</files>
|
|
<behavior>
|
|
- Engine.Register(src ReconSource) adds to internal map keyed by Name()
|
|
- Engine.List() returns sorted source names
|
|
- Engine.SweepAll(ctx, cfg) runs every enabled source in parallel via ants pool, collects Findings from a shared channel, and returns []Finding. Dedup is NOT done here (Plan 09-03 owns dedup.go); SweepAll just aggregates.
|
|
- Each source call is wrapped in its own goroutine submitted to ants.Pool; uses sync.WaitGroup to close the out channel after all sources finish
|
|
- ExampleSource.Name()="example", RateLimit()=rate.Limit(10), Burst()=1, RespectsRobots()=false, Enabled always true, Sweep emits two deterministic Findings with SourceType="recon:example"
|
|
- TestSweepAll registers ExampleSource, runs SweepAll, asserts exactly 2 findings with SourceType="recon:example"
|
|
- TestRegisterList asserts List() returns ["example"] after registering
|
|
</behavior>
|
|
<action>
|
|
Create pkg/recon/engine.go:
|
|
|
|
```go
|
|
package recon
|
|
|
|
import (
|
|
"context"
|
|
"sort"
|
|
"sync"
|
|
|
|
"github.com/panjf2000/ants/v2"
|
|
)
|
|
|
|
type Engine struct {
|
|
mu sync.RWMutex
|
|
sources map[string]ReconSource
|
|
}
|
|
|
|
func NewEngine() *Engine {
|
|
return &Engine{sources: make(map[string]ReconSource)}
|
|
}
|
|
|
|
func (e *Engine) Register(s ReconSource) {
|
|
e.mu.Lock()
|
|
defer e.mu.Unlock()
|
|
e.sources[s.Name()] = s
|
|
}
|
|
|
|
func (e *Engine) List() []string {
|
|
e.mu.RLock()
|
|
defer e.mu.RUnlock()
|
|
names := make([]string, 0, len(e.sources))
|
|
for n := range e.sources {
|
|
names = append(names, n)
|
|
}
|
|
sort.Strings(names)
|
|
return names
|
|
}
|
|
|
|
// SweepAll fans out to every enabled source in parallel via ants pool and
|
|
// returns aggregated findings. Deduplication is performed by callers using
|
|
// pkg/recon.Dedup (plan 09-03).
|
|
func (e *Engine) SweepAll(ctx context.Context, cfg Config) ([]Finding, error) {
|
|
e.mu.RLock()
|
|
active := make([]ReconSource, 0, len(e.sources))
|
|
for _, s := range e.sources {
|
|
if s.Enabled(cfg) {
|
|
active = append(active, s)
|
|
}
|
|
}
|
|
e.mu.RUnlock()
|
|
|
|
if len(active) == 0 {
|
|
return nil, nil
|
|
}
|
|
|
|
pool, err := ants.NewPool(len(active))
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
defer pool.Release()
|
|
|
|
out := make(chan Finding, 256)
|
|
var wg sync.WaitGroup
|
|
for _, s := range active {
|
|
s := s
|
|
wg.Add(1)
|
|
_ = pool.Submit(func() {
|
|
defer wg.Done()
|
|
_ = s.Sweep(ctx, cfg.Query, out)
|
|
})
|
|
}
|
|
go func() { wg.Wait(); close(out) }()
|
|
|
|
var all []Finding
|
|
for f := range out {
|
|
all = append(all, f)
|
|
}
|
|
return all, nil
|
|
}
|
|
```
|
|
|
|
Create pkg/recon/example.go with an ExampleSource emitting two deterministic Findings (SourceType="recon:example", fake masked keys, distinct Source URLs) to prove the pipeline.
|
|
|
|
```go
|
|
package recon
|
|
|
|
import (
|
|
"context"
|
|
"time"
|
|
|
|
"golang.org/x/time/rate"
|
|
)
|
|
|
|
type ExampleSource struct{}
|
|
|
|
func (ExampleSource) Name() string { return "example" }
|
|
func (ExampleSource) RateLimit() rate.Limit { return rate.Limit(10) }
|
|
func (ExampleSource) Burst() int { return 1 }
|
|
func (ExampleSource) RespectsRobots() bool { return false }
|
|
func (ExampleSource) Enabled(_ Config) bool { return true }
|
|
|
|
func (ExampleSource) Sweep(ctx context.Context, query string, out chan<- Finding) error {
|
|
fakes := []Finding{
|
|
{ProviderName: "openai", KeyMasked: "sk-examp...AAAA", Source: "https://example.invalid/a", SourceType: "recon:example", DetectedAt: time.Now()},
|
|
{ProviderName: "anthropic", KeyMasked: "sk-ant-e...BBBB", Source: "https://example.invalid/b", SourceType: "recon:example", DetectedAt: time.Now()},
|
|
}
|
|
for _, f := range fakes {
|
|
select {
|
|
case out <- f:
|
|
case <-ctx.Done():
|
|
return ctx.Err()
|
|
}
|
|
}
|
|
return nil
|
|
}
|
|
```
|
|
|
|
Create pkg/recon/engine_test.go with TestRegisterList and TestSweepAll using ExampleSource. Use testify require.
|
|
|
|
TDD: write tests first, they fail, then implement.
|
|
</action>
|
|
<verify>
|
|
<automated>cd /home/salva/Documents/apikey && go test ./pkg/recon/ -run 'TestRegisterList|TestSweepAll' -count=1</automated>
|
|
</verify>
|
|
<done>Tests pass. Engine registers ExampleSource, SweepAll returns 2 findings with SourceType="recon:example".</done>
|
|
</task>
|
|
|
|
</tasks>
|
|
|
|
<verification>
|
|
- `go build ./pkg/recon/...` succeeds
|
|
- `go test ./pkg/recon/ -count=1` passes
|
|
- `go vet ./pkg/recon/...` clean
|
|
</verification>
|
|
|
|
<success_criteria>
|
|
- ReconSource interface exported
|
|
- Engine.Register/List/SweepAll implemented and tested
|
|
- ExampleSource proves end-to-end fanout
|
|
- No cycles with pkg/engine (recon imports engine, not vice versa)
|
|
</success_criteria>
|
|
|
|
<output>
|
|
After completion, create `.planning/phases/09-osint-infrastructure/09-01-SUMMARY.md`
|
|
</output>
|
|
</content>
|