10 KiB
10 KiB
phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, must_haves
| phase | plan | type | wave | depends_on | files_modified | autonomous | requirements | must_haves | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 09-osint-infrastructure | 01 | execute | 1 |
|
true |
|
|
Purpose: Establish the interface + engine skeleton so subsequent Wave 1 plans (limiter, stealth, robots) can land in parallel without conflict, and Wave 2 can wire the CLI. Output: pkg/recon/source.go, pkg/recon/engine.go, pkg/recon/example.go, pkg/recon/engine_test.go
<execution_context> @$HOME/.claude/get-shit-done/workflows/execute-plan.md @$HOME/.claude/get-shit-done/templates/summary.md </execution_context>
@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @.planning/phases/09-osint-infrastructure/09-CONTEXT.md @pkg/engine/engine.go @pkg/engine/finding.goFrom pkg/engine/finding.go:
type Finding struct {
ProviderName string
KeyValue string
KeyMasked string
Confidence string
Source string
SourceType string // existing: "file","git","stdin","url","clipboard". New: "recon:<name>"
LineNumber int
Offset int64
DetectedAt time.Time
Verified bool
VerifyStatus string
VerifyHTTPCode int
VerifyMetadata map[string]string
VerifyError string
}
ants pool pattern (pkg/engine/engine.go): ants.NewPool(workers), pool.Submit(func(){...}), pool.Release(), coordinated via sync.WaitGroup.
```go
package recon
import (
"context"
"golang.org/x/time/rate"
"github.com/salvacybersec/keyhunter/pkg/engine"
)
// Finding is the recon package's alias for the canonical engine.Finding.
// Recon sources set SourceType = "recon:<source-name>".
type Finding = engine.Finding
// Config controls a recon sweep.
type Config struct {
Stealth bool
RespectRobots bool
EnabledSources []string // empty = all
Query string
}
// ReconSource is implemented by every OSINT source module (Phases 10-16).
// Each source owns its own rate.Limiter constructed from RateLimit()/Burst().
type ReconSource interface {
Name() string
RateLimit() rate.Limit
Burst() int
RespectsRobots() bool
Enabled(cfg Config) bool
Sweep(ctx context.Context, query string, out chan<- Finding) error
}
```
Per Config decisions in 09-CONTEXT.md. No external deps beyond golang.org/x/time/rate (already in go.mod) and pkg/engine.
```go
package recon
import (
"context"
"sort"
"sync"
"github.com/panjf2000/ants/v2"
)
type Engine struct {
mu sync.RWMutex
sources map[string]ReconSource
}
func NewEngine() *Engine {
return &Engine{sources: make(map[string]ReconSource)}
}
func (e *Engine) Register(s ReconSource) {
e.mu.Lock()
defer e.mu.Unlock()
e.sources[s.Name()] = s
}
func (e *Engine) List() []string {
e.mu.RLock()
defer e.mu.RUnlock()
names := make([]string, 0, len(e.sources))
for n := range e.sources {
names = append(names, n)
}
sort.Strings(names)
return names
}
// SweepAll fans out to every enabled source in parallel via ants pool and
// returns aggregated findings. Deduplication is performed by callers using
// pkg/recon.Dedup (plan 09-03).
func (e *Engine) SweepAll(ctx context.Context, cfg Config) ([]Finding, error) {
e.mu.RLock()
active := make([]ReconSource, 0, len(e.sources))
for _, s := range e.sources {
if s.Enabled(cfg) {
active = append(active, s)
}
}
e.mu.RUnlock()
if len(active) == 0 {
return nil, nil
}
pool, err := ants.NewPool(len(active))
if err != nil {
return nil, err
}
defer pool.Release()
out := make(chan Finding, 256)
var wg sync.WaitGroup
for _, s := range active {
s := s
wg.Add(1)
_ = pool.Submit(func() {
defer wg.Done()
_ = s.Sweep(ctx, cfg.Query, out)
})
}
go func() { wg.Wait(); close(out) }()
var all []Finding
for f := range out {
all = append(all, f)
}
return all, nil
}
```
Create pkg/recon/example.go with an ExampleSource emitting two deterministic Findings (SourceType="recon:example", fake masked keys, distinct Source URLs) to prove the pipeline.
```go
package recon
import (
"context"
"time"
"golang.org/x/time/rate"
)
type ExampleSource struct{}
func (ExampleSource) Name() string { return "example" }
func (ExampleSource) RateLimit() rate.Limit { return rate.Limit(10) }
func (ExampleSource) Burst() int { return 1 }
func (ExampleSource) RespectsRobots() bool { return false }
func (ExampleSource) Enabled(_ Config) bool { return true }
func (ExampleSource) Sweep(ctx context.Context, query string, out chan<- Finding) error {
fakes := []Finding{
{ProviderName: "openai", KeyMasked: "sk-examp...AAAA", Source: "https://example.invalid/a", SourceType: "recon:example", DetectedAt: time.Now()},
{ProviderName: "anthropic", KeyMasked: "sk-ant-e...BBBB", Source: "https://example.invalid/b", SourceType: "recon:example", DetectedAt: time.Now()},
}
for _, f := range fakes {
select {
case out <- f:
case <-ctx.Done():
return ctx.Err()
}
}
return nil
}
```
Create pkg/recon/engine_test.go with TestRegisterList and TestSweepAll using ExampleSource. Use testify require.
TDD: write tests first, they fail, then implement.
<success_criteria>
- ReconSource interface exported
- Engine.Register/List/SweepAll implemented and tested
- ExampleSource proves end-to-end fanout
- No cycles with pkg/engine (recon imports engine, not vice versa) </success_criteria>