12 KiB
12 KiB
phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, must_haves
| phase | plan | type | wave | depends_on | files_modified | autonomous | requirements | must_haves | ||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 10-osint-code-hosting | 01 | execute | 1 |
|
true |
|
Purpose: Deduplicate retry/rate-limit/backoff logic across 10 sources; centralize query
generation so providers added later automatically flow to every source.
Output: Compilable pkg/recon/sources package skeleton with tested helpers.
<execution_context> @$HOME/.claude/get-shit-done/workflows/execute-plan.md @$HOME/.claude/get-shit-done/templates/summary.md </execution_context>
@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @.planning/phases/10-osint-code-hosting/10-CONTEXT.md @pkg/recon/source.go @pkg/recon/limiter.go @pkg/dorks/github.go @pkg/providers/registry.go From pkg/recon/source.go: ```go type ReconSource interface { Name() string RateLimit() rate.Limit Burst() int RespectsRobots() bool Enabled(cfg Config) bool Sweep(ctx context.Context, query string, out chan<- Finding) error } type Finding = engine.Finding type Config struct { Stealth, RespectRobots bool; EnabledSources []string; Query string } ```From pkg/recon/limiter.go:
type LimiterRegistry struct { ... }
func NewLimiterRegistry() *LimiterRegistry
func (lr *LimiterRegistry) Wait(ctx, name, r, burst, stealth) error
From pkg/providers/registry.go:
func (r *Registry) List() []Provider
// Provider has: Name string, Keywords []string, Patterns []Pattern, Tier int
From pkg/engine/finding.go:
type Finding struct {
ProviderName, KeyValue, KeyMasked, Confidence, Source, SourceType string
LineNumber int; Offset int64; DetectedAt time.Time
Verified bool; VerifyStatus string; ...
}
Create `pkg/recon/sources/httpclient.go` exporting:
```go
package sources
import (
"context"
"errors"
"fmt"
"net/http"
"strconv"
"time"
)
// ErrUnauthorized is returned when an API rejects credentials (401).
var ErrUnauthorized = errors.New("sources: unauthorized (check credentials)")
// Client is the shared retry wrapper every Phase 10 source uses.
type Client struct {
HTTP *http.Client
MaxRetries int // default 2
UserAgent string // default "keyhunter-recon/1.0"
}
// NewClient returns a Client with a 30s timeout and 2 retries.
func NewClient() *Client {
return &Client{HTTP: &http.Client{Timeout: 30 * time.Second}, MaxRetries: 2, UserAgent: "keyhunter-recon/1.0"}
}
// Do executes req with retries on 429/403/5xx honoring Retry-After.
// 401 returns ErrUnauthorized wrapped with the response body.
// Ctx cancellation is honored during sleeps.
func (c *Client) Do(ctx context.Context, req *http.Request) (*http.Response, error) {
if req.Header.Get("User-Agent") == "" { req.Header.Set("User-Agent", c.UserAgent) }
var last *http.Response
for attempt := 0; attempt <= c.MaxRetries; attempt++ {
r, err := c.HTTP.Do(req.WithContext(ctx))
if err != nil { return nil, fmt.Errorf("sources http: %w", err) }
if r.StatusCode == http.StatusOK { return r, nil }
if r.StatusCode == http.StatusUnauthorized {
body := readBody(r)
return nil, fmt.Errorf("%w: %s", ErrUnauthorized, body)
}
retriable := r.StatusCode == 429 || r.StatusCode == 403 || r.StatusCode >= 500
if !retriable || attempt == c.MaxRetries {
body := readBody(r)
return nil, fmt.Errorf("sources http %d: %s", r.StatusCode, body)
}
sleep := ParseRetryAfter(r.Header.Get("Retry-After"))
r.Body.Close()
last = r
select {
case <-time.After(sleep):
case <-ctx.Done(): return nil, ctx.Err()
}
}
_ = last
return nil, fmt.Errorf("sources http: retries exhausted")
}
// ParseRetryAfter decodes integer-seconds Retry-After, defaulting to 1s.
func ParseRetryAfter(v string) time.Duration { ... }
// readBody reads up to 4KB of the body and closes it.
func readBody(r *http.Response) string { ... }
```
Create `pkg/recon/sources/httpclient_test.go` using `net/http/httptest`:
- Table-driven tests for each behavior above. Use an atomic counter to verify
retry attempt counts. Use `httptest.NewServer` with a handler that switches on
a request counter.
- For ctx cancellation test: set Retry-After: 10, cancel ctx inside 100ms, assert
ctx.Err() returned within 500ms.
Do NOT build a LimiterRegistry wrapper here — each source calls its own LimiterRegistry.Wait
before calling Client.Do. Keeps Client single-purpose (retry only).
import (
"fmt"
"sort"
"github.com/salvacybersec/keyhunter/pkg/providers"
)
// BuildQueries produces the search-string list a source should iterate for a
// given provider registry. Each keyword is formatted per source-specific syntax.
// Result is deterministic (sorted) for reproducible tests.
func BuildQueries(reg *providers.Registry, source string) []string {
if reg == nil { return nil }
seen := make(map[string]struct{})
for _, p := range reg.List() {
for _, k := range p.Keywords {
if k == "" { continue }
seen[k] = struct{}{}
}
}
keywords := make([]string, 0, len(seen))
for k := range seen { keywords = append(keywords, k) }
sort.Strings(keywords)
out := make([]string, 0, len(keywords))
for _, k := range keywords {
out = append(out, formatQuery(source, k))
}
return out
}
func formatQuery(source, keyword string) string {
switch source {
case "github", "gist":
return fmt.Sprintf("%q in:file", keyword)
case "gitlab":
return keyword // GitLab code search doesn't support in:file qualifier
case "bitbucket":
return keyword
case "codeberg":
return keyword
default:
return keyword
}
}
```
Create `pkg/recon/sources/queries_test.go` using `providers.NewRegistryFromProviders`
with two synthetic providers (shared keyword to test dedup).
Create `pkg/recon/sources/register.go`:
```go
package sources
import (
"github.com/salvacybersec/keyhunter/pkg/providers"
"github.com/salvacybersec/keyhunter/pkg/recon"
)
// SourcesConfig carries per-source credentials read from viper/env by cmd/recon.go.
// Plan 10-09 fleshes this out; for now it is a placeholder struct so downstream
// plans can depend on its shape.
type SourcesConfig struct {
GitHubToken string
GitLabToken string
BitbucketToken string
HuggingFaceToken string
KaggleUser string
KaggleKey string
Registry *providers.Registry
Limiters *recon.LimiterRegistry
}
// RegisterAll registers every Phase 10 code-hosting source on engine.
// Wave 2 plans append their source constructors here via additional
// registerXxx helpers in this file. Plan 10-09 writes the final list.
func RegisterAll(engine *recon.Engine, cfg SourcesConfig) {
if engine == nil { return }
// Populated by Plan 10-09 (after Wave 2 lands individual source files).
}
```
Do NOT wire this into cmd/recon.go yet — Plan 10-09 handles CLI integration after
every source exists.
<success_criteria> pkg/recon/sources package exists with httpclient.go, queries.go, register.go, doc.go and all tests green. No source implementations present yet — that is Wave 2. </success_criteria>
After completion, create `.planning/phases/10-osint-code-hosting/10-01-SUMMARY.md`.