--- phase: 10-osint-code-hosting plan: 02 type: execute wave: 2 depends_on: [10-01] files_modified: - pkg/recon/sources/github.go - pkg/recon/sources/github_test.go autonomous: true requirements: [RECON-CODE-01] must_haves: truths: - "GitHubSource.Sweep runs BuildQueries against GitHub /search/code and emits engine.Finding per match" - "GitHubSource is disabled when cfg token is empty (logs and returns nil, no error)" - "GitHubSource honors ctx cancellation mid-query and rate limiter tokens before each request" - "Each Finding has SourceType=\"recon:github\" and Source = html_url" artifacts: - path: "pkg/recon/sources/github.go" provides: "GitHubSource implementing recon.ReconSource" contains: "func (s *GitHubSource) Sweep" - path: "pkg/recon/sources/github_test.go" provides: "httptest-driven unit tests" key_links: - from: "pkg/recon/sources/github.go" to: "pkg/recon/sources/httpclient.go" via: "Client.Do" pattern: "c\\.client\\.Do" - from: "pkg/recon/sources/github.go" to: "pkg/recon/sources/queries.go" via: "BuildQueries(reg, \"github\")" pattern: "BuildQueries" --- Implement GitHubSource — the first real Phase 10 recon source. Refactors logic from pkg/dorks/github.go (Phase 8's GitHubExecutor) into a recon.ReconSource. Emits engine.Finding entries for every /search/code match, driven by provider keyword queries from pkg/recon/sources/queries.go. Purpose: RECON-CODE-01 — users can scan GitHub public code for leaked LLM keys. Output: pkg/recon/sources/github.go + green tests. @$HOME/.claude/get-shit-done/workflows/execute-plan.md @$HOME/.claude/get-shit-done/templates/summary.md @.planning/phases/10-osint-code-hosting/10-CONTEXT.md @.planning/phases/10-osint-code-hosting/10-01-SUMMARY.md @pkg/recon/source.go @pkg/recon/limiter.go @pkg/dorks/github.go @pkg/recon/sources/httpclient.go @pkg/recon/sources/queries.go @pkg/recon/sources/register.go Reference pkg/dorks/github.go for the response struct shapes (ghSearchResponse, ghCodeItem, ghRepository, ghTextMatchEntry) — copy or alias them. GitHub Code Search endpoint: GET /search/code?q=&per_page= with headers: - Accept: application/vnd.github.v3.text-match+json - Authorization: Bearer - User-Agent: keyhunter-recon Rate limit: 30 req/min authenticated → rate.Every(2*time.Second), burst 1. Task 1: GitHubSource implementation + tests pkg/recon/sources/github.go, pkg/recon/sources/github_test.go - Test A: Enabled returns false when token empty; true when token set - Test B: Sweep with empty token returns nil (no error, logs disabled) - Test C: Sweep against httptest server decodes a 2-item response, emits 2 Findings on channel with SourceType="recon:github" and Source=html_url - Test D: ProviderName is derived by matching query keyword back to provider via the registry (pass in synthetic registry) - Test E: Ctx cancellation before first request returns ctx.Err() - Test F: 401 from server returns wrapped ErrUnauthorized - Test G: Multiple queries (from BuildQueries) iterate in sorted order Create `pkg/recon/sources/github.go`: ```go package sources import ( "context" "encoding/json" "errors" "fmt" "net/http" "net/url" "time" "golang.org/x/time/rate" "github.com/salvacybersec/keyhunter/pkg/providers" "github.com/salvacybersec/keyhunter/pkg/recon" ) // GitHubSource implements recon.ReconSource against GitHub Code Search. // RECON-CODE-01. type GitHubSource struct { Token string BaseURL string // default https://api.github.com, overridable for tests Registry *providers.Registry Limiters *recon.LimiterRegistry client *Client } // NewGitHubSource constructs a source. If client is nil, NewClient() is used. func NewGitHubSource(token string, reg *providers.Registry, lim *recon.LimiterRegistry) *GitHubSource { return &GitHubSource{Token: token, BaseURL: "https://api.github.com", Registry: reg, Limiters: lim, client: NewClient()} } func (s *GitHubSource) Name() string { return "github" } func (s *GitHubSource) RateLimit() rate.Limit { return rate.Every(2 * time.Second) } func (s *GitHubSource) Burst() int { return 1 } func (s *GitHubSource) RespectsRobots() bool { return false } func (s *GitHubSource) Enabled(_ recon.Config) bool { return s.Token != "" } func (s *GitHubSource) Sweep(ctx context.Context, _ string, out chan<- recon.Finding) error { if s.Token == "" { return nil } base := s.BaseURL if base == "" { base = "https://api.github.com" } queries := BuildQueries(s.Registry, "github") kwToProvider := keywordIndex(s.Registry) for _, q := range queries { if err := ctx.Err(); err != nil { return err } if s.Limiters != nil { if err := s.Limiters.Wait(ctx, s.Name(), s.RateLimit(), s.Burst(), false); err != nil { return err } } endpoint := fmt.Sprintf("%s/search/code?q=%s&per_page=30", base, url.QueryEscape(q)) req, _ := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil) req.Header.Set("Accept", "application/vnd.github.v3.text-match+json") req.Header.Set("Authorization", "Bearer "+s.Token) resp, err := s.client.Do(ctx, req) if err != nil { if errors.Is(err, ErrUnauthorized) { return err } // Other errors: log-and-continue per CONTEXT (sources downgrade, not abort) continue } var parsed ghSearchResponse _ = json.NewDecoder(resp.Body).Decode(&parsed) resp.Body.Close() provName := kwToProvider[extractKeyword(q)] for _, it := range parsed.Items { snippet := "" if len(it.TextMatches) > 0 { snippet = it.TextMatches[0].Fragment } f := recon.Finding{ ProviderName: provName, KeyMasked: "", Confidence: "low", Source: it.HTMLURL, SourceType: "recon:github", DetectedAt: time.Now(), } _ = snippet // reserved for future content scan pass select { case out <- f: case <-ctx.Done(): return ctx.Err() } } } return nil } // Response structs mirror pkg/dorks/github.go (kept private to this file // to avoid cross-package coupling between dorks and recon/sources). type ghSearchResponse struct { Items []ghCodeItem `json:"items"` } type ghCodeItem struct { HTMLURL string `json:"html_url"` Repository ghRepository `json:"repository"` TextMatches []ghTextMatchEntry `json:"text_matches"` } type ghRepository struct { FullName string `json:"full_name"` } type ghTextMatchEntry struct { Fragment string `json:"fragment"` } // keywordIndex maps keyword -> provider name using the registry. func keywordIndex(reg *providers.Registry) map[string]string { m := make(map[string]string) if reg == nil { return m } for _, p := range reg.List() { for _, k := range p.Keywords { m[k] = p.Name } } return m } // extractKeyword parses the provider keyword out of a BuildQueries output. // For github it's `"keyword" in:file`; for bare formats it's the whole string. func extractKeyword(q string) string { ... strip quotes, trim ` in:file` suffix ... } ``` Create `pkg/recon/sources/github_test.go`: - Use `providers.NewRegistryFromProviders` with 2 synthetic providers (openai/sk-proj-, anthropic/sk-ant-) - Spin up `httptest.NewServer` that inspects `r.URL.Query().Get("q")` and returns a JSON body with two items whose html_url encodes the query - Assert 2 findings per query received on the channel within 2s using select/time.After - Separate test for empty token: NewGitHubSource("", reg, lim).Sweep returns nil immediately - Separate test for 401: server returns 401 → Sweep returns error wrapping ErrUnauthorized - Cancel-test: cancel ctx before Sweep call; assert ctx.Err() returned Leave GitHubSource unregistered (Plan 10-09 adds it to RegisterAll). cd /home/salva/Documents/apikey && go test ./pkg/recon/sources/ -run TestGitHub -v -timeout 30s GitHubSource satisfies recon.ReconSource (compile-time assert via `var _ recon.ReconSource = (*GitHubSource)(nil)`), tests green, covers happy path + empty token + 401 + cancellation. - `go build ./...` - `go test ./pkg/recon/sources/ -run TestGitHub -v` - `go vet ./pkg/recon/sources/...` RECON-CODE-01 satisfied: GitHubSource queries /search/code using provider-registry-driven keywords and emits engine.Finding. Ready for registration in Plan 10-09. After completion, create `.planning/phases/10-osint-code-hosting/10-02-SUMMARY.md`.