9.4 KiB
9.4 KiB
phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, must_haves
| phase | plan | type | wave | depends_on | files_modified | autonomous | requirements | must_haves | ||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 10-osint-code-hosting | 02 | execute | 2 |
|
|
true |
|
|
Purpose: RECON-CODE-01 — users can scan GitHub public code for leaked LLM keys. Output: pkg/recon/sources/github.go + green tests.
<execution_context> @$HOME/.claude/get-shit-done/workflows/execute-plan.md @$HOME/.claude/get-shit-done/templates/summary.md </execution_context>
@.planning/phases/10-osint-code-hosting/10-CONTEXT.md @.planning/phases/10-osint-code-hosting/10-01-SUMMARY.md @pkg/recon/source.go @pkg/recon/limiter.go @pkg/dorks/github.go @pkg/recon/sources/httpclient.go @pkg/recon/sources/queries.go @pkg/recon/sources/register.go Reference pkg/dorks/github.go for the response struct shapes (ghSearchResponse, ghCodeItem, ghRepository, ghTextMatchEntry) — copy or alias them. GitHub Code Search endpoint: GET /search/code?q=&per_page= with headers: - Accept: application/vnd.github.v3.text-match+json - Authorization: Bearer - User-Agent: keyhunter-reconRate limit: 30 req/min authenticated → rate.Every(2*time.Second), burst 1.
Task 1: GitHubSource implementation + tests pkg/recon/sources/github.go, pkg/recon/sources/github_test.go - Test A: Enabled returns false when token empty; true when token set - Test B: Sweep with empty token returns nil (no error, logs disabled) - Test C: Sweep against httptest server decodes a 2-item response, emits 2 Findings on channel with SourceType="recon:github" and Source=html_url - Test D: ProviderName is derived by matching query keyword back to provider via the registry (pass in synthetic registry) - Test E: Ctx cancellation before first request returns ctx.Err() - Test F: 401 from server returns wrapped ErrUnauthorized - Test G: Multiple queries (from BuildQueries) iterate in sorted order Create `pkg/recon/sources/github.go`: ```go package sourcesimport (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"net/url"
"time"
"golang.org/x/time/rate"
"github.com/salvacybersec/keyhunter/pkg/providers"
"github.com/salvacybersec/keyhunter/pkg/recon"
)
// GitHubSource implements recon.ReconSource against GitHub Code Search.
// RECON-CODE-01.
type GitHubSource struct {
Token string
BaseURL string // default https://api.github.com, overridable for tests
Registry *providers.Registry
Limiters *recon.LimiterRegistry
client *Client
}
// NewGitHubSource constructs a source. If client is nil, NewClient() is used.
func NewGitHubSource(token string, reg *providers.Registry, lim *recon.LimiterRegistry) *GitHubSource {
return &GitHubSource{Token: token, BaseURL: "https://api.github.com", Registry: reg, Limiters: lim, client: NewClient()}
}
func (s *GitHubSource) Name() string { return "github" }
func (s *GitHubSource) RateLimit() rate.Limit { return rate.Every(2 * time.Second) }
func (s *GitHubSource) Burst() int { return 1 }
func (s *GitHubSource) RespectsRobots() bool { return false }
func (s *GitHubSource) Enabled(_ recon.Config) bool { return s.Token != "" }
func (s *GitHubSource) Sweep(ctx context.Context, _ string, out chan<- recon.Finding) error {
if s.Token == "" { return nil }
base := s.BaseURL
if base == "" { base = "https://api.github.com" }
queries := BuildQueries(s.Registry, "github")
kwToProvider := keywordIndex(s.Registry)
for _, q := range queries {
if err := ctx.Err(); err != nil { return err }
if s.Limiters != nil {
if err := s.Limiters.Wait(ctx, s.Name(), s.RateLimit(), s.Burst(), false); err != nil { return err }
}
endpoint := fmt.Sprintf("%s/search/code?q=%s&per_page=30", base, url.QueryEscape(q))
req, _ := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
req.Header.Set("Accept", "application/vnd.github.v3.text-match+json")
req.Header.Set("Authorization", "Bearer "+s.Token)
resp, err := s.client.Do(ctx, req)
if err != nil {
if errors.Is(err, ErrUnauthorized) { return err }
// Other errors: log-and-continue per CONTEXT (sources downgrade, not abort)
continue
}
var parsed ghSearchResponse
_ = json.NewDecoder(resp.Body).Decode(&parsed)
resp.Body.Close()
provName := kwToProvider[extractKeyword(q)]
for _, it := range parsed.Items {
snippet := ""
if len(it.TextMatches) > 0 { snippet = it.TextMatches[0].Fragment }
f := recon.Finding{
ProviderName: provName,
KeyMasked: "",
Confidence: "low",
Source: it.HTMLURL,
SourceType: "recon:github",
DetectedAt: time.Now(),
}
_ = snippet // reserved for future content scan pass
select {
case out <- f:
case <-ctx.Done(): return ctx.Err()
}
}
}
return nil
}
// Response structs mirror pkg/dorks/github.go (kept private to this file
// to avoid cross-package coupling between dorks and recon/sources).
type ghSearchResponse struct { Items []ghCodeItem `json:"items"` }
type ghCodeItem struct {
HTMLURL string `json:"html_url"`
Repository ghRepository `json:"repository"`
TextMatches []ghTextMatchEntry `json:"text_matches"`
}
type ghRepository struct { FullName string `json:"full_name"` }
type ghTextMatchEntry struct { Fragment string `json:"fragment"` }
// keywordIndex maps keyword -> provider name using the registry.
func keywordIndex(reg *providers.Registry) map[string]string {
m := make(map[string]string)
if reg == nil { return m }
for _, p := range reg.List() {
for _, k := range p.Keywords { m[k] = p.Name }
}
return m
}
// extractKeyword parses the provider keyword out of a BuildQueries output.
// For github it's `"keyword" in:file`; for bare formats it's the whole string.
func extractKeyword(q string) string { ... strip quotes, trim ` in:file` suffix ... }
```
Create `pkg/recon/sources/github_test.go`:
- Use `providers.NewRegistryFromProviders` with 2 synthetic providers (openai/sk-proj-, anthropic/sk-ant-)
- Spin up `httptest.NewServer` that inspects `r.URL.Query().Get("q")` and returns
a JSON body with two items whose html_url encodes the query
- Assert 2 findings per query received on the channel within 2s using select/time.After
- Separate test for empty token: NewGitHubSource("", reg, lim).Sweep returns nil immediately
- Separate test for 401: server returns 401 → Sweep returns error wrapping ErrUnauthorized
- Cancel-test: cancel ctx before Sweep call; assert ctx.Err() returned
Leave GitHubSource unregistered (Plan 10-09 adds it to RegisterAll).
<success_criteria> RECON-CODE-01 satisfied: GitHubSource queries /search/code using provider-registry-driven keywords and emits engine.Finding. Ready for registration in Plan 10-09. </success_criteria>
After completion, create `.planning/phases/10-osint-code-hosting/10-02-SUMMARY.md`.