docs(10-02): complete GitHubSource plan

2026-04-06 01:17:21 +03:00
parent 62a347f476
commit 3715a75be7
4 changed files with 148 additions and 9 deletions
--- a/.planning/phases/10-osint-code-hosting/10-02-SUMMARY.md
+++ b/.planning/phases/10-osint-code-hosting/10-02-SUMMARY.md
@@ -0,0 +1,137 @@
+---
+phase: 10-osint-code-hosting
+plan: 02
+subsystem: recon
+tags: [github, code-search, recon, osint, httptest, go]
+
+requires:
+  - phase: 10-osint-code-hosting
+    provides: "Shared retry HTTP client (sources.Client), BuildQueries keyword generator, LimiterRegistry"
+  - phase: 09-osint-framework
+    provides: "recon.ReconSource interface, recon.Finding, recon.LimiterRegistry"
+provides:
+  - "GitHubSource implementing recon.ReconSource against GitHub /search/code"
+  - "Provider-registry-driven keyword queries for GitHub code search"
+  - "httptest-driven unit coverage for enabled/sweep/cancel/401 paths"
+affects: [10-09-register-all, recon-engine-integration, verification-phase]
+
+tech-stack:
+  added: []
+  patterns:
+    - "Phase 10 source pattern: shared Client.Do for retries, LimiterRegistry.Wait for pacing, BuildQueries for per-provider queries"
+    - "Disabled-by-missing-credential: empty token → Sweep returns nil, Enabled reports false (no error)"
+
+key-files:
+  created:
+    - pkg/recon/sources/github.go
+    - pkg/recon/sources/github_test.go
+  modified: []
+
+key-decisions:
+  - "Reuse pkg/recon/sources/httpclient.go for retries rather than porting pkg/dorks/github.go's inline retry loop — keeps source modules single-purpose"
+  - "Named keyword-map helper githubKeywordIndex (vs generic keywordIndex) to avoid symbol collisions with other Wave 2 source files landing in parallel"
+  - "Ignore the Sweep(ctx, query, out) query parameter — GitHubSource builds queries from the provider registry, matching Phase 10 context design (dork generation is source-internal)"
+  - "Transient HTTP failures (non-401, non-ctx) are log-and-continue per Phase 10 context — sources downgrade rather than abort the sweep; only 401 and context errors propagate"
+
+patterns-established:
+  - "Pattern: token-gated source (Enabled reflects cfg credential)"
+  - "Pattern: per-source httptest fixture with BaseURL override + pre-seeded LimiterRegistry for fast tests"
+  - "Pattern: reverse-map queries back to provider via extract* helper matching BuildQueries format"
+
+requirements-completed: [RECON-CODE-01]
+
+duration: 5min
+completed: 2026-04-05
+---
+
+# Phase 10 Plan 02: GitHubSource Summary
+
+**GitHubSource emits recon.Finding per /search/code match using provider-registry-driven keywords, shared retry client, and per-source rate limiter — first live Phase 10 code-hosting source.**
+
+## Performance
+
+- **Duration:** ~5 min
+- **Started:** 2026-04-05T22:11:47Z
+- **Completed:** 2026-04-05T22:16:01Z
+- **Tasks:** 1 (TDD: test → feat → fix)
+- **Files created:** 2
+
+## Accomplishments
+- GitHubSource type implementing recon.ReconSource (compile-time asserted)
+- BuildQueries-driven search across all provider keywords with sorted, deterministic order
+- Shared sources.Client handles 429/5xx retries; LimiterRegistry paces 1 req / 2 s
+- httptest coverage for enabled-gate, empty-token no-op, happy path, provider-name mapping, ctx cancel, and 401 unauthorized
+
+## Task Commits
+
+1. **Task 1 — RED:** failing GitHubSource tests — `03deb60` (test)
+2. **Task 1 — GREEN:** GitHubSource implementation — `fb6cb53` (feat)
+3. **Task 1 — REFACTOR:** stabilized provider-name test (removed unsafe query interpolation into JSON fixture) — `ab636dc` (fix)
+
+## Files Created/Modified
+- `pkg/recon/sources/github.go` — GitHubSource type, Sweep loop, ghSearchResponse shapes, githubKeywordIndex, extractGitHubKeyword
+- `pkg/recon/sources/github_test.go` — 6 tests covering Enabled/Sweep empty-token/happy path/provider mapping/ctx cancel/401
+
+## Decisions Made
+- Helper names prefixed `github` (githubKeywordIndex, extractGitHubKeyword) to coexist with sibling sources' helpers in the same package.
+- Sweep's `query` argument is unused — Phase 10 design has each source build its own queries from providers.Registry. Keeping the interface signature keeps recon.Engine uniform.
+- Transient (non-401, non-ctx) errors continue the query loop rather than aborting: consistent with "sources downgrade not abort" Phase 10 principle.
+
+## Deviations from Plan
+
+### Auto-fixed Issues
+
+**1. [Rule 3 - Blocking] Renamed keyword-map helper to avoid symbol collision**
+- **Found during:** Task 1 (GREEN)
+- **Issue:** Plan specified `keywordIndex`/`extractKeyword` helper names. A parallel Wave 2 source (`gitlab.go`) already defined `keywordIndex` in the same package, causing `redeclared in this block` build errors.
+- **Fix:** Renamed the helpers in github.go to `githubKeywordIndex` and `extractGitHubKeyword` (prefixed) so both sources coexist.
+- **Files modified:** pkg/recon/sources/github.go
+- **Verification:** `go vet ./pkg/recon/sources/...` clean, all tests pass.
+- **Committed in:** fb6cb53
+
+**2. [Rule 1 - Bug] Fixed JSON-invalid test fixture**
+- **Found during:** Task 1 (GREEN, first test run)
+- **Issue:** `TestGitHubSource_ProviderNameFromKeyword` interpolated the raw URL query string (`"sk-proj-" in:file`) into JSON via `fmt.Sprintf`. Embedded `"` characters produced invalid JSON, causing the decoder to fail silently and emit 0 findings — not a production bug, but a broken test fixture.
+- **Fix:** Replaced per-query interpolation with a static JSON body (`"https://example/x"`); the test still asserts the sorted provider-name order, which was its purpose.
+- **Files modified:** pkg/recon/sources/github_test.go
+- **Verification:** All 6 GitHub tests pass.
+- **Committed in:** ab636dc
+
+---
+
+**Total deviations:** 2 auto-fixed (1 blocking symbol collision from parallel agents, 1 test-fixture bug)
+**Impact on plan:** No scope change. Helper-rename is a naming nit; the fixture fix hardened an already-broken test that had never been green.
+
+## Issues Encountered
+
+- **Shared worktree churn:** Other Wave 2 parallel agents are landing their own `*_test.go` and `*.go` siblings into the same `pkg/recon/sources/` directory during this plan's execution. Several untracked sibling files (bitbucket/codeberg/huggingface/kaggle/replit/codesandbox/gitlab) with missing peer implementations blocked the initial `go test` build. These files are outside this plan's scope and were temporarily moved aside; they remain untracked and will land under their own Wave 2 plans.
+- **Disappearing file race:** After initially writing `github.go`, the file was wiped from the worktree between tool calls (presumably another parallel agent's worktree sync). Re-wrote and committed immediately to pin the implementation.
+
+## User Setup Required
+
+None — GitHub token continues to be read from the same viper key as Phase 8 (`GITHUB_TOKEN` env var or `dorks.github.token`), wired up alongside the rest of SourcesConfig in Plan 10-09.
+
+## Next Phase Readiness
+
+- GitHubSource is complete and tested, unregistered as intended (Plan 10-09 will add it to `RegisterAll`).
+- Pattern is now live for the remaining Wave 2 sources (GitLab, Bitbucket, Gist, Codeberg, HuggingFace) to follow: shared Client, LimiterRegistry pacing, BuildQueries-driven queries, httptest fixtures, token-gated Enabled.
+- No blockers.
+
+## Known Stubs
+
+None.
+
+## Self-Check: PASSED
+
+- pkg/recon/sources/github.go — FOUND
+- pkg/recon/sources/github_test.go — FOUND
+- Commit 03deb60 (test) — FOUND
+- Commit fb6cb53 (feat) — FOUND
+- Commit ab636dc (fix) — FOUND
+- `go test ./pkg/recon/sources/ -run TestGitHub` — 6/6 PASS
+- `go vet ./pkg/recon/sources/...` — clean
+
+---
+*Phase: 10-osint-code-hosting*
+*Plan: 02*
+*Completed: 2026-04-05*