101 lines
5.5 KiB
Markdown
101 lines
5.5 KiB
Markdown
---
|
|
phase: 10-osint-code-hosting
|
|
plan: 09
|
|
subsystem: recon
|
|
tags: [register, integration, cmd, viper, httptest]
|
|
|
|
requires:
|
|
- phase: 10-osint-code-hosting
|
|
provides: "Ten code-hosting ReconSource implementations (Plans 10-01..10-08)"
|
|
provides:
|
|
- "sources.RegisterAll wires all ten Phase 10 sources onto a recon.Engine"
|
|
- "cmd/recon.go constructs real SourcesConfig from env + viper and invokes RegisterAll"
|
|
- "End-to-end SweepAll integration test exercising every source against one multiplexed httptest server"
|
|
affects: [11-osint-pastebins, 12-osint-search-engines, cli-recon]
|
|
|
|
tech-stack:
|
|
added: []
|
|
patterns:
|
|
- "Env-var → viper fallback (firstNonEmpty) for recon credential lookup"
|
|
- "Unconditional source registration: credless sources register but Enabled()==false, uniform CLI surface"
|
|
- "Single httptest.ServeMux routing per-path fixtures for multi-source integration tests"
|
|
|
|
key-files:
|
|
created:
|
|
- pkg/recon/sources/register_test.go
|
|
- pkg/recon/sources/integration_test.go
|
|
- .planning/phases/10-osint-code-hosting/deferred-items.md
|
|
modified:
|
|
- pkg/recon/sources/register.go
|
|
- cmd/recon.go
|
|
|
|
key-decisions:
|
|
- "Register all ten sources unconditionally so `keyhunter recon list` shows the full catalog regardless of configured credentials; missing creds just flip Enabled()==false"
|
|
- "Integration test constructs sources directly with BaseURL overrides (not via RegisterAll) because RegisterAll wires production URLs"
|
|
- "Credential precedence: env var → viper config key → empty (source disabled)"
|
|
- "Single multiplexed httptest server used instead of ten separate servers — simpler and matches how recon.Engine fans out in parallel"
|
|
- "firstNonEmpty helper kept local to cmd/recon.go rather than pkg-level to avoid exporting a trivial utility"
|
|
|
|
patterns-established:
|
|
- "sources.RegisterAll(engine, cfg) is the single call cmd-layer code must make to wire Phase 10"
|
|
- "Integration tests that need to drive many sources from one server encode the sub-source into the URL path (/search/code, /api/v4/search, etc.)"
|
|
- "Struct literals for sources that lazy-init `client` in Sweep; NewXxxSource constructor for sources that don't (GitHubSource, KaggleSource, HuggingFaceSource)"
|
|
|
|
requirements-completed: [RECON-CODE-10]
|
|
|
|
duration: 12min
|
|
completed: 2026-04-05
|
|
---
|
|
|
|
# Phase 10 Plan 09: RegisterAll + cmd/recon + Integration Test Summary
|
|
|
|
**Ten Phase 10 code-hosting sources now wire onto recon.Engine via sources.RegisterAll, the CLI reads credentials from env+viper, and an end-to-end integration test drives every source through SweepAll against one multiplexed httptest server.**
|
|
|
|
## Performance
|
|
|
|
- **Duration:** ~12 min
|
|
- **Tasks:** 2 (both TDD)
|
|
- **Files created:** 3
|
|
- **Files modified:** 2
|
|
|
|
## Accomplishments
|
|
|
|
- `sources.RegisterAll` wires all ten sources (github, gitlab, bitbucket, gist, codeberg, huggingface, replit, codesandbox, sandboxes, kaggle) onto a `*recon.Engine` in one call
|
|
- Extended `SourcesConfig` with `BitbucketWorkspace` and `CodebergToken` fields to match Wave 2 constructor signatures
|
|
- `cmd/recon.go` now loads providers.Registry, constructs a full `SourcesConfig` from env vars (`GITHUB_TOKEN`, `GITLAB_TOKEN`, `BITBUCKET_TOKEN`, `BITBUCKET_WORKSPACE`, `CODEBERG_TOKEN`, `HUGGINGFACE_TOKEN`, `KAGGLE_USERNAME`, `KAGGLE_KEY`) with viper fallback keys under `recon.<source>.*`, and calls `sources.RegisterAll`
|
|
- `keyhunter recon list` now prints all eleven source names (`example` + ten Phase 10 sources)
|
|
- Integration test (`integration_test.go::TestIntegration_AllSources_SweepAll`) spins up a single `httptest` server with per-path handlers for every source's API/HTML fixture, registers all ten sources (with BaseURL overrides) on a fresh `recon.Engine`, runs `SweepAll`, and asserts at least one `Finding` was emitted for each of the ten `recon:*` `SourceType` values
|
|
- `register_test.go` covers RegisterAll contracts: exactly ten sources registered in deterministic sorted order, nil engine is a no-op, and empty credentials still produce a full registration list
|
|
|
|
## Verification
|
|
|
|
- `go test ./pkg/recon/sources/ -run TestRegisterAll -v` → 4 passing (nil, empty cfg, all-ten, missing-creds)
|
|
- `go test ./pkg/recon/sources/ -run TestIntegration_AllSources_SweepAll -v` → passing; asserts 10/10 SourceType buckets populated
|
|
- `go test ./pkg/recon/...` → all green (35s, includes pre-existing per-source suites)
|
|
- `go vet ./...` → clean
|
|
- `go build ./...` → clean
|
|
- `go run . recon list` → prints `bitbucket codeberg codesandbox example gist github gitlab huggingface kaggle replit sandboxes`
|
|
|
|
## Deviations from Plan
|
|
|
|
None — plan executed as written. One out-of-scope finding was identified and logged to `deferred-items.md` (GitHubSource.Sweep dereferences `s.client` without a nil check; safe in current code paths because `RegisterAll` uses `NewGitHubSource` which initializes it, but a latent footgun for future struct-literal callers).
|
|
|
|
## Known Stubs
|
|
|
|
None. All ten sources are production-wired through `RegisterAll` and exercised by the integration test against realistic fixtures.
|
|
|
|
## Commits
|
|
|
|
- `4628ccf` test(10-09): add failing RegisterAll wiring tests
|
|
- `fb3e573` feat(10-09): wire all ten Phase 10 sources in RegisterAll
|
|
- `8528108` test(10-09): add end-to-end SweepAll integration test across all ten sources
|
|
- `e00fb17` feat(10-09): wire sources.RegisterAll into cmd/recon with viper+env credential lookup
|
|
|
|
## Self-Check: PASSED
|
|
|
|
- pkg/recon/sources/register.go — FOUND
|
|
- pkg/recon/sources/register_test.go — FOUND
|
|
- pkg/recon/sources/integration_test.go — FOUND
|
|
- cmd/recon.go — FOUND
|
|
- commits 4628ccf, fb3e573, 8528108, e00fb17 — FOUND
|