merge: phase 14-04 register wiring
This commit is contained in:
salvacybersec
@@ -0,0 +1,162 @@
|
||||
---
|
||||
phase: 14-osint_ci_cd_logs_web_archives_frontend_leaks
|
||||
plan: 04
|
||||
subsystem: recon
|
||||
tags: [registerall, wiring, integration-test, ci-cd, archives, frontend, jsbundle]
|
||||
|
||||
requires:
|
||||
- phase: 14-osint_ci_cd_logs_web_archives_frontend_leaks
|
||||
provides: "5 frontend leak sources (sourcemap, webpack, envleak, swagger, deploypreview)"
|
||||
- phase: 13-osint-package-registries
|
||||
provides: "RegisterAll with 40 sources baseline"
|
||||
provides:
|
||||
- "TravisCISource for scraping public Travis CI build logs"
|
||||
- "GitHubActionsSource for searching Actions workflow logs"
|
||||
- "CircleCISource for scraping CircleCI pipeline logs"
|
||||
- "JenkinsSource for scraping public Jenkins console output"
|
||||
- "WaybackMachineSource for searching archived pages via CDX API"
|
||||
- "CommonCrawlSource for searching Common Crawl index"
|
||||
- "JSBundleSource for probing JS bundles for embedded API key literals"
|
||||
- "RegisterAll extended to 52 sources"
|
||||
affects: [15, 16]
|
||||
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns: ["CI log scraping pattern", "CDX index querying pattern"]
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- pkg/recon/sources/travisci.go
|
||||
- pkg/recon/sources/travisci_test.go
|
||||
- pkg/recon/sources/githubactions.go
|
||||
- pkg/recon/sources/githubactions_test.go
|
||||
- pkg/recon/sources/circleci.go
|
||||
- pkg/recon/sources/circleci_test.go
|
||||
- pkg/recon/sources/jenkins.go
|
||||
- pkg/recon/sources/jenkins_test.go
|
||||
- pkg/recon/sources/wayback.go
|
||||
- pkg/recon/sources/wayback_test.go
|
||||
- pkg/recon/sources/commoncrawl.go
|
||||
- pkg/recon/sources/commoncrawl_test.go
|
||||
- pkg/recon/sources/jsbundle.go
|
||||
- pkg/recon/sources/jsbundle_test.go
|
||||
modified:
|
||||
- pkg/recon/sources/register.go
|
||||
- pkg/recon/sources/register_test.go
|
||||
- pkg/recon/sources/integration_test.go
|
||||
- cmd/recon.go
|
||||
|
||||
key-decisions:
|
||||
- "CircleCIToken added to SourcesConfig (credential-gated); GitHubActionsSource reuses GitHubToken"
|
||||
- "TravisCI and Jenkins are credentialless (public build logs); CircleCI and GitHubActions require tokens"
|
||||
- "WaybackMachine and CommonCrawl are credentialless (public CDX APIs)"
|
||||
- "JSBundleSource complements WebpackSource by targeting raw key literals rather than env var prefixes"
|
||||
- "Integration test uses nil Limiters for Phase 14 sources to avoid rate-limit delays"
|
||||
|
||||
patterns-established:
|
||||
- "CI log scraping: fetch build list then iterate log endpoints with ciLogKeyPattern"
|
||||
- "CDX index querying: search by URL pattern then fetch archived content"
|
||||
|
||||
duration: 11min
|
||||
completed: 2026-04-06
|
||||
---
|
||||
|
||||
# Phase 14 Plan 04: RegisterAll Wiring + Integration Test Summary
|
||||
|
||||
**Wire all 12 Phase 14 sources into RegisterAll (52 total) with full integration test coverage across CI/CD logs, web archives, frontend leaks, and JS bundle analysis**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** 11 min
|
||||
- **Started:** 2026-04-06T10:23:37Z
|
||||
- **Completed:** 2026-04-06T10:34:26Z
|
||||
- **Tasks:** 2
|
||||
- **Files modified:** 18
|
||||
|
||||
## Accomplishments
|
||||
|
||||
- Created 7 new source implementations: TravisCISource, GitHubActionsSource, CircleCISource, JenkinsSource, WaybackMachineSource, CommonCrawlSource, JSBundleSource
|
||||
- Each source follows the established ReconSource pattern with httptest-based unit tests
|
||||
- RegisterAll extended from 45 to 52 sources (all Phase 10-14 sources)
|
||||
- CircleCIToken added to SourcesConfig with CIRCLECI_TOKEN env var lookup in cmd/recon.go
|
||||
- Integration test updated from 40 to 52 source validation with dedicated httptest handlers
|
||||
- All 52 sources verified end-to-end via SweepAll integration test
|
||||
|
||||
## Task Commits
|
||||
|
||||
1. **Task 1: Create 7 new Phase 14 source implementations** - `169b80b` (feat)
|
||||
2. **Task 2: Wire into RegisterAll + update tests** - `7ef6c2a` (feat)
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
### Created (14 files)
|
||||
- `pkg/recon/sources/travisci.go` - Travis CI build log scraping
|
||||
- `pkg/recon/sources/travisci_test.go` - httptest-based tests
|
||||
- `pkg/recon/sources/githubactions.go` - GitHub Actions log searching
|
||||
- `pkg/recon/sources/githubactions_test.go` - httptest-based tests
|
||||
- `pkg/recon/sources/circleci.go` - CircleCI pipeline log scraping
|
||||
- `pkg/recon/sources/circleci_test.go` - httptest-based tests
|
||||
- `pkg/recon/sources/jenkins.go` - Jenkins console output scraping
|
||||
- `pkg/recon/sources/jenkins_test.go` - httptest-based tests
|
||||
- `pkg/recon/sources/wayback.go` - Wayback Machine CDX API searching
|
||||
- `pkg/recon/sources/wayback_test.go` - httptest-based tests
|
||||
- `pkg/recon/sources/commoncrawl.go` - Common Crawl index searching
|
||||
- `pkg/recon/sources/commoncrawl_test.go` - httptest-based tests
|
||||
- `pkg/recon/sources/jsbundle.go` - JS bundle API key detection
|
||||
- `pkg/recon/sources/jsbundle_test.go` - httptest-based tests
|
||||
|
||||
### Modified (4 files)
|
||||
- `pkg/recon/sources/register.go` - Extended RegisterAll to 52 sources, added CircleCIToken to SourcesConfig
|
||||
- `pkg/recon/sources/register_test.go` - Updated expected source count and name list to 52
|
||||
- `pkg/recon/sources/integration_test.go` - Added handlers and registrations for all 12 Phase 14 sources
|
||||
- `cmd/recon.go` - Added CircleCIToken with env/viper lookup
|
||||
|
||||
## Decisions Made
|
||||
|
||||
- CircleCIToken is credential-gated (Enabled returns false without token); GitHubActionsSource reuses existing GitHubToken
|
||||
- TravisCI and Jenkins are credentialless (public build logs accessible without auth)
|
||||
- WaybackMachine and CommonCrawl are credentialless (public CDX APIs)
|
||||
- JSBundleSource targets raw key literals (apiKey:"...", Authorization:"Bearer ...") complementing WebpackSource's env var prefix detection
|
||||
- Integration test uses nil Limiters for Phase 14 sources to avoid 30s+ rate-limit delays in CI
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
### Auto-fixed Issues
|
||||
|
||||
**1. [Rule 2 - Missing Critical] Frontend leak sources missing from integration test**
|
||||
- **Found during:** Integration test update
|
||||
- **Issue:** Plan 03 added 5 frontend leak sources to RegisterAll but didn't add them to the integration test (test still counted 40 sources)
|
||||
- **Fix:** Added httptest handlers and source registrations for all 5 frontend leak sources alongside the 7 new sources
|
||||
- **Files modified:** pkg/recon/sources/integration_test.go
|
||||
- **Commit:** 7ef6c2a
|
||||
|
||||
---
|
||||
|
||||
**Total deviations:** 1 auto-fixed (missing critical)
|
||||
**Impact on plan:** Necessary for integration test correctness.
|
||||
|
||||
## Issues Encountered
|
||||
|
||||
None.
|
||||
|
||||
## User Setup Required
|
||||
|
||||
For CI/CD sources requiring credentials:
|
||||
- **GitHubActionsSource:** Set `GITHUB_TOKEN` env var (reuses existing GitHub token)
|
||||
- **CircleCISource:** Set `CIRCLECI_TOKEN` env var or `recon.circleci.token` config key
|
||||
|
||||
All other Phase 14 sources (TravisCI, Jenkins, WaybackMachine, CommonCrawl, JSBundle, SourceMap, Webpack, EnvLeak, Swagger, DeployPreview) are credentialless.
|
||||
|
||||
## Known Stubs
|
||||
|
||||
None - all sources are fully implemented with real scanning logic.
|
||||
|
||||
## Next Phase Readiness
|
||||
|
||||
- 52 sources now registered in RegisterAll across Phases 10-14
|
||||
- Phase 14 complete: CI/CD logs, web archives, frontend leaks, JS bundles all covered
|
||||
- Ready for Phase 15+ expansion
|
||||
|
||||
---
|
||||
*Phase: 14-osint_ci_cd_logs_web_archives_frontend_leaks*
|
||||
*Completed: 2026-04-06*
|
||||
Reference in New Issue
Block a user