diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 0832179..5737871 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -173,11 +173,11 @@ Requirements for initial release. Each maps to roadmap phases. ### OSINT/Recon — Frontend & JS Leaks -- [ ] **RECON-JS-01**: JavaScript source map extraction and scanning -- [ ] **RECON-JS-02**: Webpack/Vite bundle scanning for inlined env vars -- [ ] **RECON-JS-03**: Exposed .env file scanning on web servers -- [ ] **RECON-JS-04**: Exposed Swagger/OpenAPI documentation scanning -- [ ] **RECON-JS-05**: Vercel/Netlify deploy preview JS bundle scanning +- [x] **RECON-JS-01**: JavaScript source map extraction and scanning +- [x] **RECON-JS-02**: Webpack/Vite bundle scanning for inlined env vars +- [x] **RECON-JS-03**: Exposed .env file scanning on web servers +- [x] **RECON-JS-04**: Exposed Swagger/OpenAPI documentation scanning +- [x] **RECON-JS-05**: Vercel/Netlify deploy preview JS bundle scanning ### OSINT/Recon — Log Aggregators diff --git a/.planning/STATE.md b/.planning/STATE.md index 3545a01..7e1d095 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -3,14 +3,14 @@ gsd_state_version: 1.0 milestone: v1.0 milestone_name: milestone status: executing -stopped_at: Completed 13-04-PLAN.md -last_updated: "2026-04-06T10:06:43.774Z" +stopped_at: Completed 14-03-PLAN.md +last_updated: "2026-04-06T10:20:45.465Z" last_activity: 2026-04-06 progress: total_phases: 18 completed_phases: 13 total_plans: 73 - completed_plans: 74 + completed_plans: 75 percent: 20 --- @@ -96,6 +96,7 @@ Progress: [██░░░░░░░░] 20% | Phase 13 P02 | 3min | 2 tasks | 8 files | | Phase 13 P03 | 5min | 2 tasks | 11 files | | Phase 13 P04 | 5min | 2 tasks | 3 files | +| Phase 14 P03 | 5min | 2 tasks | 13 files | ## Accumulated Context @@ -142,6 +143,7 @@ Recent decisions affecting current work: - [Phase 13]: KubernetesSource uses Artifact Hub rather than Censys/Shodan dorking to avoid duplicating Phase 12 sources - [Phase 13]: RegisterAll extended to 32 sources (28 Phase 10-12 + 4 Phase 13 container/IaC) - [Phase 13]: RegisterAll extended to 40 sources (28 Phase 10-12 + 12 Phase 13); package registry sources credentialless, no new SourcesConfig fields +- [Phase 14]: RegisterAll extended to 45 sources (40 Phase 10-13 + 5 Phase 14 frontend leak sources); credentialless multi-path probing pattern ### Pending Todos @@ -156,6 +158,6 @@ None yet. ## Session Continuity -Last session: 2026-04-06T10:04:38.660Z -Stopped at: Completed 13-04-PLAN.md +Last session: 2026-04-06T10:20:45.460Z +Stopped at: Completed 14-03-PLAN.md Resume file: None diff --git a/.planning/phases/14-osint_ci_cd_logs_web_archives_frontend_leaks/14-03-SUMMARY.md b/.planning/phases/14-osint_ci_cd_logs_web_archives_frontend_leaks/14-03-SUMMARY.md new file mode 100644 index 0000000..8805e75 --- /dev/null +++ b/.planning/phases/14-osint_ci_cd_logs_web_archives_frontend_leaks/14-03-SUMMARY.md @@ -0,0 +1,152 @@ +--- +phase: 14-osint_ci_cd_logs_web_archives_frontend_leaks +plan: 03 +subsystem: recon +tags: [sourcemaps, webpack, dotenv, swagger, openapi, vercel, netlify, frontend-leaks] + +requires: + - phase: 10-osint-code-hosting + provides: "ReconSource interface, Client, BuildQueries, LimiterRegistry patterns" + - phase: 13-osint-package-registries + provides: "RegisterAll with 40 sources baseline" +provides: + - "SourceMapSource for probing .map files for original source with API keys" + - "WebpackSource for scanning JS bundles for inlined env vars" + - "EnvLeakSource for detecting exposed .env files on web servers" + - "SwaggerSource for finding API keys in OpenAPI example/default fields" + - "DeployPreviewSource for scanning Vercel/Netlify previews for leaked env vars" + - "RegisterAll extended to 45 sources" +affects: [14-04, 14-05, 15, 16] + +tech-stack: + added: [] + patterns: ["Multi-path probing pattern for credentialless web asset scanning"] + +key-files: + created: + - pkg/recon/sources/sourcemap.go + - pkg/recon/sources/sourcemap_test.go + - pkg/recon/sources/webpack.go + - pkg/recon/sources/webpack_test.go + - pkg/recon/sources/envleak.go + - pkg/recon/sources/envleak_test.go + - pkg/recon/sources/swagger.go + - pkg/recon/sources/swagger_test.go + - pkg/recon/sources/deploypreview.go + - pkg/recon/sources/deploypreview_test.go + modified: + - pkg/recon/sources/register.go + - pkg/recon/sources/register_test.go + - pkg/recon/sources/integration_test.go + +key-decisions: + - "Multi-path probing: each source probes multiple common paths per query rather than single endpoint" + - "Nil Limiters in tests: skip rate limiting in httptest to keep tests fast (<1s)" + - "RegisterAll extended to 45 sources (40 Phase 10-13 + 5 Phase 14 frontend leak sources)" + +patterns-established: + - "Multi-path probing pattern: sources that probe multiple common URL paths per domain/query hint" + - "Regex-based content scanning: compile-time regex patterns for detecting secrets in response bodies" + +requirements-completed: [RECON-JS-01, RECON-JS-02, RECON-JS-03, RECON-JS-04, RECON-JS-05] + +duration: 5min +completed: 2026-04-06 +--- + +# Phase 14 Plan 03: Frontend Leak Sources Summary + +**Five credentialless frontend leak scanners: source maps, webpack bundles, exposed .env files, Swagger docs, and deploy preview environments** + +## Performance + +- **Duration:** 5 min +- **Started:** 2026-04-06T10:13:15Z +- **Completed:** 2026-04-06T10:18:15Z +- **Tasks:** 2 +- **Files modified:** 13 + +## Accomplishments +- SourceMapSource probes 7 common .map paths, parses JSON sourcesContent for API key patterns +- WebpackSource scans JS bundles for NEXT_PUBLIC_/REACT_APP_/VITE_ prefixed env var leaks +- EnvLeakSource probes 8 common .env paths with multiline regex matching for secret key=value lines +- SwaggerSource parses OpenAPI JSON docs for API keys in example/default fields +- DeployPreviewSource scans Vercel/Netlify preview URLs for __NEXT_DATA__ and env var patterns +- RegisterAll extended from 40 to 45 sources + +## Task Commits + +Each task was committed atomically: + +1. **Task 1: SourceMapSource, WebpackSource, EnvLeakSource + tests** - `b57bd5e` (feat) +2. **Task 2: SwaggerSource, DeployPreviewSource + tests** - `7d8a418` (feat) +3. **RegisterAll wiring** - `0a8be81` (feat) + +## Files Created/Modified +- `pkg/recon/sources/sourcemap.go` - Source map file probing and content scanning +- `pkg/recon/sources/sourcemap_test.go` - httptest-based tests for source map scanning +- `pkg/recon/sources/webpack.go` - Webpack/Vite bundle env var detection +- `pkg/recon/sources/webpack_test.go` - httptest-based tests for webpack scanning +- `pkg/recon/sources/envleak.go` - Exposed .env file detection +- `pkg/recon/sources/envleak_test.go` - httptest-based tests for .env scanning +- `pkg/recon/sources/swagger.go` - Swagger/OpenAPI doc API key extraction +- `pkg/recon/sources/swagger_test.go` - httptest-based tests for Swagger scanning +- `pkg/recon/sources/deploypreview.go` - Vercel/Netlify deploy preview scanning +- `pkg/recon/sources/deploypreview_test.go` - httptest-based tests for deploy preview scanning +- `pkg/recon/sources/register.go` - Extended RegisterAll to 45 sources +- `pkg/recon/sources/register_test.go` - Updated test expectations to 45 +- `pkg/recon/sources/integration_test.go` - Updated integration test count to 45 + +## Decisions Made +- Multi-path probing: each source probes multiple common URL paths per query rather than constructing real domain URLs (sources are lead generators) +- Nil Limiters in sweep tests: rate limiter adds 3s per path probe making tests take 20+ seconds; skip in unit tests, test rate limiting separately +- envKeyValuePattern uses (?im) multiline flag for proper line-anchored matching in .env file content + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Bug] Fixed multiline regex in EnvLeakSource** +- **Found during:** Task 1 (EnvLeakSource tests) +- **Issue:** envKeyValuePattern used ^ anchor without (?m) multiline flag, failing to match lines in multi-line .env content +- **Fix:** Added (?m) flag to regex: `(?im)^[A-Z_]*(API[_]?KEY|SECRET|...)` +- **Files modified:** pkg/recon/sources/envleak.go +- **Verification:** TestEnvLeak_Sweep_ExtractsFindings passes +- **Committed in:** b57bd5e (Task 1 commit) + +**2. [Rule 1 - Bug] Removed unused imports in sourcemap.go** +- **Found during:** Task 1 (compilation) +- **Issue:** "fmt" and "strings" imported but unused +- **Fix:** Removed unused imports +- **Files modified:** pkg/recon/sources/sourcemap.go +- **Committed in:** b57bd5e (Task 1 commit) + +**3. [Rule 2 - Missing Critical] Extended RegisterAll and updated integration tests** +- **Found during:** After Task 2 (wiring sources) +- **Issue:** New sources needed registration in RegisterAll; existing tests hardcoded 40 source count +- **Fix:** Added 5 sources to RegisterAll, updated register_test.go and integration_test.go +- **Files modified:** pkg/recon/sources/register.go, register_test.go, integration_test.go +- **Committed in:** 0a8be81 + +--- + +**Total deviations:** 3 auto-fixed (2 bugs, 1 missing critical) +**Impact on plan:** All fixes necessary for correctness. No scope creep. + +## Issues Encountered +None beyond the auto-fixed deviations above. + +## User Setup Required +None - all five sources are credentialless. + +## Known Stubs +None - all sources are fully implemented with real scanning logic. + +## Next Phase Readiness +- 45 sources now registered in RegisterAll +- Frontend leak scanning vectors covered: source maps, webpack bundles, .env files, Swagger docs, deploy previews +- Ready for remaining Phase 14 plans (CI/CD log sources, web archive sources) + +--- +*Phase: 14-osint_ci_cd_logs_web_archives_frontend_leaks* +*Completed: 2026-04-06*