merge: phase 14-03 frontend leaks

.planning/REQUIREMENTS.md

@@ -173,11 +173,11 @@ Requirements for initial release. Each maps to roadmap phases.
 
 ### OSINT/Recon — Frontend & JS Leaks
 
-- [ ] **RECON-JS-01**: JavaScript source map extraction and scanning
-- [ ] **RECON-JS-02**: Webpack/Vite bundle scanning for inlined env vars
-- [ ] **RECON-JS-03**: Exposed .env file scanning on web servers
-- [ ] **RECON-JS-04**: Exposed Swagger/OpenAPI documentation scanning
-- [ ] **RECON-JS-05**: Vercel/Netlify deploy preview JS bundle scanning
+- [x] **RECON-JS-01**: JavaScript source map extraction and scanning
+- [x] **RECON-JS-02**: Webpack/Vite bundle scanning for inlined env vars
+- [x] **RECON-JS-03**: Exposed .env file scanning on web servers
+- [x] **RECON-JS-04**: Exposed Swagger/OpenAPI documentation scanning
+- [x] **RECON-JS-05**: Vercel/Netlify deploy preview JS bundle scanning
 
 ### OSINT/Recon — Log Aggregators
 

.planning/phases/14-osint_ci_cd_logs_web_archives_frontend_leaks/14-03-SUMMARY.md

@@ -0,0 +1,152 @@
+---
+phase: 14-osint_ci_cd_logs_web_archives_frontend_leaks
+plan: 03
+subsystem: recon
+tags: [sourcemaps, webpack, dotenv, swagger, openapi, vercel, netlify, frontend-leaks]
+
+requires:
+  - phase: 10-osint-code-hosting
+    provides: "ReconSource interface, Client, BuildQueries, LimiterRegistry patterns"
+  - phase: 13-osint-package-registries
+    provides: "RegisterAll with 40 sources baseline"
+provides:
+  - "SourceMapSource for probing .map files for original source with API keys"
+  - "WebpackSource for scanning JS bundles for inlined env vars"
+  - "EnvLeakSource for detecting exposed .env files on web servers"
+  - "SwaggerSource for finding API keys in OpenAPI example/default fields"
+  - "DeployPreviewSource for scanning Vercel/Netlify previews for leaked env vars"
+  - "RegisterAll extended to 45 sources"
+affects: [14-04, 14-05, 15, 16]
+
+tech-stack:
+  added: []
+  patterns: ["Multi-path probing pattern for credentialless web asset scanning"]
+
+key-files:
+  created:
+    - pkg/recon/sources/sourcemap.go
+    - pkg/recon/sources/sourcemap_test.go
+    - pkg/recon/sources/webpack.go
+    - pkg/recon/sources/webpack_test.go
+    - pkg/recon/sources/envleak.go
+    - pkg/recon/sources/envleak_test.go
+    - pkg/recon/sources/swagger.go
+    - pkg/recon/sources/swagger_test.go
+    - pkg/recon/sources/deploypreview.go
+    - pkg/recon/sources/deploypreview_test.go
+  modified:
+    - pkg/recon/sources/register.go
+    - pkg/recon/sources/register_test.go
+    - pkg/recon/sources/integration_test.go
+
+key-decisions:
+  - "Multi-path probing: each source probes multiple common paths per query rather than single endpoint"
+  - "Nil Limiters in tests: skip rate limiting in httptest to keep tests fast (<1s)"
+  - "RegisterAll extended to 45 sources (40 Phase 10-13 + 5 Phase 14 frontend leak sources)"
+
+patterns-established:
+  - "Multi-path probing pattern: sources that probe multiple common URL paths per domain/query hint"
+  - "Regex-based content scanning: compile-time regex patterns for detecting secrets in response bodies"
+
+requirements-completed: [RECON-JS-01, RECON-JS-02, RECON-JS-03, RECON-JS-04, RECON-JS-05]
+
+duration: 5min
+completed: 2026-04-06
+---
+
+# Phase 14 Plan 03: Frontend Leak Sources Summary
+
+**Five credentialless frontend leak scanners: source maps, webpack bundles, exposed .env files, Swagger docs, and deploy preview environments**
+
+## Performance
+
+- **Duration:** 5 min
+- **Started:** 2026-04-06T10:13:15Z
+- **Completed:** 2026-04-06T10:18:15Z
+- **Tasks:** 2
+- **Files modified:** 13
+
+## Accomplishments
+- SourceMapSource probes 7 common .map paths, parses JSON sourcesContent for API key patterns
+- WebpackSource scans JS bundles for NEXT_PUBLIC_/REACT_APP_/VITE_ prefixed env var leaks
+- EnvLeakSource probes 8 common .env paths with multiline regex matching for secret key=value lines
+- SwaggerSource parses OpenAPI JSON docs for API keys in example/default fields
+- DeployPreviewSource scans Vercel/Netlify preview URLs for __NEXT_DATA__ and env var patterns
+- RegisterAll extended from 40 to 45 sources
+
+## Task Commits
+
+Each task was committed atomically:
+
+1. **Task 1: SourceMapSource, WebpackSource, EnvLeakSource + tests** - `b57bd5e` (feat)
+2. **Task 2: SwaggerSource, DeployPreviewSource + tests** - `7d8a418` (feat)
+3. **RegisterAll wiring** - `0a8be81` (feat)
+
+## Files Created/Modified
+- `pkg/recon/sources/sourcemap.go` - Source map file probing and content scanning
+- `pkg/recon/sources/sourcemap_test.go` - httptest-based tests for source map scanning
+- `pkg/recon/sources/webpack.go` - Webpack/Vite bundle env var detection
+- `pkg/recon/sources/webpack_test.go` - httptest-based tests for webpack scanning
+- `pkg/recon/sources/envleak.go` - Exposed .env file detection
+- `pkg/recon/sources/envleak_test.go` - httptest-based tests for .env scanning
+- `pkg/recon/sources/swagger.go` - Swagger/OpenAPI doc API key extraction
+- `pkg/recon/sources/swagger_test.go` - httptest-based tests for Swagger scanning
+- `pkg/recon/sources/deploypreview.go` - Vercel/Netlify deploy preview scanning
+- `pkg/recon/sources/deploypreview_test.go` - httptest-based tests for deploy preview scanning
+- `pkg/recon/sources/register.go` - Extended RegisterAll to 45 sources
+- `pkg/recon/sources/register_test.go` - Updated test expectations to 45
+- `pkg/recon/sources/integration_test.go` - Updated integration test count to 45
+
+## Decisions Made
+- Multi-path probing: each source probes multiple common URL paths per query rather than constructing real domain URLs (sources are lead generators)
+- Nil Limiters in sweep tests: rate limiter adds 3s per path probe making tests take 20+ seconds; skip in unit tests, test rate limiting separately
+- envKeyValuePattern uses (?im) multiline flag for proper line-anchored matching in .env file content
+
+## Deviations from Plan
+
+### Auto-fixed Issues
+
+**1. [Rule 1 - Bug] Fixed multiline regex in EnvLeakSource**
+- **Found during:** Task 1 (EnvLeakSource tests)
+- **Issue:** envKeyValuePattern used ^ anchor without (?m) multiline flag, failing to match lines in multi-line .env content
+- **Fix:** Added (?m) flag to regex: `(?im)^[A-Z_]*(API[_]?KEY|SECRET|...)`
+- **Files modified:** pkg/recon/sources/envleak.go
+- **Verification:** TestEnvLeak_Sweep_ExtractsFindings passes
+- **Committed in:** b57bd5e (Task 1 commit)
+
+**2. [Rule 1 - Bug] Removed unused imports in sourcemap.go**
+- **Found during:** Task 1 (compilation)
+- **Issue:** "fmt" and "strings" imported but unused
+- **Fix:** Removed unused imports
+- **Files modified:** pkg/recon/sources/sourcemap.go
+- **Committed in:** b57bd5e (Task 1 commit)
+
+**3. [Rule 2 - Missing Critical] Extended RegisterAll and updated integration tests**
+- **Found during:** After Task 2 (wiring sources)
+- **Issue:** New sources needed registration in RegisterAll; existing tests hardcoded 40 source count
+- **Fix:** Added 5 sources to RegisterAll, updated register_test.go and integration_test.go
+- **Files modified:** pkg/recon/sources/register.go, register_test.go, integration_test.go
+- **Committed in:** 0a8be81
+
+---
+
+**Total deviations:** 3 auto-fixed (2 bugs, 1 missing critical)
+**Impact on plan:** All fixes necessary for correctness. No scope creep.
+
+## Issues Encountered
+None beyond the auto-fixed deviations above.
+
+## User Setup Required
+None - all five sources are credentialless.
+
+## Known Stubs
+None - all sources are fully implemented with real scanning logic.
+
+## Next Phase Readiness
+- 45 sources now registered in RegisterAll
+- Frontend leak scanning vectors covered: source maps, webpack bundles, .env files, Swagger docs, deploy previews
+- Ready for remaining Phase 14 plans (CI/CD log sources, web archive sources)
+
+---
+*Phase: 14-osint_ci_cd_logs_web_archives_frontend_leaks*
+*Completed: 2026-04-06*