salvacybersec/keyhunter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(14-03): complete frontend leak sources plan

Browse Source 
- SUMMARY.md with 5 sources, 3 commits, 13 files
- STATE.md updated with position and decisions
- Requirements RECON-JS-01 through RECON-JS-05 marked complete
This commit is contained in:
salvacybersec
2026-04-06 13:21:00 +03:00
parent 0a8be81f0c
commit 95ee768266
3 changed files with 164 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
.planning/REQUIREMENTS.md
View File

@@ -173,11 +173,11 @@ Requirements for initial release. Each maps to roadmap phases.
### OSINT/Recon — Frontend & JS Leaks ### OSINT/Recon — Frontend & JS Leaks
- [ ] **RECON-JS-01**: JavaScript source map extraction and scanning - [x] **RECON-JS-01**: JavaScript source map extraction and scanning
- [ ] **RECON-JS-02**: Webpack/Vite bundle scanning for inlined env vars - [x] **RECON-JS-02**: Webpack/Vite bundle scanning for inlined env vars
- [ ] **RECON-JS-03**: Exposed .env file scanning on web servers - [x] **RECON-JS-03**: Exposed .env file scanning on web servers
- [ ] **RECON-JS-04**: Exposed Swagger/OpenAPI documentation scanning - [x] **RECON-JS-04**: Exposed Swagger/OpenAPI documentation scanning
- [ ] **RECON-JS-05**: Vercel/Netlify deploy preview JS bundle scanning - [x] **RECON-JS-05**: Vercel/Netlify deploy preview JS bundle scanning
### OSINT/Recon — Log Aggregators ### OSINT/Recon — Log Aggregators

12
.planning/STATE.md
View File

@@ -3,14 +3,14 @@ gsd_state_version: 1.0
milestone: v1.0 milestone: v1.0
milestone_name: milestone milestone_name: milestone
status: executing status: executing
stopped_at: Completed 13-04-PLAN.md stopped_at: Completed 14-03-PLAN.md
last_updated: "2026-04-06T10:06:43.774Z" last_updated: "2026-04-06T10:20:45.465Z"
last_activity: 2026-04-06 last_activity: 2026-04-06
progress: progress:
total_phases: 18 total_phases: 18
completed_phases: 13 completed_phases: 13
total_plans: 73 total_plans: 73
completed_plans: 74 completed_plans: 75
percent: 20 percent: 20
--- ---
@@ -96,6 +96,7 @@ Progress: [██░░░░░░░░] 20%
| Phase 13 P02 | 3min | 2 tasks | 8 files | | Phase 13 P02 | 3min | 2 tasks | 8 files |
| Phase 13 P03 | 5min | 2 tasks | 11 files | | Phase 13 P03 | 5min | 2 tasks | 11 files |
| Phase 13 P04 | 5min | 2 tasks | 3 files | | Phase 13 P04 | 5min | 2 tasks | 3 files |
| Phase 14 P03 | 5min | 2 tasks | 13 files |
## Accumulated Context ## Accumulated Context
@@ -142,6 +143,7 @@ Recent decisions affecting current work:
- [Phase 13]: KubernetesSource uses Artifact Hub rather than Censys/Shodan dorking to avoid duplicating Phase 12 sources - [Phase 13]: KubernetesSource uses Artifact Hub rather than Censys/Shodan dorking to avoid duplicating Phase 12 sources
- [Phase 13]: RegisterAll extended to 32 sources (28 Phase 10-12 + 4 Phase 13 container/IaC) - [Phase 13]: RegisterAll extended to 32 sources (28 Phase 10-12 + 4 Phase 13 container/IaC)
- [Phase 13]: RegisterAll extended to 40 sources (28 Phase 10-12 + 12 Phase 13); package registry sources credentialless, no new SourcesConfig fields - [Phase 13]: RegisterAll extended to 40 sources (28 Phase 10-12 + 12 Phase 13); package registry sources credentialless, no new SourcesConfig fields
- [Phase 14]: RegisterAll extended to 45 sources (40 Phase 10-13 + 5 Phase 14 frontend leak sources); credentialless multi-path probing pattern
### Pending Todos ### Pending Todos
@@ -156,6 +158,6 @@ None yet.
## Session Continuity ## Session Continuity
Last session: 2026-04-06T10:04:38.660Z Last session: 2026-04-06T10:20:45.460Z
Stopped at: Completed 13-04-PLAN.md Stopped at: Completed 14-03-PLAN.md
Resume file: None Resume file: None

152
.planning/phases/14-osint_ci_cd_logs_web_archives_frontend_leaks/14-03-SUMMARY.md Normal file
View File

@@ -0,0 +1,152 @@
---
phase: 14-osint_ci_cd_logs_web_archives_frontend_leaks
plan: 03
subsystem: recon
tags: [sourcemaps, webpack, dotenv, swagger, openapi, vercel, netlify, frontend-leaks]
requires:
- phase: 10-osint-code-hosting
provides: "ReconSource interface, Client, BuildQueries, LimiterRegistry patterns"
- phase: 13-osint-package-registries
provides: "RegisterAll with 40 sources baseline"
provides:
- "SourceMapSource for probing .map files for original source with API keys"
- "WebpackSource for scanning JS bundles for inlined env vars"
- "EnvLeakSource for detecting exposed .env files on web servers"
- "SwaggerSource for finding API keys in OpenAPI example/default fields"
- "DeployPreviewSource for scanning Vercel/Netlify previews for leaked env vars"
- "RegisterAll extended to 45 sources"
affects: [14-04, 14-05, 15, 16]
tech-stack:
added: []
patterns: ["Multi-path probing pattern for credentialless web asset scanning"]
key-files:
created:
- pkg/recon/sources/sourcemap.go
- pkg/recon/sources/sourcemap_test.go
- pkg/recon/sources/webpack.go
- pkg/recon/sources/webpack_test.go
- pkg/recon/sources/envleak.go
- pkg/recon/sources/envleak_test.go
- pkg/recon/sources/swagger.go
- pkg/recon/sources/swagger_test.go
- pkg/recon/sources/deploypreview.go
- pkg/recon/sources/deploypreview_test.go
modified:
- pkg/recon/sources/register.go
- pkg/recon/sources/register_test.go
- pkg/recon/sources/integration_test.go
key-decisions:
- "Multi-path probing: each source probes multiple common paths per query rather than single endpoint"
- "Nil Limiters in tests: skip rate limiting in httptest to keep tests fast (<1s)"
- "RegisterAll extended to 45 sources (40 Phase 10-13 + 5 Phase 14 frontend leak sources)"
patterns-established:
- "Multi-path probing pattern: sources that probe multiple common URL paths per domain/query hint"
- "Regex-based content scanning: compile-time regex patterns for detecting secrets in response bodies"
requirements-completed: [RECON-JS-01, RECON-JS-02, RECON-JS-03, RECON-JS-04, RECON-JS-05]
duration: 5min
completed: 2026-04-06
---
# Phase 14 Plan 03: Frontend Leak Sources Summary
**Five credentialless frontend leak scanners: source maps, webpack bundles, exposed .env files, Swagger docs, and deploy preview environments**
## Performance
- **Duration:** 5 min
- **Started:** 2026-04-06T10:13:15Z
- **Completed:** 2026-04-06T10:18:15Z
- **Tasks:** 2
- **Files modified:** 13
## Accomplishments
- SourceMapSource probes 7 common .map paths, parses JSON sourcesContent for API key patterns
- WebpackSource scans JS bundles for NEXT_PUBLIC_/REACT_APP_/VITE_ prefixed env var leaks
- EnvLeakSource probes 8 common .env paths with multiline regex matching for secret key=value lines
- SwaggerSource parses OpenAPI JSON docs for API keys in example/default fields
- DeployPreviewSource scans Vercel/Netlify preview URLs for __NEXT_DATA__ and env var patterns
- RegisterAll extended from 40 to 45 sources
## Task Commits
Each task was committed atomically:
1. **Task 1: SourceMapSource, WebpackSource, EnvLeakSource + tests** - `b57bd5e` (feat)
2. **Task 2: SwaggerSource, DeployPreviewSource + tests** - `7d8a418` (feat)
3. **RegisterAll wiring** - `0a8be81` (feat)
## Files Created/Modified
- `pkg/recon/sources/sourcemap.go` - Source map file probing and content scanning
- `pkg/recon/sources/sourcemap_test.go` - httptest-based tests for source map scanning
- `pkg/recon/sources/webpack.go` - Webpack/Vite bundle env var detection
- `pkg/recon/sources/webpack_test.go` - httptest-based tests for webpack scanning
- `pkg/recon/sources/envleak.go` - Exposed .env file detection
- `pkg/recon/sources/envleak_test.go` - httptest-based tests for .env scanning
- `pkg/recon/sources/swagger.go` - Swagger/OpenAPI doc API key extraction
- `pkg/recon/sources/swagger_test.go` - httptest-based tests for Swagger scanning
- `pkg/recon/sources/deploypreview.go` - Vercel/Netlify deploy preview scanning
- `pkg/recon/sources/deploypreview_test.go` - httptest-based tests for deploy preview scanning
- `pkg/recon/sources/register.go` - Extended RegisterAll to 45 sources
- `pkg/recon/sources/register_test.go` - Updated test expectations to 45
- `pkg/recon/sources/integration_test.go` - Updated integration test count to 45
## Decisions Made
- Multi-path probing: each source probes multiple common URL paths per query rather than constructing real domain URLs (sources are lead generators)
- Nil Limiters in sweep tests: rate limiter adds 3s per path probe making tests take 20+ seconds; skip in unit tests, test rate limiting separately
- envKeyValuePattern uses (?im) multiline flag for proper line-anchored matching in .env file content
## Deviations from Plan
### Auto-fixed Issues
**1. [Rule 1 - Bug] Fixed multiline regex in EnvLeakSource**
- **Found during:** Task 1 (EnvLeakSource tests)
- **Issue:** envKeyValuePattern used ^ anchor without (?m) multiline flag, failing to match lines in multi-line .env content
- **Fix:** Added (?m) flag to regex: `(?im)^[A-Z_]*(API[_]?KEY|SECRET|...)`
- **Files modified:** pkg/recon/sources/envleak.go
- **Verification:** TestEnvLeak_Sweep_ExtractsFindings passes
- **Committed in:** b57bd5e (Task 1 commit)
**2. [Rule 1 - Bug] Removed unused imports in sourcemap.go**
- **Found during:** Task 1 (compilation)
- **Issue:** "fmt" and "strings" imported but unused
- **Fix:** Removed unused imports
- **Files modified:** pkg/recon/sources/sourcemap.go
- **Committed in:** b57bd5e (Task 1 commit)
**3. [Rule 2 - Missing Critical] Extended RegisterAll and updated integration tests**
- **Found during:** After Task 2 (wiring sources)
- **Issue:** New sources needed registration in RegisterAll; existing tests hardcoded 40 source count
- **Fix:** Added 5 sources to RegisterAll, updated register_test.go and integration_test.go
- **Files modified:** pkg/recon/sources/register.go, register_test.go, integration_test.go
- **Committed in:** 0a8be81
---
**Total deviations:** 3 auto-fixed (2 bugs, 1 missing critical)
**Impact on plan:** All fixes necessary for correctness. No scope creep.
## Issues Encountered
None beyond the auto-fixed deviations above.
## User Setup Required
None - all five sources are credentialless.
## Known Stubs
None - all sources are fully implemented with real scanning logic.
## Next Phase Readiness
- 45 sources now registered in RegisterAll
- Frontend leak scanning vectors covered: source maps, webpack bundles, .env files, Swagger docs, deploy previews
- Ready for remaining Phase 14 plans (CI/CD log sources, web archive sources)
---
*Phase: 14-osint_ci_cd_logs_web_archives_frontend_leaks*
*Completed: 2026-04-06*