docs(13-02): complete Maven/NuGet/GoProxy/Packagist plan
This commit is contained in:
salvacybersec
@@ -0,0 +1,121 @@
|
||||
---
|
||||
phase: 13-osint_package_registries_container_iac
|
||||
plan: 02
|
||||
subsystem: recon
|
||||
tags: [maven, nuget, goproxy, packagist, osint, package-registry]
|
||||
|
||||
# Dependency graph
|
||||
requires:
|
||||
- phase: 09-osint-infrastructure
|
||||
provides: ReconSource interface, LimiterRegistry, shared Client
|
||||
- phase: 10-osint-code-hosting
|
||||
provides: BuildQueries, extractAnchorHrefs HTML parsing helper
|
||||
provides:
|
||||
- MavenSource searching Maven Central Solr API
|
||||
- NuGetSource searching NuGet gallery JSON API
|
||||
- GoProxySource parsing pkg.go.dev HTML search results
|
||||
- PackagistSource searching Packagist JSON API
|
||||
affects: [13-04, register-all-wiring]
|
||||
|
||||
# Tech tracking
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns: [JSON API source pattern for Maven/NuGet/Packagist, HTML scraping reuse for GoProxy via extractAnchorHrefs]
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- pkg/recon/sources/maven.go
|
||||
- pkg/recon/sources/maven_test.go
|
||||
- pkg/recon/sources/nuget.go
|
||||
- pkg/recon/sources/nuget_test.go
|
||||
- pkg/recon/sources/goproxy.go
|
||||
- pkg/recon/sources/goproxy_test.go
|
||||
- pkg/recon/sources/packagist.go
|
||||
- pkg/recon/sources/packagist_test.go
|
||||
modified: []
|
||||
|
||||
key-decisions:
|
||||
- "GoProxy regex requires domain dot to filter non-module paths like /about"
|
||||
- "NuGet uses projectUrl with fallback to nuget.org/packages/{id} when empty"
|
||||
|
||||
patterns-established:
|
||||
- "JSON registry source: parse response, emit Finding per result, continue on HTTP errors"
|
||||
- "HTML registry source: reuse extractAnchorHrefs with domain-aware regex"
|
||||
|
||||
requirements-completed: [RECON-PKG-02, RECON-PKG-03]
|
||||
|
||||
# Metrics
|
||||
duration: 3min
|
||||
completed: 2026-04-06
|
||||
---
|
||||
|
||||
# Phase 13 Plan 02: Maven, NuGet, GoProxy, Packagist Sources Summary
|
||||
|
||||
**Four package registry ReconSources covering Java/JVM (Maven Central), .NET (NuGet), Go (pkg.go.dev), and PHP (Packagist) ecosystems**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** 3 min
|
||||
- **Started:** 2026-04-06T09:51:21Z
|
||||
- **Completed:** 2026-04-06T09:54:16Z
|
||||
- **Tasks:** 2
|
||||
- **Files modified:** 8
|
||||
|
||||
## Accomplishments
|
||||
- MavenSource queries Maven Central's Solr search API, parsing grouped artifact results
|
||||
- NuGetSource queries NuGet gallery with projectUrl fallback to nuget.org canonical URL
|
||||
- GoProxySource parses pkg.go.dev HTML search results reusing extractAnchorHrefs with domain-aware regex
|
||||
- PackagistSource queries Packagist JSON search API for PHP packages
|
||||
- All four sources: httptest fixtures, context cancellation, metadata method tests (16 tests total)
|
||||
|
||||
## Task Commits
|
||||
|
||||
Each task was committed atomically:
|
||||
|
||||
1. **Task 1: Implement MavenSource and NuGetSource** - `2361315` (feat)
|
||||
2. **Task 2: Implement GoProxySource and PackagistSource** - `018bb16` (feat)
|
||||
|
||||
## Files Created/Modified
|
||||
- `pkg/recon/sources/maven.go` - MavenSource querying Maven Central Solr API
|
||||
- `pkg/recon/sources/maven_test.go` - httptest with canned Solr JSON fixture
|
||||
- `pkg/recon/sources/nuget.go` - NuGetSource querying NuGet gallery search API
|
||||
- `pkg/recon/sources/nuget_test.go` - httptest with canned NuGet JSON, projectUrl fallback test
|
||||
- `pkg/recon/sources/goproxy.go` - GoProxySource parsing pkg.go.dev HTML search
|
||||
- `pkg/recon/sources/goproxy_test.go` - httptest with canned HTML, module path extraction test
|
||||
- `pkg/recon/sources/packagist.go` - PackagistSource querying Packagist JSON API
|
||||
- `pkg/recon/sources/packagist_test.go` - httptest with canned Packagist JSON fixture
|
||||
|
||||
## Decisions Made
|
||||
- GoProxy regex tightened to require a dot in the path (`^/[a-z][a-z0-9_-]*\.[a-z0-9./_-]+$`) to distinguish Go module paths from site navigation links like /about
|
||||
- NuGet uses projectUrl when available, falls back to canonical nuget.org URL when empty
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
### Auto-fixed Issues
|
||||
|
||||
**1. [Rule 1 - Bug] GoProxy regex too permissive**
|
||||
- **Found during:** Task 2 (GoProxySource implementation)
|
||||
- **Issue:** Original regex `^/[a-z][a-z0-9./_-]*$` matched non-module paths like /about
|
||||
- **Fix:** Tightened to require a dot character (domain separator) in the path
|
||||
- **Files modified:** pkg/recon/sources/goproxy.go
|
||||
- **Verification:** Test now correctly extracts only 2 module paths from fixture HTML
|
||||
- **Committed in:** 018bb16
|
||||
|
||||
---
|
||||
|
||||
**Total deviations:** 1 auto-fixed (1 bug)
|
||||
**Impact on plan:** Minor regex fix for correctness. No scope creep.
|
||||
|
||||
## Issues Encountered
|
||||
None
|
||||
|
||||
## User Setup Required
|
||||
None - no external service configuration required.
|
||||
|
||||
## Next Phase Readiness
|
||||
- All four package registry sources ready for RegisterAll wiring in plan 13-04
|
||||
- Sources follow established pattern: BaseURL override for tests, BuildQueries for keyword generation, LimiterRegistry for rate coordination
|
||||
|
||||
---
|
||||
*Phase: 13-osint_package_registries_container_iac*
|
||||
*Completed: 2026-04-06*
|
||||
Reference in New Issue
Block a user