docs(16-02): complete APKMirror, crt.sh, SecurityTrails plan

- SUMMARY.md with implementation details and verification results
- STATE.md updated with progress and decisions
- REQUIREMENTS.md marks RECON-MOBILE-01, RECON-DNS-01, RECON-DNS-02 complete
This commit is contained in:
salvacybersec
2026-04-06 16:46:56 +03:00
parent a195ef33a0
commit 6064902aa5
3 changed files with 96 additions and 9 deletions

View File

@@ -0,0 +1,85 @@
---
phase: 16-osint-threat-intel-mobile-dns-api-marketplaces
plan: 02
subsystem: recon-sources
tags: [osint, mobile, dns, ct-logs, securitytrails, apkmirror, crtsh]
dependency_graph:
requires: [pkg/recon/sources/httpclient.go, pkg/recon/sources/queries.go, pkg/recon/source.go]
provides: [APKMirrorSource, CrtShSource, SecurityTrailsSource]
affects: [pkg/recon/sources/register.go, cmd/recon.go]
tech_stack:
added: []
patterns: [subdomain-probe-pattern, ct-log-discovery, credential-gated-source]
key_files:
created:
- pkg/recon/sources/apkmirror.go
- pkg/recon/sources/apkmirror_test.go
- pkg/recon/sources/crtsh.go
- pkg/recon/sources/crtsh_test.go
- pkg/recon/sources/securitytrails.go
- pkg/recon/sources/securitytrails_test.go
modified:
- pkg/recon/sources/register.go
- cmd/recon.go
decisions:
- APKMirror is metadata-only scanner (no APK decompilation) since apktool/jadx require local binaries
- CrtSh and SecurityTrails share configProbeEndpoints pattern for subdomain probing
- Probe HTTP client uses 5s timeout without retries (fail fast, separate from API client)
- SecurityTrails gets dedicated SECURITYTRAILS_API_KEY env var
metrics:
duration: 3min
completed: 2026-04-06
tasks_completed: 2
tasks_total: 2
files_created: 6
files_modified: 2
---
# Phase 16 Plan 02: APKMirror, crt.sh, SecurityTrails Sources Summary
Mobile app metadata scanning via APKMirror, CT log subdomain discovery with config endpoint probing via crt.sh, and DNS intelligence subdomain enumeration with endpoint probing via SecurityTrails API.
## Completed Tasks
| Task | Name | Commit | Key Files |
|------|------|--------|-----------|
| 1 | APKMirror and crt.sh sources | 09a8d4c | apkmirror.go, crtsh.go + tests |
| 2 | SecurityTrails source | a195ef3 | securitytrails.go + test, register.go, cmd/recon.go |
## Implementation Details
### APKMirrorSource (credentialless)
- Searches APK release pages for keyword matches using BuildQueries
- Scans HTML response for ciLogKeyPattern matches in descriptions/changelogs
- Rate limited: 1 request per 5 seconds, burst 2. Respects robots.txt.
### CrtShSource (credentialless)
- Queries crt.sh JSON API for certificate transparency log entries matching `%.{domain}`
- Deduplicates subdomains (strips wildcards), limits to 20
- Probes each subdomain's /.env, /api/config, /actuator/env with 5s timeout client
- ProbeBaseURL field enables httptest-based testing
### SecurityTrailsSource (credential-gated)
- Phase 1: Enumerates subdomains via SecurityTrails API with APIKEY header
- Phase 2: Probes same three config endpoints as CrtSh (shared configProbeEndpoints)
- Phase 3: Fetches domain DNS history and checks full JSON for key patterns in TXT records
- Disabled when SECURITYTRAILS_API_KEY is empty
### RegisterAll
- Extended from 67 to 70 sources (added APKMirror, crt.sh, SecurityTrails)
- cmd/recon.go wires SecurityTrailsAPIKey from env/viper
## Deviations from Plan
None -- plan executed exactly as written.
## Known Stubs
None -- all sources fully implemented with real API integration patterns.
## Verification
```
go vet ./pkg/recon/sources/ ./cmd/ -- PASS
go test ./pkg/recon/sources/ -run "TestAPKMirror|TestCrtSh|TestSecurityTrails" -- 14/14 PASS
```