salvacybersec/keyhunter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
289 Commits 1 Branch 0 Tags
b57bd5e7d9fc4e229f222eb3f0f2b5d3a8af7c88
Commit Graph

144 Commits

Author SHA1 Message Date
salvacybersec
b57bd5e7d9 feat(14-03): implement SourceMapSource, WebpackSource, EnvLeakSource with tests 
- SourceMapSource probes .map files for original source containing API keys
- WebpackSource scans JS bundles for inlined NEXT_PUBLIC_/REACT_APP_/VITE_ env vars
- EnvLeakSource probes common .env paths for exposed environment files
- All three implement ReconSource, credentialless, with httptest-based tests
 2026-04-06 13:17:07 +03:00
salvacybersec
9b005e78bb test(13-04): add integration test handlers for all 12 Phase 13 sources (40 total) 
- Add httptest mux handlers for npm, pypi, crates, rubygems, maven, nuget, goproxy, packagist, dockerhub, k8s, terraform, helm
- Register all 12 Phase 13 sources with BaseURL prefix routing
- Update expected source types and count assertions from 28 to 40
 2026-04-06 13:03:27 +03:00
salvacybersec
c16f5feaee feat(13-04): wire all 12 Phase 13 sources into RegisterAll (40 total) 
- Add 8 package registry sources (npm, pypi, crates, rubygems, maven, nuget, goproxy, packagist)
- Update register_test to assert 40 sources in sorted list
- Update Phase 12 integration test count from 32 to 40
 2026-04-06 12:59:11 +03:00
salvacybersec
a607082131 merge: phase 13 resolve conflicts 2026-04-06 12:57:29 +03:00
salvacybersec
7e0e401266 feat(13-03): wire 4 Phase 13 sources into RegisterAll (32 total) 
- Register DockerHub, Kubernetes, Terraform, Helm as credentialless sources
- Update RegisterAll tests and integration test to expect 32 sources
 2026-04-06 12:55:52 +03:00
salvacybersec
0727b51d79 feat(13-03): implement TerraformSource and HelmSource 
- Terraform searches registry.terraform.io v1 modules API with namespace/name/provider URLs
- Helm searches artifacthub.io for charts (kind=0) with repo/chart URL construction
- Both sources: context cancellation, nil registry, httptest-based tests
 2026-04-06 12:53:58 +03:00
salvacybersec
9907e2497a feat(13-01): implement CratesIOSource and RubyGemsSource with httptest tests 
- CratesIOSource searches crates.io JSON API with custom User-Agent header
- RubyGemsSource searches rubygems.org search.json API for gem matches
- Both credentialless; CratesIO 1 req/s burst 1, RubyGems 1 req/2s burst 2
- Tests verify User-Agent header, Sweep findings, ctx cancellation, metadata
 2026-04-06 12:53:41 +03:00
salvacybersec
018bb165fe feat(13-02): implement GoProxySource and PackagistSource with tests 
- GoProxySource parses pkg.go.dev HTML search results for module paths
- PackagistSource queries Packagist JSON search API for PHP packages
- GoProxy regex requires domain dot to filter non-module paths
 2026-04-06 12:53:37 +03:00
salvacybersec
3a8123edc6 feat(13-03): implement DockerHubSource and KubernetesSource 
- DockerHub searches hub.docker.com v2 search API for repos matching provider keywords
- Kubernetes searches Artifact Hub for operators/manifests with kind-aware URL paths
- Both sources: context cancellation, nil registry, httptest-based tests
 2026-04-06 12:52:45 +03:00
salvacybersec
4b268d109f feat(13-01): implement NpmSource and PyPISource with httptest tests 
- NpmSource searches npm registry JSON API for provider keywords
- PyPISource scrapes pypi.org search HTML for project links
- Both credentialless, rate-limited at 1 req/2s, burst 2
- httptest-based tests verify Sweep, ctx cancellation, Name/Rate/Burst
 2026-04-06 12:52:31 +03:00
salvacybersec
23613150f6 feat(13-02): implement MavenSource and NuGetSource with tests 
- MavenSource queries Maven Central Solr API for provider keyword matches
- NuGetSource queries NuGet gallery search API with projectUrl fallback
- Both sources: httptest fixtures, ctx cancellation, metadata tests
 2026-04-06 12:52:27 +03:00
salvacybersec
f0f22191ef test(12-04): add end-to-end SweepAll integration test across all 28 sources 
- Extend integration test with Phase 12 IoT scanner fixtures (shodan, censys, zoomeye, fofa, netlas, binaryedge)
- Add cloud storage fixtures with correct formats (S3 XML, GCS JSON, Azure EnumerationResults XML)
- Add TestRegisterAll_Phase12 verifying 28 sources, enabled/disabled states
- Add TestRegisterAll_Phase12_SweepAllNoPanic for timeout resilience
- Update existing register_test.go from 18 to 28 expected sources
 2026-04-06 12:41:59 +03:00
salvacybersec
870431658d feat(12-04): wire all 10 Phase 12 sources into RegisterAll + cmd/recon.go credentials 
- Add 8 Phase 12 credential fields to SourcesConfig (Shodan, Censys, ZoomEye, FOFA, Netlas, BinaryEdge)
- Register all 10 Phase 12 sources in RegisterAll (6 IoT + 4 cloud storage)
- Wire env/viper credential lookup in cmd/recon.go buildReconEngine
- Update reconCmd Long description to mention Phase 12 sources
 2026-04-06 12:31:57 +03:00
salvacybersec
ade609d562 merge: phase 12 resolve conflicts 2026-04-06 12:27:23 +03:00
salvacybersec
c54e9c73ca merge: phase 12 resolve conflicts 2026-04-06 12:27:23 +03:00
salvacybersec
13905eb5ee feat(12-03): implement AzureBlobScanner, DOSpacesScanner, and all cloud scanner tests 
- AzureBlobScanner enumerates public Azure Blob containers with XML listing
- DOSpacesScanner enumerates public DO Spaces across 5 regions (S3-compatible XML)
- httptest-based tests for all four scanners: sweep, empty registry, ctx cancel, metadata
- All sources credentialless, compile-time interface assertions
 2026-04-06 12:26:01 +03:00
salvacybersec
47d542b9de feat(12-03): implement S3Scanner and GCSScanner cloud storage recon sources 
- S3Scanner enumerates public AWS S3 buckets by provider keyword + suffix pattern
- GCSScanner enumerates public GCS buckets with JSON listing format
- Shared bucketNames() helper and isConfigFile() filter for config-pattern files
- Both credentialless (anonymous HTTP), always Enabled, BaseURL override for tests
 2026-04-06 12:25:55 +03:00
salvacybersec
6443e63b9a test(12-01): add httptest tests for Shodan, Censys, ZoomEye sources 
- Each source tested with mock API server returning 2 results
- Verifies Enabled() disabled when credentials empty
- Verifies Sweep emits correct findings with proper SourceType
- Verifies context cancellation propagation
 2026-04-06 12:24:18 +03:00
salvacybersec
d6c35f4f14 test(12-02): add httptest tests for FOFA, Netlas, BinaryEdge sources 
- FOFA: mock JSON with 2 results, credential validation, context cancellation
- Netlas: mock JSON with 2 items, X-API-Key header check, context cancellation
- BinaryEdge: mock JSON with 2 events, X-Key header check, context cancellation
- All verify correct finding count, source type, and disabled state

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-06 12:24:11 +03:00
salvacybersec
270bbbfb49 feat(12-02): implement FOFA, Netlas, BinaryEdge recon sources 
- FOFASource searches FOFA API with base64-encoded queries (email+key auth)
- NetlasSource searches Netlas API with X-API-Key header auth
- BinaryEdgeSource searches BinaryEdge API with X-Key header auth
- All three implement recon.ReconSource with shared Client retry/backoff

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-06 12:24:04 +03:00
salvacybersec
f5d8470aab feat(12-01): implement Shodan, Censys, ZoomEye recon sources 
- ShodanSource searches /shodan/host/search with API key auth
- CensysSource POSTs to /v2/hosts/search with Basic Auth
- ZoomEyeSource searches /host/search with API-KEY header
- All use shared Client for retry/backoff, LimiterRegistry for rate limiting
 2026-04-06 12:23:06 +03:00
salvacybersec
bebc3e7a0b test(11-03): add end-to-end SweepAll integration test across all 18 sources 
- Extend httptest mux with fixtures for Google, Bing, DuckDuckGo, Yandex, Brave
- Add Pastebin (routed /pb/), GistPaste (/gp/), PasteSites (injected platform)
- Assert all 18 SourceTypes emit at least one finding via SweepAll
 2026-04-06 12:06:27 +03:00
salvacybersec
3250408f23 feat(11-03): wire 18 sources into RegisterAll + credential wiring in cmd/recon.go 
- Extend SourcesConfig with GoogleAPIKey, GoogleCX, BingAPIKey, YandexUser, YandexAPIKey, BraveAPIKey
- RegisterAll registers 8 Phase 11 sources alongside 10 Phase 10 sources (18 total)
- cmd/recon.go reads search engine API keys from env vars and viper config
- Guardrail tests updated to assert 18 sources
 2026-04-06 12:02:11 +03:00
salvacybersec
a53d952518 Merge branch 'worktree-agent-a27c3406' 2026-04-06 11:58:19 +03:00
salvacybersec
ed148d47e1 feat(11-02): add PasteSitesSource multi-paste aggregator 
- Aggregates dpaste, paste.ee, rentry, hastebin into single source
- Follows SandboxesSource multi-platform pattern with per-platform error isolation
- Two-phase search+raw-fetch with keyword matching against provider registry
 2026-04-06 11:55:44 +03:00
salvacybersec
770705302c feat(11-01): add DuckDuckGoSource, YandexSource, and BraveSource 
- DuckDuckGoSource scrapes HTML search (no API key, always enabled, RespectsRobots=true)
- YandexSource uses Yandex XML Search API (user+key required, XML response parsing)
- BraveSource uses Brave Search API (X-Subscription-Token header, JSON response)
- All three follow established error handling: 401 aborts, transient continues, ctx cancellation returns
 2026-04-06 11:54:42 +03:00
salvacybersec
7272e65207 feat(11-01): add GoogleDorkSource and BingDorkSource with formatQuery updates 
- GoogleDorkSource uses Google Custom Search JSON API (APIKey+CX required)
- BingDorkSource uses Bing Web Search API v7 (Ocp-Apim-Subscription-Key header)
- formatQuery now handles google/bing/duckduckgo/yandex/brave dork syntax
- Both sources follow established pattern: retry via Client, rate limit via LimiterRegistry
 2026-04-06 11:54:36 +03:00
salvacybersec
3c500b5473 feat(11-02): add PastebinSource and GistPasteSource for paste site scanning 
- PastebinSource: two-phase search+raw-fetch with keyword matching
- GistPasteSource: scrapes gist.github.com public search (no auth)
- Both implement recon.ReconSource with httptest-based tests
 2026-04-06 11:53:00 +03:00
salvacybersec
118decbb3e fix(phase-10): add --sources filter flag and DB persistence to recon full 
Closes 2 verification gaps:
1. --sources=github,gitlab flag filters registered sources before sweep
2. Findings persisted to SQLite via storage.SaveFinding after dedup

Also adds Engine.Get() method for source lookup by name.
 2026-04-06 11:36:19 +03:00
salvacybersec
8528108613 test(10-09): add end-to-end SweepAll integration test across all ten sources 2026-04-06 01:26:13 +03:00
salvacybersec
fb3e57382e feat(10-09): wire all ten Phase 10 sources in RegisterAll 2026-04-06 01:24:22 +03:00
salvacybersec
4628ccfe90 test(10-09): add failing RegisterAll wiring tests 2026-04-06 01:23:26 +03:00
salvacybersec
a034eeb14c Merge branch 'worktree-agent-ad7ef8d3' 2026-04-06 01:20:33 +03:00
salvacybersec
a0b8f99a7f Merge branch 'worktree-agent-ac81d6ab' 2026-04-06 01:20:25 +03:00
salvacybersec
430ace9a9a Merge branch 'worktree-agent-a2637f83' 2026-04-06 01:20:25 +03:00
salvacybersec
91becd961f Merge branch 'worktree-agent-a7f84823' 2026-04-06 01:20:25 +03:00
salvacybersec
6928ca4e70 Merge branch 'worktree-agent-a2fe7ff3' 2026-04-06 01:20:25 +03:00
salvacybersec
ecebffd27d feat(10-07): add SandboxesSource aggregator (codepen/jsfiddle/stackblitz/glitch/observable) 
- Single ReconSource umbrella iterating per-platform HTML or JSON search endpoints
- Per-platform failures logged and skipped (log-and-continue); ctx cancel aborts fast
- Sub-platform identifier encoded in Finding.KeyMasked as 'platform=<name>' (pragmatic slot)
- Gitpod intentionally omitted (no public search)
- 5 httptest-backed tests covering HTML+JSON extraction, platform-failure tolerance, ctx cancel
 2026-04-06 01:18:15 +03:00
salvacybersec
4fafc01052 feat(10-05): implement CodebergSource for Gitea REST API 
- Add CodebergSource targeting /api/v1/repos/search (Codeberg + any Gitea)
- Public API by default; Authorization: token <t> when Token set
- Unauth rate limit 60/hour, authenticated ~1000/hour
- Emit Findings keyed to repo html_url with SourceType=recon:codeberg
- Keyword index maps BuildQueries output back to ProviderName
- httptest coverage: name/interface, rate limits (both modes),
  sweep decoding, header presence/absence, ctx cancellation
 2026-04-06 01:17:25 +03:00
salvacybersec
0e16e8ea4c feat(10-04): add GistSource for public gist keyword recon 
- GistSource implements recon.ReconSource (RECON-CODE-04)
- Lists /gists/public?per_page=100, fetches each file's raw content,
  scans against provider keyword set, emits one Finding per matching gist
- Disabled when GitHub token empty
- Rate: rate.Every(2s), burst 1 (30 req/min GitHub limit)
- 256KB read cap per file; skips gists without keyword matches
- httptest coverage: enable gating, sweep match, no-match, 401, ctx cancel
 2026-04-06 01:17:07 +03:00
salvacybersec
62a347f476 feat(10-07): add Replit and CodeSandbox scraping sources 
- ReplitSource scrapes /search HTML extracting /@user/repl anchors
- CodeSandboxSource scrapes /search HTML extracting /s/slug anchors
- Both use golang.org/x/net/html parser, 10 req/min rate, RespectsRobots=true
- 10 httptest-backed tests covering extraction, ctx cancel, rate/name assertions
 2026-04-06 01:16:39 +03:00
salvacybersec
ab636dc5e1 fix(10-02): stabilize GitHubSource provider-name test 2026-04-06 01:15:51 +03:00
salvacybersec
0137dc57b1 feat(10-03): add GitLabSource for /api/v4/search blobs 
- Implements recon.ReconSource against GitLab Search API
- PRIVATE-TOKEN header auth; rate.Every(30ms) burst 5 (~2000/min)
- Disabled when token empty; Sweep returns nil without calls
- Emits Finding per blob with Source=/projects/<id>/-/blob/<ref>/<path>
- 401 wrapped as ErrUnauthorized; ctx cancellation honored
- httptest coverage: enabled gating, happy path, 401, ctx cancel, iface assert
 2026-04-06 01:15:49 +03:00
salvacybersec
39001f208c feat(10-06): implement HuggingFaceSource scanning Spaces and Models 
- queries /api/spaces and /api/models via Hub API
- token optional: slower rate when absent (10s vs 3.6s)
- emits Findings with SourceType=recon:huggingface and prefixed Source URLs
- compile-time assert implements recon.ReconSource
 2026-04-06 01:15:49 +03:00
salvacybersec
45f8782464 test(10-06): add failing tests for HuggingFaceSource 
- httptest server routes /api/spaces and /api/models
- assertions: enabled, both endpoints hit, URL prefixes, auth header, ctx cancel, rate-limit token mode
 2026-04-06 01:15:43 +03:00
salvacybersec
d279abf449 feat(10-04): add BitbucketSource for code search recon 
- BitbucketSource implements recon.ReconSource (RECON-CODE-03)
- Queries /2.0/workspaces/{ws}/search/code with Bearer auth
- Disabled when token OR workspace empty
- Rate: rate.Every(3.6s), burst 1 (Bitbucket 1000/hr limit)
- httptest coverage: enable gating, sweep, 401, ctx cancel
 2026-04-06 01:15:42 +03:00
salvacybersec
243b7405cd feat(10-08): add KaggleSource with HTTP Basic auth 
- KaggleSource queries /api/v1/kernels/list with SetBasicAuth(user, key)
- Disabled when either KaggleUser or KaggleKey is empty (no HTTP calls)
- Emits Findings tagged recon:kaggle with Source = <web>/code/<ref>
- 60/min rate limit via rate.Every(1s), burst 1
- httptest-driven tests cover enabled, auth header, missing creds,
  401 unauthorized, and ctx cancellation
- RECON-CODE-09
 2026-04-06 01:15:23 +03:00
salvacybersec
fb6cb53975 feat(10-02): implement GitHubSource recon.ReconSource 2026-04-06 01:14:52 +03:00
salvacybersec
03deb603b3 test(10-02): add failing tests for GitHubSource 2026-04-06 01:12:56 +03:00
salvacybersec
9273f356e6 feat(10-01): add provider-driven query generator and RegisterAll skeleton 
- BuildQueries(reg, source) dedups keywords and formats per-source syntax
- github/gist use 'keyword' in:file; others use bare keyword
- SourcesConfig placeholder struct for Wave 2 plans to depend on
- RegisterAll no-op stub (Plan 10-09 will fill)
 2026-04-06 01:09:57 +03:00