salvacybersec/keyhunter

Fork 0

Files

salvacybersec 12c402ab67 docs(10-07): complete sandbox/IDE scraping sources plan

2026-04-06 01:19:57 +03:00

7.5 KiB

Raw Blame History

gsd_state_version, milestone, milestone_name, status, stopped_at, last_updated, last_activity, progress

gsd_state_version

milestone

milestone_name

status

stopped_at

last_updated

last_activity

progress

1.0

v1.0

milestone

executing

Completed 10-07-PLAN.md

2026-04-05T22:19:41.729Z

2026-04-05

total_phases	completed_phases	total_plans	completed_plans	percent
18	9	62	57	20

Project State

Project Reference

See: .planning/PROJECT.md (updated 2026-04-04)

Core value: Detect leaked LLM API keys across more providers and more internet sources than any other tool, with active verification to confirm keys are real and alive. Current focus: Phase 10 — osint-code-hosting

Current Position

Phase: 10 (osint-code-hosting) — EXECUTING Plan: 3 of 9 Status: Ready to execute Last activity: 2026-04-05

Progress: [██░░░░░░░░] 20%

Performance Metrics

Velocity:

Total plans completed: 0
Average duration: —
Total execution time: 0 hours

By Phase:

Phase	Plans	Total	Avg/Plan
-	-	-	-

Recent Trend:

Last 5 plans: —
Trend: —

Accumulated Context

Decisions

Decisions are logged in PROJECT.md Key Decisions table. Recent decisions affecting current work:

Roadmap: CGO_ENABLED=0 throughout — modernc.org/sqlite over mattn/go-sqlite3 (see PROJECT.md)
Roadmap: Per-source rate limiter architecture (Phase 9) must precede all OSINT source modules (Phases 10-16)
Roadmap: AES-256 encryption added in Phase 1, not post-hoc — avoids migration complexity
Roadmap: Verification (Phase 5) requires consent prompt + LEGAL.md — not optional polish
[Phase 01-foundation]: Provider YAML in dual locations: providers/ (user-visible) and pkg/providers/definitions/ (embed) — Go embed cannot use '..' paths
[Phase 01-foundation]: Aho-Corasick built with DFA=true at NewRegistry() for O(n) keyword pre-filtering across all providers
[Phase 01-foundation]: pkg/types/chunk.go breaks engine<->sources circular import; ants pool with WaitGroup+Mutex for detector coordination
[Phase 01-foundation]: Per-installation salt via settings table -- no hardcoded salt in production code
[Phase 01-foundation]: Exit code semantics: 0=clean, 1=keys-found, 2=error for CI/CD integration
[Phase 02-tier-1-2-providers]: AWS Bedrock verify URL left empty — SigV4 signing deferred to Phase 5 verification engine
[Phase 03-tier-3-9-providers]: Keyword-only detection for providers without documented key prefixes (You.com, Unstructured, Runway, Midjourney) to avoid false positives.
[Phase 04]: Use 'go mod download' instead of 'go mod tidy' when bootstrapping dependencies ahead of their consumers
[Phase 04-input-sources]: GitSource walks heads+tags+remotes+stash with per-OID blob dedup
[Phase 04]: Introduced selectSource dispatcher with sourceFlags struct for testable CLI source routing
[Phase 05]: Keep legacy VerifySpec ValidStatus/InvalidStatus alongside canonical SuccessCodes/FailureCodes; Effective*() helpers pick canonical-first with fallback
[Phase 05]: Store Finding.VerifyMetadata as JSON TEXT column; legacy DBs migrated in-place via PRAGMA table_info + conditional ALTER TABLE in storage.Open()
[Phase 05-verification-engine]: LEGAL.md dual-location mirror (root + pkg/legal/) required because go:embed cannot traverse parents — mirrors Phase 1 providers pattern
[Phase 05-verification-engine]: verify.consent setting: granted is sticky across runs; declined is not — users who initially refuse can change mind without manual reset
[Phase 05-verification-engine]: Plan 05-03: HTTPVerifier classifies via YAML VerifySpec only; no per-provider branches. VerifyAll uses ants pool with per-finding Result guarantee.
[Phase 05]: Verification runs in batch mode after scan completes (collect -> verify -> persist) with Result->Finding back-assignment via provider+masked-key tuple
[Phase 06]: Registry pattern for output formatters; TableFormatter strips ANSI when writer is not a TTY via zero-value lipgloss.Style
[Phase 06]: SARIF 2.1.0 via hand-rolled structs (no library) per CLAUDE.md
[Phase 06-output-reporting]: keys export rejects SARIF (scan-only); keys show always unmasked; keys verify updates findings inline via db.SQL().Exec
[Phase 08-dork-engine]: pkg/dorks mirrors pkg/providers go:embed pattern; //go:embed definitions/* tolerates empty .gitkeep-only tree
[Phase 08-dork-engine]: Runner + Executor interface separate from Registry so 08-05 GitHub executor registers without touching YAML loader
[Phase 10-osint-code-hosting]: Client handles retry only; rate limiting is caller's responsibility via LimiterRegistry
[Phase 10-osint-code-hosting]: github/gist use 'kw' in:file; all other sources use bare keyword
[Phase 10-osint-code-hosting]: GitHubSource reuses shared sources.Client + LimiterRegistry; builds queries from providers.Registry via BuildQueries; missing token disables (not errors)

Pending Todos

None yet.

Blockers/Concerns

Phase 1: Argon2 vs PBKDF2 for database encryption key derivation — needs decision before Storage Layer implementation
Phase 1: Aho-Corasick library choice (cloudflare/ahocorasick vs bobrik/ahocorasick) — verify which TruffleHog uses
Phase 2+: Provider YAML patterns for 108 providers — lesser-known providers need targeted research (Chinese LLMs, niche APIs)
Phase 11: Google Custom Search API quota (100 queries/day free tier) vs direct scraping ToS trade-off — product decision needed

Session Continuity

Last session: 2026-04-05T22:19:41.725Z Stopped at: Completed 10-07-PLAN.md Resume file: None

7.5 KiB Raw Blame History