Files

salvacybersec d8a610758b docs(17-02): complete scheduler + storage plan

- Add 17-02-SUMMARY.md with execution results
- Update STATE.md position and metrics
- Mark SCHED-01 complete in REQUIREMENTS.md

2026-04-06 17:28:30 +03:00

9.9 KiB

Raw Blame History

gsd_state_version, milestone, milestone_name, status, stopped_at, last_updated, last_activity, progress

gsd_state_version

milestone

milestone_name

status

stopped_at

last_updated

last_activity

progress

1.0

v1.0

milestone

executing

Completed 17-02-PLAN.md

2026-04-06T14:28:16.926Z

2026-04-06

total_phases	completed_phases	total_plans	completed_plans	percent
18	14	85	84	20

Project State

Project Reference

See: .planning/PROJECT.md (updated 2026-04-04)

Core value: Detect leaked LLM API keys across more providers and more internet sources than any other tool, with active verification to confirm keys are real and alive. Current focus: Phase 13 — osint-package-registries

Current Position

Phase: 17 Plan: 2 of 5 Status: executing Last activity: 2026-04-06

Progress: [██░░░░░░░░] 20%

Performance Metrics

Velocity:

Total plans completed: 0
Average duration: —
Total execution time: 0 hours

By Phase:

Phase	Plans	Total	Avg/Plan
-	-	-	-

Recent Trend:

Last 5 plans: —
Trend: —

Accumulated Context

Decisions

Decisions are logged in PROJECT.md Key Decisions table. Recent decisions affecting current work:

Roadmap: CGO_ENABLED=0 throughout — modernc.org/sqlite over mattn/go-sqlite3 (see PROJECT.md)
Roadmap: Per-source rate limiter architecture (Phase 9) must precede all OSINT source modules (Phases 10-16)
Roadmap: AES-256 encryption added in Phase 1, not post-hoc — avoids migration complexity
Roadmap: Verification (Phase 5) requires consent prompt + LEGAL.md — not optional polish
[Phase 01-foundation]: Provider YAML in dual locations: providers/ (user-visible) and pkg/providers/definitions/ (embed) — Go embed cannot use '..' paths
[Phase 01-foundation]: Aho-Corasick built with DFA=true at NewRegistry() for O(n) keyword pre-filtering across all providers
[Phase 01-foundation]: pkg/types/chunk.go breaks engine<->sources circular import; ants pool with WaitGroup+Mutex for detector coordination
[Phase 01-foundation]: Per-installation salt via settings table -- no hardcoded salt in production code
[Phase 01-foundation]: Exit code semantics: 0=clean, 1=keys-found, 2=error for CI/CD integration
[Phase 02-tier-1-2-providers]: AWS Bedrock verify URL left empty — SigV4 signing deferred to Phase 5 verification engine
[Phase 03-tier-3-9-providers]: Keyword-only detection for providers without documented key prefixes (You.com, Unstructured, Runway, Midjourney) to avoid false positives.
[Phase 04]: Use 'go mod download' instead of 'go mod tidy' when bootstrapping dependencies ahead of their consumers
[Phase 04-input-sources]: GitSource walks heads+tags+remotes+stash with per-OID blob dedup
[Phase 04]: Introduced selectSource dispatcher with sourceFlags struct for testable CLI source routing
[Phase 05]: Keep legacy VerifySpec ValidStatus/InvalidStatus alongside canonical SuccessCodes/FailureCodes; Effective*() helpers pick canonical-first with fallback
[Phase 05]: Store Finding.VerifyMetadata as JSON TEXT column; legacy DBs migrated in-place via PRAGMA table_info + conditional ALTER TABLE in storage.Open()
[Phase 05-verification-engine]: LEGAL.md dual-location mirror (root + pkg/legal/) required because go:embed cannot traverse parents — mirrors Phase 1 providers pattern
[Phase 05-verification-engine]: verify.consent setting: granted is sticky across runs; declined is not — users who initially refuse can change mind without manual reset
[Phase 05-verification-engine]: Plan 05-03: HTTPVerifier classifies via YAML VerifySpec only; no per-provider branches. VerifyAll uses ants pool with per-finding Result guarantee.
[Phase 05]: Verification runs in batch mode after scan completes (collect -> verify -> persist) with Result->Finding back-assignment via provider+masked-key tuple
[Phase 06]: Registry pattern for output formatters; TableFormatter strips ANSI when writer is not a TTY via zero-value lipgloss.Style
[Phase 06]: SARIF 2.1.0 via hand-rolled structs (no library) per CLAUDE.md
[Phase 06-output-reporting]: keys export rejects SARIF (scan-only); keys show always unmasked; keys verify updates findings inline via db.SQL().Exec
[Phase 08-dork-engine]: pkg/dorks mirrors pkg/providers go:embed pattern; //go:embed definitions/* tolerates empty .gitkeep-only tree
[Phase 08-dork-engine]: Runner + Executor interface separate from Registry so 08-05 GitHub executor registers without touching YAML loader
[Phase 10-osint-code-hosting]: Client handles retry only; rate limiting is caller's responsibility via LimiterRegistry
[Phase 10-osint-code-hosting]: github/gist use 'kw' in:file; all other sources use bare keyword
[Phase 10-osint-code-hosting]: GitHubSource reuses shared sources.Client + LimiterRegistry; builds queries from providers.Registry via BuildQueries; missing token disables (not errors)
[Phase 10]: RegisterAll registers all ten Phase 10 sources unconditionally; missing credentials flip Enabled()==false rather than hiding sources from the CLI catalog
[Phase 11]: RegisterAll extended to 18 sources (10 Phase 10 + 8 Phase 11); paste sources use BaseURL prefix in integration test to avoid /search path collision
[Phase 11]: Integration test uses injected test platforms for PasteSites (same pattern as SandboxesSource)
[Phase 11]: All five search sources use dork query format to focus on paste/code hosting leak sites
[Phase 12]: Shodan/Censys/ZoomEye use bare keyword queries; Censys POST+BasicAuth, Shodan key param, ZoomEye API-KEY header
[Phase 12]: RegisterAll extended to 28 sources (18 Phase 10-11 + 10 Phase 12); cloud scanners credentialless, IoT scanners credential-gated
[Phase 13]: GoProxy regex requires domain dot to filter non-module paths; NuGet projectUrl fallback to nuget.org canonical
[Phase 13]: KubernetesSource uses Artifact Hub rather than Censys/Shodan dorking to avoid duplicating Phase 12 sources
[Phase 13]: RegisterAll extended to 32 sources (28 Phase 10-12 + 4 Phase 13 container/IaC)
[Phase 13]: RegisterAll extended to 40 sources (28 Phase 10-12 + 12 Phase 13); package registry sources credentialless, no new SourcesConfig fields
[Phase 14]: RegisterAll extended to 45 sources (40 Phase 10-13 + 5 Phase 14 CI/CD); CircleCI gets dedicated CIRCLECI_TOKEN
[Phase 15]: Discord/Slack use dorking approach (configurable search endpoint) since neither has public message search API
[Phase 15]: Log aggregator sources are credentialless, targeting exposed instances
[Phase 16]: VT uses x-apikey header per official API v3 spec
[Phase 16]: IX uses three-step flow: POST search, GET results, GET file content
[Phase 16]: URLhaus tag lookup with payload endpoint fallback
[Phase 17]: Scheduler.ScanFunc callback decouples from engine; OnComplete bridges to notifications; disabled jobs stay in DB

Pending Todos

None yet.

Blockers/Concerns

Phase 1: Argon2 vs PBKDF2 for database encryption key derivation — needs decision before Storage Layer implementation
Phase 1: Aho-Corasick library choice (cloudflare/ahocorasick vs bobrik/ahocorasick) — verify which TruffleHog uses
Phase 2+: Provider YAML patterns for 108 providers — lesser-known providers need targeted research (Chinese LLMs, niche APIs)
Phase 11: Google Custom Search API quota (100 queries/day free tier) vs direct scraping ToS trade-off — product decision needed

Session Continuity

Last session: 2026-04-06T14:28:16.922Z Stopped at: Completed 17-02-PLAN.md Resume file: None

9.9 KiB Raw Blame History