docs(01-02): complete provider registry plan
- SUMMARY.md: schema validation + embed loader + Aho-Corasick registry - STATE.md: updated progress (20%), decisions, metrics - ROADMAP.md: phase 01 in-progress (1/5 summaries) - REQUIREMENTS.md: marked CORE-02, CORE-03, CORE-06, PROV-10 complete
This commit is contained in:
salvacybersec
@@ -10,11 +10,11 @@ Requirements for initial release. Each maps to roadmap phases.
|
||||
### Core Engine
|
||||
|
||||
- [ ] **CORE-01**: Scanner engine detects API keys using keyword pre-filtering + regex matching pipeline
|
||||
- [ ] **CORE-02**: Provider definitions loaded from YAML files embedded at compile time via Go embed
|
||||
- [ ] **CORE-03**: Provider registry manages 108+ provider definitions with pattern, keyword, confidence, and verify metadata
|
||||
- [x] **CORE-02**: Provider definitions loaded from YAML files embedded at compile time via Go embed
|
||||
- [x] **CORE-03**: Provider registry manages 108+ provider definitions with pattern, keyword, confidence, and verify metadata
|
||||
- [ ] **CORE-04**: Entropy analysis as secondary signal for low-confidence providers (generic key formats)
|
||||
- [ ] **CORE-05**: Worker pool parallelism with configurable worker count (default: CPU count)
|
||||
- [ ] **CORE-06**: Aho-Corasick keyword pre-filter runs before regex for 10x performance on large files
|
||||
- [x] **CORE-06**: Aho-Corasick keyword pre-filter runs before regex for 10x performance on large files
|
||||
- [ ] **CORE-07**: mmap-based large file reading for memory efficiency
|
||||
|
||||
### Providers
|
||||
@@ -28,7 +28,7 @@ Requirements for initial release. Each maps to roadmap phases.
|
||||
- [ ] **PROV-07**: 10 Tier 7 Code/Dev Tools provider definitions (GitHub Copilot, Cursor, Tabnine, Codeium, Sourcegraph, CodeWhisperer, Replit AI, Codestral, watsonx, Oracle AI)
|
||||
- [ ] **PROV-08**: 10 Tier 8 Self-Hosted provider definitions (Ollama, vLLM, LocalAI, LM Studio, llama.cpp, GPT4All, text-gen-webui, TensorRT-LLM, Triton, Jan AI)
|
||||
- [ ] **PROV-09**: 8 Tier 9 Enterprise provider definitions (Salesforce Einstein, ServiceNow, SAP AI Core, Palantir, Databricks, Snowflake, Oracle GenAI, HPE GreenLake)
|
||||
- [ ] **PROV-10**: Provider YAML schema includes format_version and last_verified date for pattern health tracking
|
||||
- [x] **PROV-10**: Provider YAML schema includes format_version and last_verified date for pattern health tracking
|
||||
|
||||
### Input Sources
|
||||
|
||||
@@ -288,7 +288,7 @@ Requirements for initial release. Each maps to roadmap phases.
|
||||
| CORE-01, CORE-02, CORE-03, CORE-04, CORE-05, CORE-06, CORE-07 | Phase 1 | Pending |
|
||||
| STOR-01, STOR-02, STOR-03 | Phase 1 | Pending |
|
||||
| CLI-01, CLI-02, CLI-03, CLI-04, CLI-05 | Phase 1 | Pending |
|
||||
| PROV-10 | Phase 1 | Pending |
|
||||
| PROV-10 | Phase 1 | Complete |
|
||||
| PROV-01, PROV-02 | Phase 2 | Pending |
|
||||
| PROV-03, PROV-04, PROV-05, PROV-06, PROV-07, PROV-08, PROV-09 | Phase 3 | Pending |
|
||||
| INPUT-01, INPUT-02, INPUT-03, INPUT-04, INPUT-05, INPUT-06 | Phase 4 | Pending |
|
||||
|
||||
@@ -47,7 +47,7 @@ Decimal phases appear between their surrounding integers in numeric order.
|
||||
|
||||
Plans:
|
||||
- [ ] 01-01-PLAN.md — Go module init, dependency installation, test scaffolding and testdata fixtures
|
||||
- [ ] 01-02-PLAN.md — Provider registry: YAML schema, embed loader, Aho-Corasick automaton, Registry struct
|
||||
- [x] 01-02-PLAN.md — Provider registry: YAML schema, embed loader, Aho-Corasick automaton, Registry struct
|
||||
- [ ] 01-03-PLAN.md — Storage layer: AES-256-GCM encryption, Argon2id key derivation, SQLite + Finding CRUD
|
||||
- [ ] 01-04-PLAN.md — Scan engine pipeline: keyword pre-filter, regex+entropy detector, FileSource, ants worker pool
|
||||
- [ ] 01-05-PLAN.md — CLI wiring: scan, providers list/info/stats, config init/set/get, output table
|
||||
|
||||
@@ -1,3 +1,19 @@
|
||||
---
|
||||
gsd_state_version: 1.0
|
||||
milestone: v1.0
|
||||
milestone_name: milestone
|
||||
status: planning
|
||||
stopped_at: Completed 01-foundation 01-02-PLAN.md
|
||||
last_updated: "2026-04-04T21:12:49.099Z"
|
||||
last_activity: 2026-04-04 — Roadmap created, 18 phases defined covering 146 v1 requirements
|
||||
progress:
|
||||
total_phases: 18
|
||||
completed_phases: 0
|
||||
total_plans: 5
|
||||
completed_plans: 1
|
||||
percent: 20
|
||||
---
|
||||
|
||||
# Project State
|
||||
|
||||
## Project Reference
|
||||
@@ -14,11 +30,12 @@ Plan: 0 of ? in current phase
|
||||
Status: Ready to plan
|
||||
Last activity: 2026-04-04 — Roadmap created, 18 phases defined covering 146 v1 requirements
|
||||
|
||||
Progress: [░░░░░░░░░░░░░░░░░░░░] 0%
|
||||
Progress: [██░░░░░░░░] 20%
|
||||
|
||||
## Performance Metrics
|
||||
|
||||
**Velocity:**
|
||||
|
||||
- Total plans completed: 0
|
||||
- Average duration: —
|
||||
- Total execution time: 0 hours
|
||||
@@ -30,10 +47,12 @@ Progress: [░░░░░░░░░░░░░░░░░░░░] 0%
|
||||
| - | - | - | - |
|
||||
|
||||
**Recent Trend:**
|
||||
|
||||
- Last 5 plans: —
|
||||
- Trend: —
|
||||
|
||||
*Updated after each plan completion*
|
||||
| Phase 01-foundation P02 | 9 | 2 tasks | 11 files |
|
||||
|
||||
## Accumulated Context
|
||||
|
||||
@@ -46,6 +65,8 @@ Recent decisions affecting current work:
|
||||
- Roadmap: Per-source rate limiter architecture (Phase 9) must precede all OSINT source modules (Phases 10-16)
|
||||
- Roadmap: AES-256 encryption added in Phase 1, not post-hoc — avoids migration complexity
|
||||
- Roadmap: Verification (Phase 5) requires consent prompt + LEGAL.md — not optional polish
|
||||
- [Phase 01-foundation]: Provider YAML in dual locations: providers/ (user-visible) and pkg/providers/definitions/ (embed) — Go embed cannot use '..' paths
|
||||
- [Phase 01-foundation]: Aho-Corasick built with DFA=true at NewRegistry() for O(n) keyword pre-filtering across all providers
|
||||
|
||||
### Pending Todos
|
||||
|
||||
@@ -60,6 +81,6 @@ None yet.
|
||||
|
||||
## Session Continuity
|
||||
|
||||
Last session: 2026-04-04
|
||||
Stopped at: Roadmap written to .planning/ROADMAP.md; ready to begin Phase 1 planning
|
||||
Last session: 2026-04-04T21:12:49.095Z
|
||||
Stopped at: Completed 01-foundation 01-02-PLAN.md
|
||||
Resume file: None
|
||||
|
||||
157
.planning/phases/01-foundation/01-02-SUMMARY.md
Normal file
157
.planning/phases/01-foundation/01-02-SUMMARY.md
Normal file
@@ -0,0 +1,157 @@
|
||||
---
|
||||
phase: 01-foundation
|
||||
plan: 02
|
||||
subsystem: providers
|
||||
tags: [yaml, embed, aho-corasick, registry, go-embed, gopkg.in/yaml.v3]
|
||||
|
||||
# Dependency graph
|
||||
requires:
|
||||
- phase: 01-01
|
||||
provides: go.mod with all Phase 1 dependencies, test scaffolding, cmd/root.go stub
|
||||
provides:
|
||||
- Provider, Pattern, VerifySpec, RegistryStats Go structs with YAML validation
|
||||
- Registry with List(), Get(), Stats(), AC() methods
|
||||
- Aho-Corasick automaton built from all provider keywords at NewRegistry()
|
||||
- Three reference provider YAML definitions (openai, anthropic, huggingface)
|
||||
- Compile-time embed of provider YAML via pkg/providers/definitions/
|
||||
affects:
|
||||
- scan-engine
|
||||
- cli-providers-command
|
||||
- verification-engine
|
||||
- storage-layer
|
||||
|
||||
# Tech tracking
|
||||
tech-stack:
|
||||
added:
|
||||
- gopkg.in/yaml.v3 (UnmarshalYAML custom validation)
|
||||
- github.com/petar-dambovaliev/aho-corasick (keyword pre-filter automaton)
|
||||
- embed (stdlib) for compile-time YAML embedding
|
||||
patterns:
|
||||
- Provider YAML at providers/ (user-visible) + pkg/providers/definitions/ (embed location)
|
||||
- Type alias pattern for custom UnmarshalYAML without infinite recursion
|
||||
- Registry injected via constructor (NewRegistry), not global singleton
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- pkg/providers/schema.go
|
||||
- pkg/providers/loader.go
|
||||
- pkg/providers/registry.go
|
||||
- pkg/providers/registry_test.go
|
||||
- pkg/providers/definitions/openai.yaml
|
||||
- pkg/providers/definitions/anthropic.yaml
|
||||
- pkg/providers/definitions/huggingface.yaml
|
||||
- providers/openai.yaml
|
||||
- providers/anthropic.yaml
|
||||
- providers/huggingface.yaml
|
||||
modified: []
|
||||
|
||||
key-decisions:
|
||||
- "Provider YAML kept in dual locations: providers/ (user-visible) and pkg/providers/definitions/ (embedded) — Go embed cannot use '..' paths, so definitions/ subdirectory is canonical embed source"
|
||||
- "UnmarshalYAML validates format_version >= 1 and non-empty last_verified at parse time, not at registry use time — fail fast on malformed definitions"
|
||||
- "Aho-Corasick automaton built with DFA=true for deterministic performance — trades memory for guaranteed O(n) matching"
|
||||
- "Registry is value-safe for concurrent reads — no mutex needed since providers slice is written once at NewRegistry and never mutated"
|
||||
|
||||
patterns-established:
|
||||
- "Pattern 1: Type alias in UnmarshalYAML to avoid infinite recursion: `type ProviderAlias Provider`"
|
||||
- "Pattern 2: embed path convention — YAML at pkg/providers/definitions/, user docs at providers/"
|
||||
- "Pattern 3: Registry constructor NewRegistry() loads+validates+indexes+builds AC in one call"
|
||||
|
||||
requirements-completed: [CORE-02, CORE-03, CORE-06, PROV-10]
|
||||
|
||||
# Metrics
|
||||
duration: 9min
|
||||
completed: 2026-04-04
|
||||
---
|
||||
|
||||
# Phase 01 Plan 02: Provider Registry Summary
|
||||
|
||||
**YAML schema structs with UnmarshalYAML validation, embed.FS loader, and Aho-Corasick registry serving List/Get/Stats/AC to all downstream subsystems**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** ~9 min
|
||||
- **Started:** 2026-04-04T21:02:31Z
|
||||
- **Completed:** 2026-04-04T21:11:41Z
|
||||
- **Tasks:** 2 (both TDD)
|
||||
- **Files modified:** 10 created, 1 updated (registry_test.go)
|
||||
|
||||
## Accomplishments
|
||||
|
||||
- Provider YAML schema with compile-time validation (format_version >= 1, last_verified required, confidence enum)
|
||||
- Registry loads 3 providers from embedded YAML at startup, builds Aho-Corasick automaton over all keywords
|
||||
- Three reference provider YAML definitions with full verify specs (OpenAI, Anthropic, HuggingFace)
|
||||
- All 5 provider tests pass: TestRegistryLoad, TestRegistryGet, TestRegistryStats, TestAhoCorasickBuild, TestProviderSchemaValidation
|
||||
|
||||
## Task Commits
|
||||
|
||||
Each task was committed atomically:
|
||||
|
||||
1. **TDD RED - Failing tests for schema and registry** - `ebaf7d7` (test)
|
||||
2. **Task 1: Provider schema structs and reference YAMLs** - `4fcdc42` (feat)
|
||||
3. **Task 2: Embed loader, registry with AC, filled test stubs** - `a9859b3` (feat)
|
||||
|
||||
_Note: Bootstrap (go.mod, main.go, test stubs) was included in the RED commit since Plan 01-01 runs in parallel._
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
- `pkg/providers/schema.go` - Provider, Pattern, VerifySpec, RegistryStats structs with UnmarshalYAML validation
|
||||
- `pkg/providers/loader.go` - embed.FS declaration with //go:embed definitions/*.yaml and fs.WalkDir loader
|
||||
- `pkg/providers/registry.go` - Registry struct with List(), Get(), Stats(), AC() methods and NewRegistry() constructor
|
||||
- `pkg/providers/registry_test.go` - Full test implementation (replaced stub from Plan 01)
|
||||
- `pkg/providers/definitions/openai.yaml` - Embedded OpenAI provider definition
|
||||
- `pkg/providers/definitions/anthropic.yaml` - Embedded Anthropic provider definition
|
||||
- `pkg/providers/definitions/huggingface.yaml` - Embedded HuggingFace provider definition
|
||||
- `providers/openai.yaml` - User-visible OpenAI reference definition
|
||||
- `providers/anthropic.yaml` - User-visible Anthropic reference definition
|
||||
- `providers/huggingface.yaml` - User-visible HuggingFace reference definition
|
||||
|
||||
## Decisions Made
|
||||
|
||||
- **Dual YAML location:** providers/ for user reference, pkg/providers/definitions/ for embed — Go's embed package cannot traverse `..` paths, so definitions/ inside the package is the only valid embed location.
|
||||
- **DFA mode for Aho-Corasick:** `Opts{DFA: true}` chosen for guaranteed O(n) matching at cost of higher upfront build time — appropriate for a scanner tool that pays build cost once and scans many files.
|
||||
- **Constructor injection over globals:** NewRegistry() returns a value; callers inject it. No package-level `var Registry` global — avoids init order issues and enables testing.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
### Auto-fixed Issues
|
||||
|
||||
**1. [Rule 3 - Blocking] Bootstrapped Plan 01-01 prerequisites in this worktree**
|
||||
- **Found during:** Pre-task setup
|
||||
- **Issue:** Plan 01-02 depends on Plan 01-01 (go.mod, main.go, test stubs) which runs in parallel in a different worktree. This worktree had no go.mod.
|
||||
- **Fix:** Executed Plan 01-01 bootstrap (go mod init, go get all 10 deps, main.go, cmd/root.go, testdata fixtures, test stub files) before starting Plan 01-02 tasks.
|
||||
- **Files modified:** go.mod, go.sum, main.go, cmd/root.go, testdata/samples/*.txt, pkg/*/stub_test.go files
|
||||
- **Verification:** `go build ./...` succeeded before Plan 01-02 task execution
|
||||
- **Committed in:** ebaf7d7 (RED phase commit includes bootstrap)
|
||||
|
||||
**2. [Rule 3 - Blocking] go mod tidy required after adding production packages**
|
||||
- **Found during:** Task 2 GREEN phase
|
||||
- **Issue:** `go test` failed with "no required module provides package github.com/petar-dambovaliev/aho-corasick" even though it was in go.mod — tidy hadn't propagated it for non-test code.
|
||||
- **Fix:** Ran `go mod tidy` which resolved the module graph.
|
||||
- **Files modified:** go.mod, go.sum
|
||||
- **Verification:** `go test ./pkg/providers/...` passed after tidy
|
||||
|
||||
---
|
||||
|
||||
**Total deviations:** 2 auto-fixed (2 blocking/infrastructure)
|
||||
**Impact on plan:** Both deviations were infrastructure setup, not scope changes. Plan objectives met exactly.
|
||||
|
||||
## Issues Encountered
|
||||
|
||||
- Go embed `..` path restriction required dual YAML directory strategy (documented in plan's context, confirmed during implementation)
|
||||
- aho-corasick package name is `aho_corasick` (underscore) not `ahocorasick` — used import alias `ahocorasick` for cleaner code
|
||||
|
||||
## User Setup Required
|
||||
|
||||
None - no external service configuration required.
|
||||
|
||||
## Next Phase Readiness
|
||||
|
||||
- Registry interface is stable: NewRegistry(), List(), Get(), Stats(), AC() — downstream plans can depend on these signatures
|
||||
- Phase 03 (Storage Layer) can proceed immediately — no registry dependency
|
||||
- Phase 04 (Scan Engine) can now wire AC() for keyword pre-filtering
|
||||
- Phase 05 (CLI) can call Registry.List() for `keyhunter providers list`
|
||||
- Known: only 3 reference providers embedded; Phase 02-03 will add all 108
|
||||
|
||||
---
|
||||
*Phase: 01-foundation*
|
||||
*Completed: 2026-04-04*
|
||||
Reference in New Issue
Block a user