docs(08-01): complete dork engine foundation plan

SUMMARY, STATE, ROADMAP, and REQUIREMENTS updates for pkg/dorks foundation + custom_dorks storage (DORK-01, DORK-03).
2026-04-06 00:17:53 +03:00
parent 01062b88b1
commit 2dc7078708
4 changed files with 155 additions and 12 deletions
--- a/.planning/phases/08-dork-engine/08-01-SUMMARY.md
+++ b/.planning/phases/08-dork-engine/08-01-SUMMARY.md
@@ -0,0 +1,140 @@
+---
+phase: 08-dork-engine
+plan: 01
+subsystem: dork-engine
+tags: [dorks, yaml, go-embed, sqlite, registry, executor]
+
+requires:
+  - phase: 01-foundation
+    provides: pkg/providers go:embed loader + Registry pattern, pkg/storage.Open + schema.sql migration harness
+provides:
+  - pkg/dorks package (schema, go:embed loader, Registry, Executor interface, Runner dispatcher)
+  - ErrSourceNotImplemented / ErrMissingAuth sentinel errors for per-source executors
+  - custom_dorks SQLite table with SaveCustomDork / ListCustomDorks / GetCustomDork / GetCustomDorkByDorkID / DeleteCustomDork CRUD
+  - pkg/dorks/definitions/ tree ready for Wave 2 YAML batches
+affects: [08-02, 08-03, 08-04, 08-05, 08-06, 08-07, 09-osint, 10-osint]
+
+tech-stack:
+  added: []
+  patterns:
+    - "go:embed dir/* directive tolerating empty tree via fs.SkipAll on ErrNotExist"
+    - "Registry mirroring pkg/providers: byID + bySource + byCategory index maps"
+    - "Runner pattern for per-source Executor dispatch with sentinel-error fallbacks"
+    - "JSON-serialised tags stored in SQLite TEXT column with rowScanner interface"
+
+key-files:
+  created:
+    - pkg/dorks/schema.go
+    - pkg/dorks/loader.go
+    - pkg/dorks/registry.go
+    - pkg/dorks/executor.go
+    - pkg/dorks/registry_test.go
+    - pkg/dorks/definitions/.gitkeep
+    - dorks/.gitkeep
+    - pkg/storage/custom_dorks.go
+    - pkg/storage/custom_dorks_test.go
+  modified:
+    - pkg/storage/schema.sql
+
+key-decisions:
+  - "Used //go:embed definitions/* (not definitions/*.yaml) so the placeholder .gitkeep keeps the embed directive valid pre-YAML"
+  - "Loader tolerates a definitions tree containing only .gitkeep by skipping non-.yaml entries and returning nil dorks without error"
+  - "Runner + Executor interface split (separate from Registry) so Plan 08-05 can wire GitHub live without touching registry internals"
+  - "custom_dorks.dork_id UNIQUE constraint at SQL layer prevents user duplicate IDs; embedded dork IDs are orthogonal (read-only)"
+  - "Tags serialised as JSON in a TEXT column (no join table) to keep CRUD trivial for a user-authored data set"
+
+patterns-established:
+  - "Embedded YAML registries in KeyHunter follow the pkg/providers blueprint: schema.go + loader.go (go:embed) + registry.go with index maps + per-thing tests using NewRegistryFromX synthetic fixtures"
+  - "Per-source executor dispatch: interface + Runner.Register + ErrSourceNotImplemented sentinel for deferred backends"
+
+requirements-completed: [DORK-01, DORK-03]
+
+duration: ~15min
+completed: 2026-04-05
+---
+
+# Phase 08 Plan 01: Dork Engine Foundation Summary
+
+**pkg/dorks package mirroring the pkg/providers go:embed pattern with Registry, Executor interface, Runner dispatch, and custom_dorks SQLite CRUD — ready for Wave 2 YAML batches and the Plan 08-05 GitHub executor.**
+
+## Performance
+
+- **Duration:** ~15 min
+- **Started:** 2026-04-05
+- **Completed:** 2026-04-05
+- **Tasks:** 2
+- **Files created:** 9
+- **Files modified:** 1
+
+## Accomplishments
+- `pkg/dorks` package compiles and exposes `NewRegistry`, `NewRegistryFromDorks`, `Get`, `List`, `ListBySource`, `ListByCategory`, `Stats`, `Runner`, `Executor`, `Match`, `ErrSourceNotImplemented`, `ErrMissingAuth`
+- `go:embed definitions/*` loader tolerates an empty tree (only `.gitkeep`), so Wave 2 plans can drop YAML files into `pkg/dorks/definitions/{source}/` with zero loader changes
+- `custom_dorks` table + CRUD fully exercised via round-trip tests on `storage.Open(":memory:")`
+- All tests green, `go build ./...` clean, no regressions in existing storage tests
+
+## Task Commits
+
+1. **Task 1: pkg/dorks foundation (schema, loader, registry, executor, tests)** — `fd6efbb` (feat)
+2. **Task 2: custom_dorks storage table + CRUD + tests** — `01062b8` (feat)
+
+_Note: Tests were co-located with implementation in a single commit per task; `NewRegistryFromDorks` served as the synthetic-fixture entry point so the embedded FS was never required during testing._
+
+## Files Created/Modified
+
+- `pkg/dorks/schema.go` — `Dork` struct, `ValidSources`/`ValidCategories`, `Validate()`, `Stats`
+- `pkg/dorks/loader.go` — `//go:embed definitions/*` loader tolerating empty tree, validates every parsed dork, wraps errors with file path
+- `pkg/dorks/registry.go` — `Registry` with `byID`, `bySource`, `byCategory` indexes and `List/Get/ListBySource/ListByCategory/Stats`
+- `pkg/dorks/executor.go` — `Match`, `Executor` interface, `Runner` dispatcher, `ErrSourceNotImplemented`, `ErrMissingAuth`
+- `pkg/dorks/registry_test.go` — 9 test functions covering registry methods, `Validate`, `Runner` dispatch, empty-tree `NewRegistry`
+- `pkg/dorks/definitions/.gitkeep` — keeps embed directive valid pre-YAML
+- `dorks/.gitkeep` — placeholder at repo root per plan layout
+- `pkg/storage/schema.sql` — added `custom_dorks` CREATE TABLE + `idx_custom_dorks_source` + `idx_custom_dorks_category`
+- `pkg/storage/custom_dorks.go` — `CustomDork` struct + `SaveCustomDork / ListCustomDorks / GetCustomDork / GetCustomDorkByDorkID / DeleteCustomDork`, JSON-encoded tags, shared `rowScanner` helper
+- `pkg/storage/custom_dorks_test.go` — 7 tests: round-trip, newest-first list, not-found, by-dork-id lookup, delete + double-delete no-op, unique constraint, schema migration idempotency
+
+## Decisions Made
+
+- **`definitions/*` instead of `definitions/*.yaml`**: The `.yaml`-only form fails compilation when the directory contains only `.gitkeep`. Using `*` and filtering by extension at walk time allows the foundation plan to ship with zero YAML and for Wave 2 plans to add files incrementally.
+- **Separate `Runner` from `Registry`**: The executor plumbing is its own type so downstream plans (08-05 GitHub, 09-16 OSINT) can register implementations without coupling to the YAML index. `Registry` stays read-only and cache-safe.
+- **Tags as JSON TEXT, not a join table**: A junction table for ~150 user-authored rows worth of tags is not justified. `encoding/json` round-trip is trivial and keeps the CRUD API flat.
+- **`dork_id` UNIQUE at SQL layer**: Enforces user-visible ID uniqueness deterministically; tested via duplicate-insert failure.
+
+## Deviations from Plan
+
+None — plan executed exactly as written. The only nuance was the deliberate use of `//go:embed definitions/*` instead of `//go:embed definitions` (the plan action step called out this exact contingency and the implementation picked the tolerant form).
+
+## Issues Encountered
+
+None.
+
+## User Setup Required
+
+None — no external service configuration required for this plan.
+
+## Next Phase Readiness
+
+- Wave 2 plans (08-02 / 08-03 / 08-04 / 08-07) can drop YAML files into `pkg/dorks/definitions/{source}/` and they will load automatically via `NewRegistry()`
+- Plan 08-05 (GitHub live executor) can implement `dorks.Executor` and `Runner.Register` without modifying registry code
+- Plan 08-06 (CLI `dorks add/list/delete/info`) has `pkg/storage.CustomDork` CRUD available immediately
+- `ErrSourceNotImplemented` gives Plan 08-05's CLI a clean error path for non-GitHub sources until OSINT phases 9-16 arrive
+
+## Self-Check: PASSED
+
+- pkg/dorks/schema.go — FOUND
+- pkg/dorks/loader.go — FOUND
+- pkg/dorks/registry.go — FOUND
+- pkg/dorks/executor.go — FOUND
+- pkg/dorks/registry_test.go — FOUND
+- pkg/dorks/definitions/.gitkeep — FOUND
+- dorks/.gitkeep — FOUND
+- pkg/storage/schema.sql — FOUND (modified)
+- pkg/storage/custom_dorks.go — FOUND
+- pkg/storage/custom_dorks_test.go — FOUND
+- commit fd6efbb — FOUND
+- commit 01062b8 — FOUND
+- `go build ./...` — PASSED
+- `go test ./pkg/dorks/... ./pkg/storage/...` — PASSED
+
+---
+*Phase: 08-dork-engine*
+*Completed: 2026-04-05*