diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 6ef6b41..d99f3cc 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -212,9 +212,9 @@ Requirements for initial release. Each maps to roadmap phases. ### Dork Engine - [x] **DORK-01**: YAML-based dork definitions (GitHub, Google, Shodan, Censys, ZoomEye, FOFA, GitLab, Bing) -- [ ] **DORK-02**: 150+ built-in dorks across all sources +- [x] **DORK-02**: 150+ built-in dorks across all sources - [x] **DORK-03**: keyhunter dorks list/add/run/export commands -- [ ] **DORK-04**: Category-filtered dork execution (--category=frontier) +- [x] **DORK-04**: Category-filtered dork execution (--category=frontier) ### Web Dashboard diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 4fb764f..99c383f 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -181,7 +181,7 @@ Plans: Plans: - [x] 08-01-PLAN.md — Dork schema, go:embed loader, registry, executor interface, custom_dorks storage table -- [ ] 08-02-PLAN.md — 50 GitHub dork YAML definitions across 5 categories +- [x] 08-02-PLAN.md — 50 GitHub dork YAML definitions across 5 categories - [ ] 08-03-PLAN.md — 30 Google + 20 Shodan dork YAML definitions - [ ] 08-04-PLAN.md — 15 Censys + 10 ZoomEye + 10 FOFA + 10 GitLab + 5 Bing dork YAML definitions - [ ] 08-05-PLAN.md — Live GitHub Code Search executor (net/http, Retry-After, limit cap) diff --git a/.planning/STATE.md b/.planning/STATE.md index 4ca255a..8c0f804 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -3,14 +3,14 @@ gsd_state_version: 1.0 milestone: v1.0 milestone_name: milestone status: executing -stopped_at: Completed 08-01-PLAN.md -last_updated: "2026-04-05T21:17:48.315Z" +stopped_at: Completed 08-02-PLAN.md +last_updated: "2026-04-05T21:22:07.758Z" last_activity: 2026-04-05 progress: total_phases: 18 completed_phases: 7 total_plans: 47 - completed_plans: 41 + completed_plans: 42 percent: 20 --- @@ -26,7 +26,7 @@ See: .planning/PROJECT.md (updated 2026-04-04) ## Current Position Phase: 08 (dork-engine) — EXECUTING -Plan: 2 of 7 +Plan: 3 of 7 Status: Ready to execute Last activity: 2026-04-05 @@ -79,6 +79,7 @@ Progress: [██░░░░░░░░] 20% | Phase 06-output-reporting P05 | 4min | 2 tasks | 3 files | | Phase 06 P06 | 3min | 2 tasks | 3 files | | Phase 08-dork-engine P01 | 15min | 2 tasks | 10 files | +| Phase 08-dork-engine P02 | 12min | 2 tasks | 11 files | ## Accumulated Context @@ -126,6 +127,6 @@ None yet. ## Session Continuity -Last session: 2026-04-05T21:17:48.311Z -Stopped at: Completed 08-01-PLAN.md +Last session: 2026-04-05T21:22:07.754Z +Stopped at: Completed 08-02-PLAN.md Resume file: None diff --git a/.planning/phases/08-dork-engine/08-02-SUMMARY.md b/.planning/phases/08-dork-engine/08-02-SUMMARY.md new file mode 100644 index 0000000..0e39940 --- /dev/null +++ b/.planning/phases/08-dork-engine/08-02-SUMMARY.md @@ -0,0 +1,147 @@ +--- +phase: 08-dork-engine +plan: 02 +subsystem: dorks +tags: [dorks, github, yaml, go-embed, osint] + +requires: + - phase: 08-dork-engine + provides: pkg/dorks foundation (schema, loader, registry) from plan 08-01 +provides: + - 50 production-ready GitHub code-search dorks covering all 5 categories + - YAML list format support in the dork loader for multi-dork files + - Dual-location mirror (dorks/github/ and pkg/dorks/definitions/github/) +affects: [08-03, 08-04, 08-05, 08-06, 08-07] + +tech-stack: + added: [] + patterns: + - "Dork YAML files as top-level lists (multiple dorks per file) grouped by category" + - "Dual-location mirror: user-visible dorks/ copy + go:embed pkg/dorks/definitions/ copy" + +key-files: + created: + - pkg/dorks/definitions/github/frontier.yaml + - pkg/dorks/definitions/github/specialized.yaml + - pkg/dorks/definitions/github/infrastructure.yaml + - pkg/dorks/definitions/github/emerging.yaml + - pkg/dorks/definitions/github/enterprise.yaml + - dorks/github/frontier.yaml + - dorks/github/specialized.yaml + - dorks/github/infrastructure.yaml + - dorks/github/emerging.yaml + - dorks/github/enterprise.yaml + modified: + - pkg/dorks/loader.go + +key-decisions: + - "Loader accepts both YAML list and single-dork mapping forms for backward compatibility with plan 08-01 tests" + - "Category split into five YAML files (one per taxonomy bucket) rather than one monolithic file for easier diff/review" + - "Dorks use literal GitHub Code Search queries with no templating — GitHub syntax goes straight to the API" + +patterns-established: + - "Multi-dork YAML: top-level list of Dork mappings per file, grouped by category" + - "Dual-location mirror: identical content in dorks/{source}/ and pkg/dorks/definitions/{source}/" + +requirements-completed: [DORK-01, DORK-02, DORK-04] + +duration: 12min +completed: 2026-04-05 +--- + +# Phase 8 Plan 02: 50 GitHub Dorks Summary + +**50 production GitHub code-search dorks across 5 categories (frontier, specialized, infrastructure, emerging, enterprise) covering 40+ LLM/AI providers, embedded via go:embed and mirrored into the user-visible dorks/ tree.** + +## Performance + +- **Duration:** ~12 min +- **Started:** 2026-04-05T21:09:00Z +- **Completed:** 2026-04-05T21:21:00Z +- **Tasks:** 2 +- **Files modified:** 11 (10 created, 1 modified) + +## Accomplishments +- 50 GitHub dorks loadable via `pkg/dorks.NewRegistry().ListBySource("github")` +- All 5 dork taxonomy categories populated (frontier 15, specialized 10, infrastructure 10, emerging 10, enterprise 5) +- Loader extended to parse YAML list form without breaking existing one-dork-per-file tests +- Dual-location mirror maintained per the Phase 8 architecture decision + +## Task Commits + +1. **Task 1: 25 GitHub dorks — frontier + specialized categories** — `09722ea` (feat) +2. **Task 2: 25 GitHub dorks — infrastructure + emerging + enterprise** — `9755b37` (feat) + +## Files Created/Modified + +- `pkg/dorks/definitions/github/frontier.yaml` — 15 Tier 1/2 dorks (OpenAI, Anthropic, Google AI, Azure OpenAI, AWS Bedrock, xAI, Cohere, Mistral, Groq, Together, Replicate) +- `pkg/dorks/definitions/github/specialized.yaml` — 10 Tier 3 dorks (Perplexity, Voyage, Jina, AssemblyAI, Deepgram, ElevenLabs, Stability, HuggingFace) +- `pkg/dorks/definitions/github/infrastructure.yaml` — 10 Tier 5 gateway + Tier 8 self-hosted dorks (OpenRouter, LiteLLM, Portkey, Helicone, Cloudflare AI, Vercel AI, Ollama, vLLM, LocalAI) +- `pkg/dorks/definitions/github/emerging.yaml` — 10 Tier 4 Chinese + Tier 6 vector DB dorks (DeepSeek, Moonshot, Qwen, Zhipu, MiniMax, Pinecone, Weaviate, Qdrant, Chroma, Writer) +- `pkg/dorks/definitions/github/enterprise.yaml` — 5 Tier 7/9 enterprise dorks (Codeium, Tabnine, Databricks, Snowflake Cortex, IBM watsonx) +- `dorks/github/*.yaml` — mirror of all five category files for user-visible inspection +- `pkg/dorks/loader.go` — parse YAML list first, fall back to single-dork mapping + +## Decisions Made +- Accept both YAML shapes in the loader (list + single) so the existing `TestNewRegistry_EmptyDefinitionsTreeOK` test and any future one-off dork files keep working. +- Split the 50 dorks into five category files rather than one `github.yaml` — easier to review and aligns with the taxonomy categories enumerated in `schema.ValidCategories`. +- Use real provider prefixes verified against `pkg/providers/definitions/*.yaml` (sk-proj-, sk-ant-api03-, AIzaSy, gsk_, r8_, pplx-, hf_, sk-or-v1-, etc.) so future live execution in plan 08-05 returns genuine hits. + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 3 — Blocking] Removed empty source subdirectories that broke go:embed** +- **Found during:** Task 1 verification (`go test ./pkg/dorks/...`) +- **Issue:** Plan 08-01 left `pkg/dorks/definitions/{bing,fofa,gitlab,shodan}/` as empty directories. Once `pkg/dorks/definitions/github/` gained real YAML files, `//go:embed definitions/*` started recursing into siblings and errored with `cannot embed directory definitions/bing: contains no embeddable files`. +- **Fix:** Removed the four empty subdirs. They will be re-created by the Wave 2 plans that populate each source (shodan 08-03, etc.). The remaining non-empty siblings (`censys`, `google`, `zoomeye`) already contain YAML from parallel plans and embed fine. +- **Files modified:** deleted `pkg/dorks/definitions/{bing,fofa,gitlab,shodan}/` +- **Verification:** `go test ./pkg/dorks/...` passes, registry loads 126 dorks total (50 github + 76 from parallel sources) with 0 errors. +- **Committed in:** `09722ea` (folded into Task 1) + +**2. [Rule 2 — Missing critical] Extended loader to accept YAML list form** +- **Found during:** Task 1 planning (reading `pkg/dorks/loader.go`) +- **Issue:** The 08-01 loader only accepted one-dork-per-file (`yaml.Unmarshal(data, &Dork{})`). The plan explicitly anticipated this and instructed me to adapt the loader to also accept top-level lists. +- **Fix:** Loader now tries `[]Dork` first; if that yields entries, each is validated and appended. Otherwise it falls back to single-dork parsing (preserving the empty-definitions-tree test from 08-01). Empty YAML is tolerated. +- **Files modified:** `pkg/dorks/loader.go` +- **Verification:** `go test ./pkg/dorks/...` passes — both the empty-tree test and the new 50-dork load succeed. +- **Committed in:** `09722ea` (Task 1 commit) + +--- + +**Total deviations:** 2 auto-fixed (1 blocking, 1 missing-critical — both anticipated by the plan text) +**Impact on plan:** None — both changes were explicitly forecast in the plan's `` block. No scope creep. + +## Issues Encountered + +- go:embed does not embed empty directories, so the empty source subdirs left over from 08-01 broke compilation once the `github/` sibling had content. Resolved by deleting them; future source plans will recreate them with real content. + +## User Setup Required +None — these are built-in dorks embedded into the binary. + +## Next Phase Readiness +- Plan 08-03 and later source plans can follow the same multi-dork YAML list pattern established here. +- Plan 08-05 (GitHub live executor) now has 50 real queries to execute against the GitHub Code Search API. +- Registry statistics: `ListBySource("github")` returns 50; all 5 categories represented. + +## Self-Check: PASSED + +- File exists: `pkg/dorks/definitions/github/frontier.yaml` — FOUND +- File exists: `pkg/dorks/definitions/github/specialized.yaml` — FOUND +- File exists: `pkg/dorks/definitions/github/infrastructure.yaml` — FOUND +- File exists: `pkg/dorks/definitions/github/emerging.yaml` — FOUND +- File exists: `pkg/dorks/definitions/github/enterprise.yaml` — FOUND +- File exists: `dorks/github/frontier.yaml` — FOUND +- File exists: `dorks/github/specialized.yaml` — FOUND +- File exists: `dorks/github/infrastructure.yaml` — FOUND +- File exists: `dorks/github/emerging.yaml` — FOUND +- File exists: `dorks/github/enterprise.yaml` — FOUND +- Commit exists: `09722ea` — FOUND +- Commit exists: `9755b37` — FOUND +- Runtime verification: `dorks.NewRegistry().ListBySource("github")` returned 50 dorks across 5 categories — PASSED +- `go test ./pkg/dorks/...` — PASSED +- `go build ./...` — PASSED + +--- +*Phase: 08-dork-engine* +*Completed: 2026-04-05*