12 KiB
12 KiB
phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, must_haves
| phase | plan | type | wave | depends_on | files_modified | autonomous | requirements | must_haves | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 08-dork-engine | 01 | execute | 1 |
|
true |
|
|
Purpose: Unblock parallel Wave 2 plans (50-dork YAML batches and GitHub live executor). Output: pkg/dorks package with passing tests + custom_dorks table migration.
<execution_context> @$HOME/.claude/get-shit-done/workflows/execute-plan.md @$HOME/.claude/get-shit-done/templates/summary.md </execution_context>
@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @.planning/phases/08-dork-engine/08-CONTEXT.md @pkg/providers/loader.go @pkg/providers/registry.go @pkg/storage/db.go @pkg/storage/schema.sqlFrom pkg/providers/loader.go:
//go:embed definitions/*.yaml
var definitionsFS embed.FS
func loadProviders() ([]Provider, error) {
fs.WalkDir(definitionsFS, "definitions", func(path string, d fs.DirEntry, err error) error { ... })
}
From pkg/providers/registry.go:
type Registry struct { providers []Provider; index map[string]int; ... }
func NewRegistry() (*Registry, error)
func (r *Registry) List() []Provider
func (r *Registry) Get(name string) (Provider, bool)
func (r *Registry) Stats() RegistryStats
From pkg/storage/db.go:
type DB struct { sql *sql.DB }
func (db *DB) SQL() *sql.DB
type Dork struct {
ID string `yaml:"id"`
Name string `yaml:"name"`
Source string `yaml:"source"` // github|google|shodan|censys|zoomeye|fofa|gitlab|bing
Category string `yaml:"category"` // frontier|specialized|infrastructure|emerging|enterprise
Query string `yaml:"query"`
Description string `yaml:"description"`
Tags []string `yaml:"tags"`
}
var ValidSources = []string{"github","google","shodan","censys","zoomeye","fofa","gitlab","bing"}
func (d Dork) Validate() error { /* non-empty id/source/query + source in ValidSources */ }
```
2. Create pkg/dorks/loader.go mirroring pkg/providers/loader.go:
```go
//go:embed definitions
var definitionsFS embed.FS
func loadDorks() ([]Dork, error) {
// fs.WalkDir on "definitions", descend into {source}/ subdirs, parse *.yaml
}
```
Walk pattern: definitions/github/*.yaml, definitions/google/*.yaml, etc.
Every file decoded via yaml.Unmarshal into Dork. Call Validate() per file; wrap
errors with file path. Return combined slice.
3. Create pkg/dorks/registry.go:
```go
type Registry struct {
dorks []Dork
byID map[string]int
bySource map[string][]int
byCategory map[string][]int
}
func NewRegistry() (*Registry, error) // uses loadDorks()
func NewRegistryFromDorks(ds []Dork) *Registry // for tests
func (r *Registry) List() []Dork
func (r *Registry) Get(id string) (Dork, bool)
func (r *Registry) ListBySource(src string) []Dork
func (r *Registry) ListByCategory(cat string) []Dork
func (r *Registry) Stats() Stats // {Total int; BySource map[string]int; ByCategory map[string]int}
```
4. Create pkg/dorks/executor.go (interface + source dispatcher, stubs only —
GitHub real impl comes in Plan 08-05):
```go
var ErrSourceNotImplemented = errors.New("dork source not yet implemented")
var ErrMissingAuth = errors.New("dork source requires auth credentials")
type Match struct {
DorkID string
Source string
URL string
Snippet string // content chunk to feed into engine detector
Path string // file path in repo, if applicable
}
type Executor interface {
Source() string
Execute(ctx context.Context, d Dork, limit int) ([]Match, error)
}
type Runner struct {
executors map[string]Executor
}
func NewRunner() *Runner { return &Runner{executors: map[string]Executor{}} }
func (r *Runner) Register(e Executor) { r.executors[e.Source()] = e }
func (r *Runner) Run(ctx context.Context, d Dork, limit int) ([]Match, error) {
ex, ok := r.executors[d.Source]
if !ok { return nil, fmt.Errorf("%w: %s (coming Phase 9-16)", ErrSourceNotImplemented, d.Source) }
return ex.Execute(ctx, d, limit)
}
```
No real executors are registered here — Plan 08-05 wires the GitHub executor via
a separate constructor (NewRunnerWithGitHub or similar).
5. Create pkg/dorks/registry_test.go with the behavior cases listed above.
Use NewRegistryFromDorks for synthetic fixtures — do NOT touch the real
embedded FS (downstream plans populate it). One test MAY call NewRegistry()
and only assert err is nil or "definitions directory empty" — acceptable
either way pre-YAML.
6. Create placeholder files to make go:embed succeed with empty tree:
- pkg/dorks/definitions/.gitkeep (empty)
- dorks/.gitkeep (empty)
IMPORTANT: go:embed requires at least one matching file. If
`//go:embed definitions` fails when only .gitkeep exists, switch the directive
to `//go:embed definitions/*` and handle the empty case by returning nil
dorks (no error) when WalkDir sees only .gitkeep. Test must pass with
zero real YAML present.
CREATE INDEX IF NOT EXISTS idx_custom_dorks_source ON custom_dorks(source);
CREATE INDEX IF NOT EXISTS idx_custom_dorks_category ON custom_dorks(category);
```
2. Create pkg/storage/custom_dorks.go:
```go
type CustomDork struct {
ID int64
DorkID string
Name string
Source string
Category string
Query string
Description string
Tags []string
CreatedAt time.Time
}
func (db *DB) SaveCustomDork(d CustomDork) (int64, error)
func (db *DB) ListCustomDorks() ([]CustomDork, error)
func (db *DB) GetCustomDork(id int64) (CustomDork, error) // returns sql.ErrNoRows if missing
func (db *DB) GetCustomDorkByDorkID(dorkID string) (CustomDork, error)
func (db *DB) DeleteCustomDork(id int64) (int64, error)
```
Tags round-tripped via encoding/json (TEXT column). Dork_id UNIQUE so
user cannot create duplicate custom IDs.
3. Create pkg/storage/custom_dorks_test.go covering the behavior cases above.
Use storage.Open(":memory:") as the existing storage tests do.
<success_criteria>
- pkg/dorks.NewRegistry() compiles and runs (zero or more embedded dorks)
- Executor interface + ErrSourceNotImplemented in place for Plan 08-05 and 08-06
- custom_dorks CRUD functional; downstream
dorks add/dorks deletecommands have a storage backend to call </success_criteria>