keyhunter/.planning/phases/05-verification-engine/05-01-PLAN.md

phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, must_haves

phase	plan	type	wave	depends_on	files_modified	autonomous	requirements	must_haves
05-verification-engine	01	execute	0		go.mod go.sum pkg/providers/schema.go pkg/engine/finding.go pkg/storage/schema.sql pkg/storage/findings.go pkg/storage/findings_test.go	true	VRFY-02 VRFY-03	truths artifacts key_links VerifySpec struct carries SuccessCodes, FailureCodes, RateLimitCodes, MetadataPaths, Body Finding struct carries Verified, VerifyStatus, VerifyMetadata fields findings table has verified/verify_status/verify_metadata_json columns and existing DBs migrate without data loss github.com/tidwall/gjson is in go.mod path provides contains pkg/providers/schema.go Extended VerifySpec with SuccessCodes/FailureCodes/RateLimitCodes/MetadataPaths/Body SuccessCodes path provides contains pkg/engine/finding.go Finding with Verified/VerifyStatus/VerifyMetadata VerifyStatus path provides contains pkg/storage/schema.sql findings columns verified, verify_status, verify_metadata_json verify_status path provides contains go.mod github.com/tidwall/gjson dependency tidwall/gjson from to via pattern pkg/storage/findings.go findings table INSERT/SELECT with new verify_* columns verify_status

Wave 0 foundation for the verification engine. Extend the provider VerifySpec schema, Finding struct, and storage schema with the fields every downstream plan needs. Add the gjson dependency. No runtime verifier logic yet — just contracts so Plans 05-02, 05-03, 05-04 can run in parallel on Wave 1.

Purpose: Interface-first — downstream plans build against these types without exploring the codebase. Output: Extended schema types, migrated SQLite schema, gjson dependency wired.

<execution_context> @$HOME/.claude/get-shit-done/workflows/execute-plan.md @$HOME/.claude/get-shit-done/templates/summary.md </execution_context>

@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @.planning/phases/05-verification-engine/05-CONTEXT.md

@pkg/providers/schema.go @pkg/engine/finding.go @pkg/storage/schema.sql @pkg/storage/findings.go @pkg/storage/db.go

From pkg/providers/schema.go (current):

type VerifySpec struct {
    Method        string            `yaml:"method"`
    URL           string            `yaml:"url"`
    Headers       map[string]string `yaml:"headers"`
    ValidStatus   []int             `yaml:"valid_status"`
    InvalidStatus []int             `yaml:"invalid_status"`
}

Note: existing YAMLs use valid_status / invalid_status. Keep those fields for backward compat AND add the new canonical fields.

From pkg/engine/finding.go (current):

type Finding struct {
    ProviderName string
    KeyValue     string
    KeyMasked    string
    Confidence   string
    Source       string
    SourceType   string
    LineNumber   int
    Offset       int64
    DetectedAt   time.Time
}

From pkg/storage/schema.sql findings table: columns id, scan_id, provider_name, key_value, key_masked, confidence, source_path, source_type, line_number, created_at.

Task 1: Extend VerifySpec, Finding, and add gjson dependency go.mod, go.sum, pkg/providers/schema.go, pkg/engine/finding.go, pkg/providers/schema_test.go - VerifySpec parses new YAML fields: success_codes, failure_codes, rate_limit_codes, metadata_paths, body - Backward compat: existing YAMLs with only valid_status/invalid_status still load (no error); VerifySpec exposes both old and new fields - Finding zero value has Verified=false, VerifyStatus="", VerifyMetadata=nil - gjson is importable: `import "github.com/tidwall/gjson"` compiles 1. Add gjson dep: ``` go get github.com/tidwall/gjson@latest ``` (Do NOT run `go mod tidy`; use `go mod download` if needed per Phase 4 lesson.)

2. Extend `pkg/providers/schema.go` VerifySpec struct to:
   ```go
   type VerifySpec struct {
       Method         string            `yaml:"method"`
       URL            string            `yaml:"url"`
       Headers        map[string]string `yaml:"headers"`
       Body           string            `yaml:"body"`
       // Canonical status code fields (Phase 5)
       SuccessCodes   []int             `yaml:"success_codes"`
       FailureCodes   []int             `yaml:"failure_codes"`
       RateLimitCodes []int             `yaml:"rate_limit_codes"`
       // MetadataPaths maps display-name -> gjson path (e.g. "org" -> "organization.name")
       MetadataPaths  map[string]string `yaml:"metadata_paths"`
       // Legacy fields kept for backward compat with existing YAMLs (Phase 2-3 providers)
       ValidStatus    []int             `yaml:"valid_status"`
       InvalidStatus  []int             `yaml:"invalid_status"`
   }
   ```

   Add a method `(v VerifySpec) EffectiveSuccessCodes() []int` that returns `SuccessCodes` if non-empty else `ValidStatus` else `[]int{200}`.
   Add `(v VerifySpec) EffectiveFailureCodes() []int` returning `FailureCodes` if non-empty else `InvalidStatus` else `[]int{401, 403}`.
   Add `(v VerifySpec) EffectiveRateLimitCodes() []int` returning `RateLimitCodes` if non-empty else `[]int{429}`.

3. Extend `pkg/engine/finding.go` Finding struct — add at bottom (preserve existing fields and MaskKey function):
   ```go
   // Verification fields populated when scan --verify is set (Phase 5).
   Verified       bool              // true if verifier ran against this finding
   VerifyStatus   string            // "live", "dead", "rate_limited", "error", "unknown"
   VerifyHTTPCode int               // HTTP status code returned by verify endpoint
   VerifyMetadata map[string]string // extracted metadata from response (org, tier, etc.)
   VerifyError    string            // non-nil if VerifyStatus == "error"
   ```

4. Add `pkg/providers/schema_test.go` with:
   - `TestVerifySpec_NewFieldsParse` — YAML with success_codes/failure_codes/rate_limit_codes/metadata_paths/body unmarshals correctly
   - `TestVerifySpec_LegacyFieldsStillWork` — YAML with only valid_status/invalid_status parses and `EffectiveSuccessCodes()` returns the legacy values
   - `TestVerifySpec_Defaults` — empty VerifySpec: `EffectiveSuccessCodes()==[200]`, `EffectiveFailureCodes()==[401,403]`, `EffectiveRateLimitCodes()==[429]`
   Use `yaml.Unmarshal` directly into a small wrapper struct `struct{ Verify VerifySpec \`yaml:"verify"\` }`.

cd /home/salva/Documents/apikey && go build ./... && go test ./pkg/providers/... -run VerifySpec -v - `grep -q 'SuccessCodes' pkg/providers/schema.go` - `grep -q 'MetadataPaths' pkg/providers/schema.go` - `grep -q 'VerifyStatus' pkg/engine/finding.go` - `grep -q 'tidwall/gjson' go.mod` - `go build ./...` succeeds - All three new test cases pass VerifySpec and Finding carry all Phase 5 fields, legacy YAMLs still load, gjson is available for Plan 05-03. Task 2: Migrate storage schema and persist verify fields pkg/storage/schema.sql, pkg/storage/findings.go, pkg/storage/findings_test.go - Fresh DB: findings table has verified INTEGER, verify_status TEXT, verify_metadata_json TEXT, verify_http_code INTEGER columns - Existing DB: ALTER TABLE adds the columns idempotently (using ADD COLUMN IF NOT EXISTS pattern or pragma table_info check) - SaveFinding persists verify_* fields when set; ListFindings round-trips them - Storing a Finding with nil VerifyMetadata results in NULL verify_metadata_json; storing a populated map round-trips via JSON 1. Update `pkg/storage/schema.sql`: Add four columns to `findings` table DDL: ```sql CREATE TABLE IF NOT EXISTS findings ( id INTEGER PRIMARY KEY AUTOINCREMENT, scan_id INTEGER REFERENCES scans(id), provider_name TEXT NOT NULL, key_value BLOB NOT NULL, key_masked TEXT NOT NULL, confidence TEXT NOT NULL, source_path TEXT, source_type TEXT, line_number INTEGER, verified INTEGER NOT NULL DEFAULT 0, verify_status TEXT NOT NULL DEFAULT '', verify_http_code INTEGER NOT NULL DEFAULT 0, verify_metadata_json TEXT, created_at DATETIME DEFAULT CURRENT_TIMESTAMP ); ```

2. In `pkg/storage/db.go` Open(), after the schema exec, run an idempotent migration for existing databases. SQLite < 3.35 does not support `ADD COLUMN IF NOT EXISTS`, so check existing columns via `PRAGMA table_info(findings)` and issue `ALTER TABLE findings ADD COLUMN ...` only if missing. Add this as a helper function `migrateFindingsVerifyColumns(sqlDB *sql.DB) error` and call it from Open(). Columns to add if missing: `verified INTEGER NOT NULL DEFAULT 0`, `verify_status TEXT NOT NULL DEFAULT ''`, `verify_http_code INTEGER NOT NULL DEFAULT 0`, `verify_metadata_json TEXT`.

3. Update `pkg/storage/findings.go`:
   - Extend `Finding` struct with: `Verified bool`, `VerifyStatus string`, `VerifyHTTPCode int`, `VerifyMetadata map[string]string`
   - Update `SaveFinding` INSERT to include the four new columns. Serialize `VerifyMetadata` via `encoding/json` when non-nil; pass `sql.NullString{}` when nil
   - Update `ListFindings` SELECT to read the new columns and populate the struct (decode JSON back into map)

4. Add `pkg/storage/findings_test.go` tests (use `:memory:` DB):
   - `TestSaveFinding_VerifyFields_RoundTrip` — save with Verified=true, VerifyStatus="live", VerifyHTTPCode=200, VerifyMetadata={"org":"Acme","tier":"plus"}, then ListFindings and assert all fields equal
   - `TestSaveFinding_VerifyFields_Empty` — save finding with no verify data, ListFindings returns Verified=false, empty status, nil metadata
   - `TestOpen_MigratesExistingDB` — create an old-schema findings table manually (without verify columns), close, reopen with storage.Open, assert the four columns now exist via PRAGMA table_info

Reference existing SaveFinding/ListFindings code shape in pkg/storage/findings.go — mirror the NULL-handling pattern used for scan_id.

cd /home/salva/Documents/apikey && go test ./pkg/storage/... -run Verify -v && go test ./pkg/storage/... -run Migrate -v - `grep -q 'verify_status' pkg/storage/schema.sql` - `grep -q 'verify_metadata_json' pkg/storage/findings.go` - `grep -q 'migrateFindingsVerifyColumns' pkg/storage/db.go` - `go test ./pkg/storage/... -v` all pass - `go build ./...` succeeds Storage persists verify_* fields for fresh and existing DBs; round-trip test green. - `go build ./...` clean - `go test ./pkg/providers/... ./pkg/storage/... -v` all pass - `grep -rn "SuccessCodes\|VerifyStatus\|verify_metadata_json" pkg/` confirms all three extensions landed

<success_criteria>

• Extended VerifySpec, extended Finding, migrated SQLite schema, gjson dep present
• Backward compatibility preserved (legacy YAMLs load, old DBs migrate)
• Unit tests green on all new fields </success_criteria>

After completion, create `.planning/phases/05-verification-engine/05-01-SUMMARY.md`