---
phase: 05-verification-engine
plan: 01
type: execute
wave: 0
depends_on: []
files_modified:
- go.mod
- go.sum
- pkg/providers/schema.go
- pkg/engine/finding.go
- pkg/storage/schema.sql
- pkg/storage/findings.go
- pkg/storage/findings_test.go
autonomous: true
requirements: [VRFY-02, VRFY-03]
must_haves:
truths:
- "VerifySpec struct carries SuccessCodes, FailureCodes, RateLimitCodes, MetadataPaths, Body"
- "Finding struct carries Verified, VerifyStatus, VerifyMetadata fields"
- "findings table has verified/verify_status/verify_metadata_json columns and existing DBs migrate without data loss"
- "github.com/tidwall/gjson is in go.mod"
artifacts:
- path: "pkg/providers/schema.go"
provides: "Extended VerifySpec with SuccessCodes/FailureCodes/RateLimitCodes/MetadataPaths/Body"
contains: "SuccessCodes"
- path: "pkg/engine/finding.go"
provides: "Finding with Verified/VerifyStatus/VerifyMetadata"
contains: "VerifyStatus"
- path: "pkg/storage/schema.sql"
provides: "findings columns verified, verify_status, verify_metadata_json"
contains: "verify_status"
- path: "go.mod"
provides: "github.com/tidwall/gjson dependency"
contains: "tidwall/gjson"
key_links:
- from: "pkg/storage/findings.go"
to: "findings table"
via: "INSERT/SELECT with new verify_* columns"
pattern: "verify_status"
---
Wave 0 foundation for the verification engine. Extend the provider VerifySpec schema, Finding struct, and storage schema with the fields every downstream plan needs. Add the gjson dependency. No runtime verifier logic yet — just contracts so Plans 05-02, 05-03, 05-04 can run in parallel on Wave 1.
Purpose: Interface-first — downstream plans build against these types without exploring the codebase.
Output: Extended schema types, migrated SQLite schema, gjson dependency wired.
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/05-verification-engine/05-CONTEXT.md
@pkg/providers/schema.go
@pkg/engine/finding.go
@pkg/storage/schema.sql
@pkg/storage/findings.go
@pkg/storage/db.go
From pkg/providers/schema.go (current):
```go
type VerifySpec struct {
Method string `yaml:"method"`
URL string `yaml:"url"`
Headers map[string]string `yaml:"headers"`
ValidStatus []int `yaml:"valid_status"`
InvalidStatus []int `yaml:"invalid_status"`
}
```
Note: existing YAMLs use `valid_status` / `invalid_status`. Keep those fields for backward compat AND add the new canonical fields.
From pkg/engine/finding.go (current):
```go
type Finding struct {
ProviderName string
KeyValue string
KeyMasked string
Confidence string
Source string
SourceType string
LineNumber int
Offset int64
DetectedAt time.Time
}
```
From pkg/storage/schema.sql findings table: columns id, scan_id, provider_name, key_value, key_masked, confidence, source_path, source_type, line_number, created_at.
Task 1: Extend VerifySpec, Finding, and add gjson dependency
go.mod, go.sum, pkg/providers/schema.go, pkg/engine/finding.go, pkg/providers/schema_test.go
- VerifySpec parses new YAML fields: success_codes, failure_codes, rate_limit_codes, metadata_paths, body
- Backward compat: existing YAMLs with only valid_status/invalid_status still load (no error); VerifySpec exposes both old and new fields
- Finding zero value has Verified=false, VerifyStatus="", VerifyMetadata=nil
- gjson is importable: `import "github.com/tidwall/gjson"` compiles
1. Add gjson dep:
```
go get github.com/tidwall/gjson@latest
```
(Do NOT run `go mod tidy`; use `go mod download` if needed per Phase 4 lesson.)
2. Extend `pkg/providers/schema.go` VerifySpec struct to:
```go
type VerifySpec struct {
Method string `yaml:"method"`
URL string `yaml:"url"`
Headers map[string]string `yaml:"headers"`
Body string `yaml:"body"`
// Canonical status code fields (Phase 5)
SuccessCodes []int `yaml:"success_codes"`
FailureCodes []int `yaml:"failure_codes"`
RateLimitCodes []int `yaml:"rate_limit_codes"`
// MetadataPaths maps display-name -> gjson path (e.g. "org" -> "organization.name")
MetadataPaths map[string]string `yaml:"metadata_paths"`
// Legacy fields kept for backward compat with existing YAMLs (Phase 2-3 providers)
ValidStatus []int `yaml:"valid_status"`
InvalidStatus []int `yaml:"invalid_status"`
}
```
Add a method `(v VerifySpec) EffectiveSuccessCodes() []int` that returns `SuccessCodes` if non-empty else `ValidStatus` else `[]int{200}`.
Add `(v VerifySpec) EffectiveFailureCodes() []int` returning `FailureCodes` if non-empty else `InvalidStatus` else `[]int{401, 403}`.
Add `(v VerifySpec) EffectiveRateLimitCodes() []int` returning `RateLimitCodes` if non-empty else `[]int{429}`.
3. Extend `pkg/engine/finding.go` Finding struct — add at bottom (preserve existing fields and MaskKey function):
```go
// Verification fields populated when scan --verify is set (Phase 5).
Verified bool // true if verifier ran against this finding
VerifyStatus string // "live", "dead", "rate_limited", "error", "unknown"
VerifyHTTPCode int // HTTP status code returned by verify endpoint
VerifyMetadata map[string]string // extracted metadata from response (org, tier, etc.)
VerifyError string // non-nil if VerifyStatus == "error"
```
4. Add `pkg/providers/schema_test.go` with:
- `TestVerifySpec_NewFieldsParse` — YAML with success_codes/failure_codes/rate_limit_codes/metadata_paths/body unmarshals correctly
- `TestVerifySpec_LegacyFieldsStillWork` — YAML with only valid_status/invalid_status parses and `EffectiveSuccessCodes()` returns the legacy values
- `TestVerifySpec_Defaults` — empty VerifySpec: `EffectiveSuccessCodes()==[200]`, `EffectiveFailureCodes()==[401,403]`, `EffectiveRateLimitCodes()==[429]`
Use `yaml.Unmarshal` directly into a small wrapper struct `struct{ Verify VerifySpec \`yaml:"verify"\` }`.
cd /home/salva/Documents/apikey && go build ./... && go test ./pkg/providers/... -run VerifySpec -v
- `grep -q 'SuccessCodes' pkg/providers/schema.go`
- `grep -q 'MetadataPaths' pkg/providers/schema.go`
- `grep -q 'VerifyStatus' pkg/engine/finding.go`
- `grep -q 'tidwall/gjson' go.mod`
- `go build ./...` succeeds
- All three new test cases pass
VerifySpec and Finding carry all Phase 5 fields, legacy YAMLs still load, gjson is available for Plan 05-03.
Task 2: Migrate storage schema and persist verify fields
pkg/storage/schema.sql, pkg/storage/findings.go, pkg/storage/findings_test.go
- Fresh DB: findings table has verified INTEGER, verify_status TEXT, verify_metadata_json TEXT, verify_http_code INTEGER columns
- Existing DB: ALTER TABLE adds the columns idempotently (using ADD COLUMN IF NOT EXISTS pattern or pragma table_info check)
- SaveFinding persists verify_* fields when set; ListFindings round-trips them
- Storing a Finding with nil VerifyMetadata results in NULL verify_metadata_json; storing a populated map round-trips via JSON
1. Update `pkg/storage/schema.sql`: Add four columns to `findings` table DDL:
```sql
CREATE TABLE IF NOT EXISTS findings (
id INTEGER PRIMARY KEY AUTOINCREMENT,
scan_id INTEGER REFERENCES scans(id),
provider_name TEXT NOT NULL,
key_value BLOB NOT NULL,
key_masked TEXT NOT NULL,
confidence TEXT NOT NULL,
source_path TEXT,
source_type TEXT,
line_number INTEGER,
verified INTEGER NOT NULL DEFAULT 0,
verify_status TEXT NOT NULL DEFAULT '',
verify_http_code INTEGER NOT NULL DEFAULT 0,
verify_metadata_json TEXT,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
```
2. In `pkg/storage/db.go` Open(), after the schema exec, run an idempotent migration for existing databases. SQLite < 3.35 does not support `ADD COLUMN IF NOT EXISTS`, so check existing columns via `PRAGMA table_info(findings)` and issue `ALTER TABLE findings ADD COLUMN ...` only if missing. Add this as a helper function `migrateFindingsVerifyColumns(sqlDB *sql.DB) error` and call it from Open(). Columns to add if missing: `verified INTEGER NOT NULL DEFAULT 0`, `verify_status TEXT NOT NULL DEFAULT ''`, `verify_http_code INTEGER NOT NULL DEFAULT 0`, `verify_metadata_json TEXT`.
3. Update `pkg/storage/findings.go`:
- Extend `Finding` struct with: `Verified bool`, `VerifyStatus string`, `VerifyHTTPCode int`, `VerifyMetadata map[string]string`
- Update `SaveFinding` INSERT to include the four new columns. Serialize `VerifyMetadata` via `encoding/json` when non-nil; pass `sql.NullString{}` when nil
- Update `ListFindings` SELECT to read the new columns and populate the struct (decode JSON back into map)
4. Add `pkg/storage/findings_test.go` tests (use `:memory:` DB):
- `TestSaveFinding_VerifyFields_RoundTrip` — save with Verified=true, VerifyStatus="live", VerifyHTTPCode=200, VerifyMetadata={"org":"Acme","tier":"plus"}, then ListFindings and assert all fields equal
- `TestSaveFinding_VerifyFields_Empty` — save finding with no verify data, ListFindings returns Verified=false, empty status, nil metadata
- `TestOpen_MigratesExistingDB` — create an old-schema findings table manually (without verify columns), close, reopen with storage.Open, assert the four columns now exist via PRAGMA table_info
Reference existing SaveFinding/ListFindings code shape in pkg/storage/findings.go — mirror the NULL-handling pattern used for scan_id.
cd /home/salva/Documents/apikey && go test ./pkg/storage/... -run Verify -v && go test ./pkg/storage/... -run Migrate -v
- `grep -q 'verify_status' pkg/storage/schema.sql`
- `grep -q 'verify_metadata_json' pkg/storage/findings.go`
- `grep -q 'migrateFindingsVerifyColumns' pkg/storage/db.go`
- `go test ./pkg/storage/... -v` all pass
- `go build ./...` succeeds
Storage persists verify_* fields for fresh and existing DBs; round-trip test green.
- `go build ./...` clean
- `go test ./pkg/providers/... ./pkg/storage/... -v` all pass
- `grep -rn "SuccessCodes\|VerifyStatus\|verify_metadata_json" pkg/` confirms all three extensions landed
- Extended VerifySpec, extended Finding, migrated SQLite schema, gjson dep present
- Backward compatibility preserved (legacy YAMLs load, old DBs migrate)
- Unit tests green on all new fields