177888bfa8
Wave 0 contracts for the verification engine are in place: - VerifySpec extended with SuccessCodes/FailureCodes/RateLimitCodes/MetadataPaths/Body - Finding extended with Verified/VerifyStatus/VerifyHTTPCode/VerifyMetadata/VerifyError - findings table schema migrated with verify_* columns (fresh + legacy DBs) - gjson dep wired as direct require - VRFY-02, VRFY-03 marked complete
129 lines
7.2 KiB
Markdown
129 lines
7.2 KiB
Markdown
---
|
|
phase: 05-verification-engine
|
|
plan: 01
|
|
subsystem: verification-foundation
|
|
tags: [schema, storage, migration, verify, foundation]
|
|
requirements: [VRFY-02, VRFY-03]
|
|
dependency-graph:
|
|
requires:
|
|
- "pkg/providers/schema.go VerifySpec (Phase 2)"
|
|
- "pkg/engine/finding.go Finding (Phase 2)"
|
|
- "pkg/storage/findings.go (Phase 3)"
|
|
provides:
|
|
- "Extended VerifySpec contract for Plans 05-02/03/04"
|
|
- "Finding.Verify* fields for the verifier pipeline"
|
|
- "findings table verify_* columns + migration for existing DBs"
|
|
- "github.com/tidwall/gjson direct dependency"
|
|
affects:
|
|
- "go.mod / go.sum"
|
|
- "pkg/providers"
|
|
- "pkg/engine"
|
|
- "pkg/storage"
|
|
tech-stack:
|
|
added:
|
|
- "github.com/tidwall/gjson v1.18.0 (JSON path extraction)"
|
|
patterns:
|
|
- "PRAGMA table_info + conditional ALTER TABLE for idempotent SQLite migrations"
|
|
- "JSON-encoded TEXT column for map[string]string round-trip with NULL for nil"
|
|
- "Effective*() fallback helpers preserving legacy fields alongside canonical ones"
|
|
key-files:
|
|
created:
|
|
- "pkg/providers/schema_test.go"
|
|
- "pkg/storage/findings_test.go"
|
|
- ".planning/phases/05-verification-engine/05-01-SUMMARY.md"
|
|
modified:
|
|
- "go.mod"
|
|
- "go.sum"
|
|
- "pkg/providers/schema.go"
|
|
- "pkg/engine/finding.go"
|
|
- "pkg/storage/schema.sql"
|
|
- "pkg/storage/db.go"
|
|
- "pkg/storage/findings.go"
|
|
decisions:
|
|
- "Keep legacy ValidStatus/InvalidStatus fields AND add canonical SuccessCodes/FailureCodes; expose via Effective*() helpers so Phase 2-3 YAMLs still load."
|
|
- "Store VerifyMetadata as JSON TEXT (not a relational table) — small, read-together, and keeps the findings table self-contained."
|
|
- "Migrate existing DBs via PRAGMA table_info + ALTER TABLE ADD COLUMN (SQLite 3.35 ADD COLUMN IF NOT EXISTS is not guaranteed across modernc versions)."
|
|
- "Add ExtractMetadata skeleton in schema.go to make gjson a direct dependency now; Plan 05-03 will replace it with the full verifier extraction path."
|
|
metrics:
|
|
duration: "3m43s"
|
|
completed: "2026-04-05T12:42:56Z"
|
|
tasks_completed: 2
|
|
files_created: 3
|
|
files_modified: 7
|
|
---
|
|
|
|
# Phase 5 Plan 01: Verification Foundation Summary
|
|
|
|
Interface-first Wave 0 landing: extended VerifySpec, extended Finding, migrated SQLite schema, and wired gjson so Plans 05-02/03/04 can run in parallel on Wave 1 without touching each other's contracts.
|
|
|
|
## Tasks
|
|
|
|
### Task 1 — Extend VerifySpec, Finding, add gjson dependency
|
|
**Commits:** `499f5d5` (RED), `30c0e98` (GREEN)
|
|
|
|
- `VerifySpec` gains `Body`, `SuccessCodes`, `FailureCodes`, `RateLimitCodes`, `MetadataPaths`. `ValidStatus` / `InvalidStatus` remain for backward compat with Phase 2-3 provider YAMLs.
|
|
- `EffectiveSuccessCodes()` returns `SuccessCodes` → `ValidStatus` → `[200]`; `EffectiveFailureCodes()` returns `FailureCodes` → `InvalidStatus` → `[401, 403]`; `EffectiveRateLimitCodes()` returns `RateLimitCodes` → `[429]`.
|
|
- `ExtractMetadata([]byte) map[string]string` added as a minimal gjson-backed helper so the dep is promoted to direct in `go.mod`. Plan 05-03 will expand it.
|
|
- `engine.Finding` gains `Verified`, `VerifyStatus`, `VerifyHTTPCode`, `VerifyMetadata`, `VerifyError`.
|
|
- Tests: `TestVerifySpec_NewFieldsParse`, `TestVerifySpec_LegacyFieldsStillWork`, `TestVerifySpec_Defaults`, `TestVerifySpec_CanonicalFieldsPreferred` — all pass.
|
|
|
|
### Task 2 — Migrate storage schema and persist verify fields
|
|
**Commits:** `2654487` (RED), `aec559d` (GREEN)
|
|
|
|
- `pkg/storage/schema.sql`: findings table now carries `verified INTEGER NOT NULL DEFAULT 0`, `verify_status TEXT NOT NULL DEFAULT ''`, `verify_http_code INTEGER NOT NULL DEFAULT 0`, `verify_metadata_json TEXT`.
|
|
- `pkg/storage/db.go`: new `migrateFindingsVerifyColumns` helper called from `Open()` — walks `PRAGMA table_info(findings)`, issues `ALTER TABLE ADD COLUMN` for any missing verify column. Safe to run on fresh DBs (all columns exist, no-op) and on legacy DBs.
|
|
- `pkg/storage/findings.go`: `Finding` struct extended; `SaveFinding` serializes `VerifyMetadata` as JSON (NULL when nil) and writes `verified` as 0/1 INTEGER; `ListFindings` round-trips the same.
|
|
- Tests: `TestSaveFinding_VerifyFields_RoundTrip`, `TestSaveFinding_VerifyFields_Empty`, `TestOpen_MigratesExistingDB` — all pass.
|
|
|
|
## Verification
|
|
|
|
```
|
|
go build ./... # clean
|
|
go test ./pkg/providers/... -run VerifySpec -v # 4/4 pass
|
|
go test ./pkg/storage/... -v # 10/10 pass
|
|
go test ./... # all green
|
|
grep -q 'tidwall/gjson' go.mod # direct dep
|
|
grep -rn 'SuccessCodes|VerifyStatus|verify_metadata_json' pkg/ # extensions landed
|
|
```
|
|
|
|
## Deviations from Plan
|
|
|
|
**None material.** Two small additions made inline:
|
|
|
|
1. **Added `ExtractMetadata` helper on `VerifySpec`** (Rule 3 — blocking issue). The plan required gjson to be in `go.mod` as a usable dependency, but `go get` without an actual import in code leaves it marked `// indirect`. Rather than run `go mod tidy` (explicitly forbidden in the plan), a tiny `ExtractMetadata` method was added that uses `gjson.GetBytes`. This promotes gjson to a direct require and gives Plan 05-03 a real starting point. The helper is intentionally minimal; Plan 05-03 owns the final extraction logic.
|
|
|
|
2. **Added `TestVerifySpec_CanonicalFieldsPreferred`** (belt-and-braces test). Not in the plan's three required tests, but verifies that when both canonical and legacy fields are set, canonical wins — a contract every downstream plan depends on.
|
|
|
|
## Downstream Contracts Ready
|
|
|
|
| Consumer | Contract |
|
|
| ---------- | --------------------------------------------------------------------------------------------- |
|
|
| Plan 05-02 | `VerifySpec.Effective*Codes()`, `Finding.Verify*` fields for HTTPVerifier implementation |
|
|
| Plan 05-03 | `VerifySpec.MetadataPaths`, `ExtractMetadata` skeleton, gjson available |
|
|
| Plan 05-04 | `storage.Finding.Verified/VerifyStatus/VerifyMetadata` round-trip, migration is automatic |
|
|
| Plan 05-05 | YAML schema ready for Tier 1 provider verify-block enrichment |
|
|
|
|
## Known Stubs
|
|
|
|
- `VerifySpec.ExtractMetadata` is a minimal gjson extraction — only handles flat string values via `GetBytes`. **Plan 05-03 will replace this** with full metadata extraction (nested paths, type coercion, Content-Type gating).
|
|
|
|
## Self-Check: PASSED
|
|
|
|
Files verified:
|
|
- `pkg/providers/schema.go` — FOUND (SuccessCodes, MetadataPaths, ExtractMetadata, Effective* present)
|
|
- `pkg/providers/schema_test.go` — FOUND (4 tests)
|
|
- `pkg/engine/finding.go` — FOUND (Verified/VerifyStatus/VerifyHTTPCode/VerifyMetadata/VerifyError present)
|
|
- `pkg/storage/schema.sql` — FOUND (verify_metadata_json column present)
|
|
- `pkg/storage/db.go` — FOUND (migrateFindingsVerifyColumns present)
|
|
- `pkg/storage/findings.go` — FOUND (verify_* columns in SELECT/INSERT)
|
|
- `pkg/storage/findings_test.go` — FOUND (3 tests)
|
|
- `go.mod` — FOUND (`github.com/tidwall/gjson v1.18.0` in direct require block)
|
|
|
|
Commits verified in `git log`:
|
|
- `499f5d5` — test(05-01) VerifySpec RED
|
|
- `30c0e98` — feat(05-01) VerifySpec GREEN
|
|
- `2654487` — test(05-01) storage RED
|
|
- `aec559d` — feat(05-01) storage GREEN
|
|
|
|
Build + tests: `go build ./...` clean, `go test ./...` all pass.
|