From 026d3050267daad0fa75fe8f460b763e1a8db7ff Mon Sep 17 00:00:00 2001 From: salvacybersec Date: Sun, 5 Apr 2026 15:48:15 +0300 Subject: [PATCH] docs(05-04): complete Tier 1 verify specs plan --- .planning/ROADMAP.md | 2 +- .planning/STATE.md | 13 +- .../05-verification-engine/05-04-SUMMARY.md | 156 ++++++++++++++++++ 3 files changed, 164 insertions(+), 7 deletions(-) create mode 100644 .planning/phases/05-verification-engine/05-04-SUMMARY.md diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 4960e5d..2ef669c 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -126,7 +126,7 @@ Plans: - [x] 05-01-PLAN.md — Wave 0: extend VerifySpec schema, Finding struct, storage schema; add gjson dep - [ ] 05-02-PLAN.md — LEGAL.md + pkg/legal embed + consent prompt + keyhunter legal command - [ ] 05-03-PLAN.md — pkg/verify HTTPVerifier: template sub, gjson metadata extraction, ants pool -- [ ] 05-04-PLAN.md — Update 12 Tier 1 provider YAMLs with extended verify specs + guardrail test +- [x] 05-04-PLAN.md — Update 12 Tier 1 provider YAMLs with extended verify specs + guardrail test - [ ] 05-05-PLAN.md — cmd/scan.go --verify wiring + --verify-timeout/workers flags + output verify column ### Phase 6: Output, Reporting & Key Management diff --git a/.planning/STATE.md b/.planning/STATE.md index 96c5e7a..bbda817 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -3,14 +3,14 @@ gsd_state_version: 1.0 milestone: v1.0 milestone_name: milestone status: executing -stopped_at: Completed 05-01-PLAN.md -last_updated: "2026-04-05T12:44:11.076Z" +stopped_at: Completed 05-04-PLAN.md +last_updated: "2026-04-05T12:48:10.887Z" last_activity: 2026-04-05 progress: total_phases: 18 completed_phases: 4 total_plans: 28 - completed_plans: 24 + completed_plans: 25 percent: 20 --- @@ -26,7 +26,7 @@ See: .planning/PROJECT.md (updated 2026-04-04) ## Current Position Phase: 05 (verification-engine) — EXECUTING -Plan: 2 of 5 +Plan: 3 of 5 Status: Ready to execute Last activity: 2026-04-05 @@ -70,6 +70,7 @@ Progress: [██░░░░░░░░] 20% | Phase 04 P02 | 4min | 1 tasks | 3 files | | Phase 04 P05 | 3min | 1 tasks | 2 files | | Phase 05 P01 | 3m43s | 2 tasks | 10 files | +| Phase 05 P04 | 10m | 2 tasks | 25 files | ## Accumulated Context @@ -108,6 +109,6 @@ None yet. ## Session Continuity -Last session: 2026-04-05T12:44:04.063Z -Stopped at: Completed 05-01-PLAN.md +Last session: 2026-04-05T12:48:10.879Z +Stopped at: Completed 05-04-PLAN.md Resume file: None diff --git a/.planning/phases/05-verification-engine/05-04-SUMMARY.md b/.planning/phases/05-verification-engine/05-04-SUMMARY.md new file mode 100644 index 0000000..cb59da3 --- /dev/null +++ b/.planning/phases/05-verification-engine/05-04-SUMMARY.md @@ -0,0 +1,156 @@ +--- +phase: 05-verification-engine +plan: 04 +subsystem: providers +tags: [providers, verification, yaml, guardrail] +one-liner: "Extended 12 Tier 1 provider YAMLs with full verify specs (success_codes, metadata_paths, {{KEY}} templates) and guardrail tests enforcing completeness." +requires: + - VerifySpec extended fields from 05-01 (success_codes, failure_codes, rate_limit_codes, metadata_paths) +provides: + - Tier 1 providers ready for HTTPVerifier consumption (Plan 05-03) + - Guardrail tests blocking regressions on Tier 1 verify specs +affects: + - pkg/verify HTTPVerifier will find well-formed specs for all Tier 1 providers +tech-stack: + added: [] + patterns: + - "{{KEY}} double-brace template substitution (canonical for Phase 5)" + - "Dual-location YAML mirror: providers/ + pkg/providers/definitions/" +key-files: + created: [] + modified: + - providers/openai.yaml + - providers/anthropic.yaml + - providers/google-ai.yaml + - providers/cohere.yaml + - providers/mistral.yaml + - providers/groq.yaml + - providers/xai.yaml + - providers/ai21.yaml + - providers/inflection.yaml + - providers/perplexity.yaml + - providers/deepseek.yaml + - providers/together.yaml + - pkg/providers/definitions/openai.yaml + - pkg/providers/definitions/anthropic.yaml + - pkg/providers/definitions/google-ai.yaml + - pkg/providers/definitions/cohere.yaml + - pkg/providers/definitions/mistral.yaml + - pkg/providers/definitions/groq.yaml + - pkg/providers/definitions/xai.yaml + - pkg/providers/definitions/ai21.yaml + - pkg/providers/definitions/inflection.yaml + - pkg/providers/definitions/perplexity.yaml + - pkg/providers/definitions/deepseek.yaml + - pkg/providers/definitions/together.yaml + - pkg/providers/registry_test.go +decisions: + - "Followed the plan's explicit task list (openai, anthropic, google-ai, cohere, mistral, groq, xai, ai21, inflection, perplexity, deepseek, together) rather than the brief objective mentioning vertex-ai/aws-bedrock/azure-openai/meta-ai — the block is authoritative and these providers either lack a public verify endpoint usable without region/tenant-specific config (Vertex, Bedrock, Azure, Meta)." + - "Anthropic uses POST /v1/messages with a minimal 1-token body rather than an unauthenticated GET — Anthropic has no public GET endpoint that cleanly differentiates authenticated vs unauthenticated keys. 529 added to rate_limit_codes (Anthropic overload)." + - "Inflection verify URL left empty — plan specifies no public endpoint; HTTPVerifier will return StatusUnknown." + - "Google AI failure_codes include 400 because the API returns 400 for bad API keys rather than 401." +metrics: + tasks-completed: 2 + files-modified: 25 + commits: 2 + duration: "~10m" + completed: "2026-04-05" +--- + +# Phase 05 Plan 04: Tier 1 Provider Verify Specs Summary + +## Objective Recap + +Update the 12 Tier 1 provider YAMLs so each carries a complete Phase 5 verify spec (success_codes, failure_codes, rate_limit_codes, metadata_paths, `{{KEY}}` header/URL/body substitution), with dual-location sync between `providers/` and `pkg/providers/definitions/`. Add a guardrail test that blocks future regressions. + +## What Was Built + +### Task 1: Provider YAML Updates + +Rewrote 12 provider YAMLs in both locations (24 files total) to adopt the canonical Phase 5 verify schema. + +| Provider | Method | Endpoint | metadata_paths | +| ----------- | ------ | -------------------------------------------------------- | -------------- | +| openai | GET | api.openai.com/v1/models | yes (2 fields) | +| anthropic | POST | api.anthropic.com/v1/messages (1-token body) | yes | +| google-ai | GET | generativelanguage.googleapis.com/v1/models?key={{KEY}} | yes | +| cohere | GET | api.cohere.ai/v1/models | yes | +| mistral | GET | api.mistral.ai/v1/models | yes | +| groq | GET | api.groq.com/openai/v1/models | yes | +| xai | GET | api.x.ai/v1/api-key | yes (2 fields) | +| ai21 | GET | api.ai21.com/studio/v1/models | no | +| inflection | GET | "" (no public endpoint) | no | +| perplexity | POST | api.perplexity.ai/chat/completions (sonar, 1 token) | no | +| deepseek | GET | api.deepseek.com/v1/models | yes | +| together | GET | api.together.xyz/v1/models | yes | + +Key design points: +- All templates use `{{KEY}}` (double brace), replacing legacy `{KEY}` single-brace form. +- `success_codes: [200]`, `failure_codes: [401, 403]`, `rate_limit_codes: [429]` is the baseline, with per-provider overrides (google-ai adds 400 to failure; anthropic adds 529 to rate_limit). +- Legacy `valid_status`/`invalid_status` fields removed from these 12; schema back-compat in pkg/providers/schema.go still supports other tiers that keep them. +- `diff providers/X.yaml pkg/providers/definitions/X.yaml` returns no mismatches for all 12 files. + +**Commit:** `f3ae8f0` — feat(05-04): extend Tier 1 provider verify specs + +### Task 2: Guardrail Tests + +Added two tests to `pkg/providers/registry_test.go`: + +1. `TestTier1VerifySpecs_Complete` — iterates the 11 Tier 1 providers that must have verify endpoints, asserts each has a non-empty HTTPS URL and non-empty `EffectiveSuccessCodes()`. +2. `TestInflection_NoVerifyEndpoint` — locks in the intentional empty-URL state for inflection so a well-meaning future edit cannot silently add a bogus endpoint. + +**Commit:** `6a94ce5` — test(05-04): guardrail tests for Tier 1 verify spec completeness + +## Verification Results + +``` +go test ./pkg/providers/... -v +... +--- PASS: TestTier1VerifySpecs_Complete (0.04s) +--- PASS: TestInflection_NoVerifyEndpoint (0.04s) +PASS +ok github.com/salvacybersec/keyhunter/pkg/providers 0.931s +``` + +Acceptance criteria audit: +- `grep -l '{{KEY}}' providers/*.yaml | wc -l` → 11 (inflection excluded, as specified) +- `grep -l 'success_codes:' providers/*.yaml | wc -l` → 12 +- `grep -l 'metadata_paths:' providers/*.yaml | wc -l` → 9 (≥8 required) +- `diff` between both locations returns 0 mismatches for all 12 providers +- `go test ./pkg/providers/...` passes (all 9 tests, including the two new guardrails) +- `go build ./...` succeeds + +## Deviations from Plan + +None — plan executed exactly as written. The plan's `` block was taken as authoritative over the phase-level objective prose that briefly referenced vertex-ai/aws-bedrock/azure-openai/meta-ai; see Decisions for rationale. + +## Authentication Gates + +None. + +## Known Stubs + +None. Inflection's empty verify URL is intentional and documented via `TestInflection_NoVerifyEndpoint`. + +## Deferred Issues + +None. + +## Self-Check: PASSED + +- providers/openai.yaml: FOUND +- providers/anthropic.yaml: FOUND +- providers/google-ai.yaml: FOUND +- providers/cohere.yaml: FOUND +- providers/mistral.yaml: FOUND +- providers/groq.yaml: FOUND +- providers/xai.yaml: FOUND +- providers/ai21.yaml: FOUND +- providers/inflection.yaml: FOUND +- providers/perplexity.yaml: FOUND +- providers/deepseek.yaml: FOUND +- providers/together.yaml: FOUND +- pkg/providers/definitions/ mirrors: FOUND (all 12) +- pkg/providers/registry_test.go: FOUND (updated) +- commit f3ae8f0: FOUND +- commit 6a94ce5: FOUND