--- phase: 05-verification-engine plan: 04 subsystem: providers tags: [providers, verification, yaml, guardrail] one-liner: "Extended 12 Tier 1 provider YAMLs with full verify specs (success_codes, metadata_paths, {{KEY}} templates) and guardrail tests enforcing completeness." requires: - VerifySpec extended fields from 05-01 (success_codes, failure_codes, rate_limit_codes, metadata_paths) provides: - Tier 1 providers ready for HTTPVerifier consumption (Plan 05-03) - Guardrail tests blocking regressions on Tier 1 verify specs affects: - pkg/verify HTTPVerifier will find well-formed specs for all Tier 1 providers tech-stack: added: [] patterns: - "{{KEY}} double-brace template substitution (canonical for Phase 5)" - "Dual-location YAML mirror: providers/ + pkg/providers/definitions/" key-files: created: [] modified: - providers/openai.yaml - providers/anthropic.yaml - providers/google-ai.yaml - providers/cohere.yaml - providers/mistral.yaml - providers/groq.yaml - providers/xai.yaml - providers/ai21.yaml - providers/inflection.yaml - providers/perplexity.yaml - providers/deepseek.yaml - providers/together.yaml - pkg/providers/definitions/openai.yaml - pkg/providers/definitions/anthropic.yaml - pkg/providers/definitions/google-ai.yaml - pkg/providers/definitions/cohere.yaml - pkg/providers/definitions/mistral.yaml - pkg/providers/definitions/groq.yaml - pkg/providers/definitions/xai.yaml - pkg/providers/definitions/ai21.yaml - pkg/providers/definitions/inflection.yaml - pkg/providers/definitions/perplexity.yaml - pkg/providers/definitions/deepseek.yaml - pkg/providers/definitions/together.yaml - pkg/providers/registry_test.go decisions: - "Followed the plan's explicit task list (openai, anthropic, google-ai, cohere, mistral, groq, xai, ai21, inflection, perplexity, deepseek, together) rather than the brief objective mentioning vertex-ai/aws-bedrock/azure-openai/meta-ai — the block is authoritative and these providers either lack a public verify endpoint usable without region/tenant-specific config (Vertex, Bedrock, Azure, Meta)." - "Anthropic uses POST /v1/messages with a minimal 1-token body rather than an unauthenticated GET — Anthropic has no public GET endpoint that cleanly differentiates authenticated vs unauthenticated keys. 529 added to rate_limit_codes (Anthropic overload)." - "Inflection verify URL left empty — plan specifies no public endpoint; HTTPVerifier will return StatusUnknown." - "Google AI failure_codes include 400 because the API returns 400 for bad API keys rather than 401." metrics: tasks-completed: 2 files-modified: 25 commits: 2 duration: "~10m" completed: "2026-04-05" --- # Phase 05 Plan 04: Tier 1 Provider Verify Specs Summary ## Objective Recap Update the 12 Tier 1 provider YAMLs so each carries a complete Phase 5 verify spec (success_codes, failure_codes, rate_limit_codes, metadata_paths, `{{KEY}}` header/URL/body substitution), with dual-location sync between `providers/` and `pkg/providers/definitions/`. Add a guardrail test that blocks future regressions. ## What Was Built ### Task 1: Provider YAML Updates Rewrote 12 provider YAMLs in both locations (24 files total) to adopt the canonical Phase 5 verify schema. | Provider | Method | Endpoint | metadata_paths | | ----------- | ------ | -------------------------------------------------------- | -------------- | | openai | GET | api.openai.com/v1/models | yes (2 fields) | | anthropic | POST | api.anthropic.com/v1/messages (1-token body) | yes | | google-ai | GET | generativelanguage.googleapis.com/v1/models?key={{KEY}} | yes | | cohere | GET | api.cohere.ai/v1/models | yes | | mistral | GET | api.mistral.ai/v1/models | yes | | groq | GET | api.groq.com/openai/v1/models | yes | | xai | GET | api.x.ai/v1/api-key | yes (2 fields) | | ai21 | GET | api.ai21.com/studio/v1/models | no | | inflection | GET | "" (no public endpoint) | no | | perplexity | POST | api.perplexity.ai/chat/completions (sonar, 1 token) | no | | deepseek | GET | api.deepseek.com/v1/models | yes | | together | GET | api.together.xyz/v1/models | yes | Key design points: - All templates use `{{KEY}}` (double brace), replacing legacy `{KEY}` single-brace form. - `success_codes: [200]`, `failure_codes: [401, 403]`, `rate_limit_codes: [429]` is the baseline, with per-provider overrides (google-ai adds 400 to failure; anthropic adds 529 to rate_limit). - Legacy `valid_status`/`invalid_status` fields removed from these 12; schema back-compat in pkg/providers/schema.go still supports other tiers that keep them. - `diff providers/X.yaml pkg/providers/definitions/X.yaml` returns no mismatches for all 12 files. **Commit:** `f3ae8f0` — feat(05-04): extend Tier 1 provider verify specs ### Task 2: Guardrail Tests Added two tests to `pkg/providers/registry_test.go`: 1. `TestTier1VerifySpecs_Complete` — iterates the 11 Tier 1 providers that must have verify endpoints, asserts each has a non-empty HTTPS URL and non-empty `EffectiveSuccessCodes()`. 2. `TestInflection_NoVerifyEndpoint` — locks in the intentional empty-URL state for inflection so a well-meaning future edit cannot silently add a bogus endpoint. **Commit:** `6a94ce5` — test(05-04): guardrail tests for Tier 1 verify spec completeness ## Verification Results ``` go test ./pkg/providers/... -v ... --- PASS: TestTier1VerifySpecs_Complete (0.04s) --- PASS: TestInflection_NoVerifyEndpoint (0.04s) PASS ok github.com/salvacybersec/keyhunter/pkg/providers 0.931s ``` Acceptance criteria audit: - `grep -l '{{KEY}}' providers/*.yaml | wc -l` → 11 (inflection excluded, as specified) - `grep -l 'success_codes:' providers/*.yaml | wc -l` → 12 - `grep -l 'metadata_paths:' providers/*.yaml | wc -l` → 9 (≥8 required) - `diff` between both locations returns 0 mismatches for all 12 providers - `go test ./pkg/providers/...` passes (all 9 tests, including the two new guardrails) - `go build ./...` succeeds ## Deviations from Plan None — plan executed exactly as written. The plan's `` block was taken as authoritative over the phase-level objective prose that briefly referenced vertex-ai/aws-bedrock/azure-openai/meta-ai; see Decisions for rationale. ## Authentication Gates None. ## Known Stubs None. Inflection's empty verify URL is intentional and documented via `TestInflection_NoVerifyEndpoint`. ## Deferred Issues None. ## Self-Check: PASSED - providers/openai.yaml: FOUND - providers/anthropic.yaml: FOUND - providers/google-ai.yaml: FOUND - providers/cohere.yaml: FOUND - providers/mistral.yaml: FOUND - providers/groq.yaml: FOUND - providers/xai.yaml: FOUND - providers/ai21.yaml: FOUND - providers/inflection.yaml: FOUND - providers/perplexity.yaml: FOUND - providers/deepseek.yaml: FOUND - providers/together.yaml: FOUND - pkg/providers/definitions/ mirrors: FOUND (all 12) - pkg/providers/registry_test.go: FOUND (updated) - commit f3ae8f0: FOUND - commit 6a94ce5: FOUND