157 lines
7.4 KiB
Markdown
157 lines
7.4 KiB
Markdown
---
|
|
phase: 05-verification-engine
|
|
plan: 04
|
|
subsystem: providers
|
|
tags: [providers, verification, yaml, guardrail]
|
|
one-liner: "Extended 12 Tier 1 provider YAMLs with full verify specs (success_codes, metadata_paths, {{KEY}} templates) and guardrail tests enforcing completeness."
|
|
requires:
|
|
- VerifySpec extended fields from 05-01 (success_codes, failure_codes, rate_limit_codes, metadata_paths)
|
|
provides:
|
|
- Tier 1 providers ready for HTTPVerifier consumption (Plan 05-03)
|
|
- Guardrail tests blocking regressions on Tier 1 verify specs
|
|
affects:
|
|
- pkg/verify HTTPVerifier will find well-formed specs for all Tier 1 providers
|
|
tech-stack:
|
|
added: []
|
|
patterns:
|
|
- "{{KEY}} double-brace template substitution (canonical for Phase 5)"
|
|
- "Dual-location YAML mirror: providers/ + pkg/providers/definitions/"
|
|
key-files:
|
|
created: []
|
|
modified:
|
|
- providers/openai.yaml
|
|
- providers/anthropic.yaml
|
|
- providers/google-ai.yaml
|
|
- providers/cohere.yaml
|
|
- providers/mistral.yaml
|
|
- providers/groq.yaml
|
|
- providers/xai.yaml
|
|
- providers/ai21.yaml
|
|
- providers/inflection.yaml
|
|
- providers/perplexity.yaml
|
|
- providers/deepseek.yaml
|
|
- providers/together.yaml
|
|
- pkg/providers/definitions/openai.yaml
|
|
- pkg/providers/definitions/anthropic.yaml
|
|
- pkg/providers/definitions/google-ai.yaml
|
|
- pkg/providers/definitions/cohere.yaml
|
|
- pkg/providers/definitions/mistral.yaml
|
|
- pkg/providers/definitions/groq.yaml
|
|
- pkg/providers/definitions/xai.yaml
|
|
- pkg/providers/definitions/ai21.yaml
|
|
- pkg/providers/definitions/inflection.yaml
|
|
- pkg/providers/definitions/perplexity.yaml
|
|
- pkg/providers/definitions/deepseek.yaml
|
|
- pkg/providers/definitions/together.yaml
|
|
- pkg/providers/registry_test.go
|
|
decisions:
|
|
- "Followed the plan's explicit task list (openai, anthropic, google-ai, cohere, mistral, groq, xai, ai21, inflection, perplexity, deepseek, together) rather than the brief objective mentioning vertex-ai/aws-bedrock/azure-openai/meta-ai — the <tasks> block is authoritative and these providers either lack a public verify endpoint usable without region/tenant-specific config (Vertex, Bedrock, Azure, Meta)."
|
|
- "Anthropic uses POST /v1/messages with a minimal 1-token body rather than an unauthenticated GET — Anthropic has no public GET endpoint that cleanly differentiates authenticated vs unauthenticated keys. 529 added to rate_limit_codes (Anthropic overload)."
|
|
- "Inflection verify URL left empty — plan specifies no public endpoint; HTTPVerifier will return StatusUnknown."
|
|
- "Google AI failure_codes include 400 because the API returns 400 for bad API keys rather than 401."
|
|
metrics:
|
|
tasks-completed: 2
|
|
files-modified: 25
|
|
commits: 2
|
|
duration: "~10m"
|
|
completed: "2026-04-05"
|
|
---
|
|
|
|
# Phase 05 Plan 04: Tier 1 Provider Verify Specs Summary
|
|
|
|
## Objective Recap
|
|
|
|
Update the 12 Tier 1 provider YAMLs so each carries a complete Phase 5 verify spec (success_codes, failure_codes, rate_limit_codes, metadata_paths, `{{KEY}}` header/URL/body substitution), with dual-location sync between `providers/` and `pkg/providers/definitions/`. Add a guardrail test that blocks future regressions.
|
|
|
|
## What Was Built
|
|
|
|
### Task 1: Provider YAML Updates
|
|
|
|
Rewrote 12 provider YAMLs in both locations (24 files total) to adopt the canonical Phase 5 verify schema.
|
|
|
|
| Provider | Method | Endpoint | metadata_paths |
|
|
| ----------- | ------ | -------------------------------------------------------- | -------------- |
|
|
| openai | GET | api.openai.com/v1/models | yes (2 fields) |
|
|
| anthropic | POST | api.anthropic.com/v1/messages (1-token body) | yes |
|
|
| google-ai | GET | generativelanguage.googleapis.com/v1/models?key={{KEY}} | yes |
|
|
| cohere | GET | api.cohere.ai/v1/models | yes |
|
|
| mistral | GET | api.mistral.ai/v1/models | yes |
|
|
| groq | GET | api.groq.com/openai/v1/models | yes |
|
|
| xai | GET | api.x.ai/v1/api-key | yes (2 fields) |
|
|
| ai21 | GET | api.ai21.com/studio/v1/models | no |
|
|
| inflection | GET | "" (no public endpoint) | no |
|
|
| perplexity | POST | api.perplexity.ai/chat/completions (sonar, 1 token) | no |
|
|
| deepseek | GET | api.deepseek.com/v1/models | yes |
|
|
| together | GET | api.together.xyz/v1/models | yes |
|
|
|
|
Key design points:
|
|
- All templates use `{{KEY}}` (double brace), replacing legacy `{KEY}` single-brace form.
|
|
- `success_codes: [200]`, `failure_codes: [401, 403]`, `rate_limit_codes: [429]` is the baseline, with per-provider overrides (google-ai adds 400 to failure; anthropic adds 529 to rate_limit).
|
|
- Legacy `valid_status`/`invalid_status` fields removed from these 12; schema back-compat in pkg/providers/schema.go still supports other tiers that keep them.
|
|
- `diff providers/X.yaml pkg/providers/definitions/X.yaml` returns no mismatches for all 12 files.
|
|
|
|
**Commit:** `f3ae8f0` — feat(05-04): extend Tier 1 provider verify specs
|
|
|
|
### Task 2: Guardrail Tests
|
|
|
|
Added two tests to `pkg/providers/registry_test.go`:
|
|
|
|
1. `TestTier1VerifySpecs_Complete` — iterates the 11 Tier 1 providers that must have verify endpoints, asserts each has a non-empty HTTPS URL and non-empty `EffectiveSuccessCodes()`.
|
|
2. `TestInflection_NoVerifyEndpoint` — locks in the intentional empty-URL state for inflection so a well-meaning future edit cannot silently add a bogus endpoint.
|
|
|
|
**Commit:** `6a94ce5` — test(05-04): guardrail tests for Tier 1 verify spec completeness
|
|
|
|
## Verification Results
|
|
|
|
```
|
|
go test ./pkg/providers/... -v
|
|
...
|
|
--- PASS: TestTier1VerifySpecs_Complete (0.04s)
|
|
--- PASS: TestInflection_NoVerifyEndpoint (0.04s)
|
|
PASS
|
|
ok github.com/salvacybersec/keyhunter/pkg/providers 0.931s
|
|
```
|
|
|
|
Acceptance criteria audit:
|
|
- `grep -l '{{KEY}}' providers/*.yaml | wc -l` → 11 (inflection excluded, as specified)
|
|
- `grep -l 'success_codes:' providers/*.yaml | wc -l` → 12
|
|
- `grep -l 'metadata_paths:' providers/*.yaml | wc -l` → 9 (≥8 required)
|
|
- `diff` between both locations returns 0 mismatches for all 12 providers
|
|
- `go test ./pkg/providers/...` passes (all 9 tests, including the two new guardrails)
|
|
- `go build ./...` succeeds
|
|
|
|
## Deviations from Plan
|
|
|
|
None — plan executed exactly as written. The plan's `<tasks>` block was taken as authoritative over the phase-level objective prose that briefly referenced vertex-ai/aws-bedrock/azure-openai/meta-ai; see Decisions for rationale.
|
|
|
|
## Authentication Gates
|
|
|
|
None.
|
|
|
|
## Known Stubs
|
|
|
|
None. Inflection's empty verify URL is intentional and documented via `TestInflection_NoVerifyEndpoint`.
|
|
|
|
## Deferred Issues
|
|
|
|
None.
|
|
|
|
## Self-Check: PASSED
|
|
|
|
- providers/openai.yaml: FOUND
|
|
- providers/anthropic.yaml: FOUND
|
|
- providers/google-ai.yaml: FOUND
|
|
- providers/cohere.yaml: FOUND
|
|
- providers/mistral.yaml: FOUND
|
|
- providers/groq.yaml: FOUND
|
|
- providers/xai.yaml: FOUND
|
|
- providers/ai21.yaml: FOUND
|
|
- providers/inflection.yaml: FOUND
|
|
- providers/perplexity.yaml: FOUND
|
|
- providers/deepseek.yaml: FOUND
|
|
- providers/together.yaml: FOUND
|
|
- pkg/providers/definitions/ mirrors: FOUND (all 12)
|
|
- pkg/providers/registry_test.go: FOUND (updated)
|
|
- commit f3ae8f0: FOUND
|
|
- commit 6a94ce5: FOUND
|