keyhunter/.planning/phases/05-verification-engine/05-04-PLAN.md

phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, must_haves

phase	plan	type	wave	depends_on	files_modified	autonomous	requirements	must_haves
05-verification-engine	04	execute	1	05-01	providers/openai.yaml providers/anthropic.yaml providers/google-ai.yaml providers/cohere.yaml providers/mistral.yaml providers/groq.yaml providers/xai.yaml providers/ai21.yaml providers/inflection.yaml providers/perplexity.yaml providers/deepseek.yaml providers/together.yaml pkg/providers/definitions/openai.yaml pkg/providers/definitions/anthropic.yaml pkg/providers/definitions/google-ai.yaml pkg/providers/definitions/cohere.yaml pkg/providers/definitions/mistral.yaml pkg/providers/definitions/groq.yaml pkg/providers/definitions/xai.yaml pkg/providers/definitions/ai21.yaml pkg/providers/definitions/inflection.yaml pkg/providers/definitions/perplexity.yaml pkg/providers/definitions/deepseek.yaml pkg/providers/definitions/together.yaml pkg/providers/registry_test.go	true	VRFY-02 VRFY-03	truths artifacts key_links All 12 Tier 1 provider YAMLs include success_codes, failure_codes, rate_limit_codes fields (extended from legacy valid_status/invalid_status) {{KEY}} template is used in verify.headers (Bearer token) or verify.url (query key) Providers with known metadata endpoints include metadata_paths mapping Dual-location sync: providers/ and pkg/providers/definitions/ kept identical All YAMLs still load via providers.NewRegistry() with no parse errors path provides contains providers/openai.yaml OpenAI verify spec with success_codes, {{KEY}} header substitution {{KEY}} from to via pattern providers/*.yaml pkg/providers/definitions/*.yaml dual-location mirror format_version

Update Tier 1 provider YAMLs so each carries a complete verify spec usable by the new HTTPVerifier: `{{KEY}}` template in headers or URL, `success_codes`, `failure_codes`, `rate_limit_codes`, and `metadata_paths` where the provider API returns useful metadata. Must maintain the dual-location sync between `providers/` (user-visible) and `pkg/providers/definitions/` (embed).

Purpose: VRFY-03 requires that provider YAMLs carry verification metadata. Without this update the verifier from Plan 05-03 would have no endpoints to hit. Output: 12 Tier 1 provider YAMLs updated in both locations; guardrail test asserts presence of new fields.

<execution_context> @$HOME/.claude/get-shit-done/workflows/execute-plan.md @$HOME/.claude/get-shit-done/templates/summary.md </execution_context>

@.planning/phases/05-verification-engine/05-CONTEXT.md @providers/openai.yaml @pkg/providers/schema.go Extended VerifySpec (from Plan 05-01) accepts these YAML keys under `verify:`: ```yaml verify: method: GET url: https://api.provider.com/v1/models headers: Authorization: "Bearer {{KEY}}" body: "" # optional, can use {{KEY}} success_codes: [200] failure_codes: [401, 403] rate_limit_codes: [429] metadata_paths: # display-name -> gjson path org: "organization.name" tier: "rate_limit.tier" ```

Legacy fields valid_status/invalid_status still parse (backward compat) but new YAMLs should use the canonical success_codes/failure_codes.

Dual-location rule (from Phase 1 decisions): every YAML in providers/ must have an identical copy in pkg/providers/definitions/ because go:embed cannot traverse ...

Task 1: Update 12 Tier 1 provider YAMLs with extended verify specs (both locations) providers/openai.yaml, providers/anthropic.yaml, providers/google-ai.yaml, providers/cohere.yaml, providers/mistral.yaml, providers/groq.yaml, providers/xai.yaml, providers/ai21.yaml, providers/inflection.yaml, providers/perplexity.yaml, providers/deepseek.yaml, providers/together.yaml, and mirrors in pkg/providers/definitions/ For each of the 12 providers below, update BOTH `providers/{name}.yaml` AND `pkg/providers/definitions/{name}.yaml` to have an identical `verify:` block as specified. Preserve existing `format_version`, `name`, `display_name`, `tier`, `last_verified`, `keywords`, and `patterns` — only touch the `verify:` block.

First, check each file exists in both locations — if a provider file is named differently in `pkg/providers/definitions/` than in `providers/`, match the `providers/` location's naming. If a file is missing from `pkg/providers/definitions/`, add it as a copy.

Use `{{KEY}}` (double brace) as the template marker. Set `last_verified: "2026-04-05"` on every updated file.

**1. openai** — `Bearer {{KEY}}` header, `GET /v1/models`
```yaml
verify:
  method: GET
  url: https://api.openai.com/v1/models
  headers:
    Authorization: "Bearer {{KEY}}"
  success_codes: [200]
  failure_codes: [401, 403]
  rate_limit_codes: [429]
  metadata_paths:
    first_model: "data.0.id"
    object_type: "object"
```

**2. anthropic** — POST /v1/messages with minimal body; Anthropic requires `x-api-key` header and `anthropic-version`
```yaml
verify:
  method: POST
  url: https://api.anthropic.com/v1/messages
  headers:
    x-api-key: "{{KEY}}"
    anthropic-version: "2023-06-01"
    content-type: "application/json"
  body: '{"model":"claude-haiku-4-5","max_tokens":1,"messages":[{"role":"user","content":"hi"}]}'
  success_codes: [200]
  failure_codes: [401, 403]
  rate_limit_codes: [429, 529]
  metadata_paths:
    model: "model"
    stop_reason: "stop_reason"
```

**3. google-ai** — key goes in URL query string
```yaml
verify:
  method: GET
  url: https://generativelanguage.googleapis.com/v1/models?key={{KEY}}
  success_codes: [200]
  failure_codes: [400, 401, 403]
  rate_limit_codes: [429]
  metadata_paths:
    first_model: "models.0.name"
```
(Note: Google returns 400 for bad key, not 401 — include 400 in failure_codes.)

**4. cohere**
```yaml
verify:
  method: GET
  url: https://api.cohere.ai/v1/models
  headers:
    Authorization: "Bearer {{KEY}}"
  success_codes: [200]
  failure_codes: [401, 403]
  rate_limit_codes: [429]
  metadata_paths:
    first_model: "models.0.name"
```

**5. mistral**
```yaml
verify:
  method: GET
  url: https://api.mistral.ai/v1/models
  headers:
    Authorization: "Bearer {{KEY}}"
  success_codes: [200]
  failure_codes: [401, 403]
  rate_limit_codes: [429]
  metadata_paths:
    first_model: "data.0.id"
```

**6. groq**
```yaml
verify:
  method: GET
  url: https://api.groq.com/openai/v1/models
  headers:
    Authorization: "Bearer {{KEY}}"
  success_codes: [200]
  failure_codes: [401, 403]
  rate_limit_codes: [429]
  metadata_paths:
    first_model: "data.0.id"
```

**7. xai**
```yaml
verify:
  method: GET
  url: https://api.x.ai/v1/api-key
  headers:
    Authorization: "Bearer {{KEY}}"
  success_codes: [200]
  failure_codes: [401, 403]
  rate_limit_codes: [429]
  metadata_paths:
    name: "name"
    acls: "acls"
```

**8. ai21**
```yaml
verify:
  method: GET
  url: https://api.ai21.com/studio/v1/models
  headers:
    Authorization: "Bearer {{KEY}}"
  success_codes: [200]
  failure_codes: [401, 403]
  rate_limit_codes: [429]
```

**9. inflection** — leave URL empty (no public endpoint) → verifier will return StatusUnknown
```yaml
verify:
  method: GET
  url: ""
  success_codes: [200]
  failure_codes: [401, 403]
  rate_limit_codes: [429]
```

**10. perplexity**
```yaml
verify:
  method: POST
  url: https://api.perplexity.ai/chat/completions
  headers:
    Authorization: "Bearer {{KEY}}"
    content-type: "application/json"
  body: '{"model":"sonar","messages":[{"role":"user","content":"hi"}],"max_tokens":1}'
  success_codes: [200]
  failure_codes: [401, 403]
  rate_limit_codes: [429]
```

**11. deepseek**
```yaml
verify:
  method: GET
  url: https://api.deepseek.com/v1/models
  headers:
    Authorization: "Bearer {{KEY}}"
  success_codes: [200]
  failure_codes: [401, 403]
  rate_limit_codes: [429]
  metadata_paths:
    first_model: "data.0.id"
```

**12. together**
```yaml
verify:
  method: GET
  url: https://api.together.xyz/v1/models
  headers:
    Authorization: "Bearer {{KEY}}"
  success_codes: [200]
  failure_codes: [401, 403]
  rate_limit_codes: [429]
  metadata_paths:
    first_model: "0.id"
```

After updating, `diff providers/openai.yaml pkg/providers/definitions/openai.yaml` should return nothing (identical files). Verify for each of the 12.

If a provider file in `providers/` has a slightly different filename in `pkg/providers/definitions/` (e.g. `google_ai.yaml` vs `google-ai.yaml`), investigate `pkg/providers/definitions/` directory listing first via `Read` or `Bash ls` to get exact names, then update the matching file.

cd /home/salva/Documents/apikey && go test ./pkg/providers/... -run Registry -v && for p in openai anthropic google-ai cohere mistral groq xai ai21 perplexity deepseek together; do diff "providers/$p.yaml" "pkg/providers/definitions/$p.yaml" || echo "MISMATCH $p"; done - `grep -l '{{KEY}}' providers/*.yaml | wc -l` returns at least 11 (inflection has empty URL, so no key template) - `grep -l 'success_codes:' providers/*.yaml | wc -l` returns at least 12 - `grep -l 'metadata_paths:' providers/*.yaml | wc -l` returns at least 8 - All Tier 1 provider files identical between `providers/` and `pkg/providers/definitions/` - `go test ./pkg/providers/...` passes (existing guardrail tests load YAMLs) All 12 Tier 1 provider YAMLs carry Phase 5 verify specs in both locations. Task 2: Guardrail test — verify spec completeness for Tier 1 pkg/providers/registry_test.go - Test asserts that all 12 Tier 1 providers in the loaded registry have VerifySpec.URL set (except inflection, which is allowed to be empty) - Test asserts that providers with a URL have SuccessCodes populated (either via new field or legacy ValidStatus) - Test asserts that all non-empty verify URLs start with "https://" Append to `pkg/providers/registry_test.go` (or create if absent — check first):

```go
func TestTier1VerifySpecs_Complete(t *testing.T) {
    reg, err := NewRegistry()
    if err != nil {
        t.Fatalf("NewRegistry: %v", err)
    }
    tier1 := []string{"openai", "anthropic", "google-ai", "cohere", "mistral", "groq", "xai", "ai21", "perplexity", "deepseek", "together"}
    // Note: inflection intentionally excluded — no public verify endpoint.
    for _, name := range tier1 {
        p, ok := reg.Get(name) // adjust to match actual Registry method
        if !ok {
            t.Errorf("provider %q not in registry", name)
            continue
        }
        if p.Verify.URL == "" {
            t.Errorf("provider %q: verify.url must be set", name)
            continue
        }
        if !strings.HasPrefix(p.Verify.URL, "https://") {
            t.Errorf("provider %q: verify.url must be HTTPS, got %q", name, p.Verify.URL)
        }
        if len(p.Verify.EffectiveSuccessCodes()) == 0 {
            t.Errorf("provider %q: no success codes configured", name)
        }
    }
}

func TestInflection_NoVerifyEndpoint(t *testing.T) {
    reg, err := NewRegistry()
    if err != nil {
        t.Fatalf("NewRegistry: %v", err)
    }
    p, ok := reg.Get("inflection")
    if !ok {
        t.Skip("inflection provider not loaded")
    }
    if p.Verify.URL != "" {
        t.Errorf("inflection should have empty verify.url (no public endpoint), got %q", p.Verify.URL)
    }
}
```

Adjust `reg.Get(name)` to match the actual method on Registry (check pkg/providers/registry.go first — may be `Find`, `ByName`, or a map access). If Registry exposes the providers via an exported field or method like `All()`, iterate from that.

cd /home/salva/Documents/apikey && go test ./pkg/providers/... -run Tier1VerifySpecs -v && go test ./pkg/providers/... -run Inflection -v - Both new tests pass - `go build ./...` succeeds Guardrail test protects Tier 1 verify spec quality on future edits. - `go test ./pkg/providers/... -v` all pass - `diff` between providers/ and pkg/providers/definitions/ copies returns no mismatches for Tier 1 files - Guardrail test catches any regression

<success_criteria>

• 12 Tier 1 providers carry complete verify specs
• Dual-location sync maintained
• New guardrail test prevents future drift </success_criteria>

After completion, create `.planning/phases/05-verification-engine/05-04-SUMMARY.md`