keyhunter/.planning/phases/05-verification-engine/05-04-PLAN.md

---
phase: 05-verification-engine
plan: 04
type: execute
wave: 1
depends_on: [05-01]
files_modified:
  - providers/openai.yaml
  - providers/anthropic.yaml
  - providers/google-ai.yaml
  - providers/cohere.yaml
  - providers/mistral.yaml
  - providers/groq.yaml
  - providers/xai.yaml
  - providers/ai21.yaml
  - providers/inflection.yaml
  - providers/perplexity.yaml
  - providers/deepseek.yaml
  - providers/together.yaml
  - pkg/providers/definitions/openai.yaml
  - pkg/providers/definitions/anthropic.yaml
  - pkg/providers/definitions/google-ai.yaml
  - pkg/providers/definitions/cohere.yaml
  - pkg/providers/definitions/mistral.yaml
  - pkg/providers/definitions/groq.yaml
  - pkg/providers/definitions/xai.yaml
  - pkg/providers/definitions/ai21.yaml
  - pkg/providers/definitions/inflection.yaml
  - pkg/providers/definitions/perplexity.yaml
  - pkg/providers/definitions/deepseek.yaml
  - pkg/providers/definitions/together.yaml
  - pkg/providers/registry_test.go
autonomous: true
requirements: [VRFY-02, VRFY-03]
must_haves:
  truths:
    - "All 12 Tier 1 provider YAMLs include success_codes, failure_codes, rate_limit_codes fields (extended from legacy valid_status/invalid_status)"
    - "{{KEY}} template is used in verify.headers (Bearer token) or verify.url (query key)"
    - "Providers with known metadata endpoints include metadata_paths mapping"
    - "Dual-location sync: providers/ and pkg/providers/definitions/ kept identical"
    - "All YAMLs still load via providers.NewRegistry() with no parse errors"
  artifacts:
    - path: "providers/openai.yaml"
      provides: "OpenAI verify spec with success_codes, {{KEY}} header substitution"
      contains: "{{KEY}}"
  key_links:
    - from: "providers/*.yaml"
      to: "pkg/providers/definitions/*.yaml"
      via: "dual-location mirror"
      pattern: "format_version"
---

<objective>
Update Tier 1 provider YAMLs so each carries a complete verify spec usable by the new HTTPVerifier: `{{KEY}}` template in headers or URL, `success_codes`, `failure_codes`, `rate_limit_codes`, and `metadata_paths` where the provider API returns useful metadata. Must maintain the dual-location sync between `providers/` (user-visible) and `pkg/providers/definitions/` (embed).

Purpose: VRFY-03 requires that provider YAMLs carry verification metadata. Without this update the verifier from Plan 05-03 would have no endpoints to hit.
Output: 12 Tier 1 provider YAMLs updated in both locations; guardrail test asserts presence of new fields.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/phases/05-verification-engine/05-CONTEXT.md
@providers/openai.yaml
@pkg/providers/schema.go

<interfaces>
Extended VerifySpec (from Plan 05-01) accepts these YAML keys under `verify:`:
```yaml
verify:
  method: GET
  url: https://api.provider.com/v1/models
  headers:
    Authorization: "Bearer {{KEY}}"
  body: ""                                        # optional, can use {{KEY}}
  success_codes: [200]
  failure_codes: [401, 403]
  rate_limit_codes: [429]
  metadata_paths:                                 # display-name -> gjson path
    org: "organization.name"
    tier: "rate_limit.tier"
```

Legacy fields `valid_status`/`invalid_status` still parse (backward compat) but new YAMLs should use the canonical `success_codes`/`failure_codes`.

**Dual-location rule** (from Phase 1 decisions): every YAML in `providers/` must have an identical copy in `pkg/providers/definitions/` because `go:embed` cannot traverse `..`.
</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Update 12 Tier 1 provider YAMLs with extended verify specs (both locations)</name>
  <files>providers/openai.yaml, providers/anthropic.yaml, providers/google-ai.yaml, providers/cohere.yaml, providers/mistral.yaml, providers/groq.yaml, providers/xai.yaml, providers/ai21.yaml, providers/inflection.yaml, providers/perplexity.yaml, providers/deepseek.yaml, providers/together.yaml, and mirrors in pkg/providers/definitions/</files>
  <action>
    For each of the 12 providers below, update BOTH `providers/{name}.yaml` AND `pkg/providers/definitions/{name}.yaml` to have an identical `verify:` block as specified. Preserve existing `format_version`, `name`, `display_name`, `tier`, `last_verified`, `keywords`, and `patterns` — only touch the `verify:` block.

    First, check each file exists in both locations — if a provider file is named differently in `pkg/providers/definitions/` than in `providers/`, match the `providers/` location's naming. If a file is missing from `pkg/providers/definitions/`, add it as a copy.

    Use `{{KEY}}` (double brace) as the template marker. Set `last_verified: "2026-04-05"` on every updated file.

    **1. openai** — `Bearer {{KEY}}` header, `GET /v1/models`
    ```yaml
    verify:
      method: GET
      url: https://api.openai.com/v1/models
      headers:
        Authorization: "Bearer {{KEY}}"
      success_codes: [200]
      failure_codes: [401, 403]
      rate_limit_codes: [429]
      metadata_paths:
        first_model: "data.0.id"
        object_type: "object"
    ```

    **2. anthropic** — POST /v1/messages with minimal body; Anthropic requires `x-api-key` header and `anthropic-version`
    ```yaml
    verify:
      method: POST
      url: https://api.anthropic.com/v1/messages
      headers:
        x-api-key: "{{KEY}}"
        anthropic-version: "2023-06-01"
        content-type: "application/json"
      body: '{"model":"claude-haiku-4-5","max_tokens":1,"messages":[{"role":"user","content":"hi"}]}'
      success_codes: [200]
      failure_codes: [401, 403]
      rate_limit_codes: [429, 529]
      metadata_paths:
        model: "model"
        stop_reason: "stop_reason"
    ```

    **3. google-ai** — key goes in URL query string
    ```yaml
    verify:
      method: GET
      url: https://generativelanguage.googleapis.com/v1/models?key={{KEY}}
      success_codes: [200]
      failure_codes: [400, 401, 403]
      rate_limit_codes: [429]
      metadata_paths:
        first_model: "models.0.name"
    ```
    (Note: Google returns 400 for bad key, not 401 — include 400 in failure_codes.)

    **4. cohere**
    ```yaml
    verify:
      method: GET
      url: https://api.cohere.ai/v1/models
      headers:
        Authorization: "Bearer {{KEY}}"
      success_codes: [200]
      failure_codes: [401, 403]
      rate_limit_codes: [429]
      metadata_paths:
        first_model: "models.0.name"
    ```

    **5. mistral**
    ```yaml
    verify:
      method: GET
      url: https://api.mistral.ai/v1/models
      headers:
        Authorization: "Bearer {{KEY}}"
      success_codes: [200]
      failure_codes: [401, 403]
      rate_limit_codes: [429]
      metadata_paths:
        first_model: "data.0.id"
    ```

    **6. groq**
    ```yaml
    verify:
      method: GET
      url: https://api.groq.com/openai/v1/models
      headers:
        Authorization: "Bearer {{KEY}}"
      success_codes: [200]
      failure_codes: [401, 403]
      rate_limit_codes: [429]
      metadata_paths:
        first_model: "data.0.id"
    ```

    **7. xai**
    ```yaml
    verify:
      method: GET
      url: https://api.x.ai/v1/api-key
      headers:
        Authorization: "Bearer {{KEY}}"
      success_codes: [200]
      failure_codes: [401, 403]
      rate_limit_codes: [429]
      metadata_paths:
        name: "name"
        acls: "acls"
    ```

    **8. ai21**
    ```yaml
    verify:
      method: GET
      url: https://api.ai21.com/studio/v1/models
      headers:
        Authorization: "Bearer {{KEY}}"
      success_codes: [200]
      failure_codes: [401, 403]
      rate_limit_codes: [429]
    ```

    **9. inflection** — leave URL empty (no public endpoint) → verifier will return StatusUnknown
    ```yaml
    verify:
      method: GET
      url: ""
      success_codes: [200]
      failure_codes: [401, 403]
      rate_limit_codes: [429]
    ```

    **10. perplexity**
    ```yaml
    verify:
      method: POST
      url: https://api.perplexity.ai/chat/completions
      headers:
        Authorization: "Bearer {{KEY}}"
        content-type: "application/json"
      body: '{"model":"sonar","messages":[{"role":"user","content":"hi"}],"max_tokens":1}'
      success_codes: [200]
      failure_codes: [401, 403]
      rate_limit_codes: [429]
    ```

    **11. deepseek**
    ```yaml
    verify:
      method: GET
      url: https://api.deepseek.com/v1/models
      headers:
        Authorization: "Bearer {{KEY}}"
      success_codes: [200]
      failure_codes: [401, 403]
      rate_limit_codes: [429]
      metadata_paths:
        first_model: "data.0.id"
    ```

    **12. together**
    ```yaml
    verify:
      method: GET
      url: https://api.together.xyz/v1/models
      headers:
        Authorization: "Bearer {{KEY}}"
      success_codes: [200]
      failure_codes: [401, 403]
      rate_limit_codes: [429]
      metadata_paths:
        first_model: "0.id"
    ```

    After updating, `diff providers/openai.yaml pkg/providers/definitions/openai.yaml` should return nothing (identical files). Verify for each of the 12.

    If a provider file in `providers/` has a slightly different filename in `pkg/providers/definitions/` (e.g. `google_ai.yaml` vs `google-ai.yaml`), investigate `pkg/providers/definitions/` directory listing first via `Read` or `Bash ls` to get exact names, then update the matching file.
  </action>
  <verify>
    <automated>cd /home/salva/Documents/apikey && go test ./pkg/providers/... -run Registry -v && for p in openai anthropic google-ai cohere mistral groq xai ai21 perplexity deepseek together; do diff "providers/$p.yaml" "pkg/providers/definitions/$p.yaml" || echo "MISMATCH $p"; done</automated>
  </verify>
  <acceptance_criteria>
    - `grep -l '{{KEY}}' providers/*.yaml | wc -l` returns at least 11 (inflection has empty URL, so no key template)
    - `grep -l 'success_codes:' providers/*.yaml | wc -l` returns at least 12
    - `grep -l 'metadata_paths:' providers/*.yaml | wc -l` returns at least 8
    - All Tier 1 provider files identical between `providers/` and `pkg/providers/definitions/`
    - `go test ./pkg/providers/...` passes (existing guardrail tests load YAMLs)
  </acceptance_criteria>
  <done>All 12 Tier 1 provider YAMLs carry Phase 5 verify specs in both locations.</done>
</task>

<task type="auto" tdd="true">
  <name>Task 2: Guardrail test — verify spec completeness for Tier 1</name>
  <files>pkg/providers/registry_test.go</files>
  <behavior>
    - Test asserts that all 12 Tier 1 providers in the loaded registry have VerifySpec.URL set (except inflection, which is allowed to be empty)
    - Test asserts that providers with a URL have SuccessCodes populated (either via new field or legacy ValidStatus)
    - Test asserts that all non-empty verify URLs start with "https://"
  </behavior>
  <action>
    Append to `pkg/providers/registry_test.go` (or create if absent — check first):

    ```go
    func TestTier1VerifySpecs_Complete(t *testing.T) {
        reg, err := NewRegistry()
        if err != nil {
            t.Fatalf("NewRegistry: %v", err)
        }
        tier1 := []string{"openai", "anthropic", "google-ai", "cohere", "mistral", "groq", "xai", "ai21", "perplexity", "deepseek", "together"}
        // Note: inflection intentionally excluded — no public verify endpoint.
        for _, name := range tier1 {
            p, ok := reg.Get(name) // adjust to match actual Registry method
            if !ok {
                t.Errorf("provider %q not in registry", name)
                continue
            }
            if p.Verify.URL == "" {
                t.Errorf("provider %q: verify.url must be set", name)
                continue
            }
            if !strings.HasPrefix(p.Verify.URL, "https://") {
                t.Errorf("provider %q: verify.url must be HTTPS, got %q", name, p.Verify.URL)
            }
            if len(p.Verify.EffectiveSuccessCodes()) == 0 {
                t.Errorf("provider %q: no success codes configured", name)
            }
        }
    }

    func TestInflection_NoVerifyEndpoint(t *testing.T) {
        reg, err := NewRegistry()
        if err != nil {
            t.Fatalf("NewRegistry: %v", err)
        }
        p, ok := reg.Get("inflection")
        if !ok {
            t.Skip("inflection provider not loaded")
        }
        if p.Verify.URL != "" {
            t.Errorf("inflection should have empty verify.url (no public endpoint), got %q", p.Verify.URL)
        }
    }
    ```

    Adjust `reg.Get(name)` to match the actual method on Registry (check pkg/providers/registry.go first — may be `Find`, `ByName`, or a map access). If Registry exposes the providers via an exported field or method like `All()`, iterate from that.
  </action>
  <verify>
    <automated>cd /home/salva/Documents/apikey && go test ./pkg/providers/... -run Tier1VerifySpecs -v && go test ./pkg/providers/... -run Inflection -v</automated>
  </verify>
  <acceptance_criteria>
    - Both new tests pass
    - `go build ./...` succeeds
  </acceptance_criteria>
  <done>Guardrail test protects Tier 1 verify spec quality on future edits.</done>
</task>

</tasks>

<verification>
- `go test ./pkg/providers/... -v` all pass
- `diff` between providers/ and pkg/providers/definitions/ copies returns no mismatches for Tier 1 files
- Guardrail test catches any regression
</verification>

<success_criteria>
- 12 Tier 1 providers carry complete verify specs
- Dual-location sync maintained
- New guardrail test prevents future drift
</success_criteria>

<output>
After completion, create `.planning/phases/05-verification-engine/05-04-SUMMARY.md`
</output>