salvacybersec/keyhunter

Fork 0

Files

salvacybersec 026d305026 docs(05-04): complete Tier 1 verify specs plan

2026-04-05 15:48:15 +03:00

7.4 KiB

Raw Blame History

phase, plan, subsystem, tags, one-liner, requires, provides, affects, tech-stack, key-files, decisions, metrics

phase

plan

subsystem

tags

one-liner

requires

provides

affects

tech-stack

key-files

decisions

metrics

05-verification-engine

providers

verification

yaml

guardrail

Extended 12 Tier 1 provider YAMLs with full verify specs (success_codes, metadata_paths, {{KEY}} templates) and guardrail tests enforcing completeness.

VerifySpec extended fields from 05-01 (success_codes, failure_codes, rate_limit_codes, metadata_paths)

Tier 1 providers ready for HTTPVerifier consumption (Plan 05-03)

Guardrail tests blocking regressions on Tier 1 verify specs

pkg/verify HTTPVerifier will find well-formed specs for all Tier 1 providers

added

patterns

{{KEY}} double-brace template substitution (canonical for Phase 5)

Dual-location YAML mirror: providers/ + pkg/providers/definitions/

created

modified

providers/openai.yaml

providers/anthropic.yaml

providers/google-ai.yaml

providers/cohere.yaml

providers/mistral.yaml

providers/groq.yaml

providers/xai.yaml

providers/ai21.yaml

providers/inflection.yaml

providers/perplexity.yaml

providers/deepseek.yaml

providers/together.yaml

pkg/providers/definitions/openai.yaml

pkg/providers/definitions/anthropic.yaml

pkg/providers/definitions/google-ai.yaml

pkg/providers/definitions/cohere.yaml

pkg/providers/definitions/mistral.yaml

pkg/providers/definitions/groq.yaml

pkg/providers/definitions/xai.yaml

pkg/providers/definitions/ai21.yaml

pkg/providers/definitions/inflection.yaml

pkg/providers/definitions/perplexity.yaml

pkg/providers/definitions/deepseek.yaml

pkg/providers/definitions/together.yaml

pkg/providers/registry_test.go

Followed the plan's explicit task list (openai, anthropic, google-ai, cohere, mistral, groq, xai, ai21, inflection, perplexity, deepseek, together) rather than the brief objective mentioning vertex-ai/aws-bedrock/azure-openai/meta-ai — the <tasks> block is authoritative and these providers either lack a public verify endpoint usable without region/tenant-specific config (Vertex, Bedrock, Azure, Meta).

Anthropic uses POST /v1/messages with a minimal 1-token body rather than an unauthenticated GET — Anthropic has no public GET endpoint that cleanly differentiates authenticated vs unauthenticated keys. 529 added to rate_limit_codes (Anthropic overload).

Inflection verify URL left empty — plan specifies no public endpoint; HTTPVerifier will return StatusUnknown.

Google AI failure_codes include 400 because the API returns 400 for bad API keys rather than 401.

tasks-completed	files-modified	commits	duration	completed
2	25	2	~10m	2026-04-05

Phase 05 Plan 04: Tier 1 Provider Verify Specs Summary

Objective Recap

Update the 12 Tier 1 provider YAMLs so each carries a complete Phase 5 verify spec (success_codes, failure_codes, rate_limit_codes, metadata_paths, {{KEY}} header/URL/body substitution), with dual-location sync between providers/ and pkg/providers/definitions/. Add a guardrail test that blocks future regressions.

What Was Built

Task 1: Provider YAML Updates

Rewrote 12 provider YAMLs in both locations (24 files total) to adopt the canonical Phase 5 verify schema.

Provider	Method	Endpoint	metadata_paths
openai	GET	api.openai.com/v1/models	yes (2 fields)
anthropic	POST	api.anthropic.com/v1/messages (1-token body)	yes
google-ai	GET	generativelanguage.googleapis.com/v1/models?key={{KEY}}	yes
cohere	GET	api.cohere.ai/v1/models	yes
mistral	GET	api.mistral.ai/v1/models	yes
groq	GET	api.groq.com/openai/v1/models	yes
xai	GET	api.x.ai/v1/api-key	yes (2 fields)
ai21	GET	api.ai21.com/studio/v1/models	no
inflection	GET	"" (no public endpoint)	no
perplexity	POST	api.perplexity.ai/chat/completions (sonar, 1 token)	no
deepseek	GET	api.deepseek.com/v1/models	yes
together	GET	api.together.xyz/v1/models	yes

Key design points:

All templates use {{KEY}} (double brace), replacing legacy {KEY} single-brace form.
success_codes: [200], failure_codes: [401, 403], rate_limit_codes: [429] is the baseline, with per-provider overrides (google-ai adds 400 to failure; anthropic adds 529 to rate_limit).
Legacy valid_status/invalid_status fields removed from these 12; schema back-compat in pkg/providers/schema.go still supports other tiers that keep them.
diff providers/X.yaml pkg/providers/definitions/X.yaml returns no mismatches for all 12 files.

Commit: f3ae8f0 — feat(05-04): extend Tier 1 provider verify specs

Task 2: Guardrail Tests

Added two tests to pkg/providers/registry_test.go:

TestTier1VerifySpecs_Complete — iterates the 11 Tier 1 providers that must have verify endpoints, asserts each has a non-empty HTTPS URL and non-empty EffectiveSuccessCodes().
TestInflection_NoVerifyEndpoint — locks in the intentional empty-URL state for inflection so a well-meaning future edit cannot silently add a bogus endpoint.

Commit: 6a94ce5 — test(05-04): guardrail tests for Tier 1 verify spec completeness

Verification Results

go test ./pkg/providers/... -v
...
--- PASS: TestTier1VerifySpecs_Complete (0.04s)
--- PASS: TestInflection_NoVerifyEndpoint (0.04s)
PASS
ok  github.com/salvacybersec/keyhunter/pkg/providers  0.931s

Acceptance criteria audit:

grep -l '{{KEY}}' providers/*.yaml | wc -l → 11 (inflection excluded, as specified)
grep -l 'success_codes:' providers/*.yaml | wc -l → 12
grep -l 'metadata_paths:' providers/*.yaml | wc -l → 9 (≥8 required)
diff between both locations returns 0 mismatches for all 12 providers
go test ./pkg/providers/... passes (all 9 tests, including the two new guardrails)
go build ./... succeeds

Deviations from Plan

None — plan executed exactly as written. The plan's <tasks> block was taken as authoritative over the phase-level objective prose that briefly referenced vertex-ai/aws-bedrock/azure-openai/meta-ai; see Decisions for rationale.

Authentication Gates

None.

Known Stubs

None. Inflection's empty verify URL is intentional and documented via TestInflection_NoVerifyEndpoint.

Deferred Issues