docs(05-04): complete Tier 1 verify specs plan
This commit is contained in:
salvacybersec
@@ -126,7 +126,7 @@ Plans:
|
||||
- [x] 05-01-PLAN.md — Wave 0: extend VerifySpec schema, Finding struct, storage schema; add gjson dep
|
||||
- [ ] 05-02-PLAN.md — LEGAL.md + pkg/legal embed + consent prompt + keyhunter legal command
|
||||
- [ ] 05-03-PLAN.md — pkg/verify HTTPVerifier: template sub, gjson metadata extraction, ants pool
|
||||
- [ ] 05-04-PLAN.md — Update 12 Tier 1 provider YAMLs with extended verify specs + guardrail test
|
||||
- [x] 05-04-PLAN.md — Update 12 Tier 1 provider YAMLs with extended verify specs + guardrail test
|
||||
- [ ] 05-05-PLAN.md — cmd/scan.go --verify wiring + --verify-timeout/workers flags + output verify column
|
||||
|
||||
### Phase 6: Output, Reporting & Key Management
|
||||
|
||||
@@ -3,14 +3,14 @@ gsd_state_version: 1.0
|
||||
milestone: v1.0
|
||||
milestone_name: milestone
|
||||
status: executing
|
||||
stopped_at: Completed 05-01-PLAN.md
|
||||
last_updated: "2026-04-05T12:44:11.076Z"
|
||||
stopped_at: Completed 05-04-PLAN.md
|
||||
last_updated: "2026-04-05T12:48:10.887Z"
|
||||
last_activity: 2026-04-05
|
||||
progress:
|
||||
total_phases: 18
|
||||
completed_phases: 4
|
||||
total_plans: 28
|
||||
completed_plans: 24
|
||||
completed_plans: 25
|
||||
percent: 20
|
||||
---
|
||||
|
||||
@@ -26,7 +26,7 @@ See: .planning/PROJECT.md (updated 2026-04-04)
|
||||
## Current Position
|
||||
|
||||
Phase: 05 (verification-engine) — EXECUTING
|
||||
Plan: 2 of 5
|
||||
Plan: 3 of 5
|
||||
Status: Ready to execute
|
||||
Last activity: 2026-04-05
|
||||
|
||||
@@ -70,6 +70,7 @@ Progress: [██░░░░░░░░] 20%
|
||||
| Phase 04 P02 | 4min | 1 tasks | 3 files |
|
||||
| Phase 04 P05 | 3min | 1 tasks | 2 files |
|
||||
| Phase 05 P01 | 3m43s | 2 tasks | 10 files |
|
||||
| Phase 05 P04 | 10m | 2 tasks | 25 files |
|
||||
|
||||
## Accumulated Context
|
||||
|
||||
@@ -108,6 +109,6 @@ None yet.
|
||||
|
||||
## Session Continuity
|
||||
|
||||
Last session: 2026-04-05T12:44:04.063Z
|
||||
Stopped at: Completed 05-01-PLAN.md
|
||||
Last session: 2026-04-05T12:48:10.879Z
|
||||
Stopped at: Completed 05-04-PLAN.md
|
||||
Resume file: None
|
||||
|
||||
156
.planning/phases/05-verification-engine/05-04-SUMMARY.md
Normal file
156
.planning/phases/05-verification-engine/05-04-SUMMARY.md
Normal file
@@ -0,0 +1,156 @@
|
||||
---
|
||||
phase: 05-verification-engine
|
||||
plan: 04
|
||||
subsystem: providers
|
||||
tags: [providers, verification, yaml, guardrail]
|
||||
one-liner: "Extended 12 Tier 1 provider YAMLs with full verify specs (success_codes, metadata_paths, {{KEY}} templates) and guardrail tests enforcing completeness."
|
||||
requires:
|
||||
- VerifySpec extended fields from 05-01 (success_codes, failure_codes, rate_limit_codes, metadata_paths)
|
||||
provides:
|
||||
- Tier 1 providers ready for HTTPVerifier consumption (Plan 05-03)
|
||||
- Guardrail tests blocking regressions on Tier 1 verify specs
|
||||
affects:
|
||||
- pkg/verify HTTPVerifier will find well-formed specs for all Tier 1 providers
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns:
|
||||
- "{{KEY}} double-brace template substitution (canonical for Phase 5)"
|
||||
- "Dual-location YAML mirror: providers/ + pkg/providers/definitions/"
|
||||
key-files:
|
||||
created: []
|
||||
modified:
|
||||
- providers/openai.yaml
|
||||
- providers/anthropic.yaml
|
||||
- providers/google-ai.yaml
|
||||
- providers/cohere.yaml
|
||||
- providers/mistral.yaml
|
||||
- providers/groq.yaml
|
||||
- providers/xai.yaml
|
||||
- providers/ai21.yaml
|
||||
- providers/inflection.yaml
|
||||
- providers/perplexity.yaml
|
||||
- providers/deepseek.yaml
|
||||
- providers/together.yaml
|
||||
- pkg/providers/definitions/openai.yaml
|
||||
- pkg/providers/definitions/anthropic.yaml
|
||||
- pkg/providers/definitions/google-ai.yaml
|
||||
- pkg/providers/definitions/cohere.yaml
|
||||
- pkg/providers/definitions/mistral.yaml
|
||||
- pkg/providers/definitions/groq.yaml
|
||||
- pkg/providers/definitions/xai.yaml
|
||||
- pkg/providers/definitions/ai21.yaml
|
||||
- pkg/providers/definitions/inflection.yaml
|
||||
- pkg/providers/definitions/perplexity.yaml
|
||||
- pkg/providers/definitions/deepseek.yaml
|
||||
- pkg/providers/definitions/together.yaml
|
||||
- pkg/providers/registry_test.go
|
||||
decisions:
|
||||
- "Followed the plan's explicit task list (openai, anthropic, google-ai, cohere, mistral, groq, xai, ai21, inflection, perplexity, deepseek, together) rather than the brief objective mentioning vertex-ai/aws-bedrock/azure-openai/meta-ai — the <tasks> block is authoritative and these providers either lack a public verify endpoint usable without region/tenant-specific config (Vertex, Bedrock, Azure, Meta)."
|
||||
- "Anthropic uses POST /v1/messages with a minimal 1-token body rather than an unauthenticated GET — Anthropic has no public GET endpoint that cleanly differentiates authenticated vs unauthenticated keys. 529 added to rate_limit_codes (Anthropic overload)."
|
||||
- "Inflection verify URL left empty — plan specifies no public endpoint; HTTPVerifier will return StatusUnknown."
|
||||
- "Google AI failure_codes include 400 because the API returns 400 for bad API keys rather than 401."
|
||||
metrics:
|
||||
tasks-completed: 2
|
||||
files-modified: 25
|
||||
commits: 2
|
||||
duration: "~10m"
|
||||
completed: "2026-04-05"
|
||||
---
|
||||
|
||||
# Phase 05 Plan 04: Tier 1 Provider Verify Specs Summary
|
||||
|
||||
## Objective Recap
|
||||
|
||||
Update the 12 Tier 1 provider YAMLs so each carries a complete Phase 5 verify spec (success_codes, failure_codes, rate_limit_codes, metadata_paths, `{{KEY}}` header/URL/body substitution), with dual-location sync between `providers/` and `pkg/providers/definitions/`. Add a guardrail test that blocks future regressions.
|
||||
|
||||
## What Was Built
|
||||
|
||||
### Task 1: Provider YAML Updates
|
||||
|
||||
Rewrote 12 provider YAMLs in both locations (24 files total) to adopt the canonical Phase 5 verify schema.
|
||||
|
||||
| Provider | Method | Endpoint | metadata_paths |
|
||||
| ----------- | ------ | -------------------------------------------------------- | -------------- |
|
||||
| openai | GET | api.openai.com/v1/models | yes (2 fields) |
|
||||
| anthropic | POST | api.anthropic.com/v1/messages (1-token body) | yes |
|
||||
| google-ai | GET | generativelanguage.googleapis.com/v1/models?key={{KEY}} | yes |
|
||||
| cohere | GET | api.cohere.ai/v1/models | yes |
|
||||
| mistral | GET | api.mistral.ai/v1/models | yes |
|
||||
| groq | GET | api.groq.com/openai/v1/models | yes |
|
||||
| xai | GET | api.x.ai/v1/api-key | yes (2 fields) |
|
||||
| ai21 | GET | api.ai21.com/studio/v1/models | no |
|
||||
| inflection | GET | "" (no public endpoint) | no |
|
||||
| perplexity | POST | api.perplexity.ai/chat/completions (sonar, 1 token) | no |
|
||||
| deepseek | GET | api.deepseek.com/v1/models | yes |
|
||||
| together | GET | api.together.xyz/v1/models | yes |
|
||||
|
||||
Key design points:
|
||||
- All templates use `{{KEY}}` (double brace), replacing legacy `{KEY}` single-brace form.
|
||||
- `success_codes: [200]`, `failure_codes: [401, 403]`, `rate_limit_codes: [429]` is the baseline, with per-provider overrides (google-ai adds 400 to failure; anthropic adds 529 to rate_limit).
|
||||
- Legacy `valid_status`/`invalid_status` fields removed from these 12; schema back-compat in pkg/providers/schema.go still supports other tiers that keep them.
|
||||
- `diff providers/X.yaml pkg/providers/definitions/X.yaml` returns no mismatches for all 12 files.
|
||||
|
||||
**Commit:** `f3ae8f0` — feat(05-04): extend Tier 1 provider verify specs
|
||||
|
||||
### Task 2: Guardrail Tests
|
||||
|
||||
Added two tests to `pkg/providers/registry_test.go`:
|
||||
|
||||
1. `TestTier1VerifySpecs_Complete` — iterates the 11 Tier 1 providers that must have verify endpoints, asserts each has a non-empty HTTPS URL and non-empty `EffectiveSuccessCodes()`.
|
||||
2. `TestInflection_NoVerifyEndpoint` — locks in the intentional empty-URL state for inflection so a well-meaning future edit cannot silently add a bogus endpoint.
|
||||
|
||||
**Commit:** `6a94ce5` — test(05-04): guardrail tests for Tier 1 verify spec completeness
|
||||
|
||||
## Verification Results
|
||||
|
||||
```
|
||||
go test ./pkg/providers/... -v
|
||||
...
|
||||
--- PASS: TestTier1VerifySpecs_Complete (0.04s)
|
||||
--- PASS: TestInflection_NoVerifyEndpoint (0.04s)
|
||||
PASS
|
||||
ok github.com/salvacybersec/keyhunter/pkg/providers 0.931s
|
||||
```
|
||||
|
||||
Acceptance criteria audit:
|
||||
- `grep -l '{{KEY}}' providers/*.yaml | wc -l` → 11 (inflection excluded, as specified)
|
||||
- `grep -l 'success_codes:' providers/*.yaml | wc -l` → 12
|
||||
- `grep -l 'metadata_paths:' providers/*.yaml | wc -l` → 9 (≥8 required)
|
||||
- `diff` between both locations returns 0 mismatches for all 12 providers
|
||||
- `go test ./pkg/providers/...` passes (all 9 tests, including the two new guardrails)
|
||||
- `go build ./...` succeeds
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
None — plan executed exactly as written. The plan's `<tasks>` block was taken as authoritative over the phase-level objective prose that briefly referenced vertex-ai/aws-bedrock/azure-openai/meta-ai; see Decisions for rationale.
|
||||
|
||||
## Authentication Gates
|
||||
|
||||
None.
|
||||
|
||||
## Known Stubs
|
||||
|
||||
None. Inflection's empty verify URL is intentional and documented via `TestInflection_NoVerifyEndpoint`.
|
||||
|
||||
## Deferred Issues
|
||||
|
||||
None.
|
||||
|
||||
## Self-Check: PASSED
|
||||
|
||||
- providers/openai.yaml: FOUND
|
||||
- providers/anthropic.yaml: FOUND
|
||||
- providers/google-ai.yaml: FOUND
|
||||
- providers/cohere.yaml: FOUND
|
||||
- providers/mistral.yaml: FOUND
|
||||
- providers/groq.yaml: FOUND
|
||||
- providers/xai.yaml: FOUND
|
||||
- providers/ai21.yaml: FOUND
|
||||
- providers/inflection.yaml: FOUND
|
||||
- providers/perplexity.yaml: FOUND
|
||||
- providers/deepseek.yaml: FOUND
|
||||
- providers/together.yaml: FOUND
|
||||
- pkg/providers/definitions/ mirrors: FOUND (all 12)
|
||||
- pkg/providers/registry_test.go: FOUND (updated)
|
||||
- commit f3ae8f0: FOUND
|
||||
- commit 6a94ce5: FOUND
|
||||
Reference in New Issue
Block a user