docs(02): create phase 2 tier 1-2 provider plans
This commit is contained in:
salvacybersec
240
.planning/phases/02-tier-1-2-providers/02-05-PLAN.md
Normal file
240
.planning/phases/02-tier-1-2-providers/02-05-PLAN.md
Normal file
@@ -0,0 +1,240 @@
|
||||
---
|
||||
phase: 02-tier-1-2-providers
|
||||
plan: 05
|
||||
type: execute
|
||||
wave: 2
|
||||
depends_on: ["02-01", "02-02", "02-03", "02-04"]
|
||||
files_modified:
|
||||
- pkg/providers/tier12_test.go
|
||||
autonomous: true
|
||||
requirements: [PROV-01, PROV-02]
|
||||
must_haves:
|
||||
truths:
|
||||
- "Registry loads exactly 26 Tier 1+2 providers plus any pre-existing extras"
|
||||
- "Tier 1 count = 12, Tier 2 count = 14"
|
||||
- "Every regex pattern in the registry compiles under Go RE2"
|
||||
- "`keyhunter providers stats` (via Registry.Stats()) returns correct counts"
|
||||
artifacts:
|
||||
- path: "pkg/providers/tier12_test.go"
|
||||
provides: "Integration test asserting tier counts and regex compilation"
|
||||
min_lines: 50
|
||||
key_links:
|
||||
- from: "tier12_test.go"
|
||||
to: "pkg/providers/registry.go NewRegistry()"
|
||||
via: "load-all + count-by-tier"
|
||||
pattern: "NewRegistry"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Write a guardrail integration test that verifies the Phase 2 deliverable: exactly 12 Tier 1 and 14 Tier 2 providers are loaded by the registry, every regex pattern compiles, and no keywords are empty.
|
||||
|
||||
Purpose: Goal-backward verification. Phase 2 success criteria state "26 providers loaded with pattern and keyword counts" — this test locks that in so regressions in Phase 3+ cannot quietly break Phase 2 coverage.
|
||||
|
||||
Output: Single test file that runs in the existing `go test ./pkg/providers/...` pipeline.
|
||||
|
||||
Addresses PROV-01 and PROV-02 verification.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@pkg/providers/schema.go
|
||||
@pkg/providers/registry.go
|
||||
@pkg/providers/registry_test.go
|
||||
@.planning/phases/02-tier-1-2-providers/02-01-PLAN.md
|
||||
@.planning/phases/02-tier-1-2-providers/02-02-PLAN.md
|
||||
@.planning/phases/02-tier-1-2-providers/02-03-PLAN.md
|
||||
@.planning/phases/02-tier-1-2-providers/02-04-PLAN.md
|
||||
|
||||
<interfaces>
|
||||
From pkg/providers/registry.go (verify exact API via Read during execution):
|
||||
- `NewRegistry() (*Registry, error)` or similar factory
|
||||
- `Registry.Stats() RegistryStats` returning ByTier map[int]int
|
||||
- `Registry.All() []Provider` or iteration helper
|
||||
|
||||
RegistryStats (from schema.go):
|
||||
```go
|
||||
type RegistryStats struct {
|
||||
Total int
|
||||
ByTier map[int]int
|
||||
ByConfidence map[string]int
|
||||
}
|
||||
```
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 1: Write tier1/tier2 count + regex compilation guardrail test</name>
|
||||
<files>pkg/providers/tier12_test.go</files>
|
||||
<read_first>
|
||||
- pkg/providers/registry.go (to find exact NewRegistry constructor + accessor names)
|
||||
- pkg/providers/registry_test.go (to mirror existing test style)
|
||||
- pkg/providers/schema.go (Pattern, Provider, RegistryStats types)
|
||||
</read_first>
|
||||
<behavior>
|
||||
- Test 1: TestTier1Count — registry.Stats().ByTier[1] == 12
|
||||
- Test 2: TestTier2Count — registry.Stats().ByTier[2] == 14
|
||||
- Test 3: TestAllPatternsCompile — every Pattern.Regex across all providers compiles via regexp.Compile without error
|
||||
- Test 4: TestAllProvidersHaveKeywords — every Provider.Keywords slice has len > 0 (Aho-Corasick pre-filter requires keywords)
|
||||
- Test 5: TestTier1ProviderNames — registry contains providers with these names: openai, anthropic, google-ai, vertex-ai, aws-bedrock, azure-openai, meta-ai, xai, cohere, mistral, inflection, ai21
|
||||
- Test 6: TestTier2ProviderNames — registry contains: groq, replicate, anyscale, together, fireworks, baseten, deepinfra, lepton, modal, cerebrium, novita, sambanova, octoai, friendli
|
||||
</behavior>
|
||||
<action>
|
||||
Create pkg/providers/tier12_test.go. Use the existing registry_test.go for the exact constructor name and iteration API — likely `providers.NewRegistry()` returning `(*Registry, error)`.
|
||||
|
||||
Skeleton:
|
||||
|
||||
```go
|
||||
package providers
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// expectedTier1 lists all 12 Tier 1 Frontier provider names (PROV-01).
|
||||
var expectedTier1 = []string{
|
||||
"openai", "anthropic", "google-ai", "vertex-ai", "aws-bedrock",
|
||||
"azure-openai", "meta-ai", "xai", "cohere", "mistral",
|
||||
"inflection", "ai21",
|
||||
}
|
||||
|
||||
// expectedTier2 lists all 14 Tier 2 Inference Platform provider names (PROV-02).
|
||||
var expectedTier2 = []string{
|
||||
"groq", "replicate", "anyscale", "together", "fireworks",
|
||||
"baseten", "deepinfra", "lepton", "modal", "cerebrium",
|
||||
"novita", "sambanova", "octoai", "friendli",
|
||||
}
|
||||
|
||||
func TestTier1Count(t *testing.T) {
|
||||
reg, err := NewRegistry()
|
||||
if err != nil {
|
||||
t.Fatalf("NewRegistry failed: %v", err)
|
||||
}
|
||||
stats := reg.Stats()
|
||||
if got := stats.ByTier[1]; got != 12 {
|
||||
t.Errorf("Tier 1 count = %d, want 12", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTier2Count(t *testing.T) {
|
||||
reg, err := NewRegistry()
|
||||
if err != nil {
|
||||
t.Fatalf("NewRegistry failed: %v", err)
|
||||
}
|
||||
stats := reg.Stats()
|
||||
if got := stats.ByTier[2]; got != 14 {
|
||||
t.Errorf("Tier 2 count = %d, want 14", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAllPatternsCompile(t *testing.T) {
|
||||
reg, err := NewRegistry()
|
||||
if err != nil {
|
||||
t.Fatalf("NewRegistry failed: %v", err)
|
||||
}
|
||||
for _, p := range reg.All() { // adjust to real accessor name
|
||||
for i, pat := range p.Patterns {
|
||||
if _, err := regexp.Compile(pat.Regex); err != nil {
|
||||
t.Errorf("provider %q pattern[%d] regex %q failed to compile: %v",
|
||||
p.Name, i, pat.Regex, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAllProvidersHaveKeywords(t *testing.T) {
|
||||
reg, err := NewRegistry()
|
||||
if err != nil {
|
||||
t.Fatalf("NewRegistry failed: %v", err)
|
||||
}
|
||||
for _, p := range reg.All() {
|
||||
if len(p.Keywords) == 0 {
|
||||
t.Errorf("provider %q has no keywords (breaks AC pre-filter)", p.Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTier1ProviderNames(t *testing.T) {
|
||||
reg, err := NewRegistry()
|
||||
if err != nil {
|
||||
t.Fatalf("NewRegistry failed: %v", err)
|
||||
}
|
||||
for _, name := range expectedTier1 {
|
||||
p, ok := reg.Get(name) // adjust to real accessor
|
||||
if !ok {
|
||||
t.Errorf("Tier 1 provider %q not found in registry", name)
|
||||
continue
|
||||
}
|
||||
if p.Tier != 1 {
|
||||
t.Errorf("provider %q tier = %d, want 1", name, p.Tier)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTier2ProviderNames(t *testing.T) {
|
||||
reg, err := NewRegistry()
|
||||
if err != nil {
|
||||
t.Fatalf("NewRegistry failed: %v", err)
|
||||
}
|
||||
for _, name := range expectedTier2 {
|
||||
p, ok := reg.Get(name)
|
||||
if !ok {
|
||||
t.Errorf("Tier 2 provider %q not found in registry", name)
|
||||
continue
|
||||
}
|
||||
if p.Tier != 2 {
|
||||
t.Errorf("provider %q tier = %d, want 2", name, p.Tier)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
IMPORTANT: Before writing the file, Read pkg/providers/registry.go to confirm the EXACT names of:
|
||||
1. Constructor (NewRegistry? LoadRegistry?)
|
||||
2. Iteration method (All? Providers? Each?)
|
||||
3. Lookup method (Get? Find? Lookup?)
|
||||
|
||||
If the real API differs, adjust the skeleton above to use the real method names. The behavior and assertions must stay the same.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd /home/salva/Documents/apikey && go test ./pkg/providers/... -run 'TestTier1Count|TestTier2Count|TestAllPatternsCompile|TestAllProvidersHaveKeywords|TestTier1ProviderNames|TestTier2ProviderNames' -v -count=1</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `test -f pkg/providers/tier12_test.go`
|
||||
- `grep -q 'expectedTier1' pkg/providers/tier12_test.go`
|
||||
- `grep -q 'expectedTier2' pkg/providers/tier12_test.go`
|
||||
- All six test functions present: `grep -c '^func Test' pkg/providers/tier12_test.go` >= 6
|
||||
- `go test ./pkg/providers/... -run TestTier1Count -count=1` passes (Tier 1 = 12)
|
||||
- `go test ./pkg/providers/... -run TestTier2Count -count=1` passes (Tier 2 = 14)
|
||||
- `go test ./pkg/providers/... -run TestAllPatternsCompile -count=1` passes (all regex compile)
|
||||
- `go test ./pkg/providers/... -count=1` passes (full suite green)
|
||||
</acceptance_criteria>
|
||||
<done>Guardrail test locks in 12 Tier 1 + 14 Tier 2 providers, regex compilation, and keyword presence. Future phase regressions that drop a provider will fail this test.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
Final phase sanity:
|
||||
- `go test ./... -count=1` passes
|
||||
- `ls providers/*.yaml | wc -l` >= 27 (26 new + huggingface)
|
||||
- `ls pkg/providers/definitions/*.yaml | wc -l` matches providers/ directory count
|
||||
- `diff <(ls providers/*.yaml | xargs -n1 basename) <(ls pkg/providers/definitions/*.yaml | xargs -n1 basename)` returns empty
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- Registry loads exactly 12 Tier 1 + 14 Tier 2 providers (+ pre-existing tier-less huggingface)
|
||||
- All patterns compile under RE2
|
||||
- All providers have non-empty keywords
|
||||
- Test file committed and passing
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
After completion, create `.planning/phases/02-tier-1-2-providers/02-05-SUMMARY.md`
|
||||
</output>
|
||||
Reference in New Issue
Block a user