241 lines
8.3 KiB
Markdown
241 lines
8.3 KiB
Markdown
---
|
|
phase: 02-tier-1-2-providers
|
|
plan: 05
|
|
type: execute
|
|
wave: 2
|
|
depends_on: ["02-01", "02-02", "02-03", "02-04"]
|
|
files_modified:
|
|
- pkg/providers/tier12_test.go
|
|
autonomous: true
|
|
requirements: [PROV-01, PROV-02]
|
|
must_haves:
|
|
truths:
|
|
- "Registry loads exactly 26 Tier 1+2 providers plus any pre-existing extras"
|
|
- "Tier 1 count = 12, Tier 2 count = 14"
|
|
- "Every regex pattern in the registry compiles under Go RE2"
|
|
- "`keyhunter providers stats` (via Registry.Stats()) returns correct counts"
|
|
artifacts:
|
|
- path: "pkg/providers/tier12_test.go"
|
|
provides: "Integration test asserting tier counts and regex compilation"
|
|
min_lines: 50
|
|
key_links:
|
|
- from: "tier12_test.go"
|
|
to: "pkg/providers/registry.go NewRegistry()"
|
|
via: "load-all + count-by-tier"
|
|
pattern: "NewRegistry"
|
|
---
|
|
|
|
<objective>
|
|
Write a guardrail integration test that verifies the Phase 2 deliverable: exactly 12 Tier 1 and 14 Tier 2 providers are loaded by the registry, every regex pattern compiles, and no keywords are empty.
|
|
|
|
Purpose: Goal-backward verification. Phase 2 success criteria state "26 providers loaded with pattern and keyword counts" — this test locks that in so regressions in Phase 3+ cannot quietly break Phase 2 coverage.
|
|
|
|
Output: Single test file that runs in the existing `go test ./pkg/providers/...` pipeline.
|
|
|
|
Addresses PROV-01 and PROV-02 verification.
|
|
</objective>
|
|
|
|
<execution_context>
|
|
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
|
@$HOME/.claude/get-shit-done/templates/summary.md
|
|
</execution_context>
|
|
|
|
<context>
|
|
@pkg/providers/schema.go
|
|
@pkg/providers/registry.go
|
|
@pkg/providers/registry_test.go
|
|
@.planning/phases/02-tier-1-2-providers/02-01-PLAN.md
|
|
@.planning/phases/02-tier-1-2-providers/02-02-PLAN.md
|
|
@.planning/phases/02-tier-1-2-providers/02-03-PLAN.md
|
|
@.planning/phases/02-tier-1-2-providers/02-04-PLAN.md
|
|
|
|
<interfaces>
|
|
From pkg/providers/registry.go (verify exact API via Read during execution):
|
|
- `NewRegistry() (*Registry, error)` or similar factory
|
|
- `Registry.Stats() RegistryStats` returning ByTier map[int]int
|
|
- `Registry.All() []Provider` or iteration helper
|
|
|
|
RegistryStats (from schema.go):
|
|
```go
|
|
type RegistryStats struct {
|
|
Total int
|
|
ByTier map[int]int
|
|
ByConfidence map[string]int
|
|
}
|
|
```
|
|
</interfaces>
|
|
</context>
|
|
|
|
<tasks>
|
|
|
|
<task type="auto" tdd="true">
|
|
<name>Task 1: Write tier1/tier2 count + regex compilation guardrail test</name>
|
|
<files>pkg/providers/tier12_test.go</files>
|
|
<read_first>
|
|
- pkg/providers/registry.go (to find exact NewRegistry constructor + accessor names)
|
|
- pkg/providers/registry_test.go (to mirror existing test style)
|
|
- pkg/providers/schema.go (Pattern, Provider, RegistryStats types)
|
|
</read_first>
|
|
<behavior>
|
|
- Test 1: TestTier1Count — registry.Stats().ByTier[1] == 12
|
|
- Test 2: TestTier2Count — registry.Stats().ByTier[2] == 14
|
|
- Test 3: TestAllPatternsCompile — every Pattern.Regex across all providers compiles via regexp.Compile without error
|
|
- Test 4: TestAllProvidersHaveKeywords — every Provider.Keywords slice has len > 0 (Aho-Corasick pre-filter requires keywords)
|
|
- Test 5: TestTier1ProviderNames — registry contains providers with these names: openai, anthropic, google-ai, vertex-ai, aws-bedrock, azure-openai, meta-ai, xai, cohere, mistral, inflection, ai21
|
|
- Test 6: TestTier2ProviderNames — registry contains: groq, replicate, anyscale, together, fireworks, baseten, deepinfra, lepton, modal, cerebrium, novita, sambanova, octoai, friendli
|
|
</behavior>
|
|
<action>
|
|
Create pkg/providers/tier12_test.go. Use the existing registry_test.go for the exact constructor name and iteration API — likely `providers.NewRegistry()` returning `(*Registry, error)`.
|
|
|
|
Skeleton:
|
|
|
|
```go
|
|
package providers
|
|
|
|
import (
|
|
"regexp"
|
|
"testing"
|
|
)
|
|
|
|
// expectedTier1 lists all 12 Tier 1 Frontier provider names (PROV-01).
|
|
var expectedTier1 = []string{
|
|
"openai", "anthropic", "google-ai", "vertex-ai", "aws-bedrock",
|
|
"azure-openai", "meta-ai", "xai", "cohere", "mistral",
|
|
"inflection", "ai21",
|
|
}
|
|
|
|
// expectedTier2 lists all 14 Tier 2 Inference Platform provider names (PROV-02).
|
|
var expectedTier2 = []string{
|
|
"groq", "replicate", "anyscale", "together", "fireworks",
|
|
"baseten", "deepinfra", "lepton", "modal", "cerebrium",
|
|
"novita", "sambanova", "octoai", "friendli",
|
|
}
|
|
|
|
func TestTier1Count(t *testing.T) {
|
|
reg, err := NewRegistry()
|
|
if err != nil {
|
|
t.Fatalf("NewRegistry failed: %v", err)
|
|
}
|
|
stats := reg.Stats()
|
|
if got := stats.ByTier[1]; got != 12 {
|
|
t.Errorf("Tier 1 count = %d, want 12", got)
|
|
}
|
|
}
|
|
|
|
func TestTier2Count(t *testing.T) {
|
|
reg, err := NewRegistry()
|
|
if err != nil {
|
|
t.Fatalf("NewRegistry failed: %v", err)
|
|
}
|
|
stats := reg.Stats()
|
|
if got := stats.ByTier[2]; got != 14 {
|
|
t.Errorf("Tier 2 count = %d, want 14", got)
|
|
}
|
|
}
|
|
|
|
func TestAllPatternsCompile(t *testing.T) {
|
|
reg, err := NewRegistry()
|
|
if err != nil {
|
|
t.Fatalf("NewRegistry failed: %v", err)
|
|
}
|
|
for _, p := range reg.All() { // adjust to real accessor name
|
|
for i, pat := range p.Patterns {
|
|
if _, err := regexp.Compile(pat.Regex); err != nil {
|
|
t.Errorf("provider %q pattern[%d] regex %q failed to compile: %v",
|
|
p.Name, i, pat.Regex, err)
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
func TestAllProvidersHaveKeywords(t *testing.T) {
|
|
reg, err := NewRegistry()
|
|
if err != nil {
|
|
t.Fatalf("NewRegistry failed: %v", err)
|
|
}
|
|
for _, p := range reg.All() {
|
|
if len(p.Keywords) == 0 {
|
|
t.Errorf("provider %q has no keywords (breaks AC pre-filter)", p.Name)
|
|
}
|
|
}
|
|
}
|
|
|
|
func TestTier1ProviderNames(t *testing.T) {
|
|
reg, err := NewRegistry()
|
|
if err != nil {
|
|
t.Fatalf("NewRegistry failed: %v", err)
|
|
}
|
|
for _, name := range expectedTier1 {
|
|
p, ok := reg.Get(name) // adjust to real accessor
|
|
if !ok {
|
|
t.Errorf("Tier 1 provider %q not found in registry", name)
|
|
continue
|
|
}
|
|
if p.Tier != 1 {
|
|
t.Errorf("provider %q tier = %d, want 1", name, p.Tier)
|
|
}
|
|
}
|
|
}
|
|
|
|
func TestTier2ProviderNames(t *testing.T) {
|
|
reg, err := NewRegistry()
|
|
if err != nil {
|
|
t.Fatalf("NewRegistry failed: %v", err)
|
|
}
|
|
for _, name := range expectedTier2 {
|
|
p, ok := reg.Get(name)
|
|
if !ok {
|
|
t.Errorf("Tier 2 provider %q not found in registry", name)
|
|
continue
|
|
}
|
|
if p.Tier != 2 {
|
|
t.Errorf("provider %q tier = %d, want 2", name, p.Tier)
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
IMPORTANT: Before writing the file, Read pkg/providers/registry.go to confirm the EXACT names of:
|
|
1. Constructor (NewRegistry? LoadRegistry?)
|
|
2. Iteration method (All? Providers? Each?)
|
|
3. Lookup method (Get? Find? Lookup?)
|
|
|
|
If the real API differs, adjust the skeleton above to use the real method names. The behavior and assertions must stay the same.
|
|
</action>
|
|
<verify>
|
|
<automated>cd /home/salva/Documents/apikey && go test ./pkg/providers/... -run 'TestTier1Count|TestTier2Count|TestAllPatternsCompile|TestAllProvidersHaveKeywords|TestTier1ProviderNames|TestTier2ProviderNames' -v -count=1</automated>
|
|
</verify>
|
|
<acceptance_criteria>
|
|
- `test -f pkg/providers/tier12_test.go`
|
|
- `grep -q 'expectedTier1' pkg/providers/tier12_test.go`
|
|
- `grep -q 'expectedTier2' pkg/providers/tier12_test.go`
|
|
- All six test functions present: `grep -c '^func Test' pkg/providers/tier12_test.go` >= 6
|
|
- `go test ./pkg/providers/... -run TestTier1Count -count=1` passes (Tier 1 = 12)
|
|
- `go test ./pkg/providers/... -run TestTier2Count -count=1` passes (Tier 2 = 14)
|
|
- `go test ./pkg/providers/... -run TestAllPatternsCompile -count=1` passes (all regex compile)
|
|
- `go test ./pkg/providers/... -count=1` passes (full suite green)
|
|
</acceptance_criteria>
|
|
<done>Guardrail test locks in 12 Tier 1 + 14 Tier 2 providers, regex compilation, and keyword presence. Future phase regressions that drop a provider will fail this test.</done>
|
|
</task>
|
|
|
|
</tasks>
|
|
|
|
<verification>
|
|
Final phase sanity:
|
|
- `go test ./... -count=1` passes
|
|
- `ls providers/*.yaml | wc -l` >= 27 (26 new + huggingface)
|
|
- `ls pkg/providers/definitions/*.yaml | wc -l` matches providers/ directory count
|
|
- `diff <(ls providers/*.yaml | xargs -n1 basename) <(ls pkg/providers/definitions/*.yaml | xargs -n1 basename)` returns empty
|
|
</verification>
|
|
|
|
<success_criteria>
|
|
- Registry loads exactly 12 Tier 1 + 14 Tier 2 providers (+ pre-existing tier-less huggingface)
|
|
- All patterns compile under RE2
|
|
- All providers have non-empty keywords
|
|
- Test file committed and passing
|
|
</success_criteria>
|
|
|
|
<output>
|
|
After completion, create `.planning/phases/02-tier-1-2-providers/02-05-SUMMARY.md`
|
|
</output>
|