8.2 KiB
phase, verified, status, score
| phase | verified | status | score |
|---|---|---|---|
| 03-tier-3-9-providers | 2026-04-05T00:00:00Z | passed | 4/4 success criteria verified |
Phase 3: Tier 3-9 Providers Verification Report
Phase Goal: All 108+ LLM provider definitions exist — specialized models, Chinese/regional providers, infrastructure gateways, emerging tools, code assistants, self-hosted runtimes, and enterprise platforms Verified: 2026-04-05 Status: passed Re-verification: No — initial verification
Goal Achievement
Observable Truths (from ROADMAP Success Criteria)
| # | Truth | Status | Evidence |
|---|---|---|---|
| 1 | keyhunter providers stats shows 108+ total providers across all tiers |
VERIFIED | Live CLI run: Total providers: 108 with T1=12, T2=14, T3=12, T4=16, T5=11, T6=15, T7=10, T8=10, T9=8 |
| 2 | Chinese/regional provider keys (DeepSeek, Zhipu, Moonshot, Qwen, Baidu, ByteDance) detected via keyword-based matching | VERIFIED | All 6 YAMLs exist with substantive keyword lists (env vars, domains, model names). DeepSeek/Moonshot/Qwen also have sk- prefix regex. Baidu/ByteDance keyword-only per Phase 2 false-positive mitigation. |
| 3 | Self-hosted provider definitions (Ollama, vLLM, LocalAI) include patterns for API key authentication | VERIFIED | All three YAMLs include API key authentication keywords: OLLAMA_API_KEY, VLLM_API_KEY, LOCALAI_API_KEY, plus SDK-specific markers (--api-key, vllm.entrypoints, go-skynet). Keyword-only detection is intentional per phase mitigation strategy for providers without documented key formats. |
| 4 | Enterprise providers include Salesforce, ServiceNow, SAP, Palantir, Databricks, Snowflake, Oracle, HPE | VERIFIED | All 8 Tier 9 YAMLs exist: salesforce-einstein, servicenow, sap-ai-core, palantir, databricks, snowflake, oracle-genai, hpe-greenlake |
Score: 4/4 success criteria verified
Registry Load Verification (Key Link: YAML → Aho-Corasick)
| From | To | Via | Status |
|---|---|---|---|
| providers/*.yaml (108 files) | reg.Stats().Total |
NewRegistry() embed.FS load |
WIRED — returns 108 live |
| pkg/providers/definitions/*.yaml | Aho-Corasick keyword automaton | NewRegistry() keyword indexing |
WIRED — TestTier39ProviderNames asserts per-name Get() + correct tier for all 63 T3-T9 providers |
| providers/*.yaml | pkg/providers/definitions/*.yaml | verbatim copy (dual-location) | WIRED — diff -q providers/ pkg/providers/definitions/ returns no diffs |
Required Artifacts
| Tier | Expected Count | Actual Count | Status |
|---|---|---|---|
| Tier 3 Specialized (PROV-03) | 12 | 12 | VERIFIED (perplexity, you, voyage, jina, unstructured, assemblyai, deepgram, elevenlabs, stability, runway, midjourney, huggingface) |
| Tier 4 Chinese/Regional (PROV-04) | 16 | 16 | VERIFIED (deepseek, zhipu, moonshot, qwen, baidu, bytedance, 01ai, minimax, baichuan, stepfun, sensetime, iflytek, tencent, siliconflow, 360ai, kuaishou) |
| Tier 5 Infrastructure (PROV-05) | 11 | 11 | VERIFIED (openrouter, litellm, cloudflare-ai, vercel-ai, portkey, helicone, martian, kong, bricksai, aether, notdiamond) |
| Tier 6 Emerging (PROV-06) | 15 | 15 | VERIFIED (reka, aleph-alpha, lamini, writer, jasper, typeface, comet, wandb, langsmith, pinecone, weaviate, qdrant, chroma, milvus, neon) |
| Tier 7 Code/Dev Tools (PROV-07) | 10 | 10 | VERIFIED (github-copilot, cursor, tabnine, codeium, sourcegraph, codewhisperer, replit-ai, codestral, watsonx, oracle-ai) |
| Tier 8 Self-Hosted (PROV-08) | 10 | 10 | VERIFIED (ollama, vllm, localai, lmstudio, llamacpp, gpt4all, text-gen-webui, tensorrt-llm, triton, jan) |
| Tier 9 Enterprise (PROV-09) | 8 | 8 | VERIFIED (salesforce-einstein, servicenow, sap-ai-core, palantir, databricks, snowflake, oracle-genai, hpe-greenlake) |
Each provider has dual-location files (providers/X.yaml AND pkg/providers/definitions/X.yaml).
Data-Flow Trace (Level 4)
| Artifact | Data Variable | Source | Produces Real Data | Status |
|---|---|---|---|---|
| pkg/providers/definitions/*.yaml (108 files) | Registry.providers map |
embed.FS.ReadDir("definitions") → yaml.Unmarshal in NewRegistry() |
Yes — reg.Stats().Total == 108 at runtime |
FLOWING |
| Registry keywords | Aho-Corasick automaton | NewRegistry() indexing step |
Yes — engine tests pass, no empty-keyword panics | FLOWING |
CLI providers stats |
reg.Stats() |
Calls NewRegistry() → prints real counts |
Yes — live output shows 108 with correct per-tier distribution | FLOWING |
Behavioral Spot-Checks
| Behavior | Command | Result | Status |
|---|---|---|---|
| Registry loads 108 providers | go run . providers stats |
Total providers: 108 |
PASS |
| Per-tier distribution correct | (same command) | T1=12, T2=14, T3=12, T4=16, T5=11, T6=15, T7=10, T8=10, T9=8 | PASS |
| Dual-location consistent | diff -q providers/ pkg/providers/definitions/ |
no diffs | PASS |
| Provider test suite green | go test ./pkg/providers/... -count=1 |
ok ... 0.971s |
PASS |
| Engine test suite green (no regression from 82 new providers) | go test ./pkg/engine/... -count=1 |
ok ... 0.287s |
PASS |
| Full test suite green | go test ./... -count=1 |
all packages ok (engine, providers, storage) | PASS |
| Tier 3-9 guardrail test exists | pkg/providers/tier39_test.go |
8 tier-count tests + TestTotalProviderCount + TestTier39ProviderNames asserting all 63 names per tier | PASS |
Requirements Coverage
| Requirement | Source | Description | Status | Evidence |
|---|---|---|---|---|
| PROV-03 | 03-02-PLAN (assumed) | 12 Tier 3 Specialized providers | SATISFIED | All 12 names in registry; tier 3: 12 from grep and TestTier3Count passes |
| PROV-04 | 03-01-PLAN | 16 Tier 4 Chinese/Regional providers | SATISFIED | All 16 names verified; TestTier4Count passes; keyword-only strategy applied per Phase 2 lessons |
| PROV-05 | 03-03-PLAN (assumed) | 11 Tier 5 Infrastructure/Gateway | SATISFIED | All 11 names; TestTier5Count passes |
| PROV-06 | 03-04-PLAN (assumed) | 15 Tier 6 Emerging/Niche | SATISFIED | All 15 names; TestTier6Count passes |
| PROV-07 | 03-05-PLAN (assumed) | 10 Tier 7 Code/Dev Tools | SATISFIED | All 10 names; TestTier7Count passes |
| PROV-08 | 03-06-PLAN (assumed) | 10 Tier 8 Self-Hosted | SATISFIED | All 10 names; TestTier8Count passes |
| PROV-09 | 03-07-PLAN (assumed) | 8 Tier 9 Enterprise | SATISFIED | All 8 names; TestTier9Count passes |
REQUIREMENTS.md already marks PROV-03 through PROV-09 as [x] checked.
Anti-Patterns Found
| File | Pattern | Severity | Impact |
|---|---|---|---|
| — | No TODO/FIXME/PLACEHOLDER in providers/ or pkg/providers/definitions/ | — | — |
| — | All 108 YAMLs have non-empty keywords: field (Aho-Corasick requirement) |
— | — |
Scan results: zero blocker, zero warning, zero info anti-patterns. Several providers have empty verify: stubs (url: "", valid_status: []) — this is intentional per PLAN design for providers without public verification endpoints (Baidu, ByteDance, SenseTime, iFlytek, Tencent, 360ai, Kuaishou, Ollama, vLLM, LocalAI, etc.) and is compatible with registry load (verify block is optional at runtime).
Gaps Summary
No gaps. All 4 ROADMAP success criteria verified, all 7 requirement IDs (PROV-03..PROV-09) satisfied, registry loads exactly 108 providers with correct per-tier distribution, dual-location diff-clean, comprehensive guardrail test (tier39_test.go) asserts structure, and full test suite passes. CLI providers stats produces the expected live output.
One interpretive note on Success Criterion 3: the literal wording says "patterns for API key authentication" for Ollama/vLLM/LocalAI. The actual definitions use keyword-only detection (OLLAMA_API_KEY, VLLM_API_KEY, LOCALAI_API_KEY, --api-key, etc.) rather than regex patterns. This is the documented and correct approach per Phase 2's false-positive mitigation lessons (03-CONTEXT.md lines 20-27) because these runtimes have no standardized key format. Counted as verified because the criterion's intent — detection of API key authentication in self-hosted runtimes — is achieved.
Verified: 2026-04-05 Verifier: Claude (gsd-verifier)