salvacybersec/keyhunter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(03-02): complete Tier 3 specialized providers plan

Browse Source 
11 new Tier 3 providers (search, embeddings, voice, image/video). PROV-03 satisfied.
This commit is contained in:
salvacybersec
2026-04-05 14:43:01 +03:00
parent a019ba9a3d
commit e9948f4ccf
4 changed files with 119 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
.planning/REQUIREMENTS.md
View File

@@ -21,13 +21,13 @@ Requirements for initial release. Each maps to roadmap phases.
- [x] **PROV-01**: 12 Tier 1 Frontier provider YAML definitions (OpenAI, Anthropic, Google AI, Vertex, AWS Bedrock, Azure OpenAI, Meta AI, xAI, Cohere, Mistral, Inflection, AI21)
- [x] **PROV-02**: 14 Tier 2 Inference Platform provider definitions (Together, Fireworks, Groq, Replicate, Anyscale, DeepInfra, Lepton, Modal, Baseten, Cerebrium, NovitaAI, Sambanova, OctoAI, Friendli)
- [ ] **PROV-03**: 12 Tier 3 Specialized provider definitions (Perplexity, You.com, Voyage, Jina, Unstructured, AssemblyAI, Deepgram, ElevenLabs, Stability, Runway, Midjourney, HuggingFace)
- [x] **PROV-03**: 12 Tier 3 Specialized provider definitions (Perplexity, You.com, Voyage, Jina, Unstructured, AssemblyAI, Deepgram, ElevenLabs, Stability, Runway, Midjourney, HuggingFace)
- [ ] **PROV-04**: 16 Tier 4 Chinese/Regional provider definitions (DeepSeek, Baichuan, Zhipu, Moonshot, Yi, Qwen, Baidu, ByteDance, SenseTime, iFlytek, MiniMax, Stepfun, 360 AI, Kuaishou, Tencent, SiliconFlow)
- [ ] **PROV-05**: 11 Tier 5 Infrastructure/Gateway provider definitions (Cloudflare AI, Vercel AI, LiteLLM, Portkey, Helicone, OpenRouter, Martian, Kong, BricksAI, Aether, Not Diamond)
- [ ] **PROV-06**: 15 Tier 6 Emerging/Niche provider definitions (Reka, Aleph Alpha, Writer, Jasper, Typeface, Comet, W&B, LangSmith, Pinecone, Weaviate, Qdrant, Chroma, Milvus, Neon, Lamini)
- [x] **PROV-07**: 10 Tier 7 Code/Dev Tools provider definitions (GitHub Copilot, Cursor, Tabnine, Codeium, Sourcegraph, CodeWhisperer, Replit AI, Codestral, watsonx, Oracle AI)
- [ ] **PROV-08**: 10 Tier 8 Self-Hosted provider definitions (Ollama, vLLM, LocalAI, LM Studio, llama.cpp, GPT4All, text-gen-webui, TensorRT-LLM, Triton, Jan AI)
- [ ] **PROV-09**: 8 Tier 9 Enterprise provider definitions (Salesforce Einstein, ServiceNow, SAP AI Core, Palantir, Databricks, Snowflake, Oracle GenAI, HPE GreenLake)
- [x] **PROV-09**: 8 Tier 9 Enterprise provider definitions (Salesforce Einstein, ServiceNow, SAP AI Core, Palantir, Databricks, Snowflake, Oracle GenAI, HPE GreenLake)
- [x] **PROV-10**: Provider YAML schema includes format_version and last_verified date for pattern health tracking
### Input Sources

4
.planning/ROADMAP.md
View File

@@ -83,11 +83,11 @@ Plans:
Plans:
- [ ] 03-01-PLAN.md — Tier 4 Chinese/regional providers (DeepSeek, Zhipu, Moonshot, Qwen, Baidu, ByteDance, 01.AI, MiniMax, Baichuan, StepFun, SenseTime, iFlytek, Tencent, SiliconFlow, 360 AI, Kuaishou)
- [ ] 03-02-PLAN.md — Tier 3 Specialized (Perplexity, You.com, Voyage, Jina, Unstructured, AssemblyAI, Deepgram, ElevenLabs, Stability, Runway, Midjourney)
- [x] 03-02-PLAN.md — Tier 3 Specialized (Perplexity, You.com, Voyage, Jina, Unstructured, AssemblyAI, Deepgram, ElevenLabs, Stability, Runway, Midjourney)
- [x] 03-03-PLAN.md — Tier 5 Infrastructure/Gateway (OpenRouter, LiteLLM, Cloudflare AI, Vercel AI, Portkey, Helicone, Martian, Kong, BricksAI, Aether, Not Diamond)
- [x] 03-04-PLAN.md — Tier 7 Code/Dev Tools (GitHub Copilot, Cursor, Tabnine, Codeium, Sourcegraph, CodeWhisperer, Replit AI, Codestral, watsonx, Oracle AI)
- [ ] 03-05-PLAN.md — Tier 8 Self-Hosted runtimes (Ollama, vLLM, LocalAI, LM Studio, llama.cpp, GPT4All, text-gen-webui, TensorRT-LLM, Triton, Jan)
- [ ] 03-06-PLAN.md — Tier 9 Enterprise (Salesforce Einstein, ServiceNow, SAP AI Core, Palantir, Databricks, Snowflake, Oracle GenAI, HPE GreenLake)
- [x] 03-06-PLAN.md — Tier 9 Enterprise (Salesforce Einstein, ServiceNow, SAP AI Core, Palantir, Databricks, Snowflake, Oracle GenAI, HPE GreenLake)
- [ ] 03-07-PLAN.md — Tier 6 Emerging/Niche (Reka, Aleph Alpha, Lamini, Writer, Jasper, Typeface, Comet, W&B, LangSmith, Pinecone, Weaviate, Qdrant, Chroma, Milvus, Neon)
- [ ] 03-08-PLAN.md — Tier 3-9 guardrail test: lock 108 total providers, per-tier counts, and name sets

15
.planning/STATE.md
View File

@@ -3,14 +3,14 @@ gsd_state_version: 1.0
milestone: v1.0
milestone_name: milestone
status: executing
stopped_at: Completed 02-tier-1-2-providers 02-05-PLAN.md
last_updated: "2026-04-05T11:42:39.202Z"
stopped_at: Completed 03-06-PLAN.md
last_updated: "2026-04-05T11:42:57.673Z"
last_activity: 2026-04-05
progress:
total_phases: 18
completed_phases: 2
total_plans: 18
completed_plans: 12
completed_plans: 14
percent: 20
---
@@ -26,7 +26,7 @@ See: .planning/PROJECT.md (updated 2026-04-04)
## Current Position
Phase: 03 (tier-3-9-providers) — EXECUTING
Plan: 2 of 8
Plan: 4 of 8
Status: Ready to execute
Last activity: 2026-04-05
@@ -61,6 +61,8 @@ Progress: [██░░░░░░░░] 20%
| Phase 02-tier-1-2-providers P04 | 1min | 2 tasks tasks | 14 files files |
| Phase 02-tier-1-2-providers P05 | 2min | 1 tasks | 1 files |
| Phase 03-tier-3-9-providers P04 | 3m | 2 tasks | 20 files |
| Phase 03-tier-3-9-providers P02 | 70 | 2 tasks | 22 files |
| Phase 03-tier-3-9-providers P06 | 3m | 2 tasks | 16 files |
## Accumulated Context
@@ -79,6 +81,7 @@ Recent decisions affecting current work:
- [Phase 01-foundation]: Per-installation salt via settings table -- no hardcoded salt in production code
- [Phase 01-foundation]: Exit code semantics: 0=clean, 1=keys-found, 2=error for CI/CD integration
- [Phase 02-tier-1-2-providers]: AWS Bedrock verify URL left empty — SigV4 signing deferred to Phase 5 verification engine
- [Phase 03-tier-3-9-providers]: Keyword-only detection for providers without documented key prefixes (You.com, Unstructured, Runway, Midjourney) to avoid false positives.
### Pending Todos
@@ -93,6 +96,6 @@ None yet.
## Session Continuity
Last session: 2026-04-05T11:16:08.292Z
Stopped at: Completed 02-tier-1-2-providers 02-05-PLAN.md
Last session: 2026-04-05T11:42:57.670Z
Stopped at: Completed 03-06-PLAN.md
Resume file: None

106
.planning/phases/03-tier-3-9-providers/03-02-SUMMARY.md Normal file
View File

@@ -0,0 +1,106 @@
---
phase: 03-tier-3-9-providers
plan: 02
subsystem: providers
tags: [providers, tier-3, specialized, voice, image, embeddings]
requires: [PROV-10-schema, embed-loader]
provides: [PROV-03]
affects: [pkg/providers/registry, engine/detector]
tech_stack_added: []
patterns: [dual-location-yaml, keyword-only-fallback, tight-prefix-regex]
files_created:
- providers/perplexity.yaml
- providers/you.yaml
- providers/voyage.yaml
- providers/jina.yaml
- providers/unstructured.yaml
- providers/assemblyai.yaml
- providers/deepgram.yaml
- providers/elevenlabs.yaml
- providers/stability.yaml
- providers/runway.yaml
- providers/midjourney.yaml
- pkg/providers/definitions/perplexity.yaml
- pkg/providers/definitions/you.yaml
- pkg/providers/definitions/voyage.yaml
- pkg/providers/definitions/jina.yaml
- pkg/providers/definitions/unstructured.yaml
- pkg/providers/definitions/assemblyai.yaml
- pkg/providers/definitions/deepgram.yaml
- pkg/providers/definitions/elevenlabs.yaml
- pkg/providers/definitions/stability.yaml
- pkg/providers/definitions/runway.yaml
- pkg/providers/definitions/midjourney.yaml
files_modified: []
decisions:
- "Providers without documented key prefixes (You.com, Unstructured, Runway, Midjourney) use keyword-only detection (no regex) to avoid Phase 2 false-positive regression."
- "Providers with documented prefixes (Perplexity pplx-, Jina jina_, Voyage pa-, Stability sk-) use tight regex with high/medium confidence."
- "ElevenLabs/Deepgram/AssemblyAI use hex alphanumeric patterns with low confidence + entropy_min 4.0 — keyword pre-filter guards against noise."
- "Midjourney has no official API; verify block uses empty URL as sentinel (no active verification possible)."
metrics:
duration_seconds: 70
tasks_completed: 2
files_changed: 22
completed_at: "2026-04-05T11:42:06Z"
---
# Phase 3 Plan 02: Tier 3 Specialized Providers Summary
11 specialized Tier 3 LLM/AI providers added (search, embeddings, voice, image/video) across dual-location YAML, bringing total Tier 3 count to 12 with pre-existing huggingface.
## What Was Built
### Task 1: Search + Embeddings (commit `7ad9588`)
Added 6 providers covering search APIs and embedding/document-processing services:
| Provider | Type | Detection |
|----------|------|-----------|
| Perplexity AI | Search LLM | `pplx-[A-Za-z0-9]{48,}` (high) |
| You.com | Search | keyword-only |
| Voyage AI | Embeddings | `pa-[A-Za-z0-9_\-]{40,}` (medium) |
| Jina AI | Embeddings | `jina_[A-Za-z0-9]{40,}` (high) |
| Unstructured.io | Doc processing | keyword-only |
| AssemblyAI | Voice (STT) | `[a-f0-9]{32}` (low) |
### Task 2: Voice + Image/Video (commit `0ac12e5`)
Added 5 providers covering speech, image, and video generation:
| Provider | Type | Detection |
|----------|------|-----------|
| Deepgram | Voice (STT) | `[a-f0-9]{40}` (low) |
| ElevenLabs | Voice (TTS) | `[a-f0-9]{32}` (low), `XI_API_KEY` |
| Stability AI | Image | `sk-[A-Za-z0-9]{48}` (medium) |
| Runway | Video | keyword-only |
| Midjourney | Image | keyword-only (no official API) |
All 11 provider YAMLs dual-located (`providers/` + `pkg/providers/definitions/`) to satisfy the embed loader contract.
## Key Decisions
- **Keyword-only where no documented format exists.** Per Phase 3 lessons-learned, providers without distinctive prefixes (You.com, Unstructured, Runway, Midjourney) rely solely on keyword pre-filtering to avoid false positives.
- **Tight regex for documented prefixes.** Perplexity (`pplx-`), Jina (`jina_`), Voyage (`pa-`), Stability (`sk-`) use prefix-anchored regex with high/medium confidence.
- **Low-confidence hex patterns backed by keyword pre-filter.** ElevenLabs, Deepgram, and AssemblyAI use hex-alphanumeric regex (32 or 40 chars) with `confidence: low` and `entropy_min: 4.0` — the Aho-Corasick keyword filter ensures these only fire on matched contexts.
- **Midjourney verify sentinel.** Midjourney has no first-party API; VerifySpec uses empty URL/status fields as a sentinel for "cannot actively verify."
## Verification
- `go test ./pkg/providers/... -count=1` → **PASS**
- `go test ./pkg/engine/... -count=1` → **PASS**
- `diff providers/<name>.yaml pkg/providers/definitions/<name>.yaml` for all 11 providers → identical
- `grep -l 'tier: 3' providers/*.yaml | wc -l` → **12** (PROV-03 satisfied)
## Deviations from Plan
None — plan executed exactly as written.
## Requirements Satisfied
- **PROV-03**: 12 Tier 3 Specialized providers (11 new + pre-existing huggingface)
## Self-Check: PASSED
- All 22 files present on disk.
- Commits `7ad9588` and `0ac12e5` exist on current branch.
- `go test ./pkg/providers/... ./pkg/engine/...` green after each task.