docs(03-tier-3-9-providers): create phase plan

2026-04-05 14:39:54 +03:00
parent 19f55ffeb3
commit a318b9d89f
9 changed files with 3231 additions and 1 deletions
--- a/.planning/phases/03-tier-3-9-providers/03-02-PLAN.md
+++ b/.planning/phases/03-tier-3-9-providers/03-02-PLAN.md
@@ -0,0 +1,418 @@
+---
+phase: 03-tier-3-9-providers
+plan: 02
+type: execute
+wave: 1
+depends_on: []
+files_modified:
+  - providers/perplexity.yaml
+  - providers/you.yaml
+  - providers/voyage.yaml
+  - providers/jina.yaml
+  - providers/unstructured.yaml
+  - providers/assemblyai.yaml
+  - providers/deepgram.yaml
+  - providers/elevenlabs.yaml
+  - providers/stability.yaml
+  - providers/runway.yaml
+  - providers/midjourney.yaml
+  - pkg/providers/definitions/perplexity.yaml
+  - pkg/providers/definitions/you.yaml
+  - pkg/providers/definitions/voyage.yaml
+  - pkg/providers/definitions/jina.yaml
+  - pkg/providers/definitions/unstructured.yaml
+  - pkg/providers/definitions/assemblyai.yaml
+  - pkg/providers/definitions/deepgram.yaml
+  - pkg/providers/definitions/elevenlabs.yaml
+  - pkg/providers/definitions/stability.yaml
+  - pkg/providers/definitions/runway.yaml
+  - pkg/providers/definitions/midjourney.yaml
+autonomous: true
+requirements: [PROV-03]
+must_haves:
+  truths:
+    - "11 new Tier 3 Specialized provider YAMLs load (huggingface already exists → 12 total)"
+    - "Providers with documented prefixes (pplx-, xai-voyage, sk-, jina_) use tight regex"
+    - "Voice/image providers without documented prefixes use keyword-only"
+  artifacts:
+    - path: "providers/perplexity.yaml"
+      provides: "Perplexity pplx- prefix pattern"
+      contains: "pplx-"
+    - path: "providers/elevenlabs.yaml"
+      provides: "ElevenLabs API key keyword detection"
+      contains: "elevenlabs"
+    - path: "providers/assemblyai.yaml"
+      provides: "AssemblyAI keyword detection"
+      contains: "assemblyai"
+  key_links:
+    - from: "provider keywords[]"
+      to: "Registry Aho-Corasick automaton"
+      via: "NewRegistry()"
+      pattern: "keywords"
+---
+
+<objective>
+Create 11 Tier 3 Specialized LLM/AI provider YAMLs — search (Perplexity, You.com), embeddings (Voyage, Jina, Unstructured), voice (AssemblyAI, Deepgram, ElevenLabs), and image/video (Stability, Runway, Midjourney). Huggingface is pre-existing from Phase 2 and must NOT be recreated. Total Tier 3 = 12 after this plan.
+
+Purpose: Satisfy PROV-03 (12 Tier 3 Specialized providers).
+
+Output: 22 YAML files (11 providers × 2 locations).
+
+Addresses PROV-03.
+</objective>
+
+<execution_context>
+@$HOME/.claude/get-shit-done/workflows/execute-plan.md
+@$HOME/.claude/get-shit-done/templates/summary.md
+</execution_context>
+
+<context>
+@.planning/ROADMAP.md
+@.planning/phases/03-tier-3-9-providers/03-CONTEXT.md
+@pkg/providers/schema.go
+@providers/huggingface.yaml
+@providers/cohere.yaml
+
+<interfaces>
+Schema: pkg/providers/schema.go. No `category` field. Confidence: high|medium|low.
+Patterns field may be omitted for keyword-only providers. Keywords ≥3 required.
+Dual-location required (providers/ + pkg/providers/definitions/).
+RE2 regex only.
+
+IMPORTANT: providers/huggingface.yaml ALREADY EXISTS (tier: 3). Do NOT modify or recreate it.
+</interfaces>
+</context>
+
+<tasks>
+
+<task type="auto">
+  <name>Task 1: Search + embeddings providers (Perplexity, You.com, Voyage, Jina, Unstructured, AssemblyAI)</name>
+  <files>providers/perplexity.yaml, providers/you.yaml, providers/voyage.yaml, providers/jina.yaml, providers/unstructured.yaml, providers/assemblyai.yaml, pkg/providers/definitions/perplexity.yaml, pkg/providers/definitions/you.yaml, pkg/providers/definitions/voyage.yaml, pkg/providers/definitions/jina.yaml, pkg/providers/definitions/unstructured.yaml, pkg/providers/definitions/assemblyai.yaml</files>
+  <read_first>
+    - pkg/providers/schema.go
+    - providers/huggingface.yaml (existing tier 3 reference — DO NOT MODIFY)
+  </read_first>
+  <action>
+Create each file in providers/ AND copy verbatim to pkg/providers/definitions/.
+
+providers/perplexity.yaml:
+```yaml
+format_version: 1
+name: perplexity
+display_name: Perplexity AI
+tier: 3
+last_verified: "2026-04-05"
+keywords:
+  - "perplexity"
+  - "PERPLEXITY_API_KEY"
+  - "pplx-"
+  - "api.perplexity.ai"
+patterns:
+  - regex: 'pplx-[A-Za-z0-9]{48,}'
+    entropy_min: 4.0
+    confidence: high
+verify:
+  method: POST
+  url: https://api.perplexity.ai/chat/completions
+  headers:
+    Authorization: "Bearer {KEY}"
+  valid_status: [200, 400]
+  invalid_status: [401, 403]
+```
+
+providers/you.yaml:
+```yaml
+format_version: 1
+name: you
+display_name: You.com
+tier: 3
+last_verified: "2026-04-05"
+keywords:
+  - "you.com"
+  - "YDC_API_KEY"
+  - "YOU_API_KEY"
+  - "api.ydc-index.io"
+verify:
+  method: GET
+  url: https://api.ydc-index.io/search
+  headers:
+    X-API-Key: "{KEY}"
+  valid_status: [200, 400]
+  invalid_status: [401, 403]
+```
+
+providers/voyage.yaml:
+```yaml
+format_version: 1
+name: voyage
+display_name: Voyage AI
+tier: 3
+last_verified: "2026-04-05"
+keywords:
+  - "voyage"
+  - "VOYAGE_API_KEY"
+  - "voyageai"
+  - "api.voyageai.com"
+patterns:
+  - regex: 'pa-[A-Za-z0-9_\-]{40,}'
+    entropy_min: 4.0
+    confidence: medium
+verify:
+  method: POST
+  url: https://api.voyageai.com/v1/embeddings
+  headers:
+    Authorization: "Bearer {KEY}"
+  valid_status: [200, 400]
+  invalid_status: [401, 403]
+```
+
+providers/jina.yaml:
+```yaml
+format_version: 1
+name: jina
+display_name: Jina AI
+tier: 3
+last_verified: "2026-04-05"
+keywords:
+  - "jina"
+  - "JINA_API_KEY"
+  - "api.jina.ai"
+  - "jinaai"
+patterns:
+  - regex: 'jina_[A-Za-z0-9]{40,}'
+    entropy_min: 4.0
+    confidence: high
+verify:
+  method: GET
+  url: https://api.jina.ai/v1/models
+  headers:
+    Authorization: "Bearer {KEY}"
+  valid_status: [200]
+  invalid_status: [401, 403]
+```
+
+providers/unstructured.yaml:
+```yaml
+format_version: 1
+name: unstructured
+display_name: Unstructured.io
+tier: 3
+last_verified: "2026-04-05"
+keywords:
+  - "unstructured"
+  - "UNSTRUCTURED_API_KEY"
+  - "api.unstructured.io"
+  - "unstructuredio"
+verify:
+  method: GET
+  url: https://api.unstructured.io/general/v0/general
+  headers:
+    unstructured-api-key: "{KEY}"
+  valid_status: [200, 400]
+  invalid_status: [401, 403]
+```
+
+providers/assemblyai.yaml:
+```yaml
+format_version: 1
+name: assemblyai
+display_name: AssemblyAI
+tier: 3
+last_verified: "2026-04-05"
+keywords:
+  - "assemblyai"
+  - "ASSEMBLYAI_API_KEY"
+  - "api.assemblyai.com"
+  - "assembly.ai"
+patterns:
+  - regex: '[a-f0-9]{32}'
+    entropy_min: 4.0
+    confidence: low
+verify:
+  method: GET
+  url: https://api.assemblyai.com/v2/transcript
+  headers:
+    authorization: "{KEY}"
+  valid_status: [200]
+  invalid_status: [401, 403]
+```
+
+Copy all 6 files verbatim to pkg/providers/definitions/.
+  </action>
+  <verify>
+    <automated>cd /home/salva/Documents/apikey && for f in perplexity you voyage jina unstructured assemblyai; do diff providers/$f.yaml pkg/providers/definitions/$f.yaml || exit 1; done && go test ./pkg/providers/... -count=1 && go test ./pkg/engine/... -count=1</automated>
+  </verify>
+  <acceptance_criteria>
+    - All 12 files exist (6 providers × 2 locations)
+    - `grep -q 'pplx-' providers/perplexity.yaml`
+    - `grep -q 'jina_' providers/jina.yaml`
+    - `grep -q 'api.voyageai.com' providers/voyage.yaml`
+    - `diff providers/perplexity.yaml pkg/providers/definitions/perplexity.yaml` returns no diff
+    - `go test ./pkg/providers/... -count=1` passes
+    - `go test ./pkg/engine/... -count=1` passes
+  </acceptance_criteria>
+  <done>6 search/embeddings providers dual-located.</done>
+</task>
+
+<task type="auto">
+  <name>Task 2: Voice + image/video providers (Deepgram, ElevenLabs, Stability, Runway, Midjourney)</name>
+  <files>providers/deepgram.yaml, providers/elevenlabs.yaml, providers/stability.yaml, providers/runway.yaml, providers/midjourney.yaml, pkg/providers/definitions/deepgram.yaml, pkg/providers/definitions/elevenlabs.yaml, pkg/providers/definitions/stability.yaml, pkg/providers/definitions/runway.yaml, pkg/providers/definitions/midjourney.yaml</files>
+  <read_first>
+    - pkg/providers/schema.go
+  </read_first>
+  <action>
+providers/deepgram.yaml:
+```yaml
+format_version: 1
+name: deepgram
+display_name: Deepgram
+tier: 3
+last_verified: "2026-04-05"
+keywords:
+  - "deepgram"
+  - "DEEPGRAM_API_KEY"
+  - "api.deepgram.com"
+  - "dg-api"
+patterns:
+  - regex: '[a-f0-9]{40}'
+    entropy_min: 4.0
+    confidence: low
+verify:
+  method: GET
+  url: https://api.deepgram.com/v1/projects
+  headers:
+    Authorization: "Token {KEY}"
+  valid_status: [200]
+  invalid_status: [401, 403]
+```
+
+providers/elevenlabs.yaml:
+```yaml
+format_version: 1
+name: elevenlabs
+display_name: ElevenLabs
+tier: 3
+last_verified: "2026-04-05"
+keywords:
+  - "elevenlabs"
+  - "ELEVENLABS_API_KEY"
+  - "ELEVEN_API_KEY"
+  - "XI_API_KEY"
+  - "api.elevenlabs.io"
+patterns:
+  - regex: '[a-f0-9]{32}'
+    entropy_min: 4.0
+    confidence: low
+verify:
+  method: GET
+  url: https://api.elevenlabs.io/v1/user
+  headers:
+    xi-api-key: "{KEY}"
+  valid_status: [200]
+  invalid_status: [401, 403]
+```
+
+providers/stability.yaml:
+```yaml
+format_version: 1
+name: stability
+display_name: Stability AI
+tier: 3
+last_verified: "2026-04-05"
+keywords:
+  - "stability"
+  - "STABILITY_API_KEY"
+  - "STABILITY_KEY"
+  - "api.stability.ai"
+  - "stable-diffusion"
+patterns:
+  - regex: 'sk-[A-Za-z0-9]{48}'
+    entropy_min: 4.0
+    confidence: medium
+verify:
+  method: GET
+  url: https://api.stability.ai/v1/user/account
+  headers:
+    Authorization: "Bearer {KEY}"
+  valid_status: [200]
+  invalid_status: [401, 403]
+```
+
+providers/runway.yaml:
+```yaml
+format_version: 1
+name: runway
+display_name: Runway
+tier: 3
+last_verified: "2026-04-05"
+keywords:
+  - "runway"
+  - "runwayml"
+  - "RUNWAY_API_KEY"
+  - "RUNWAYML_API_SECRET"
+  - "api.runwayml.com"
+  - "dev.runwayml.com"
+verify:
+  method: GET
+  url: https://api.dev.runwayml.com/v1/tasks
+  headers:
+    Authorization: "Bearer {KEY}"
+  valid_status: [200, 400]
+  invalid_status: [401, 403]
+```
+
+providers/midjourney.yaml:
+```yaml
+format_version: 1
+name: midjourney
+display_name: Midjourney
+tier: 3
+last_verified: "2026-04-05"
+keywords:
+  - "midjourney"
+  - "MIDJOURNEY_API_KEY"
+  - "MJ_API_KEY"
+  - "midjourney.com"
+  - "useapi.net"
+verify:
+  method: GET
+  url: ""
+  headers: {}
+  valid_status: []
+  invalid_status: []
+```
+
+Copy all 5 files verbatim to pkg/providers/definitions/.
+  </action>
+  <verify>
+    <automated>cd /home/salva/Documents/apikey && for f in deepgram elevenlabs stability runway midjourney; do diff providers/$f.yaml pkg/providers/definitions/$f.yaml || exit 1; done && go test ./pkg/providers/... -count=1 && go test ./pkg/engine/... -count=1</automated>
+  </verify>
+  <acceptance_criteria>
+    - All 10 files exist
+    - `grep -q 'XI_API_KEY' providers/elevenlabs.yaml`
+    - `grep -q 'stable-diffusion' providers/stability.yaml`
+    - `grep -q 'runwayml' providers/runway.yaml`
+    - `grep -L 'patterns:' providers/runway.yaml providers/midjourney.yaml` returns both (keyword-only)
+    - Total Tier 3 count: `grep -l 'tier: 3' providers/*.yaml | wc -l` → 12 (11 new + pre-existing huggingface)
+    - `go test ./pkg/providers/... -count=1` passes
+    - `go test ./pkg/engine/... -count=1` passes
+  </acceptance_criteria>
+  <done>All 11 new Tier 3 Specialized providers dual-located. PROV-03 satisfied (12 total with pre-existing huggingface).</done>
+</task>
+
+</tasks>
+
+<verification>
+`grep -l 'tier: 3' providers/*.yaml | wc -l` returns 12.
+</verification>
+
+<success_criteria>
+- 11 new Tier 3 providers added (huggingface pre-existing, total 12)
+- All dual-located
+- Documented prefixes use tight regex; undocumented use keyword-only
+- No engine regression
+</success_criteria>
+
+<output>
+After completion, create `.planning/phases/03-tier-3-9-providers/03-02-SUMMARY.md`
+</output>