617199ba44
Adds guardrail summary and advances phase 02 state. Notes pre-existing Tier 2 regex over-match regression in pkg/engine as a phase-2 blocker to be handled in a follow-up plan.
116 lines
5.1 KiB
Markdown
116 lines
5.1 KiB
Markdown
---
|
||
phase: 02-tier-1-2-providers
|
||
plan: 05
|
||
subsystem: testing
|
||
tags: [providers, guardrail, integration-test, re2, aho-corasick]
|
||
|
||
requires:
|
||
- phase: 02-tier-1-2-providers (plans 01-04)
|
||
provides: 26 Tier 1+2 provider YAML definitions embedded in registry
|
||
provides:
|
||
- Compile-time guardrail locking Tier 1 (12) and Tier 2 (14) provider counts
|
||
- Runtime verification that every provider regex compiles under Go RE2
|
||
- Runtime verification that every provider has non-empty keywords (AC pre-filter)
|
||
affects: [03-tier-3-providers, all future phases touching pkg/providers]
|
||
|
||
tech-stack:
|
||
added: []
|
||
patterns:
|
||
- "Name-list guardrail: test enumerates expected provider names so silent deletions fail loudly"
|
||
|
||
key-files:
|
||
created:
|
||
- pkg/providers/tier12_test.go
|
||
modified: []
|
||
|
||
key-decisions:
|
||
- "Guardrail uses Registry.List() and Registry.Get() public API (no reflection / internal poking)"
|
||
- "Test scoped to pkg/providers only — unrelated pkg/engine regression pre-exists this plan"
|
||
|
||
patterns-established:
|
||
- "Tier-count regression test: lock phase deliverables with explicit count assertions"
|
||
- "Name-enumeration test: silent provider removal breaks TestTier{1,2}ProviderNames"
|
||
|
||
requirements-completed: [PROV-01, PROV-02]
|
||
|
||
duration: 2min
|
||
completed: 2026-04-05
|
||
---
|
||
|
||
# Phase 02 Plan 05: Tier 1+2 Provider Guardrail Test Summary
|
||
|
||
**Six-function integration test that locks 12 Tier 1 + 14 Tier 2 provider counts, RE2 regex compilation, and keyword presence against future regressions.**
|
||
|
||
## Performance
|
||
|
||
- **Duration:** ~2 min
|
||
- **Started:** 2026-04-05T11:14:00Z
|
||
- **Completed:** 2026-04-05T11:16:00Z
|
||
- **Tasks:** 1
|
||
- **Files modified:** 1
|
||
|
||
## Accomplishments
|
||
|
||
- Added `pkg/providers/tier12_test.go` with 6 guardrail tests
|
||
- All 6 tests pass against current registry state (12 Tier 1 + 14 Tier 2 providers)
|
||
- Full `pkg/providers/...` test suite remains green
|
||
- Phase 2 success criteria now machine-verified — any future drop of a provider from the registry fails CI loudly
|
||
|
||
## Task Commits
|
||
|
||
1. **Task 1: tier1/tier2 guardrail test** - `58f302b` (test)
|
||
|
||
_Note: Plan was a single TDD-style task. Because the target tests assert state already built by plans 02-01 through 02-04, RED was skipped — tests went directly to GREEN and stayed there. No refactor needed._
|
||
|
||
## Files Created/Modified
|
||
|
||
- `pkg/providers/tier12_test.go` — 103 lines, 6 test functions asserting Tier 1/2 counts, regex compilation, keyword non-emptiness, and expected provider names
|
||
|
||
## Decisions Made
|
||
|
||
- Used `Registry.List()` rather than a hypothetical `All()`/`Each()` helper — matches the actual public API
|
||
- Used `reg.Get(name)` for name lookups — already exported
|
||
- Kept expected name lists as package-level `var` so future phases can reuse them if needed
|
||
|
||
## Deviations from Plan
|
||
|
||
None in scope. Plan executed exactly as written.
|
||
|
||
## Issues Encountered
|
||
|
||
### Pre-existing regression (OUT OF SCOPE — documented per orchestrator instructions)
|
||
|
||
`go test ./pkg/engine/...` currently fails with `TestScannerPipelineOpenAI`, `TestScannerPipelineAnthropic`, and `TestScannerPipelineMultipleKeys`. Root cause: several Tier 2 provider regex patterns introduced in Wave 1 (plans 02-03 and 02-04 — ai21, baseten, cerebrium, cohere, deepinfra, fireworks, friendli, inflection, lepton, mistral, novita, octoai, sambanova) are too broad and match the OpenAI/Anthropic fixtures in `testdata/samples/`, producing 14–15 findings where exactly 1 is expected. `TestScannerPipelineMultipleKeys` additionally does not even find the anthropic provider by name in the over-matched result set.
|
||
|
||
Per orchestrator instructions, this regression was **NOT** fixed in plan 02-05. It pre-existed this plan's creation and requires either:
|
||
1. Tightening the over-broad Tier 2 regexes (prefixes, length bounds, character classes), or
|
||
2. Updating the engine test expectations to tolerate multi-provider matches on ambiguous keys.
|
||
|
||
This is a blocker for the phase-level success criterion `go test ./... -count=1 passes` and should be handled as a follow-up plan or verifier-driven fix in Phase 2 closeout.
|
||
|
||
The guardrail test added in this plan **does not depend on** and **is not affected by** the engine regression — it runs cleanly via `go test ./pkg/providers/... -count=1`.
|
||
|
||
## User Setup Required
|
||
|
||
None — pure test code, no external services.
|
||
|
||
## Next Phase Readiness
|
||
|
||
**Ready for phase closeout verification** with one blocker:
|
||
|
||
- Phase 2 guardrail in place (this plan) — DONE
|
||
- Pre-existing Tier 2 regex over-match breaking `pkg/engine` tests — BLOCKER for full `go test ./...`
|
||
|
||
Phase 3 (Tier 3 providers) should NOT start until the engine regression is resolved, because adding more providers on top of the current over-broad patterns will compound the problem.
|
||
|
||
## Self-Check: PASSED
|
||
|
||
- File exists: `pkg/providers/tier12_test.go` — FOUND
|
||
- Commit exists: `58f302b` — FOUND
|
||
- Test execution: `go test ./pkg/providers/... -count=1` — PASSING
|
||
- Six test functions present and passing: TestTier1Count, TestTier2Count, TestAllPatternsCompile, TestAllProvidersHaveKeywords, TestTier1ProviderNames, TestTier2ProviderNames
|
||
|
||
---
|
||
*Phase: 02-tier-1-2-providers*
|
||
*Completed: 2026-04-05*
|