merge: phase 18 API+SSE

.planning/phases/18-web-dashboard/18-02-SUMMARY.md

@@ -0,0 +1,131 @@
+---
+phase: 18-web-dashboard
+plan: 02
+subsystem: api
+tags: [chi, rest-api, sse, json, http, server-sent-events]
+
+requires:
+  - phase: 01-foundation
+    provides: "storage DB, providers registry, encryption"
+  - phase: 08-dork-engine
+    provides: "dorks registry and custom dork storage"
+  - phase: 09-osint-infrastructure
+    provides: "recon engine"
+provides:
+  - "REST API at /api/v1/* for keys, providers, scan, recon, dorks, config"
+  - "SSE hub for live scan/recon progress streaming"
+  - "Server struct with dependency injection for all web handlers"
+affects: [18-web-dashboard, serve-command]
+
+tech-stack:
+  added: [chi-v5]
+  patterns: [api-json-wrappers, sse-hub-broadcast, dependency-injected-server]
+
+key-files:
+  created:
+    - pkg/web/server.go
+    - pkg/web/api.go
+    - pkg/web/sse.go
+    - pkg/web/api_test.go
+    - pkg/web/sse_test.go
+  modified:
+    - pkg/storage/schema.sql
+    - go.mod
+    - go.sum
+
+key-decisions:
+  - "JSON wrapper structs (apiKey, apiProvider, apiDork) with explicit JSON tags since domain structs only have yaml tags"
+  - "API never exposes raw key values -- KeyValue always empty string in JSON responses"
+  - "Single SSEHub shared between scan and recon progress endpoints, events distinguished by Type prefix"
+
+patterns-established:
+  - "API wrapper pattern: domain structs -> apiX structs with JSON tags for consistent camelCase API"
+  - "writeJSON/readJSON helpers for DRY HTTP response handling"
+  - "ServerConfig struct for dependency injection into all web handlers"
+
+requirements-completed: [WEB-03, WEB-09, WEB-11]
+
+duration: 7min
+completed: 2026-04-06
+---
+
+# Phase 18 Plan 02: REST API + SSE Hub Summary
+
+**Complete REST API at /api/v1/* with 14 endpoints (keys, providers, scan, recon, dorks, config) plus SSE hub for live event streaming**
+
+## Performance
+
+- **Duration:** 7 min
+- **Started:** 2026-04-06T14:59:58Z
+- **Completed:** 2026-04-06T15:06:51Z
+- **Tasks:** 2
+- **Files modified:** 7
+
+## Accomplishments
+- Full REST API with 14 endpoints covering stats, keys CRUD, providers, scan/recon triggers, dorks, and config
+- SSE hub with subscribe/unsubscribe/broadcast lifecycle and non-blocking buffered channels
+- 23 passing tests (16 API + 7 SSE) covering happy paths and error cases
+
+## Task Commits
+
+Each task was committed atomically:
+
+1. **Task 1: REST API handlers for /api/v1/*** - `76601b1` (feat)
+2. **Task 2: SSE hub for live scan/recon progress** - `d557c73` (feat)
+
+## Files Created/Modified
+- `pkg/web/server.go` - Server struct with ServerConfig dependency injection
+- `pkg/web/api.go` - All 14 REST API handlers with JSON wrapper types
+- `pkg/web/sse.go` - SSEHub with Subscribe/Unsubscribe/Broadcast + HTTP handlers
+- `pkg/web/api_test.go` - 16 tests for all API endpoints
+- `pkg/web/sse_test.go` - 7 tests for SSE hub lifecycle and HTTP streaming
+- `pkg/storage/schema.sql` - Resolved merge conflict (HEAD version kept)
+- `go.mod` / `go.sum` - Added chi v5.2.5
+
+## Decisions Made
+- JSON wrapper structs (apiKey, apiProvider, apiDork) with explicit JSON tags since domain structs only have yaml tags -- ensures consistent camelCase JSON API
+- API never exposes raw key values -- KeyValue always empty string in JSON responses for security
+- Single SSEHub shared between scan and recon progress endpoints, events distinguished by Type prefix (scan:*, recon:*)
+- DisallowUnknownFields removed from readJSON to avoid overly strict request parsing
+
+## Deviations from Plan
+
+### Auto-fixed Issues
+
+**1. [Rule 3 - Blocking] Resolved merge conflict in schema.sql**
+- **Found during:** Task 1
+- **Issue:** schema.sql had unresolved git merge conflict markers between two versions of scheduled_jobs table
+- **Fix:** Kept HEAD version (includes subscribers table + scheduled_jobs with scan_command column) and added missing index
+- **Files modified:** pkg/storage/schema.sql
+- **Verification:** All tests pass with resolved schema
+- **Committed in:** 76601b1
+
+**2. [Rule 1 - Bug] Added JSON wrapper structs for domain types**
+- **Found during:** Task 1
+- **Issue:** Provider, Dork, and Finding structs only have yaml tags -- json.Marshal would produce PascalCase field names inconsistent with REST API conventions
+- **Fix:** Created apiKey, apiProvider, apiDork structs with explicit JSON tags and converter functions
+- **Files modified:** pkg/web/api.go
+- **Verification:** Tests check exact JSON field names (providerName, name, etc.)
+- **Committed in:** 76601b1
+
+---
+
+**Total deviations:** 2 auto-fixed (1 blocking, 1 bug)
+**Impact on plan:** Both fixes necessary for correctness. No scope creep.
+
+## Issues Encountered
+None beyond the auto-fixed deviations above.
+
+## User Setup Required
+None - no external service configuration required.
+
+## Known Stubs
+None - all endpoints are fully wired to their backing registries/database.
+
+## Next Phase Readiness
+- REST API and SSE infrastructure ready for Plan 18-03 (HTML pages with htmx consuming these endpoints)
+- Server struct ready to be wired into cmd/serve.go
+
+---
+*Phase: 18-web-dashboard*
+*Completed: 2026-04-06*