docs(18-02): complete REST API + SSE hub plan

- 18-02-SUMMARY.md with 2 task commits - STATE.md updated with position and decisions - Requirements WEB-03, WEB-09, WEB-11 marked complete
2026-04-06 18:08:19 +03:00
parent d557c7303d
commit 79ec763233
3 changed files with 142 additions and 7 deletions
--- a/.planning/REQUIREMENTS.md
+++ b/.planning/REQUIREMENTS.md
@@ -220,15 +220,15 @@ Requirements for initial release. Each maps to roadmap phases.

 - [ ] **WEB-01**: Embedded HTTP server (chi + htmx + Tailwind CSS)
 - [ ] **WEB-02**: Dashboard overview page with summary statistics
- [ ] **WEB-03**: Scan history and scan detail pages
+- [x] **WEB-03**: Scan history and scan detail pages
 - [ ] **WEB-04**: Key listing page with filtering and "Reveal Key" toggle
 - [ ] **WEB-05**: OSINT/Recon launcher and results page
 - [ ] **WEB-06**: Provider listing and statistics page
 - [ ] **WEB-07**: Dork management page
 - [ ] **WEB-08**: Settings configuration page
- [ ] **WEB-09**: REST API (/api/v1/*) for programmatic access
+- [x] **WEB-09**: REST API (/api/v1/*) for programmatic access
 - [ ] **WEB-10**: Optional basic auth / token auth
- [ ] **WEB-11**: Server-Sent Events for live scan progress
+- [x] **WEB-11**: Server-Sent Events for live scan progress

 ### Telegram Bot

--- a/.planning/STATE.md
+++ b/.planning/STATE.md
@@ -3,8 +3,8 @@ gsd_state_version: 1.0
 milestone: v1.0
 milestone_name: milestone
 status: executing
-stopped_at: Completed 17-04-PLAN.md
-last_updated: "2026-04-06T14:50:49.687Z"
+stopped_at: Completed 18-02-PLAN.md
+last_updated: "2026-04-06T15:07:44.687Z"
 last_activity: 2026-04-06
 progress:
  total_phases: 18
@@ -102,6 +102,7 @@ Progress: [██░░░░░░░░] 20%
 | Phase 16 P01 | 4min | 2 tasks | 6 files |
 | Phase 17 P01 | 3min | 2 tasks | 4 files |
 | Phase 17 P04 | 3min | 2 tasks | 4 files |
+| Phase 18 P02 | 7min | 2 tasks | 7 files |

 ## Accumulated Context

@@ -156,6 +157,9 @@ Recent decisions affecting current work:
 - [Phase 16]: URLhaus tag lookup with payload endpoint fallback
 - [Phase 17]: telego v1.8.0 promoted from indirect to direct; context cancellation for graceful shutdown; rate limit 60s scan/verify/recon, 5s others
 - [Phase 17]: Separated format from send for testable notifications without telego mock
+- [Phase 18]: JSON wrapper structs (apiKey, apiProvider, apiDork) with explicit JSON tags since domain structs only have yaml tags
+- [Phase 18]: API never exposes raw key values -- KeyValue always empty string in JSON responses
+- [Phase 18]: Single SSEHub shared between scan and recon progress endpoints, events distinguished by Type prefix

 ### Pending Todos

@@ -170,6 +174,6 @@ None yet.

 ## Session Continuity

-Last session: 2026-04-06T14:34:18.710Z
-Stopped at: Completed 17-04-PLAN.md
+Last session: 2026-04-06T15:07:44.683Z
+Stopped at: Completed 18-02-PLAN.md
 Resume file: None
--- a/.planning/phases/18-web-dashboard/18-02-SUMMARY.md
+++ b/.planning/phases/18-web-dashboard/18-02-SUMMARY.md
@@ -0,0 +1,131 @@
+---
+phase: 18-web-dashboard
+plan: 02
+subsystem: api
+tags: [chi, rest-api, sse, json, http, server-sent-events]
+
+requires:
+  - phase: 01-foundation
+    provides: "storage DB, providers registry, encryption"
+  - phase: 08-dork-engine
+    provides: "dorks registry and custom dork storage"
+  - phase: 09-osint-infrastructure
+    provides: "recon engine"
+provides:
+  - "REST API at /api/v1/* for keys, providers, scan, recon, dorks, config"
+  - "SSE hub for live scan/recon progress streaming"
+  - "Server struct with dependency injection for all web handlers"
+affects: [18-web-dashboard, serve-command]
+
+tech-stack:
+  added: [chi-v5]
+  patterns: [api-json-wrappers, sse-hub-broadcast, dependency-injected-server]
+
+key-files:
+  created:
+    - pkg/web/server.go
+    - pkg/web/api.go
+    - pkg/web/sse.go
+    - pkg/web/api_test.go
+    - pkg/web/sse_test.go
+  modified:
+    - pkg/storage/schema.sql
+    - go.mod
+    - go.sum
+
+key-decisions:
+  - "JSON wrapper structs (apiKey, apiProvider, apiDork) with explicit JSON tags since domain structs only have yaml tags"
+  - "API never exposes raw key values -- KeyValue always empty string in JSON responses"
+  - "Single SSEHub shared between scan and recon progress endpoints, events distinguished by Type prefix"
+
+patterns-established:
+  - "API wrapper pattern: domain structs -> apiX structs with JSON tags for consistent camelCase API"
+  - "writeJSON/readJSON helpers for DRY HTTP response handling"
+  - "ServerConfig struct for dependency injection into all web handlers"
+
+requirements-completed: [WEB-03, WEB-09, WEB-11]
+
+duration: 7min
+completed: 2026-04-06
+---
+
+# Phase 18 Plan 02: REST API + SSE Hub Summary
+
+**Complete REST API at /api/v1/* with 14 endpoints (keys, providers, scan, recon, dorks, config) plus SSE hub for live event streaming**
+
+## Performance
+
+- **Duration:** 7 min
+- **Started:** 2026-04-06T14:59:58Z
+- **Completed:** 2026-04-06T15:06:51Z
+- **Tasks:** 2
+- **Files modified:** 7
+
+## Accomplishments
+- Full REST API with 14 endpoints covering stats, keys CRUD, providers, scan/recon triggers, dorks, and config
+- SSE hub with subscribe/unsubscribe/broadcast lifecycle and non-blocking buffered channels
+- 23 passing tests (16 API + 7 SSE) covering happy paths and error cases
+
+## Task Commits
+
+Each task was committed atomically:
+
+1. **Task 1: REST API handlers for /api/v1/*** - `76601b1` (feat)
+2. **Task 2: SSE hub for live scan/recon progress** - `d557c73` (feat)
+
+## Files Created/Modified
+- `pkg/web/server.go` - Server struct with ServerConfig dependency injection
+- `pkg/web/api.go` - All 14 REST API handlers with JSON wrapper types
+- `pkg/web/sse.go` - SSEHub with Subscribe/Unsubscribe/Broadcast + HTTP handlers
+- `pkg/web/api_test.go` - 16 tests for all API endpoints
+- `pkg/web/sse_test.go` - 7 tests for SSE hub lifecycle and HTTP streaming
+- `pkg/storage/schema.sql` - Resolved merge conflict (HEAD version kept)
+- `go.mod` / `go.sum` - Added chi v5.2.5
+
+## Decisions Made
+- JSON wrapper structs (apiKey, apiProvider, apiDork) with explicit JSON tags since domain structs only have yaml tags -- ensures consistent camelCase JSON API
+- API never exposes raw key values -- KeyValue always empty string in JSON responses for security
+- Single SSEHub shared between scan and recon progress endpoints, events distinguished by Type prefix (scan:*, recon:*)
+- DisallowUnknownFields removed from readJSON to avoid overly strict request parsing
+
+## Deviations from Plan
+
+### Auto-fixed Issues
+
+**1. [Rule 3 - Blocking] Resolved merge conflict in schema.sql**
+- **Found during:** Task 1
+- **Issue:** schema.sql had unresolved git merge conflict markers between two versions of scheduled_jobs table
+- **Fix:** Kept HEAD version (includes subscribers table + scheduled_jobs with scan_command column) and added missing index
+- **Files modified:** pkg/storage/schema.sql
+- **Verification:** All tests pass with resolved schema
+- **Committed in:** 76601b1
+
+**2. [Rule 1 - Bug] Added JSON wrapper structs for domain types**
+- **Found during:** Task 1
+- **Issue:** Provider, Dork, and Finding structs only have yaml tags -- json.Marshal would produce PascalCase field names inconsistent with REST API conventions
+- **Fix:** Created apiKey, apiProvider, apiDork structs with explicit JSON tags and converter functions
+- **Files modified:** pkg/web/api.go
+- **Verification:** Tests check exact JSON field names (providerName, name, etc.)
+- **Committed in:** 76601b1
+
+---
+
+**Total deviations:** 2 auto-fixed (1 blocking, 1 bug)
+**Impact on plan:** Both fixes necessary for correctness. No scope creep.
+
+## Issues Encountered
+None beyond the auto-fixed deviations above.
+
+## User Setup Required
+None - no external service configuration required.
+
+## Known Stubs
+None - all endpoints are fully wired to their backing registries/database.
+
+## Next Phase Readiness
+- REST API and SSE infrastructure ready for Plan 18-03 (HTML pages with htmx consuming these endpoints)
+- Server struct ready to be wired into cmd/serve.go
+
+---
+*Phase: 18-web-dashboard*
+*Completed: 2026-04-06*