From 79ec763233f96da4d9db5c9fea076aaec6b2b202 Mon Sep 17 00:00:00 2001 From: salvacybersec Date: Mon, 6 Apr 2026 18:08:19 +0300 Subject: [PATCH] docs(18-02): complete REST API + SSE hub plan - 18-02-SUMMARY.md with 2 task commits - STATE.md updated with position and decisions - Requirements WEB-03, WEB-09, WEB-11 marked complete --- .planning/REQUIREMENTS.md | 6 +- .planning/STATE.md | 12 +- .../phases/18-web-dashboard/18-02-SUMMARY.md | 131 ++++++++++++++++++ 3 files changed, 142 insertions(+), 7 deletions(-) create mode 100644 .planning/phases/18-web-dashboard/18-02-SUMMARY.md diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index af3bbee..4e9f075 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -220,15 +220,15 @@ Requirements for initial release. Each maps to roadmap phases. - [ ] **WEB-01**: Embedded HTTP server (chi + htmx + Tailwind CSS) - [ ] **WEB-02**: Dashboard overview page with summary statistics -- [ ] **WEB-03**: Scan history and scan detail pages +- [x] **WEB-03**: Scan history and scan detail pages - [ ] **WEB-04**: Key listing page with filtering and "Reveal Key" toggle - [ ] **WEB-05**: OSINT/Recon launcher and results page - [ ] **WEB-06**: Provider listing and statistics page - [ ] **WEB-07**: Dork management page - [ ] **WEB-08**: Settings configuration page -- [ ] **WEB-09**: REST API (/api/v1/*) for programmatic access +- [x] **WEB-09**: REST API (/api/v1/*) for programmatic access - [ ] **WEB-10**: Optional basic auth / token auth -- [ ] **WEB-11**: Server-Sent Events for live scan progress +- [x] **WEB-11**: Server-Sent Events for live scan progress ### Telegram Bot diff --git a/.planning/STATE.md b/.planning/STATE.md index 5ec92af..f8817fd 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -3,8 +3,8 @@ gsd_state_version: 1.0 milestone: v1.0 milestone_name: milestone status: executing -stopped_at: Completed 17-04-PLAN.md -last_updated: "2026-04-06T14:50:49.687Z" +stopped_at: Completed 18-02-PLAN.md +last_updated: "2026-04-06T15:07:44.687Z" last_activity: 2026-04-06 progress: total_phases: 18 @@ -102,6 +102,7 @@ Progress: [██░░░░░░░░] 20% | Phase 16 P01 | 4min | 2 tasks | 6 files | | Phase 17 P01 | 3min | 2 tasks | 4 files | | Phase 17 P04 | 3min | 2 tasks | 4 files | +| Phase 18 P02 | 7min | 2 tasks | 7 files | ## Accumulated Context @@ -156,6 +157,9 @@ Recent decisions affecting current work: - [Phase 16]: URLhaus tag lookup with payload endpoint fallback - [Phase 17]: telego v1.8.0 promoted from indirect to direct; context cancellation for graceful shutdown; rate limit 60s scan/verify/recon, 5s others - [Phase 17]: Separated format from send for testable notifications without telego mock +- [Phase 18]: JSON wrapper structs (apiKey, apiProvider, apiDork) with explicit JSON tags since domain structs only have yaml tags +- [Phase 18]: API never exposes raw key values -- KeyValue always empty string in JSON responses +- [Phase 18]: Single SSEHub shared between scan and recon progress endpoints, events distinguished by Type prefix ### Pending Todos @@ -170,6 +174,6 @@ None yet. ## Session Continuity -Last session: 2026-04-06T14:34:18.710Z -Stopped at: Completed 17-04-PLAN.md +Last session: 2026-04-06T15:07:44.683Z +Stopped at: Completed 18-02-PLAN.md Resume file: None diff --git a/.planning/phases/18-web-dashboard/18-02-SUMMARY.md b/.planning/phases/18-web-dashboard/18-02-SUMMARY.md new file mode 100644 index 0000000..2456a5c --- /dev/null +++ b/.planning/phases/18-web-dashboard/18-02-SUMMARY.md @@ -0,0 +1,131 @@ +--- +phase: 18-web-dashboard +plan: 02 +subsystem: api +tags: [chi, rest-api, sse, json, http, server-sent-events] + +requires: + - phase: 01-foundation + provides: "storage DB, providers registry, encryption" + - phase: 08-dork-engine + provides: "dorks registry and custom dork storage" + - phase: 09-osint-infrastructure + provides: "recon engine" +provides: + - "REST API at /api/v1/* for keys, providers, scan, recon, dorks, config" + - "SSE hub for live scan/recon progress streaming" + - "Server struct with dependency injection for all web handlers" +affects: [18-web-dashboard, serve-command] + +tech-stack: + added: [chi-v5] + patterns: [api-json-wrappers, sse-hub-broadcast, dependency-injected-server] + +key-files: + created: + - pkg/web/server.go + - pkg/web/api.go + - pkg/web/sse.go + - pkg/web/api_test.go + - pkg/web/sse_test.go + modified: + - pkg/storage/schema.sql + - go.mod + - go.sum + +key-decisions: + - "JSON wrapper structs (apiKey, apiProvider, apiDork) with explicit JSON tags since domain structs only have yaml tags" + - "API never exposes raw key values -- KeyValue always empty string in JSON responses" + - "Single SSEHub shared between scan and recon progress endpoints, events distinguished by Type prefix" + +patterns-established: + - "API wrapper pattern: domain structs -> apiX structs with JSON tags for consistent camelCase API" + - "writeJSON/readJSON helpers for DRY HTTP response handling" + - "ServerConfig struct for dependency injection into all web handlers" + +requirements-completed: [WEB-03, WEB-09, WEB-11] + +duration: 7min +completed: 2026-04-06 +--- + +# Phase 18 Plan 02: REST API + SSE Hub Summary + +**Complete REST API at /api/v1/* with 14 endpoints (keys, providers, scan, recon, dorks, config) plus SSE hub for live event streaming** + +## Performance + +- **Duration:** 7 min +- **Started:** 2026-04-06T14:59:58Z +- **Completed:** 2026-04-06T15:06:51Z +- **Tasks:** 2 +- **Files modified:** 7 + +## Accomplishments +- Full REST API with 14 endpoints covering stats, keys CRUD, providers, scan/recon triggers, dorks, and config +- SSE hub with subscribe/unsubscribe/broadcast lifecycle and non-blocking buffered channels +- 23 passing tests (16 API + 7 SSE) covering happy paths and error cases + +## Task Commits + +Each task was committed atomically: + +1. **Task 1: REST API handlers for /api/v1/*** - `76601b1` (feat) +2. **Task 2: SSE hub for live scan/recon progress** - `d557c73` (feat) + +## Files Created/Modified +- `pkg/web/server.go` - Server struct with ServerConfig dependency injection +- `pkg/web/api.go` - All 14 REST API handlers with JSON wrapper types +- `pkg/web/sse.go` - SSEHub with Subscribe/Unsubscribe/Broadcast + HTTP handlers +- `pkg/web/api_test.go` - 16 tests for all API endpoints +- `pkg/web/sse_test.go` - 7 tests for SSE hub lifecycle and HTTP streaming +- `pkg/storage/schema.sql` - Resolved merge conflict (HEAD version kept) +- `go.mod` / `go.sum` - Added chi v5.2.5 + +## Decisions Made +- JSON wrapper structs (apiKey, apiProvider, apiDork) with explicit JSON tags since domain structs only have yaml tags -- ensures consistent camelCase JSON API +- API never exposes raw key values -- KeyValue always empty string in JSON responses for security +- Single SSEHub shared between scan and recon progress endpoints, events distinguished by Type prefix (scan:*, recon:*) +- DisallowUnknownFields removed from readJSON to avoid overly strict request parsing + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 3 - Blocking] Resolved merge conflict in schema.sql** +- **Found during:** Task 1 +- **Issue:** schema.sql had unresolved git merge conflict markers between two versions of scheduled_jobs table +- **Fix:** Kept HEAD version (includes subscribers table + scheduled_jobs with scan_command column) and added missing index +- **Files modified:** pkg/storage/schema.sql +- **Verification:** All tests pass with resolved schema +- **Committed in:** 76601b1 + +**2. [Rule 1 - Bug] Added JSON wrapper structs for domain types** +- **Found during:** Task 1 +- **Issue:** Provider, Dork, and Finding structs only have yaml tags -- json.Marshal would produce PascalCase field names inconsistent with REST API conventions +- **Fix:** Created apiKey, apiProvider, apiDork structs with explicit JSON tags and converter functions +- **Files modified:** pkg/web/api.go +- **Verification:** Tests check exact JSON field names (providerName, name, etc.) +- **Committed in:** 76601b1 + +--- + +**Total deviations:** 2 auto-fixed (1 blocking, 1 bug) +**Impact on plan:** Both fixes necessary for correctness. No scope creep. + +## Issues Encountered +None beyond the auto-fixed deviations above. + +## User Setup Required +None - no external service configuration required. + +## Known Stubs +None - all endpoints are fully wired to their backing registries/database. + +## Next Phase Readiness +- REST API and SSE infrastructure ready for Plan 18-03 (HTML pages with htmx consuming these endpoints) +- Server struct ready to be wired into cmd/serve.go + +--- +*Phase: 18-web-dashboard* +*Completed: 2026-04-06*