keyhunter/.planning/phases/18-web-dashboard/18-02-PLAN.md

phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, must_haves

phase	plan	type	wave	depends_on	files_modified	autonomous	requirements	must_haves
18-web-dashboard	02	execute	1		pkg/web/api.go pkg/web/sse.go pkg/web/api_test.go pkg/web/sse_test.go	true	WEB-03 WEB-09 WEB-11	truths artifacts key_links REST API at /api/v1/* returns JSON for keys, providers, scan, recon, dorks, config SSE endpoint streams live scan/recon progress events API endpoints support filtering, pagination, and proper HTTP status codes path provides exports pkg/web/api.go All REST API handlers under /api/v1 mountAPI path provides exports pkg/web/sse.go SSE hub and endpoint handlers for live progress SSEHub NewSSEHub path provides pkg/web/api_test.go HTTP tests for all API endpoints path provides pkg/web/sse_test.go SSE connection and event broadcast tests from to via pattern pkg/web/api.go pkg/storage DB queries for findings, config s.cfg.DB. from to via pattern pkg/web/api.go pkg/providers Provider listing and stats s.cfg.Providers. from to via pattern pkg/web/sse.go pkg/web/api.go scan/recon handlers publish events to SSEHub s.sse.Broadcast

Implement all REST API endpoints (/api/v1/*) for programmatic access and the SSE hub for live scan/recon progress streaming.

Purpose: Provides the JSON data layer that both external API consumers and the htmx HTML pages (Plan 03) will use. Output: Complete REST API + SSE infrastructure in pkg/web.

<execution_context> @$HOME/.claude/get-shit-done/workflows/execute-plan.md @$HOME/.claude/get-shit-done/templates/summary.md </execution_context>

@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @.planning/phases/18-web-dashboard/18-CONTEXT.md

From pkg/storage/db.go + findings.go + queries.go:

type DB struct { ... }
func (db *DB) SQL() *sql.DB
func (db *DB) ListFindingsFiltered(encKey []byte, f Filters) ([]Finding, error)
func (db *DB) GetFinding(id int64, encKey []byte) (*Finding, error)
func (db *DB) DeleteFinding(id int64) (int64, error)
func (db *DB) SaveFinding(f Finding, encKey []byte) (int64, error)
type Filters struct { Provider, Confidence, SourceType string; Verified *bool; Limit, Offset int }
type Finding struct { ID, ScanID int64; ProviderName, KeyValue, KeyMasked, Confidence, SourcePath, SourceType string; LineNumber int; CreatedAt time.Time; Verified bool; VerifyStatus string; VerifyHTTPCode int; VerifyMetadata map[string]string }

From pkg/providers/registry.go + schema.go:

func (r *Registry) List() []Provider
func (r *Registry) Get(name string) (Provider, bool)
func (r *Registry) Stats() RegistryStats
type Provider struct { Name, DisplayName, Category, Confidence string; ... }
type RegistryStats struct { Total, ByCategory map[string]int; ... }

From pkg/dorks/registry.go + schema.go:

func (r *Registry) List() []Dork
func (r *Registry) Get(id string) (Dork, bool)
func (r *Registry) ListBySource(source string) []Dork
func (r *Registry) Stats() Stats
type Dork struct { ID, Source, Category, Query, Description string; ... }
type Stats struct { Total int; BySource map[string]int }

From pkg/storage/custom_dorks.go:

func (db *DB) SaveCustomDork(d CustomDork) (int64, error)
func (db *DB) ListCustomDorks() ([]CustomDork, error)

From pkg/recon/engine.go + source.go:

func (e *Engine) SweepAll(ctx context.Context, cfg Config) ([]Finding, error)
func (e *Engine) List() []string
type Config struct { Stealth, RespectRobots bool; EnabledSources []string; Query string }

From pkg/engine/engine.go:

func NewEngine(registry *providers.Registry) *Engine
func (e *Engine) Scan(ctx context.Context, src sources.Source, cfg ScanConfig) (<-chan Finding, error)
type ScanConfig struct { Workers int; Verify bool; VerifyTimeout time.Duration }

From pkg/storage/settings.go (viper config):

// Config is managed via viper — read/write with viper.GetString/viper.Set

Task 1: REST API handlers for /api/v1/* pkg/web/api.go, pkg/web/api_test.go - Test: GET /api/v1/stats returns JSON with totalKeys, totalProviders, reconSources fields - Test: GET /api/v1/keys returns JSON array of findings (masked by default) - Test: GET /api/v1/keys?provider=openai filters by provider - Test: GET /api/v1/keys/:id returns single finding JSON or 404 - Test: DELETE /api/v1/keys/:id returns 204 on success, 404 if not found - Test: GET /api/v1/providers returns JSON array of providers - Test: GET /api/v1/providers/:name returns single provider or 404 - Test: POST /api/v1/scan with JSON body returns 202 Accepted (async) - Test: POST /api/v1/recon with JSON body returns 202 Accepted (async) - Test: GET /api/v1/dorks returns JSON array of dorks - Test: POST /api/v1/dorks with valid JSON returns 201 - Test: GET /api/v1/config returns JSON config - Test: PUT /api/v1/config updates config and returns 200 1. Create `pkg/web/api.go`: - `func (s *Server) mountAPI(r chi.Router)` — sub-router under `/api/v1` - All handlers set `Content-Type: application/json` - Use `encoding/json` for marshal/unmarshal. Use `chi.URLParam(r, "id")` for path params.

2. Stats endpoint:

   • GET /api/v1/stats -> handleAPIStats
   • Query DB for total key count (SELECT COUNT(*) FROM findings), provider count from registry, recon source count from engine
   • Return {"totalKeys": N, "totalProviders": N, "reconSources": N, "lastScan": "..."}

3. Keys endpoints:

   • GET /api/v1/keys -> handleAPIListKeys — accepts query params: provider, confidence, limit (default 50), offset. Returns findings with KeyValue ALWAYS masked (API never exposes raw keys — use CLI keys show for that). Map Filters from query params.
   • GET /api/v1/keys/{id} -> handleAPIGetKey — parse id from URL, call GetFinding, return masked. 404 if nil.
   • DELETE /api/v1/keys/{id} -> handleAPIDeleteKey — call DeleteFinding, return 204. If rows=0, return 404.

4. Providers endpoints:

   • GET /api/v1/providers -> handleAPIListProviders — return registry.List() as JSON
   • GET /api/v1/providers/{name} -> handleAPIGetProvider — registry.Get(name), 404 if not found

5. Scan endpoint:

   • POST /api/v1/scan -> handleAPIScan — accepts JSON {"path": "/some/dir", "verify": false, "workers": 4}. Launches scan in background goroutine. Returns 202 with {"status": "started", "message": "scan initiated"}. Progress sent via SSE (Plan 18-02 SSE hub). If scan engine or DB is nil, return 503.

6. Recon endpoint:

   • POST /api/v1/recon -> handleAPIRecon — accepts JSON {"query": "openai", "sources": ["github","shodan"], "stealth": false}. Launches recon in background goroutine. Returns 202. Progress via SSE.

7. Dorks endpoints:

   • GET /api/v1/dorks -> handleAPIListDorks — accepts optional query param source for filtering. Return dorks registry list.
   • POST /api/v1/dorks -> handleAPIAddDork — accepts JSON with dork fields, saves as custom dork to DB. Returns 201.

8. Config endpoints:

   • GET /api/v1/config -> handleAPIGetConfig — return viper.AllSettings() as JSON
   • PUT /api/v1/config -> handleAPIUpdateConfig — accepts JSON object, iterate keys, call viper.Set for each. Write config with viper.WriteConfig(). Return 200.

9. Helper: func writeJSON(w http.ResponseWriter, status int, v interface{}) and func readJSON(r *http.Request, v interface{}) error for DRY request/response handling.

10. Create pkg/web/api_test.go:

    • Use httptest against a Server with in-memory SQLite DB, real providers registry, nil-safe recon engine
    • Test each endpoint for happy path + error cases (404, bad input)
    • For scan/recon POST tests, just verify 202 response (actual execution is async) cd /home/salva/Documents/apikey && go test ./pkg/web/... -run TestAPI -v -count=1 All /api/v1/* endpoints return correct JSON responses, proper HTTP status codes, filtering works, scan/recon return 202 for async operations

Task 2: SSE hub for live scan/recon progress pkg/web/sse.go, pkg/web/sse_test.go - Test: SSE client connects to /api/v1/scan/progress and receives events - Test: Broadcasting an event delivers to all connected clients - Test: Client disconnect removes from subscriber list - Test: SSE event format is "event: {type}\ndata: {json}\n\n" 1. Create `pkg/web/sse.go`: - `type SSEEvent struct { Type string; Data interface{} }` — Type is "scan:progress", "scan:finding", "scan:complete", "recon:progress", "recon:finding", "recon:complete" - `type SSEHub struct { clients map[chan SSEEvent]struct{}; mu sync.RWMutex }` - `func NewSSEHub() *SSEHub` - `func (h *SSEHub) Subscribe() chan SSEEvent` — creates buffered channel (cap 32), adds to clients map, returns - `func (h *SSEHub) Unsubscribe(ch chan SSEEvent)` — removes from map, closes channel - `func (h *SSEHub) Broadcast(evt SSEEvent)` — sends to all clients, skip if client buffer full (non-blocking send) - `func (s *Server) handleSSEScanProgress(w http.ResponseWriter, r *http.Request)` — standard SSE handler: - Set headers: `Content-Type: text/event-stream`, `Cache-Control: no-cache`, `Connection: keep-alive` - Flush with `http.Flusher` - Subscribe to hub, defer Unsubscribe - Loop: read from channel, format as `event: {type}\ndata: {json}\n\n`, flush - Break on request context done - `func (s *Server) handleSSEReconProgress(w http.ResponseWriter, r *http.Request)` — same pattern, same hub (events distinguish scan vs recon via Type prefix) - Add SSEHub field to Server struct, initialize in NewServer

2. Wire SSE into scan/recon handlers:

   • In handleAPIScan (from api.go), the background goroutine should: iterate findings channel from engine.Scan, broadcast SSEEvent{Type: "scan:finding", Data: finding} for each, then broadcast SSEEvent{Type: "scan:complete", Data: summary} when done
   • In handleAPIRecon, similar: broadcast recon progress events

3. Mount routes in mountAPI:

   • GET /api/v1/scan/progress -> handleSSEScanProgress
   • GET /api/v1/recon/progress -> handleSSEReconProgress

4. Create pkg/web/sse_test.go:

   • Test hub subscribe/broadcast/unsubscribe lifecycle
   • Test SSE HTTP handler using httptest — connect, send event via hub.Broadcast, verify SSE format in response body
   • Test client disconnect (cancel request context, verify unsubscribed) cd /home/salva/Documents/apikey && go test ./pkg/web/... -run TestSSE -v -count=1 SSE hub broadcasts events to connected clients, scan/recon progress streams in real-time, client disconnect is handled cleanly, event format matches SSE spec

- `go test ./pkg/web/... -v` — all API and SSE tests pass - `go vet ./pkg/web/...` — no issues - Manual: `curl http://localhost:8080/api/v1/stats` returns JSON (when server wired in Plan 03)

<success_criteria>

• GET /api/v1/stats returns JSON with totalKeys, totalProviders, reconSources
• GET /api/v1/keys returns filtered, paginated JSON array (always masked)
• GET/DELETE /api/v1/keys/{id} work with proper 404 handling
• GET /api/v1/providers and /api/v1/providers/{name} return provider data
• POST /api/v1/scan and /api/v1/recon return 202 and launch async work
• GET /api/v1/dorks returns dork list, POST /api/v1/dorks creates custom dork
• GET/PUT /api/v1/config read/write viper config
• SSE endpoints stream events in proper text/event-stream format
• All tests pass </success_criteria>

After completion, create `.planning/phases/18-web-dashboard/18-02-SUMMARY.md`