---
phase: 18-web-dashboard
plan: 02
type: execute
wave: 1
depends_on: []
files_modified:
- pkg/web/api.go
- pkg/web/sse.go
- pkg/web/api_test.go
- pkg/web/sse_test.go
autonomous: true
requirements: [WEB-03, WEB-09, WEB-11]
must_haves:
truths:
- "REST API at /api/v1/* returns JSON for keys, providers, scan, recon, dorks, config"
- "SSE endpoint streams live scan/recon progress events"
- "API endpoints support filtering, pagination, and proper HTTP status codes"
artifacts:
- path: "pkg/web/api.go"
provides: "All REST API handlers under /api/v1"
exports: ["mountAPI"]
- path: "pkg/web/sse.go"
provides: "SSE hub and endpoint handlers for live progress"
exports: ["SSEHub", "NewSSEHub"]
- path: "pkg/web/api_test.go"
provides: "HTTP tests for all API endpoints"
- path: "pkg/web/sse_test.go"
provides: "SSE connection and event broadcast tests"
key_links:
- from: "pkg/web/api.go"
to: "pkg/storage"
via: "DB queries for findings, config"
pattern: "s\\.cfg\\.DB\\."
- from: "pkg/web/api.go"
to: "pkg/providers"
via: "Provider listing and stats"
pattern: "s\\.cfg\\.Providers\\."
- from: "pkg/web/sse.go"
to: "pkg/web/api.go"
via: "scan/recon handlers publish events to SSEHub"
pattern: "s\\.sse\\.Broadcast"
---
Implement all REST API endpoints (/api/v1/*) for programmatic access and the SSE hub for live scan/recon progress streaming.
Purpose: Provides the JSON data layer that both external API consumers and the htmx HTML pages (Plan 03) will use.
Output: Complete REST API + SSE infrastructure in pkg/web.
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/18-web-dashboard/18-CONTEXT.md
From pkg/storage/db.go + findings.go + queries.go:
```go
type DB struct { ... }
func (db *DB) SQL() *sql.DB
func (db *DB) ListFindingsFiltered(encKey []byte, f Filters) ([]Finding, error)
func (db *DB) GetFinding(id int64, encKey []byte) (*Finding, error)
func (db *DB) DeleteFinding(id int64) (int64, error)
func (db *DB) SaveFinding(f Finding, encKey []byte) (int64, error)
type Filters struct { Provider, Confidence, SourceType string; Verified *bool; Limit, Offset int }
type Finding struct { ID, ScanID int64; ProviderName, KeyValue, KeyMasked, Confidence, SourcePath, SourceType string; LineNumber int; CreatedAt time.Time; Verified bool; VerifyStatus string; VerifyHTTPCode int; VerifyMetadata map[string]string }
```
From pkg/providers/registry.go + schema.go:
```go
func (r *Registry) List() []Provider
func (r *Registry) Get(name string) (Provider, bool)
func (r *Registry) Stats() RegistryStats
type Provider struct { Name, DisplayName, Category, Confidence string; ... }
type RegistryStats struct { Total, ByCategory map[string]int; ... }
```
From pkg/dorks/registry.go + schema.go:
```go
func (r *Registry) List() []Dork
func (r *Registry) Get(id string) (Dork, bool)
func (r *Registry) ListBySource(source string) []Dork
func (r *Registry) Stats() Stats
type Dork struct { ID, Source, Category, Query, Description string; ... }
type Stats struct { Total int; BySource map[string]int }
```
From pkg/storage/custom_dorks.go:
```go
func (db *DB) SaveCustomDork(d CustomDork) (int64, error)
func (db *DB) ListCustomDorks() ([]CustomDork, error)
```
From pkg/recon/engine.go + source.go:
```go
func (e *Engine) SweepAll(ctx context.Context, cfg Config) ([]Finding, error)
func (e *Engine) List() []string
type Config struct { Stealth, RespectRobots bool; EnabledSources []string; Query string }
```
From pkg/engine/engine.go:
```go
func NewEngine(registry *providers.Registry) *Engine
func (e *Engine) Scan(ctx context.Context, src sources.Source, cfg ScanConfig) (<-chan Finding, error)
type ScanConfig struct { Workers int; Verify bool; VerifyTimeout time.Duration }
```
From pkg/storage/settings.go (viper config):
```go
// Config is managed via viper — read/write with viper.GetString/viper.Set
```
Task 1: REST API handlers for /api/v1/*
pkg/web/api.go, pkg/web/api_test.go
- Test: GET /api/v1/stats returns JSON with totalKeys, totalProviders, reconSources fields
- Test: GET /api/v1/keys returns JSON array of findings (masked by default)
- Test: GET /api/v1/keys?provider=openai filters by provider
- Test: GET /api/v1/keys/:id returns single finding JSON or 404
- Test: DELETE /api/v1/keys/:id returns 204 on success, 404 if not found
- Test: GET /api/v1/providers returns JSON array of providers
- Test: GET /api/v1/providers/:name returns single provider or 404
- Test: POST /api/v1/scan with JSON body returns 202 Accepted (async)
- Test: POST /api/v1/recon with JSON body returns 202 Accepted (async)
- Test: GET /api/v1/dorks returns JSON array of dorks
- Test: POST /api/v1/dorks with valid JSON returns 201
- Test: GET /api/v1/config returns JSON config
- Test: PUT /api/v1/config updates config and returns 200
1. Create `pkg/web/api.go`:
- `func (s *Server) mountAPI(r chi.Router)` — sub-router under `/api/v1`
- All handlers set `Content-Type: application/json`
- Use `encoding/json` for marshal/unmarshal. Use `chi.URLParam(r, "id")` for path params.
2. Stats endpoint:
- `GET /api/v1/stats` -> `handleAPIStats`
- Query DB for total key count (SELECT COUNT(*) FROM findings), provider count from registry, recon source count from engine
- Return `{"totalKeys": N, "totalProviders": N, "reconSources": N, "lastScan": "..."}`
3. Keys endpoints:
- `GET /api/v1/keys` -> `handleAPIListKeys` — accepts query params: provider, confidence, limit (default 50), offset. Returns findings with KeyValue ALWAYS masked (API never exposes raw keys — use CLI `keys show` for that). Map Filters from query params.
- `GET /api/v1/keys/{id}` -> `handleAPIGetKey` — parse id from URL, call GetFinding, return masked. 404 if nil.
- `DELETE /api/v1/keys/{id}` -> `handleAPIDeleteKey` — call DeleteFinding, return 204. If rows=0, return 404.
4. Providers endpoints:
- `GET /api/v1/providers` -> `handleAPIListProviders` — return registry.List() as JSON
- `GET /api/v1/providers/{name}` -> `handleAPIGetProvider` — registry.Get(name), 404 if not found
5. Scan endpoint:
- `POST /api/v1/scan` -> `handleAPIScan` — accepts JSON `{"path": "/some/dir", "verify": false, "workers": 4}`. Launches scan in background goroutine. Returns 202 with `{"status": "started", "message": "scan initiated"}`. Progress sent via SSE (Plan 18-02 SSE hub). If scan engine or DB is nil, return 503.
6. Recon endpoint:
- `POST /api/v1/recon` -> `handleAPIRecon` — accepts JSON `{"query": "openai", "sources": ["github","shodan"], "stealth": false}`. Launches recon in background goroutine. Returns 202. Progress via SSE.
7. Dorks endpoints:
- `GET /api/v1/dorks` -> `handleAPIListDorks` — accepts optional query param `source` for filtering. Return dorks registry list.
- `POST /api/v1/dorks` -> `handleAPIAddDork` — accepts JSON with dork fields, saves as custom dork to DB. Returns 201.
8. Config endpoints:
- `GET /api/v1/config` -> `handleAPIGetConfig` — return viper.AllSettings() as JSON
- `PUT /api/v1/config` -> `handleAPIUpdateConfig` — accepts JSON object, iterate keys, call viper.Set for each. Write config with viper.WriteConfig(). Return 200.
9. Helper: `func writeJSON(w http.ResponseWriter, status int, v interface{})` and `func readJSON(r *http.Request, v interface{}) error` for DRY request/response handling.
10. Create `pkg/web/api_test.go`:
- Use httptest against a Server with in-memory SQLite DB, real providers registry, nil-safe recon engine
- Test each endpoint for happy path + error cases (404, bad input)
- For scan/recon POST tests, just verify 202 response (actual execution is async)
cd /home/salva/Documents/apikey && go test ./pkg/web/... -run TestAPI -v -count=1
All /api/v1/* endpoints return correct JSON responses, proper HTTP status codes, filtering works, scan/recon return 202 for async operations
Task 2: SSE hub for live scan/recon progress
pkg/web/sse.go, pkg/web/sse_test.go
- Test: SSE client connects to /api/v1/scan/progress and receives events
- Test: Broadcasting an event delivers to all connected clients
- Test: Client disconnect removes from subscriber list
- Test: SSE event format is "event: {type}\ndata: {json}\n\n"
1. Create `pkg/web/sse.go`:
- `type SSEEvent struct { Type string; Data interface{} }` — Type is "scan:progress", "scan:finding", "scan:complete", "recon:progress", "recon:finding", "recon:complete"
- `type SSEHub struct { clients map[chan SSEEvent]struct{}; mu sync.RWMutex }`
- `func NewSSEHub() *SSEHub`
- `func (h *SSEHub) Subscribe() chan SSEEvent` — creates buffered channel (cap 32), adds to clients map, returns
- `func (h *SSEHub) Unsubscribe(ch chan SSEEvent)` — removes from map, closes channel
- `func (h *SSEHub) Broadcast(evt SSEEvent)` — sends to all clients, skip if client buffer full (non-blocking send)
- `func (s *Server) handleSSEScanProgress(w http.ResponseWriter, r *http.Request)` — standard SSE handler:
- Set headers: `Content-Type: text/event-stream`, `Cache-Control: no-cache`, `Connection: keep-alive`
- Flush with `http.Flusher`
- Subscribe to hub, defer Unsubscribe
- Loop: read from channel, format as `event: {type}\ndata: {json}\n\n`, flush
- Break on request context done
- `func (s *Server) handleSSEReconProgress(w http.ResponseWriter, r *http.Request)` — same pattern, same hub (events distinguish scan vs recon via Type prefix)
- Add SSEHub field to Server struct, initialize in NewServer
2. Wire SSE into scan/recon handlers:
- In handleAPIScan (from api.go), the background goroutine should: iterate findings channel from engine.Scan, broadcast `SSEEvent{Type: "scan:finding", Data: finding}` for each, then broadcast `SSEEvent{Type: "scan:complete", Data: summary}` when done
- In handleAPIRecon, similar: broadcast recon progress events
3. Mount routes in mountAPI:
- `GET /api/v1/scan/progress` -> handleSSEScanProgress
- `GET /api/v1/recon/progress` -> handleSSEReconProgress
4. Create `pkg/web/sse_test.go`:
- Test hub subscribe/broadcast/unsubscribe lifecycle
- Test SSE HTTP handler using httptest — connect, send event via hub.Broadcast, verify SSE format in response body
- Test client disconnect (cancel request context, verify unsubscribed)
cd /home/salva/Documents/apikey && go test ./pkg/web/... -run TestSSE -v -count=1
SSE hub broadcasts events to connected clients, scan/recon progress streams in real-time, client disconnect is handled cleanly, event format matches SSE spec
- `go test ./pkg/web/... -v` — all API and SSE tests pass
- `go vet ./pkg/web/...` — no issues
- Manual: `curl http://localhost:8080/api/v1/stats` returns JSON (when server wired in Plan 03)
- GET /api/v1/stats returns JSON with totalKeys, totalProviders, reconSources
- GET /api/v1/keys returns filtered, paginated JSON array (always masked)
- GET/DELETE /api/v1/keys/{id} work with proper 404 handling
- GET /api/v1/providers and /api/v1/providers/{name} return provider data
- POST /api/v1/scan and /api/v1/recon return 202 and launch async work
- GET /api/v1/dorks returns dork list, POST /api/v1/dorks creates custom dork
- GET/PUT /api/v1/config read/write viper config
- SSE endpoints stream events in proper text/event-stream format
- All tests pass