260 lines
12 KiB
Markdown
260 lines
12 KiB
Markdown
---
|
|
phase: 18-web-dashboard
|
|
plan: 02
|
|
type: execute
|
|
wave: 1
|
|
depends_on: []
|
|
files_modified:
|
|
- pkg/web/api.go
|
|
- pkg/web/sse.go
|
|
- pkg/web/api_test.go
|
|
- pkg/web/sse_test.go
|
|
autonomous: true
|
|
requirements: [WEB-03, WEB-09, WEB-11]
|
|
|
|
must_haves:
|
|
truths:
|
|
- "REST API at /api/v1/* returns JSON for keys, providers, scan, recon, dorks, config"
|
|
- "SSE endpoint streams live scan/recon progress events"
|
|
- "API endpoints support filtering, pagination, and proper HTTP status codes"
|
|
artifacts:
|
|
- path: "pkg/web/api.go"
|
|
provides: "All REST API handlers under /api/v1"
|
|
exports: ["mountAPI"]
|
|
- path: "pkg/web/sse.go"
|
|
provides: "SSE hub and endpoint handlers for live progress"
|
|
exports: ["SSEHub", "NewSSEHub"]
|
|
- path: "pkg/web/api_test.go"
|
|
provides: "HTTP tests for all API endpoints"
|
|
- path: "pkg/web/sse_test.go"
|
|
provides: "SSE connection and event broadcast tests"
|
|
key_links:
|
|
- from: "pkg/web/api.go"
|
|
to: "pkg/storage"
|
|
via: "DB queries for findings, config"
|
|
pattern: "s\\.cfg\\.DB\\."
|
|
- from: "pkg/web/api.go"
|
|
to: "pkg/providers"
|
|
via: "Provider listing and stats"
|
|
pattern: "s\\.cfg\\.Providers\\."
|
|
- from: "pkg/web/sse.go"
|
|
to: "pkg/web/api.go"
|
|
via: "scan/recon handlers publish events to SSEHub"
|
|
pattern: "s\\.sse\\.Broadcast"
|
|
---
|
|
|
|
<objective>
|
|
Implement all REST API endpoints (/api/v1/*) for programmatic access and the SSE hub for live scan/recon progress streaming.
|
|
|
|
Purpose: Provides the JSON data layer that both external API consumers and the htmx HTML pages (Plan 03) will use.
|
|
Output: Complete REST API + SSE infrastructure in pkg/web.
|
|
</objective>
|
|
|
|
<execution_context>
|
|
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
|
@$HOME/.claude/get-shit-done/templates/summary.md
|
|
</execution_context>
|
|
|
|
<context>
|
|
@.planning/PROJECT.md
|
|
@.planning/ROADMAP.md
|
|
@.planning/STATE.md
|
|
@.planning/phases/18-web-dashboard/18-CONTEXT.md
|
|
|
|
<interfaces>
|
|
<!-- Key types and contracts the executor needs. -->
|
|
|
|
From pkg/storage/db.go + findings.go + queries.go:
|
|
```go
|
|
type DB struct { ... }
|
|
func (db *DB) SQL() *sql.DB
|
|
func (db *DB) ListFindingsFiltered(encKey []byte, f Filters) ([]Finding, error)
|
|
func (db *DB) GetFinding(id int64, encKey []byte) (*Finding, error)
|
|
func (db *DB) DeleteFinding(id int64) (int64, error)
|
|
func (db *DB) SaveFinding(f Finding, encKey []byte) (int64, error)
|
|
type Filters struct { Provider, Confidence, SourceType string; Verified *bool; Limit, Offset int }
|
|
type Finding struct { ID, ScanID int64; ProviderName, KeyValue, KeyMasked, Confidence, SourcePath, SourceType string; LineNumber int; CreatedAt time.Time; Verified bool; VerifyStatus string; VerifyHTTPCode int; VerifyMetadata map[string]string }
|
|
```
|
|
|
|
From pkg/providers/registry.go + schema.go:
|
|
```go
|
|
func (r *Registry) List() []Provider
|
|
func (r *Registry) Get(name string) (Provider, bool)
|
|
func (r *Registry) Stats() RegistryStats
|
|
type Provider struct { Name, DisplayName, Category, Confidence string; ... }
|
|
type RegistryStats struct { Total, ByCategory map[string]int; ... }
|
|
```
|
|
|
|
From pkg/dorks/registry.go + schema.go:
|
|
```go
|
|
func (r *Registry) List() []Dork
|
|
func (r *Registry) Get(id string) (Dork, bool)
|
|
func (r *Registry) ListBySource(source string) []Dork
|
|
func (r *Registry) Stats() Stats
|
|
type Dork struct { ID, Source, Category, Query, Description string; ... }
|
|
type Stats struct { Total int; BySource map[string]int }
|
|
```
|
|
|
|
From pkg/storage/custom_dorks.go:
|
|
```go
|
|
func (db *DB) SaveCustomDork(d CustomDork) (int64, error)
|
|
func (db *DB) ListCustomDorks() ([]CustomDork, error)
|
|
```
|
|
|
|
From pkg/recon/engine.go + source.go:
|
|
```go
|
|
func (e *Engine) SweepAll(ctx context.Context, cfg Config) ([]Finding, error)
|
|
func (e *Engine) List() []string
|
|
type Config struct { Stealth, RespectRobots bool; EnabledSources []string; Query string }
|
|
```
|
|
|
|
From pkg/engine/engine.go:
|
|
```go
|
|
func NewEngine(registry *providers.Registry) *Engine
|
|
func (e *Engine) Scan(ctx context.Context, src sources.Source, cfg ScanConfig) (<-chan Finding, error)
|
|
type ScanConfig struct { Workers int; Verify bool; VerifyTimeout time.Duration }
|
|
```
|
|
|
|
From pkg/storage/settings.go (viper config):
|
|
```go
|
|
// Config is managed via viper — read/write with viper.GetString/viper.Set
|
|
```
|
|
</interfaces>
|
|
</context>
|
|
|
|
<tasks>
|
|
|
|
<task type="auto" tdd="true">
|
|
<name>Task 1: REST API handlers for /api/v1/*</name>
|
|
<files>pkg/web/api.go, pkg/web/api_test.go</files>
|
|
<behavior>
|
|
- Test: GET /api/v1/stats returns JSON with totalKeys, totalProviders, reconSources fields
|
|
- Test: GET /api/v1/keys returns JSON array of findings (masked by default)
|
|
- Test: GET /api/v1/keys?provider=openai filters by provider
|
|
- Test: GET /api/v1/keys/:id returns single finding JSON or 404
|
|
- Test: DELETE /api/v1/keys/:id returns 204 on success, 404 if not found
|
|
- Test: GET /api/v1/providers returns JSON array of providers
|
|
- Test: GET /api/v1/providers/:name returns single provider or 404
|
|
- Test: POST /api/v1/scan with JSON body returns 202 Accepted (async)
|
|
- Test: POST /api/v1/recon with JSON body returns 202 Accepted (async)
|
|
- Test: GET /api/v1/dorks returns JSON array of dorks
|
|
- Test: POST /api/v1/dorks with valid JSON returns 201
|
|
- Test: GET /api/v1/config returns JSON config
|
|
- Test: PUT /api/v1/config updates config and returns 200
|
|
</behavior>
|
|
<action>
|
|
1. Create `pkg/web/api.go`:
|
|
- `func (s *Server) mountAPI(r chi.Router)` — sub-router under `/api/v1`
|
|
- All handlers set `Content-Type: application/json`
|
|
- Use `encoding/json` for marshal/unmarshal. Use `chi.URLParam(r, "id")` for path params.
|
|
|
|
2. Stats endpoint:
|
|
- `GET /api/v1/stats` -> `handleAPIStats`
|
|
- Query DB for total key count (SELECT COUNT(*) FROM findings), provider count from registry, recon source count from engine
|
|
- Return `{"totalKeys": N, "totalProviders": N, "reconSources": N, "lastScan": "..."}`
|
|
|
|
3. Keys endpoints:
|
|
- `GET /api/v1/keys` -> `handleAPIListKeys` — accepts query params: provider, confidence, limit (default 50), offset. Returns findings with KeyValue ALWAYS masked (API never exposes raw keys — use CLI `keys show` for that). Map Filters from query params.
|
|
- `GET /api/v1/keys/{id}` -> `handleAPIGetKey` — parse id from URL, call GetFinding, return masked. 404 if nil.
|
|
- `DELETE /api/v1/keys/{id}` -> `handleAPIDeleteKey` — call DeleteFinding, return 204. If rows=0, return 404.
|
|
|
|
4. Providers endpoints:
|
|
- `GET /api/v1/providers` -> `handleAPIListProviders` — return registry.List() as JSON
|
|
- `GET /api/v1/providers/{name}` -> `handleAPIGetProvider` — registry.Get(name), 404 if not found
|
|
|
|
5. Scan endpoint:
|
|
- `POST /api/v1/scan` -> `handleAPIScan` — accepts JSON `{"path": "/some/dir", "verify": false, "workers": 4}`. Launches scan in background goroutine. Returns 202 with `{"status": "started", "message": "scan initiated"}`. Progress sent via SSE (Plan 18-02 SSE hub). If scan engine or DB is nil, return 503.
|
|
|
|
6. Recon endpoint:
|
|
- `POST /api/v1/recon` -> `handleAPIRecon` — accepts JSON `{"query": "openai", "sources": ["github","shodan"], "stealth": false}`. Launches recon in background goroutine. Returns 202. Progress via SSE.
|
|
|
|
7. Dorks endpoints:
|
|
- `GET /api/v1/dorks` -> `handleAPIListDorks` — accepts optional query param `source` for filtering. Return dorks registry list.
|
|
- `POST /api/v1/dorks` -> `handleAPIAddDork` — accepts JSON with dork fields, saves as custom dork to DB. Returns 201.
|
|
|
|
8. Config endpoints:
|
|
- `GET /api/v1/config` -> `handleAPIGetConfig` — return viper.AllSettings() as JSON
|
|
- `PUT /api/v1/config` -> `handleAPIUpdateConfig` — accepts JSON object, iterate keys, call viper.Set for each. Write config with viper.WriteConfig(). Return 200.
|
|
|
|
9. Helper: `func writeJSON(w http.ResponseWriter, status int, v interface{})` and `func readJSON(r *http.Request, v interface{}) error` for DRY request/response handling.
|
|
|
|
10. Create `pkg/web/api_test.go`:
|
|
- Use httptest against a Server with in-memory SQLite DB, real providers registry, nil-safe recon engine
|
|
- Test each endpoint for happy path + error cases (404, bad input)
|
|
- For scan/recon POST tests, just verify 202 response (actual execution is async)
|
|
</action>
|
|
<verify>
|
|
<automated>cd /home/salva/Documents/apikey && go test ./pkg/web/... -run TestAPI -v -count=1</automated>
|
|
</verify>
|
|
<done>All /api/v1/* endpoints return correct JSON responses, proper HTTP status codes, filtering works, scan/recon return 202 for async operations</done>
|
|
</task>
|
|
|
|
<task type="auto" tdd="true">
|
|
<name>Task 2: SSE hub for live scan/recon progress</name>
|
|
<files>pkg/web/sse.go, pkg/web/sse_test.go</files>
|
|
<behavior>
|
|
- Test: SSE client connects to /api/v1/scan/progress and receives events
|
|
- Test: Broadcasting an event delivers to all connected clients
|
|
- Test: Client disconnect removes from subscriber list
|
|
- Test: SSE event format is "event: {type}\ndata: {json}\n\n"
|
|
</behavior>
|
|
<action>
|
|
1. Create `pkg/web/sse.go`:
|
|
- `type SSEEvent struct { Type string; Data interface{} }` — Type is "scan:progress", "scan:finding", "scan:complete", "recon:progress", "recon:finding", "recon:complete"
|
|
- `type SSEHub struct { clients map[chan SSEEvent]struct{}; mu sync.RWMutex }`
|
|
- `func NewSSEHub() *SSEHub`
|
|
- `func (h *SSEHub) Subscribe() chan SSEEvent` — creates buffered channel (cap 32), adds to clients map, returns
|
|
- `func (h *SSEHub) Unsubscribe(ch chan SSEEvent)` — removes from map, closes channel
|
|
- `func (h *SSEHub) Broadcast(evt SSEEvent)` — sends to all clients, skip if client buffer full (non-blocking send)
|
|
- `func (s *Server) handleSSEScanProgress(w http.ResponseWriter, r *http.Request)` — standard SSE handler:
|
|
- Set headers: `Content-Type: text/event-stream`, `Cache-Control: no-cache`, `Connection: keep-alive`
|
|
- Flush with `http.Flusher`
|
|
- Subscribe to hub, defer Unsubscribe
|
|
- Loop: read from channel, format as `event: {type}\ndata: {json}\n\n`, flush
|
|
- Break on request context done
|
|
- `func (s *Server) handleSSEReconProgress(w http.ResponseWriter, r *http.Request)` — same pattern, same hub (events distinguish scan vs recon via Type prefix)
|
|
- Add SSEHub field to Server struct, initialize in NewServer
|
|
|
|
2. Wire SSE into scan/recon handlers:
|
|
- In handleAPIScan (from api.go), the background goroutine should: iterate findings channel from engine.Scan, broadcast `SSEEvent{Type: "scan:finding", Data: finding}` for each, then broadcast `SSEEvent{Type: "scan:complete", Data: summary}` when done
|
|
- In handleAPIRecon, similar: broadcast recon progress events
|
|
|
|
3. Mount routes in mountAPI:
|
|
- `GET /api/v1/scan/progress` -> handleSSEScanProgress
|
|
- `GET /api/v1/recon/progress` -> handleSSEReconProgress
|
|
|
|
4. Create `pkg/web/sse_test.go`:
|
|
- Test hub subscribe/broadcast/unsubscribe lifecycle
|
|
- Test SSE HTTP handler using httptest — connect, send event via hub.Broadcast, verify SSE format in response body
|
|
- Test client disconnect (cancel request context, verify unsubscribed)
|
|
</action>
|
|
<verify>
|
|
<automated>cd /home/salva/Documents/apikey && go test ./pkg/web/... -run TestSSE -v -count=1</automated>
|
|
</verify>
|
|
<done>SSE hub broadcasts events to connected clients, scan/recon progress streams in real-time, client disconnect is handled cleanly, event format matches SSE spec</done>
|
|
</task>
|
|
|
|
</tasks>
|
|
|
|
<verification>
|
|
- `go test ./pkg/web/... -v` — all API and SSE tests pass
|
|
- `go vet ./pkg/web/...` — no issues
|
|
- Manual: `curl http://localhost:8080/api/v1/stats` returns JSON (when server wired in Plan 03)
|
|
</verification>
|
|
|
|
<success_criteria>
|
|
- GET /api/v1/stats returns JSON with totalKeys, totalProviders, reconSources
|
|
- GET /api/v1/keys returns filtered, paginated JSON array (always masked)
|
|
- GET/DELETE /api/v1/keys/{id} work with proper 404 handling
|
|
- GET /api/v1/providers and /api/v1/providers/{name} return provider data
|
|
- POST /api/v1/scan and /api/v1/recon return 202 and launch async work
|
|
- GET /api/v1/dorks returns dork list, POST /api/v1/dorks creates custom dork
|
|
- GET/PUT /api/v1/config read/write viper config
|
|
- SSE endpoints stream events in proper text/event-stream format
|
|
- All tests pass
|
|
</success_criteria>
|
|
|
|
<output>
|
|
After completion, create `.planning/phases/18-web-dashboard/18-02-SUMMARY.md`
|
|
</output>
|