docs(18): create web dashboard phase plan

This commit is contained in:
salvacybersec
2026-04-06 17:58:13 +03:00
parent cd93703620
commit e2f87a62ef
4 changed files with 826 additions and 6 deletions

View File

@@ -358,14 +358,13 @@ Plans:
3. The keys page lists all findings with masked values and a "Reveal Key" toggle that shows the full key on demand
4. The recon page allows launching a recon sweep with source selection and shows live progress via Server-Sent Events
5. The REST API at `/api/v1/*` accepts and returns JSON for all dashboard actions; optional basic auth or token auth is configurable via settings page
**Plans**: 5 plans
**Plans**: 3 plans
Plans:
- [ ] 17-01-PLAN.md — Bot package skeleton: telego dependency, Bot struct, long polling, auth middleware
- [ ] 17-02-PLAN.md — Scheduler package + storage tables: gocron wrapper, subscribers/scheduled_jobs CRUD
- [ ] 17-03-PLAN.md — Bot command handlers: /scan, /verify, /recon, /status, /stats, /providers, /help, /key
- [ ] 17-04-PLAN.md — Subscribe/unsubscribe handlers + notification dispatcher (scheduler→bot bridge)
- [ ] 17-05-PLAN.md — CLI wiring: cmd/serve.go + cmd/schedule.go replacing stubs
- [ ] 18-01-PLAN.md — pkg/web foundation: chi router, go:embed static, layout template, overview page, auth middleware
- [ ] 18-02-PLAN.md — REST API handlers (/api/v1/*) + SSE hub for live progress
- [ ] 18-03-PLAN.md — HTML pages (keys, providers, scan, recon, dorks, settings) + cmd/serve.go wiring
**UI hint**: yes
## Progress

View File

@@ -0,0 +1,245 @@
---
phase: 18-web-dashboard
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- pkg/web/server.go
- pkg/web/auth.go
- pkg/web/handlers.go
- pkg/web/embed.go
- pkg/web/static/htmx.min.js
- pkg/web/static/style.css
- pkg/web/templates/layout.html
- pkg/web/templates/overview.html
- pkg/web/server_test.go
autonomous: true
requirements: [WEB-01, WEB-02, WEB-10]
must_haves:
truths:
- "chi v5 HTTP server starts on configurable port and serves embedded static assets"
- "Overview page renders with summary statistics from database"
- "Optional basic auth / token auth blocks unauthenticated requests when configured"
artifacts:
- path: "pkg/web/server.go"
provides: "chi router setup, middleware stack, NewServer constructor"
exports: ["Server", "NewServer", "Config"]
- path: "pkg/web/auth.go"
provides: "Basic auth and bearer token auth middleware"
exports: ["AuthMiddleware"]
- path: "pkg/web/handlers.go"
provides: "Overview page handler with stats aggregation"
exports: ["handleOverview"]
- path: "pkg/web/embed.go"
provides: "go:embed directives for static/ and templates/"
exports: ["staticFS", "templateFS"]
- path: "pkg/web/server_test.go"
provides: "Integration tests for server, auth, overview"
key_links:
- from: "pkg/web/server.go"
to: "pkg/storage"
via: "DB dependency in Config struct"
pattern: "storage\\.DB"
- from: "pkg/web/handlers.go"
to: "pkg/web/templates/overview.html"
via: "html/template rendering"
pattern: "template\\..*Execute"
- from: "pkg/web/server.go"
to: "pkg/web/static/"
via: "go:embed + http.FileServer"
pattern: "http\\.FileServer"
---
<objective>
Create the pkg/web package foundation: chi v5 router, go:embed static assets (htmx.min.js, Tailwind CDN reference), html/template-based layout, overview dashboard page with stats, and optional auth middleware.
Purpose: Establishes the HTTP server skeleton that Plans 02 and 03 build upon.
Output: Working `pkg/web` package with chi router, static serving, layout template, overview page, auth middleware.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/18-web-dashboard/18-CONTEXT.md
<interfaces>
<!-- Key types and contracts the executor needs. -->
From pkg/storage/db.go:
```go
type DB struct { ... }
func Open(path string) (*DB, error)
func (db *DB) Close() error
func (db *DB) SQL() *sql.DB
```
From pkg/storage/findings.go:
```go
type Finding struct {
ID, ScanID int64
ProviderName string
KeyValue, KeyMasked, Confidence string
SourcePath, SourceType string
LineNumber int
CreatedAt time.Time
Verified bool
VerifyStatus string
VerifyHTTPCode int
VerifyMetadata map[string]string
}
func (db *DB) ListFindings(encKey []byte) ([]Finding, error)
func (db *DB) SaveFinding(f Finding, encKey []byte) (int64, error)
```
From pkg/storage/queries.go:
```go
type Filters struct {
Provider, Confidence, SourceType string
Verified *bool
Limit, Offset int
}
func (db *DB) ListFindingsFiltered(encKey []byte, f Filters) ([]Finding, error)
func (db *DB) GetFinding(id int64, encKey []byte) (*Finding, error)
func (db *DB) DeleteFinding(id int64) (int64, error)
```
From pkg/providers/registry.go:
```go
type Registry struct { ... }
func NewRegistry() (*Registry, error)
func (r *Registry) List() []Provider
func (r *Registry) Stats() RegistryStats
```
From pkg/dorks/registry.go:
```go
type Registry struct { ... }
func NewRegistry() (*Registry, error)
func (r *Registry) List() []Dork
func (r *Registry) Stats() Stats
```
From pkg/recon/engine.go:
```go
type Engine struct { ... }
func NewEngine() *Engine
func (e *Engine) SweepAll(ctx context.Context, cfg Config) ([]Finding, error)
func (e *Engine) List() []string
```
</interfaces>
</context>
<tasks>
<task type="auto">
<name>Task 1: chi v5 dependency + go:embed static assets + layout template</name>
<files>pkg/web/embed.go, pkg/web/static/htmx.min.js, pkg/web/static/style.css, pkg/web/templates/layout.html, pkg/web/templates/overview.html</files>
<action>
1. Run `go get github.com/go-chi/chi/v5@v5.2.5` to add chi v5 to go.mod.
2. Create `pkg/web/embed.go`:
- `//go:embed static/*` into `var staticFiles embed.FS`
- `//go:embed templates/*` into `var templateFiles embed.FS`
- Export both via package-level vars.
3. Download htmx v2.0.4 minified JS (curl from unpkg.com/htmx.org@2.0.4/dist/htmx.min.js) and save to `pkg/web/static/htmx.min.js`.
4. Create `pkg/web/static/style.css` with minimal custom styles (body font, table styling, card class). The layout will load Tailwind v4 from CDN (`https://cdn.tailwindcss.com`) per the CONTEXT.md deferred decision. The local style.css is for overrides only.
5. Create `pkg/web/templates/layout.html` — html/template (NOT templ, per deferred decision):
- DOCTYPE, html, head with Tailwind CDN link, htmx.min.js script tag (served from /static/htmx.min.js), local style.css link
- Navigation bar: KeyHunter brand, links to Overview (/), Keys (/keys), Providers (/providers), Recon (/recon), Dorks (/dorks), Settings (/settings)
- `{{block "content" .}}{{end}}` placeholder for page content
- Use `{{define "layout"}}...{{end}}` wrapping pattern so pages extend it
6. Create `pkg/web/templates/overview.html` extending layout:
- `{{template "layout" .}}` with `{{define "content"}}` block
- Four stat cards in a Tailwind grid (lg:grid-cols-4, sm:grid-cols-2): Total Keys, Providers Loaded, Recon Sources, Last Scan
- Recent findings table showing last 10 keys (masked): Provider, Masked Key, Source, Confidence, Date
- Data struct: `OverviewData{TotalKeys int, TotalProviders int, ReconSources int, LastScan string, RecentFindings []storage.Finding}`
</action>
<verify>
<automated>cd /home/salva/Documents/apikey && go build ./pkg/web/...</automated>
</verify>
<done>pkg/web/embed.go compiles with go:embed directives, htmx.min.js is vendored, layout.html and overview.html parse without errors, chi v5 is in go.mod</done>
</task>
<task type="auto" tdd="true">
<name>Task 2: Server struct, auth middleware, overview handler, and tests</name>
<files>pkg/web/server.go, pkg/web/auth.go, pkg/web/handlers.go, pkg/web/server_test.go</files>
<behavior>
- Test: GET / returns 200 with "KeyHunter" in body (overview page renders)
- Test: GET /static/htmx.min.js returns 200 with JS content
- Test: GET / with auth enabled but no credentials returns 401
- Test: GET / with correct basic auth returns 200
- Test: GET / with correct bearer token returns 200
- Test: Overview page shows provider count and key count from injected data
</behavior>
<action>
1. Create `pkg/web/server.go`:
- `type Config struct { DB *storage.DB; EncKey []byte; Providers *providers.Registry; Dorks *dorks.Registry; ReconEngine *recon.Engine; Port int; AuthUser string; AuthPass string; AuthToken string }` — all fields the server needs
- `type Server struct { router chi.Router; cfg Config; tmpl *template.Template }`
- `func NewServer(cfg Config) (*Server, error)` — parses all templates from templateFiles embed.FS, builds chi.Router
- Router setup: `chi.NewRouter()`, use `middleware.Logger`, `middleware.Recoverer`, `middleware.RealIP`
- If AuthUser or AuthToken is set, apply AuthMiddleware (from auth.go)
- Mount `/static/` serving from staticFiles embed.FS (use `http.StripPrefix` + `http.FileServer(http.FS(...))`)
- Register routes: `GET /` -> handleOverview
- `func (s *Server) ListenAndServe() error` — starts `http.Server` on `cfg.Port`
- `func (s *Server) Router() chi.Router` — expose for testing
2. Create `pkg/web/auth.go`:
- `func AuthMiddleware(user, pass, token string) func(http.Handler) http.Handler`
- Check Authorization header: if "Bearer <token>" matches configured token, pass through
- If "Basic <base64>" matches user:pass, pass through
- Otherwise return 401 with `WWW-Authenticate: Basic realm="keyhunter"` header
- If all auth fields are empty strings, middleware is a no-op passthrough
3. Create `pkg/web/handlers.go`:
- `type OverviewData struct { TotalKeys, TotalProviders, ReconSources int; LastScan string; RecentFindings []storage.Finding; PageTitle string }`
- `func (s *Server) handleOverview(w http.ResponseWriter, r *http.Request)`
- Query: count findings via `len(db.ListFindingsFiltered(encKey, Filters{Limit: 10}))` for recent, run a COUNT query on the SQL for total
- Provider count from `s.cfg.Providers.Stats().Total` (or `len(s.cfg.Providers.List())`)
- Recon sources from `len(s.cfg.ReconEngine.List())`
- Render overview template with OverviewData
4. Create `pkg/web/server_test.go`:
- Use `httptest.NewRecorder` + `httptest.NewRequest` against `s.Router()`
- Test overview returns 200 with "KeyHunter" in body
- Test static asset serving
- Test auth middleware (401 without creds, 200 with basic auth, 200 with bearer token)
- For DB-dependent tests, use in-memory SQLite (`storage.Open(":memory:")`) or skip DB and test the router/auth independently with a nil-safe overview (show zeroes when DB is nil)
</action>
<verify>
<automated>cd /home/salva/Documents/apikey && go test ./pkg/web/... -v -count=1</automated>
</verify>
<done>Server starts with chi router, static assets served via go:embed, overview page renders with stats, auth middleware blocks unauthenticated requests when configured, all tests pass</done>
</task>
</tasks>
<verification>
- `go build ./pkg/web/...` compiles without errors
- `go test ./pkg/web/... -v` — all tests pass
- `go vet ./pkg/web/...` — no issues
</verification>
<success_criteria>
- chi v5.2.5 in go.mod
- pkg/web/server.go exports Server, NewServer, Config
- GET / returns overview HTML with stat cards
- GET /static/htmx.min.js returns vendored htmx
- Auth middleware returns 401 when credentials missing (when auth configured)
- Auth middleware passes with valid basic auth or bearer token
</success_criteria>
<output>
After completion, create `.planning/phases/18-web-dashboard/18-01-SUMMARY.md`
</output>

View File

@@ -0,0 +1,259 @@
---
phase: 18-web-dashboard
plan: 02
type: execute
wave: 1
depends_on: []
files_modified:
- pkg/web/api.go
- pkg/web/sse.go
- pkg/web/api_test.go
- pkg/web/sse_test.go
autonomous: true
requirements: [WEB-03, WEB-09, WEB-11]
must_haves:
truths:
- "REST API at /api/v1/* returns JSON for keys, providers, scan, recon, dorks, config"
- "SSE endpoint streams live scan/recon progress events"
- "API endpoints support filtering, pagination, and proper HTTP status codes"
artifacts:
- path: "pkg/web/api.go"
provides: "All REST API handlers under /api/v1"
exports: ["mountAPI"]
- path: "pkg/web/sse.go"
provides: "SSE hub and endpoint handlers for live progress"
exports: ["SSEHub", "NewSSEHub"]
- path: "pkg/web/api_test.go"
provides: "HTTP tests for all API endpoints"
- path: "pkg/web/sse_test.go"
provides: "SSE connection and event broadcast tests"
key_links:
- from: "pkg/web/api.go"
to: "pkg/storage"
via: "DB queries for findings, config"
pattern: "s\\.cfg\\.DB\\."
- from: "pkg/web/api.go"
to: "pkg/providers"
via: "Provider listing and stats"
pattern: "s\\.cfg\\.Providers\\."
- from: "pkg/web/sse.go"
to: "pkg/web/api.go"
via: "scan/recon handlers publish events to SSEHub"
pattern: "s\\.sse\\.Broadcast"
---
<objective>
Implement all REST API endpoints (/api/v1/*) for programmatic access and the SSE hub for live scan/recon progress streaming.
Purpose: Provides the JSON data layer that both external API consumers and the htmx HTML pages (Plan 03) will use.
Output: Complete REST API + SSE infrastructure in pkg/web.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/18-web-dashboard/18-CONTEXT.md
<interfaces>
<!-- Key types and contracts the executor needs. -->
From pkg/storage/db.go + findings.go + queries.go:
```go
type DB struct { ... }
func (db *DB) SQL() *sql.DB
func (db *DB) ListFindingsFiltered(encKey []byte, f Filters) ([]Finding, error)
func (db *DB) GetFinding(id int64, encKey []byte) (*Finding, error)
func (db *DB) DeleteFinding(id int64) (int64, error)
func (db *DB) SaveFinding(f Finding, encKey []byte) (int64, error)
type Filters struct { Provider, Confidence, SourceType string; Verified *bool; Limit, Offset int }
type Finding struct { ID, ScanID int64; ProviderName, KeyValue, KeyMasked, Confidence, SourcePath, SourceType string; LineNumber int; CreatedAt time.Time; Verified bool; VerifyStatus string; VerifyHTTPCode int; VerifyMetadata map[string]string }
```
From pkg/providers/registry.go + schema.go:
```go
func (r *Registry) List() []Provider
func (r *Registry) Get(name string) (Provider, bool)
func (r *Registry) Stats() RegistryStats
type Provider struct { Name, DisplayName, Category, Confidence string; ... }
type RegistryStats struct { Total, ByCategory map[string]int; ... }
```
From pkg/dorks/registry.go + schema.go:
```go
func (r *Registry) List() []Dork
func (r *Registry) Get(id string) (Dork, bool)
func (r *Registry) ListBySource(source string) []Dork
func (r *Registry) Stats() Stats
type Dork struct { ID, Source, Category, Query, Description string; ... }
type Stats struct { Total int; BySource map[string]int }
```
From pkg/storage/custom_dorks.go:
```go
func (db *DB) SaveCustomDork(d CustomDork) (int64, error)
func (db *DB) ListCustomDorks() ([]CustomDork, error)
```
From pkg/recon/engine.go + source.go:
```go
func (e *Engine) SweepAll(ctx context.Context, cfg Config) ([]Finding, error)
func (e *Engine) List() []string
type Config struct { Stealth, RespectRobots bool; EnabledSources []string; Query string }
```
From pkg/engine/engine.go:
```go
func NewEngine(registry *providers.Registry) *Engine
func (e *Engine) Scan(ctx context.Context, src sources.Source, cfg ScanConfig) (<-chan Finding, error)
type ScanConfig struct { Workers int; Verify bool; VerifyTimeout time.Duration }
```
From pkg/storage/settings.go (viper config):
```go
// Config is managed via viper — read/write with viper.GetString/viper.Set
```
</interfaces>
</context>
<tasks>
<task type="auto" tdd="true">
<name>Task 1: REST API handlers for /api/v1/*</name>
<files>pkg/web/api.go, pkg/web/api_test.go</files>
<behavior>
- Test: GET /api/v1/stats returns JSON with totalKeys, totalProviders, reconSources fields
- Test: GET /api/v1/keys returns JSON array of findings (masked by default)
- Test: GET /api/v1/keys?provider=openai filters by provider
- Test: GET /api/v1/keys/:id returns single finding JSON or 404
- Test: DELETE /api/v1/keys/:id returns 204 on success, 404 if not found
- Test: GET /api/v1/providers returns JSON array of providers
- Test: GET /api/v1/providers/:name returns single provider or 404
- Test: POST /api/v1/scan with JSON body returns 202 Accepted (async)
- Test: POST /api/v1/recon with JSON body returns 202 Accepted (async)
- Test: GET /api/v1/dorks returns JSON array of dorks
- Test: POST /api/v1/dorks with valid JSON returns 201
- Test: GET /api/v1/config returns JSON config
- Test: PUT /api/v1/config updates config and returns 200
</behavior>
<action>
1. Create `pkg/web/api.go`:
- `func (s *Server) mountAPI(r chi.Router)` — sub-router under `/api/v1`
- All handlers set `Content-Type: application/json`
- Use `encoding/json` for marshal/unmarshal. Use `chi.URLParam(r, "id")` for path params.
2. Stats endpoint:
- `GET /api/v1/stats` -> `handleAPIStats`
- Query DB for total key count (SELECT COUNT(*) FROM findings), provider count from registry, recon source count from engine
- Return `{"totalKeys": N, "totalProviders": N, "reconSources": N, "lastScan": "..."}`
3. Keys endpoints:
- `GET /api/v1/keys` -> `handleAPIListKeys` — accepts query params: provider, confidence, limit (default 50), offset. Returns findings with KeyValue ALWAYS masked (API never exposes raw keys — use CLI `keys show` for that). Map Filters from query params.
- `GET /api/v1/keys/{id}` -> `handleAPIGetKey` — parse id from URL, call GetFinding, return masked. 404 if nil.
- `DELETE /api/v1/keys/{id}` -> `handleAPIDeleteKey` — call DeleteFinding, return 204. If rows=0, return 404.
4. Providers endpoints:
- `GET /api/v1/providers` -> `handleAPIListProviders` — return registry.List() as JSON
- `GET /api/v1/providers/{name}` -> `handleAPIGetProvider` — registry.Get(name), 404 if not found
5. Scan endpoint:
- `POST /api/v1/scan` -> `handleAPIScan` — accepts JSON `{"path": "/some/dir", "verify": false, "workers": 4}`. Launches scan in background goroutine. Returns 202 with `{"status": "started", "message": "scan initiated"}`. Progress sent via SSE (Plan 18-02 SSE hub). If scan engine or DB is nil, return 503.
6. Recon endpoint:
- `POST /api/v1/recon` -> `handleAPIRecon` — accepts JSON `{"query": "openai", "sources": ["github","shodan"], "stealth": false}`. Launches recon in background goroutine. Returns 202. Progress via SSE.
7. Dorks endpoints:
- `GET /api/v1/dorks` -> `handleAPIListDorks` — accepts optional query param `source` for filtering. Return dorks registry list.
- `POST /api/v1/dorks` -> `handleAPIAddDork` — accepts JSON with dork fields, saves as custom dork to DB. Returns 201.
8. Config endpoints:
- `GET /api/v1/config` -> `handleAPIGetConfig` — return viper.AllSettings() as JSON
- `PUT /api/v1/config` -> `handleAPIUpdateConfig` — accepts JSON object, iterate keys, call viper.Set for each. Write config with viper.WriteConfig(). Return 200.
9. Helper: `func writeJSON(w http.ResponseWriter, status int, v interface{})` and `func readJSON(r *http.Request, v interface{}) error` for DRY request/response handling.
10. Create `pkg/web/api_test.go`:
- Use httptest against a Server with in-memory SQLite DB, real providers registry, nil-safe recon engine
- Test each endpoint for happy path + error cases (404, bad input)
- For scan/recon POST tests, just verify 202 response (actual execution is async)
</action>
<verify>
<automated>cd /home/salva/Documents/apikey && go test ./pkg/web/... -run TestAPI -v -count=1</automated>
</verify>
<done>All /api/v1/* endpoints return correct JSON responses, proper HTTP status codes, filtering works, scan/recon return 202 for async operations</done>
</task>
<task type="auto" tdd="true">
<name>Task 2: SSE hub for live scan/recon progress</name>
<files>pkg/web/sse.go, pkg/web/sse_test.go</files>
<behavior>
- Test: SSE client connects to /api/v1/scan/progress and receives events
- Test: Broadcasting an event delivers to all connected clients
- Test: Client disconnect removes from subscriber list
- Test: SSE event format is "event: {type}\ndata: {json}\n\n"
</behavior>
<action>
1. Create `pkg/web/sse.go`:
- `type SSEEvent struct { Type string; Data interface{} }` — Type is "scan:progress", "scan:finding", "scan:complete", "recon:progress", "recon:finding", "recon:complete"
- `type SSEHub struct { clients map[chan SSEEvent]struct{}; mu sync.RWMutex }`
- `func NewSSEHub() *SSEHub`
- `func (h *SSEHub) Subscribe() chan SSEEvent` — creates buffered channel (cap 32), adds to clients map, returns
- `func (h *SSEHub) Unsubscribe(ch chan SSEEvent)` — removes from map, closes channel
- `func (h *SSEHub) Broadcast(evt SSEEvent)` — sends to all clients, skip if client buffer full (non-blocking send)
- `func (s *Server) handleSSEScanProgress(w http.ResponseWriter, r *http.Request)` — standard SSE handler:
- Set headers: `Content-Type: text/event-stream`, `Cache-Control: no-cache`, `Connection: keep-alive`
- Flush with `http.Flusher`
- Subscribe to hub, defer Unsubscribe
- Loop: read from channel, format as `event: {type}\ndata: {json}\n\n`, flush
- Break on request context done
- `func (s *Server) handleSSEReconProgress(w http.ResponseWriter, r *http.Request)` — same pattern, same hub (events distinguish scan vs recon via Type prefix)
- Add SSEHub field to Server struct, initialize in NewServer
2. Wire SSE into scan/recon handlers:
- In handleAPIScan (from api.go), the background goroutine should: iterate findings channel from engine.Scan, broadcast `SSEEvent{Type: "scan:finding", Data: finding}` for each, then broadcast `SSEEvent{Type: "scan:complete", Data: summary}` when done
- In handleAPIRecon, similar: broadcast recon progress events
3. Mount routes in mountAPI:
- `GET /api/v1/scan/progress` -> handleSSEScanProgress
- `GET /api/v1/recon/progress` -> handleSSEReconProgress
4. Create `pkg/web/sse_test.go`:
- Test hub subscribe/broadcast/unsubscribe lifecycle
- Test SSE HTTP handler using httptest — connect, send event via hub.Broadcast, verify SSE format in response body
- Test client disconnect (cancel request context, verify unsubscribed)
</action>
<verify>
<automated>cd /home/salva/Documents/apikey && go test ./pkg/web/... -run TestSSE -v -count=1</automated>
</verify>
<done>SSE hub broadcasts events to connected clients, scan/recon progress streams in real-time, client disconnect is handled cleanly, event format matches SSE spec</done>
</task>
</tasks>
<verification>
- `go test ./pkg/web/... -v` — all API and SSE tests pass
- `go vet ./pkg/web/...` — no issues
- Manual: `curl http://localhost:8080/api/v1/stats` returns JSON (when server wired in Plan 03)
</verification>
<success_criteria>
- GET /api/v1/stats returns JSON with totalKeys, totalProviders, reconSources
- GET /api/v1/keys returns filtered, paginated JSON array (always masked)
- GET/DELETE /api/v1/keys/{id} work with proper 404 handling
- GET /api/v1/providers and /api/v1/providers/{name} return provider data
- POST /api/v1/scan and /api/v1/recon return 202 and launch async work
- GET /api/v1/dorks returns dork list, POST /api/v1/dorks creates custom dork
- GET/PUT /api/v1/config read/write viper config
- SSE endpoints stream events in proper text/event-stream format
- All tests pass
</success_criteria>
<output>
After completion, create `.planning/phases/18-web-dashboard/18-02-SUMMARY.md`
</output>

View File

@@ -0,0 +1,317 @@
---
phase: 18-web-dashboard
plan: 03
type: execute
wave: 2
depends_on: ["18-01", "18-02"]
files_modified:
- pkg/web/templates/keys.html
- pkg/web/templates/providers.html
- pkg/web/templates/recon.html
- pkg/web/templates/dorks.html
- pkg/web/templates/settings.html
- pkg/web/templates/scan.html
- pkg/web/handlers.go
- pkg/web/server.go
- cmd/serve.go
- pkg/web/handlers_test.go
autonomous: false
requirements: [WEB-03, WEB-04, WEB-05, WEB-06, WEB-07, WEB-08]
must_haves:
truths:
- "User can browse keys with filtering, click Reveal to unmask, click Copy"
- "User can view provider list with statistics"
- "User can launch recon sweep from web UI and see live results via SSE"
- "User can view and manage dorks"
- "User can view and edit settings"
- "User can trigger scan from web UI and see live progress"
- "keyhunter serve --port=8080 starts full web dashboard"
artifacts:
- path: "pkg/web/templates/keys.html"
provides: "Keys listing page with filter, reveal, copy"
- path: "pkg/web/templates/providers.html"
provides: "Provider listing with stats"
- path: "pkg/web/templates/recon.html"
provides: "Recon launcher with SSE live results"
- path: "pkg/web/templates/dorks.html"
provides: "Dork listing and management"
- path: "pkg/web/templates/settings.html"
provides: "Config editor"
- path: "pkg/web/templates/scan.html"
provides: "Scan launcher with SSE live progress"
- path: "cmd/serve.go"
provides: "HTTP server wired into CLI"
key_links:
- from: "pkg/web/templates/keys.html"
to: "/api/v1/keys"
via: "htmx hx-get for filtering and pagination"
pattern: "hx-get.*api/v1/keys"
- from: "pkg/web/templates/recon.html"
to: "/api/v1/recon/progress"
via: "EventSource SSE connection"
pattern: "EventSource.*recon/progress"
- from: "pkg/web/templates/scan.html"
to: "/api/v1/scan/progress"
via: "EventSource SSE connection"
pattern: "EventSource.*scan/progress"
- from: "cmd/serve.go"
to: "pkg/web"
via: "web.NewServer(cfg) + ListenAndServe"
pattern: "web\\.NewServer"
---
<objective>
Create all remaining HTML pages (keys, providers, recon, dorks, scan, settings) using htmx for interactivity and SSE for live updates, then wire the HTTP server into cmd/serve.go so `keyhunter serve` launches the full dashboard.
Purpose: Completes the user-facing web dashboard and makes it accessible via the CLI.
Output: Full dashboard with all pages + cmd/serve.go wiring.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/18-web-dashboard/18-CONTEXT.md
@.planning/phases/18-web-dashboard/18-01-SUMMARY.md
@.planning/phases/18-web-dashboard/18-02-SUMMARY.md
<interfaces>
<!-- From Plan 18-01 (Server foundation): -->
```go
// pkg/web/server.go
type Config struct {
DB *storage.DB
EncKey []byte
Providers *providers.Registry
Dorks *dorks.Registry
ReconEngine *recon.Engine
Port int
AuthUser string
AuthPass string
AuthToken string
}
type Server struct { router chi.Router; cfg Config; tmpl *template.Template; sse *SSEHub }
func NewServer(cfg Config) (*Server, error)
func (s *Server) ListenAndServe() error
func (s *Server) Router() chi.Router
```
```go
// pkg/web/embed.go
var staticFiles embed.FS // //go:embed static/*
var templateFiles embed.FS // //go:embed templates/*
```
```go
// pkg/web/auth.go
func AuthMiddleware(user, pass, token string) func(http.Handler) http.Handler
```
<!-- From Plan 18-02 (API + SSE): -->
```go
// pkg/web/api.go
func (s *Server) mountAPI(r chi.Router) // mounts /api/v1/*
func writeJSON(w http.ResponseWriter, status int, v interface{})
```
```go
// pkg/web/sse.go
type SSEHub struct { ... }
func NewSSEHub() *SSEHub
func (h *SSEHub) Broadcast(evt SSEEvent)
type SSEEvent struct { Type string; Data interface{} }
```
<!-- From cmd/serve.go (existing): -->
```go
var servePort int
var serveTelegram bool
var serveCmd = &cobra.Command{ Use: "serve", ... }
// Currently only starts Telegram bot — needs HTTP server wiring
```
<!-- From cmd/ helpers (existing pattern): -->
```go
func openDBWithKey() (*storage.DB, []byte, error) // returns DB + encryption key
```
</interfaces>
</context>
<tasks>
<task type="auto">
<name>Task 1: HTML pages with htmx interactivity + page handlers</name>
<files>pkg/web/templates/keys.html, pkg/web/templates/providers.html, pkg/web/templates/recon.html, pkg/web/templates/dorks.html, pkg/web/templates/settings.html, pkg/web/templates/scan.html, pkg/web/handlers.go, pkg/web/server.go, pkg/web/handlers_test.go</files>
<action>
1. **keys.html** — extends layout (WEB-04):
- Filter bar: provider dropdown (populated server-side from registry), confidence dropdown, text filter. Use `hx-get="/keys" hx-target="#keys-table" hx-include="[name='provider'],[name='confidence']"` for htmx-driven filtering.
- Keys table: ID, Provider, Masked Key, Source, Confidence, Verified, Date columns
- "Reveal" button per row: uses a small inline script or htmx `hx-get="/api/v1/keys/{id}"` that replaces the masked value cell. Since API always returns masked, the Reveal button uses a `data-key` attribute with the masked key from server render; for actual reveal, a dedicated handler `/keys/{id}/reveal` renders the unmasked key value (server-side, not API — the web dashboard can show unmasked to authenticated users).
- "Copy" button: `navigator.clipboard.writeText()` on the revealed key value
- "Delete" button: `hx-delete="/api/v1/keys/{id}" hx-confirm="Delete this key?" hx-target="closest tr" hx-swap="outerHTML"` — removes row on success
- Pagination: "Load more" button via `hx-get="/keys?offset=N" hx-target="#keys-table" hx-swap="beforeend"`
2. **providers.html** — extends layout (WEB-06):
- Stats summary bar: total count, per-category counts in badges
- Provider table: Name, Category, Confidence, Keywords count, Has Verify
- Filter by category via htmx dropdown
- Click provider name -> expand row with details (patterns, verify endpoint) via `hx-get="/api/v1/providers/{name}" hx-target="#detail-{name}"`
3. **scan.html** — extends layout (WEB-03):
- Form: Path input, verify checkbox, workers number input
- "Start Scan" button: `hx-post="/api/v1/scan"` with JSON body, shows progress section
- Progress section (hidden until scan starts): connects to SSE via inline script:
`const es = new EventSource('/api/v1/scan/progress');`
`es.addEventListener('scan:finding', (e) => { /* append row */ });`
`es.addEventListener('scan:complete', (e) => { es.close(); });`
- Results table: populated live via SSE events
4. **recon.html** — extends layout (WEB-05):
- Source checkboxes: populated from `recon.Engine.List()`, grouped by category
- Query input, stealth toggle, respect-robots toggle
- "Sweep" button: `hx-post="/api/v1/recon"` triggers sweep
- Live results via SSE (same pattern as scan.html with recon event types)
- Results displayed as cards showing provider, masked key, source
5. **dorks.html** — extends layout (WEB-07):
- Dork list table: ID, Source, Category, Query (truncated), Description
- Filter by source dropdown
- "Add Dork" form: source, category, query, description fields. `hx-post="/api/v1/dorks"` to create.
- Stats bar: total dorks, per-source counts
6. **settings.html** — extends layout (WEB-08):
- Config form populated from viper settings (rendered server-side)
- Key fields: database path, encryption, telegram token (masked), default workers, verify timeout
- "Save" button: `hx-put="/api/v1/config"` with form data as JSON
- Success/error toast notification via htmx `hx-swap-oob`
7. **Update handlers.go** — add page handlers:
- `handleKeys(w, r)` — render keys.html with initial data (first 50 findings, provider list for filter dropdown)
- `handleKeyReveal(w, r)` — GET /keys/{id}/reveal — returns unmasked key value as HTML fragment (for htmx swap)
- `handleProviders(w, r)` — render providers.html with provider list + stats
- `handleScan(w, r)` — render scan.html
- `handleRecon(w, r)` — render recon.html with source list
- `handleDorks(w, r)` — render dorks.html with dork list + stats
- `handleSettings(w, r)` — render settings.html with current config
8. **Update server.go** — register new routes in the router:
- `GET /keys` -> handleKeys
- `GET /keys/{id}/reveal` -> handleKeyReveal
- `GET /providers` -> handleProviders
- `GET /scan` -> handleScan
- `GET /recon` -> handleRecon
- `GET /dorks` -> handleDorks
- `GET /settings` -> handleSettings
9. **Create handlers_test.go**:
- Test each page handler returns 200 with expected content
- Test keys page contains "keys-table" div
- Test providers page lists provider names
- Test key reveal returns unmasked value
</action>
<verify>
<automated>cd /home/salva/Documents/apikey && go test ./pkg/web/... -v -count=1</automated>
</verify>
<done>All 6 page templates render correctly, htmx attributes are present for interactive features, SSE JavaScript is embedded in scan and recon pages, page handlers serve data from real packages, all tests pass</done>
</task>
<task type="auto">
<name>Task 2: Wire HTTP server into cmd/serve.go</name>
<files>cmd/serve.go</files>
<action>
1. Update cmd/serve.go RunE function:
- Import `github.com/salvacybersec/keyhunter/pkg/web`
- Import `github.com/salvacybersec/keyhunter/pkg/dorks`
- After existing DB/provider/recon setup, create web server:
```go
reg, err := providers.NewRegistry()
dorkReg, err := dorks.NewRegistry()
reconEng := recon.NewEngine()
// ... (register recon sources if needed)
srv, err := web.NewServer(web.Config{
DB: db,
EncKey: encKey,
Providers: reg,
Dorks: dorkReg,
ReconEngine: reconEng,
Port: servePort,
AuthUser: viper.GetString("web.auth_user"),
AuthPass: viper.GetString("web.auth_pass"),
AuthToken: viper.GetString("web.auth_token"),
})
```
- Start HTTP server in a goroutine: `go srv.ListenAndServe()`
- Keep existing Telegram bot start logic (conditioned on --telegram flag)
- Update the port message: `fmt.Printf("KeyHunter dashboard running at http://localhost:%d\n", servePort)`
- The existing `<-ctx.Done()` already handles graceful shutdown
2. Add serve flags:
- `--no-web` flag (default false) to disable web dashboard (for telegram-only mode)
- `--auth-user`, `--auth-pass`, `--auth-token` flags bound to viper `web.auth_user`, `web.auth_pass`, `web.auth_token`
3. Ensure the DB is opened unconditionally (it currently only opens when --telegram is set):
- Move `openDBWithKey()` call before the telegram conditional
- Both web server and telegram bot share the same DB instance
</action>
<verify>
<automated>cd /home/salva/Documents/apikey && go build -o /dev/null ./cmd/... && echo "build OK"</automated>
</verify>
<done>`keyhunter serve` starts HTTP server on port 8080 with full dashboard, --telegram additionally starts bot, --port changes listen port, --auth-user/pass/token enable auth, `go build ./cmd/...` succeeds</done>
</task>
<task type="checkpoint:human-verify" gate="blocking">
<name>Task 3: Visual verification of complete web dashboard</name>
<action>Human verifies the full dashboard renders and functions correctly in browser.</action>
<verify>
<automated>cd /home/salva/Documents/apikey && go build -o /dev/null ./cmd/... && go test ./pkg/web/... -count=1</automated>
</verify>
<done>All pages render, navigation works, API returns JSON, server starts and stops cleanly</done>
<what-built>Complete web dashboard: overview, keys (with reveal/copy/delete), providers, scan (with SSE live progress), recon (with SSE live results), dorks, and settings pages. HTTP server wired into `keyhunter serve`.</what-built>
<how-to-verify>
1. Run: `cd /home/salva/Documents/apikey && go run . serve --port=9090`
2. Open browser: http://localhost:9090
3. Verify overview page shows stat cards and navigation bar
4. Click "Keys" — verify table renders (may be empty if no scans done)
5. Click "Providers" — verify 108+ providers listed with categories
6. Click "Dorks" — verify dork list renders
7. Click "Settings" — verify config form renders
8. Test API: `curl http://localhost:9090/api/v1/stats` — verify JSON response
9. Test API: `curl http://localhost:9090/api/v1/providers | head -c 200` — verify provider JSON
10. Stop server with Ctrl+C — verify clean shutdown
</how-to-verify>
<resume-signal>Type "approved" or describe issues</resume-signal>
</task>
</tasks>
<verification>
- `go build ./cmd/...` compiles without errors
- `go test ./pkg/web/... -v` — all tests pass
- `keyhunter serve --port=9090` starts and serves dashboard at http://localhost:9090
- All 7 pages render (overview, keys, providers, scan, recon, dorks, settings)
- Navigation links work
- htmx interactions work (filtering, delete)
- SSE streams work (scan and recon progress)
- API endpoints return proper JSON
</verification>
<success_criteria>
- All 7 HTML pages render with proper layout and navigation
- Keys page supports filtering, reveal, copy, delete via htmx
- Scan and recon pages show live progress via SSE
- Providers page shows 108+ providers with stats
- Settings page reads/writes config
- cmd/serve.go starts HTTP server + optional Telegram bot
- Auth middleware protects dashboard when credentials configured
</success_criteria>
<output>
After completion, create `.planning/phases/18-web-dashboard/18-03-SUMMARY.md`
</output>