diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 712fdba..2034e4a 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -358,14 +358,13 @@ Plans: 3. The keys page lists all findings with masked values and a "Reveal Key" toggle that shows the full key on demand 4. The recon page allows launching a recon sweep with source selection and shows live progress via Server-Sent Events 5. The REST API at `/api/v1/*` accepts and returns JSON for all dashboard actions; optional basic auth or token auth is configurable via settings page -**Plans**: 5 plans +**Plans**: 3 plans Plans: -- [ ] 17-01-PLAN.md — Bot package skeleton: telego dependency, Bot struct, long polling, auth middleware -- [ ] 17-02-PLAN.md — Scheduler package + storage tables: gocron wrapper, subscribers/scheduled_jobs CRUD -- [ ] 17-03-PLAN.md — Bot command handlers: /scan, /verify, /recon, /status, /stats, /providers, /help, /key -- [ ] 17-04-PLAN.md — Subscribe/unsubscribe handlers + notification dispatcher (scheduler→bot bridge) -- [ ] 17-05-PLAN.md — CLI wiring: cmd/serve.go + cmd/schedule.go replacing stubs +- [ ] 18-01-PLAN.md — pkg/web foundation: chi router, go:embed static, layout template, overview page, auth middleware +- [ ] 18-02-PLAN.md — REST API handlers (/api/v1/*) + SSE hub for live progress +- [ ] 18-03-PLAN.md — HTML pages (keys, providers, scan, recon, dorks, settings) + cmd/serve.go wiring + **UI hint**: yes ## Progress diff --git a/.planning/phases/18-web-dashboard/18-01-PLAN.md b/.planning/phases/18-web-dashboard/18-01-PLAN.md new file mode 100644 index 0000000..c6f4949 --- /dev/null +++ b/.planning/phases/18-web-dashboard/18-01-PLAN.md @@ -0,0 +1,245 @@ +--- +phase: 18-web-dashboard +plan: 01 +type: execute +wave: 1 +depends_on: [] +files_modified: + - pkg/web/server.go + - pkg/web/auth.go + - pkg/web/handlers.go + - pkg/web/embed.go + - pkg/web/static/htmx.min.js + - pkg/web/static/style.css + - pkg/web/templates/layout.html + - pkg/web/templates/overview.html + - pkg/web/server_test.go +autonomous: true +requirements: [WEB-01, WEB-02, WEB-10] + +must_haves: + truths: + - "chi v5 HTTP server starts on configurable port and serves embedded static assets" + - "Overview page renders with summary statistics from database" + - "Optional basic auth / token auth blocks unauthenticated requests when configured" + artifacts: + - path: "pkg/web/server.go" + provides: "chi router setup, middleware stack, NewServer constructor" + exports: ["Server", "NewServer", "Config"] + - path: "pkg/web/auth.go" + provides: "Basic auth and bearer token auth middleware" + exports: ["AuthMiddleware"] + - path: "pkg/web/handlers.go" + provides: "Overview page handler with stats aggregation" + exports: ["handleOverview"] + - path: "pkg/web/embed.go" + provides: "go:embed directives for static/ and templates/" + exports: ["staticFS", "templateFS"] + - path: "pkg/web/server_test.go" + provides: "Integration tests for server, auth, overview" + key_links: + - from: "pkg/web/server.go" + to: "pkg/storage" + via: "DB dependency in Config struct" + pattern: "storage\\.DB" + - from: "pkg/web/handlers.go" + to: "pkg/web/templates/overview.html" + via: "html/template rendering" + pattern: "template\\..*Execute" + - from: "pkg/web/server.go" + to: "pkg/web/static/" + via: "go:embed + http.FileServer" + pattern: "http\\.FileServer" +--- + + +Create the pkg/web package foundation: chi v5 router, go:embed static assets (htmx.min.js, Tailwind CDN reference), html/template-based layout, overview dashboard page with stats, and optional auth middleware. + +Purpose: Establishes the HTTP server skeleton that Plans 02 and 03 build upon. +Output: Working `pkg/web` package with chi router, static serving, layout template, overview page, auth middleware. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md +@.planning/phases/18-web-dashboard/18-CONTEXT.md + + + + +From pkg/storage/db.go: +```go +type DB struct { ... } +func Open(path string) (*DB, error) +func (db *DB) Close() error +func (db *DB) SQL() *sql.DB +``` + +From pkg/storage/findings.go: +```go +type Finding struct { + ID, ScanID int64 + ProviderName string + KeyValue, KeyMasked, Confidence string + SourcePath, SourceType string + LineNumber int + CreatedAt time.Time + Verified bool + VerifyStatus string + VerifyHTTPCode int + VerifyMetadata map[string]string +} +func (db *DB) ListFindings(encKey []byte) ([]Finding, error) +func (db *DB) SaveFinding(f Finding, encKey []byte) (int64, error) +``` + +From pkg/storage/queries.go: +```go +type Filters struct { + Provider, Confidence, SourceType string + Verified *bool + Limit, Offset int +} +func (db *DB) ListFindingsFiltered(encKey []byte, f Filters) ([]Finding, error) +func (db *DB) GetFinding(id int64, encKey []byte) (*Finding, error) +func (db *DB) DeleteFinding(id int64) (int64, error) +``` + +From pkg/providers/registry.go: +```go +type Registry struct { ... } +func NewRegistry() (*Registry, error) +func (r *Registry) List() []Provider +func (r *Registry) Stats() RegistryStats +``` + +From pkg/dorks/registry.go: +```go +type Registry struct { ... } +func NewRegistry() (*Registry, error) +func (r *Registry) List() []Dork +func (r *Registry) Stats() Stats +``` + +From pkg/recon/engine.go: +```go +type Engine struct { ... } +func NewEngine() *Engine +func (e *Engine) SweepAll(ctx context.Context, cfg Config) ([]Finding, error) +func (e *Engine) List() []string +``` + + + + + + + Task 1: chi v5 dependency + go:embed static assets + layout template + pkg/web/embed.go, pkg/web/static/htmx.min.js, pkg/web/static/style.css, pkg/web/templates/layout.html, pkg/web/templates/overview.html + +1. Run `go get github.com/go-chi/chi/v5@v5.2.5` to add chi v5 to go.mod. + +2. Create `pkg/web/embed.go`: + - `//go:embed static/*` into `var staticFiles embed.FS` + - `//go:embed templates/*` into `var templateFiles embed.FS` + - Export both via package-level vars. + +3. Download htmx v2.0.4 minified JS (curl from unpkg.com/htmx.org@2.0.4/dist/htmx.min.js) and save to `pkg/web/static/htmx.min.js`. + +4. Create `pkg/web/static/style.css` with minimal custom styles (body font, table styling, card class). The layout will load Tailwind v4 from CDN (`https://cdn.tailwindcss.com`) per the CONTEXT.md deferred decision. The local style.css is for overrides only. + +5. Create `pkg/web/templates/layout.html` — html/template (NOT templ, per deferred decision): + - DOCTYPE, html, head with Tailwind CDN link, htmx.min.js script tag (served from /static/htmx.min.js), local style.css link + - Navigation bar: KeyHunter brand, links to Overview (/), Keys (/keys), Providers (/providers), Recon (/recon), Dorks (/dorks), Settings (/settings) + - `{{block "content" .}}{{end}}` placeholder for page content + - Use `{{define "layout"}}...{{end}}` wrapping pattern so pages extend it + +6. Create `pkg/web/templates/overview.html` extending layout: + - `{{template "layout" .}}` with `{{define "content"}}` block + - Four stat cards in a Tailwind grid (lg:grid-cols-4, sm:grid-cols-2): Total Keys, Providers Loaded, Recon Sources, Last Scan + - Recent findings table showing last 10 keys (masked): Provider, Masked Key, Source, Confidence, Date + - Data struct: `OverviewData{TotalKeys int, TotalProviders int, ReconSources int, LastScan string, RecentFindings []storage.Finding}` + + + cd /home/salva/Documents/apikey && go build ./pkg/web/... + + pkg/web/embed.go compiles with go:embed directives, htmx.min.js is vendored, layout.html and overview.html parse without errors, chi v5 is in go.mod + + + + Task 2: Server struct, auth middleware, overview handler, and tests + pkg/web/server.go, pkg/web/auth.go, pkg/web/handlers.go, pkg/web/server_test.go + + - Test: GET / returns 200 with "KeyHunter" in body (overview page renders) + - Test: GET /static/htmx.min.js returns 200 with JS content + - Test: GET / with auth enabled but no credentials returns 401 + - Test: GET / with correct basic auth returns 200 + - Test: GET / with correct bearer token returns 200 + - Test: Overview page shows provider count and key count from injected data + + +1. Create `pkg/web/server.go`: + - `type Config struct { DB *storage.DB; EncKey []byte; Providers *providers.Registry; Dorks *dorks.Registry; ReconEngine *recon.Engine; Port int; AuthUser string; AuthPass string; AuthToken string }` — all fields the server needs + - `type Server struct { router chi.Router; cfg Config; tmpl *template.Template }` + - `func NewServer(cfg Config) (*Server, error)` — parses all templates from templateFiles embed.FS, builds chi.Router + - Router setup: `chi.NewRouter()`, use `middleware.Logger`, `middleware.Recoverer`, `middleware.RealIP` + - If AuthUser or AuthToken is set, apply AuthMiddleware (from auth.go) + - Mount `/static/` serving from staticFiles embed.FS (use `http.StripPrefix` + `http.FileServer(http.FS(...))`) + - Register routes: `GET /` -> handleOverview + - `func (s *Server) ListenAndServe() error` — starts `http.Server` on `cfg.Port` + - `func (s *Server) Router() chi.Router` — expose for testing + +2. Create `pkg/web/auth.go`: + - `func AuthMiddleware(user, pass, token string) func(http.Handler) http.Handler` + - Check Authorization header: if "Bearer " matches configured token, pass through + - If "Basic " matches user:pass, pass through + - Otherwise return 401 with `WWW-Authenticate: Basic realm="keyhunter"` header + - If all auth fields are empty strings, middleware is a no-op passthrough + +3. Create `pkg/web/handlers.go`: + - `type OverviewData struct { TotalKeys, TotalProviders, ReconSources int; LastScan string; RecentFindings []storage.Finding; PageTitle string }` + - `func (s *Server) handleOverview(w http.ResponseWriter, r *http.Request)` + - Query: count findings via `len(db.ListFindingsFiltered(encKey, Filters{Limit: 10}))` for recent, run a COUNT query on the SQL for total + - Provider count from `s.cfg.Providers.Stats().Total` (or `len(s.cfg.Providers.List())`) + - Recon sources from `len(s.cfg.ReconEngine.List())` + - Render overview template with OverviewData + +4. Create `pkg/web/server_test.go`: + - Use `httptest.NewRecorder` + `httptest.NewRequest` against `s.Router()` + - Test overview returns 200 with "KeyHunter" in body + - Test static asset serving + - Test auth middleware (401 without creds, 200 with basic auth, 200 with bearer token) + - For DB-dependent tests, use in-memory SQLite (`storage.Open(":memory:")`) or skip DB and test the router/auth independently with a nil-safe overview (show zeroes when DB is nil) + + + cd /home/salva/Documents/apikey && go test ./pkg/web/... -v -count=1 + + Server starts with chi router, static assets served via go:embed, overview page renders with stats, auth middleware blocks unauthenticated requests when configured, all tests pass + + + + + +- `go build ./pkg/web/...` compiles without errors +- `go test ./pkg/web/... -v` — all tests pass +- `go vet ./pkg/web/...` — no issues + + + +- chi v5.2.5 in go.mod +- pkg/web/server.go exports Server, NewServer, Config +- GET / returns overview HTML with stat cards +- GET /static/htmx.min.js returns vendored htmx +- Auth middleware returns 401 when credentials missing (when auth configured) +- Auth middleware passes with valid basic auth or bearer token + + + +After completion, create `.planning/phases/18-web-dashboard/18-01-SUMMARY.md` + diff --git a/.planning/phases/18-web-dashboard/18-02-PLAN.md b/.planning/phases/18-web-dashboard/18-02-PLAN.md new file mode 100644 index 0000000..e51cec9 --- /dev/null +++ b/.planning/phases/18-web-dashboard/18-02-PLAN.md @@ -0,0 +1,259 @@ +--- +phase: 18-web-dashboard +plan: 02 +type: execute +wave: 1 +depends_on: [] +files_modified: + - pkg/web/api.go + - pkg/web/sse.go + - pkg/web/api_test.go + - pkg/web/sse_test.go +autonomous: true +requirements: [WEB-03, WEB-09, WEB-11] + +must_haves: + truths: + - "REST API at /api/v1/* returns JSON for keys, providers, scan, recon, dorks, config" + - "SSE endpoint streams live scan/recon progress events" + - "API endpoints support filtering, pagination, and proper HTTP status codes" + artifacts: + - path: "pkg/web/api.go" + provides: "All REST API handlers under /api/v1" + exports: ["mountAPI"] + - path: "pkg/web/sse.go" + provides: "SSE hub and endpoint handlers for live progress" + exports: ["SSEHub", "NewSSEHub"] + - path: "pkg/web/api_test.go" + provides: "HTTP tests for all API endpoints" + - path: "pkg/web/sse_test.go" + provides: "SSE connection and event broadcast tests" + key_links: + - from: "pkg/web/api.go" + to: "pkg/storage" + via: "DB queries for findings, config" + pattern: "s\\.cfg\\.DB\\." + - from: "pkg/web/api.go" + to: "pkg/providers" + via: "Provider listing and stats" + pattern: "s\\.cfg\\.Providers\\." + - from: "pkg/web/sse.go" + to: "pkg/web/api.go" + via: "scan/recon handlers publish events to SSEHub" + pattern: "s\\.sse\\.Broadcast" +--- + + +Implement all REST API endpoints (/api/v1/*) for programmatic access and the SSE hub for live scan/recon progress streaming. + +Purpose: Provides the JSON data layer that both external API consumers and the htmx HTML pages (Plan 03) will use. +Output: Complete REST API + SSE infrastructure in pkg/web. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md +@.planning/phases/18-web-dashboard/18-CONTEXT.md + + + + +From pkg/storage/db.go + findings.go + queries.go: +```go +type DB struct { ... } +func (db *DB) SQL() *sql.DB +func (db *DB) ListFindingsFiltered(encKey []byte, f Filters) ([]Finding, error) +func (db *DB) GetFinding(id int64, encKey []byte) (*Finding, error) +func (db *DB) DeleteFinding(id int64) (int64, error) +func (db *DB) SaveFinding(f Finding, encKey []byte) (int64, error) +type Filters struct { Provider, Confidence, SourceType string; Verified *bool; Limit, Offset int } +type Finding struct { ID, ScanID int64; ProviderName, KeyValue, KeyMasked, Confidence, SourcePath, SourceType string; LineNumber int; CreatedAt time.Time; Verified bool; VerifyStatus string; VerifyHTTPCode int; VerifyMetadata map[string]string } +``` + +From pkg/providers/registry.go + schema.go: +```go +func (r *Registry) List() []Provider +func (r *Registry) Get(name string) (Provider, bool) +func (r *Registry) Stats() RegistryStats +type Provider struct { Name, DisplayName, Category, Confidence string; ... } +type RegistryStats struct { Total, ByCategory map[string]int; ... } +``` + +From pkg/dorks/registry.go + schema.go: +```go +func (r *Registry) List() []Dork +func (r *Registry) Get(id string) (Dork, bool) +func (r *Registry) ListBySource(source string) []Dork +func (r *Registry) Stats() Stats +type Dork struct { ID, Source, Category, Query, Description string; ... } +type Stats struct { Total int; BySource map[string]int } +``` + +From pkg/storage/custom_dorks.go: +```go +func (db *DB) SaveCustomDork(d CustomDork) (int64, error) +func (db *DB) ListCustomDorks() ([]CustomDork, error) +``` + +From pkg/recon/engine.go + source.go: +```go +func (e *Engine) SweepAll(ctx context.Context, cfg Config) ([]Finding, error) +func (e *Engine) List() []string +type Config struct { Stealth, RespectRobots bool; EnabledSources []string; Query string } +``` + +From pkg/engine/engine.go: +```go +func NewEngine(registry *providers.Registry) *Engine +func (e *Engine) Scan(ctx context.Context, src sources.Source, cfg ScanConfig) (<-chan Finding, error) +type ScanConfig struct { Workers int; Verify bool; VerifyTimeout time.Duration } +``` + +From pkg/storage/settings.go (viper config): +```go +// Config is managed via viper — read/write with viper.GetString/viper.Set +``` + + + + + + + Task 1: REST API handlers for /api/v1/* + pkg/web/api.go, pkg/web/api_test.go + + - Test: GET /api/v1/stats returns JSON with totalKeys, totalProviders, reconSources fields + - Test: GET /api/v1/keys returns JSON array of findings (masked by default) + - Test: GET /api/v1/keys?provider=openai filters by provider + - Test: GET /api/v1/keys/:id returns single finding JSON or 404 + - Test: DELETE /api/v1/keys/:id returns 204 on success, 404 if not found + - Test: GET /api/v1/providers returns JSON array of providers + - Test: GET /api/v1/providers/:name returns single provider or 404 + - Test: POST /api/v1/scan with JSON body returns 202 Accepted (async) + - Test: POST /api/v1/recon with JSON body returns 202 Accepted (async) + - Test: GET /api/v1/dorks returns JSON array of dorks + - Test: POST /api/v1/dorks with valid JSON returns 201 + - Test: GET /api/v1/config returns JSON config + - Test: PUT /api/v1/config updates config and returns 200 + + +1. Create `pkg/web/api.go`: + - `func (s *Server) mountAPI(r chi.Router)` — sub-router under `/api/v1` + - All handlers set `Content-Type: application/json` + - Use `encoding/json` for marshal/unmarshal. Use `chi.URLParam(r, "id")` for path params. + +2. Stats endpoint: + - `GET /api/v1/stats` -> `handleAPIStats` + - Query DB for total key count (SELECT COUNT(*) FROM findings), provider count from registry, recon source count from engine + - Return `{"totalKeys": N, "totalProviders": N, "reconSources": N, "lastScan": "..."}` + +3. Keys endpoints: + - `GET /api/v1/keys` -> `handleAPIListKeys` — accepts query params: provider, confidence, limit (default 50), offset. Returns findings with KeyValue ALWAYS masked (API never exposes raw keys — use CLI `keys show` for that). Map Filters from query params. + - `GET /api/v1/keys/{id}` -> `handleAPIGetKey` — parse id from URL, call GetFinding, return masked. 404 if nil. + - `DELETE /api/v1/keys/{id}` -> `handleAPIDeleteKey` — call DeleteFinding, return 204. If rows=0, return 404. + +4. Providers endpoints: + - `GET /api/v1/providers` -> `handleAPIListProviders` — return registry.List() as JSON + - `GET /api/v1/providers/{name}` -> `handleAPIGetProvider` — registry.Get(name), 404 if not found + +5. Scan endpoint: + - `POST /api/v1/scan` -> `handleAPIScan` — accepts JSON `{"path": "/some/dir", "verify": false, "workers": 4}`. Launches scan in background goroutine. Returns 202 with `{"status": "started", "message": "scan initiated"}`. Progress sent via SSE (Plan 18-02 SSE hub). If scan engine or DB is nil, return 503. + +6. Recon endpoint: + - `POST /api/v1/recon` -> `handleAPIRecon` — accepts JSON `{"query": "openai", "sources": ["github","shodan"], "stealth": false}`. Launches recon in background goroutine. Returns 202. Progress via SSE. + +7. Dorks endpoints: + - `GET /api/v1/dorks` -> `handleAPIListDorks` — accepts optional query param `source` for filtering. Return dorks registry list. + - `POST /api/v1/dorks` -> `handleAPIAddDork` — accepts JSON with dork fields, saves as custom dork to DB. Returns 201. + +8. Config endpoints: + - `GET /api/v1/config` -> `handleAPIGetConfig` — return viper.AllSettings() as JSON + - `PUT /api/v1/config` -> `handleAPIUpdateConfig` — accepts JSON object, iterate keys, call viper.Set for each. Write config with viper.WriteConfig(). Return 200. + +9. Helper: `func writeJSON(w http.ResponseWriter, status int, v interface{})` and `func readJSON(r *http.Request, v interface{}) error` for DRY request/response handling. + +10. Create `pkg/web/api_test.go`: + - Use httptest against a Server with in-memory SQLite DB, real providers registry, nil-safe recon engine + - Test each endpoint for happy path + error cases (404, bad input) + - For scan/recon POST tests, just verify 202 response (actual execution is async) + + + cd /home/salva/Documents/apikey && go test ./pkg/web/... -run TestAPI -v -count=1 + + All /api/v1/* endpoints return correct JSON responses, proper HTTP status codes, filtering works, scan/recon return 202 for async operations + + + + Task 2: SSE hub for live scan/recon progress + pkg/web/sse.go, pkg/web/sse_test.go + + - Test: SSE client connects to /api/v1/scan/progress and receives events + - Test: Broadcasting an event delivers to all connected clients + - Test: Client disconnect removes from subscriber list + - Test: SSE event format is "event: {type}\ndata: {json}\n\n" + + +1. Create `pkg/web/sse.go`: + - `type SSEEvent struct { Type string; Data interface{} }` — Type is "scan:progress", "scan:finding", "scan:complete", "recon:progress", "recon:finding", "recon:complete" + - `type SSEHub struct { clients map[chan SSEEvent]struct{}; mu sync.RWMutex }` + - `func NewSSEHub() *SSEHub` + - `func (h *SSEHub) Subscribe() chan SSEEvent` — creates buffered channel (cap 32), adds to clients map, returns + - `func (h *SSEHub) Unsubscribe(ch chan SSEEvent)` — removes from map, closes channel + - `func (h *SSEHub) Broadcast(evt SSEEvent)` — sends to all clients, skip if client buffer full (non-blocking send) + - `func (s *Server) handleSSEScanProgress(w http.ResponseWriter, r *http.Request)` — standard SSE handler: + - Set headers: `Content-Type: text/event-stream`, `Cache-Control: no-cache`, `Connection: keep-alive` + - Flush with `http.Flusher` + - Subscribe to hub, defer Unsubscribe + - Loop: read from channel, format as `event: {type}\ndata: {json}\n\n`, flush + - Break on request context done + - `func (s *Server) handleSSEReconProgress(w http.ResponseWriter, r *http.Request)` — same pattern, same hub (events distinguish scan vs recon via Type prefix) + - Add SSEHub field to Server struct, initialize in NewServer + +2. Wire SSE into scan/recon handlers: + - In handleAPIScan (from api.go), the background goroutine should: iterate findings channel from engine.Scan, broadcast `SSEEvent{Type: "scan:finding", Data: finding}` for each, then broadcast `SSEEvent{Type: "scan:complete", Data: summary}` when done + - In handleAPIRecon, similar: broadcast recon progress events + +3. Mount routes in mountAPI: + - `GET /api/v1/scan/progress` -> handleSSEScanProgress + - `GET /api/v1/recon/progress` -> handleSSEReconProgress + +4. Create `pkg/web/sse_test.go`: + - Test hub subscribe/broadcast/unsubscribe lifecycle + - Test SSE HTTP handler using httptest — connect, send event via hub.Broadcast, verify SSE format in response body + - Test client disconnect (cancel request context, verify unsubscribed) + + + cd /home/salva/Documents/apikey && go test ./pkg/web/... -run TestSSE -v -count=1 + + SSE hub broadcasts events to connected clients, scan/recon progress streams in real-time, client disconnect is handled cleanly, event format matches SSE spec + + + + + +- `go test ./pkg/web/... -v` — all API and SSE tests pass +- `go vet ./pkg/web/...` — no issues +- Manual: `curl http://localhost:8080/api/v1/stats` returns JSON (when server wired in Plan 03) + + + +- GET /api/v1/stats returns JSON with totalKeys, totalProviders, reconSources +- GET /api/v1/keys returns filtered, paginated JSON array (always masked) +- GET/DELETE /api/v1/keys/{id} work with proper 404 handling +- GET /api/v1/providers and /api/v1/providers/{name} return provider data +- POST /api/v1/scan and /api/v1/recon return 202 and launch async work +- GET /api/v1/dorks returns dork list, POST /api/v1/dorks creates custom dork +- GET/PUT /api/v1/config read/write viper config +- SSE endpoints stream events in proper text/event-stream format +- All tests pass + + + +After completion, create `.planning/phases/18-web-dashboard/18-02-SUMMARY.md` + diff --git a/.planning/phases/18-web-dashboard/18-03-PLAN.md b/.planning/phases/18-web-dashboard/18-03-PLAN.md new file mode 100644 index 0000000..e72d931 --- /dev/null +++ b/.planning/phases/18-web-dashboard/18-03-PLAN.md @@ -0,0 +1,317 @@ +--- +phase: 18-web-dashboard +plan: 03 +type: execute +wave: 2 +depends_on: ["18-01", "18-02"] +files_modified: + - pkg/web/templates/keys.html + - pkg/web/templates/providers.html + - pkg/web/templates/recon.html + - pkg/web/templates/dorks.html + - pkg/web/templates/settings.html + - pkg/web/templates/scan.html + - pkg/web/handlers.go + - pkg/web/server.go + - cmd/serve.go + - pkg/web/handlers_test.go +autonomous: false +requirements: [WEB-03, WEB-04, WEB-05, WEB-06, WEB-07, WEB-08] + +must_haves: + truths: + - "User can browse keys with filtering, click Reveal to unmask, click Copy" + - "User can view provider list with statistics" + - "User can launch recon sweep from web UI and see live results via SSE" + - "User can view and manage dorks" + - "User can view and edit settings" + - "User can trigger scan from web UI and see live progress" + - "keyhunter serve --port=8080 starts full web dashboard" + artifacts: + - path: "pkg/web/templates/keys.html" + provides: "Keys listing page with filter, reveal, copy" + - path: "pkg/web/templates/providers.html" + provides: "Provider listing with stats" + - path: "pkg/web/templates/recon.html" + provides: "Recon launcher with SSE live results" + - path: "pkg/web/templates/dorks.html" + provides: "Dork listing and management" + - path: "pkg/web/templates/settings.html" + provides: "Config editor" + - path: "pkg/web/templates/scan.html" + provides: "Scan launcher with SSE live progress" + - path: "cmd/serve.go" + provides: "HTTP server wired into CLI" + key_links: + - from: "pkg/web/templates/keys.html" + to: "/api/v1/keys" + via: "htmx hx-get for filtering and pagination" + pattern: "hx-get.*api/v1/keys" + - from: "pkg/web/templates/recon.html" + to: "/api/v1/recon/progress" + via: "EventSource SSE connection" + pattern: "EventSource.*recon/progress" + - from: "pkg/web/templates/scan.html" + to: "/api/v1/scan/progress" + via: "EventSource SSE connection" + pattern: "EventSource.*scan/progress" + - from: "cmd/serve.go" + to: "pkg/web" + via: "web.NewServer(cfg) + ListenAndServe" + pattern: "web\\.NewServer" +--- + + +Create all remaining HTML pages (keys, providers, recon, dorks, scan, settings) using htmx for interactivity and SSE for live updates, then wire the HTTP server into cmd/serve.go so `keyhunter serve` launches the full dashboard. + +Purpose: Completes the user-facing web dashboard and makes it accessible via the CLI. +Output: Full dashboard with all pages + cmd/serve.go wiring. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md +@.planning/phases/18-web-dashboard/18-CONTEXT.md +@.planning/phases/18-web-dashboard/18-01-SUMMARY.md +@.planning/phases/18-web-dashboard/18-02-SUMMARY.md + + + +```go +// pkg/web/server.go +type Config struct { + DB *storage.DB + EncKey []byte + Providers *providers.Registry + Dorks *dorks.Registry + ReconEngine *recon.Engine + Port int + AuthUser string + AuthPass string + AuthToken string +} +type Server struct { router chi.Router; cfg Config; tmpl *template.Template; sse *SSEHub } +func NewServer(cfg Config) (*Server, error) +func (s *Server) ListenAndServe() error +func (s *Server) Router() chi.Router +``` + +```go +// pkg/web/embed.go +var staticFiles embed.FS // //go:embed static/* +var templateFiles embed.FS // //go:embed templates/* +``` + +```go +// pkg/web/auth.go +func AuthMiddleware(user, pass, token string) func(http.Handler) http.Handler +``` + + +```go +// pkg/web/api.go +func (s *Server) mountAPI(r chi.Router) // mounts /api/v1/* +func writeJSON(w http.ResponseWriter, status int, v interface{}) +``` + +```go +// pkg/web/sse.go +type SSEHub struct { ... } +func NewSSEHub() *SSEHub +func (h *SSEHub) Broadcast(evt SSEEvent) +type SSEEvent struct { Type string; Data interface{} } +``` + + +```go +var servePort int +var serveTelegram bool +var serveCmd = &cobra.Command{ Use: "serve", ... } +// Currently only starts Telegram bot — needs HTTP server wiring +``` + + +```go +func openDBWithKey() (*storage.DB, []byte, error) // returns DB + encryption key +``` + + + + + + + Task 1: HTML pages with htmx interactivity + page handlers + pkg/web/templates/keys.html, pkg/web/templates/providers.html, pkg/web/templates/recon.html, pkg/web/templates/dorks.html, pkg/web/templates/settings.html, pkg/web/templates/scan.html, pkg/web/handlers.go, pkg/web/server.go, pkg/web/handlers_test.go + +1. **keys.html** — extends layout (WEB-04): + - Filter bar: provider dropdown (populated server-side from registry), confidence dropdown, text filter. Use `hx-get="/keys" hx-target="#keys-table" hx-include="[name='provider'],[name='confidence']"` for htmx-driven filtering. + - Keys table: ID, Provider, Masked Key, Source, Confidence, Verified, Date columns + - "Reveal" button per row: uses a small inline script or htmx `hx-get="/api/v1/keys/{id}"` that replaces the masked value cell. Since API always returns masked, the Reveal button uses a `data-key` attribute with the masked key from server render; for actual reveal, a dedicated handler `/keys/{id}/reveal` renders the unmasked key value (server-side, not API — the web dashboard can show unmasked to authenticated users). + - "Copy" button: `navigator.clipboard.writeText()` on the revealed key value + - "Delete" button: `hx-delete="/api/v1/keys/{id}" hx-confirm="Delete this key?" hx-target="closest tr" hx-swap="outerHTML"` — removes row on success + - Pagination: "Load more" button via `hx-get="/keys?offset=N" hx-target="#keys-table" hx-swap="beforeend"` + +2. **providers.html** — extends layout (WEB-06): + - Stats summary bar: total count, per-category counts in badges + - Provider table: Name, Category, Confidence, Keywords count, Has Verify + - Filter by category via htmx dropdown + - Click provider name -> expand row with details (patterns, verify endpoint) via `hx-get="/api/v1/providers/{name}" hx-target="#detail-{name}"` + +3. **scan.html** — extends layout (WEB-03): + - Form: Path input, verify checkbox, workers number input + - "Start Scan" button: `hx-post="/api/v1/scan"` with JSON body, shows progress section + - Progress section (hidden until scan starts): connects to SSE via inline script: + `const es = new EventSource('/api/v1/scan/progress');` + `es.addEventListener('scan:finding', (e) => { /* append row */ });` + `es.addEventListener('scan:complete', (e) => { es.close(); });` + - Results table: populated live via SSE events + +4. **recon.html** — extends layout (WEB-05): + - Source checkboxes: populated from `recon.Engine.List()`, grouped by category + - Query input, stealth toggle, respect-robots toggle + - "Sweep" button: `hx-post="/api/v1/recon"` triggers sweep + - Live results via SSE (same pattern as scan.html with recon event types) + - Results displayed as cards showing provider, masked key, source + +5. **dorks.html** — extends layout (WEB-07): + - Dork list table: ID, Source, Category, Query (truncated), Description + - Filter by source dropdown + - "Add Dork" form: source, category, query, description fields. `hx-post="/api/v1/dorks"` to create. + - Stats bar: total dorks, per-source counts + +6. **settings.html** — extends layout (WEB-08): + - Config form populated from viper settings (rendered server-side) + - Key fields: database path, encryption, telegram token (masked), default workers, verify timeout + - "Save" button: `hx-put="/api/v1/config"` with form data as JSON + - Success/error toast notification via htmx `hx-swap-oob` + +7. **Update handlers.go** — add page handlers: + - `handleKeys(w, r)` — render keys.html with initial data (first 50 findings, provider list for filter dropdown) + - `handleKeyReveal(w, r)` — GET /keys/{id}/reveal — returns unmasked key value as HTML fragment (for htmx swap) + - `handleProviders(w, r)` — render providers.html with provider list + stats + - `handleScan(w, r)` — render scan.html + - `handleRecon(w, r)` — render recon.html with source list + - `handleDorks(w, r)` — render dorks.html with dork list + stats + - `handleSettings(w, r)` — render settings.html with current config + +8. **Update server.go** — register new routes in the router: + - `GET /keys` -> handleKeys + - `GET /keys/{id}/reveal` -> handleKeyReveal + - `GET /providers` -> handleProviders + - `GET /scan` -> handleScan + - `GET /recon` -> handleRecon + - `GET /dorks` -> handleDorks + - `GET /settings` -> handleSettings + +9. **Create handlers_test.go**: + - Test each page handler returns 200 with expected content + - Test keys page contains "keys-table" div + - Test providers page lists provider names + - Test key reveal returns unmasked value + + + cd /home/salva/Documents/apikey && go test ./pkg/web/... -v -count=1 + + All 6 page templates render correctly, htmx attributes are present for interactive features, SSE JavaScript is embedded in scan and recon pages, page handlers serve data from real packages, all tests pass + + + + Task 2: Wire HTTP server into cmd/serve.go + cmd/serve.go + +1. Update cmd/serve.go RunE function: + - Import `github.com/salvacybersec/keyhunter/pkg/web` + - Import `github.com/salvacybersec/keyhunter/pkg/dorks` + - After existing DB/provider/recon setup, create web server: + ```go + reg, err := providers.NewRegistry() + dorkReg, err := dorks.NewRegistry() + reconEng := recon.NewEngine() + // ... (register recon sources if needed) + + srv, err := web.NewServer(web.Config{ + DB: db, + EncKey: encKey, + Providers: reg, + Dorks: dorkReg, + ReconEngine: reconEng, + Port: servePort, + AuthUser: viper.GetString("web.auth_user"), + AuthPass: viper.GetString("web.auth_pass"), + AuthToken: viper.GetString("web.auth_token"), + }) + ``` + - Start HTTP server in a goroutine: `go srv.ListenAndServe()` + - Keep existing Telegram bot start logic (conditioned on --telegram flag) + - Update the port message: `fmt.Printf("KeyHunter dashboard running at http://localhost:%d\n", servePort)` + - The existing `<-ctx.Done()` already handles graceful shutdown + +2. Add serve flags: + - `--no-web` flag (default false) to disable web dashboard (for telegram-only mode) + - `--auth-user`, `--auth-pass`, `--auth-token` flags bound to viper `web.auth_user`, `web.auth_pass`, `web.auth_token` + +3. Ensure the DB is opened unconditionally (it currently only opens when --telegram is set): + - Move `openDBWithKey()` call before the telegram conditional + - Both web server and telegram bot share the same DB instance + + + cd /home/salva/Documents/apikey && go build -o /dev/null ./cmd/... && echo "build OK" + + `keyhunter serve` starts HTTP server on port 8080 with full dashboard, --telegram additionally starts bot, --port changes listen port, --auth-user/pass/token enable auth, `go build ./cmd/...` succeeds + + + + Task 3: Visual verification of complete web dashboard + Human verifies the full dashboard renders and functions correctly in browser. + + cd /home/salva/Documents/apikey && go build -o /dev/null ./cmd/... && go test ./pkg/web/... -count=1 + + All pages render, navigation works, API returns JSON, server starts and stops cleanly + Complete web dashboard: overview, keys (with reveal/copy/delete), providers, scan (with SSE live progress), recon (with SSE live results), dorks, and settings pages. HTTP server wired into `keyhunter serve`. + + 1. Run: `cd /home/salva/Documents/apikey && go run . serve --port=9090` + 2. Open browser: http://localhost:9090 + 3. Verify overview page shows stat cards and navigation bar + 4. Click "Keys" — verify table renders (may be empty if no scans done) + 5. Click "Providers" — verify 108+ providers listed with categories + 6. Click "Dorks" — verify dork list renders + 7. Click "Settings" — verify config form renders + 8. Test API: `curl http://localhost:9090/api/v1/stats` — verify JSON response + 9. Test API: `curl http://localhost:9090/api/v1/providers | head -c 200` — verify provider JSON + 10. Stop server with Ctrl+C — verify clean shutdown + + Type "approved" or describe issues + + + + + +- `go build ./cmd/...` compiles without errors +- `go test ./pkg/web/... -v` — all tests pass +- `keyhunter serve --port=9090` starts and serves dashboard at http://localhost:9090 +- All 7 pages render (overview, keys, providers, scan, recon, dorks, settings) +- Navigation links work +- htmx interactions work (filtering, delete) +- SSE streams work (scan and recon progress) +- API endpoints return proper JSON + + + +- All 7 HTML pages render with proper layout and navigation +- Keys page supports filtering, reveal, copy, delete via htmx +- Scan and recon pages show live progress via SSE +- Providers page shows 108+ providers with stats +- Settings page reads/writes config +- cmd/serve.go starts HTTP server + optional Telegram bot +- Auth middleware protects dashboard when credentials configured + + + +After completion, create `.planning/phases/18-web-dashboard/18-03-SUMMARY.md` +