salvacybersec/keyhunter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: create roadmap (18 phases)

Browse Source
This commit is contained in:
salvacybersec
2026-04-04 19:12:41 +03:00
parent 6803863833
commit ee92aad4cf
4 changed files with 531 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

160
CLAUDE.md Normal file
View File

@@ -0,0 +1,160 @@
<!-- GSD:project-start source:PROJECT.md -->
## Project
**KeyHunter**
KeyHunter is a comprehensive, modular API key scanner built in Go, focused on detecting and validating API keys from 108+ LLM/AI providers. It combines native scanning with external tool integration (TruffleHog, Gitleaks), OSINT/recon across 80+ internet sources, a web dashboard, and Telegram bot notifications. Designed for red teams, DevSecOps, bug bounty hunters, and security researchers.
**Core Value:** Detect leaked LLM API keys across more providers and more internet sources than any other tool, with active verification to confirm keys are real and alive.
### Constraints
- **Language**: Go 1.22+ — single binary distribution, performance, TruffleHog/Gitleaks ecosystem alignment
- **Architecture**: Plugin-based — providers as YAML files, compile-time embedded via Go embed
- **Storage**: SQLite — zero-dependency embedded database, AES-256 encrypted
- **Web stack**: htmx + Tailwind CSS — no JS framework dependency, embedded in binary
- **CLI framework**: Cobra — industry standard for Go CLIs
- **Verification**: Must be opt-in (`--verify` flag) — passive scanning by default for legal safety
- **Key masking**: Default masked output, `--unmask` for full keys — shoulder surfing protection
<!-- GSD:project-end -->
<!-- GSD:stack-start source:research/STACK.md -->
## Technology Stack
## Recommended Stack
### Core CLI Framework
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `github.com/spf13/cobra` | v1.10.2 | CLI command tree (scan, verify, recon, keys, serve, dorks, providers, config, hook, schedule) | Industry standard for Go CLIs. Used by Kubernetes, Docker, GitHub CLI. Sub-command hierarchy, persistent flags, shell completion, man-page generation are all built in. No viable alternative — it IS the Go CLI standard. |
| `github.com/spf13/viper` | v1.21.0 | Configuration management (YAML/JSON/env/flags binding) | Designed to pair with Cobra. Handles config file + env var + CLI flag precedence chain automatically. v1.21.0 switched to maintained yaml lib, cleaning supply-chain issues. |
### Web Dashboard
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `github.com/go-chi/chi/v5` | v5.2.5 | HTTP router for dashboard and API | 100% net/http compatible — no custom context or handler types. Zero external dependencies. Routes embed naturally into `go:embed` serving. Used by major Go projects. Requires Go 1.22+ (matches project constraint). |
| `github.com/a-h/templ` | v0.3.1001 | Type-safe HTML template compilation | `.templ` files compile to Go — template errors are caught at compile time, not runtime. Composes naturally with HTMX. Significantly safer than `html/template` for a project with a public-facing dashboard. |
| htmx | v2.x (CDN or vendored) | Frontend interactivity without JS framework | Server-rendered with AJAX behavior. No build step. Aligns with "embed in binary" architecture constraint. Use `go:embed` to bundle the htmx.min.js into the binary. |
| Tailwind CSS | v4.x (standalone CLI) | Utility-first styling | v4 ships a standalone binary — no Node.js required. Use `@tailwindcss/cli` to compile a single CSS file, then `go:embed` it. Air watches both `.templ` and CSS changes during development. |
### Database
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `modernc.org/sqlite` | v1.35.x (SQLite 3.51.2 inside) | Embedded database for scan results, keys, recon data | Pure Go — no CGo, no C compiler requirement. Cross-compiles cleanly for Linux/macOS/ARM. Actively maintained (updated 2026-03-17). Zero external process dependency for single-binary distribution. |
| `database/sql` (stdlib) | — | SQL interface layer | Use standard library interface over `modernc.org/sqlite` directly — driver is registered as `"sqlite"`. No ORM needed for a tool of this scope. Raw SQL gives full control and avoids ORM magic bugs. |
### Concurrency
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `github.com/panjf2000/ants/v2` | v2.12.0 | Worker pool for parallel scanning across files, sources, and verification requests | Mature, battle-tested goroutine pool. Dynamically resizable. Handles thousands of concurrent tasks without goroutine explosion. Used in high-throughput Go systems. v2.12.0 adds ReleaseContext for clean shutdown. |
| `golang.org/x/time/rate` | latest x/ | Per-source rate limiting for OSINT/recon sources | Official Go extended library. Token bucket algorithm. Rate-limit each external source (Shodan, GitHub, etc.) independently. Used by TruffleHog for the same purpose. |
| `sync`, `context` (stdlib) | — | Cancellation, mutex, waitgroups | Standard library is sufficient for coordination between pool and caller. No additional abstraction needed. |
### YAML Provider/Dork Engine
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `gopkg.in/yaml.v3` | v3.x | Parse provider YAML definitions and dork YAML files embedded via `go:embed` | Direct, well-understood API. v3 handles inline/anchored structs correctly. The Cobra v1.10.2 release migrated away from it to `go.yaml.in/yaml/v3` — however for provider YAML parsing, gopkg.in/yaml.v3 remains stable and appropriate. |
| `embed` (stdlib) | — | Compile-time embedding of `/providers/*.yaml` and `/dorks/*.yaml` | Go 1.16+ native. No external dependency. Providers and dorks are baked into the binary at compile time — no runtime filesystem access needed. |
### Telegram Bot
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `github.com/mymmrac/telego` | v1.8.0 | Telegram bot for /scan, /verify, /recon, /status, /stats, /subscribe, /key commands | One-to-one Telegram Bot API v9.6 mapping. Supports long polling and webhooks. Type-safe API surface. v1.8.0 released 2026-04-03 with API v9.6 support. Actively maintained. |
### Scheduled Scanning
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `github.com/go-co-op/gocron/v2` | v2.19.1 | Cron-based recurring scans | Modern API, v2 has full context support and job lifecycle management. Race condition fix in v2.19.1 (important for scheduler reliability). Better API than robfig/cron v3. |
### Output and Formatting
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `github.com/charmbracelet/lipgloss` | latest | Colored terminal table output, status indicators | Declarative style definitions. Composes with any output stream. Used across the Go security tool ecosystem (TruffleHog uses it indirectly). |
| `github.com/charmbracelet/bubbles` | latest | Progress bars for long scans, spinners during verification | Pre-built terminal UI components. Pairs with lipgloss. Less overhead than full Bubble Tea TUI — use components only. |
| `encoding/json` (stdlib) | — | JSON output format | Standard library is sufficient. No external JSON library needed. |
| SARIF output | custom | CI/CD integration output | Implement SARIF 2.1.0 format directly — it is a straightforward JSON schema. Do not use a library; gosec's SARIF package is not designed for import. ~200 lines of struct definitions covers the needed schema. |
### HTTP Client (OSINT/Recon/Verification)
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `net/http` (stdlib) | — | All outbound HTTP requests for verification and OSINT | Standard library client is sufficient. Supports custom TLS, proxy settings, timeouts. Avoid adding httpclient wrappers that hide behavior. |
| `golang.org/x/time/rate` | latest | Per-source rate limiting on outbound requests | Already listed under concurrency — same package serves both purposes. |
### Development Tooling
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `github.com/air-verse/air` | latest | Hot reload during dashboard development | Watches Go + templ files, rebuilds on change. Industry standard for Go web dev loops. |
| `@tailwindcss/cli` | v4.x standalone | CSS compilation without Node.js | v4 standalone binary eliminates Node dependency entirely. Run as `tailwindcss -i input.css -o dist/style.css --watch` alongside air. |
| `golangci-lint` | latest | Static analysis and linting | Multi-linter runner. Include gosec, staticcheck, errcheck at minimum. |
| `go test` (stdlib) | — | Testing | Standard library testing is sufficient. Use `testify` for assertions only. |
| `github.com/stretchr/testify` | v1.x | Test assertions | Assert/require packages only. No mocking framework needed at this scope. |
### Build and Distribution
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `go build` with `-ldflags` | — | Single binary compilation | `CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-s -w"` produces a stripped static binary. modernc.org/sqlite makes CGO=0 possible. |
| `goreleaser` | v2.x | Multi-platform release builds (Linux amd64/arm64, macOS amd64/arm64) | Standard tool for Go binary releases. Produces checksums, archives, and optionally Homebrew taps. |
## Alternatives Considered
| Category | Recommended | Alternative | Why Not |
|----------|-------------|-------------|---------|
| Web framework | chi v5 | Fiber | Fiber uses fasthttp (not net/http) — breaks standard middleware and `go:embed` serving patterns |
| Web framework | chi v5 | Echo | Echo works with net/http but adds unnecessary abstraction for a dashboard-only use case |
| Templating | templ | html/template (stdlib) | stdlib templates have no compile-time type checking; errors surface at runtime, not build time |
| SQLite driver | modernc.org/sqlite | mattn/go-sqlite3 | mattn requires CGo — breaks cross-compilation and single-binary distribution goals |
| SQLite encryption | application-level AES-256 | SQLCipher (go-sqlcipher) | SQLCipher requires CGo, reintroducing the problem modernc.org/sqlite solves |
| Config | viper | koanf | koanf is cleaner but viper's Cobra integration is tight; viper v1.21 fixed the main key-casing issues |
| Concurrency | ants | pond | pond is simpler but less battle-tested at high concurrency (80+ simultaneous OSINT sources) |
| Scheduler | gocron v2 | robfig/cron v3 | robfig unmaintained since 2020 with known panic bugs in production |
| Telegram | telego | telebot v4 | telebot has better DX but less complete API coverage; telego's 1:1 mapping matters for a bot sending scan results |
| SARIF | custom structs | gosec/v2/report/sarif | gosec SARIF package is internal to gosec, not a published importable library |
| Terminal UI | lipgloss + bubbles | Full Bubble Tea | Full TUI event loop is overkill; components-only approach is simpler and sufficient |
## Canonical go.mod Dependencies
- Pin `cobra`, `chi`, `templ`, `telego`, `ants`, `gocron` to exact versions above (verified current).
- Use `go get -u` on `golang.org/x/time` — x/ packages track Go versions not semver.
- `modernc.org/sqlite` — pin to whatever `go get modernc.org/sqlite@latest` resolves at project init.
## Build Commands
# Development (dashboard with hot reload)
# Test
# Production binary (Linux amd64, CGO-free)
# Release (all platforms)
## Sources
- Cobra releases: https://github.com/spf13/cobra/releases (v1.10.2 confirmed)
- Chi releases: https://github.com/go-chi/chi/releases (v5.2.5 confirmed)
- Templ releases: https://github.com/a-h/templ/releases (v0.3.1001 confirmed)
- Telego releases: https://github.com/mymmrac/telego/releases (v1.8.0, 2026-04-03)
- Ants releases: https://github.com/panjf2000/ants/releases (v2.12.0 confirmed)
- gocron releases: https://github.com/go-co-op/gocron/releases (v2.19.1 confirmed)
- Viper releases: https://github.com/spf13/viper/releases (v1.21.0 confirmed)
- modernc.org/sqlite: https://pkg.go.dev/modernc.org/sqlite (SQLite 3.51.2, updated 2026-03-17)
- Chi router comparison 2025: https://blog.logrocket.com/top-go-frameworks-2025/
- Go web stack 2025 (chi + templ + htmx): https://www.ersin.nz/articles/a-great-web-stack-for-go
- Tailwind v4 standalone (no Node): https://dev.to/getjv/tailwind-css-with-air-and-go-no-node-no-problem-3j92
- SQLite driver comparison: https://github.com/cvilsmeier/go-sqlite-bench
- robfig/cron maintenance status: https://github.com/netresearch/go-cron (unmaintained since 2020 note)
- Viper vs koanf: https://itnext.io/golang-configuration-management-library-viper-vs-koanf-eea60a652a22
- TruffleHog output formats: https://deepwiki.com/trufflesecurity/trufflehog/6-output-and-results
- Gitleaks output formats: https://appsecsanta.com/sast-tools/gitleaks-vs-trufflehog
<!-- GSD:stack-end -->
<!-- GSD:conventions-start source:CONVENTIONS.md -->
## Conventions
Conventions not yet established. Will populate as patterns emerge during development.
<!-- GSD:conventions-end -->
<!-- GSD:architecture-start source:ARCHITECTURE.md -->
## Architecture
Architecture not yet mapped. Follow existing patterns found in the codebase.
<!-- GSD:architecture-end -->
<!-- GSD:workflow-start source:GSD defaults -->
## GSD Workflow Enforcement
Before using Edit, Write, or other file-changing tools, start work through a GSD command so planning artifacts and execution context stay in sync.
Use these entry points:
- `/gsd:quick` for small fixes, doc updates, and ad-hoc tasks
- `/gsd:debug` for investigation and bug fixing
- `/gsd:execute-phase` for planned phase work
Do not make direct repo edits outside a GSD workflow unless the user explicitly asks to bypass it.
<!-- GSD:workflow-end -->
<!-- GSD:profile-start -->
## Developer Profile
> Profile not yet configured. Run `/gsd:profile-user` to generate your developer profile.
> This section is managed by `generate-claude-profile` -- do not edit manually.
<!-- GSD:profile-end -->