salvacybersec/keyhunter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: complete project research

Browse Source
This commit is contained in:
salvacybersec
2026-04-04 19:03:12 +03:00
parent c8e744cb48
commit 6c3a84b1ff
5 changed files with 1645 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

259
.planning/research/STACK.md Normal file
View File

@@ -0,0 +1,259 @@
# Technology Stack
**Project:** KeyHunter — Go-based API Key Scanner
**Researched:** 2026-04-04
**Research Mode:** Ecosystem
---
## Recommended Stack
### Core CLI Framework
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `github.com/spf13/cobra` | v1.10.2 | CLI command tree (scan, verify, recon, keys, serve, dorks, providers, config, hook, schedule) | Industry standard for Go CLIs. Used by Kubernetes, Docker, GitHub CLI. Sub-command hierarchy, persistent flags, shell completion, man-page generation are all built in. No viable alternative — it IS the Go CLI standard. |
| `github.com/spf13/viper` | v1.21.0 | Configuration management (YAML/JSON/env/flags binding) | Designed to pair with Cobra. Handles config file + env var + CLI flag precedence chain automatically. v1.21.0 switched to maintained yaml lib, cleaning supply-chain issues. |
**Confidence: HIGH** — Verified via GitHub releases. Used in TruffleHog and Gitleaks themselves.
---
### Web Dashboard
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `github.com/go-chi/chi/v5` | v5.2.5 | HTTP router for dashboard and API | 100% net/http compatible — no custom context or handler types. Zero external dependencies. Routes embed naturally into `go:embed` serving. Used by major Go projects. Requires Go 1.22+ (matches project constraint). |
| `github.com/a-h/templ` | v0.3.1001 | Type-safe HTML template compilation | `.templ` files compile to Go — template errors are caught at compile time, not runtime. Composes naturally with HTMX. Significantly safer than `html/template` for a project with a public-facing dashboard. |
| htmx | v2.x (CDN or vendored) | Frontend interactivity without JS framework | Server-rendered with AJAX behavior. No build step. Aligns with "embed in binary" architecture constraint. Use `go:embed` to bundle the htmx.min.js into the binary. |
| Tailwind CSS | v4.x (standalone CLI) | Utility-first styling | v4 ships a standalone binary — no Node.js required. Use `@tailwindcss/cli` to compile a single CSS file, then `go:embed` it. Air watches both `.templ` and CSS changes during development. |
**Confidence: HIGH** — Verified via GitHub releases and multiple 2025 tutorial ecosystem sources.
**Do NOT use:** Fiber or Echo for this project. Both introduce custom handler types that break `net/http` compatibility and complicate `go:embed` static file serving. Chi's zero-dependency, stdlib-native approach is correct for a single-binary security tool.
---
### Database
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `modernc.org/sqlite` | v1.35.x (SQLite 3.51.2 inside) | Embedded database for scan results, keys, recon data | Pure Go — no CGo, no C compiler requirement. Cross-compiles cleanly for Linux/macOS/ARM. Actively maintained (updated 2026-03-17). Zero external process dependency for single-binary distribution. |
| `database/sql` (stdlib) | — | SQL interface layer | Use standard library interface over `modernc.org/sqlite` directly — driver is registered as `"sqlite"`. No ORM needed for a tool of this scope. Raw SQL gives full control and avoids ORM magic bugs. |
**Encryption approach:** Application-level AES-256 using Go's `crypto/aes` + `crypto/cipher` stdlib. Encrypt key material fields before writing to SQLite, decrypt on read. This avoids the CGo dependency of SQLCipher while achieving the same security goal for a local tool. The database file itself stays portable.
**Confidence: HIGH (modernc.org/sqlite choice) / MEDIUM (application-level encryption approach).**
**Do NOT use:** `mattn/go-sqlite3` — requires CGo, breaks cross-compilation, adds C toolchain dependency. `SQLCipher` wrappers — all require CGo, reintroducing the problem modernc solves.
---
### Concurrency
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `github.com/panjf2000/ants/v2` | v2.12.0 | Worker pool for parallel scanning across files, sources, and verification requests | Mature, battle-tested goroutine pool. Dynamically resizable. Handles thousands of concurrent tasks without goroutine explosion. Used in high-throughput Go systems. v2.12.0 adds ReleaseContext for clean shutdown. |
| `golang.org/x/time/rate` | latest x/ | Per-source rate limiting for OSINT/recon sources | Official Go extended library. Token bucket algorithm. Rate-limit each external source (Shodan, GitHub, etc.) independently. Used by TruffleHog for the same purpose. |
| `sync`, `context` (stdlib) | — | Cancellation, mutex, waitgroups | Standard library is sufficient for coordination between pool and caller. No additional abstraction needed. |
**Confidence: HIGH (ants) / HIGH (x/time/rate).**
**Do NOT use:** `pond` or `conc` — viable alternatives but ants has more production mileage at the scale this tool will operate (thousands of files, 80+ concurrent OSINT sources). `conc` is excellent for structured concurrency but adds abstraction that ants doesn't need.
---
### YAML Provider/Dork Engine
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `gopkg.in/yaml.v3` | v3.x | Parse provider YAML definitions and dork YAML files embedded via `go:embed` | Direct, well-understood API. v3 handles inline/anchored structs correctly. The Cobra v1.10.2 release migrated away from it to `go.yaml.in/yaml/v3` — however for provider YAML parsing, gopkg.in/yaml.v3 remains stable and appropriate. |
| `embed` (stdlib) | — | Compile-time embedding of `/providers/*.yaml` and `/dorks/*.yaml` | Go 1.16+ native. No external dependency. Providers and dorks are baked into the binary at compile time — no runtime filesystem access needed. |
**Confidence: HIGH.**
**Do NOT use:** `sigs.k8s.io/yaml` — converts YAML to JSON internally, adding unnecessary round-trip for this use case. `goccy/go-yaml` — performance advantage is irrelevant for config parsing; adds dependency.
---
### Telegram Bot
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `github.com/mymmrac/telego` | v1.8.0 | Telegram bot for /scan, /verify, /recon, /status, /stats, /subscribe, /key commands | One-to-one Telegram Bot API v9.6 mapping. Supports long polling and webhooks. Type-safe API surface. v1.8.0 released 2026-04-03 with API v9.6 support. Actively maintained. |
**Confidence: HIGH** — Latest release confirmed day before research date.
**Alternative considered:** `gopkg.in/telebot.v4` — higher-level abstraction, good DX, but less full API coverage. For a security tool where controlling exact API behavior matters, telego's 1:1 mapping is preferable.
---
### Scheduled Scanning
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `github.com/go-co-op/gocron/v2` | v2.19.1 | Cron-based recurring scans | Modern API, v2 has full context support and job lifecycle management. Race condition fix in v2.19.1 (important for scheduler reliability). Better API than robfig/cron v3. |
**Confidence: MEDIUM** — robfig/cron v3 also viable; gocron v2 has cleaner modern API. The netresearch/go-cron fork note (robfig unmaintained since 2020) makes gocron the better default.
**Do NOT use:** `robfig/cron` directly — unmaintained since 2020, 50+ open PRs, known panic bugs in production.
---
### Output and Formatting
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `github.com/charmbracelet/lipgloss` | latest | Colored terminal table output, status indicators | Declarative style definitions. Composes with any output stream. Used across the Go security tool ecosystem (TruffleHog uses it indirectly). |
| `github.com/charmbracelet/bubbles` | latest | Progress bars for long scans, spinners during verification | Pre-built terminal UI components. Pairs with lipgloss. Less overhead than full Bubble Tea TUI — use components only. |
| `encoding/json` (stdlib) | — | JSON output format | Standard library is sufficient. No external JSON library needed. |
| SARIF output | custom | CI/CD integration output | Implement SARIF 2.1.0 format directly — it is a straightforward JSON schema. Do not use a library; gosec's SARIF package is not designed for import. ~200 lines of struct definitions covers the needed schema. |
**Confidence: HIGH (lipgloss/bubbles) / HIGH (stdlib JSON) / MEDIUM (custom SARIF).**
**Do NOT use:** Full Bubble Tea TUI — KeyHunter is a scanning tool, not an interactive terminal application. The TUI overhead (event loop, model/update/view) is inappropriate. Use lipgloss + bubbles components directly via tea.Program only for progress output.
---
### HTTP Client (OSINT/Recon/Verification)
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `net/http` (stdlib) | — | All outbound HTTP requests for verification and OSINT | Standard library client is sufficient. Supports custom TLS, proxy settings, timeouts. Avoid adding httpclient wrappers that hide behavior. |
| `golang.org/x/time/rate` | latest | Per-source rate limiting on outbound requests | Already listed under concurrency — same package serves both purposes. |
**Confidence: HIGH.**
**Do NOT use:** `resty`, `go-retryablehttp`, or other HTTP client wrappers — they add abstractions that make debugging OSINT source failures harder. Implement retry/backoff directly using stdlib + time/rate.
---
### Development Tooling
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `github.com/air-verse/air` | latest | Hot reload during dashboard development | Watches Go + templ files, rebuilds on change. Industry standard for Go web dev loops. |
| `@tailwindcss/cli` | v4.x standalone | CSS compilation without Node.js | v4 standalone binary eliminates Node dependency entirely. Run as `tailwindcss -i input.css -o dist/style.css --watch` alongside air. |
| `golangci-lint` | latest | Static analysis and linting | Multi-linter runner. Include gosec, staticcheck, errcheck at minimum. |
| `go test` (stdlib) | — | Testing | Standard library testing is sufficient. Use `testify` for assertions only. |
| `github.com/stretchr/testify` | v1.x | Test assertions | Assert/require packages only. No mocking framework needed at this scope. |
**Confidence: HIGH.**
---
### Build and Distribution
| Technology | Version | Purpose | Why |
|------------|---------|---------|-----|
| `go build` with `-ldflags` | — | Single binary compilation | `CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-s -w"` produces a stripped static binary. modernc.org/sqlite makes CGO=0 possible. |
| `goreleaser` | v2.x | Multi-platform release builds (Linux amd64/arm64, macOS amd64/arm64) | Standard tool for Go binary releases. Produces checksums, archives, and optionally Homebrew taps. |
**Confidence: HIGH.**
---
## Alternatives Considered
| Category | Recommended | Alternative | Why Not |
|----------|-------------|-------------|---------|
| Web framework | chi v5 | Fiber | Fiber uses fasthttp (not net/http) — breaks standard middleware and `go:embed` serving patterns |
| Web framework | chi v5 | Echo | Echo works with net/http but adds unnecessary abstraction for a dashboard-only use case |
| Templating | templ | html/template (stdlib) | stdlib templates have no compile-time type checking; errors surface at runtime, not build time |
| SQLite driver | modernc.org/sqlite | mattn/go-sqlite3 | mattn requires CGo — breaks cross-compilation and single-binary distribution goals |
| SQLite encryption | application-level AES-256 | SQLCipher (go-sqlcipher) | SQLCipher requires CGo, reintroducing the problem modernc.org/sqlite solves |
| Config | viper | koanf | koanf is cleaner but viper's Cobra integration is tight; viper v1.21 fixed the main key-casing issues |
| Concurrency | ants | pond | pond is simpler but less battle-tested at high concurrency (80+ simultaneous OSINT sources) |
| Scheduler | gocron v2 | robfig/cron v3 | robfig unmaintained since 2020 with known panic bugs in production |
| Telegram | telego | telebot v4 | telebot has better DX but less complete API coverage; telego's 1:1 mapping matters for a bot sending scan results |
| SARIF | custom structs | gosec/v2/report/sarif | gosec SARIF package is internal to gosec, not a published importable library |
| Terminal UI | lipgloss + bubbles | Full Bubble Tea | Full TUI event loop is overkill; components-only approach is simpler and sufficient |
---
## Canonical go.mod Dependencies
```go
module github.com/yourusername/keyhunter
go 1.22
require (
// CLI
github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0
// Web dashboard
github.com/go-chi/chi/v5 v5.2.5
github.com/a-h/templ v0.3.1001
// Database
modernc.org/sqlite v1.35.x
// YAML provider/dork engine
gopkg.in/yaml.v3 v3.0.1
// Concurrency + rate limiting
github.com/panjf2000/ants/v2 v2.12.0
golang.org/x/time latest
// Telegram bot
github.com/mymmrac/telego v1.8.0
// Scheduler
github.com/go-co-op/gocron/v2 v2.19.1
// Terminal output
github.com/charmbracelet/lipgloss latest
github.com/charmbracelet/bubbles latest
// Testing
github.com/stretchr/testify v1.x
)
```
**Notes on pinning:**
- Pin `cobra`, `chi`, `templ`, `telego`, `ants`, `gocron` to exact versions above (verified current).
- Use `go get -u` on `golang.org/x/time` — x/ packages track Go versions not semver.
- `modernc.org/sqlite` — pin to whatever `go get modernc.org/sqlite@latest` resolves at project init.
---
## Build Commands
```bash
# Development (dashboard with hot reload)
air & # hot-reload Go + templ
tailwindcss -i web/input.css -o web/dist/style.css --watch & # CSS watch
# Test
go test ./... -race -cover
# Production binary (Linux amd64, CGO-free)
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
go build -ldflags="-s -w" -o keyhunter ./cmd/keyhunter
# Release (all platforms)
goreleaser release --clean
```
---
## Sources
- Cobra releases: https://github.com/spf13/cobra/releases (v1.10.2 confirmed)
- Chi releases: https://github.com/go-chi/chi/releases (v5.2.5 confirmed)
- Templ releases: https://github.com/a-h/templ/releases (v0.3.1001 confirmed)
- Telego releases: https://github.com/mymmrac/telego/releases (v1.8.0, 2026-04-03)
- Ants releases: https://github.com/panjf2000/ants/releases (v2.12.0 confirmed)
- gocron releases: https://github.com/go-co-op/gocron/releases (v2.19.1 confirmed)
- Viper releases: https://github.com/spf13/viper/releases (v1.21.0 confirmed)
- modernc.org/sqlite: https://pkg.go.dev/modernc.org/sqlite (SQLite 3.51.2, updated 2026-03-17)
- Chi router comparison 2025: https://blog.logrocket.com/top-go-frameworks-2025/
- Go web stack 2025 (chi + templ + htmx): https://www.ersin.nz/articles/a-great-web-stack-for-go
- Tailwind v4 standalone (no Node): https://dev.to/getjv/tailwind-css-with-air-and-go-no-node-no-problem-3j92
- SQLite driver comparison: https://github.com/cvilsmeier/go-sqlite-bench
- robfig/cron maintenance status: https://github.com/netresearch/go-cron (unmaintained since 2020 note)
- Viper vs koanf: https://itnext.io/golang-configuration-management-library-viper-vs-koanf-eea60a652a22
- TruffleHog output formats: https://deepwiki.com/trufflesecurity/trufflehog/6-output-and-results
- Gitleaks output formats: https://appsecsanta.com/sast-tools/gitleaks-vs-trufflehog