From 43aeb8985d722d20027e57c3305f57eee16a7691 Mon Sep 17 00:00:00 2001 From: salvacybersec Date: Sun, 5 Apr 2026 00:07:24 +0300 Subject: [PATCH] =?UTF-8?q?docs(01-foundation-03):=20complete=20storage=20?= =?UTF-8?q?layer=20plan=20=E2=80=94=20SUMMARY,=20STATE,=20ROADMAP,=20REQUI?= =?UTF-8?q?REMENTS=20updated?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - 01-03-SUMMARY.md: AES-256-GCM + Argon2id + SQLite CRUD layer complete - STATE.md: progress 20%, decisions logged, session updated - ROADMAP.md: Phase 1 In Progress (1/5 summaries) - REQUIREMENTS.md: STOR-01, STOR-02, STOR-03 marked complete --- .planning/REQUIREMENTS.md | 6 +- .planning/ROADMAP.md | 4 +- .planning/STATE.md | 27 +++- .../phases/01-foundation/01-03-SUMMARY.md | 139 ++++++++++++++++++ 4 files changed, 168 insertions(+), 8 deletions(-) create mode 100644 .planning/phases/01-foundation/01-03-SUMMARY.md diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 32296ba..262ad00 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -74,9 +74,9 @@ Requirements for initial release. Each maps to roadmap phases. ### Storage -- [ ] **STOR-01**: SQLite database for persisting scan results, keys, recon history -- [ ] **STOR-02**: Application-level AES-256 encryption for stored keys and sensitive config -- [ ] **STOR-03**: Encryption key derived from user passphrase via Argon2 +- [x] **STOR-01**: SQLite database for persisting scan results, keys, recon history +- [x] **STOR-02**: Application-level AES-256 encryption for stored keys and sensitive config +- [x] **STOR-03**: Encryption key derived from user passphrase via Argon2 ### CLI diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 53c9549..ccd920a 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -48,7 +48,7 @@ Decimal phases appear between their surrounding integers in numeric order. Plans: - [ ] 01-01-PLAN.md — Go module init, dependency installation, test scaffolding and testdata fixtures - [ ] 01-02-PLAN.md — Provider registry: YAML schema, embed loader, Aho-Corasick automaton, Registry struct -- [ ] 01-03-PLAN.md — Storage layer: AES-256-GCM encryption, Argon2id key derivation, SQLite + Finding CRUD +- [x] 01-03-PLAN.md — Storage layer: AES-256-GCM encryption, Argon2id key derivation, SQLite + Finding CRUD - [ ] 01-04-PLAN.md — Scan engine pipeline: keyword pre-filter, regex+entropy detector, FileSource, ants worker pool - [ ] 01-05-PLAN.md — CLI wiring: scan, providers list/info/stats, config init/set/get, output table @@ -255,7 +255,7 @@ Phases execute in numeric order: 1 → 2 → 3 → ... → 18 | Phase | Plans Complete | Status | Completed | |-------|----------------|--------|-----------| -| 1. Foundation | 0/5 | Planning complete | - | +| 1. Foundation | 1/5 | In Progress| | | 2. Tier 1-2 Providers | 0/? | Not started | - | | 3. Tier 3-9 Providers | 0/? | Not started | - | | 4. Input Sources | 0/? | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index cde2084..2f9bdb8 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -1,3 +1,19 @@ +--- +gsd_state_version: 1.0 +milestone: v1.0 +milestone_name: milestone +status: planning +stopped_at: Completed 01-foundation-03-PLAN.md +last_updated: "2026-04-04T21:07:04.658Z" +last_activity: 2026-04-04 — Roadmap created, 18 phases defined covering 146 v1 requirements +progress: + total_phases: 18 + completed_phases: 0 + total_plans: 5 + completed_plans: 1 + percent: 20 +--- + # Project State ## Project Reference @@ -14,11 +30,12 @@ Plan: 0 of ? in current phase Status: Ready to plan Last activity: 2026-04-04 — Roadmap created, 18 phases defined covering 146 v1 requirements -Progress: [░░░░░░░░░░░░░░░░░░░░] 0% +Progress: [██░░░░░░░░] 20% ## Performance Metrics **Velocity:** + - Total plans completed: 0 - Average duration: — - Total execution time: 0 hours @@ -30,10 +47,12 @@ Progress: [░░░░░░░░░░░░░░░░░░░░] 0% | - | - | - | - | **Recent Trend:** + - Last 5 plans: — - Trend: — *Updated after each plan completion* +| Phase 01-foundation P03 | 3 | 2 tasks | 7 files | ## Accumulated Context @@ -46,6 +65,8 @@ Recent decisions affecting current work: - Roadmap: Per-source rate limiter architecture (Phase 9) must precede all OSINT source modules (Phases 10-16) - Roadmap: AES-256 encryption added in Phase 1, not post-hoc — avoids migration complexity - Roadmap: Verification (Phase 5) requires consent prompt + LEGAL.md — not optional polish +- [Phase 01-foundation]: Storage 01-03: Argon2id selected over PBKDF2 — memory-hard RFC 9106 params, resolves STATE.md blocker +- [Phase 01-foundation]: Storage 01-03: AES-256-GCM nonce prepended to ciphertext in single BLOB column — no separate nonce column needed ### Pending Todos @@ -60,6 +81,6 @@ None yet. ## Session Continuity -Last session: 2026-04-04 -Stopped at: Roadmap written to .planning/ROADMAP.md; ready to begin Phase 1 planning +Last session: 2026-04-04T21:07:04.654Z +Stopped at: Completed 01-foundation-03-PLAN.md Resume file: None diff --git a/.planning/phases/01-foundation/01-03-SUMMARY.md b/.planning/phases/01-foundation/01-03-SUMMARY.md new file mode 100644 index 0000000..5418f34 --- /dev/null +++ b/.planning/phases/01-foundation/01-03-SUMMARY.md @@ -0,0 +1,139 @@ +--- +phase: 01-foundation +plan: 03 +subsystem: database +tags: [sqlite, aes-256-gcm, argon2id, encryption, storage, modernc-sqlite] + +# Dependency graph +requires: + - phase: 01-foundation-01 + provides: go.mod with modernc.org/sqlite and golang.org/x/crypto dependencies + +provides: + - AES-256-GCM column encryption (Encrypt/Decrypt) with random nonce prepended + - Argon2id key derivation (DeriveKey/NewSalt) using RFC 9106 parameters + - SQLite database Open() with WAL mode and embedded schema migration + - Finding CRUD: SaveFinding encrypts key_value at boundary, ListFindings decrypts transparently + - MaskKey helper: first8...last4 display format + - schema.sql with findings, scans, settings tables and performance indexes + +affects: [01-04-scanner, 01-05-cli, 17-dashboard, 18-telegram] + +# Tech tracking +tech-stack: + added: + - modernc.org/sqlite v1.48.1 (pure Go SQLite, CGO-free) + - golang.org/x/crypto (argon2.IDKey for key derivation) + - crypto/aes + crypto/cipher (stdlib AES-256-GCM) + patterns: + - "Encrypt-at-boundary: SaveFinding encrypts, ListFindings decrypts — storage layer handles all crypto transparently" + - "go:embed schema.sql — schema migrated on Open(), idempotent via CREATE TABLE IF NOT EXISTS" + - "WAL mode enabled on every Open() for concurrent read performance" + - "NULL scan_id: zero-value ScanID stored as SQL NULL to satisfy FK constraint" + +key-files: + created: + - pkg/storage/encrypt.go + - pkg/storage/crypto.go + - pkg/storage/db.go + - pkg/storage/findings.go + - pkg/storage/schema.sql + - pkg/storage/db_test.go + modified: [] + +key-decisions: + - "Argon2id over PBKDF2: RFC 9106 recommended, memory-hard, resolves blocker from STATE.md" + - "NULL scan_id for findings without parent scan — FK constraint satisfied without mandatory scan creation" + - "Nonce prepended to ciphertext in single []byte — simplifies storage (no separate column needed)" + - "MaskKey returns first8...last4 — consistent with plan spec, 12-char minimum before masking" + +patterns-established: + - "Pattern: Encrypt-at-boundary — pkg/storage is the only layer that sees encrypted bytes" + - "Pattern: sql.NullInt64 for nullable FK columns in scan results" + - "Pattern: go:embed for all embedded assets — schema.sql embedded in db.go" + +requirements-completed: [STOR-01, STOR-02, STOR-03] + +# Metrics +duration: 3min +completed: 2026-04-04 +--- + +# Phase 1 Plan 3: Storage Layer Summary + +**AES-256-GCM column encryption with Argon2id key derivation and SQLite CRUD — raw BLOB verified to contain no plaintext key data** + +## Performance + +- **Duration:** 3 min +- **Started:** 2026-04-04T21:02:00Z +- **Completed:** 2026-04-04T21:06:06Z +- **Tasks:** 2 +- **Files modified:** 7 (5 created, go.mod + go.sum updated) + +## Accomplishments + +- AES-256-GCM Encrypt/Decrypt with prepended random nonce — non-deterministic, wrong-key fails with GCM auth error +- Argon2id DeriveKey using RFC 9106 Section 7.3 params (time=1, memory=64MB, threads=4, keyLen=32) — resolves the Argon2 vs PBKDF2 blocker from STATE.md +- SQLite opens with WAL mode, foreign keys, and embedded schema.sql migration — works with `:memory:` for tests +- SaveFinding/ListFindings transparently encrypt/decrypt key_value at storage boundary +- All 7 tests pass including raw-BLOB assertion confirming plaintext is not stored + +## Task Commits + +Each task was committed atomically: + +1. **TDD RED: Failing test suite** - `2ef54f7` (test) +2. **Task 1: AES-256-GCM + Argon2id** - `239e2c2` (feat) +3. **Task 2: SQLite DB + schema + CRUD** - `3334633` (feat) + +## Files Created/Modified + +- `pkg/storage/encrypt.go` - Encrypt(plaintext, key) and Decrypt(ciphertext, key) using AES-256-GCM +- `pkg/storage/crypto.go` - DeriveKey(passphrase, salt) using Argon2id RFC 9106, NewSalt() 16-byte random +- `pkg/storage/db.go` - DB struct with Open(), Close(), SQL() — WAL mode, FK, embedded schema migration +- `pkg/storage/findings.go` - Finding struct, SaveFinding, ListFindings, MaskKey helper +- `pkg/storage/schema.sql` - CREATE TABLE for findings, scans, settings + 3 indexes +- `pkg/storage/db_test.go` - 7 tests including raw-BLOB encryption verification + +## Decisions Made + +- Argon2id selected over PBKDF2 (resolves STATE.md blocker) — memory-hard, RFC 9106 recommended +- NULL scan_id: zero-value ScanID stored as SQL NULL so findings can exist without a parent scan +- Single []byte for nonce+ciphertext — no separate nonce column needed, simplifies schema + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Bug] FK constraint failed when ScanID = 0** +- **Found during:** Task 2 (TestSaveFindingEncrypted) +- **Issue:** Go zero value `ScanID: 0` sent as integer 0 to SQLite, failing FK constraint (no scan with id=0) +- **Fix:** SaveFinding converts zero ScanID to sql.NullInt64{} (NULL), ListFindings uses sql.NullInt64 for scan +- **Files modified:** pkg/storage/findings.go +- **Verification:** TestSaveFindingEncrypted passes after fix +- **Committed in:** 3334633 (Task 2 commit) + +--- + +**Total deviations:** 1 auto-fixed (Rule 1 - Bug) +**Impact on plan:** Necessary correctness fix — plan spec allows NULL scan_id (no NOT NULL in schema). No scope creep. + +## Issues Encountered + +None beyond the FK constraint bug documented above. + +## User Setup Required + +None - no external service configuration required. + +## Next Phase Readiness + +- Storage layer fully functional with transparent encryption +- pkg/storage exports: Encrypt, Decrypt, DeriveKey, NewSalt, Open, DB, Finding, SaveFinding, ListFindings, MaskKey +- Scanner (Plan 04) can call SaveFinding to persist findings +- CLI (Plan 05) can call ListFindings to display findings + +--- +*Phase: 01-foundation* +*Completed: 2026-04-04*