docs(01-foundation-03): complete storage layer plan — SUMMARY, STATE, ROADMAP, REQUIREMENTS updated
- 01-03-SUMMARY.md: AES-256-GCM + Argon2id + SQLite CRUD layer complete - STATE.md: progress 20%, decisions logged, session updated - ROADMAP.md: Phase 1 In Progress (1/5 summaries) - REQUIREMENTS.md: STOR-01, STOR-02, STOR-03 marked complete
This commit is contained in:
@@ -74,9 +74,9 @@ Requirements for initial release. Each maps to roadmap phases.
|
|||||||
|
|
||||||
### Storage
|
### Storage
|
||||||
|
|
||||||
- [ ] **STOR-01**: SQLite database for persisting scan results, keys, recon history
|
- [x] **STOR-01**: SQLite database for persisting scan results, keys, recon history
|
||||||
- [ ] **STOR-02**: Application-level AES-256 encryption for stored keys and sensitive config
|
- [x] **STOR-02**: Application-level AES-256 encryption for stored keys and sensitive config
|
||||||
- [ ] **STOR-03**: Encryption key derived from user passphrase via Argon2
|
- [x] **STOR-03**: Encryption key derived from user passphrase via Argon2
|
||||||
|
|
||||||
### CLI
|
### CLI
|
||||||
|
|
||||||
|
|||||||
@@ -48,7 +48,7 @@ Decimal phases appear between their surrounding integers in numeric order.
|
|||||||
Plans:
|
Plans:
|
||||||
- [ ] 01-01-PLAN.md — Go module init, dependency installation, test scaffolding and testdata fixtures
|
- [ ] 01-01-PLAN.md — Go module init, dependency installation, test scaffolding and testdata fixtures
|
||||||
- [ ] 01-02-PLAN.md — Provider registry: YAML schema, embed loader, Aho-Corasick automaton, Registry struct
|
- [ ] 01-02-PLAN.md — Provider registry: YAML schema, embed loader, Aho-Corasick automaton, Registry struct
|
||||||
- [ ] 01-03-PLAN.md — Storage layer: AES-256-GCM encryption, Argon2id key derivation, SQLite + Finding CRUD
|
- [x] 01-03-PLAN.md — Storage layer: AES-256-GCM encryption, Argon2id key derivation, SQLite + Finding CRUD
|
||||||
- [ ] 01-04-PLAN.md — Scan engine pipeline: keyword pre-filter, regex+entropy detector, FileSource, ants worker pool
|
- [ ] 01-04-PLAN.md — Scan engine pipeline: keyword pre-filter, regex+entropy detector, FileSource, ants worker pool
|
||||||
- [ ] 01-05-PLAN.md — CLI wiring: scan, providers list/info/stats, config init/set/get, output table
|
- [ ] 01-05-PLAN.md — CLI wiring: scan, providers list/info/stats, config init/set/get, output table
|
||||||
|
|
||||||
@@ -255,7 +255,7 @@ Phases execute in numeric order: 1 → 2 → 3 → ... → 18
|
|||||||
|
|
||||||
| Phase | Plans Complete | Status | Completed |
|
| Phase | Plans Complete | Status | Completed |
|
||||||
|-------|----------------|--------|-----------|
|
|-------|----------------|--------|-----------|
|
||||||
| 1. Foundation | 0/5 | Planning complete | - |
|
| 1. Foundation | 1/5 | In Progress| |
|
||||||
| 2. Tier 1-2 Providers | 0/? | Not started | - |
|
| 2. Tier 1-2 Providers | 0/? | Not started | - |
|
||||||
| 3. Tier 3-9 Providers | 0/? | Not started | - |
|
| 3. Tier 3-9 Providers | 0/? | Not started | - |
|
||||||
| 4. Input Sources | 0/? | Not started | - |
|
| 4. Input Sources | 0/? | Not started | - |
|
||||||
|
|||||||
@@ -1,3 +1,19 @@
|
|||||||
|
---
|
||||||
|
gsd_state_version: 1.0
|
||||||
|
milestone: v1.0
|
||||||
|
milestone_name: milestone
|
||||||
|
status: planning
|
||||||
|
stopped_at: Completed 01-foundation-03-PLAN.md
|
||||||
|
last_updated: "2026-04-04T21:07:04.658Z"
|
||||||
|
last_activity: 2026-04-04 — Roadmap created, 18 phases defined covering 146 v1 requirements
|
||||||
|
progress:
|
||||||
|
total_phases: 18
|
||||||
|
completed_phases: 0
|
||||||
|
total_plans: 5
|
||||||
|
completed_plans: 1
|
||||||
|
percent: 20
|
||||||
|
---
|
||||||
|
|
||||||
# Project State
|
# Project State
|
||||||
|
|
||||||
## Project Reference
|
## Project Reference
|
||||||
@@ -14,11 +30,12 @@ Plan: 0 of ? in current phase
|
|||||||
Status: Ready to plan
|
Status: Ready to plan
|
||||||
Last activity: 2026-04-04 — Roadmap created, 18 phases defined covering 146 v1 requirements
|
Last activity: 2026-04-04 — Roadmap created, 18 phases defined covering 146 v1 requirements
|
||||||
|
|
||||||
Progress: [░░░░░░░░░░░░░░░░░░░░] 0%
|
Progress: [██░░░░░░░░] 20%
|
||||||
|
|
||||||
## Performance Metrics
|
## Performance Metrics
|
||||||
|
|
||||||
**Velocity:**
|
**Velocity:**
|
||||||
|
|
||||||
- Total plans completed: 0
|
- Total plans completed: 0
|
||||||
- Average duration: —
|
- Average duration: —
|
||||||
- Total execution time: 0 hours
|
- Total execution time: 0 hours
|
||||||
@@ -30,10 +47,12 @@ Progress: [░░░░░░░░░░░░░░░░░░░░] 0%
|
|||||||
| - | - | - | - |
|
| - | - | - | - |
|
||||||
|
|
||||||
**Recent Trend:**
|
**Recent Trend:**
|
||||||
|
|
||||||
- Last 5 plans: —
|
- Last 5 plans: —
|
||||||
- Trend: —
|
- Trend: —
|
||||||
|
|
||||||
*Updated after each plan completion*
|
*Updated after each plan completion*
|
||||||
|
| Phase 01-foundation P03 | 3 | 2 tasks | 7 files |
|
||||||
|
|
||||||
## Accumulated Context
|
## Accumulated Context
|
||||||
|
|
||||||
@@ -46,6 +65,8 @@ Recent decisions affecting current work:
|
|||||||
- Roadmap: Per-source rate limiter architecture (Phase 9) must precede all OSINT source modules (Phases 10-16)
|
- Roadmap: Per-source rate limiter architecture (Phase 9) must precede all OSINT source modules (Phases 10-16)
|
||||||
- Roadmap: AES-256 encryption added in Phase 1, not post-hoc — avoids migration complexity
|
- Roadmap: AES-256 encryption added in Phase 1, not post-hoc — avoids migration complexity
|
||||||
- Roadmap: Verification (Phase 5) requires consent prompt + LEGAL.md — not optional polish
|
- Roadmap: Verification (Phase 5) requires consent prompt + LEGAL.md — not optional polish
|
||||||
|
- [Phase 01-foundation]: Storage 01-03: Argon2id selected over PBKDF2 — memory-hard RFC 9106 params, resolves STATE.md blocker
|
||||||
|
- [Phase 01-foundation]: Storage 01-03: AES-256-GCM nonce prepended to ciphertext in single BLOB column — no separate nonce column needed
|
||||||
|
|
||||||
### Pending Todos
|
### Pending Todos
|
||||||
|
|
||||||
@@ -60,6 +81,6 @@ None yet.
|
|||||||
|
|
||||||
## Session Continuity
|
## Session Continuity
|
||||||
|
|
||||||
Last session: 2026-04-04
|
Last session: 2026-04-04T21:07:04.654Z
|
||||||
Stopped at: Roadmap written to .planning/ROADMAP.md; ready to begin Phase 1 planning
|
Stopped at: Completed 01-foundation-03-PLAN.md
|
||||||
Resume file: None
|
Resume file: None
|
||||||
|
|||||||
139
.planning/phases/01-foundation/01-03-SUMMARY.md
Normal file
139
.planning/phases/01-foundation/01-03-SUMMARY.md
Normal file
@@ -0,0 +1,139 @@
|
|||||||
|
---
|
||||||
|
phase: 01-foundation
|
||||||
|
plan: 03
|
||||||
|
subsystem: database
|
||||||
|
tags: [sqlite, aes-256-gcm, argon2id, encryption, storage, modernc-sqlite]
|
||||||
|
|
||||||
|
# Dependency graph
|
||||||
|
requires:
|
||||||
|
- phase: 01-foundation-01
|
||||||
|
provides: go.mod with modernc.org/sqlite and golang.org/x/crypto dependencies
|
||||||
|
|
||||||
|
provides:
|
||||||
|
- AES-256-GCM column encryption (Encrypt/Decrypt) with random nonce prepended
|
||||||
|
- Argon2id key derivation (DeriveKey/NewSalt) using RFC 9106 parameters
|
||||||
|
- SQLite database Open() with WAL mode and embedded schema migration
|
||||||
|
- Finding CRUD: SaveFinding encrypts key_value at boundary, ListFindings decrypts transparently
|
||||||
|
- MaskKey helper: first8...last4 display format
|
||||||
|
- schema.sql with findings, scans, settings tables and performance indexes
|
||||||
|
|
||||||
|
affects: [01-04-scanner, 01-05-cli, 17-dashboard, 18-telegram]
|
||||||
|
|
||||||
|
# Tech tracking
|
||||||
|
tech-stack:
|
||||||
|
added:
|
||||||
|
- modernc.org/sqlite v1.48.1 (pure Go SQLite, CGO-free)
|
||||||
|
- golang.org/x/crypto (argon2.IDKey for key derivation)
|
||||||
|
- crypto/aes + crypto/cipher (stdlib AES-256-GCM)
|
||||||
|
patterns:
|
||||||
|
- "Encrypt-at-boundary: SaveFinding encrypts, ListFindings decrypts — storage layer handles all crypto transparently"
|
||||||
|
- "go:embed schema.sql — schema migrated on Open(), idempotent via CREATE TABLE IF NOT EXISTS"
|
||||||
|
- "WAL mode enabled on every Open() for concurrent read performance"
|
||||||
|
- "NULL scan_id: zero-value ScanID stored as SQL NULL to satisfy FK constraint"
|
||||||
|
|
||||||
|
key-files:
|
||||||
|
created:
|
||||||
|
- pkg/storage/encrypt.go
|
||||||
|
- pkg/storage/crypto.go
|
||||||
|
- pkg/storage/db.go
|
||||||
|
- pkg/storage/findings.go
|
||||||
|
- pkg/storage/schema.sql
|
||||||
|
- pkg/storage/db_test.go
|
||||||
|
modified: []
|
||||||
|
|
||||||
|
key-decisions:
|
||||||
|
- "Argon2id over PBKDF2: RFC 9106 recommended, memory-hard, resolves blocker from STATE.md"
|
||||||
|
- "NULL scan_id for findings without parent scan — FK constraint satisfied without mandatory scan creation"
|
||||||
|
- "Nonce prepended to ciphertext in single []byte — simplifies storage (no separate column needed)"
|
||||||
|
- "MaskKey returns first8...last4 — consistent with plan spec, 12-char minimum before masking"
|
||||||
|
|
||||||
|
patterns-established:
|
||||||
|
- "Pattern: Encrypt-at-boundary — pkg/storage is the only layer that sees encrypted bytes"
|
||||||
|
- "Pattern: sql.NullInt64 for nullable FK columns in scan results"
|
||||||
|
- "Pattern: go:embed for all embedded assets — schema.sql embedded in db.go"
|
||||||
|
|
||||||
|
requirements-completed: [STOR-01, STOR-02, STOR-03]
|
||||||
|
|
||||||
|
# Metrics
|
||||||
|
duration: 3min
|
||||||
|
completed: 2026-04-04
|
||||||
|
---
|
||||||
|
|
||||||
|
# Phase 1 Plan 3: Storage Layer Summary
|
||||||
|
|
||||||
|
**AES-256-GCM column encryption with Argon2id key derivation and SQLite CRUD — raw BLOB verified to contain no plaintext key data**
|
||||||
|
|
||||||
|
## Performance
|
||||||
|
|
||||||
|
- **Duration:** 3 min
|
||||||
|
- **Started:** 2026-04-04T21:02:00Z
|
||||||
|
- **Completed:** 2026-04-04T21:06:06Z
|
||||||
|
- **Tasks:** 2
|
||||||
|
- **Files modified:** 7 (5 created, go.mod + go.sum updated)
|
||||||
|
|
||||||
|
## Accomplishments
|
||||||
|
|
||||||
|
- AES-256-GCM Encrypt/Decrypt with prepended random nonce — non-deterministic, wrong-key fails with GCM auth error
|
||||||
|
- Argon2id DeriveKey using RFC 9106 Section 7.3 params (time=1, memory=64MB, threads=4, keyLen=32) — resolves the Argon2 vs PBKDF2 blocker from STATE.md
|
||||||
|
- SQLite opens with WAL mode, foreign keys, and embedded schema.sql migration — works with `:memory:` for tests
|
||||||
|
- SaveFinding/ListFindings transparently encrypt/decrypt key_value at storage boundary
|
||||||
|
- All 7 tests pass including raw-BLOB assertion confirming plaintext is not stored
|
||||||
|
|
||||||
|
## Task Commits
|
||||||
|
|
||||||
|
Each task was committed atomically:
|
||||||
|
|
||||||
|
1. **TDD RED: Failing test suite** - `2ef54f7` (test)
|
||||||
|
2. **Task 1: AES-256-GCM + Argon2id** - `239e2c2` (feat)
|
||||||
|
3. **Task 2: SQLite DB + schema + CRUD** - `3334633` (feat)
|
||||||
|
|
||||||
|
## Files Created/Modified
|
||||||
|
|
||||||
|
- `pkg/storage/encrypt.go` - Encrypt(plaintext, key) and Decrypt(ciphertext, key) using AES-256-GCM
|
||||||
|
- `pkg/storage/crypto.go` - DeriveKey(passphrase, salt) using Argon2id RFC 9106, NewSalt() 16-byte random
|
||||||
|
- `pkg/storage/db.go` - DB struct with Open(), Close(), SQL() — WAL mode, FK, embedded schema migration
|
||||||
|
- `pkg/storage/findings.go` - Finding struct, SaveFinding, ListFindings, MaskKey helper
|
||||||
|
- `pkg/storage/schema.sql` - CREATE TABLE for findings, scans, settings + 3 indexes
|
||||||
|
- `pkg/storage/db_test.go` - 7 tests including raw-BLOB encryption verification
|
||||||
|
|
||||||
|
## Decisions Made
|
||||||
|
|
||||||
|
- Argon2id selected over PBKDF2 (resolves STATE.md blocker) — memory-hard, RFC 9106 recommended
|
||||||
|
- NULL scan_id: zero-value ScanID stored as SQL NULL so findings can exist without a parent scan
|
||||||
|
- Single []byte for nonce+ciphertext — no separate nonce column needed, simplifies schema
|
||||||
|
|
||||||
|
## Deviations from Plan
|
||||||
|
|
||||||
|
### Auto-fixed Issues
|
||||||
|
|
||||||
|
**1. [Rule 1 - Bug] FK constraint failed when ScanID = 0**
|
||||||
|
- **Found during:** Task 2 (TestSaveFindingEncrypted)
|
||||||
|
- **Issue:** Go zero value `ScanID: 0` sent as integer 0 to SQLite, failing FK constraint (no scan with id=0)
|
||||||
|
- **Fix:** SaveFinding converts zero ScanID to sql.NullInt64{} (NULL), ListFindings uses sql.NullInt64 for scan
|
||||||
|
- **Files modified:** pkg/storage/findings.go
|
||||||
|
- **Verification:** TestSaveFindingEncrypted passes after fix
|
||||||
|
- **Committed in:** 3334633 (Task 2 commit)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Total deviations:** 1 auto-fixed (Rule 1 - Bug)
|
||||||
|
**Impact on plan:** Necessary correctness fix — plan spec allows NULL scan_id (no NOT NULL in schema). No scope creep.
|
||||||
|
|
||||||
|
## Issues Encountered
|
||||||
|
|
||||||
|
None beyond the FK constraint bug documented above.
|
||||||
|
|
||||||
|
## User Setup Required
|
||||||
|
|
||||||
|
None - no external service configuration required.
|
||||||
|
|
||||||
|
## Next Phase Readiness
|
||||||
|
|
||||||
|
- Storage layer fully functional with transparent encryption
|
||||||
|
- pkg/storage exports: Encrypt, Decrypt, DeriveKey, NewSalt, Open, DB, Finding, SaveFinding, ListFindings, MaskKey
|
||||||
|
- Scanner (Plan 04) can call SaveFinding to persist findings
|
||||||
|
- CLI (Plan 05) can call ListFindings to display findings
|
||||||
|
|
||||||
|
---
|
||||||
|
*Phase: 01-foundation*
|
||||||
|
*Completed: 2026-04-04*
|
||||||
Reference in New Issue
Block a user