diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 0abc997..246b54b 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -51,9 +51,9 @@ Requirements for initial release. Each maps to roadmap phases. ### Output & Reporting - [x] **OUT-01**: Colored terminal table output (default) -- [ ] **OUT-02**: JSON output format +- [x] **OUT-02**: JSON output format - [x] **OUT-03**: SARIF output format (CI/CD compatible) -- [ ] **OUT-04**: CSV output format +- [x] **OUT-04**: CSV output format - [ ] **OUT-05**: Key masking by default (first 8 + last 4 chars) with --unmask flag for full keys - [x] **OUT-06**: Exit codes: 0=clean, 1=keys found, 2=error diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 290de80..5288443 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -143,7 +143,7 @@ Plans: Plans: - [x] 06-01-PLAN.md — Wave 0: Formatter interface, colors.go (TTY/NO_COLOR), refactor TableFormatter -- [ ] 06-02-PLAN.md — JSONFormatter + CSVFormatter (full Finding fields, Unmask option) +- [x] 06-02-PLAN.md — JSONFormatter + CSVFormatter (full Finding fields, Unmask option) - [x] 06-03-PLAN.md — SARIF 2.1.0 formatter with custom structs (rule dedup, level mapping) - [x] 06-04-PLAN.md — pkg/storage/queries.go: Filters, ListFindingsFiltered, GetFinding, DeleteFinding - [ ] 06-05-PLAN.md — cmd/keys.go command tree: list/show/export/copy/delete/verify (KEYS-01..06) diff --git a/.planning/STATE.md b/.planning/STATE.md index 9d6f21e..78f112b 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -4,13 +4,13 @@ milestone: v1.0 milestone_name: milestone status: executing stopped_at: Completed 06-03-PLAN.md -last_updated: "2026-04-05T20:32:29.678Z" +last_updated: "2026-04-05T20:34:48.003Z" last_activity: 2026-04-05 progress: total_phases: 18 completed_phases: 5 total_plans: 34 - completed_plans: 31 + completed_plans: 32 percent: 20 --- diff --git a/.planning/phases/06-output-reporting/06-02-SUMMARY.md b/.planning/phases/06-output-reporting/06-02-SUMMARY.md new file mode 100644 index 0000000..dce1b9c --- /dev/null +++ b/.planning/phases/06-output-reporting/06-02-SUMMARY.md @@ -0,0 +1,152 @@ +--- +phase: 06-output-reporting +plan: 02 +subsystem: pkg/output +tags: [output, formatter, json, csv, unmask] +requirements: [OUT-02, OUT-04] +dependency-graph: + requires: + - "pkg/output.Formatter" + - "pkg/output.Options" + - "pkg/output.Register" + - "pkg/engine.Finding" + provides: + - "output.JSONFormatter (registered as \"json\")" + - "output.CSVFormatter (registered as \"csv\")" + affects: + - "cmd/scan.go --output=json|csv (wired in later plan)" +tech-stack: + added: [] + patterns: + - "encoding/json Encoder with SetIndent for pretty output" + - "encoding/csv Writer for stable, auto-quoted flat rows" + - "omitempty on verify_* JSON fields to keep unverified output compact" + - "init()-time registration into the shared output.Registry" +key-files: + created: + - pkg/output/json.go + - pkg/output/json_test.go + - pkg/output/csv.go + - pkg/output/csv_test.go + modified: [] +decisions: + - "JSON `key` field carries masked value by default; `key_masked` always carries masked form regardless of Unmask so consumers can tell which mode produced the file." + - "Verification fields use `omitempty` so unverified scans stay compact; `verified: false` is always emitted as a required discriminator." + - "CSV `id` column is the zero-based slice index, not the DB rowid — formatter must run on in-memory findings before persistence." + - "CSV header order is frozen (id,provider,confidence,key,source,line,detected_at,verified,verify_status) — downstream consumers may index by position." + - "DetectedAt is serialised as RFC3339 in both formats for cross-format consistency." +metrics: + duration: ~10m + completed: 2026-04-05 + tasks: 2 + commits: 4 +--- + +# Phase 06 Plan 02: JSON + CSV Formatters Summary + +Added `JSONFormatter` and `CSVFormatter` to `pkg/output`, both implementing the `Formatter` interface established in Plan 01 and both honoring the `Unmask` option. These are the machine-readable siblings of the colored table output: JSON for pipelines and tooling, CSV for spreadsheets and ad-hoc grep. + +## What Was Built + +### 1. JSONFormatter (`pkg/output/json.go`) + +- Renders `[]engine.Finding` as a JSON array with 2-space indent via `json.Encoder.SetIndent("", " ")`. +- Empty input produces `"[]\n"` exactly (important for deterministic CI diffs). +- Private `jsonFinding` struct controls field names and serialization order: `provider, key, key_masked, confidence, source, source_type, line, offset, detected_at, verified, verify_status?, verify_http_code?, verify_metadata?, verify_error?`. +- `key` column: masked by default, full `KeyValue` when `Options.Unmask == true`. `key_masked` is always the masked form. +- `detected_at` serialized as RFC3339. +- Verification fields tagged `omitempty` so unverified scans don't carry dead columns. +- Registers itself as `"json"` in the output registry via `init()`. + +### 2. CSVFormatter (`pkg/output/csv.go`) + +- Writes a frozen header row followed by one flat row per finding using `encoding/csv.Writer`. +- Header (stable, index-indexable): `id,provider,confidence,key,source,line,detected_at,verified,verify_status`. +- `id` is the zero-based slice index (scan-time id; DB rowid is intentionally not used so this formatter can run on pre-persisted results). +- `key` column honors `Options.Unmask` identically to JSON. +- `verified` rendered as `"true"`/`"false"`, `line` as a plain integer, `detected_at` as RFC3339. +- Commas, quotes, and newlines in source paths are auto-escaped by `encoding/csv`. +- Empty input still writes the header row (one line) — consumers can unconditionally parse a header. +- Registers itself as `"csv"` in the output registry via `init()`. + +## Test Coverage + +### `pkg/output/json_test.go` +- `TestJSONFormatter_EmptyFindings` — exact `"[]\n"` output. +- `TestJSONFormatter_RegisteredUnderJSON` — registry lookup returns `JSONFormatter`. +- `TestJSONFormatter_MaskedByDefault` — `key == KeyMasked`, verify fields absent (omitempty). +- `TestJSONFormatter_UnmaskRevealsKey` — `key == KeyValue`, `key_masked` still masked. +- `TestJSONFormatter_VerifyFieldsPresent` — verify_status/code/metadata round-trip when `Verified=true`. +- `TestJSONFormatter_Indented` — 2-space array item indent, 4-space nested field indent. + +### `pkg/output/csv_test.go` +- `TestCSVFormatter_RegisteredUnderCSV` — registry lookup. +- `TestCSVFormatter_HeaderOnly` — empty findings produce exactly the 9-column header. +- `TestCSVFormatter_RowMasked` — full row round-trips via `csv.NewReader`, masked key, correct types. +- `TestCSVFormatter_Unmask` — Unmask=true reveals `KeyValue`. +- `TestCSVFormatter_QuotesCommaInSource` — `"path, with, commas.txt"` round-trips verbatim. +- `TestCSVFormatter_VerifiedRow` — verified/verify_status columns populated. +- `TestCSVFormatter_MultipleRowsIncrementID` — id column increments 0,1,… with slice position. + +## Commits + +| Task | Phase | Hash | Message | +| ---- | ----- | --------- | -------------------------------------------------- | +| 1 | RED | `c933673` | `test(06-02): add failing tests for JSONFormatter` | +| 1 | GREEN | `1644771` | `feat(06-02): implement JSONFormatter with Unmask support` | +| 2 | RED | `b35881a` | `test(06-02): add failing tests for CSVFormatter` | +| 2 | GREEN | `03249fb` | `feat(06-02): implement CSVFormatter with Unmask support` | + +## Verification + +``` +$ go test ./pkg/output/... -count=1 +ok github.com/salvacybersec/keyhunter/pkg/output 0.005s + +$ go build ./... +(no output — clean build) + +$ grep -h "Register(" pkg/output/*.go + Register("csv", CSVFormatter{}) + Register("json", JSONFormatter{}) + Register("sarif", SARIFFormatter{}) + Register("table", TableFormatter{}) +``` + +All four formatters (`table`, `json`, `csv`, `sarif`) now live in the registry — `--output=json|csv` works end to end once the scan command is wired up in a later plan. + +## Deviations from Plan + +### Fixed During Execution + +**1. [Rule 1 - Test bug] Incorrect indent assertion in initial JSON test** +- **Found during:** Task 1 GREEN run +- **Issue:** `TestJSONFormatter_Indented` asserted `"\n \"provider\""` (2-space indent) but `json.Encoder.SetIndent("", " ")` produces 4-space indent for nested object fields (array item at 2 spaces + 2 more for nested field). +- **Fix:** Assertion now verifies both indent levels: `"\n {"` for array items and `"\n \"provider\""` for nested fields. +- **Files modified:** `pkg/output/json_test.go` +- **Commit:** `1644771` (rolled into GREEN commit since the RED had not yet been runnable) + +Plan body (the JSON encoder behavior) was implemented exactly as specified — only the test expectation was off by one indent level. + +### Out of Scope / Observed + +- During execution, a `pkg/output/sarif_test.go` file from the parallel Plan 03 executor was present in the working tree before `pkg/output/sarif.go` existed, causing the package to fail compilation for unrelated tests. Handled by temporarily moving it aside while running `go test -run TestJSONFormatter|TestCSVFormatter`, then restoring. By the time the final package-wide test run happened, Plan 03's `sarif.go` had landed and the full package compiled and tested cleanly. No files were modified as part of this mitigation; this is a parallel-execution artifact and not a deviation from the plan. + +## Known Stubs + +None. JSON and CSV formatters emit real data from `engine.Finding` and wire through `Options.Unmask`. No placeholder fields, no TODO markers. + +## Self-Check: PASSED + +- FOUND: pkg/output/json.go +- FOUND: pkg/output/json_test.go +- FOUND: pkg/output/csv.go +- FOUND: pkg/output/csv_test.go +- FOUND: c933673 (test JSON RED) +- FOUND: 1644771 (feat JSON GREEN) +- FOUND: b35881a (test CSV RED) +- FOUND: 03249fb (feat CSV GREEN) +- FOUND: `Register("json"` in pkg/output/json.go +- FOUND: `Register("csv"` in pkg/output/csv.go +- VERIFIED: `go test ./pkg/output/... -count=1` → ok +- VERIFIED: `go build ./...` → clean