docs(06-02): complete JSON + CSV formatter plan
This commit is contained in:
salvacybersec
@@ -51,9 +51,9 @@ Requirements for initial release. Each maps to roadmap phases.
|
||||
### Output & Reporting
|
||||
|
||||
- [x] **OUT-01**: Colored terminal table output (default)
|
||||
- [ ] **OUT-02**: JSON output format
|
||||
- [x] **OUT-02**: JSON output format
|
||||
- [x] **OUT-03**: SARIF output format (CI/CD compatible)
|
||||
- [ ] **OUT-04**: CSV output format
|
||||
- [x] **OUT-04**: CSV output format
|
||||
- [ ] **OUT-05**: Key masking by default (first 8 + last 4 chars) with --unmask flag for full keys
|
||||
- [x] **OUT-06**: Exit codes: 0=clean, 1=keys found, 2=error
|
||||
|
||||
|
||||
@@ -143,7 +143,7 @@ Plans:
|
||||
|
||||
Plans:
|
||||
- [x] 06-01-PLAN.md — Wave 0: Formatter interface, colors.go (TTY/NO_COLOR), refactor TableFormatter
|
||||
- [ ] 06-02-PLAN.md — JSONFormatter + CSVFormatter (full Finding fields, Unmask option)
|
||||
- [x] 06-02-PLAN.md — JSONFormatter + CSVFormatter (full Finding fields, Unmask option)
|
||||
- [x] 06-03-PLAN.md — SARIF 2.1.0 formatter with custom structs (rule dedup, level mapping)
|
||||
- [x] 06-04-PLAN.md — pkg/storage/queries.go: Filters, ListFindingsFiltered, GetFinding, DeleteFinding
|
||||
- [ ] 06-05-PLAN.md — cmd/keys.go command tree: list/show/export/copy/delete/verify (KEYS-01..06)
|
||||
|
||||
@@ -4,13 +4,13 @@ milestone: v1.0
|
||||
milestone_name: milestone
|
||||
status: executing
|
||||
stopped_at: Completed 06-03-PLAN.md
|
||||
last_updated: "2026-04-05T20:32:29.678Z"
|
||||
last_updated: "2026-04-05T20:34:48.003Z"
|
||||
last_activity: 2026-04-05
|
||||
progress:
|
||||
total_phases: 18
|
||||
completed_phases: 5
|
||||
total_plans: 34
|
||||
completed_plans: 31
|
||||
completed_plans: 32
|
||||
percent: 20
|
||||
---
|
||||
|
||||
|
||||
152
.planning/phases/06-output-reporting/06-02-SUMMARY.md
Normal file
152
.planning/phases/06-output-reporting/06-02-SUMMARY.md
Normal file
@@ -0,0 +1,152 @@
|
||||
---
|
||||
phase: 06-output-reporting
|
||||
plan: 02
|
||||
subsystem: pkg/output
|
||||
tags: [output, formatter, json, csv, unmask]
|
||||
requirements: [OUT-02, OUT-04]
|
||||
dependency-graph:
|
||||
requires:
|
||||
- "pkg/output.Formatter"
|
||||
- "pkg/output.Options"
|
||||
- "pkg/output.Register"
|
||||
- "pkg/engine.Finding"
|
||||
provides:
|
||||
- "output.JSONFormatter (registered as \"json\")"
|
||||
- "output.CSVFormatter (registered as \"csv\")"
|
||||
affects:
|
||||
- "cmd/scan.go --output=json|csv (wired in later plan)"
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns:
|
||||
- "encoding/json Encoder with SetIndent for pretty output"
|
||||
- "encoding/csv Writer for stable, auto-quoted flat rows"
|
||||
- "omitempty on verify_* JSON fields to keep unverified output compact"
|
||||
- "init()-time registration into the shared output.Registry"
|
||||
key-files:
|
||||
created:
|
||||
- pkg/output/json.go
|
||||
- pkg/output/json_test.go
|
||||
- pkg/output/csv.go
|
||||
- pkg/output/csv_test.go
|
||||
modified: []
|
||||
decisions:
|
||||
- "JSON `key` field carries masked value by default; `key_masked` always carries masked form regardless of Unmask so consumers can tell which mode produced the file."
|
||||
- "Verification fields use `omitempty` so unverified scans stay compact; `verified: false` is always emitted as a required discriminator."
|
||||
- "CSV `id` column is the zero-based slice index, not the DB rowid — formatter must run on in-memory findings before persistence."
|
||||
- "CSV header order is frozen (id,provider,confidence,key,source,line,detected_at,verified,verify_status) — downstream consumers may index by position."
|
||||
- "DetectedAt is serialised as RFC3339 in both formats for cross-format consistency."
|
||||
metrics:
|
||||
duration: ~10m
|
||||
completed: 2026-04-05
|
||||
tasks: 2
|
||||
commits: 4
|
||||
---
|
||||
|
||||
# Phase 06 Plan 02: JSON + CSV Formatters Summary
|
||||
|
||||
Added `JSONFormatter` and `CSVFormatter` to `pkg/output`, both implementing the `Formatter` interface established in Plan 01 and both honoring the `Unmask` option. These are the machine-readable siblings of the colored table output: JSON for pipelines and tooling, CSV for spreadsheets and ad-hoc grep.
|
||||
|
||||
## What Was Built
|
||||
|
||||
### 1. JSONFormatter (`pkg/output/json.go`)
|
||||
|
||||
- Renders `[]engine.Finding` as a JSON array with 2-space indent via `json.Encoder.SetIndent("", " ")`.
|
||||
- Empty input produces `"[]\n"` exactly (important for deterministic CI diffs).
|
||||
- Private `jsonFinding` struct controls field names and serialization order: `provider, key, key_masked, confidence, source, source_type, line, offset, detected_at, verified, verify_status?, verify_http_code?, verify_metadata?, verify_error?`.
|
||||
- `key` column: masked by default, full `KeyValue` when `Options.Unmask == true`. `key_masked` is always the masked form.
|
||||
- `detected_at` serialized as RFC3339.
|
||||
- Verification fields tagged `omitempty` so unverified scans don't carry dead columns.
|
||||
- Registers itself as `"json"` in the output registry via `init()`.
|
||||
|
||||
### 2. CSVFormatter (`pkg/output/csv.go`)
|
||||
|
||||
- Writes a frozen header row followed by one flat row per finding using `encoding/csv.Writer`.
|
||||
- Header (stable, index-indexable): `id,provider,confidence,key,source,line,detected_at,verified,verify_status`.
|
||||
- `id` is the zero-based slice index (scan-time id; DB rowid is intentionally not used so this formatter can run on pre-persisted results).
|
||||
- `key` column honors `Options.Unmask` identically to JSON.
|
||||
- `verified` rendered as `"true"`/`"false"`, `line` as a plain integer, `detected_at` as RFC3339.
|
||||
- Commas, quotes, and newlines in source paths are auto-escaped by `encoding/csv`.
|
||||
- Empty input still writes the header row (one line) — consumers can unconditionally parse a header.
|
||||
- Registers itself as `"csv"` in the output registry via `init()`.
|
||||
|
||||
## Test Coverage
|
||||
|
||||
### `pkg/output/json_test.go`
|
||||
- `TestJSONFormatter_EmptyFindings` — exact `"[]\n"` output.
|
||||
- `TestJSONFormatter_RegisteredUnderJSON` — registry lookup returns `JSONFormatter`.
|
||||
- `TestJSONFormatter_MaskedByDefault` — `key == KeyMasked`, verify fields absent (omitempty).
|
||||
- `TestJSONFormatter_UnmaskRevealsKey` — `key == KeyValue`, `key_masked` still masked.
|
||||
- `TestJSONFormatter_VerifyFieldsPresent` — verify_status/code/metadata round-trip when `Verified=true`.
|
||||
- `TestJSONFormatter_Indented` — 2-space array item indent, 4-space nested field indent.
|
||||
|
||||
### `pkg/output/csv_test.go`
|
||||
- `TestCSVFormatter_RegisteredUnderCSV` — registry lookup.
|
||||
- `TestCSVFormatter_HeaderOnly` — empty findings produce exactly the 9-column header.
|
||||
- `TestCSVFormatter_RowMasked` — full row round-trips via `csv.NewReader`, masked key, correct types.
|
||||
- `TestCSVFormatter_Unmask` — Unmask=true reveals `KeyValue`.
|
||||
- `TestCSVFormatter_QuotesCommaInSource` — `"path, with, commas.txt"` round-trips verbatim.
|
||||
- `TestCSVFormatter_VerifiedRow` — verified/verify_status columns populated.
|
||||
- `TestCSVFormatter_MultipleRowsIncrementID` — id column increments 0,1,… with slice position.
|
||||
|
||||
## Commits
|
||||
|
||||
| Task | Phase | Hash | Message |
|
||||
| ---- | ----- | --------- | -------------------------------------------------- |
|
||||
| 1 | RED | `c933673` | `test(06-02): add failing tests for JSONFormatter` |
|
||||
| 1 | GREEN | `1644771` | `feat(06-02): implement JSONFormatter with Unmask support` |
|
||||
| 2 | RED | `b35881a` | `test(06-02): add failing tests for CSVFormatter` |
|
||||
| 2 | GREEN | `03249fb` | `feat(06-02): implement CSVFormatter with Unmask support` |
|
||||
|
||||
## Verification
|
||||
|
||||
```
|
||||
$ go test ./pkg/output/... -count=1
|
||||
ok github.com/salvacybersec/keyhunter/pkg/output 0.005s
|
||||
|
||||
$ go build ./...
|
||||
(no output — clean build)
|
||||
|
||||
$ grep -h "Register(" pkg/output/*.go
|
||||
Register("csv", CSVFormatter{})
|
||||
Register("json", JSONFormatter{})
|
||||
Register("sarif", SARIFFormatter{})
|
||||
Register("table", TableFormatter{})
|
||||
```
|
||||
|
||||
All four formatters (`table`, `json`, `csv`, `sarif`) now live in the registry — `--output=json|csv` works end to end once the scan command is wired up in a later plan.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
### Fixed During Execution
|
||||
|
||||
**1. [Rule 1 - Test bug] Incorrect indent assertion in initial JSON test**
|
||||
- **Found during:** Task 1 GREEN run
|
||||
- **Issue:** `TestJSONFormatter_Indented` asserted `"\n \"provider\""` (2-space indent) but `json.Encoder.SetIndent("", " ")` produces 4-space indent for nested object fields (array item at 2 spaces + 2 more for nested field).
|
||||
- **Fix:** Assertion now verifies both indent levels: `"\n {"` for array items and `"\n \"provider\""` for nested fields.
|
||||
- **Files modified:** `pkg/output/json_test.go`
|
||||
- **Commit:** `1644771` (rolled into GREEN commit since the RED had not yet been runnable)
|
||||
|
||||
Plan body (the JSON encoder behavior) was implemented exactly as specified — only the test expectation was off by one indent level.
|
||||
|
||||
### Out of Scope / Observed
|
||||
|
||||
- During execution, a `pkg/output/sarif_test.go` file from the parallel Plan 03 executor was present in the working tree before `pkg/output/sarif.go` existed, causing the package to fail compilation for unrelated tests. Handled by temporarily moving it aside while running `go test -run TestJSONFormatter|TestCSVFormatter`, then restoring. By the time the final package-wide test run happened, Plan 03's `sarif.go` had landed and the full package compiled and tested cleanly. No files were modified as part of this mitigation; this is a parallel-execution artifact and not a deviation from the plan.
|
||||
|
||||
## Known Stubs
|
||||
|
||||
None. JSON and CSV formatters emit real data from `engine.Finding` and wire through `Options.Unmask`. No placeholder fields, no TODO markers.
|
||||
|
||||
## Self-Check: PASSED
|
||||
|
||||
- FOUND: pkg/output/json.go
|
||||
- FOUND: pkg/output/json_test.go
|
||||
- FOUND: pkg/output/csv.go
|
||||
- FOUND: pkg/output/csv_test.go
|
||||
- FOUND: c933673 (test JSON RED)
|
||||
- FOUND: 1644771 (feat JSON GREEN)
|
||||
- FOUND: b35881a (test CSV RED)
|
||||
- FOUND: 03249fb (feat CSV GREEN)
|
||||
- FOUND: `Register("json"` in pkg/output/json.go
|
||||
- FOUND: `Register("csv"` in pkg/output/csv.go
|
||||
- VERIFIED: `go test ./pkg/output/... -count=1` → ok
|
||||
- VERIFIED: `go build ./...` → clean
|
||||
Reference in New Issue
Block a user