diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 01c3d80..3588c00 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -70,7 +70,7 @@ Requirements for initial release. Each maps to roadmap phases. - [ ] **IMP-01**: TruffleHog JSON output parser and importer - [ ] **IMP-02**: Gitleaks JSON output parser and importer -- [ ] **IMP-03**: Generic CSV import for custom tool output +- [x] **IMP-03**: Generic CSV import for custom tool output ### Storage @@ -89,7 +89,7 @@ Requirements for initial release. Each maps to roadmap phases. ### CI/CD Integration - [ ] **CICD-01**: keyhunter hook install/uninstall for git pre-commit hooks -- [ ] **CICD-02**: SARIF output uploadable to GitHub Security tab +- [x] **CICD-02**: SARIF output uploadable to GitHub Security tab ### OSINT/Recon — IoT & Internet Scanners diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index f44dbc2..d9ffc04 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -161,9 +161,9 @@ Plans: **Plans**: 6 plans Plans: -- [ ] 07-01-PLAN.md — pkg/importer Importer interface + TruffleHog v3 JSON parser + fixtures (IMP-01) -- [ ] 07-02-PLAN.md — Gitleaks JSON + CSV parsers (IMP-02) -- [ ] 07-03-PLAN.md — Dedup helper + SARIF GitHub Code Scanning validation test (IMP-03, CICD-02) +- [x] 07-01-PLAN.md — pkg/importer Importer interface + TruffleHog v3 JSON parser + fixtures (IMP-01) +- [x] 07-02-PLAN.md — Gitleaks JSON + CSV parsers (IMP-02) +- [x] 07-03-PLAN.md — Dedup helper + SARIF GitHub Code Scanning validation test (IMP-03, CICD-02) - [ ] 07-04-PLAN.md — cmd/import.go wiring format dispatch, dedup, DB persistence (IMP-01/02/03) - [ ] 07-05-PLAN.md — cmd/hook.go install/uninstall with embedded pre-commit script (CICD-01) - [ ] 07-06-PLAN.md — docs/CI-CD.md + README CI/CD section with GitHub Actions workflow (CICD-01, CICD-02) diff --git a/.planning/STATE.md b/.planning/STATE.md index 8e3be49..1aa7015 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -4,13 +4,13 @@ milestone: v1.0 milestone_name: milestone status: executing stopped_at: Completed 06-06-PLAN.md -last_updated: "2026-04-05T20:46:26.438Z" -last_activity: 2026-04-05 +last_updated: "2026-04-05T20:56:37.008Z" +last_activity: 2026-04-05 -- Phase 07 execution started progress: total_phases: 18 completed_phases: 6 - total_plans: 34 - completed_plans: 34 + total_plans: 40 + completed_plans: 37 percent: 20 --- @@ -21,14 +21,14 @@ progress: See: .planning/PROJECT.md (updated 2026-04-04) **Core value:** Detect leaked LLM API keys across more providers and more internet sources than any other tool, with active verification to confirm keys are real and alive. -**Current focus:** Phase 06 — output-reporting +**Current focus:** Phase 07 — import-cicd ## Current Position -Phase: 7 -Plan: Not started -Status: Ready to execute -Last activity: 2026-04-05 +Phase: 07 (import-cicd) — EXECUTING +Plan: 1 of 6 +Status: Executing Phase 07 +Last activity: 2026-04-05 -- Phase 07 execution started Progress: [██░░░░░░░░] 20% diff --git a/.planning/phases/07-import-cicd/07-03-SUMMARY.md b/.planning/phases/07-import-cicd/07-03-SUMMARY.md new file mode 100644 index 0000000..898867a --- /dev/null +++ b/.planning/phases/07-import-cicd/07-03-SUMMARY.md @@ -0,0 +1,104 @@ +--- +phase: 07-import-cicd +plan: 03 +subsystem: importer+output +tags: [importer, dedup, sarif, cicd, github-code-scanning] +requires: + - pkg/engine/finding.go + - pkg/output/sarif.go +provides: + - pkg/importer.FindingKey + - pkg/importer.Dedup + - testdata/sarif/sarif-2.1.0-minimal-schema.json + - pkg/output.TestSARIFGitHubValidation +affects: + - pkg/importer + - pkg/output +tech-stack: + added: [] + patterns: + - "SHA-256 stable hash for dedup identity (provider\\x00masked\\x00source\\x00line)" + - "Fixture-driven schema validation via JSON field list, stdlib only" +key-files: + created: + - pkg/importer/dedup.go + - pkg/importer/dedup_test.go + - pkg/output/sarif_github_test.go + - testdata/sarif/sarif-2.1.0-minimal-schema.json + modified: [] +decisions: + - "FindingKey uses provider+masked+source+line as identity tuple; DetectedAt and Confidence intentionally excluded so re-imports collapse" + - "SARIF validation uses a hand-rolled required-fields fixture rather than the full 500KB SARIF 2.1.0 schema — zero external deps, targets GitHub's enforced surface" + - "Test file walks ../.. to locate repo root rather than hardcoding paths, keeping the fixture relocatable" +metrics: + duration: "~4 min" + completed: 2026-04-05 +requirements: [IMP-03, CICD-02] +--- + +# Phase 7 Plan 03: Dedup Helper & SARIF GitHub Validation Summary + +Adds a stable dedup identity hash for imported findings (IMP-03) and a standalone test proving Phase 6's `SARIFFormatter` output satisfies GitHub Code Scanning's required-field surface (CICD-02). + +## What Was Built + +### Task 1: Dedup helper (`pkg/importer/dedup.go`) +- `FindingKey(engine.Finding) string`: hex-encoded SHA-256 over `provider\x00masked\x00source\x00line`. Fields outside that tuple (DetectedAt, Confidence, VerifyStatus, ...) do not participate — re-running the same import later must collapse onto the original finding. +- `Dedup([]engine.Finding) ([]engine.Finding, int)`: preserves first-seen order, returns the deduplicated slice plus count of duplicates dropped. Callers use the drop count to surface `"Imported N findings (M new, K duplicates)"`. + +Tests (`pkg/importer/dedup_test.go`, 8 total, all passing): +- `TestFindingKey_Stable` +- `TestFindingKey_DiffersByProvider / ByMasked / BySource / ByLine` +- `TestDedup_PreservesOrder` — `[A,B,A,C,B] -> [A,B,C]`, dropped=2 +- `TestDedup_Empty` — nil input yields empty slice, 0 dropped +- `TestDedup_IgnoresUnrelatedFields` — identical identity but different `DetectedAt`/`Confidence` collapses to one, first-seen wins + +### Task 2: SARIF GitHub validation test +- `testdata/sarif/sarif-2.1.0-minimal-schema.json`: required-fields subset for GitHub Code Scanning (top-level, run, tool.driver, result, location.physicalLocation, region, allowed result levels). Hand-curated from https://docs.github.com/en/code-security/code-scanning/integrating-with-code-scanning/sarif-support-for-code-scanning rather than shipping the 500KB full schema. +- `pkg/output/sarif_github_test.go`: + - `TestSARIFGitHubValidation`: renders 3 findings (high/medium/low confidence) through `SARIFFormatter.Format`, unmarshals to `map[string]any`, then walks the document and asserts every required field in the fixture exists, `version == "2.1.0"`, `$schema` is a non-empty `https://` URL, exactly one run, `tool.driver.name == "keyhunter"`, each result has `ruleId`, `level ∈ allowed_levels`, non-empty `message.text`, non-empty `locations`, and every `physicalLocation.region.startLine >= 1`. Includes one finding with `LineNumber: 0` to prove the Phase 6 `startLine` floor converts to 1. + - `TestSARIFGitHubValidation_EmptyFindings`: empty input still produces a valid document with `runs[0].results == []` (not `null`) and tool.driver populated. +- Uses stdlib only (`encoding/json`, `os`, `path/filepath`, `strings`, `testing`). Test walks `../../testdata/sarif/...` from the package directory so the fixture stays relocatable. + +## Commits + +| Task | Commit | Description | +| ---- | ------- | ---------------------------------------- | +| 1 | 6a3d5b0 | feat(07-03): dedup helper for imported findings | +| 2 | bd8eb9b | test(07-03): SARIF GitHub code scanning validation | + +## Verification + +``` +$ go test ./pkg/importer/... ./pkg/output/... +ok github.com/salvacybersec/keyhunter/pkg/importer (cached) +ok github.com/salvacybersec/keyhunter/pkg/output 0.008s +``` + +All 8 importer dedup tests + 2 SARIF validation tests pass. Both targeted `go test` invocations from the plan's verify blocks pass first-try with no deviations. + +## Deviations from Plan + +None - plan executed exactly as written. + +## Downstream Consumers + +- **Plan 07-04** (import CLI command): will call `importer.Dedup` after parsing TruffleHog/Gitleaks output and report `(unique, dropped)` to the user before inserting into storage. +- **CICD-02 closure**: `TestSARIFGitHubValidation` functions as a regression guard — any future change to `SARIFFormatter` that breaks GitHub Code Scanning upload compatibility will fail this test before reaching users. + +## Key Decisions + +1. **Identity tuple excludes verification/timing fields.** A verified finding re-imported later should still collapse onto the original. Only provider + masked key + source + line number define identity. +2. **Hand-rolled required-fields fixture over full SARIF schema.** GitHub enforces a small, well-documented subset. Shipping the full 500KB JSON Schema and a validator library would bloat the test binary without catching more bugs that matter to GitHub uploads. +3. **Line number floor asserted in test.** Makes the Phase 6 `startLine = max(line, 1)` behavior a contract rather than an incidental implementation detail — future refactors can't silently reintroduce `startLine: 0`, which GitHub rejects. + +## Self-Check: PASSED + +- FOUND: pkg/importer/dedup.go +- FOUND: pkg/importer/dedup_test.go +- FOUND: pkg/output/sarif_github_test.go +- FOUND: testdata/sarif/sarif-2.1.0-minimal-schema.json +- FOUND commit: 6a3d5b0 +- FOUND commit: bd8eb9b +- Tests verified green: `go test ./pkg/importer/... ./pkg/output/...` +- No stubs introduced.