Replace the cmd/import stub with a fully wired command that parses external scanner output (via pkg/importer), deduplicates, and persists findings to the KeyHunter SQLite database.
Purpose: Delivers IMP-01/02/03 end-to-end from CLI. Users can consolidate TruffleHog and Gitleaks scans into the unified KeyHunter database.
Output: Working keyhunter import command with tests.
@.planning/phases/07-import-cicd/07-CONTEXT.md
@cmd/stubs.go
@cmd/root.go
@pkg/storage/findings.go
From pkg/importer (Plans 07-01, 07-02, 07-03):
```go
type Importer interface {
Name() string
Import(r io.Reader) ([]engine.Finding, error)
}
type TruffleHogImporter struct{}
type GitleaksImporter struct{}
type GitleaksCSVImporter struct{}
func FindingKey(f engine.Finding) string
func Dedup(in []engine.Finding) (unique []engine.Finding, duplicates int)
```
From pkg/storage/findings.go:
```go
func (db *DB) SaveFinding(f storage.Finding, encKey []byte) (int64, error)
```
storage.Finding fields: ProviderName, KeyValue, KeyMasked, Confidence, SourcePath, SourceType, LineNumber, Verified, VerifyStatus, VerifyHTTPCode, VerifyMetadata, ScanID.
Note field name difference: storage uses SourcePath; engine uses Source. Conversion required.
Task 1: Implement cmd/import.go with format dispatch and dedup
cmd/import.go, cmd/stubs.go, cmd/import_test.go
Remove the `importCmd` stub from cmd/stubs.go (delete the `var importCmd = &cobra.Command{...}` block). Leave all other stubs intact.
Create cmd/import.go:
```go
package cmd
import (
"fmt"
"io"
"os"
"time"
"github.com/spf13/cobra"
"github.com/salvacybersec/keyhunter/pkg/engine"
"github.com/salvacybersec/keyhunter/pkg/importer"
"github.com/salvacybersec/keyhunter/pkg/storage"
)
var (
importFormat string
)
var importCmd = &cobra.Command{
Use: "import <file>",
Short: "Import findings from TruffleHog or Gitleaks output",
Long: `Import scan output from external secret scanners into the KeyHunter database. Supported formats: trufflehog (v3 JSON), gitleaks (JSON), gitleaks-csv.`,
Args: cobra.ExactArgs(1),
RunE: runImport,
}
func init() {
importCmd.Flags().StringVar(&importFormat, "format", "", "input format: trufflehog | gitleaks | gitleaks-csv (required)")
_ = importCmd.MarkFlagRequired("format")
}
func runImport(cmd *cobra.Command, args []string) error {
path := args[0]
imp, err := selectImporter(importFormat)
if err != nil {
return err
}
f, err := os.Open(path)
if err != nil {
return fmt.Errorf("opening %s: %w", path, err)
}
defer f.Close()
findings, err := imp.Import(f)
if err != nil {
return fmt.Errorf("parsing %s output: %w", imp.Name(), err)
}
unique, dupes := importer.Dedup(findings)
db, encKey, err := openDBForImport()
if err != nil {
return err
}
defer db.Close()
newCount := 0
dbDupes := 0
for _, finding := range unique {
sf := engineToStorage(finding)
// Defense against cross-import duplicates already in DB:
exists, err := findingExistsInDB(db, finding)
if err != nil {
return err
}
if exists {
dbDupes++
continue
}
if _, err := db.SaveFinding(sf, encKey); err != nil {
return fmt.Errorf("saving finding: %w", err)
}
newCount++
}
totalDupes := dupes + dbDupes
fmt.Fprintf(cmd.OutOrStdout(), "Imported %d findings (%d new, %d duplicates)\n", len(findings), newCount, totalDupes)
return nil
}
func selectImporter(format string) (importer.Importer, error) {
switch format {
case "trufflehog":
return importer.TruffleHogImporter{}, nil
case "gitleaks":
return importer.GitleaksImporter{}, nil
case "gitleaks-csv":
return importer.GitleaksCSVImporter{}, nil
default:
return nil, fmt.Errorf("unknown format %q (want trufflehog | gitleaks | gitleaks-csv)", format)
}
}
func engineToStorage(f engine.Finding) storage.Finding {
if f.DetectedAt.IsZero() {
f.DetectedAt = time.Now()
}
return storage.Finding{
ProviderName: f.ProviderName,
KeyValue: f.KeyValue,
KeyMasked: f.KeyMasked,
Confidence: f.Confidence,
SourcePath: f.Source,
SourceType: f.SourceType,
LineNumber: f.LineNumber,
Verified: f.Verified,
VerifyStatus: f.VerifyStatus,
VerifyHTTPCode: f.VerifyHTTPCode,
VerifyMetadata: f.VerifyMetadata,
}
}
// openDBForImport opens the configured DB using the same helpers as scan/keys.
// Reuse whatever helper already exists in cmd/ (e.g., openDBWithKey from keys.go).
// If no shared helper exists, extract one from cmd/scan.go.
func openDBForImport() (*storage.DB, []byte, error) {
// TODO-executor: reuse existing DB-open helper from cmd/scan.go or cmd/keys.go.
// Do NOT duplicate encryption key derivation — call into the existing helper.
return nil, nil, fmt.Errorf("not yet wired")
}
// findingExistsInDB checks if a finding with the same provider + masked key + source + line
// already exists. Uses importer.FindingKey-style logic via a DB query against findings table.
func findingExistsInDB(db *storage.DB, f engine.Finding) (bool, error) {
// Executor: add a storage helper or use db.SQL() with:
// SELECT 1 FROM findings WHERE provider_name=? AND key_masked=? AND source_path=? AND line_number=? LIMIT 1
return false, nil
}
```
CRITICAL executor notes:
1. Inspect cmd/scan.go and cmd/keys.go to find the existing DB-open + passphrase helper (e.g., `openDBWithPassphrase` or similar). Use that helper — do not reimplement encryption key derivation. Replace the `openDBForImport` body accordingly.
2. Inspect pkg/storage for an existing "find by key" helper. If none, add a thin method `func (db *DB) FindingExistsByKey(provider, masked, sourcePath string, line int) (bool, error)` to pkg/storage/queries.go that runs the SELECT above. If you add this method, update pkg/storage/queries.go to include it, and add a test in pkg/storage (simple in-memory roundtrip).
3. Register importCmd: it's already added in cmd/root.go via `rootCmd.AddCommand(importCmd)`. Since you removed the stub, your new `var importCmd` declaration takes over the identifier — no root.go change needed.
Create cmd/import_test.go:
- TestSelectImporter: table — {"trufflehog", TruffleHogImporter}, {"gitleaks", GitleaksImporter}, {"gitleaks-csv", GitleaksCSVImporter}, {"bogus", error}.
- TestEngineToStorage: converts engine.Finding (with Source="a.yml", LineNumber=5, Verified=true) to storage.Finding (SourcePath="a.yml", LineNumber=5, Verified=true).
- TestRunImport_EndToEnd (integration-style):
* Create a temp DB via existing test helpers (look for one in cmd/*_test.go or pkg/storage/*_test.go).
* Write a tiny TruffleHog JSON file to a temp path.
* Invoke importCmd.Execute() with args `["import", "--format=trufflehog", tmpPath]`.
* Assert stdout contains "Imported" and "new".
* Assert db.ListFindings returns at least 1 finding with ProviderName set.
* Re-run the same command → assert output reports "0 new" and dupe count equals prior insert count.
* If a shared test DB helper is not discoverable, mark this subtest with t.Skip("needs shared test DB helper") but still ship TestSelectImporter and TestEngineToStorage.
cd /home/salva/Documents/apikey && go build ./... && go test ./cmd/... -run Import -v
- cmd/import.go replaces the stub; stub removed from cmd/stubs.go
- `keyhunter import --format=trufflehog sample.json` inserts findings
- Re-running the same import reports all as duplicates
- Unit tests pass; build succeeds
Manual smoke test:
```
go run ./cmd/keyhunter import --format=trufflehog pkg/importer/testdata/trufflehog-sample.json
# Expect: "Imported 3 findings (3 new, 0 duplicates)"
go run ./cmd/keyhunter import --format=trufflehog pkg/importer/testdata/trufflehog-sample.json
# Expect: "Imported 3 findings (0 new, 3 duplicates)"
```
<success_criteria>
IMP-01, IMP-02, IMP-03 delivered end-to-end: external scanner output can be imported, deduped, and persisted; repeat imports are idempotent.
</success_criteria>
After completion, create `.planning/phases/07-import-cicd/07-04-SUMMARY.md`.
