178 lines
7.9 KiB
Markdown
178 lines
7.9 KiB
Markdown
---
|
|
phase: 07-import-cicd
|
|
plan: 01
|
|
type: execute
|
|
wave: 1
|
|
depends_on: []
|
|
files_modified:
|
|
- pkg/importer/importer.go
|
|
- pkg/importer/trufflehog.go
|
|
- pkg/importer/trufflehog_test.go
|
|
- pkg/importer/testdata/trufflehog-sample.json
|
|
autonomous: true
|
|
requirements: [IMP-01]
|
|
must_haves:
|
|
truths:
|
|
- "TruffleHog v3 JSON output can be parsed into []engine.Finding"
|
|
- "Detector names from TruffleHog are normalized to lowercase KeyHunter provider names"
|
|
- "Verified flag in TruffleHog JSON maps to Finding.Verified + VerifyStatus"
|
|
artifacts:
|
|
- path: pkg/importer/importer.go
|
|
provides: "Importer interface"
|
|
contains: "type Importer interface"
|
|
- path: pkg/importer/trufflehog.go
|
|
provides: "TruffleHog v3 JSON parser"
|
|
contains: "func (TruffleHogImporter) Import"
|
|
- path: pkg/importer/testdata/trufflehog-sample.json
|
|
provides: "Test fixture matching TruffleHog v3 JSON schema"
|
|
key_links:
|
|
- from: pkg/importer/trufflehog.go
|
|
to: pkg/engine/finding.go
|
|
via: "constructs engine.Finding from TruffleHog records"
|
|
pattern: "engine\\.Finding\\{"
|
|
---
|
|
|
|
<objective>
|
|
Create the pkg/importer package with the Importer interface and the TruffleHog v3 JSON adapter.
|
|
|
|
Purpose: External tool import requires a uniform contract so the CLI command (Plan 07-04) can dispatch by format flag. TruffleHog is the most widely used scanner; its v3 JSON output is the canonical import target (IMP-01).
|
|
Output: Importer interface, TruffleHogImporter implementation, unit test, JSON fixture.
|
|
</objective>
|
|
|
|
<execution_context>
|
|
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
|
@$HOME/.claude/get-shit-done/templates/summary.md
|
|
</execution_context>
|
|
|
|
<context>
|
|
@.planning/PROJECT.md
|
|
@.planning/ROADMAP.md
|
|
@.planning/STATE.md
|
|
@.planning/phases/07-import-cicd/07-CONTEXT.md
|
|
@pkg/engine/finding.go
|
|
|
|
<interfaces>
|
|
From pkg/engine/finding.go:
|
|
```go
|
|
type Finding struct {
|
|
ProviderName string
|
|
KeyValue string
|
|
KeyMasked string
|
|
Confidence string
|
|
Source string
|
|
SourceType string
|
|
LineNumber int
|
|
Offset int64
|
|
DetectedAt time.Time
|
|
Verified bool
|
|
VerifyStatus string
|
|
VerifyHTTPCode int
|
|
VerifyMetadata map[string]string
|
|
VerifyError string
|
|
}
|
|
func MaskKey(key string) string
|
|
```
|
|
</interfaces>
|
|
</context>
|
|
|
|
<tasks>
|
|
|
|
<task type="auto" tdd="true">
|
|
<name>Task 1: Importer interface + TruffleHog parser + fixtures</name>
|
|
<files>pkg/importer/importer.go, pkg/importer/trufflehog.go, pkg/importer/testdata/trufflehog-sample.json</files>
|
|
<behavior>
|
|
- Importer interface declares: Import(r io.Reader) ([]engine.Finding, error) and Name() string
|
|
- TruffleHogImporter parses a JSON array of objects with fields {SourceID, SourceName, SourceMetadata (object), DetectorName, DetectorType, Verified (bool), Raw (string), Redacted (string), ExtraData (object)}
|
|
- Each record maps to engine.Finding:
|
|
ProviderName = normalizeTruffleHogName(DetectorName) // e.g. "OpenAI" -> "openai", "GitHubV2" -> "github", "AWS" -> "aws"
|
|
KeyValue = Raw
|
|
KeyMasked = engine.MaskKey(Raw)
|
|
Confidence = "high" if Verified else "medium"
|
|
SourceType = "import:trufflehog"
|
|
Source = extractSourcePath(SourceMetadata) — traverses SourceMetadata.Data.{Git,Filesystem,Github}.file/link/repository, falls back to SourceName
|
|
LineNumber = extracted from SourceMetadata.Data.Git.line if present, else 0
|
|
Verified = Verified
|
|
VerifyStatus = "live" if Verified else "unverified"
|
|
DetectedAt = time.Now()
|
|
- normalizeTruffleHogName(): lowercase, strip trailing version digits ("GithubV2" -> "github"), map known aliases (AWS -> aws, GCP -> gcp). Unknown names: lowercased as-is.
|
|
- Invalid JSON returns wrapped error
|
|
- Empty array returns empty slice, nil error
|
|
- Name() returns "trufflehog"
|
|
</behavior>
|
|
<action>
|
|
Create pkg/importer/importer.go:
|
|
```go
|
|
package importer
|
|
|
|
import (
|
|
"io"
|
|
|
|
"github.com/salvacybersec/keyhunter/pkg/engine"
|
|
)
|
|
|
|
// Importer parses output from an external secret scanner and returns
|
|
// normalized engine.Finding records. Implementations must be stateless
|
|
// and safe for reuse across calls.
|
|
type Importer interface {
|
|
Name() string
|
|
Import(r io.Reader) ([]engine.Finding, error)
|
|
}
|
|
```
|
|
|
|
Create pkg/importer/trufflehog.go:
|
|
- Define `type TruffleHogImporter struct{}`
|
|
- Define `type trufflehogRecord struct` with JSON tags matching TruffleHog v3: SourceID, SourceName, SourceMetadata (json.RawMessage), DetectorName, DetectorType int, Verified bool, Raw, Redacted string, ExtraData json.RawMessage.
|
|
- Implement `Name() string` returning "trufflehog".
|
|
- Implement `Import(r io.Reader) ([]engine.Finding, error)`:
|
|
1. json.NewDecoder(r).Decode(&records) — if error, wrap: fmt.Errorf("decoding trufflehog json: %w", err)
|
|
2. For each record: build engine.Finding per behavior spec. Skip records with empty Raw (log-skip count via return? no — just skip silently).
|
|
3. Return slice + nil.
|
|
- Implement `normalizeTruffleHogName(detector string) string`:
|
|
- lowercased := strings.ToLower(detector)
|
|
- trim trailing "v\d+" via regexp (package-level var `var tfhVersionSuffix = regexp.MustCompile(`v\d+$`)`)
|
|
- apply alias map: {"gcp": "gcp", "aws": "aws", "openai": "openai", "anthropic": "anthropic", "huggingface": "huggingface"}
|
|
- return trimmed
|
|
- Implement `extractSourcePath(meta json.RawMessage) (path string, line int)`:
|
|
- Unmarshal into `struct{ Data struct{ Git *struct{ File, Repository, Commit string; Line int } ; Filesystem *struct{ File string }; Github *struct{ File, Link, Repository string } } }`
|
|
- Return first non-empty in priority: Git.File, Filesystem.File, Github.File, Github.Link, Git.Repository, Github.Repository. Line from Git.Line.
|
|
- On unmarshal error: return "", 0 (not fatal).
|
|
|
|
Create pkg/importer/testdata/trufflehog-sample.json with a realistic fixture containing 3 records:
|
|
- record 1: DetectorName "OpenAI", Verified true, Raw "sk-proj-abcdef1234567890abcdef", SourceMetadata.Data.Git.File "src/config.py", Line 42
|
|
- record 2: DetectorName "AnthropicV2", Verified false, Raw "sk-ant-api03-xxxxxxxxxxxxxxxx", SourceMetadata.Data.Filesystem.File "/tmp/leaked.env"
|
|
- record 3: DetectorName "AWS", Verified true, Raw "AKIAIOSFODNN7EXAMPLE", SourceMetadata.Data.Github.Link "https://github.com/foo/bar/blob/main/a.yml"
|
|
|
|
Create pkg/importer/trufflehog_test.go:
|
|
- TestTruffleHogImporter_Import: open testdata, call Import, assert len==3, assert findings[0].ProviderName=="openai", Confidence=="high", Verified==true, Source=="src/config.py", LineNumber==42.
|
|
- TestTruffleHogImporter_NormalizeName: table test — {"OpenAI","openai"}, {"GithubV2","github"}, {"AnthropicV2","anthropic"}, {"AWS","aws"}, {"UnknownDetector","unknowndetector"}.
|
|
- TestTruffleHogImporter_EmptyArray: Import(strings.NewReader("[]")) returns empty slice, nil error.
|
|
- TestTruffleHogImporter_InvalidJSON: Import(strings.NewReader("not json")) returns error.
|
|
- TestTruffleHogImporter_Name: asserts "trufflehog".
|
|
|
|
All TruffleHog v3 field decisions per 07-CONTEXT.md decisions block.
|
|
</action>
|
|
<verify>
|
|
<automated>cd /home/salva/Documents/apikey && go test ./pkg/importer/... -run TruffleHog -v</automated>
|
|
</verify>
|
|
<done>
|
|
- pkg/importer/importer.go declares Importer interface
|
|
- pkg/importer/trufflehog.go implements Importer for TruffleHog v3 JSON
|
|
- All 5 tests pass
|
|
- go build ./pkg/importer/... succeeds
|
|
</done>
|
|
</task>
|
|
|
|
</tasks>
|
|
|
|
<verification>
|
|
go test ./pkg/importer/... -v passes. go vet ./pkg/importer/... clean.
|
|
</verification>
|
|
|
|
<success_criteria>
|
|
TruffleHog v3 JSON can be loaded from disk and converted to []engine.Finding with correct provider name normalization and verify status mapping.
|
|
</success_criteria>
|
|
|
|
<output>
|
|
After completion, create `.planning/phases/07-import-cicd/07-01-SUMMARY.md`.
|
|
</output>
|