137 lines
7.4 KiB
Markdown
137 lines
7.4 KiB
Markdown
---
|
|
phase: 04-input-sources
|
|
plan: 02
|
|
subsystem: input-sources
|
|
tags: [sources, dir, mmap, wave-1]
|
|
requires:
|
|
- 04-01 (golang.org/x/exp/mmap module available)
|
|
- pkg/engine/sources/source.go Source interface
|
|
- pkg/types/chunk.go Chunk struct
|
|
provides:
|
|
- DirSource recursive directory scanner implementing Source interface
|
|
- Shared emitChunks/isBinary helpers for all sources
|
|
- mmap-backed large file reading for FileSource and DirSource
|
|
- Default exclusion set for generated/vendored dirs
|
|
affects:
|
|
- pkg/engine/sources/dir.go
|
|
- pkg/engine/sources/dir_test.go
|
|
- pkg/engine/sources/file.go
|
|
tech-stack:
|
|
added:
|
|
- golang.org/x/exp/mmap (promoted from indirect to direct use)
|
|
patterns:
|
|
- Collect-then-sort path list for deterministic walk order
|
|
- Shared package-level emitChunks helper reused by FileSource and DirSource
|
|
- Binary detection via NUL sniff in leading 512 bytes
|
|
key-files:
|
|
created:
|
|
- pkg/engine/sources/dir.go
|
|
- pkg/engine/sources/dir_test.go
|
|
- .planning/phases/04-input-sources/deferred-items.md
|
|
modified:
|
|
- pkg/engine/sources/file.go
|
|
decisions:
|
|
- Ordered paths by sort.Strings after full walk rather than streaming, so the public contract guarantees reproducible emission order even if filepath.WalkDir's internal ordering changes
|
|
- Added NewDirSourceRaw (no-defaults) alongside NewDirSource (defaults-merged) to keep tests free of leakage from DefaultExcludes while giving callers an ergonomic constructor
|
|
- Handled `dir/**` glob style with an explicit suffix check — filepath.Match does not support ** natively, and we need .git/**, node_modules/**, vendor/** to match nested paths
|
|
- Per-file errors during emission are swallowed (continue walking) rather than aborting the entire scan — the engine logs elsewhere; context errors are still propagated
|
|
- Refactored FileSource to share emitChunks/isBinary so single-file and directory paths stay behaviorally identical (binary skip + mmap threshold)
|
|
metrics:
|
|
duration: "~4 min"
|
|
completed: 2026-04-05
|
|
tasks_completed: 1
|
|
files_modified: 3
|
|
files_created: 3
|
|
---
|
|
|
|
# Phase 4 Plan 2: DirSource Recursive Directory Scanner Summary
|
|
|
|
DirSource walks any directory recursively with glob-based exclusions, binary-file skipping, and mmap-backed reads for files over 10MB — giving the scan pipeline its primary input source. FileSource was refactored to share the same emit and mmap helpers so both paths stay consistent.
|
|
|
|
## Objective
|
|
|
|
Implement `DirSource` — a recursive directory scanner that walks a root path via `filepath.WalkDir`, honors glob exclusion patterns, detects and skips binary files, and uses memory-mapped I/O for large files. Satisfies INPUT-01 (directory/recursive scanning with exclusions) and CORE-07 (mmap large file reading).
|
|
|
|
## What Was Built
|
|
|
|
### pkg/engine/sources/dir.go (218 lines)
|
|
|
|
- `DirSource` struct with `Root`, `Excludes`, `ChunkSize` fields
|
|
- `NewDirSource(root, extraExcludes...)` — merges DefaultExcludes with user extras
|
|
- `NewDirSourceRaw(root, excludes)` — bypasses defaults entirely (used by tests)
|
|
- `Chunks(ctx, out)` — validates root is a directory, walks via `filepath.WalkDir`, collects non-excluded file paths, sorts them, and emits each file's chunks in order
|
|
- `isExcluded(rel, base)` — matches glob patterns against both basename and relative path, with explicit `dir/**` suffix handling
|
|
- `emitFile(ctx, path, out)` — reads file via `mmap.Open` if size ≥ MmapThreshold (10 MB), else `os.ReadFile`; skips binary files
|
|
- `isBinary(data)` — NUL byte sniff in the first 512 bytes (shared helper)
|
|
- `emitChunks(ctx, data, source, chunkSize, out)` — shared overlapping chunk emitter (4096 size / 256 overlap)
|
|
- Constants: `MmapThreshold = 10MB`, `BinarySniffSize = 512`, `DefaultExcludes = [.git/**, node_modules/**, vendor/**, *.min.js, *.map]`
|
|
|
|
### pkg/engine/sources/file.go (refactored, 60 lines)
|
|
|
|
- FileSource now shares `emitChunks`, `isBinary`, and the mmap read path with DirSource
|
|
- Files ≥ 10 MB are read via `golang.org/x/exp/mmap`
|
|
- Binary files are skipped before emission
|
|
|
|
### pkg/engine/sources/dir_test.go (146 lines)
|
|
|
|
Eight subtests, all passing under `-race`:
|
|
|
|
| Test | Status | What it proves |
|
|
|------|--------|----------------|
|
|
| TestDirSource_RecursiveWalk | PASS | Emits 3 chunks for 3 nested files; sources are sorted |
|
|
| TestDirSource_DefaultExcludes | PASS | `.git/config`, `node_modules/foo.js`, `vendor/bar.go`, `app.min.js`, `app.js.map` all skipped; `keep.txt` emitted |
|
|
| TestDirSource_UserExclude | PASS | `*.log` user glob skips `drop.log`, keeps `keep.txt` |
|
|
| TestDirSource_BinarySkipped | PASS | File with NUL in first bytes skipped; text file emitted |
|
|
| TestDirSource_MmapLargeFile | PASS | >10MB file emits chunks via mmap path |
|
|
| TestDirSource_DeterministicOrdering | PASS | Two consecutive runs produce identical source order |
|
|
| TestDirSource_MissingRoot | PASS | Non-existent root returns error |
|
|
| TestDirSource_CtxCancellation | PASS | Pre-cancelled ctx returns `context.Canceled` |
|
|
|
|
## Verification
|
|
|
|
```
|
|
$ go build ./pkg/engine/sources/... # exit 0
|
|
$ go vet ./pkg/engine/sources/... # clean
|
|
$ go test ./pkg/engine/sources/... -run 'TestDirSource|TestFileSource' -race -count=1
|
|
ok github.com/salvacybersec/keyhunter/pkg/engine/sources 1.120s
|
|
```
|
|
|
|
Acceptance greps all hit:
|
|
- `mmap.Open` — 2 hits (dir.go:157, file.go:41)
|
|
- `filepath.WalkDir` — 1 hit (dir.go:77)
|
|
- `DefaultExcludes` — 4 hits (dir.go)
|
|
- `isBinary` — present in dir.go:180
|
|
|
|
## Deviations from Plan
|
|
|
|
### Minor exclude-semantics adjustment
|
|
|
|
The plan's `isExcluded` implementation matched `dir/**` only against the relative path. I added one extra check so `base == prefix` also matches — this lets `.git/**` correctly exclude the top-level `.git` directory entry itself when WalkDir reports it (the `rel` for that directory is just `.git`, which already matched, but the defensive check protects against filepath quirks on Windows).
|
|
|
|
### Root-directory skip guard
|
|
|
|
Added an early `if path == d.Root { return nil }` inside the WalkDir callback. Without it, an exclusion pattern that accidentally matches an empty relative path (`.`) could cause `SkipDir` on the root, halting the walk before it starts. This is defensive, not spec-changing.
|
|
|
|
## Deferred Issues
|
|
|
|
### TestClipboardSource_ReaderError (pre-existing, out of scope)
|
|
|
|
`pkg/engine/sources/clipboard_test.go:42` fails because the test expects an error message containing the substring `"clipboard"`, but the current clipboard source returns `"ClipboardSource: read: no xclip installed"`. This failure is in a different source (clipboard, owned by a later plan) and unrelated to DirSource. Logged to `.planning/phases/04-input-sources/deferred-items.md` for the plan that owns clipboard source.
|
|
|
|
Scoped test runs (`-run 'TestDirSource|TestFileSource'`) are green under `-race`.
|
|
|
|
## Follow-up Hooks for Later Plans
|
|
|
|
- `cmd/scan.go` needs wiring: select `NewDirSource(path, excludeFlags...)` when the positional argument is a directory. Plan 04-06 (scan command integration).
|
|
- Exclude-flag plumbing: `--exclude` CLI flag to append to DefaultExcludes.
|
|
- Per-file error logging: today errors are swallowed; once the engine has a structured logger, DirSource should emit a warning for each skipped file.
|
|
|
|
## Self-Check: PASSED
|
|
|
|
- pkg/engine/sources/dir.go — FOUND (218 lines)
|
|
- pkg/engine/sources/dir_test.go — FOUND (146 lines)
|
|
- pkg/engine/sources/file.go — FOUND (refactored, 60 lines)
|
|
- RED commit ce6298f — verified in git log
|
|
- GREEN commit 6f834c9 — verified in git log
|
|
- All TestDirSource subtests pass under -race
|