109 lines
5.0 KiB
Markdown
109 lines
5.0 KiB
Markdown
---
|
|
phase: 09-osint-infrastructure
|
|
plan: 04
|
|
subsystem: recon
|
|
tags: [recon, robots, http, cache]
|
|
requires: [github.com/temoto/robotstxt]
|
|
provides:
|
|
- pkg/recon.RobotsCache
|
|
- pkg/recon.NewRobotsCache
|
|
affects: [pkg/recon]
|
|
tech-stack:
|
|
added: [github.com/temoto/robotstxt@v1.1.2]
|
|
patterns: [per-host TTL cache, default-allow on error, injectable http.Client]
|
|
key-files:
|
|
created:
|
|
- pkg/recon/robots.go
|
|
- pkg/recon/robots_test.go
|
|
modified:
|
|
- go.mod
|
|
- go.sum
|
|
decisions:
|
|
- default-allow on fetch/parse error so a broken robots endpoint does not silently disable sources
|
|
- cache key is host (not full URL) to share TTL across all paths on a site
|
|
- UA matched is "keyhunter"; temoto/robotstxt TestAgent handles group precedence
|
|
- RobotsCache.Client field lets tests inject httptest.Server.Client()
|
|
metrics:
|
|
duration: ~6 min
|
|
completed: 2026-04-05
|
|
tasks: 2
|
|
files: 4
|
|
requirements: [RECON-INFRA-07]
|
|
---
|
|
|
|
# Phase 9 Plan 04: robots.txt Parser with 1h Cache Summary
|
|
|
|
RobotsCache parses robots.txt via temoto/robotstxt and caches results per host for 1 hour, with default-allow on network/parse errors and injectable HTTP client for tests.
|
|
|
|
## Objective
|
|
|
|
Foundation for every web-scraping recon source (Phase 11 paste, Phase 15 forums, etc.): a thread-safe per-host robots.txt cache that only sources with `RespectsRobots()==true` consult. Satisfies RECON-INFRA-07.
|
|
|
|
## What Was Built
|
|
|
|
- `pkg/recon/robots.go` (95 lines) — `RobotsCache` struct with `NewRobotsCache()` constructor and `Allowed(ctx, rawURL) (bool, error)` method.
|
|
- `pkg/recon/robots_test.go` (118 lines) — 5 tests covering allow/disallow/cache-hit/network-error/keyhunter-UA.
|
|
- `github.com/temoto/robotstxt@v1.1.2` added to `go.mod` and `go.sum`.
|
|
|
|
## Behavior
|
|
|
|
- Parses URL, looks up host in `sync.Mutex`-guarded map.
|
|
- Cache hit within 1h TTL returns `entry.data.TestAgent(path, "keyhunter")` without network I/O.
|
|
- Cache miss fetches `<scheme>://<host>/robots.txt`; on success, stores `robotstxt.RobotsData`.
|
|
- Network error, HTTP 4xx/5xx, read error, or parse error => returns `true, nil` (default-allow).
|
|
- `Client` field (nil -> `http.DefaultClient`) lets tests inject `httptest.Server.Client()`.
|
|
|
|
## Tasks
|
|
|
|
| # | Name | Commit | Files |
|
|
|---|------|--------|-------|
|
|
| 1 | Add temoto/robotstxt dependency | c3b9fb4 | go.mod, go.sum |
|
|
| 2a | Failing tests for RobotsCache (RED) | 4bd6c6b | pkg/recon/robots_test.go |
|
|
| 2b | Implement RobotsCache (GREEN) | 0373931 | pkg/recon/robots.go |
|
|
|
|
## Verification
|
|
|
|
- `go test ./pkg/recon/ -run TestRobots -count=1 -v` — 5/5 PASS
|
|
- TestRobotsAllowed
|
|
- TestRobotsDisallowed
|
|
- TestRobotsCacheHit (request counter == 1 after 3 calls)
|
|
- TestRobotsNetworkError (500 -> allowed)
|
|
- TestRobotsUAKeyhunter (keyhunter group matched over wildcard)
|
|
- `go vet ./pkg/recon/...` — clean
|
|
- `go build ./...` — clean
|
|
|
|
## Decisions Made
|
|
|
|
1. **Default-allow on error** — preferred over fail-closed to avoid silently disabling recon sources when a target has a misconfigured/500ing robots endpoint. Documented inline and in tests.
|
|
2. **Host-keyed cache** — a single robots.txt governs all paths on a host; caching by host shares TTL across every request to that site.
|
|
3. **Injectable Client** — `Client *http.Client` field on `RobotsCache` instead of a functional-option constructor to keep the surface minimal; tests pass `srv.Client()` directly.
|
|
4. **UA "keyhunter" literal** — constant `robotsUA = "keyhunter"` so upstream sources do not need to pass it. temoto/robotstxt `TestAgent` handles group precedence (specific UA beats wildcard).
|
|
|
|
## Deviations from Plan
|
|
|
|
### Out-of-scope discovery (NOT fixed, logged only)
|
|
|
|
**Untracked `pkg/recon/engine_test.go`** — present in the working tree from parallel wave work (plan 09-05 scaffold). It references `NewEngine`/`ExampleSource` which do not yet exist, causing the test binary for `pkg/recon` to fail to build until those types land.
|
|
|
|
- **Action taken:** Temporarily moved the file aside during my test run, verified my 5 tests pass + `go vet` clean + `go build ./...` clean, then restored the file unchanged.
|
|
- **Why not fixed:** Not caused by this plan (files_modified is `pkg/recon/robots.go` + `pkg/recon/robots_test.go` only). Belongs to plan 09-05 (wave 2, depends on 09-04). Fixing would be scope creep.
|
|
- **Impact on 09-04 verification:** None — my tests, vet, and full build all pass with the file in place only after 09-05's types exist. Under the current repository state, running `go test ./pkg/recon/...` will fail on the orphan test file, not on my code. Plan 09-05 will resolve this when it lands `engine.go` and `example.go`.
|
|
|
|
Otherwise: plan executed exactly as written.
|
|
|
|
## Known Stubs
|
|
|
|
None.
|
|
|
|
## Self-Check: PASSED
|
|
|
|
- pkg/recon/robots.go — FOUND
|
|
- pkg/recon/robots_test.go — FOUND
|
|
- go.mod contains github.com/temoto/robotstxt — VERIFIED
|
|
- Commit c3b9fb4 (Task 1) — FOUND
|
|
- Commit 4bd6c6b (Task 2 RED) — FOUND
|
|
- Commit 0373931 (Task 2 GREEN) — FOUND
|
|
- `go test ./pkg/recon/ -run TestRobots` — 5/5 PASS
|
|
- `go vet ./pkg/recon/...` — clean
|
|
- `go build ./...` — clean
|