197 lines
7.3 KiB
Markdown
197 lines
7.3 KiB
Markdown
---
|
|
phase: 09-osint-infrastructure
|
|
plan: 04
|
|
type: execute
|
|
wave: 1
|
|
depends_on: []
|
|
files_modified:
|
|
- pkg/recon/robots.go
|
|
- pkg/recon/robots_test.go
|
|
- go.mod
|
|
- go.sum
|
|
autonomous: true
|
|
requirements: [RECON-INFRA-07]
|
|
must_haves:
|
|
truths:
|
|
- "pkg/recon.RobotsCache parses and caches robots.txt per host for 1 hour"
|
|
- "Allowed(host, path) returns true if robots.txt permits `keyhunter` UA on that path"
|
|
- "Cache hit avoids a second HTTP fetch for the same host within TTL"
|
|
- "Network errors degrade safely: default-allow (so a broken robots.txt fetch does not silently block sweeps)"
|
|
artifacts:
|
|
- path: "pkg/recon/robots.go"
|
|
provides: "RobotsCache with Allowed(ctx, url) bool + 1h per-host TTL"
|
|
contains: "type RobotsCache"
|
|
- path: "pkg/recon/robots_test.go"
|
|
provides: "Tests for parse/allowed/disallowed/cache-hit/network-fail"
|
|
key_links:
|
|
- from: "pkg/recon/robots.go"
|
|
to: "github.com/temoto/robotstxt"
|
|
via: "robotstxt.FromBytes"
|
|
pattern: "robotstxt\\."
|
|
---
|
|
|
|
<objective>
|
|
Add robots.txt parser and per-host cache for web-scraping sources. Satisfies RECON-INFRA-07 ("`keyhunter recon full --respect-robots` respects robots.txt for web-scraping sources before making any requests"). Only sources with RespectsRobots()==true consult the cache.
|
|
|
|
Purpose: Foundation for every later web-scraping source (Phase 11 paste, Phase 15 forums, etc.). Adds github.com/temoto/robotstxt dependency.
|
|
Output: pkg/recon/robots.go, pkg/recon/robots_test.go, go.mod/go.sum updated
|
|
</objective>
|
|
|
|
<execution_context>
|
|
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
|
@$HOME/.claude/get-shit-done/templates/summary.md
|
|
</execution_context>
|
|
|
|
<context>
|
|
@.planning/phases/09-osint-infrastructure/09-CONTEXT.md
|
|
@go.mod
|
|
</context>
|
|
|
|
<tasks>
|
|
|
|
<task type="auto">
|
|
<name>Task 1: Add temoto/robotstxt dependency</name>
|
|
<files>go.mod, go.sum</files>
|
|
<action>
|
|
Run `go get github.com/temoto/robotstxt@latest` from the repo root. This updates go.mod and go.sum. Do NOT run `go mod tidy` yet — downstream tasks in this plan consume the dep and tidy will fail if tests are not written. Prefer `go mod download github.com/temoto/robotstxt` if only population of go.sum is needed, but `go get` is canonical.
|
|
|
|
Verify the dep appears in go.mod `require` block.
|
|
</action>
|
|
<verify>
|
|
<automated>cd /home/salva/Documents/apikey && grep -q "github.com/temoto/robotstxt" go.mod</automated>
|
|
</verify>
|
|
<done>go.mod contains github.com/temoto/robotstxt; go.sum populated.</done>
|
|
</task>
|
|
|
|
<task type="auto" tdd="true">
|
|
<name>Task 2: RobotsCache with 1h TTL and default-allow on error</name>
|
|
<files>pkg/recon/robots.go, pkg/recon/robots_test.go</files>
|
|
<behavior>
|
|
- RobotsCache.Allowed(ctx, rawURL) (bool, error): parse URL -> host, fetch https://host/robots.txt (or use injected http.Client for tests), cache parsed result for 1 hour per host
|
|
- UA used for matching is "keyhunter"
|
|
- On fetch error or parse error: return true, nil (default-allow) so a broken robots endpoint does not silently disable a recon source
|
|
- Cache key is host (not full URL)
|
|
- Second call for same host within TTL does NOT trigger another HTTP request
|
|
- Tests use httptest.Server to serve robots.txt and inject a custom http.Client via RobotsCache.Client field
|
|
- Tests:
|
|
- TestRobotsAllowed: robots.txt says "User-agent: * / Disallow:" and path /public -> Allowed returns true
|
|
- TestRobotsDisallowed: robots.txt says "User-agent: * / Disallow: /private" and path /private -> false
|
|
- TestRobotsCacheHit: after first call, second call hits cache (use an atomic counter in the httptest handler and assert count == 1)
|
|
- TestRobotsNetworkError: server returns 500 -> Allowed returns true (default-allow)
|
|
- TestRobotsUAKeyhunter: robots.txt has "User-agent: keyhunter / Disallow: /blocked" -> path /blocked returns false
|
|
</behavior>
|
|
<action>
|
|
Create pkg/recon/robots.go:
|
|
|
|
```go
|
|
package recon
|
|
|
|
import (
|
|
"context"
|
|
"io"
|
|
"net/http"
|
|
"net/url"
|
|
"sync"
|
|
"time"
|
|
|
|
"github.com/temoto/robotstxt"
|
|
)
|
|
|
|
const (
|
|
robotsTTL = 1 * time.Hour
|
|
robotsUA = "keyhunter"
|
|
)
|
|
|
|
type robotsEntry struct {
|
|
data *robotstxt.RobotsData
|
|
fetched time.Time
|
|
}
|
|
|
|
// RobotsCache fetches and caches per-host robots.txt for 1 hour.
|
|
// Sources whose RespectsRobots() returns true should call Allowed before each request.
|
|
type RobotsCache struct {
|
|
mu sync.Mutex
|
|
cache map[string]robotsEntry
|
|
Client *http.Client // nil -> http.DefaultClient
|
|
}
|
|
|
|
func NewRobotsCache() *RobotsCache {
|
|
return &RobotsCache{cache: make(map[string]robotsEntry)}
|
|
}
|
|
|
|
// Allowed reports whether `keyhunter` may fetch rawURL per the host's robots.txt.
|
|
// On fetch/parse error the function returns true (default-allow) to avoid silently
|
|
// disabling recon sources when a site has a broken robots endpoint.
|
|
func (rc *RobotsCache) Allowed(ctx context.Context, rawURL string) (bool, error) {
|
|
u, err := url.Parse(rawURL)
|
|
if err != nil {
|
|
return true, nil
|
|
}
|
|
host := u.Host
|
|
|
|
rc.mu.Lock()
|
|
entry, ok := rc.cache[host]
|
|
if ok && time.Since(entry.fetched) < robotsTTL {
|
|
rc.mu.Unlock()
|
|
return entry.data.TestAgent(u.Path, robotsUA), nil
|
|
}
|
|
rc.mu.Unlock()
|
|
|
|
client := rc.Client
|
|
if client == nil {
|
|
client = http.DefaultClient
|
|
}
|
|
req, _ := http.NewRequestWithContext(ctx, "GET", u.Scheme+"://"+host+"/robots.txt", nil)
|
|
resp, err := client.Do(req)
|
|
if err != nil {
|
|
return true, nil // default-allow on network error
|
|
}
|
|
defer resp.Body.Close()
|
|
if resp.StatusCode >= 400 {
|
|
return true, nil // default-allow on 4xx/5xx
|
|
}
|
|
body, err := io.ReadAll(resp.Body)
|
|
if err != nil {
|
|
return true, nil
|
|
}
|
|
data, err := robotstxt.FromBytes(body)
|
|
if err != nil {
|
|
return true, nil
|
|
}
|
|
rc.mu.Lock()
|
|
rc.cache[host] = robotsEntry{data: data, fetched: time.Now()}
|
|
rc.mu.Unlock()
|
|
return data.TestAgent(u.Path, robotsUA), nil
|
|
}
|
|
```
|
|
|
|
Create pkg/recon/robots_test.go using httptest.NewServer. Inject the test server's client into RobotsCache.Client (use `server.Client()`). For TestRobotsCacheHit, use `atomic.Int32` incremented inside the handler.
|
|
|
|
Note on test URL: since httptest.Server has a dynamic host, build rawURL from `server.URL + "/public"`. The cache key will be the httptest host:port — both calls share the same host, so cache hit is testable.
|
|
</action>
|
|
<verify>
|
|
<automated>cd /home/salva/Documents/apikey && go test ./pkg/recon/ -run TestRobots -count=1</automated>
|
|
</verify>
|
|
<done>All 5 robots tests pass. Cache hit verified via request counter. Default-allow on 500 verified.</done>
|
|
</task>
|
|
|
|
</tasks>
|
|
|
|
<verification>
|
|
- `go test ./pkg/recon/ -run TestRobots -count=1` passes
|
|
- `go build ./...` passes (robotstxt dep resolved)
|
|
- `go vet ./pkg/recon/...` clean
|
|
</verification>
|
|
|
|
<success_criteria>
|
|
- RobotsCache implemented with 1h TTL
|
|
- UA "keyhunter" matching
|
|
- Default-allow on network/parse errors
|
|
- github.com/temoto/robotstxt added to go.mod
|
|
</success_criteria>
|
|
|
|
<output>
|
|
After completion, create `.planning/phases/09-osint-infrastructure/09-04-SUMMARY.md`
|
|
</output>
|
|
</content>
|