salvacybersec/keyhunter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
377 Commits 1 Branch 0 Tags
main
Commit Graph

1 Commits

Author SHA1 Message Date
salvacybersec
0373931490 feat(09-04): implement RobotsCache with 1h per-host TTL 
- Parses robots.txt via temoto/robotstxt
- Caches per host for 1 hour; second call within TTL skips HTTP fetch
- Default-allow on network/parse/4xx/5xx errors
- Matches 'keyhunter' user-agent against disallowed paths
- Client field allows httptest injection

Satisfies RECON-INFRA-07.
 2026-04-06 00:42:33 +03:00