docs(09-04): complete robots.txt cache plan

Adds SUMMARY, marks RECON-INFRA-07 complete, updates phase 9 roadmap.
This commit is contained in:
salvacybersec
2026-04-06 00:43:49 +03:00
parent 851b2432b8
commit 4dbc38dcc5
4 changed files with 122 additions and 13 deletions

View File

@@ -206,7 +206,7 @@ Requirements for initial release. Each maps to roadmap phases.
- [x] **RECON-INFRA-05**: Per-source rate limiter with configurable limits
- [ ] **RECON-INFRA-06**: Stealth mode (--stealth) with UA rotation and increased delays
- [ ] **RECON-INFRA-07**: robots.txt respect (--respect-robots, default on)
- [x] **RECON-INFRA-07**: robots.txt respect (--respect-robots, default on)
- [ ] **RECON-INFRA-08**: Recon full command — parallel sweep across all sources with deduplication
### Dork Engine