docs(15-01): complete forum/discussion sources plan

- SUMMARY.md with 6 sources, 2 tasks, 13 files
- STATE.md advanced, ROADMAP.md updated, requirements marked
This commit is contained in:
salvacybersec
2026-04-06 16:30:49 +03:00
parent fcc1a769c5
commit 77a2a0b531
4 changed files with 134 additions and 14 deletions

View File

@@ -0,0 +1,118 @@
---
phase: 15-osint_forums_collaboration_log_aggregators
plan: 01
subsystem: recon
tags: [stackoverflow, reddit, hackernews, discord, slack, devto, osint, forums]
requires:
- phase: 10-osint-code-hosting
provides: "ReconSource interface, Client, BuildQueries, ciLogKeyPattern, RegisterAll"
provides:
- "StackOverflowSource searching SE API v2.3 for leaked keys"
- "RedditSource searching Reddit JSON API for leaked keys"
- "HackerNewsSource searching Algolia HN API for leaked keys"
- "DiscordSource using dorking for indexed Discord content"
- "SlackSource using dorking for indexed Slack archives"
- "DevToSource searching dev.to API articles for leaked keys"
affects: [recon-engine, register-all, phase-15-plans]
tech-stack:
added: []
patterns: [dorking-based-search-for-closed-platforms]
key-files:
created:
- pkg/recon/sources/stackoverflow.go
- pkg/recon/sources/stackoverflow_test.go
- pkg/recon/sources/reddit.go
- pkg/recon/sources/reddit_test.go
- pkg/recon/sources/hackernews.go
- pkg/recon/sources/hackernews_test.go
- pkg/recon/sources/discord.go
- pkg/recon/sources/discord_test.go
- pkg/recon/sources/slack.go
- pkg/recon/sources/slack_test.go
- pkg/recon/sources/devto.go
- pkg/recon/sources/devto_test.go
modified:
- pkg/recon/sources/register.go
key-decisions:
- "Discord and Slack use dorking approach (configurable search endpoint) since neither has public message search API"
- "DevTo fetches article list then detail endpoint for body_markdown, limited to first 5 articles per keyword"
- "Reddit sets custom User-Agent to avoid blocking by Reddit's default UA filter"
patterns-established:
- "Dorking pattern: for platforms without public search APIs, use configurable search endpoint with site: prefix queries"
requirements-completed: [RECON-FORUM-01, RECON-FORUM-02, RECON-FORUM-03, RECON-FORUM-04, RECON-FORUM-05, RECON-FORUM-06]
duration: 3min
completed: 2026-04-06
---
# Phase 15 Plan 01: Forum/Discussion Sources Summary
**Six forum ReconSources (StackOverflow, Reddit, HackerNews, Discord, Slack, DevTo) scanning developer discussions for leaked API keys**
## Performance
- **Duration:** 3 min
- **Started:** 2026-04-06T13:27:19Z
- **Completed:** 2026-04-06T13:30:02Z
- **Tasks:** 2
- **Files modified:** 13
## Accomplishments
- Three API-based sources (StackOverflow SE API, Reddit JSON, HackerNews Algolia) for direct forum search
- Two dorking-based sources (Discord, Slack) for platforms without public search APIs
- DevTo two-phase search (article list + detail fetch) with rate limit protection
- RegisterAll extended with all 6 new forum sources
## Task Commits
Each task was committed atomically:
1. **Task 1: StackOverflow, Reddit, HackerNews sources** - `282c145` (feat)
2. **Task 2: Discord, Slack, DevTo sources + RegisterAll wiring** - `fcc1a76` (feat)
## Files Created/Modified
- `pkg/recon/sources/stackoverflow.go` - SE API v2.3 search/excerpts source
- `pkg/recon/sources/stackoverflow_test.go` - httptest mock tests
- `pkg/recon/sources/reddit.go` - Reddit JSON API search source with custom UA
- `pkg/recon/sources/reddit_test.go` - httptest mock tests
- `pkg/recon/sources/hackernews.go` - Algolia HN Search API source
- `pkg/recon/sources/hackernews_test.go` - httptest mock tests
- `pkg/recon/sources/discord.go` - Dorking-based Discord content search
- `pkg/recon/sources/discord_test.go` - httptest mock tests
- `pkg/recon/sources/slack.go` - Dorking-based Slack archive search
- `pkg/recon/sources/slack_test.go` - httptest mock tests
- `pkg/recon/sources/devto.go` - dev.to API article list + detail search
- `pkg/recon/sources/devto_test.go` - httptest mock tests with list+detail endpoints
- `pkg/recon/sources/register.go` - Extended RegisterAll with 6 forum sources
## Decisions Made
- Discord and Slack use configurable search endpoint dorking since neither platform has public message search APIs
- DevTo limits to first 5 articles per keyword to stay within rate limits
- Reddit requires custom User-Agent header to avoid 429 blocking
- Discord/Slack findings marked as "low" confidence (indirect via search indexers); API-based sources marked "medium"
## Deviations from Plan
None - plan executed exactly as written.
## Issues Encountered
None.
## User Setup Required
None - all six sources are credentialless and always enabled.
## Next Phase Readiness
- All forum/discussion sources registered in RegisterAll
- Ready for Phase 15 Plan 02+ (collaboration tools, log aggregators)
---
*Phase: 15-osint_forums_collaboration_log_aggregators*
*Completed: 2026-04-06*