docs(15-01): complete forum/discussion sources plan
- SUMMARY.md with 6 sources, 2 tasks, 13 files - STATE.md advanced, ROADMAP.md updated, requirements marked
This commit is contained in:
@@ -0,0 +1,118 @@
|
||||
---
|
||||
phase: 15-osint_forums_collaboration_log_aggregators
|
||||
plan: 01
|
||||
subsystem: recon
|
||||
tags: [stackoverflow, reddit, hackernews, discord, slack, devto, osint, forums]
|
||||
|
||||
requires:
|
||||
- phase: 10-osint-code-hosting
|
||||
provides: "ReconSource interface, Client, BuildQueries, ciLogKeyPattern, RegisterAll"
|
||||
provides:
|
||||
- "StackOverflowSource searching SE API v2.3 for leaked keys"
|
||||
- "RedditSource searching Reddit JSON API for leaked keys"
|
||||
- "HackerNewsSource searching Algolia HN API for leaked keys"
|
||||
- "DiscordSource using dorking for indexed Discord content"
|
||||
- "SlackSource using dorking for indexed Slack archives"
|
||||
- "DevToSource searching dev.to API articles for leaked keys"
|
||||
affects: [recon-engine, register-all, phase-15-plans]
|
||||
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns: [dorking-based-search-for-closed-platforms]
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- pkg/recon/sources/stackoverflow.go
|
||||
- pkg/recon/sources/stackoverflow_test.go
|
||||
- pkg/recon/sources/reddit.go
|
||||
- pkg/recon/sources/reddit_test.go
|
||||
- pkg/recon/sources/hackernews.go
|
||||
- pkg/recon/sources/hackernews_test.go
|
||||
- pkg/recon/sources/discord.go
|
||||
- pkg/recon/sources/discord_test.go
|
||||
- pkg/recon/sources/slack.go
|
||||
- pkg/recon/sources/slack_test.go
|
||||
- pkg/recon/sources/devto.go
|
||||
- pkg/recon/sources/devto_test.go
|
||||
modified:
|
||||
- pkg/recon/sources/register.go
|
||||
|
||||
key-decisions:
|
||||
- "Discord and Slack use dorking approach (configurable search endpoint) since neither has public message search API"
|
||||
- "DevTo fetches article list then detail endpoint for body_markdown, limited to first 5 articles per keyword"
|
||||
- "Reddit sets custom User-Agent to avoid blocking by Reddit's default UA filter"
|
||||
|
||||
patterns-established:
|
||||
- "Dorking pattern: for platforms without public search APIs, use configurable search endpoint with site: prefix queries"
|
||||
|
||||
requirements-completed: [RECON-FORUM-01, RECON-FORUM-02, RECON-FORUM-03, RECON-FORUM-04, RECON-FORUM-05, RECON-FORUM-06]
|
||||
|
||||
duration: 3min
|
||||
completed: 2026-04-06
|
||||
---
|
||||
|
||||
# Phase 15 Plan 01: Forum/Discussion Sources Summary
|
||||
|
||||
**Six forum ReconSources (StackOverflow, Reddit, HackerNews, Discord, Slack, DevTo) scanning developer discussions for leaked API keys**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** 3 min
|
||||
- **Started:** 2026-04-06T13:27:19Z
|
||||
- **Completed:** 2026-04-06T13:30:02Z
|
||||
- **Tasks:** 2
|
||||
- **Files modified:** 13
|
||||
|
||||
## Accomplishments
|
||||
- Three API-based sources (StackOverflow SE API, Reddit JSON, HackerNews Algolia) for direct forum search
|
||||
- Two dorking-based sources (Discord, Slack) for platforms without public search APIs
|
||||
- DevTo two-phase search (article list + detail fetch) with rate limit protection
|
||||
- RegisterAll extended with all 6 new forum sources
|
||||
|
||||
## Task Commits
|
||||
|
||||
Each task was committed atomically:
|
||||
|
||||
1. **Task 1: StackOverflow, Reddit, HackerNews sources** - `282c145` (feat)
|
||||
2. **Task 2: Discord, Slack, DevTo sources + RegisterAll wiring** - `fcc1a76` (feat)
|
||||
|
||||
## Files Created/Modified
|
||||
- `pkg/recon/sources/stackoverflow.go` - SE API v2.3 search/excerpts source
|
||||
- `pkg/recon/sources/stackoverflow_test.go` - httptest mock tests
|
||||
- `pkg/recon/sources/reddit.go` - Reddit JSON API search source with custom UA
|
||||
- `pkg/recon/sources/reddit_test.go` - httptest mock tests
|
||||
- `pkg/recon/sources/hackernews.go` - Algolia HN Search API source
|
||||
- `pkg/recon/sources/hackernews_test.go` - httptest mock tests
|
||||
- `pkg/recon/sources/discord.go` - Dorking-based Discord content search
|
||||
- `pkg/recon/sources/discord_test.go` - httptest mock tests
|
||||
- `pkg/recon/sources/slack.go` - Dorking-based Slack archive search
|
||||
- `pkg/recon/sources/slack_test.go` - httptest mock tests
|
||||
- `pkg/recon/sources/devto.go` - dev.to API article list + detail search
|
||||
- `pkg/recon/sources/devto_test.go` - httptest mock tests with list+detail endpoints
|
||||
- `pkg/recon/sources/register.go` - Extended RegisterAll with 6 forum sources
|
||||
|
||||
## Decisions Made
|
||||
- Discord and Slack use configurable search endpoint dorking since neither platform has public message search APIs
|
||||
- DevTo limits to first 5 articles per keyword to stay within rate limits
|
||||
- Reddit requires custom User-Agent header to avoid 429 blocking
|
||||
- Discord/Slack findings marked as "low" confidence (indirect via search indexers); API-based sources marked "medium"
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
None - plan executed exactly as written.
|
||||
|
||||
## Issues Encountered
|
||||
|
||||
None.
|
||||
|
||||
## User Setup Required
|
||||
|
||||
None - all six sources are credentialless and always enabled.
|
||||
|
||||
## Next Phase Readiness
|
||||
- All forum/discussion sources registered in RegisterAll
|
||||
- Ready for Phase 15 Plan 02+ (collaboration tools, log aggregators)
|
||||
|
||||
---
|
||||
*Phase: 15-osint_forums_collaboration_log_aggregators*
|
||||
*Completed: 2026-04-06*
|
||||
Reference in New Issue
Block a user