121 lines
4.4 KiB
Markdown
121 lines
4.4 KiB
Markdown
---
|
|
phase: 10-osint-code-hosting
|
|
plan: 03
|
|
type: execute
|
|
wave: 2
|
|
depends_on: [10-01]
|
|
files_modified:
|
|
- pkg/recon/sources/gitlab.go
|
|
- pkg/recon/sources/gitlab_test.go
|
|
autonomous: true
|
|
requirements: [RECON-CODE-02]
|
|
must_haves:
|
|
truths:
|
|
- "GitLabSource.Sweep queries GitLab /api/v4/search?scope=blobs and emits Findings"
|
|
- "Disabled when token empty; enabled otherwise"
|
|
- "Findings have SourceType=\"recon:gitlab\" and Source = web_url of blob"
|
|
artifacts:
|
|
- path: "pkg/recon/sources/gitlab.go"
|
|
provides: "GitLabSource implementing recon.ReconSource"
|
|
- path: "pkg/recon/sources/gitlab_test.go"
|
|
provides: "httptest tests"
|
|
key_links:
|
|
- from: "pkg/recon/sources/gitlab.go"
|
|
to: "pkg/recon/sources/httpclient.go"
|
|
via: "c.client.Do(ctx, req)"
|
|
pattern: "client\\.Do"
|
|
---
|
|
|
|
<objective>
|
|
Implement GitLabSource against GitLab's Search API (/api/v4/search?scope=blobs).
|
|
Honors PRIVATE-TOKEN header auth, 2000 req/min rate limit.
|
|
|
|
Purpose: RECON-CODE-02.
|
|
Output: pkg/recon/sources/gitlab.go + tests.
|
|
</objective>
|
|
|
|
<execution_context>
|
|
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
|
@$HOME/.claude/get-shit-done/templates/summary.md
|
|
</execution_context>
|
|
|
|
<context>
|
|
@.planning/phases/10-osint-code-hosting/10-CONTEXT.md
|
|
@.planning/phases/10-osint-code-hosting/10-01-SUMMARY.md
|
|
@pkg/recon/source.go
|
|
@pkg/recon/sources/httpclient.go
|
|
@pkg/recon/sources/queries.go
|
|
|
|
<interfaces>
|
|
GitLab Search API (docs: https://docs.gitlab.com/ee/api/search.html):
|
|
GET /api/v4/search?scope=blobs&search=<query>&per_page=20
|
|
Header: PRIVATE-TOKEN: <token>
|
|
Response (array of blob objects):
|
|
[{ "basename": "...", "data": "matched snippet", "path": "...", "project_id": 123,
|
|
"ref": "main", "startline": 42 }, ...]
|
|
Project web_url must be constructed from project_id → fetch /api/v4/projects/<id> (or
|
|
just use basename+path with a placeholder Source — keep it minimal: Source =
|
|
"https://gitlab.com/projects/<project_id>/-/blob/<ref>/<path>").
|
|
|
|
Rate limit: 2000 req/min → rate.Every(30 * time.Millisecond) ≈ 2000/min, burst 5.
|
|
</interfaces>
|
|
</context>
|
|
|
|
<tasks>
|
|
|
|
<task type="auto" tdd="true">
|
|
<name>Task 1: GitLabSource implementation + tests</name>
|
|
<files>pkg/recon/sources/gitlab.go, pkg/recon/sources/gitlab_test.go</files>
|
|
<behavior>
|
|
- Test A: Enabled false when token empty
|
|
- Test B: Sweep queries /api/v4/search with scope=blobs, PRIVATE-TOKEN header set
|
|
- Test C: Decodes array response, emits one Finding per blob with Source containing project_id + path + ref
|
|
- Test D: 401 returns wrapped ErrUnauthorized
|
|
- Test E: Ctx cancellation respected
|
|
- Test F: Empty token → Sweep returns nil with no calls
|
|
</behavior>
|
|
<action>
|
|
Create `pkg/recon/sources/gitlab.go` with struct `GitLabSource { Token, BaseURL string; Registry *providers.Registry; Limiters *recon.LimiterRegistry; client *Client }`.
|
|
|
|
Default BaseURL: `https://gitlab.com`.
|
|
Name: "gitlab". RateLimit: `rate.Every(30 * time.Millisecond)`. Burst: 5. RespectsRobots: false.
|
|
|
|
Sweep loop:
|
|
- For each query from BuildQueries(reg, "gitlab"):
|
|
- Build `base + /api/v4/search?scope=blobs&search=<url-escaped>&per_page=20`
|
|
- Set header `PRIVATE-TOKEN: <token>`
|
|
- limiters.Wait, then client.Do
|
|
- Decode `[]glBlob` where glBlob has ProjectID int, Path, Ref, Data, Startline
|
|
- Emit Finding with Source = fmt.Sprintf("%s/projects/%d/-/blob/%s/%s", base, b.ProjectID, b.Ref, b.Path), SourceType="recon:gitlab", Confidence="low", ProviderName derived via keywordIndex(reg)
|
|
- Respect ctx.Done on send
|
|
|
|
Add compile-time assert: `var _ recon.ReconSource = (*GitLabSource)(nil)`.
|
|
|
|
Create `pkg/recon/sources/gitlab_test.go` with httptest server returning a JSON
|
|
array of two blob objects. Assert both Findings received, Source URLs contain
|
|
project IDs, ctx cancellation test, 401 test, empty-token test. Use synthetic
|
|
registry with 2 providers.
|
|
</action>
|
|
<verify>
|
|
<automated>cd /home/salva/Documents/apikey && go test ./pkg/recon/sources/ -run TestGitLab -v -timeout 30s</automated>
|
|
</verify>
|
|
<done>
|
|
GitLabSource compiles, implements ReconSource, all test behaviors covered.
|
|
</done>
|
|
</task>
|
|
|
|
</tasks>
|
|
|
|
<verification>
|
|
- `go build ./...`
|
|
- `go test ./pkg/recon/sources/ -run TestGitLab -v`
|
|
</verification>
|
|
|
|
<success_criteria>
|
|
RECON-CODE-02 satisfied.
|
|
</success_criteria>
|
|
|
|
<output>
|
|
After completion, create `.planning/phases/10-osint-code-hosting/10-03-SUMMARY.md`.
|
|
</output>
|