salvacybersec/anything-llm-rag
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
8 Commits 1 Branch 0 Tags
3176ebf102c64234acb677ba80fecf92b6884086
Commit Graph

8 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
salvacybersec
3176ebf102 Fix batch size (20→5) and script detection in monitor 
- Reduce embed batch to 5 — AnythingLLM hangs on batches >10
- Fix check_script_running() to properly detect setup.py process
  (was returning false because pgrep matched monitor.py too)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-07 00:33:35 +03:00
salvacybersec
1028d11507 Add structured logging + log panel to monitor 
- setup.py: logging module with file (setup.log) + console output
  - Line-buffered output (fixes background execution buffering)
  - API calls with timeout (300s), retry (3x), debug logging
  - Per-batch progress: [1/29] persona batch 1/20 (20 docs)
  - --verbose flag for debug-level console
- monitor.py: log tail in CLI + web dashboard
  - CLI: colorized last 15 log lines
  - Web: scrollable log panel with level-based colors
- Smaller embed batches (20 instead of 50) for reliability

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-07 00:30:29 +03:00
salvacybersec
9105c03b4b Add monitor.py: CLI + web dashboard for embedding progress 
Three modes:
  python3 monitor.py          # one-shot CLI
  python3 monitor.py --watch  # auto-refresh 2s
  python3 monitor.py --web    # web dashboard on :8899

Shows per-persona progress bars, vector sizes, API/script status,
cluster grouping with color coding. Web mode auto-polls /api/status.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-07 00:24:07 +03:00
salvacybersec
98ed69653d Update all docs: 29 personas, 88 paths, 39K files, --reassign mode 
Sync README, skill, memory, and Obsidian note with current state:
- 29 persona workspaces across 5 clusters
- 88 mapped paths covering 39,754 files (67 GB)
- New --reassign --reset mode for fast vector recovery
- Expanded skip_extensions list
- Gitea repo reference added

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-07 00:19:27 +03:00
salvacybersec
e54ed045fe Map remaining 4 files: _oneshots + KonferansSlaytlari to Neo 
Full library coverage complete — 0 unmapped content folders.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-07 00:15:34 +03:00
salvacybersec
24f22b5b6c Expand config coverage from 73 to 86 mapped paths across 39K files 
- Add SiyasetVeTeori (1262 files) to Tribune + Sage personas
- Add Marketing (12 files) to Herald persona
- Add 7 regional FOIA-CIA folders to Frodo + Scribe
- Add NATO/FOIA-NATO to Scribe + Warden
- Add MobilGuvenlik to Neo + Specter, KonferansSunumlari to Neo
- Add HHSGuvenlikEgitimi to Bastion
- Expand skip_extensions: add .djvu .mobi .azw3 .json .log .py .js
  .jsx .htm .png .gif .mp3 .flac .gov .org .db

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-07 00:13:30 +03:00
salvacybersec
c45efcb261 Add --reassign mode for fast vector recovery without disk scanning 
Skips the slow folder scan (50K+ files) and upload phases — directly
re-embeds already-uploaded documents to workspaces using progress state.
Use with --reset to clear assignment tracking first.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-07 00:04:36 +03:00
salvacybersec
9e9b75e0b3 Initial commit: AnythingLLM persona RAG integration 
28 persona workspace with document upload, OCR pipeline, and vector embedding
assignment via AnythingLLM API. Supports 5 clusters (intel, cyber, military,
humanities, engineering) with batch processing and resume capability.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-06 23:07:44 +03:00