Patrick Robertson
2c5e138263
Add a note on disabling the auto-update for yt-dlp
2025-03-07 11:44:24 +00:00
Patrick Robertson
478f0b2171
Tidy-ups to auto-updating code
2025-03-07 09:59:18 +00:00
Patrick Robertson
358884c5d1
Fix unit tests for yt-dlp update
2025-03-04 17:04:23 +00:00
Patrick Robertson
be09aa927d
Make 'STARTED' command INFO not warning
2025-03-04 16:51:17 +00:00
Patrick Robertson
0eb112431b
Auto-update yt-dlp based on generic_extractor.ytdlp_update_interval (default=5 days)
2025-03-04 16:43:46 +00:00
erinhmclark
8124bb831d
Merge branch 'main' into small_issues
...
# Conflicts:
# src/auto_archiver/core/base_module.py
# src/auto_archiver/utils/misc.py
2025-02-26 13:19:49 +00:00
erinhmclark
9bc6dd5c3c
Add set_content into generic_extractor.py.
2025-02-25 20:07:00 +00:00
erinhmclark
cf1219f798
Add text content into gsheet.
2025-02-25 20:06:44 +00:00
Patrick Robertson
d10c7fbe55
Better documentation based on the discord feedbackgst
2025-02-24 22:42:57 +00:00
Patrick Robertson
4174285898
Fix unit tests
2025-02-20 13:18:06 +00:00
Patrick Robertson
eda359a1ef
Fix json loader - it should go in 'validators' not 'utils'
...
Fixes #214
2025-02-20 13:10:39 +00:00
Patrick Robertson
40488e0869
Use 'Auto Archiver' naming for consistency.
...
auto-archiver is reserved in the docs for when talking about the command line usage
2025-02-20 11:50:29 +00:00
Patrick Robertson
cbea551876
Better display name for wayback machine to emphasise it's typically used as an enricher
2025-02-20 11:46:57 +00:00
Patrick Robertson
b978484a89
Rename wacz_enricher to wacz_extractor_enricher. Fixes #205
2025-02-20 11:46:57 +00:00
Patrick Robertson
7dde8d609d
Merge main
2025-02-20 10:29:57 +00:00
Patrick Robertson
5211c5de18
Merge pull request #210 from bellingcat/logger_fix
...
Fix issue #200 + Refactor _LAZY_LOADED_MODULES
2025-02-19 15:11:42 +00:00
Patrick Robertson
a9802dd004
Remove the global _LAZY_LOADED_MODULES and allow each instance of ArchivingOrchestrator to load its own modules
2025-02-19 12:25:35 +00:00
erinhmclark
a8ffb19325
Fix auth key name for cookies_from_browser.
2025-02-19 10:40:54 +00:00
Patrick Robertson
222a94563f
WIP: Docs tidyups+add howto on logging and authentication
...
(Authentication is WIP)
2025-02-19 10:37:04 +00:00
erinhmclark
ddf2e76624
Include Atlos Storage __init__.py for module recognition.
2025-02-19 09:24:34 +00:00
erinhmclark
10a5ad62b8
Include Atlos tests, metadata fixture.
2025-02-19 09:18:41 +00:00
erinhmclark
657fbd357d
Merge branch 'main' into tests/add_module_tests
2025-02-18 19:47:47 +00:00
Patrick Robertson
3c543a3a6a
Various fixes for issues with new architecture ( #208 )
...
* Add formatters to the TOC - fixes #204
* Add 'steps' settings to the example YAML in the docs. Fixes #206
* Improved docs on authentication architecture
* Fix setting modules on the command line - they now override any module settings in the orchestration as opposed to appending
* Fix tests for gsheet-feeder: add a test service_account.json (note: not real keys in there)
* Rename the command line entrypoint to _command_line_run
Also: make it clear that code implementation should not call this
Make sure the command line entry returns (we don't want a generator)
* Fix unit tests to use now code-entry points
* Version bump
* Move iterating of generator up to __main__
* Breakpoint
* two minor fixes
* Fix unit tests + add new '__main__' entry point implementation test
* Skip youtube tests if running on CI. Should still run them locally
* Fix full implementation run on GH actions
* Fix skipif test for GH Actions CI
* Add skipifs for truth - it blocks GH:
---------
Co-authored-by: msramalho <19508417+msramalho@users.noreply.github.com >
2025-02-18 19:10:09 +00:00
erinhmclark
ce5a200d1f
Added tests, updated instagram_tbot_extractor.py raise failure.
2025-02-18 12:59:10 +00:00
erinhmclark
319c1e8f92
Add more tests.
2025-02-14 09:48:37 +00:00
erinhmclark
3fce593aad
Merge branch 'main' into tests/add_module_tests
2025-02-12 19:33:29 +00:00
erinhmclark
cbe98c729d
Enricher tests
2025-02-12 19:32:40 +00:00
erinhmclark
d9d936c2ca
Thumbnail enricher fix seconds to minutes.
2025-02-12 12:22:27 +00:00
Patrick Robertson
460a71649c
Merge pull request #190 from bellingcat/docs_update
...
Docs improvement
2025-02-12 12:38:04 +01:00
Patrick Robertson
5b481f72ab
Tidy ups to manifests for docs
2025-02-11 20:03:53 +00:00
Patrick Robertson
4c119b4db8
Add missing manifest for atlos_storage
2025-02-11 20:03:45 +00:00
Patrick Robertson
1ee7981c6e
Add YAML config to the module docs
2025-02-11 19:42:03 +00:00
Patrick Robertson
1b976f4c09
Remove unused atlos util functions
2025-02-11 18:49:54 +00:00
Patrick Robertson
3787577a96
Screenshot enricher depends on geckodriver not chromedriver
2025-02-11 18:18:52 +00:00
Patrick Robertson
ea728a7a97
TODO on facebook dropin not working
2025-02-11 15:56:12 +00:00
msramalho
91f1ebf7b3
fix temp for yandex new shortlink
2025-02-11 15:23:16 +00:00
msramalho
c720541de2
merge conflicts
2025-02-11 15:22:06 +00:00
msramalho
e507fc81d2
improves mimetype guessing, previously file.sub.something would not have an extension
2025-02-11 15:02:49 +00:00
msramalho
5478ed3860
bsky fix media fetching
2025-02-11 15:02:00 +00:00
msramalho
47d1dc9d47
typing warnings fixed
2025-02-11 15:01:37 +00:00
Patrick Robertson
29901da601
Merge branch 'load_modules' into docs_update
2025-02-11 14:10:56 +00:00
Patrick Robertson
7d87b858d6
Merge branch 'load_modules' into docs_update
2025-02-11 13:09:38 +00:00
erinhmclark
c8cd7ea63c
Merge branch 'load_modules' into add_module_tests
...
# Conflicts:
# src/auto_archiver/modules/telethon_extractor/telethon_extractor.py
2025-02-11 13:08:08 +00:00
msramalho
977618b4ce
doc: adds note about telethon vs telegram extractors
2025-02-11 13:04:59 +00:00
msramalho
d90d3cec28
fix telethon_extractor setup
2025-02-11 13:03:18 +00:00
msramalho
977f06c37a
renames api_db property for clarity
2025-02-11 12:56:33 +00:00
msramalho
5c59029221
updates api_db for new API endpoint
2025-02-11 12:53:58 +00:00
msramalho
4eeb39477c
improves gsheetdb feedback on retrieve sheet failure
2025-02-11 12:53:46 +00:00
msramalho
6fdd5f0e66
fix cases of single : vs :: in entrypoint
2025-02-11 12:53:12 +00:00
Patrick Robertson
2650cd8fb2
Use a script to auto-generate documentation for the core modules from the manifest file
2025-02-10 22:51:04 +00:00
