erinhmclark
fb56aac15e
Catch edge case to ensure iterator is reached in instagram_tbot_extractor.py
2025-03-07 11:24:25 +00:00
Patrick Robertson
478f0b2171
Tidy-ups to auto-updating code
2025-03-07 09:59:18 +00:00
erinhmclark
fa1e65f54c
Fix instagram_extractor.py typo, add warning to docs, and add basic regex test.
2025-03-06 16:25:38 +00:00
erinhmclark
b9c2f98f46
Update Atlos tests
2025-03-05 21:24:38 +00:00
erinhmclark
0f911543cd
Atlos refactor
2025-03-05 13:49:11 +00:00
erinhmclark
6cb7afefdc
Initial Atlos merge
2025-03-05 10:24:54 +00:00
Patrick Robertson
358884c5d1
Fix unit tests for yt-dlp update
2025-03-04 17:04:23 +00:00
Patrick Robertson
be09aa927d
Make 'STARTED' command INFO not warning
2025-03-04 16:51:17 +00:00
Patrick Robertson
0eb112431b
Auto-update yt-dlp based on generic_extractor.ytdlp_update_interval (default=5 days)
2025-03-04 16:43:46 +00:00
erinhmclark
d1c8d4ba0e
Initial merge of Atlos Feeder and DB
2025-03-04 14:06:46 +00:00
erinhmclark
077b56c150
Merge GSheet Feeder and Database.
2025-03-04 14:05:19 +00:00
erinhmclark
7e4b44883b
Add temp options for testing
2025-03-04 14:03:39 +00:00
erinhmclark
77b517cfc1
Merge remote-tracking branch 'origin/feat/yt-dlp-pots' into feat/yt-dlp-pots
2025-03-03 22:02:14 +00:00
erinhmclark
dd07b0b830
Allow flexible extractor_args in generic_extractor.py.
2025-03-03 21:11:34 +00:00
erinhmclark
a705a78632
Fix instagram_extractor.py typo in config value.
2025-03-03 21:06:09 +00:00
Patrick Robertson
a0869bb3b2
Fixed up timestamp verifying - waiting on issue with rfc-client to be fixed
...
Ref: https://github.com/trailofbits/rfc3161-client/issues/104#issuecomment-2693890607
2025-03-03 10:28:30 +00:00
Patrick Robertson
65a9885d86
A few more manifest types
2025-02-27 21:33:04 +00:00
Patrick Robertson
1e92c03b1d
Tweaks to settings page + more declarations in manifests
2025-02-27 15:21:11 +00:00
Patrick Robertson
efe9fdf915
Tidy ups to config editor page
2025-02-27 13:02:50 +00:00
Patrick Robertson
f58f110436
Check at least 1 URL provided for new cli_feeder module rewrite
2025-02-26 17:59:13 +00:00
Patrick Robertson
70d89c71ce
Fully-working settings page editor
2025-02-26 17:02:49 +00:00
Patrick Robertson
bb961b131c
Turn cli_feeder *back* into a module, it's better like this for settings etc, documentation etc.
2025-02-26 15:41:33 +00:00
erinhmclark
8124bb831d
Merge branch 'main' into small_issues
...
# Conflicts:
# src/auto_archiver/core/base_module.py
# src/auto_archiver/utils/misc.py
2025-02-26 13:19:49 +00:00
Patrick Robertson
afc117a229
Get downloading certs working
2025-02-26 09:33:56 +00:00
erinhmclark
9bc6dd5c3c
Add set_content into generic_extractor.py.
2025-02-25 20:07:00 +00:00
erinhmclark
cf1219f798
Add text content into gsheet.
2025-02-25 20:06:44 +00:00
Patrick Robertson
4dcb77c29f
Merge branch 'main' into timestamping_rewrite
2025-02-25 17:10:55 +00:00
Patrick Robertson
898faf6fe4
Further WIP - currently working on verify_signed
2025-02-25 12:08:08 +00:00
Patrick Robertson
f8e846d59a
Create facebook dropin - working for images + text. CAVEAT: only gets the first ~100 chars of the post at the moment
2025-02-25 11:44:35 +00:00
Patrick Robertson
d10c7fbe55
Better documentation based on the discord feedbackgst
2025-02-24 22:42:57 +00:00
Patrick Robertson
01bf88a695
Merge branch 'main' into timestamping_rewrite
2025-02-24 12:03:14 +00:00
erinhmclark
c5127f5fd1
Allow flexible extractor_args in generic_extractor.py.
2025-02-24 11:40:44 +00:00
Patrick Robertson
4174285898
Fix unit tests
2025-02-20 13:18:06 +00:00
Patrick Robertson
eda359a1ef
Fix json loader - it should go in 'validators' not 'utils'
...
Fixes #214
2025-02-20 13:10:39 +00:00
Patrick Robertson
40488e0869
Use 'Auto Archiver' naming for consistency.
...
auto-archiver is reserved in the docs for when talking about the command line usage
2025-02-20 11:50:29 +00:00
Patrick Robertson
cbea551876
Better display name for wayback machine to emphasise it's typically used as an enricher
2025-02-20 11:46:57 +00:00
Patrick Robertson
b978484a89
Rename wacz_enricher to wacz_extractor_enricher. Fixes #205
2025-02-20 11:46:57 +00:00
Patrick Robertson
7dde8d609d
Merge main
2025-02-20 10:29:57 +00:00
Patrick Robertson
5211c5de18
Merge pull request #210 from bellingcat/logger_fix
...
Fix issue #200 + Refactor _LAZY_LOADED_MODULES
2025-02-19 15:11:42 +00:00
Patrick Robertson
a9802dd004
Remove the global _LAZY_LOADED_MODULES and allow each instance of ArchivingOrchestrator to load its own modules
2025-02-19 12:25:35 +00:00
erinhmclark
a8ffb19325
Fix auth key name for cookies_from_browser.
2025-02-19 10:40:54 +00:00
Patrick Robertson
222a94563f
WIP: Docs tidyups+add howto on logging and authentication
...
(Authentication is WIP)
2025-02-19 10:37:04 +00:00
erinhmclark
ddf2e76624
Include Atlos Storage __init__.py for module recognition.
2025-02-19 09:24:34 +00:00
erinhmclark
10a5ad62b8
Include Atlos tests, metadata fixture.
2025-02-19 09:18:41 +00:00
erinhmclark
657fbd357d
Merge branch 'main' into tests/add_module_tests
2025-02-18 19:47:47 +00:00
Patrick Robertson
3c543a3a6a
Various fixes for issues with new architecture ( #208 )
...
* Add formatters to the TOC - fixes #204
* Add 'steps' settings to the example YAML in the docs. Fixes #206
* Improved docs on authentication architecture
* Fix setting modules on the command line - they now override any module settings in the orchestration as opposed to appending
* Fix tests for gsheet-feeder: add a test service_account.json (note: not real keys in there)
* Rename the command line entrypoint to _command_line_run
Also: make it clear that code implementation should not call this
Make sure the command line entry returns (we don't want a generator)
* Fix unit tests to use now code-entry points
* Version bump
* Move iterating of generator up to __main__
* Breakpoint
* two minor fixes
* Fix unit tests + add new '__main__' entry point implementation test
* Skip youtube tests if running on CI. Should still run them locally
* Fix full implementation run on GH actions
* Fix skipif test for GH Actions CI
* Add skipifs for truth - it blocks GH:
---------
Co-authored-by: msramalho <19508417+msramalho@users.noreply.github.com >
2025-02-18 19:10:09 +00:00
erinhmclark
ce5a200d1f
Added tests, updated instagram_tbot_extractor.py raise failure.
2025-02-18 12:59:10 +00:00
erinhmclark
319c1e8f92
Add more tests.
2025-02-14 09:48:37 +00:00
erinhmclark
3fce593aad
Merge branch 'main' into tests/add_module_tests
2025-02-12 19:33:29 +00:00
erinhmclark
cbe98c729d
Enricher tests
2025-02-12 19:32:40 +00:00
