Patrick Robertson
b8da7607e8
Merge branch 'main' into opentimestamps
2025-03-14 12:36:03 +00:00
erinhmclark
8673bc5979
Fix unused imports and include rule.
2025-03-13 13:55:31 +00:00
erinhmclark
e76551ba22
Add documentation, pre-commit hook, more make commands and
2025-03-13 13:21:32 +00:00
erinhmclark
6e52a534e7
More fixes from Bugbear suggestions
2025-03-12 16:07:05 +00:00
Patrick Robertson
1423c10363
Finish off timestamping module
2025-03-12 10:24:57 +00:00
erinhmclark
8ca7698fa0
Move Makefile and fix import error with unused import.
2025-03-11 19:58:02 +00:00
erinhmclark
81aa343f21
Merge main.
2025-03-11 10:45:07 +00:00
erinhmclark
441f341139
Merge branch 'main' into linting_etc
...
# Conflicts:
# src/auto_archiver/core/consts.py
# src/auto_archiver/core/orchestrator.py
# src/auto_archiver/core/storage.py
# src/auto_archiver/modules/local_storage/local_storage.py
# src/auto_archiver/modules/s3_storage/s3_storage.py
# tests/storages/test_S3_storage.py
# tests/storages/test_local_storage.py
# tests/storages/test_storage_base.py
2025-03-11 10:39:47 +00:00
erinhmclark
e7fa88f1c7
Implementing ruff suggestions.
2025-03-10 21:45:30 +00:00
erinhmclark
ca44a40b88
Ruff fix on src.
2025-03-10 19:03:45 +00:00
erinhmclark
85abe1837a
Ruff format with defaults.
2025-03-10 18:44:54 +00:00
Patrick Robertson
a9c3477289
Improve docs on the path_generator and filename_generator config options
2025-03-10 16:43:14 +00:00
Patrick Robertson
770f4c8a3d
Refactoring of storage code:
...
1. Fix some bugs in local_storage
2. Refactor unit tests to not set Media.key explicitly (unless it's well-known beforehand, which it isn't)
3. Limit length of URL for 'url' type path_generator
4. Throw an error if 'save_to' of local storage is too long
5. A few other tidyups
2025-03-10 16:39:48 +00:00
Patrick Robertson
e89a8da3b4
Unit tests for storage types + fix storage too long issues for local storage
2025-03-10 11:30:15 +00:00
Patrick Robertson
3fac353407
Merge pull request #217 from bellingcat/settings_page
...
Settings page user interface
2025-03-07 16:10:50 +00:00
Erin Clark
85a75755e2
Merge pull request #236 from bellingcat/cleanup_fixes
...
Cleanup fixes
2025-03-07 15:37:05 +00:00
Patrick Robertson
333201acec
Merge branch 'main' into settings_page
2025-03-07 15:17:42 +00:00
Patrick Robertson
027985024b
Merge pull request #234 from bellingcat/update_suggestions
...
Auto Updates
2025-03-07 15:12:03 +00:00
erinhmclark
bdd35408ce
Fix ref before assignment in orchestrator.py
2025-03-07 11:23:51 +00:00
Patrick Robertson
478f0b2171
Tidy-ups to auto-updating code
2025-03-07 09:59:18 +00:00
Patrick Robertson
e6a578e60e
Check for auto-archiver updates and present warning if there's a newer version available
2025-03-04 16:51:17 +00:00
Patrick Robertson
0b5a0fcb32
Better error logs if users have XXXX_archiver modules enabled in config
2025-03-03 19:57:09 +00:00
Patrick Robertson
1fe023cd70
Throw a nicer error if a user has an orchestration.yaml file in the old format (feeder: / archivers: / formatter: )
2025-03-03 19:51:55 +00:00
Patrick Robertson
dea0a49600
Download correct gecko-driver for the platform + fix setting executable path when running in Docker
...
Fixes #232
2025-03-03 15:41:44 +00:00
Patrick Robertson
bb961b131c
Turn cli_feeder *back* into a module, it's better like this for settings etc, documentation etc.
2025-02-26 15:41:33 +00:00
erinhmclark
8124bb831d
Merge branch 'main' into small_issues
...
# Conflicts:
# src/auto_archiver/core/base_module.py
# src/auto_archiver/utils/misc.py
2025-02-26 13:19:49 +00:00
erinhmclark
1df5129268
Small typos.
2025-02-25 14:08:38 +00:00
Patrick Robertson
ca1ed418aa
Throw an error for invalid __manifest__ syntax + fix: allow default values of False/None
2025-02-24 21:46:24 +00:00
Patrick Robertson
091a19e25c
Further docs improvements/tidy ups
2025-02-21 16:52:30 +00:00
Patrick Robertson
9661e90a05
Allow disabling logging in auto_archiver with logging: enabled: false
2025-02-20 15:45:32 +00:00
Patrick Robertson
0bec71d203
Finish how to on authentication
2025-02-20 15:33:50 +00:00
Patrick Robertson
eda359a1ef
Fix json loader - it should go in 'validators' not 'utils'
...
Fixes #214
2025-02-20 13:10:39 +00:00
Patrick Robertson
40488e0869
Use 'Auto Archiver' naming for consistency.
...
auto-archiver is reserved in the docs for when talking about the command line usage
2025-02-20 11:50:29 +00:00
Patrick Robertson
49b6c32058
Fix the 'full' mode which creates a complete config file
2025-02-20 11:34:05 +00:00
Patrick Robertson
4b51ec9ad5
Remove dangling import
2025-02-20 11:20:16 +00:00
Patrick Robertson
7734a551fa
Move 'assert_valid_url' out into utils, don't use assert but raise
...
assert is recommended only for debugging
2025-02-20 11:19:29 +00:00
Patrick Robertson
77b2b099c6
Replace exit() with raise exceptions. Better for code implementations
...
exit() is reserved solely for command line-called areas now
also assert is only recommended for debugging
2025-02-20 11:19:13 +00:00
Patrick Robertson
7dde8d609d
Merge main
2025-02-20 10:29:57 +00:00
Patrick Robertson
a9802dd004
Remove the global _LAZY_LOADED_MODULES and allow each instance of ArchivingOrchestrator to load its own modules
2025-02-19 12:25:35 +00:00
Patrick Robertson
222a94563f
WIP: Docs tidyups+add howto on logging and authentication
...
(Authentication is WIP)
2025-02-19 10:37:04 +00:00
Patrick Robertson
eb60b271b9
Fix issue #200
2025-02-19 10:35:14 +00:00
Patrick Robertson
3c543a3a6a
Various fixes for issues with new architecture ( #208 )
...
* Add formatters to the TOC - fixes #204
* Add 'steps' settings to the example YAML in the docs. Fixes #206
* Improved docs on authentication architecture
* Fix setting modules on the command line - they now override any module settings in the orchestration as opposed to appending
* Fix tests for gsheet-feeder: add a test service_account.json (note: not real keys in there)
* Rename the command line entrypoint to _command_line_run
Also: make it clear that code implementation should not call this
Make sure the command line entry returns (we don't want a generator)
* Fix unit tests to use now code-entry points
* Version bump
* Move iterating of generator up to __main__
* Breakpoint
* two minor fixes
* Fix unit tests + add new '__main__' entry point implementation test
* Skip youtube tests if running on CI. Should still run them locally
* Fix full implementation run on GH actions
* Fix skipif test for GH Actions CI
* Add skipifs for truth - it blocks GH:
---------
Co-authored-by: msramalho <19508417+msramalho@users.noreply.github.com >
2025-02-18 19:10:09 +00:00
Patrick Robertson
6d43bc7d4d
Fix generator programmatic setup ( #197 )
...
* Fix returning a generator of a generator
* Move download test test to pytest.mark.download
2025-02-15 17:36:44 +00:00
Miguel Sozinho Ramalho
9297697ef5
makes orchestrator.run return the results to allow for code integration ( #196 )
2025-02-15 12:41:26 +00:00
Patrick Robertson
460a71649c
Merge pull request #190 from bellingcat/docs_update
...
Docs improvement
2025-02-12 12:38:04 +01:00
Patrick Robertson
a0c4a82825
Improved docstrings for base modules
2025-02-12 11:32:13 +00:00
msramalho
e507fc81d2
improves mimetype guessing, previously file.sub.something would not have an extension
2025-02-11 15:02:49 +00:00
Patrick Robertson
29901da601
Merge branch 'load_modules' into docs_update
2025-02-11 14:10:56 +00:00
Patrick Robertson
2f51d3917a
Further addition to docs: creating modules, configurations, installation
2025-02-11 13:49:30 +00:00
erinhmclark
c8cd7ea63c
Merge branch 'load_modules' into add_module_tests
...
# Conflicts:
# src/auto_archiver/modules/telethon_extractor/telethon_extractor.py
2025-02-11 13:08:08 +00:00
