erinhmclark
0bef78b0b4
Remove autouse property of mock_sleep.
2025-03-13 15:23:35 +00:00
erinhmclark
16012df30b
Revert exception check in test.
2025-03-13 13:57:04 +00:00
erinhmclark
8673bc5979
Fix unused imports and include rule.
2025-03-13 13:55:31 +00:00
erinhmclark
e76551ba22
Add documentation, pre-commit hook, more make commands and
2025-03-13 13:21:32 +00:00
erinhmclark
753c3c6214
Linting tests
2025-03-12 14:27:45 +00:00
erinhmclark
79f576be1d
Run fix on tests.
2025-03-12 10:38:16 +00:00
erinhmclark
8ca7698fa0
Move Makefile and fix import error with unused import.
2025-03-11 19:58:02 +00:00
erinhmclark
441f341139
Merge branch 'main' into linting_etc
...
# Conflicts:
# src/auto_archiver/core/consts.py
# src/auto_archiver/core/orchestrator.py
# src/auto_archiver/core/storage.py
# src/auto_archiver/modules/local_storage/local_storage.py
# src/auto_archiver/modules/s3_storage/s3_storage.py
# tests/storages/test_S3_storage.py
# tests/storages/test_local_storage.py
# tests/storages/test_storage_base.py
2025-03-11 10:39:47 +00:00
Patrick Robertson
e2442b2f6b
Merge pull request #243 from bellingcat/fix-long-path-names
...
Unit tests for storage types + fix storage too long issues for local storage
2025-03-11 10:05:09 +00:00
erinhmclark
85abe1837a
Ruff format with defaults.
2025-03-10 18:44:54 +00:00
Patrick Robertson
2b91dc9514
Fix up unit tests
2025-03-10 16:51:16 +00:00
Patrick Robertson
770f4c8a3d
Refactoring of storage code:
...
1. Fix some bugs in local_storage
2. Refactor unit tests to not set Media.key explicitly (unless it's well-known beforehand, which it isn't)
3. Limit length of URL for 'url' type path_generator
4. Throw an error if 'save_to' of local storage is too long
5. A few other tidyups
2025-03-10 16:39:48 +00:00
Miguel Sozinho Ramalho
58bd38e292
Adds new extractor for tiktok via unofficial API ( #237 )
...
* minor update to defaults in api_db
* readme typo
* adds and tests new tikwm tiktok downloader
* addresses PR comments
2025-03-10 11:56:45 +00:00
Patrick Robertson
e89a8da3b4
Unit tests for storage types + fix storage too long issues for local storage
2025-03-10 11:30:15 +00:00
Patrick Robertson
be513e95aa
Merge branch 'main' into merge_modules
2025-03-07 16:19:51 +00:00
erinhmclark
fa1e65f54c
Fix instagram_extractor.py typo, add warning to docs, and add basic regex test.
2025-03-06 16:25:38 +00:00
erinhmclark
b9c2f98f46
Update Atlos tests
2025-03-05 21:24:38 +00:00
erinhmclark
d775e4612e
Update gsheet_feeder references in tests.
2025-03-04 14:06:46 +00:00
erinhmclark
077b56c150
Merge GSheet Feeder and Database.
2025-03-04 14:05:19 +00:00
erinhmclark
a705a78632
Fix instagram_extractor.py typo in config value.
2025-03-03 21:06:09 +00:00
erinhmclark
4280791f07
Fix mocking in test_wayback_enricher.py.
2025-02-27 11:25:58 +00:00
erinhmclark
8124bb831d
Merge branch 'main' into small_issues
...
# Conflicts:
# src/auto_archiver/core/base_module.py
# src/auto_archiver/utils/misc.py
2025-02-26 13:19:49 +00:00
erinhmclark
b2e654aef9
Remove context manager from test_pdq_hash_enricher.py
2025-02-26 12:57:33 +00:00
erinhmclark
cf1219f798
Add text content into gsheet.
2025-02-25 20:06:44 +00:00
erinhmclark
73b434aafc
Tests for test_vk_extractor.py.
2025-02-25 14:08:28 +00:00
erinhmclark
2d276cb9c4
Fix tmp test file.
2025-02-25 14:08:14 +00:00
Patrick Robertson
73a2e2d752
Fix tests for moving orchestration to secrets/orchestration.yaml
2025-02-21 19:05:39 +00:00
Patrick Robertson
4174285898
Fix unit tests
2025-02-20 13:18:06 +00:00
Patrick Robertson
40b8359348
Implementation test with 2 x orchestrators with different configs
2025-02-20 11:18:28 +00:00
Patrick Robertson
7dde8d609d
Merge main
2025-02-20 10:29:57 +00:00
Patrick Robertson
5211c5de18
Merge pull request #210 from bellingcat/logger_fix
...
Fix issue #200 + Refactor _LAZY_LOADED_MODULES
2025-02-19 15:11:42 +00:00
erinhmclark
47a634fc63
Add WACZ, Wayback and local storage tests.
2025-02-19 13:14:08 +00:00
Patrick Robertson
a9802dd004
Remove the global _LAZY_LOADED_MODULES and allow each instance of ArchivingOrchestrator to load its own modules
2025-02-19 12:25:35 +00:00
Patrick Robertson
222a94563f
WIP: Docs tidyups+add howto on logging and authentication
...
(Authentication is WIP)
2025-02-19 10:37:04 +00:00
erinhmclark
10a5ad62b8
Include Atlos tests, metadata fixture.
2025-02-19 09:18:41 +00:00
erinhmclark
f0fd9bf445
Updates tests to use pytest-mock.
2025-02-18 23:32:03 +00:00
erinhmclark
657fbd357d
Merge branch 'main' into tests/add_module_tests
2025-02-18 19:47:47 +00:00
erinhmclark
7b88df72cb
Update test_metadata_enricher.py
2025-02-18 19:46:57 +00:00
Patrick Robertson
3c543a3a6a
Various fixes for issues with new architecture ( #208 )
...
* Add formatters to the TOC - fixes #204
* Add 'steps' settings to the example YAML in the docs. Fixes #206
* Improved docs on authentication architecture
* Fix setting modules on the command line - they now override any module settings in the orchestration as opposed to appending
* Fix tests for gsheet-feeder: add a test service_account.json (note: not real keys in there)
* Rename the command line entrypoint to _command_line_run
Also: make it clear that code implementation should not call this
Make sure the command line entry returns (we don't want a generator)
* Fix unit tests to use now code-entry points
* Version bump
* Move iterating of generator up to __main__
* Breakpoint
* two minor fixes
* Fix unit tests + add new '__main__' entry point implementation test
* Skip youtube tests if running on CI. Should still run them locally
* Fix full implementation run on GH actions
* Fix skipif test for GH Actions CI
* Add skipifs for truth - it blocks GH:
---------
Co-authored-by: msramalho <19508417+msramalho@users.noreply.github.com >
2025-02-18 19:10:09 +00:00
erinhmclark
ce5a200d1f
Added tests, updated instagram_tbot_extractor.py raise failure.
2025-02-18 12:59:10 +00:00
erinhmclark
f4c623b11b
Merge branch 'main' into tests/add_module_tests
2025-02-17 09:03:04 +00:00
Patrick Robertson
6d43bc7d4d
Fix generator programmatic setup ( #197 )
...
* Fix returning a generator of a generator
* Move download test test to pytest.mark.download
2025-02-15 17:36:44 +00:00
erinhmclark
71b41dd901
Remove accidental path, yet again.
2025-02-14 10:05:32 +00:00
erinhmclark
b0756a6a34
Remove accidental full path.
2025-02-14 09:57:44 +00:00
erinhmclark
319c1e8f92
Add more tests.
2025-02-14 09:48:37 +00:00
erinhmclark
3fce593aad
Merge branch 'main' into tests/add_module_tests
2025-02-12 19:33:29 +00:00
erinhmclark
cbe98c729d
Enricher tests
2025-02-12 19:32:40 +00:00
erinhmclark
d9d936c2ca
Thumbnail enricher fix seconds to minutes.
2025-02-12 12:22:27 +00:00
Patrick Robertson
29901da601
Merge branch 'load_modules' into docs_update
2025-02-11 14:10:56 +00:00
Patrick Robertson
2f51d3917a
Further addition to docs: creating modules, configurations, installation
2025-02-11 13:49:30 +00:00
