Patrick Robertson
5e5e1c43a1
When loading modules, check they have been added to the right 'step' in the config
...
Fixes an issue seen on discord where a user accidentally set up metadata_enricher under 'extractors'
2025-03-20 18:09:26 +04:00
Patrick Robertson
89e387030d
Tests for suitable URLs for tikwm
2025-03-18 10:04:03 +00:00
Patrick Robertson
8ec053ed1b
Refactor the dropin 'is_suitable' method + fix tikwm implementation
...
Makes it easier to maintain/understand.
2025-03-18 09:14:14 +00:00
Patrick Robertson
3d4056ef70
Merge pull request #223 from bellingcat/facebook_extractor
...
Create facebook dropin - working for images + text.
2025-03-17 12:45:05 +00:00
Patrick Robertson
0765640bff
Fix up tiktok dropin for slightly modified generic_extractor format
2025-03-17 10:31:22 +00:00
Patrick Robertson
06b1f4c0ca
Fix lingering merge conflict issues
2025-03-17 10:12:55 +00:00
Patrick Robertson
59b910ec30
Merge main
2025-03-17 10:05:11 +00:00
Patrick Robertson
7e360240bf
Copy ytdlp code into AA project - seems like ytdlp won't be merged anytime soon
2025-03-17 09:57:05 +00:00
Patrick Robertson
7badf89c28
Create the 'secrets' folder if it doesn't exist on first run
...
Easier setup for users
2025-03-17 09:40:46 +00:00
Patrick Robertson
d59530c8e7
Fix if logic bug
2025-03-17 09:40:27 +00:00
Patrick Robertson
0ec5451f66
Nicer error log when no URLs provided for CLI feeder - don't need the stacktrace
2025-03-17 09:34:33 +00:00
Patrick Robertson
99e9ac2465
Fix 'Syntax Error' warning in python3.12+
2025-03-17 09:29:51 +00:00
Patrick Robertson
42162c5e3f
Various docs improvements based on Friday Office Hours discussion
2025-03-17 09:23:43 +00:00
Patrick Robertson
17463de937
Merge pull request #247 from bellingcat/opentimestamps
...
Opentimestamps Module
2025-03-14 13:41:46 +00:00
Patrick Robertson
a8e5585e6c
github format
2025-03-14 12:52:01 +00:00
Patrick Robertson
19715c8ec2
Merge branch 'main' into webdriver-cookies
2025-03-14 12:44:48 +00:00
Patrick Robertson
17ae75fb95
Ruff fixes
2025-03-14 12:38:12 +00:00
Patrick Robertson
b8da7607e8
Merge branch 'main' into opentimestamps
2025-03-14 12:36:03 +00:00
erinhmclark
72f48f0147
Fix merge conflicts.
2025-03-14 12:11:24 +00:00
erinhmclark
846474a4e2
Merge branch 'main' into linting_etc
2025-03-14 10:50:13 +00:00
Patrick Robertson
f504d2e304
Merge branch 'main' into webdriver-cookies
2025-03-14 09:37:12 +00:00
msramalho
4d67dce4c8
minor log fix
2025-03-13 19:24:05 +00:00
Patrick Robertson
f6b13327f0
Tweaks and additional debug logging
2025-03-13 17:41:41 +00:00
Patrick Robertson
589c834047
Fix parsing ytdlp args - we should first run them through the parse_options method
2025-03-13 17:41:40 +00:00
Patrick Robertson
0efeaaabb1
Revert to using time.sleep and .click() - since we only want to be waiting the first time (for the page to load)
2025-03-13 17:41:16 +00:00
Patrick Robertson
b908655cc8
Remove references to litecoin + several tidy-ups
2025-03-13 17:40:00 +00:00
Patrick Robertson
10ceb7aa15
Move tikwm extractor into a droping for the generic extractor
2025-03-13 15:59:42 +00:00
Patrick Robertson
e7489ac4c4
Tidy up opentimestamps
...
* Simplify
* Don't add fake (pending) attestations if the calendar urls all have issues
* Remove unnecessary configs
* Improve docs on upgrading + verifying
2025-03-13 14:30:33 +00:00
erinhmclark
8673bc5979
Fix unused imports and include rule.
2025-03-13 13:55:31 +00:00
erinhmclark
e76551ba22
Add documentation, pre-commit hook, more make commands and
2025-03-13 13:21:32 +00:00
erinhmclark
6e52a534e7
More fixes from Bugbear suggestions
2025-03-12 16:07:05 +00:00
erinhmclark
753c3c6214
Linting tests
2025-03-12 14:27:45 +00:00
Patrick Robertson
1d664524eb
Add info on last check/last updated to the metadata
2025-03-12 11:54:25 +00:00
Patrick Robertson
394b8b2dd1
Improvements to opentimestamps enricher - make OTS file a sub-file of original media
2025-03-12 11:45:13 +00:00
Patrick Robertson
1423c10363
Finish off timestamping module
2025-03-12 10:24:57 +00:00
erinhmclark
8ca7698fa0
Move Makefile and fix import error with unused import.
2025-03-11 19:58:02 +00:00
Patrick Robertson
b70ed97ffd
Create opentimestamps module
2025-03-11 17:28:28 +00:00
Patrick Robertson
94543e9a67
Merge branch 'main' into opentimestamps
2025-03-11 17:21:34 +00:00
Patrick Robertson
37eac64442
Remove desc
2025-03-11 17:10:44 +00:00
Patrick Robertson
7a81ab617a
Better checking of cookies to add to webdriver
2025-03-11 11:57:25 +00:00
erinhmclark
81aa343f21
Merge main.
2025-03-11 10:45:07 +00:00
erinhmclark
441f341139
Merge branch 'main' into linting_etc
...
# Conflicts:
# src/auto_archiver/core/consts.py
# src/auto_archiver/core/orchestrator.py
# src/auto_archiver/core/storage.py
# src/auto_archiver/modules/local_storage/local_storage.py
# src/auto_archiver/modules/s3_storage/s3_storage.py
# tests/storages/test_S3_storage.py
# tests/storages/test_local_storage.py
# tests/storages/test_storage_base.py
2025-03-11 10:39:47 +00:00
Patrick Robertson
e2442b2f6b
Merge pull request #243 from bellingcat/fix-long-path-names
...
Unit tests for storage types + fix storage too long issues for local storage
2025-03-11 10:05:09 +00:00
erinhmclark
e7fa88f1c7
Implementing ruff suggestions.
2025-03-10 21:45:30 +00:00
erinhmclark
ca44a40b88
Ruff fix on src.
2025-03-10 19:03:45 +00:00
erinhmclark
85abe1837a
Ruff format with defaults.
2025-03-10 18:44:54 +00:00
Miguel Sozinho Ramalho
3fcec57492
minor string fix
2025-03-10 17:17:59 +00:00
Patrick Robertson
a9c3477289
Improve docs on the path_generator and filename_generator config options
2025-03-10 16:43:14 +00:00
Patrick Robertson
770f4c8a3d
Refactoring of storage code:
...
1. Fix some bugs in local_storage
2. Refactor unit tests to not set Media.key explicitly (unless it's well-known beforehand, which it isn't)
3. Limit length of URL for 'url' type path_generator
4. Throw an error if 'save_to' of local storage is too long
5. A few other tidyups
2025-03-10 16:39:48 +00:00
Miguel Sozinho Ramalho
58bd38e292
Adds new extractor for tiktok via unofficial API ( #237 )
...
* minor update to defaults in api_db
* readme typo
* adds and tests new tikwm tiktok downloader
* addresses PR comments
2025-03-10 11:56:45 +00:00
