Patrick Robertson
19715c8ec2
Merge branch 'main' into webdriver-cookies
2025-03-14 12:44:48 +00:00
erinhmclark
72f48f0147
Fix merge conflicts.
2025-03-14 12:11:24 +00:00
erinhmclark
846474a4e2
Merge branch 'main' into linting_etc
2025-03-14 10:50:13 +00:00
Patrick Robertson
f504d2e304
Merge branch 'main' into webdriver-cookies
2025-03-14 09:37:12 +00:00
msramalho
4d67dce4c8
minor log fix
2025-03-13 19:24:05 +00:00
Patrick Robertson
f6b13327f0
Tweaks and additional debug logging
2025-03-13 17:41:41 +00:00
Patrick Robertson
589c834047
Fix parsing ytdlp args - we should first run them through the parse_options method
2025-03-13 17:41:40 +00:00
Patrick Robertson
0efeaaabb1
Revert to using time.sleep and .click() - since we only want to be waiting the first time (for the page to load)
2025-03-13 17:41:16 +00:00
Patrick Robertson
10ceb7aa15
Move tikwm extractor into a droping for the generic extractor
2025-03-13 15:59:42 +00:00
erinhmclark
8673bc5979
Fix unused imports and include rule.
2025-03-13 13:55:31 +00:00
erinhmclark
e76551ba22
Add documentation, pre-commit hook, more make commands and
2025-03-13 13:21:32 +00:00
erinhmclark
6e52a534e7
More fixes from Bugbear suggestions
2025-03-12 16:07:05 +00:00
erinhmclark
753c3c6214
Linting tests
2025-03-12 14:27:45 +00:00
erinhmclark
8ca7698fa0
Move Makefile and fix import error with unused import.
2025-03-11 19:58:02 +00:00
Patrick Robertson
7a81ab617a
Better checking of cookies to add to webdriver
2025-03-11 11:57:25 +00:00
erinhmclark
81aa343f21
Merge main.
2025-03-11 10:45:07 +00:00
erinhmclark
441f341139
Merge branch 'main' into linting_etc
...
# Conflicts:
# src/auto_archiver/core/consts.py
# src/auto_archiver/core/orchestrator.py
# src/auto_archiver/core/storage.py
# src/auto_archiver/modules/local_storage/local_storage.py
# src/auto_archiver/modules/s3_storage/s3_storage.py
# tests/storages/test_S3_storage.py
# tests/storages/test_local_storage.py
# tests/storages/test_storage_base.py
2025-03-11 10:39:47 +00:00
Patrick Robertson
e2442b2f6b
Merge pull request #243 from bellingcat/fix-long-path-names
...
Unit tests for storage types + fix storage too long issues for local storage
2025-03-11 10:05:09 +00:00
erinhmclark
e7fa88f1c7
Implementing ruff suggestions.
2025-03-10 21:45:30 +00:00
erinhmclark
ca44a40b88
Ruff fix on src.
2025-03-10 19:03:45 +00:00
erinhmclark
85abe1837a
Ruff format with defaults.
2025-03-10 18:44:54 +00:00
Miguel Sozinho Ramalho
3fcec57492
minor string fix
2025-03-10 17:17:59 +00:00
Patrick Robertson
a9c3477289
Improve docs on the path_generator and filename_generator config options
2025-03-10 16:43:14 +00:00
Patrick Robertson
770f4c8a3d
Refactoring of storage code:
...
1. Fix some bugs in local_storage
2. Refactor unit tests to not set Media.key explicitly (unless it's well-known beforehand, which it isn't)
3. Limit length of URL for 'url' type path_generator
4. Throw an error if 'save_to' of local storage is too long
5. A few other tidyups
2025-03-10 16:39:48 +00:00
Miguel Sozinho Ramalho
58bd38e292
Adds new extractor for tiktok via unofficial API ( #237 )
...
* minor update to defaults in api_db
* readme typo
* adds and tests new tikwm tiktok downloader
* addresses PR comments
2025-03-10 11:56:45 +00:00
Patrick Robertson
e89a8da3b4
Unit tests for storage types + fix storage too long issues for local storage
2025-03-10 11:30:15 +00:00
Patrick Robertson
e519ba2433
Add 'reject all' cookie button
2025-03-07 16:40:34 +00:00
Patrick Robertson
be513e95aa
Merge branch 'main' into merge_modules
2025-03-07 16:19:51 +00:00
Patrick Robertson
3fac353407
Merge pull request #217 from bellingcat/settings_page
...
Settings page user interface
2025-03-07 16:10:50 +00:00
erinhmclark
8fcec692b7
Add comments to highlight different steps of atlos_feeder_db_storage.py
2025-03-07 15:42:20 +00:00
erinhmclark
65109e377f
Remove raising exception in atlos_feeder_db_storage.py
2025-03-07 15:39:15 +00:00
Erin Clark
85a75755e2
Merge pull request #236 from bellingcat/cleanup_fixes
...
Cleanup fixes
2025-03-07 15:37:05 +00:00
Patrick Robertson
333201acec
Merge branch 'main' into settings_page
2025-03-07 15:17:42 +00:00
Patrick Robertson
027985024b
Merge pull request #234 from bellingcat/update_suggestions
...
Auto Updates
2025-03-07 15:12:03 +00:00
Patrick Robertson
48b29d43f7
Merge pull request #233 from bellingcat/docker-webdriver-aarch64
...
Docker webdriver aarch64
2025-03-07 15:04:45 +00:00
erinhmclark
4df03255a4
Fix typo in __manifest__.py
2025-03-07 14:56:35 +00:00
Patrick Robertson
503ba3d1c1
Add note on auto updates to readme
2025-03-07 14:46:50 +00:00
erinhmclark
40e5fe7a7e
Update __manifest__.py for merged Atlos module.
2025-03-07 13:46:09 +00:00
erinhmclark
89d2a8bb54
Update the __manifest__.py of the Instagram Extractor.
2025-03-07 12:34:19 +00:00
Patrick Robertson
e72b3e14ba
Change default height of screenshots to attempt to capture more information
2025-03-07 12:08:29 +00:00
Patrick Robertson
dba44b1ac1
Use WebDriverWait when waiting for elements in screenshot enricher
2025-03-07 12:07:54 +00:00
Patrick Robertson
2c5e138263
Add a note on disabling the auto-update for yt-dlp
2025-03-07 11:44:24 +00:00
erinhmclark
fb56aac15e
Catch edge case to ensure iterator is reached in instagram_tbot_extractor.py
2025-03-07 11:24:25 +00:00
erinhmclark
bdd35408ce
Fix ref before assignment in orchestrator.py
2025-03-07 11:23:51 +00:00
Patrick Robertson
478f0b2171
Tidy-ups to auto-updating code
2025-03-07 09:59:18 +00:00
erinhmclark
fa1e65f54c
Fix instagram_extractor.py typo, add warning to docs, and add basic regex test.
2025-03-06 16:25:38 +00:00
erinhmclark
b9c2f98f46
Update Atlos tests
2025-03-05 21:24:38 +00:00
erinhmclark
0f911543cd
Atlos refactor
2025-03-05 13:49:11 +00:00
erinhmclark
6cb7afefdc
Initial Atlos merge
2025-03-05 10:24:54 +00:00
Patrick Robertson
358884c5d1
Fix unit tests for yt-dlp update
2025-03-04 17:04:23 +00:00
