Patrick Robertson
2b91dc9514
Fix up unit tests
2025-03-10 16:51:16 +00:00
Patrick Robertson
a9c3477289
Improve docs on the path_generator and filename_generator config options
2025-03-10 16:43:14 +00:00
Patrick Robertson
770f4c8a3d
Refactoring of storage code:
...
1. Fix some bugs in local_storage
2. Refactor unit tests to not set Media.key explicitly (unless it's well-known beforehand, which it isn't)
3. Limit length of URL for 'url' type path_generator
4. Throw an error if 'save_to' of local storage is too long
5. A few other tidyups
2025-03-10 16:39:48 +00:00
Patrick Robertson
e89a8da3b4
Unit tests for storage types + fix storage too long issues for local storage
2025-03-10 11:30:15 +00:00
Erin Clark
4c21795d5f
Merge pull request #226 from bellingcat/merge_modules
...
Merge modules with multi-functionality:
- gsheet_feeder and gsheet_db are now one module, gsheet_feeder_db
- atlos_feeder, atlos_db and atlos_storage are now one module, atlos_feeder_db_storage.
This pull request also add documentation and updates references.
2025-03-07 16:47:30 +00:00
Patrick Robertson
e519ba2433
Add 'reject all' cookie button
2025-03-07 16:40:34 +00:00
Patrick Robertson
a8fcd0b9a0
Further info in how to for the new config format
2025-03-07 16:37:58 +00:00
Patrick Robertson
09e09e9ab9
Document module renames in 'upgrading from 0.12' how to
2025-03-07 16:28:17 +00:00
Patrick Robertson
be513e95aa
Merge branch 'main' into merge_modules
2025-03-07 16:19:51 +00:00
Patrick Robertson
3fac353407
Merge pull request #217 from bellingcat/settings_page
...
Settings page user interface
2025-03-07 16:10:50 +00:00
Erin Clark
928c6f88a9
Merge pull request #239 from bellingcat/fix_docker_registry_ref
...
Fix docker registry reference
2025-03-07 15:57:29 +00:00
erinhmclark
8fcec692b7
Add comments to highlight different steps of atlos_feeder_db_storage.py
2025-03-07 15:42:20 +00:00
erinhmclark
65109e377f
Remove raising exception in atlos_feeder_db_storage.py
2025-03-07 15:39:15 +00:00
Erin Clark
85a75755e2
Merge pull request #236 from bellingcat/cleanup_fixes
...
Cleanup fixes
2025-03-07 15:37:05 +00:00
erinhmclark
4949e9bcd2
Update docker-publish.yaml to use docker registry.
2025-03-07 15:34:56 +00:00
erinhmclark
3877b538be
Update docker-publish.yaml to use docker registry.
2025-03-07 15:31:25 +00:00
Erin Clark
2e0e989793
Merge pull request #238 from bellingcat/cache_docker_to_registry
...
Add cache-from and cache-to to docker-publish.yaml, using Dockerhub Registry as the cache.
2025-03-07 15:23:50 +00:00
Patrick Robertson
333201acec
Merge branch 'main' into settings_page
2025-03-07 15:17:42 +00:00
erinhmclark
87ab98c270
Update docker/build-push version
2025-03-07 15:14:20 +00:00
Patrick Robertson
027985024b
Merge pull request #234 from bellingcat/update_suggestions
...
Auto Updates
2025-03-07 15:12:03 +00:00
erinhmclark
7bbf0da0d1
Add cache-from and cache-to to docker-publish.yaml.
2025-03-07 15:09:10 +00:00
Patrick Robertson
48b29d43f7
Merge pull request #233 from bellingcat/docker-webdriver-aarch64
...
Docker webdriver aarch64
2025-03-07 15:04:45 +00:00
Erin Clark
8ae3d9c031
Merge pull request #235 from bellingcat/instagram_extractor_bugfix
...
Instagram extractor bugfix:
- Fix typo from config changes
- Add warning message to documentation to alert to it not being maintained.
2025-03-07 15:02:05 +00:00
Patrick Robertson
158e6be0b1
Don't force enable a module under all types - allows modules to be set as only feeder/only extractor etc.
2025-03-07 14:59:11 +00:00
erinhmclark
4df03255a4
Fix typo in __manifest__.py
2025-03-07 14:56:35 +00:00
Patrick Robertson
503ba3d1c1
Add note on auto updates to readme
2025-03-07 14:46:50 +00:00
erinhmclark
40e5fe7a7e
Update __manifest__.py for merged Atlos module.
2025-03-07 13:46:09 +00:00
Patrick Robertson
f6f397700e
Fix typo + use hidden password fields for secret info
2025-03-07 12:41:09 +00:00
erinhmclark
89d2a8bb54
Update the __manifest__.py of the Instagram Extractor.
2025-03-07 12:34:19 +00:00
Patrick Robertson
e72b3e14ba
Change default height of screenshots to attempt to capture more information
2025-03-07 12:08:29 +00:00
Patrick Robertson
dba44b1ac1
Use WebDriverWait when waiting for elements in screenshot enricher
2025-03-07 12:07:54 +00:00
Patrick Robertson
e756f1504f
Remove geckodriver .tar file
2025-03-07 11:52:14 +00:00
Patrick Robertson
2c5e138263
Add a note on disabling the auto-update for yt-dlp
2025-03-07 11:44:24 +00:00
erinhmclark
fb56aac15e
Catch edge case to ensure iterator is reached in instagram_tbot_extractor.py
2025-03-07 11:24:25 +00:00
erinhmclark
bdd35408ce
Fix ref before assignment in orchestrator.py
2025-03-07 11:23:51 +00:00
Patrick Robertson
478f0b2171
Tidy-ups to auto-updating code
2025-03-07 09:59:18 +00:00
erinhmclark
32329c6b2c
Update Google Sheet how to docs.
2025-03-07 00:11:43 +00:00
erinhmclark
fa1e65f54c
Fix instagram_extractor.py typo, add warning to docs, and add basic regex test.
2025-03-06 16:25:38 +00:00
erinhmclark
b9c2f98f46
Update Atlos tests
2025-03-05 21:24:38 +00:00
erinhmclark
0f911543cd
Atlos refactor
2025-03-05 13:49:11 +00:00
erinhmclark
6cb7afefdc
Initial Atlos merge
2025-03-05 10:24:54 +00:00
Patrick Robertson
358884c5d1
Fix unit tests for yt-dlp update
2025-03-04 17:04:23 +00:00
Patrick Robertson
be09aa927d
Make 'STARTED' command INFO not warning
2025-03-04 16:51:17 +00:00
Patrick Robertson
e6a578e60e
Check for auto-archiver updates and present warning if there's a newer version available
2025-03-04 16:51:17 +00:00
Patrick Robertson
0eb112431b
Auto-update yt-dlp based on generic_extractor.ytdlp_update_interval (default=5 days)
2025-03-04 16:43:46 +00:00
erinhmclark
22932645aa
Merge remote-tracking branch 'origin/merge_modules' into merge_modules
2025-03-04 14:07:12 +00:00
erinhmclark
d1c8d4ba0e
Initial merge of Atlos Feeder and DB
2025-03-04 14:06:46 +00:00
erinhmclark
d775e4612e
Update gsheet_feeder references in tests.
2025-03-04 14:06:46 +00:00
erinhmclark
077b56c150
Merge GSheet Feeder and Database.
2025-03-04 14:05:19 +00:00
Patrick Robertson
f54d6519a8
Fix sorting of steps in the outputted file
2025-03-04 11:51:26 +00:00
