748 Commits

Author	SHA1	Message	Date
erinhmclark	d1c8d4ba0e	Initial merge of Atlos Feeder and DB	2025-03-04 14:06:46 +00:00
erinhmclark	077b56c150	Merge GSheet Feeder and Database.	2025-03-04 14:05:19 +00:00
erinhmclark	7e4b44883b	Add temp options for testing	2025-03-04 14:03:39 +00:00
erinhmclark	77b517cfc1	Merge remote-tracking branch 'origin/feat/yt-dlp-pots' into feat/yt-dlp-pots	2025-03-03 22:02:14 +00:00
erinhmclark	dd07b0b830	Allow flexible extractor_args in generic_extractor.py.	2025-03-03 21:11:34 +00:00
erinhmclark	a705a78632	Fix instagram_extractor.py typo in config value.	2025-03-03 21:06:09 +00:00
Patrick Robertson	0b5a0fcb32	Better error logs if users have XXXX_archiver modules enabled in config	2025-03-03 19:57:09 +00:00
Patrick Robertson	1fe023cd70	Throw a nicer error if a user has an orchestration.yaml file in the old format (feeder: / archivers: / formatter: )	2025-03-03 19:51:55 +00:00
Patrick Robertson	0dfab2d1bc	Add some code to attempt to click the cookies banners on various websites	2025-03-03 15:55:04 +00:00
Patrick Robertson	dea0a49600	Download correct gecko-driver for the platform + fix setting executable path when running in Docker ... Fixes #232	2025-03-03 15:41:44 +00:00
Patrick Robertson	a0869bb3b2	Fixed up timestamp verifying - waiting on issue with rfc-client to be fixed ... Ref: https://github.com/trailofbits/rfc3161-client/issues/104#issuecomment-2693890607	2025-03-03 10:28:30 +00:00
Patrick Robertson	65a9885d86	A few more manifest types	2025-02-27 21:33:04 +00:00
Patrick Robertson	1e92c03b1d	Tweaks to settings page + more declarations in manifests	2025-02-27 15:21:11 +00:00
Patrick Robertson	efe9fdf915	Tidy ups to config editor page	2025-02-27 13:02:50 +00:00
Patrick Robertson	f58f110436	Check at least 1 URL provided for new cli_feeder module rewrite	2025-02-26 17:59:13 +00:00
Patrick Robertson	70d89c71ce	Fully-working settings page editor	2025-02-26 17:02:49 +00:00
Patrick Robertson	bb961b131c	Turn cli_feeder *back* into a module, it's better like this for settings etc, documentation etc.	2025-02-26 15:41:33 +00:00
erinhmclark	8124bb831d	Merge branch 'main' into small_issues ... # Conflicts: # src/auto_archiver/core/base_module.py # src/auto_archiver/utils/misc.py	2025-02-26 13:19:49 +00:00
erinhmclark	9157846930	Add docstrings to explain date formats.	2025-02-26 10:01:52 +00:00
Patrick Robertson	afc117a229	Get downloading certs working	2025-02-26 09:33:56 +00:00
erinhmclark	83a08dd215	Update date parsing to use dateutil.parser in misc.py	2025-02-25 20:17:31 +00:00
erinhmclark	9bc6dd5c3c	Add set_content into generic_extractor.py.	2025-02-25 20:07:00 +00:00
erinhmclark	cf1219f798	Add text content into gsheet.	2025-02-25 20:06:44 +00:00
Patrick Robertson	4dcb77c29f	Merge branch 'main' into timestamping_rewrite	2025-02-25 17:10:55 +00:00
erinhmclark	1df5129268	Small typos.	2025-02-25 14:08:38 +00:00
Patrick Robertson	898faf6fe4	Further WIP - currently working on verify_signed	2025-02-25 12:08:08 +00:00
Patrick Robertson	f8e846d59a	Create facebook dropin - working for images + text. CAVEAT: only gets the first ~100 chars of the post at the moment	2025-02-25 11:44:35 +00:00
Patrick Robertson	d10c7fbe55	Better documentation based on the discord feedbackgst	2025-02-24 22:42:57 +00:00
Patrick Robertson	ca1ed418aa	Throw an error for invalid __manifest__ syntax + fix: allow default values of False/None	2025-02-24 21:46:24 +00:00
Patrick Robertson	01bf88a695	Merge branch 'main' into timestamping_rewrite	2025-02-24 12:03:14 +00:00
erinhmclark	c5127f5fd1	Allow flexible extractor_args in generic_extractor.py.	2025-02-24 11:40:44 +00:00
Patrick Robertson	091a19e25c	Further docs improvements/tidy ups	2025-02-21 16:52:30 +00:00
Patrick Robertson	9661e90a05	Allow disabling logging in auto_archiver with logging: enabled: false	2025-02-20 15:45:32 +00:00
Patrick Robertson	0bec71d203	Finish how to on authentication	2025-02-20 15:33:50 +00:00
Patrick Robertson	4174285898	Fix unit tests	2025-02-20 13:18:06 +00:00
Patrick Robertson	eda359a1ef	Fix json loader - it should go in 'validators' not 'utils' ... Fixes #214	2025-02-20 13:10:39 +00:00
Patrick Robertson	40488e0869	Use 'Auto Archiver' naming for consistency. ... auto-archiver is reserved in the docs for when talking about the command line usage	2025-02-20 11:50:29 +00:00
Patrick Robertson	cbea551876	Better display name for wayback machine to emphasise it's typically used as an enricher	2025-02-20 11:46:57 +00:00
Patrick Robertson	b978484a89	Rename wacz_enricher to wacz_extractor_enricher. Fixes #205	2025-02-20 11:46:57 +00:00
Patrick Robertson	49b6c32058	Fix the 'full' mode which creates a complete config file	2025-02-20 11:34:05 +00:00
Patrick Robertson	4b51ec9ad5	Remove dangling import	2025-02-20 11:20:16 +00:00
Patrick Robertson	7734a551fa	Move 'assert_valid_url' out into utils, don't use assert but raise ... assert is recommended only for debugging	2025-02-20 11:19:29 +00:00
Patrick Robertson	77b2b099c6	Replace exit() with raise exceptions. Better for code implementations ... exit() is reserved solely for command line-called areas now also assert is only recommended for debugging	2025-02-20 11:19:13 +00:00
Patrick Robertson	7dde8d609d	Merge main	2025-02-20 10:29:57 +00:00
Patrick Robertson	5211c5de18	Merge pull request #210 from bellingcat/logger_fix ... Fix issue #200 + Refactor _LAZY_LOADED_MODULES	2025-02-19 15:11:42 +00:00
Patrick Robertson	a9802dd004	Remove the global _LAZY_LOADED_MODULES and allow each instance of ArchivingOrchestrator to load its own modules	2025-02-19 12:25:35 +00:00
erinhmclark	a8ffb19325	Fix auth key name for cookies_from_browser.	2025-02-19 10:40:54 +00:00
Patrick Robertson	222a94563f	WIP: Docs tidyups+add howto on logging and authentication ... (Authentication is WIP)	2025-02-19 10:37:04 +00:00
Patrick Robertson	eb60b271b9	Fix issue #200	2025-02-19 10:35:14 +00:00
erinhmclark	ddf2e76624	Include Atlos Storage __init__.py for module recognition.	2025-02-19 09:24:34 +00:00