erinhmclark
72f48f0147
Fix merge conflicts.
2025-03-14 12:11:24 +00:00
erinhmclark
846474a4e2
Merge branch 'main' into linting_etc
2025-03-14 10:50:13 +00:00
msramalho
4d67dce4c8
minor log fix
2025-03-13 19:24:05 +00:00
Patrick Robertson
10ceb7aa15
Move tikwm extractor into a droping for the generic extractor
2025-03-13 15:59:42 +00:00
erinhmclark
ca44a40b88
Ruff fix on src.
2025-03-10 19:03:45 +00:00
erinhmclark
85abe1837a
Ruff format with defaults.
2025-03-10 18:44:54 +00:00
Patrick Robertson
503ba3d1c1
Add note on auto updates to readme
2025-03-07 14:46:50 +00:00
Patrick Robertson
2c5e138263
Add a note on disabling the auto-update for yt-dlp
2025-03-07 11:44:24 +00:00
Patrick Robertson
478f0b2171
Tidy-ups to auto-updating code
2025-03-07 09:59:18 +00:00
Patrick Robertson
358884c5d1
Fix unit tests for yt-dlp update
2025-03-04 17:04:23 +00:00
Patrick Robertson
0eb112431b
Auto-update yt-dlp based on generic_extractor.ytdlp_update_interval (default=5 days)
2025-03-04 16:43:46 +00:00
erinhmclark
8124bb831d
Merge branch 'main' into small_issues
...
# Conflicts:
# src/auto_archiver/core/base_module.py
# src/auto_archiver/utils/misc.py
2025-02-26 13:19:49 +00:00
erinhmclark
9bc6dd5c3c
Add set_content into generic_extractor.py.
2025-02-25 20:07:00 +00:00
Patrick Robertson
7dde8d609d
Merge main
2025-02-20 10:29:57 +00:00
erinhmclark
a8ffb19325
Fix auth key name for cookies_from_browser.
2025-02-19 10:40:54 +00:00
Patrick Robertson
222a94563f
WIP: Docs tidyups+add howto on logging and authentication
...
(Authentication is WIP)
2025-02-19 10:37:04 +00:00
Patrick Robertson
ea728a7a97
TODO on facebook dropin not working
2025-02-11 15:56:12 +00:00
msramalho
91f1ebf7b3
fix temp for yandex new shortlink
2025-02-11 15:23:16 +00:00
msramalho
5478ed3860
bsky fix media fetching
2025-02-11 15:02:00 +00:00
msramalho
47d1dc9d47
typing warnings fixed
2025-02-11 15:01:37 +00:00
Patrick Robertson
63aba6ad39
Fix sphinx-autoapi imports
2025-02-07 21:54:49 +01:00
Patrick Robertson
91ca325fd5
Update yt-dlp to latest version + remove code no longer needed from bluesky dropin
2025-02-04 17:46:46 +01:00
Patrick Robertson
c574b694ed
Set up screenshot enricher to use authentication/cookies
2025-02-03 17:25:59 +01:00
Patrick Robertson
7ec328ab40
Remove cookie options from generic_extractor - it now uses 'authentication' global settings :D
2025-02-03 16:04:36 +01:00
Patrick Robertson
7a2be5a0da
Add cookie extraction to 'authentication' options, get generic_extractor working using this info
2025-02-03 16:03:07 +01:00
Patrick Robertson
c25d5cae84
Remove ArchivingContext completely
...
Context for a specific url/item is now passed around via the metadata (metadata.set_context('key', 'val') and metadata.get_context('key', default='something')
The only other thing that was passed around in ArchivingContext was the storage info, which is already accessible now via self.config
2025-01-30 17:50:54 +01:00
Patrick Robertson
d6b4b7a932
Further cleanup
...
* Removes (partly) the ArchivingOrchestrator
* Removes the cli_feeder module, and makes it the 'default', allowing you to pass URLs directly on the command line, without having to use the cumbersome --cli_feeder.urls. Just do auto-archiver https://my.url.com
* More unit tests
* Improved error handling
2025-01-30 16:44:40 +01:00
Patrick Robertson
7a4871db6b
Fix up unit tests for new structure
2025-01-28 14:40:12 +01:00
Patrick Robertson
1d2a1d4db7
Allow framework for config settings that should not be stored in config (e.g. cli_feeder.urls
...
Use 'do_not_store': True in the config settings to apply this. Also: fix up generic archiver dropins loading + local_storage defaults (same as what's in example orchestration)
2025-01-28 11:14:12 +01:00
erinhmclark
e1a9373336
Refactoring for new config setup
2025-01-27 19:03:02 +00:00
Patrick Robertson
7fd95866a1
Further fixes/changes to loading 'types' for config + manifest edits
2025-01-27 11:48:04 +01:00
Patrick Robertson
14e2479599
Merge branch 'more_mainifests' into load_modules
2025-01-27 11:05:56 +01:00
Patrick Robertson
f1e9ab6751
Merge branch 'main' into load_modules
2025-01-24 15:23:15 +01:00
erinhmclark
aa7ca93a43
Update manifests and modules
2025-01-24 12:58:16 +00:00
Patrick Robertson
9befb9776c
Fix loading modules when entry_point isn't set
2025-01-23 21:08:54 +01:00
erinhmclark
1274a1b231
More manifests, base modules and rename from archiver to extractor.
2025-01-23 16:40:48 +00:00
Patrick Robertson
65ef46d01e
Fix loading already loaded modules - don't load them twice
2025-01-23 00:09:39 +01:00
Patrick Robertson
550097ab7b
Get module loading working properly
2025-01-22 23:54:21 +01:00
Patrick Robertson
54995ad6ab
Further tweaks based on __manifest__.py files
...
Loading configs now works
2025-01-22 13:11:43 +01:00
Patrick Robertson
4830f99300
Get parsing of manifest and combining with config file working
2025-01-21 20:03:10 +01:00
Patrick Robertson
241b35002c
Initial changes to move to '__manifest__' format
2025-01-21 19:02:38 +01:00
