Commit Graph

17 Commits

Author SHA1 Message Date
Patrick Robertson
7a2be5a0da Add cookie extraction to 'authentication' options, get generic_extractor working using this info 2025-02-03 16:03:07 +01:00
Patrick Robertson
c25d5cae84 Remove ArchivingContext completely
Context for a specific url/item is now passed around via the metadata (metadata.set_context('key', 'val') and metadata.get_context('key', default='something')
The only other thing that was passed around in ArchivingContext was the storage info, which is already accessible now via self.config
2025-01-30 17:50:54 +01:00
Patrick Robertson
d6b4b7a932 Further cleanup
* Removes (partly) the ArchivingOrchestrator
* Removes the cli_feeder module, and makes it the 'default', allowing you to pass URLs directly on the command line, without having to use the cumbersome --cli_feeder.urls. Just do auto-archiver https://my.url.com
* More unit tests
* Improved error handling
2025-01-30 16:44:40 +01:00
Patrick Robertson
7a4871db6b Fix up unit tests for new structure 2025-01-28 14:40:12 +01:00
Patrick Robertson
1d2a1d4db7 Allow framework for config settings that should not be stored in config (e.g. cli_feeder.urls
Use 'do_not_store': True in the config settings to apply this. Also: fix up generic archiver dropins loading + local_storage defaults (same as what's in example orchestration)
2025-01-28 11:14:12 +01:00
erinhmclark
e1a9373336 Refactoring for new config setup 2025-01-27 19:03:02 +00:00
Patrick Robertson
7fd95866a1 Further fixes/changes to loading 'types' for config + manifest edits 2025-01-27 11:48:04 +01:00
Patrick Robertson
14e2479599 Merge branch 'more_mainifests' into load_modules 2025-01-27 11:05:56 +01:00
Patrick Robertson
f1e9ab6751 Merge branch 'main' into load_modules 2025-01-24 15:23:15 +01:00
erinhmclark
aa7ca93a43 Update manifests and modules 2025-01-24 12:58:16 +00:00
Patrick Robertson
9befb9776c Fix loading modules when entry_point isn't set 2025-01-23 21:08:54 +01:00
erinhmclark
1274a1b231 More manifests, base modules and rename from archiver to extractor. 2025-01-23 16:40:48 +00:00
Patrick Robertson
65ef46d01e Fix loading already loaded modules - don't load them twice 2025-01-23 00:09:39 +01:00
Patrick Robertson
550097ab7b Get module loading working properly 2025-01-22 23:54:21 +01:00
Patrick Robertson
54995ad6ab Further tweaks based on __manifest__.py files
Loading configs now works
2025-01-22 13:11:43 +01:00
Patrick Robertson
4830f99300 Get parsing of manifest and combining with config file working 2025-01-21 20:03:10 +01:00
Patrick Robertson
241b35002c Initial changes to move to '__manifest__' format 2025-01-21 19:02:38 +01:00