Commit Graph

14 Commits

Author SHA1 Message Date
Patrick Robertson
034197a81f Fix typos in csv feeder docs (in manifest) 2025-02-04 13:40:07 +01:00
Patrick Robertson
78e6418249 Unit tests for csv feeder + fix some bugs 2025-02-04 13:37:26 +01:00
Patrick Robertson
b301f60ea3 Fix using validators set in __manifest__.py
E.g. you can use the validator 'is_file' to check if a config is a valid file
2025-02-04 13:37:26 +01:00
Patrick Robertson
c25d5cae84 Remove ArchivingContext completely
Context for a specific url/item is now passed around via the metadata (metadata.set_context('key', 'val') and metadata.get_context('key', default='something')
The only other thing that was passed around in ArchivingContext was the storage info, which is already accessible now via self.config
2025-01-30 17:50:54 +01:00
Patrick Robertson
d6b4b7a932 Further cleanup
* Removes (partly) the ArchivingOrchestrator
* Removes the cli_feeder module, and makes it the 'default', allowing you to pass URLs directly on the command line, without having to use the cumbersome --cli_feeder.urls. Just do auto-archiver https://my.url.com
* More unit tests
* Improved error handling
2025-01-30 16:44:40 +01:00
Patrick Robertson
00a7018f36 Fix up dependency checking (use 'dependencies' instead of 'external_dependencies' -> simpler/easier to remember 2025-01-29 19:25:22 +01:00
erinhmclark
e1a9373336 Refactoring for new config setup 2025-01-27 19:03:02 +00:00
Patrick Robertson
7fd95866a1 Further fixes/changes to loading 'types' for config + manifest edits 2025-01-27 11:48:04 +01:00
erinhmclark
dd402b456f Fix and add types to manifest 2025-01-24 18:50:11 +00:00
erinhmclark
024fe58377 fix config parsing in manifests, remove module level configs 2025-01-24 13:33:12 +00:00
erinhmclark
0453d95f56 fix config parsing in manifests 2025-01-24 13:24:54 +00:00
erinhmclark
aa7ca93a43 Update manifests and modules 2025-01-24 12:58:16 +00:00
erinhmclark
1274a1b231 More manifests, base modules and rename from archiver to extractor. 2025-01-23 16:40:48 +00:00
erinhmclark
79684f8348 Set up feeder manifests (not merged by source yet) 2025-01-23 09:16:42 +00:00