msramalho
ab6cf52533
fixes bad hash initialization
2025-02-10 16:45:28 +00:00
msramalho
15abf686b1
decouples s3_storage from hash_enricher
2025-02-10 15:48:54 +00:00
Patrick Robertson
c25d5cae84
Remove ArchivingContext completely
...
Context for a specific url/item is now passed around via the metadata (metadata.set_context('key', 'val') and metadata.get_context('key', default='something')
The only other thing that was passed around in ArchivingContext was the storage info, which is already accessible now via self.config
2025-01-30 17:50:54 +01:00
Patrick Robertson
00a7018f36
Fix up dependency checking (use 'dependencies' instead of 'external_dependencies' -> simpler/easier to remember
2025-01-29 19:25:22 +01:00
Patrick Robertson
3d37c494aa
Tidy ups + unit tests:
...
1. Allow loading modules from --module_paths=/extra/path/here
2. Improved unit tests for module loading
3. Further small tidy ups/clean ups
2025-01-29 18:42:49 +01:00
erinhmclark
e1a9373336
Refactoring for new config setup
2025-01-27 19:03:02 +00:00
Patrick Robertson
7fd95866a1
Further fixes/changes to loading 'types' for config + manifest edits
2025-01-27 11:48:04 +01:00
erinhmclark
21a7ff0520
Fix types in manifests
2025-01-27 08:43:18 +00:00
erinhmclark
aa7ca93a43
Update manifests and modules
2025-01-24 12:58:16 +00:00
erinhmclark
1274a1b231
More manifests, base modules and rename from archiver to extractor.
2025-01-23 16:40:48 +00:00
erinhmclark
79684f8348
Set up feeder manifests (not merged by source yet)
2025-01-23 09:16:42 +00:00
