erinhmclark
cddae65a90
Update modules for new core structure.
2025-01-30 08:42:23 +00:00
Patrick Robertson
00a7018f36
Fix up dependency checking (use 'dependencies' instead of 'external_dependencies' -> simpler/easier to remember
2025-01-29 19:25:22 +01:00
Patrick Robertson
3d37c494aa
Tidy ups + unit tests:
...
1. Allow loading modules from --module_paths=/extra/path/here
2. Improved unit tests for module loading
3. Further small tidy ups/clean ups
2025-01-29 18:42:49 +01:00
Patrick Robertson
dcd5576f29
set metadata enricher to requires_setup=True (requires exiftool which isn't installed by default on most machines)
2025-01-29 00:10:40 +01:00
Patrick Robertson
7a4871db6b
Fix up unit tests for new structure
2025-01-28 14:40:12 +01:00
Patrick Robertson
27b25c5bd4
Validate orchestration.yaml file inputs - so if a user enters invalid values, it also validates them
2025-01-28 11:37:23 +01:00
Patrick Robertson
1d2a1d4db7
Allow framework for config settings that should not be stored in config (e.g. cli_feeder.urls
...
Use 'do_not_store': True in the config settings to apply this. Also: fix up generic archiver dropins loading + local_storage defaults (same as what's in example orchestration)
2025-01-28 11:14:12 +01:00
erinhmclark
57b3bec935
Google sheets feeder and database implemented.
2025-01-27 20:13:12 +00:00
erinhmclark
6c67effd8c
remove name reference in local_storage.py
2025-01-27 19:17:18 +00:00
erinhmclark
e1a9373336
Refactoring for new config setup
2025-01-27 19:03:02 +00:00
Patrick Robertson
f68e2726f2
Refactor loader + step into module, use LazyBaseModule and BaseModule
2025-01-27 14:01:36 +01:00
Patrick Robertson
7fd95866a1
Further fixes/changes to loading 'types' for config + manifest edits
2025-01-27 11:48:04 +01:00
Patrick Robertson
14e2479599
Merge branch 'more_mainifests' into load_modules
2025-01-27 11:05:56 +01:00
erinhmclark
ebebd27897
Fix archiver to extractor naming
2025-01-27 09:11:45 +00:00
erinhmclark
21a7ff0520
Fix types in manifests
2025-01-27 08:43:18 +00:00
erinhmclark
dd402b456f
Fix and add types to manifest
2025-01-24 18:50:11 +00:00
Patrick Robertson
f1e9ab6751
Merge branch 'main' into load_modules
2025-01-24 15:23:15 +01:00
erinhmclark
024fe58377
fix config parsing in manifests, remove module level configs
2025-01-24 13:33:12 +00:00
erinhmclark
0453d95f56
fix config parsing in manifests
2025-01-24 13:24:54 +00:00
erinhmclark
aa7ca93a43
Update manifests and modules
2025-01-24 12:58:16 +00:00
Patrick Robertson
9befb9776c
Fix loading modules when entry_point isn't set
2025-01-23 21:08:54 +01:00
erinhmclark
50f4ebcdc3
Move storage configs into individual manifests, assert format on useage.
2025-01-23 17:01:30 +00:00
erinhmclark
c3403ced26
Rename storages for clarity
2025-01-23 16:51:17 +00:00
erinhmclark
1274a1b231
More manifests, base modules and rename from archiver to extractor.
2025-01-23 16:40:48 +00:00
erinhmclark
9db26cdfc2
Merge branch 'load_modules' into more_mainifests
...
# Conflicts:
# src/auto_archiver/core/orchestrator.py
2025-01-23 09:19:54 +00:00
erinhmclark
79684f8348
Set up feeder manifests (not merged by source yet)
2025-01-23 09:16:42 +00:00
Patrick Robertson
65ef46d01e
Fix loading already loaded modules - don't load them twice
2025-01-23 00:09:39 +01:00
Patrick Robertson
550097ab7b
Get module loading working properly
2025-01-22 23:54:21 +01:00
erinhmclark
99c8c69085
Manifests for databases
2025-01-22 18:18:13 +00:00
Patrick Robertson
54995ad6ab
Further tweaks based on __manifest__.py files
...
Loading configs now works
2025-01-22 13:11:43 +01:00
erinhmclark
7b3a1468cd
Create manifest files for archiver modules.
2025-01-22 10:21:27 +01:00
Patrick Robertson
4830f99300
Get parsing of manifest and combining with config file working
2025-01-21 20:03:10 +01:00
Patrick Robertson
241b35002c
Initial changes to move to '__manifest__' format
2025-01-21 19:02:38 +01:00
