erinhmclark
c4bb667cec
Merge branch 'load_modules' into add_module_tests
...
# Conflicts:
# src/auto_archiver/modules/s3_storage/s3_storage.py
# src/auto_archiver/utils/gsheet.py
# src/auto_archiver/utils/misc.py
2025-02-10 16:17:08 +00:00
msramalho
15abf686b1
decouples s3_storage from hash_enricher
2025-02-10 15:48:54 +00:00
msramalho
7c848046e8
adds better info about wrong/missing modules
2025-02-10 14:59:32 +00:00
Patrick Robertson
74207d7821
Implementation tests for auto-archiver
2025-02-10 13:27:11 +01:00
Patrick Robertson
e9dd321dcd
Fix setting cli_feeder as default feeder on clean install
2025-02-10 13:06:24 +01:00
Patrick Robertson
1fad37fd93
Remove blank file
2025-02-07 23:08:30 +01:00
Patrick Robertson
63aba6ad39
Fix sphinx-autoapi imports
2025-02-07 21:54:49 +01:00
erinhmclark
e9ad1e1b85
Pass media to storage cdn_call
2025-02-06 22:01:55 +00:00
Patrick Robertson
a506f2a88f
Clarify that an extractor's method can also return False if no valid data was found
2025-02-06 10:20:05 +01:00
Patrick Robertson
6ab8fd2ee4
Tidy up setting modules as Orchestrator attributes on startup.
...
Don't override the values in config['steps'] – the config should be left as is
2025-02-06 10:20:05 +01:00
Patrick Robertson
b301f60ea3
Fix using validators set in __manifest__.py
...
E.g. you can use the validator 'is_file' to check if a config is a valid file
2025-02-04 13:37:26 +01:00
Patrick Robertson
c574b694ed
Set up screenshot enricher to use authentication/cookies
2025-02-03 17:25:59 +01:00
Patrick Robertson
7a2be5a0da
Add cookie extraction to 'authentication' options, get generic_extractor working using this info
2025-02-03 16:03:07 +01:00
Patrick Robertson
9a8c94b641
Fix getting/setting folder context for metadata
2025-02-03 16:02:17 +01:00
Patrick Robertson
c25d5cae84
Remove ArchivingContext completely
...
Context for a specific url/item is now passed around via the metadata (metadata.set_context('key', 'val') and metadata.get_context('key', default='something')
The only other thing that was passed around in ArchivingContext was the storage info, which is already accessible now via self.config
2025-01-30 17:50:54 +01:00
Patrick Robertson
d6b4b7a932
Further cleanup
...
* Removes (partly) the ArchivingOrchestrator
* Removes the cli_feeder module, and makes it the 'default', allowing you to pass URLs directly on the command line, without having to use the cumbersome --cli_feeder.urls. Just do auto-archiver https://my.url.com
* More unit tests
* Improved error handling
2025-01-30 16:44:40 +01:00
Patrick Robertson
953011f368
Don't make modules 'dataclasses'
2025-01-30 16:44:40 +01:00
Patrick Robertson
fade68c6f4
Fix up unit tests - dataclass + subclasses not having @dataclass was breaking it
2025-01-30 13:45:24 +01:00
Patrick Robertson
b7d9145f6c
Further tidyups + refactoring for new structure
...
* Add implementation tests for orchestrator + logging tests
* Standardise method/class vars for extractors to see if they are suitable
* Fix bugs with removing default loguru logger (allows further customisation)
* Fix bug loading required fields from file
*
2025-01-30 13:21:10 +01:00
erinhmclark
cddae65a90
Update modules for new core structure.
2025-01-30 08:42:23 +00:00
Patrick Robertson
00a7018f36
Fix up dependency checking (use 'dependencies' instead of 'external_dependencies' -> simpler/easier to remember
2025-01-29 19:25:22 +01:00
Patrick Robertson
3d37c494aa
Tidy ups + unit tests:
...
1. Allow loading modules from --module_paths=/extra/path/here
2. Improved unit tests for module loading
3. Further small tidy ups/clean ups
2025-01-29 18:42:49 +01:00
Patrick Robertson
7a4871db6b
Fix up unit tests for new structure
2025-01-28 14:40:12 +01:00
Patrick Robertson
9635449ac0
more user friendly error logging when config issues are found
2025-01-28 11:44:52 +01:00
Patrick Robertson
27b25c5bd4
Validate orchestration.yaml file inputs - so if a user enters invalid values, it also validates them
2025-01-28 11:37:23 +01:00
Patrick Robertson
1d2a1d4db7
Allow framework for config settings that should not be stored in config (e.g. cli_feeder.urls
...
Use 'do_not_store': True in the config settings to apply this. Also: fix up generic archiver dropins loading + local_storage defaults (same as what's in example orchestration)
2025-01-28 11:14:12 +01:00
erinhmclark
e1a9373336
Refactoring for new config setup
2025-01-27 19:03:02 +00:00
Patrick Robertson
e3074013d0
Fix loading/saving to orchestration file with comments
2025-01-27 14:28:04 +01:00
Patrick Robertson
f68e2726f2
Refactor loader + step into module, use LazyBaseModule and BaseModule
2025-01-27 14:01:36 +01:00
Patrick Robertson
7fd95866a1
Further fixes/changes to loading 'types' for config + manifest edits
2025-01-27 11:48:04 +01:00
Patrick Robertson
14e2479599
Merge branch 'more_mainifests' into load_modules
2025-01-27 11:05:56 +01:00
Patrick Robertson
0b03f54f4e
Fix up config validation, and allow for custom 'validators'
2025-01-27 11:00:52 +01:00
Patrick Robertson
3fc6ddfe85
Tweaks to logging strings
2025-01-24 15:30:00 +01:00
Patrick Robertson
9befb9776c
Fix loading modules when entry_point isn't set
2025-01-23 21:08:54 +01:00
Patrick Robertson
06f6e34d9d
Revert changes to orchestrator to avoid merge conflicts
2025-01-23 20:38:36 +01:00
Patrick Robertson
b27bf8ffeb
Fix up loading/storing configs + unit tests
2025-01-23 20:32:19 +01:00
erinhmclark
1274a1b231
More manifests, base modules and rename from archiver to extractor.
2025-01-23 16:40:48 +00:00
erinhmclark
9db26cdfc2
Merge branch 'load_modules' into more_mainifests
...
# Conflicts:
# src/auto_archiver/core/orchestrator.py
2025-01-23 09:19:54 +00:00
erinhmclark
79684f8348
Set up feeder manifests (not merged by source yet)
2025-01-23 09:16:42 +00:00
Patrick Robertson
65ef46d01e
Fix loading already loaded modules - don't load them twice
2025-01-23 00:09:39 +01:00
Patrick Robertson
550097ab7b
Get module loading working properly
2025-01-22 23:54:21 +01:00
Patrick Robertson
ade5ea0f6f
Tidy up imports + start on loading modules - program now starts much faster
2025-01-22 18:45:58 +01:00
Patrick Robertson
b6b085854c
Switch back to using yaml with dot notation
...
(two simple helper functions to convert between dot and dict notation)
2025-01-22 17:40:51 +01:00
Patrick Robertson
54995ad6ab
Further tweaks based on __manifest__.py files
...
Loading configs now works
2025-01-22 13:11:43 +01:00
Patrick Robertson
4830f99300
Get parsing of manifest and combining with config file working
2025-01-21 20:03:10 +01:00
Patrick Robertson
241b35002c
Initial changes to move to '__manifest__' format
2025-01-21 19:02:38 +01:00
Patrick Robertson
c41d93a634
Use already implemented helper to get version
2025-01-21 17:53:37 +01:00
Patrick Robertson
6388983815
Merge branch 'main' into youtubedlp-rewrite
2025-01-21 16:43:14 +01:00
Patrick Robertson
5b20288d06
Add a 'version' arg to get the current running version
2025-01-17 16:59:57 +01:00
erinhmclark
d3eec5d90f
Basic docs structure for RTD
2025-01-15 21:45:29 +00:00
