Commit Graph

40 Commits

Author SHA1 Message Date
Patrick Robertson
091a19e25c Further docs improvements/tidy ups 2025-02-21 16:52:30 +00:00
Patrick Robertson
eda359a1ef Fix json loader - it should go in 'validators' not 'utils'
Fixes #214
2025-02-20 13:10:39 +00:00
Patrick Robertson
49b6c32058 Fix the 'full' mode which creates a complete config file 2025-02-20 11:34:05 +00:00
Patrick Robertson
77b2b099c6 Replace exit() with raise exceptions. Better for code implementations
exit() is reserved solely for command line-called areas now
also assert is only recommended for debugging
2025-02-20 11:19:13 +00:00
Patrick Robertson
7dde8d609d Merge main 2025-02-20 10:29:57 +00:00
Patrick Robertson
a9802dd004 Remove the global _LAZY_LOADED_MODULES and allow each instance of ArchivingOrchestrator to load its own modules 2025-02-19 12:25:35 +00:00
Patrick Robertson
222a94563f WIP: Docs tidyups+add howto on logging and authentication
(Authentication is WIP)
2025-02-19 10:37:04 +00:00
Patrick Robertson
3c543a3a6a Various fixes for issues with new architecture (#208)
* Add formatters to the TOC - fixes #204

* Add 'steps' settings to the example YAML in the docs. Fixes #206

* Improved docs on authentication architecture

* Fix setting modules on the command line - they now override any module settings in the orchestration as opposed to appending

* Fix tests for gsheet-feeder: add a test service_account.json (note: not real keys in there)

* Rename the command line entrypoint to _command_line_run

Also: make it clear that code implementation should not call this
Make sure the command line entry returns (we don't want a generator)

* Fix unit tests to use now code-entry points

* Version bump

* Move iterating of generator up to __main__

* Breakpoint

* two minor fixes

* Fix unit tests + add new '__main__' entry point implementation test

* Skip youtube tests if running on CI. Should still run them locally

* Fix full implementation run on GH actions

* Fix skipif test for GH Actions CI

* Add skipifs for truth - it blocks GH:

---------

Co-authored-by: msramalho <19508417+msramalho@users.noreply.github.com>
2025-02-18 19:10:09 +00:00
Patrick Robertson
2f51d3917a Further addition to docs: creating modules, configurations, installation 2025-02-11 13:49:30 +00:00
Patrick Robertson
ed81dcdaf0 Remove dangling 'b = ' from config.py 2025-02-10 23:07:03 +00:00
Patrick Robertson
e9dd321dcd Fix setting cli_feeder as default feeder on clean install 2025-02-10 13:06:24 +01:00
Patrick Robertson
d6b4b7a932 Further cleanup
* Removes (partly) the ArchivingOrchestrator
* Removes the cli_feeder module, and makes it the 'default', allowing you to pass URLs directly on the command line, without having to use the cumbersome --cli_feeder.urls. Just do auto-archiver https://my.url.com
* More unit tests
* Improved error handling
2025-01-30 16:44:40 +01:00
Patrick Robertson
b7d9145f6c Further tidyups + refactoring for new structure
* Add implementation tests for orchestrator + logging tests
* Standardise method/class vars for extractors to see if they are suitable
* Fix bugs with removing default loguru logger (allows further customisation)
* Fix bug loading required fields from file
*
2025-01-30 13:21:10 +01:00
Patrick Robertson
3d37c494aa Tidy ups + unit tests:
1. Allow loading modules from --module_paths=/extra/path/here
2. Improved unit tests for module loading
3. Further small tidy ups/clean ups
2025-01-29 18:42:49 +01:00
Patrick Robertson
9635449ac0 more user friendly error logging when config issues are found 2025-01-28 11:44:52 +01:00
Patrick Robertson
27b25c5bd4 Validate orchestration.yaml file inputs - so if a user enters invalid values, it also validates them 2025-01-28 11:37:23 +01:00
Patrick Robertson
1d2a1d4db7 Allow framework for config settings that should not be stored in config (e.g. cli_feeder.urls
Use 'do_not_store': True in the config settings to apply this. Also: fix up generic archiver dropins loading + local_storage defaults (same as what's in example orchestration)
2025-01-28 11:14:12 +01:00
Patrick Robertson
e3074013d0 Fix loading/saving to orchestration file with comments 2025-01-27 14:28:04 +01:00
Patrick Robertson
f68e2726f2 Refactor loader + step into module, use LazyBaseModule and BaseModule 2025-01-27 14:01:36 +01:00
Patrick Robertson
7fd95866a1 Further fixes/changes to loading 'types' for config + manifest edits 2025-01-27 11:48:04 +01:00
Patrick Robertson
14e2479599 Merge branch 'more_mainifests' into load_modules 2025-01-27 11:05:56 +01:00
Patrick Robertson
9befb9776c Fix loading modules when entry_point isn't set 2025-01-23 21:08:54 +01:00
Patrick Robertson
b27bf8ffeb Fix up loading/storing configs + unit tests 2025-01-23 20:32:19 +01:00
erinhmclark
1274a1b231 More manifests, base modules and rename from archiver to extractor. 2025-01-23 16:40:48 +00:00
Patrick Robertson
ade5ea0f6f Tidy up imports + start on loading modules - program now starts much faster 2025-01-22 18:45:58 +01:00
Patrick Robertson
b6b085854c Switch back to using yaml with dot notation
(two simple helper functions to convert between dot and dict notation)
2025-01-22 17:40:51 +01:00
Patrick Robertson
54995ad6ab Further tweaks based on __manifest__.py files
Loading configs now works
2025-01-22 13:11:43 +01:00
Patrick Robertson
4830f99300 Get parsing of manifest and combining with config file working 2025-01-21 20:03:10 +01:00
Patrick Robertson
241b35002c Initial changes to move to '__manifest__' format 2025-01-21 19:02:38 +01:00
Patrick Robertson
c41d93a634 Use already implemented helper to get version 2025-01-21 17:53:37 +01:00
Patrick Robertson
6388983815 Merge branch 'main' into youtubedlp-rewrite 2025-01-21 16:43:14 +01:00
Patrick Robertson
5b20288d06 Add a 'version' arg to get the current running version 2025-01-17 16:59:57 +01:00
erinhmclark
d3eec5d90f Basic docs structure for RTD 2025-01-15 21:45:29 +00:00
msramalho
613b1f1e50 properly overwrite configs 2023-05-19 12:35:19 +01:00
msramalho
68e9d2a2ce allows yaml config to be overwritten 2023-05-19 11:49:02 +01:00
msramalho
2a7ece5dcc cleanups and docs 2023-02-08 22:13:19 +00:00
msramalho
d1e4dde3f6 fixing imports 2023-01-27 00:19:58 +00:00
msramalho
f5b7c3a5ea mute formatter and docker 2023-01-26 23:38:58 +00:00
msramalho
746f6a333e further cleanup 2023-01-21 19:57:54 +00:00
msramalho
b763fc4188 final naming cleanup + new feeders/dbs 2023-01-21 19:44:12 +00:00