erinhmclark
5b0bad832f
Updated test, test metadata
2025-02-06 10:11:56 +00:00
erinhmclark
52542812dc
Merge tests from version with context.
2025-02-05 16:42:58 +00:00
Patrick Robertson
91ca325fd5
Update yt-dlp to latest version + remove code no longer needed from bluesky dropin
2025-02-04 17:46:46 +01:00
Patrick Robertson
0633e17998
Close the facebook 'login' window if it's there - to allow for proper screenshots
2025-02-04 14:18:46 +01:00
Patrick Robertson
034197a81f
Fix typos in csv feeder docs (in manifest)
2025-02-04 13:40:07 +01:00
Patrick Robertson
78e6418249
Unit tests for csv feeder + fix some bugs
2025-02-04 13:37:26 +01:00
Patrick Robertson
b301f60ea3
Fix using validators set in __manifest__.py
...
E.g. you can use the validator 'is_file' to check if a config is a valid file
2025-02-04 13:37:26 +01:00
Patrick Robertson
a873e56b87
Remove old csv_feeder file - now inside a module
2025-02-04 12:57:35 +01:00
Patrick Robertson
72b5ea9ab6
Restore headless arg
2025-02-03 17:40:40 +01:00
Patrick Robertson
c574b694ed
Set up screenshot enricher to use authentication/cookies
2025-02-03 17:25:59 +01:00
Patrick Robertson
7ec328ab40
Remove cookie options from generic_extractor - it now uses 'authentication' global settings :D
2025-02-03 16:04:36 +01:00
Patrick Robertson
7a2be5a0da
Add cookie extraction to 'authentication' options, get generic_extractor working using this info
2025-02-03 16:03:07 +01:00
Patrick Robertson
9c9e9b370e
Remove lingering reference to ArchivingContext
2025-02-03 16:02:38 +01:00
Patrick Robertson
9a8c94b641
Fix getting/setting folder context for metadata
2025-02-03 16:02:17 +01:00
Patrick Robertson
c25d5cae84
Remove ArchivingContext completely
...
Context for a specific url/item is now passed around via the metadata (metadata.set_context('key', 'val') and metadata.get_context('key', default='something')
The only other thing that was passed around in ArchivingContext was the storage info, which is already accessible now via self.config
2025-01-30 17:50:54 +01:00
Patrick Robertson
d76063c3f3
Fix unit tests
2025-01-30 16:46:53 +01:00
Patrick Robertson
d6b4b7a932
Further cleanup
...
* Removes (partly) the ArchivingOrchestrator
* Removes the cli_feeder module, and makes it the 'default', allowing you to pass URLs directly on the command line, without having to use the cumbersome --cli_feeder.urls. Just do auto-archiver https://my.url.com
* More unit tests
* Improved error handling
2025-01-30 16:44:40 +01:00
Patrick Robertson
953011f368
Don't make modules 'dataclasses'
2025-01-30 16:44:40 +01:00
erinhmclark
527438826c
Fix manifests for required configs.
2025-01-30 13:04:51 +00:00
Patrick Robertson
fade68c6f4
Fix up unit tests - dataclass + subclasses not having @dataclass was breaking it
2025-01-30 13:45:24 +01:00
Patrick Robertson
b7d9145f6c
Further tidyups + refactoring for new structure
...
* Add implementation tests for orchestrator + logging tests
* Standardise method/class vars for extractors to see if they are suitable
* Fix bugs with removing default loguru logger (allows further customisation)
* Fix bug loading required fields from file
*
2025-01-30 13:21:10 +01:00
erinhmclark
cddae65a90
Update modules for new core structure.
2025-01-30 08:42:23 +00:00
Patrick Robertson
18ff36ce15
Add ruamel to dependencies (replaces pyyaml)
2025-01-29 19:37:41 +01:00
Patrick Robertson
00a7018f36
Fix up dependency checking (use 'dependencies' instead of 'external_dependencies' -> simpler/easier to remember
2025-01-29 19:25:22 +01:00
Patrick Robertson
3d37c494aa
Tidy ups + unit tests:
...
1. Allow loading modules from --module_paths=/extra/path/here
2. Improved unit tests for module loading
3. Further small tidy ups/clean ups
2025-01-29 18:42:49 +01:00
Patrick Robertson
dcd5576f29
set metadata enricher to requires_setup=True (requires exiftool which isn't installed by default on most machines)
2025-01-29 00:10:40 +01:00
Patrick Robertson
7a4871db6b
Fix up unit tests for new structure
2025-01-28 14:40:12 +01:00
Patrick Robertson
9635449ac0
more user friendly error logging when config issues are found
2025-01-28 11:44:52 +01:00
Patrick Robertson
27b25c5bd4
Validate orchestration.yaml file inputs - so if a user enters invalid values, it also validates them
2025-01-28 11:37:23 +01:00
Patrick Robertson
1d2a1d4db7
Allow framework for config settings that should not be stored in config (e.g. cli_feeder.urls
...
Use 'do_not_store': True in the config settings to apply this. Also: fix up generic archiver dropins loading + local_storage defaults (same as what's in example orchestration)
2025-01-28 11:14:12 +01:00
erinhmclark
57b3bec935
Google sheets feeder and database implemented.
2025-01-27 20:13:12 +00:00
erinhmclark
6c67effd8c
remove name reference in local_storage.py
2025-01-27 19:17:18 +00:00
erinhmclark
e1a9373336
Refactoring for new config setup
2025-01-27 19:03:02 +00:00
Patrick Robertson
e3074013d0
Fix loading/saving to orchestration file with comments
2025-01-27 14:28:04 +01:00
Patrick Robertson
f68e2726f2
Refactor loader + step into module, use LazyBaseModule and BaseModule
2025-01-27 14:01:36 +01:00
Patrick Robertson
7fd95866a1
Further fixes/changes to loading 'types' for config + manifest edits
2025-01-27 11:48:04 +01:00
Patrick Robertson
14e2479599
Merge branch 'more_mainifests' into load_modules
2025-01-27 11:05:56 +01:00
Patrick Robertson
0b03f54f4e
Fix up config validation, and allow for custom 'validators'
2025-01-27 11:00:52 +01:00
erinhmclark
ebebd27897
Fix archiver to extractor naming
2025-01-27 09:11:45 +00:00
erinhmclark
21a7ff0520
Fix types in manifests
2025-01-27 08:43:18 +00:00
erinhmclark
96b35a272c
Rm gsheet references in utils
2025-01-24 18:51:15 +00:00
erinhmclark
dd402b456f
Fix and add types to manifest
2025-01-24 18:50:11 +00:00
Patrick Robertson
3fc6ddfe85
Tweaks to logging strings
2025-01-24 15:30:00 +01:00
Patrick Robertson
f1e9ab6751
Merge branch 'main' into load_modules
2025-01-24 15:23:15 +01:00
Patrick Robertson
e8138eac1c
Add ubuntu-latest to the matrix of test runners ( #181 )
...
* Don't clutter logs with info about generic dropin
* Add ubuntu-latest to unit tests
This is currently failing due to an issue with oscrypto and newer openssl https://github.com/wbond/oscrypto/issues/78#issuecomment-1756317472
* fix oscrypto version for ubuntu 24 compatibility (boto3 too see #180 )
---------
Co-authored-by: msramalho <19508417+msramalho@users.noreply.github.com >
2025-01-24 14:03:55 +00:00
Miguel Sozinho Ramalho
a6fc4e1bb1
modifies base docker image to use browsertrix 1.4.2 ( #182 )
...
* modifies base image to newest browsertrix version
* modify browsertrix cmd args based on recent experience
2025-01-24 13:59:29 +00:00
erinhmclark
1942e8b819
Gsheets utility revert
2025-01-24 13:34:30 +00:00
erinhmclark
024fe58377
fix config parsing in manifests, remove module level configs
2025-01-24 13:33:12 +00:00
erinhmclark
0453d95f56
fix config parsing in manifests
2025-01-24 13:24:54 +00:00
erinhmclark
aa7ca93a43
Update manifests and modules
2025-01-24 12:58:16 +00:00
