bellingcat/auto-archiver
1
0
Fork 0
mirror of https://github.com/bellingcat/auto-archiver.git synced 2026-06-11 20:58:29 +03:00
Code Issues Packages Projects Releases Wiki Activity

Various fixes for issues with new architecture (#208)

Browse Source 
* Add formatters to the TOC - fixes #204

* Add 'steps' settings to the example YAML in the docs. Fixes #206

* Improved docs on authentication architecture

* Fix setting modules on the command line - they now override any module settings in the orchestration as opposed to appending

* Fix tests for gsheet-feeder: add a test service_account.json (note: not real keys in there)

* Rename the command line entrypoint to _command_line_run

Also: make it clear that code implementation should not call this
Make sure the command line entry returns (we don't want a generator)

* Fix unit tests to use now code-entry points

* Version bump

* Move iterating of generator up to __main__

* Breakpoint

* two minor fixes

* Fix unit tests + add new '__main__' entry point implementation test

* Skip youtube tests if running on CI. Should still run them locally

* Fix full implementation run on GH actions

* Fix skipif test for GH Actions CI

* Add skipifs for truth - it blocks GH:

---------

Co-authored-by: msramalho <19508417+msramalho@users.noreply.github.com>
This commit is contained in:
Patrick Robertson
2025-02-18 19:10:09 +00:00
committed by GitHub
parent 6d43bc7d4d
commit 3c543a3a6a
18 changed files with 314 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
docs/scripts/scripts.py
View File

@@ -19,6 +19,19 @@ type_color = {
TABLE_HEADER = ("Option", "Description", "Default", "Type")
EXAMPLE_YAML = """
# steps configuration
steps:
...
{steps_str}
...
# module configuration
...
{config_string}
"""
def generate_module_docs():
yaml = YAML()
SAVE_FOLDER.mkdir(exist_ok=True)
@@ -45,11 +58,14 @@ def generate_module_docs():
```
{description}
"""
steps_str = "\n".join(f" {t}s:\n - {module.name}" for t in manifest['type'])
if not manifest['configs']:
readme_str += "\n*This module has no configuration options.*\n"
config_string = f"# No configuration options for {module.name}.*\n"
else:
config_yaml = {}
config_table = header_row
config_yaml = {}
for key, value in manifest['configs'].items():
type = value.get('type', 'string')
if type == 'auto_archiver.utils.json_loader':
@@ -65,11 +81,14 @@ def generate_module_docs():
configs_cheatsheet += f"| `{module.name}.{key}` | {help} | {default} | {type} |\n"
readme_str += "\n## Configuration Options\n"
readme_str += "\n### YAML\n"
yaml_string = io.BytesIO()
yaml.dump({module.name: config_yaml}, yaml_string)
readme_str += f"```{{code}} yaml\n{yaml_string.getvalue().decode('utf-8')}\n```\n"
config_string = io.BytesIO()
yaml.dump({module.name: config_yaml}, config_string)
config_string = config_string.getvalue().decode('utf-8')
yaml_string = EXAMPLE_YAML.format(steps_str=steps_str, config_string=config_string)
readme_str += f"```{{code}} yaml\n{yaml_string}\n```\n"
if manifest['configs']:
readme_str += "\n### Command Line:\n"
readme_str += config_table
@@ -103,3 +122,7 @@ def generate_index(modules_by_type):
with open(SAVE_FOLDER / "module_list.md", "w") as f:
print("writing", SAVE_FOLDER / "module_list.md")
f.write(readme_str)
if __name__ == "__main__":
generate_module_docs()