bellingcat/auto-archiver
1
0
Fork 0
mirror of https://github.com/bellingcat/auto-archiver.git synced 2026-06-10 12:18:30 +03:00
Code Issues Packages Projects Releases Wiki Activity
Files
600f43e790d956faf2ddf2ca4a01dcf3dfd802a9
auto-archiver/docs/source/core_modules.md
Patrick Robertson 3c543a3a6a Various fixes for issues with new architecture (#208) 
* Add formatters to the TOC - fixes #204

* Add 'steps' settings to the example YAML in the docs. Fixes #206

* Improved docs on authentication architecture

* Fix setting modules on the command line - they now override any module settings in the orchestration as opposed to appending

* Fix tests for gsheet-feeder: add a test service_account.json (note: not real keys in there)

* Rename the command line entrypoint to _command_line_run

Also: make it clear that code implementation should not call this
Make sure the command line entry returns (we don't want a generator)

* Fix unit tests to use now code-entry points

* Version bump

* Move iterating of generator up to __main__

* Breakpoint

* two minor fixes

* Fix unit tests + add new '__main__' entry point implementation test

* Skip youtube tests if running on CI. Should still run them locally

* Fix full implementation run on GH actions

* Fix skipif test for GH Actions CI

* Add skipifs for truth - it blocks GH:

---------

Co-authored-by: msramalho <19508417+msramalho@users.noreply.github.com>
2025-02-18 19:10:09 +00:00

928 B
Raw Blame History

Module Documentation

These pages describe the core modules that come with auto-archiver and provide the main functionality for archiving websites on the internet. There are five core module types:

  1. Feeders - these 'feed' information (the URLs) from various sources to the auto-archiver for processing
  2. Extractors - these 'extract' the page data for a given URL that is fed in by a feeder
  3. Enrichers - these 'enrich' the data extracted in the previous step with additional information
  4. Storage - these 'store' the data in a persistent location (on disk, Google Drive etc.)
  5. Databases - these 'store' the status of the entire archiving process in a log file or database.
:maxdepth: 1
:caption: Core Modules
:hidden:

modules/config_cheatsheet
modules/feeder
modules/extractor
modules/enricher
modules/storage
modules/database
modules/formatter