bellingcat/auto-archiver
1
0
Fork 0
mirror of https://github.com/bellingcat/auto-archiver.git synced 2026-06-11 20:58:29 +03:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' into opentimestamps

Browse Source
This commit is contained in:
Patrick Robertson
2025-03-11 17:21:34 +00:00
parent 37eac64442 e2442b2f6b
commit 94543e9a67
193 changed files with 61842 additions and 3011 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
.github/workflows/docker-publish.yaml vendored
View File

@@ -11,7 +11,7 @@ on:
env:
# Use docker.io for Docker Hub if empty
REGISTRY: ghcr.io
REGISTRY: docker.io
# github.repository as <account>/<repo>
IMAGE_NAME: ${{ github.repository }}
@@ -45,10 +45,12 @@ jobs:
images: bellingcat/auto-archiver
- name: Build and push Docker image
uses: docker/build-push-action@v2
uses: docker/build-push-action@v6
with:
context: .
platforms: linux/amd64,linux/arm64
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:cache
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:cache,mode=max

7
.github/workflows/tests-core.yaml vendored
View File

@@ -5,9 +5,13 @@ on:
branches: [ main ]
paths:
- src/**
- poetry.lock
- pyproject.toml
pull_request:
paths:
- src/**
- poetry.lock
- pyproject.toml
jobs:
tests:
@@ -16,7 +20,8 @@ jobs:
fail-fast: false
matrix:
python-version: ["3.10", "3.11", "3.12"]
os: [ubuntu-22.04, ubuntu-latest]
os: [ubuntu-22.04]
#TODO: re-enable ubuntu-latest, this is disabled as oscrypto cannot be pinned to github commit and pushed to pypi
defaults:
run:
working-directory: ./

3
.gitignore vendored
View File

@@ -32,3 +32,6 @@ archived/
dist*
docs/_build/
docs/source/autoapi/
docs/source/modules/autogen/
scripts/settings_page.html
.vite

8
.readthedocs.yaml
View File

@@ -9,13 +9,19 @@ build:
os: ubuntu-22.04
tools:
python: "3.10"
nodejs: "22"
jobs:
post_install:
- pip install poetry
# https://python-poetry.org/docs/managing-dependencies/#dependency-groups
# VIRTUAL_ENV needs to be set manually for now.
# See https://github.com/readthedocs/readthedocs.org/pull/11152/
- VIRTUAL_ENV=$READTHEDOCS_VIRTUALENV_PATH poetry install --only docs
- VIRTUAL_ENV=$READTHEDOCS_VIRTUALENV_PATH poetry install --with docs
# generate the config editor page. Schema then HTML
- VIRTUAL_ENV=$READTHEDOCS_VIRTUALENV_PATH poetry run python scripts/generate_settings_schema.py
# install node dependencies and build the settings
- cd scripts/settings && npm install && npm run build && yes | cp dist/index.html ../../docs/source/installation/settings_base.html && cd ../..
sphinx:

49
CONTRIBUTING.md Normal file
View File

@@ -0,0 +1,49 @@
# Contributing to Auto Archiver
Thank you for your interest in contributing to Auto Archiver! Your contributions help improve the project and make it more useful for everyone. Please follow the guidelines below to ensure a smooth collaboration.
### 1. Reporting a Bug
If you encounter a bug, please create an issue on GitHub with the following details:
* Describe the bug: Provide a clear and concise description of the issue.
* Steps to reproduce: Include the steps needed to reproduce the bug.
* Expected behavior: Describe what you expected to happen.
* Actual behavior: Explain what actually happened.
* Screenshots/logs: If applicable, attach screenshots or logs to help diagnose the problem.
* Environment: Mention the OS, Ruby version, and any other relevant details.
### 2. Writing a Patch/Fix and Submitting Pull Requests
If you’d like to fix a bug or improve existing code:
1. Open a pull request on GitHub and link it to the relevant issue.
2. Make sure to document your pull request with a clear description of what changes were made and why.
3. Wait for review and make any requested changes.
### 3. Creating New Modules
If you want to add a new module to Auto Archiver:
1. Ensure your module follows the existing [coding style and project structure](https://auto-archiver.readthedocs.io/en/latest/development/creating_modules.html).
2. Write clear documentation explaining what your module does and how to use it.
3. Ideally, include unit tests for your module!
4. Follow the steps in Section 2 to submit a pull request.
### 4. Do You Have Questions About the Source Code?
If you have any questions about how the source code works or need help using Auto Archiver
📝 Check the [Auto Archiver](https://auto-archiver.readthedocs.io/en/latest/) documentation.
👉 Ask your questions in the [Bellingcat Discord](https://www.bellingcat.com/follow-bellingcat-on-social-media/).
### 5. Do You Want to Contribute to the Documentation?
We welcome contributions to the documentation!
📖 Please read [Contributing to the Auto Archiver Documentation](https://auto-archiver.readthedocs.io/en/latest/development/docs.html) to learn how you can help improve the project's documentation.
------------------
Thank you for contributing to Auto Archiver! 🚀

15
Dockerfile
View File

@@ -7,13 +7,24 @@ ENV RUNNING_IN_DOCKER=1 \
PYTHONFAULTHANDLER=1 \
PATH="/root/.local/bin:$PATH"
ARG TARGETARCH
# Installing system dependencies
RUN add-apt-repository ppa:mozillateam/ppa && \
apt-get update && \
apt-get install -y --no-install-recommends gcc ffmpeg fonts-noto exiftool && \
apt-get install -y --no-install-recommends firefox-esr && \
ln -s /usr/bin/firefox-esr /usr/bin/firefox && \
wget https://github.com/mozilla/geckodriver/releases/download/v0.35.0/geckodriver-v0.35.0-linux64.tar.gz && \
ln -s /usr/bin/firefox-esr /usr/bin/firefox
ARG GECKODRIVER_VERSION=0.36.0
RUN if [ $(uname -m) = "aarch64" ]; then \
GECKODRIVER_ARCH=linux-aarch64; \
else \
GECKODRIVER_ARCH=linux64; \
fi && \
wget https://github.com/mozilla/geckodriver/releases/download/v${GECKODRIVER_VERSION}/geckodriver-v${GECKODRIVER_VERSION}-${GECKODRIVER_ARCH}.tar.gz && \
tar -xvzf geckodriver* -C /usr/local/bin && \
chmod +x /usr/local/bin/geckodriver && \
rm geckodriver-v* && \

330
README.md
View File

@@ -9,327 +9,33 @@
<!-- [![Documentation Status](https://readthedocs.org/projects/vk-url-scraper/badge/?version=latest)](https://vk-url-scraper.readthedocs.io/en/latest/?badge=latest) -->
Auto Archiver is a Python tool to automatically archive content on the web in a secure and verifiable way. It takes URLs from different sources (e.g. a CSV file, Google Sheets, command line etc.) and archives the content of each one. It can archive social media posts, videos, images and webpages. Content can be enriched, then saved either locally or remotely (S3 bucket, Google Drive). The status of the archiving process can be appended to a CSV report, or if using Google Sheets – back to the original sheet.
<div class="hidden_rtd">
**[See the Auto Archiver documentation for more information.](https://auto-archiver.readthedocs.io/en/latest/)**
</div>
Read the [article about Auto Archiver on bellingcat.com](https://www.bellingcat.com/resources/2022/09/22/preserve-vital-online-content-with-bellingcats-auto-archiver-tool/).
Python tool to automatically archive social media posts, videos, and images from a Google Sheets, the console, and more. Uses different archivers depending on the platform, and can save content to local storage, S3 bucket (Digital Ocean Spaces, AWS, ...), and Google Drive. If using Google Sheets as the source for links, it will be updated with information about the archived content. It can be run manually or on an automated basis.
## Installation
There are 3 ways to use the auto-archiver:
1. (easiest installation) via docker
2. (local python install) `pip install auto-archiver`
3. (legacy/development) clone and manually install from repo (see legacy [tutorial video](https://youtu.be/VfAhcuV2tLQ))
View the [Installation Guide](https://auto-archiver.readthedocs.io/en/latest/installation/installation.html) for full instructions
But **you always need a configuration/orchestration file**, which is where you'll configure where/what/how to archive. Make sure you read [orchestration](#orchestration).
**Advanced:**
To get started quickly using Docker:
## How to install and run the auto-archiver
`docker pull bellingcat/auto-archiver && docker run --rm -v secrets:/app/secrets bellingcat/auto-archiver --config secrets/orchestration.yaml`
### Option 1 - docker
Or pip:
[![dockeri.co](https://dockerico.blankenship.io/image/bellingcat/auto-archiver)](https://hub.docker.com/r/bellingcat/auto-archiver)
`pip install auto-archiver && auto-archiver --help`
Docker works like a virtual machine running inside your computer, it isolates everything and makes installation simple. Since it is an isolated environment when you need to pass it your orchestration file or get downloaded media out of docker you will need to connect folders on your machine with folders inside docker with the `-v` volume flag.
## Contributing
We welcome contributions to the Auto Archiver project! See the [Contributing Guide](https://auto-archiver.readthedocs.io/en/latest/contributing.html) for how to get involved!
1. install [docker](https://docs.docker.com/get-docker/)
2. pull the auto-archiver docker [image](https://hub.docker.com/r/bellingcat/auto-archiver) with `docker pull bellingcat/auto-archiver`
3. run the docker image locally in a container: `docker run --rm -v $PWD/secrets:/app/secrets -v $PWD/local_archive:/app/local_archive bellingcat/auto-archiver --config secrets/orchestration.yaml` breaking this command down:
1. `docker run` tells docker to start a new container (an instance of the image)
2. `--rm` makes sure this container is removed after execution (less garbage locally)
3. `-v $PWD/secrets:/app/secrets` - your secrets folder
1. `-v` is a volume flag which means a folder that you have on your computer will be connected to a folder inside the docker container
2. `$PWD/secrets` points to a `secrets/` folder in your current working directory (where your console points to), we use this folder as a best practice to hold all the secrets/tokens/passwords/... you use
3. `/app/secrets` points to the path the docker container where this image can be found
4. `-v $PWD/local_archive:/app/local_archive` - (optional) if you use local_storage
1. `-v` same as above, this is a volume instruction
2. `$PWD/local_archive` is a folder `local_archive/` in case you want to archive locally and have the files accessible outside docker
3. `/app/local_archive` is a folder inside docker that you can reference in your orchestration.yml file
### Option 2 - python package
<details><summary><code>Python package instructions</code></summary>
1. make sure you have python 3.10 or higher installed
2. install the package with your preferred package manager: `pip/pipenv/conda install auto-archiver` or `poetry add auto-archiver`
3. test it's installed with `auto-archiver --help`
4. run it with your orchestration file and pass any flags you want in the command line `auto-archiver --config secrets/orchestration.yaml` if your orchestration file is inside a `secrets/`, which we advise
You will also need [ffmpeg](https://www.ffmpeg.org/), [firefox](https://www.mozilla.org/en-US/firefox/new/) and [geckodriver](https://github.com/mozilla/geckodriver/releases), and optionally [fonts-noto](https://fonts.google.com/noto). Similar to the local installation.
</details>
### Option 3 - local installation
This can also be used for development.
<details><summary><code>Legacy instructions, only use if docker/package is not an option</code></summary>
Install the following locally:
1. [ffmpeg](https://www.ffmpeg.org/) must also be installed locally for this tool to work.
2. [firefox](https://www.mozilla.org/en-US/firefox/new/) and [geckodriver](https://github.com/mozilla/geckodriver/releases) on a path folder like `/usr/local/bin`.
3. [Poetry](https://python-poetry.org/docs/#installation) for dependency management and packaging.
4. (optional) [fonts-noto](https://fonts.google.com/noto) to deal with multiple unicode characters during selenium/geckodriver's screenshots: `sudo apt install fonts-noto -y`.
Clone and run:
1. `git clone https://github.com/bellingcat/auto-archiver`
2. `poetry install`
3. `poetry run python -m src.auto_archiver --config secrets/orchestration.yaml`
Note: Add the plugin [poetry-shell-plugin](https://github.com/python-poetry/poetry-plugin-shell) and run `poetry shell` to activate the virtual environment.
This allows you to run the auto-archiver without the `poetry run` prefix.
</details><br/>
# Orchestration
The archiver work is orchestrated by the following workflow (we call each a **step**):
1. **Feeder** gets the links (from a spreadsheet, from the console, ...)
2. **Archiver** tries to archive the link (twitter, youtube, ...)
3. **Enricher** adds more info to the content (hashes, thumbnails, ...)
4. **Formatter** creates a report from all the archived content (HTML, PDF, ...)
5. **Database** knows what's been archived and also stores the archive result (spreadsheet, CSV, or just the console)
To setup an auto-archiver instance create an `orchestration.yaml` which contains the workflow you would like. We advise you put this file into a `secrets/` folder and do not share it with others because it will contain passwords and other secrets.
The structure of orchestration file is split into 2 parts: `steps` (what **steps** to use) and `configurations` (how those steps should behave), here's a simplification:
```yaml
# orchestration.yaml content
steps:
feeder: gsheet_feeder
archivers: # order matters
- youtubedl_archiver
enrichers:
- thumbnail_enricher
formatter: html_formatter
storages:
- local_storage
databases:
- gsheet_db
configurations:
gsheet_feeder:
sheet: "your google sheet name"
header: 2 # row with header for your sheet
# ... configurations for the other steps here ...
```
To see all available `steps` (which archivers, storages, databases, ...) exist check the [example.orchestration.yaml](example.orchestration.yaml).
All the `configurations` in the `orchestration.yaml` file (you can name it differently but need to pass it in the `--config FILENAME` argument) can be seen in the console by using the `--help` flag. They can also be overwritten, for example if you are using the `cli_feeder` to archive from the command line and want to provide the URLs you should do:
```bash
auto-archiver --config secrets/orchestration.yaml --cli_feeder.urls="url1,url2,url3"
```
Here's the complete workflow that the auto-archiver goes through:
```{mermaid}
graph TD
s((start)) --> F(fa:fa-table Feeder)
F -->|get and clean URL| D1{fa:fa-database Database}
D1 -->|is already archived| e((end))
D1 -->|not yet archived| a(fa:fa-download Archivers)
a -->|got media| E(fa:fa-chart-line Enrichers)
E --> S[fa:fa-box-archive Storages]
E --> Fo(fa:fa-code Formatter)
Fo --> S
Fo -->|update database| D2(fa:fa-database Database)
D2 --> e
```
## Orchestration checklist
Use this to make sure you help making sure you did all the required steps:
* [ ] you have a `/secrets` folder with all your configuration files including
* [ ] a orchestration file eg: `orchestration.yaml` pointing to the correct location of other files
* [ ] (optional if you use GoogleSheets) you have a `service_account.json` (see [how-to](https://gspread.readthedocs.io/en/latest/oauth2.html#for-bots-using-service-account))
* [ ] (optional for telegram) a `anon.session` which appears after the 1st run where you login to telegram
* if you use private channels you need to add `channel_invites` and set `join_channels=true` at least once
* [ ] (optional for VK) a `vk_config.v2.json`
* [ ] (optional for using GoogleDrive storage) `gd-token.json` (see [help script](scripts/create_update_gdrive_oauth_token.py))
* [ ] (optional for instagram) `instaloader.session` file which appears after the 1st run and login in instagram
* [ ] (optional for browsertrix) `profile.tar.gz` file
#### Example invocations
The recommended way to run the auto-archiver is through Docker. The invocations below will run the auto-archiver Docker image using a configuration file that you have specified
```bash
# all the configurations come from ./secrets/orchestration.yaml
docker run --rm -v $PWD/secrets:/app/secrets -v $PWD/local_archive:/app/local_archive bellingcat/auto-archiver --config secrets/orchestration.yaml
# uses the same configurations but for another google docs sheet
# with a header on row 2 and with some different column names
# notice that columns is a dictionary so you need to pass it as JSON and it will override only the values provided
docker run --rm -v $PWD/secrets:/app/secrets -v $PWD/local_archive:/app/local_archive bellingcat/auto-archiver --config secrets/orchestration.yaml --gsheet_feeder.sheet="use it on another sheets doc" --gsheet_feeder.header=2 --gsheet_feeder.columns='{"url": "link"}'
# all the configurations come from orchestration.yaml and specifies that s3 files should be private
docker run --rm -v $PWD/secrets:/app/secrets -v $PWD/local_archive:/app/local_archive bellingcat/auto-archiver --config secrets/orchestration.yaml --s3_storage.private=1
```
The auto-archiver can also be run locally, if pre-requisites are correctly configured. Equivalent invocations are below.
```bash
# all the configurations come from ./secrets/orchestration.yaml
auto-archiver --config secrets/orchestration.yaml
# uses the same configurations but for another google docs sheet
# with a header on row 2 and with some different column names
# notice that columns is a dictionary so you need to pass it as JSON and it will override only the values provided
auto-archiver --config secrets/orchestration.yaml --gsheet_feeder.sheet="use it on another sheets doc" --gsheet_feeder.header=2 --gsheet_feeder.columns='{"url": "link"}'
# all the configurations come from orchestration.yaml and specifies that s3 files should be private
auto-archiver --config secrets/orchestration.yaml --s3_storage.private=1
```
### Extra notes on configuration
#### Google Drive
To use Google Drive storage you need the id of the shared folder in the `config.yaml` file which must be shared with the service account eg `autoarchiverservice@auto-archiver-111111.iam.gserviceaccount.com` and then you can use `--storage=gd`
#### Telethon + Instagram with telegram bot
The first time you run, you will be prompted to do a authentication with the phone number associated, alternatively you can put your `anon.session` in the root.
#### Atlos
When integrating with [Atlos](https://atlos.org), you will need to provide an API token in your configuration. You can learn more about Atlos and how to get an API token [here](https://docs.atlos.org/technical/api). You will have to provide this token to the `atlos_feeder`, `atlos_storage`, and `atlos_db` steps in your orchestration file. If you use a custom or self-hosted Atlos instance, you can also specify the `atlos_url` option to point to your custom instance's URL. For example:
```yaml
# orchestration.yaml content
steps:
feeder: atlos_feeder
archivers: # order matters
- youtubedl_archiver
enrichers:
- thumbnail_enricher
- hash_enricher
formatter: html_formatter
storages:
- atlos_storage
databases:
- console_db
- atlos_db
configurations:
atlos_feeder:
atlos_url: "https://platform.atlos.org" # optional
api_token: "...your API token..."
atlos_db:
atlos_url: "https://platform.atlos.org" # optional
api_token: "...your API token..."
atlos_storage:
atlos_url: "https://platform.atlos.org" # optional
api_token: "...your API token..."
hash_enricher:
algorithm: "SHA-256"
```
## Running on Google Sheets Feeder (gsheet_feeder)
The `--gsheet_feeder.sheet` property is the name of the Google Sheet to check for URLs.
This sheet must have been shared with the Google Service account used by `gspread`.
This sheet must also have specific columns (case-insensitive) in the `header` as specified in [gsheet_feeder.__manifest__.py](src/auto_archiver/modules/gsheet_feeder/__manifest__.py). The default names of these columns and their purpose is:
Inputs:
* **Link** *(required)*: the URL of the post to archive
* **Destination folder**: custom folder for archived file (regardless of storage)
Outputs:
* **Archive status** *(required)*: Status of archive operation
* **Archive location**: URL of archived post
* **Archive date**: Date archived
* **Thumbnail**: Embeds a thumbnail for the post in the spreadsheet
* **Timestamp**: Timestamp of original post
* **Title**: Post title
* **Text**: Post text
* **Screenshot**: Link to screenshot of post
* **Hash**: Hash of archived HTML file (which contains hashes of post media) - for checksums/verification
* **Perceptual Hash**: Perceptual hashes of found images - these can be used for de-duplication of content
* **WACZ**: Link to a WACZ web archive of post
* **ReplayWebpage**: Link to a ReplayWebpage viewer of the WACZ archive
For example, this is a spreadsheet configured with all of the columns for the auto archiver and a few URLs to archive. (Note that the column names are not case sensitive.)
![A screenshot of a Google Spreadsheet with column headers defined as above, and several Youtube and Twitter URLs in the "Link" column](docs/demo-before.png)
Now the auto archiver can be invoked, with this command in this example: `docker run --rm -v $PWD/secrets:/app/secrets -v $PWD/local_archive:/app/local_archive bellingcat/auto-archiver:dockerize --config secrets/orchestration-global.yaml --gsheet_feeder.sheet "Auto archive test 2023-2"`. Note that the sheet name has been overridden/specified in the command line invocation.
When the auto archiver starts running, it updates the "Archive status" column.
![A screenshot of a Google Spreadsheet with column headers defined as above, and several Youtube and Twitter URLs in the "Link" column. The auto archiver has added "archive in progress" to one of the status columns.](docs/demo-progress.png)
The links are downloaded and archived, and the spreadsheet is updated to the following:
![A screenshot of a Google Spreadsheet with videos archived and metadata added per the description of the columns above.](docs/demo-after.png)
Note that the first row is skipped, as it is assumed to be a header row (`--gsheet_feeder.header=1` and you can change it if you use more rows above). Rows with an empty URL column, or a non-empty archive column are also skipped. All sheets in the document will be checked.
The "archive location" link contains the path of the archived file, in local storage, S3, or in Google Drive.
![The archive result for a link in the demo sheet.](docs/demo-archive.png)
---
## Development
Use `python -m src.auto_archiver --config secrets/orchestration.yaml` to run from the local development environment.
### Testing
Tests are split using `pytest.mark` into 'core' and 'download' tests. Download tests will hit the network and make API calls (e.g. Twitter, Bluesky etc.) and should be run regularly to make sure that APIs have not changed.
Tests can be run as follows:
```
# run core tests
pytest -ra -v -m "not download" # or poetry run pytest -ra -v -m "not download"
# run download tests
pytest -ra -v -m "download" # or poetry run pytest -ra -v -m "download"
# run all tests
pytest -ra -v # or poetry run pytest -ra -v
```
#### Docker development
working with docker locally:
* `docker compose up` to build the first time and run a local image with the settings in `secrets/orchestration.yaml`
* To modify/pass additional command line args, use `docker compose run auto-archiver --config secrets/orchestration.yaml [OTHER ARGUMENTS]`
* To rebuild after code changes, just pass the `--build` flag, e.g. `docker compose up --build`
manual release to docker hub
* `docker image tag auto-archiver bellingcat/auto-archiver:latest`
* `docker push bellingcat/auto-archiver`
### Building the Docs
The documentation is built using [Sphinx](https://www.sphinx-doc.org/en/master/) and [AutoAPI](https://sphinx-autoapi.readthedocs.io/en/latest/) and hosted on ReadTheDocs.
To build the documentation locally, run the following commands:
**Install required dependencies:**
- Install the docs group of dependencies:
```shell
# only the docs dependencies
poetry install --only docs
# or for all dependencies
poetry install
```
- Either use [poetry-plugin-shell](https://github.com/python-poetry/poetry-plugin-shell) to activate the virtual environment: `poetry shell`
- Or prepend the following commands with `poetry run`
**Create the documentation:**
- Build the documentation:
```
# Using makefile (Linux/macOS):
make -C docs html
# or using sphinx directly (Windows/Linux/macOS):
sphinx-build -b html docs/source docs/_build/html
```
- If you make significant changes and want a fresh build run: `make -C docs clean` to remove the old build files.
**Viewing the documentation:**
```shell
# to open the documentation in your browser.
open docs/_build/html/index.html
# or run autobuild to automatically update the documentation when you make changes
sphinx-autobuild docs/source docs/_build/html
```
#### RELEASE
* update version in [version.py](src/auto_archiver/version.py)
* go to github releases > new release > use `vx.y.z` for matching version notation
* package is automatically updated in pypi
* docker image is automatically pushed to dockerhup

4
docs/_static/custom.css vendored Normal file
View File

@@ -0,0 +1,4 @@
.hidden_rtd {
display:none;
}

46
docs/_templates/autoapi/index.rst vendored Normal file
View File

@@ -0,0 +1,46 @@
API Reference
=============
These pages are intended for developers of the `auto-archiver` package,
and include documentation on the core classes and functions used by
the auto-archiver
Core Classes
------------
.. toctree::
:titlesonly:
{% for page in pages|selectattr("is_top_level_object") %}
{% if page.name == 'core' %}
{{ page.include_path }}
{% endif %}
{% endfor %}
Util Functions
--------------
.. toctree::
:titlesonly:
{% for page in pages|selectattr("is_top_level_object") %}
{% if page.name == 'utils' %}
{{ page.include_path }}
{% endif %}
{% endfor %}
Core Modules
------------
.. toctree::
:titlesonly:
{% for page in pages|selectattr("is_top_level_object") %}
{% if page.name != 'core' and page.name != 'utils' %}
{{ page.include_path }}
{% endif %}
{% endfor %}

1
docs/_templates/autoapi/python/attribute.rst vendored Normal file
View File

@@ -0,0 +1 @@
{% extends "python/data.rst" %}

104
docs/_templates/autoapi/python/class.rst vendored Normal file
View File

@@ -0,0 +1,104 @@
{% if obj.display %}
{% if is_own_page %}
{{ obj.id }}
{{ "=" * obj.id | length }}
{% endif %}
{% set visible_children = obj.children|selectattr("display")|list %}
{% set own_page_children = visible_children|selectattr("type", "in", own_page_types)|list %}
{% if is_own_page and own_page_children %}
.. toctree::
:hidden:
{% for child in own_page_children %}
{{ child.include_path }}
{% endfor %}
{% endif %}
.. py:{{ obj.type }}:: {% if is_own_page %}{{ obj.id }}{% else %}{{ obj.short_name }}{% endif %}{% if obj.args %}({{ obj.args }}){% endif %}
{% for (args, return_annotation) in obj.overloads %}
{{ " " * (obj.type | length) }} {{ obj.short_name }}{% if args %}({{ args }}){% endif %}
{% endfor %}
{% if obj.bases %}
{% if "show-inheritance" in autoapi_options %}
Bases: {% for base in obj.bases %}{{ base|link_objs }}{% if not loop.last %}, {% endif %}{% endfor %}
{% endif %}
{% if "show-inheritance-diagram" in autoapi_options and obj.bases != ["object"] %}
.. autoapi-inheritance-diagram:: {{ obj.obj["full_name"] }}
:parts: 1
{% if "private-members" in autoapi_options %}
:private-bases:
{% endif %}
{% endif %}
{% endif %}
{% if obj.docstring %}
{{ obj.docstring|indent(3) }}
{% endif %}
{% for obj_item in visible_children %}
{% if obj_item.type not in own_page_types %}
{{ obj_item.render()|indent(3) }}
{% endif %}
{% endfor %}
{% if is_own_page and own_page_children %}
{% set visible_attributes = own_page_children|selectattr("type", "equalto", "attribute")|list %}
{% if visible_attributes %}
Attributes
----------
.. autoapisummary::
{% for attribute in visible_attributes %}
{{ attribute.id }}
{% endfor %}
{% endif %}
{% set visible_exceptions = own_page_children|selectattr("type", "equalto", "exception")|list %}
{% if visible_exceptions %}
Exceptions
----------
.. autoapisummary::
{% for exception in visible_exceptions %}
{{ exception.id }}
{% endfor %}
{% endif %}
{% set visible_classes = own_page_children|selectattr("type", "equalto", "class")|list %}
{% if visible_classes %}
Classes
-------
.. autoapisummary::
{% for klass in visible_classes %}
{{ klass.id }}
{% endfor %}
{% endif %}
{% set visible_methods = own_page_children|selectattr("type", "equalto", "method")|list %}
{% if visible_methods %}
Methods
-------
.. autoapisummary::
{% for method in visible_methods %}
{{ method.id }}
{% endfor %}
{% endif %}
{% endif %}
{% endif %}

38
docs/_templates/autoapi/python/data.rst vendored Normal file
View File

@@ -0,0 +1,38 @@
{% if obj.display %}
{% if is_own_page %}
{{ obj.id }}
{{ "=" * obj.id | length }}
{% endif %}
.. py:{{ obj.type }}:: {% if is_own_page %}{{ obj.id }}{% else %}{{ obj.name }}{% endif %}
{% if obj.annotation is not none %}
:type: {% if obj.annotation %} {{ obj.annotation }}{% endif %}
{% endif %}
{% if obj.value is not none %}
{% if obj.value.splitlines()|count > 1 %}
:value: Multiline-String
.. raw:: html
<details><summary>Show Value</summary>
.. code-block:: python
{{ obj.value|indent(width=6,blank=true) }}
.. raw:: html
</details>
{% else %}
:value: {{ obj.value|truncate(100) }}
{% endif %}
{% endif %}
{% if obj.docstring %}
{{ obj.docstring|indent(3) }}
{% endif %}
{% endif %}

1
docs/_templates/autoapi/python/exception.rst vendored Normal file
View File

@@ -0,0 +1 @@
{% extends "python/class.rst" %}

21
docs/_templates/autoapi/python/function.rst vendored Normal file
View File

@@ -0,0 +1,21 @@
{% if obj.display %}
{% if is_own_page %}
{{ obj.id }}
{{ "=" * obj.id | length }}
{% endif %}
.. py:function:: {% if is_own_page %}{{ obj.id }}{% else %}{{ obj.short_name }}{% endif %}({{ obj.args }}){% if obj.return_annotation is not none %} -> {{ obj.return_annotation }}{% endif %}
{% for (args, return_annotation) in obj.overloads %}
{%+ if is_own_page %}{{ obj.id }}{% else %}{{ obj.short_name }}{% endif %}({{ args }}){% if return_annotation is not none %} -> {{ return_annotation }}{% endif %}
{% endfor %}
{% for property in obj.properties %}
:{{ property }}:
{% endfor %}
{% if obj.docstring %}
{{ obj.docstring|indent(3) }}
{% endif %}
{% endif %}

21
docs/_templates/autoapi/python/method.rst vendored Normal file
View File

@@ -0,0 +1,21 @@
{% if obj.display %}
{% if is_own_page %}
{{ obj.id }}
{{ "=" * obj.id | length }}
{% endif %}
.. py:method:: {% if is_own_page %}{{ obj.id }}{% else %}{{ obj.short_name }}{% endif %}({{ obj.args }}){% if obj.return_annotation is not none %} -> {{ obj.return_annotation }}{% endif %}
{% for (args, return_annotation) in obj.overloads %}
{%+ if is_own_page %}{{ obj.id }}{% else %}{{ obj.short_name }}{% endif %}({{ args }}){% if return_annotation is not none %} -> {{ return_annotation }}{% endif %}
{% endfor %}
{% for property in obj.properties %}
:{{ property }}:
{% endfor %}
{% if obj.docstring %}
{{ obj.docstring|indent(3) }}
{% endif %}
{% endif %}

156
docs/_templates/autoapi/python/module.rst vendored Normal file
View File

@@ -0,0 +1,156 @@
{% if obj.display %}
{% if is_own_page %}
{{ obj.id }}
{{ "=" * obj.id|length }}
.. py:module:: {{ obj.name }}
{% if obj.docstring %}
.. autoapi-nested-parse::
{{ obj.docstring|indent(3) }}
{% endif %}
{% block submodules %}
{% set visible_subpackages = obj.subpackages|selectattr("display")|list %}
{% set visible_submodules = obj.submodules|selectattr("display")|list %}
{% set visible_submodules = (visible_subpackages + visible_submodules)|sort %}
{% if visible_submodules %}
Submodules
----------
.. toctree::
:maxdepth: 1
{% for submodule in visible_submodules %}
{{ submodule.include_path }}
{% endfor %}
{% endif %}
{% endblock %}
{% block content %}
{% set visible_children = obj.children|selectattr("display")|list %}
{% if visible_children %}
{% set visible_attributes = visible_children|selectattr("type", "equalto", "data")|list %}
{% if visible_attributes %}
{% if "attribute" in own_page_types or "show-module-summary" in autoapi_options %}
Attributes
----------
{% if "attribute" in own_page_types %}
.. toctree::
:hidden:
{% for attribute in visible_attributes %}
{{ attribute.include_path }}
{% endfor %}
{% endif %}
.. autoapisummary::
{% for attribute in visible_attributes %}
{{ attribute.id }}
{% endfor %}
{% endif %}
{% endif %}
{% set visible_exceptions = visible_children|selectattr("type", "equalto", "exception")|list %}
{% if visible_exceptions %}
{% if "exception" in own_page_types or "show-module-summary" in autoapi_options %}
Exceptions
----------
{% if "exception" in own_page_types %}
.. toctree::
:hidden:
{% for exception in visible_exceptions %}
{{ exception.include_path }}
{% endfor %}
{% endif %}
.. autoapisummary::
{% for exception in visible_exceptions %}
{{ exception.id }}
{% endfor %}
{% endif %}
{% endif %}
{% set visible_classes = visible_children|selectattr("type", "equalto", "class")|list %}
{% if visible_classes %}
{% if "class" in own_page_types or "show-module-summary" in autoapi_options %}
Classes
-------
{% if "class" in own_page_types %}
.. toctree::
:hidden:
{% for klass in visible_classes %}
{{ klass.include_path }}
{% endfor %}
{% endif %}
.. autoapisummary::
{% for klass in visible_classes %}
{{ klass.id }}
{% endfor %}
{% endif %}
{% endif %}
{% set visible_functions = visible_children|selectattr("type", "equalto", "function")|list %}
{% if visible_functions %}
{% if "function" in own_page_types or "show-module-summary" in autoapi_options %}
Functions
---------
{% if "function" in own_page_types %}
.. toctree::
:hidden:
{% for function in visible_functions %}
{{ function.include_path }}
{% endfor %}
{% endif %}
.. autoapisummary::
{% for function in visible_functions %}
{{ function.id }}
{% endfor %}
{% endif %}
{% endif %}
{% set this_page_children = visible_children|rejectattr("type", "in", own_page_types)|list %}
{% if this_page_children %}
{{ obj.type|title }} Contents
{{ "-" * obj.type|length }}---------
{% for obj_item in this_page_children %}
{{ obj_item.render()|indent(0) }}
{% endfor %}
{% endif %}
{% endif %}
{% endblock %}
{% else %}
.. py:module:: {{ obj.name }}
{% if obj.docstring %}
.. autoapi-nested-parse::
{{ obj.docstring|indent(6) }}
{% endif %}
{% for obj_item in visible_children %}
{{ obj_item.render()|indent(3) }}
{% endfor %}
{% endif %}
{% endif %}

1
docs/_templates/autoapi/python/package.rst vendored Normal file
View File

@@ -0,0 +1 @@
{% extends "python/module.rst" %}

21
docs/_templates/autoapi/python/property.rst vendored Normal file
View File

@@ -0,0 +1,21 @@
{% if obj.display %}
{% if is_own_page %}
{{ obj.id }}
{{ "=" * obj.id | length }}
{% endif %}
.. py:property:: {% if is_own_page %}{{ obj.id}}{% else %}{{ obj.short_name }}{% endif %}
{% if obj.annotation %}
:type: {{ obj.annotation }}
{% endif %}
{% for property in obj.properties %}
:{{ property }}:
{% endfor %}
{% if obj.docstring %}
{{ obj.docstring|indent(3) }}
{% endif %}
{% endif %}

1
docs/scripts/__init__.py Normal file
View File

@@ -0,0 +1 @@
from scripts import generate_module_docs

147
docs/scripts/scripts.py Normal file
View File

@@ -0,0 +1,147 @@
# iterate through all the modules in auto_archiver.modules and turn the __manifest__.py file into a markdown table
from pathlib import Path
from auto_archiver.core.module import ModuleFactory
from auto_archiver.core.base_module import BaseModule
from ruamel.yaml import YAML
from ruamel.yaml.comments import CommentedMap
import io
MODULES_FOLDER = Path(__file__).parent.parent.parent.parent / "src" / "auto_archiver" / "modules"
SAVE_FOLDER = Path(__file__).parent.parent / "source" / "modules" / "autogen"
type_color = {
'feeder': "<span style='color: #FFA500'>[feeder](/core_modules.md#feeder-modules)</a></span>",
'extractor': "<span style='color: #00FF00'>[extractor](/core_modules.md#extractor-modules)</a></span>",
'enricher': "<span style='color: #0000FF'>[enricher](/core_modules.md#enricher-modules)</a></span>",
'database': "<span style='color: #FF00FF'>[database](/core_modules.md#database-modules)</a></span>",
'storage': "<span style='color: #FFFF00'>[storage](/core_modules.md#storage-modules)</a></span>",
'formatter': "<span style='color: #00FFFF'>[formatter](/core_modules.md#formatter-modules)</a></span>",
}
TABLE_HEADER = ("Option", "Description", "Default", "Type")
EXAMPLE_YAML = """
# steps configuration
steps:
...
{steps_str}
...
# module configuration
...
{config_string}
"""
def generate_module_docs():
yaml = YAML()
SAVE_FOLDER.mkdir(exist_ok=True)
modules_by_type = {}
header_row = "| " + " | ".join(TABLE_HEADER) + "|\n" + "| --- " * len(TABLE_HEADER) + "|\n"
global_table = "\n## Configuration Options\n" + header_row
global_yaml = yaml.load("""\n# Module configuration\nplaceholder: {}""")
for module in sorted(ModuleFactory().available_modules(), key=lambda x: (x.requires_setup, x.name)):
# generate the markdown file from the __manifest__.py file.
manifest = module.manifest
for type in manifest['type']:
modules_by_type.setdefault(type, []).append(module)
description = "\n".join(l.lstrip() for l in manifest['description'].split("\n"))
types = ", ".join(type_color[t] for t in manifest['type'])
readme_str = f"""
# {manifest['name']}
```{{admonition}} Module type
{types}
```
{description}
"""
steps_str = "\n".join(f" {t}s:\n - {module.name}" for t in manifest['type'])
if not manifest['configs']:
config_string = f"# No configuration options for {module.name}.*\n"
else:
config_table = header_row
config_yaml = {}
global_yaml[module.name] = CommentedMap()
global_yaml.yaml_set_comment_before_after_key(module.name, f"\n\n{module.display_name} configuration options")
for key, value in manifest['configs'].items():
type = value.get('type', 'string')
if type == 'json_loader':
value['type'] = 'json'
elif type == 'str':
type = "string"
default = value.get('default', '')
config_yaml[key] = default
global_yaml[module.name][key] = default
if value.get('help', ''):
global_yaml[module.name].yaml_add_eol_comment(value.get('help', ''), key)
help = "**Required**. " if value.get('required', False) else "Optional. "
help += value.get('help', '')
config_table += f"| `{module.name}.{key}` | {help} | {value.get('default', '')} | {type} |\n"
global_table += f"| `{module.name}.{key}` | {help} | {default} | {type} |\n"
readme_str += "\n## Configuration Options\n"
readme_str += "\n### YAML\n"
config_string = io.BytesIO()
yaml.dump({module.name: config_yaml}, config_string)
config_string = config_string.getvalue().decode('utf-8')
yaml_string = EXAMPLE_YAML.format(steps_str=steps_str, config_string=config_string)
readme_str += f"```{{code}} yaml\n{yaml_string}\n```\n"
if manifest['configs']:
readme_str += "\n### Command Line:\n"
readme_str += config_table
# add a link to the autodoc refs
readme_str += f"\n[API Reference](../../../autoapi/{module.name}/index)\n"
# create the module.type folder, use the first type just for where to store the file
for type in manifest['type']:
type_folder = SAVE_FOLDER / type
type_folder.mkdir(exist_ok=True)
with open(type_folder / f"{module.name}.md", "w") as f:
print("writing", SAVE_FOLDER)
f.write(readme_str)
generate_index(modules_by_type)
del global_yaml['placeholder']
global_string = io.BytesIO()
global_yaml = yaml.dump(global_yaml, global_string)
global_string = global_string.getvalue().decode('utf-8')
global_yaml = f"```yaml\n{global_string}\n```"
with open(SAVE_FOLDER / "configs_cheatsheet.md", "w") as f:
f.write("### Configuration File\n" + global_yaml + "\n### Command Line\n" + global_table)
def generate_index(modules_by_type):
readme_str = ""
for type in BaseModule.MODULE_TYPES:
modules = modules_by_type.get(type, [])
module_str = f"## {type.capitalize()} Modules\n"
for module in modules:
module_str += f"\n[{module.manifest['name']}](/modules/autogen/{module.type[0]}/{module.name}.md)\n"
with open(SAVE_FOLDER / f"{type}.md", "w") as f:
print("writing", SAVE_FOLDER / f"{type}.md")
f.write(module_str)
readme_str += module_str
with open(SAVE_FOLDER / "module_list.md", "w") as f:
print("writing", SAVE_FOLDER / "module_list.md")
f.write(readme_str)
if __name__ == "__main__":
generate_module_docs()

742
docs/source/_auto/configs.rst
View File

@@ -1,742 +0,0 @@
Configs
-------
This section documents all configuration options available for various components.
InstagramAPIArchiver
--------------------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - access_token
- None
- a valid instagrapi-api token
* - api_endpoint
- None
- API endpoint to use
* - full_profile
- False
- if true, will download all posts, tagged posts, stories, and highlights for a profile, if false, will only download the profile pic and information.
* - full_profile_max_posts
- 0
- Use to limit the number of posts to download when full_profile is true. 0 means no limit. limit is applied softly since posts are fetched in batch, once to: posts, tagged posts, and highlights
* - minimize_json_output
- True
- if true, will remove empty values from the json output
InstagramArchiver
-----------------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - username
- None
- a valid Instagram username
* - password
- None
- the corresponding Instagram account password
* - download_folder
- instaloader
- name of a folder to temporarily download content to
* - session_file
- secrets/instaloader.session
- path to the instagram session which saves session credentials
InstagramTbotArchiver
---------------------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - api_id
- None
- telegram API_ID value, go to https://my.telegram.org/apps
* - api_hash
- None
- telegram API_HASH value, go to https://my.telegram.org/apps
* - session_file
- secrets/anon-insta
- optional, records the telegram login session for future usage, '.session' will be appended to the provided value.
* - timeout
- 45
- timeout to fetch the instagram content in seconds.
TelethonArchiver
----------------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - api_id
- None
- telegram API_ID value, go to https://my.telegram.org/apps
* - api_hash
- None
- telegram API_HASH value, go to https://my.telegram.org/apps
* - bot_token
- None
- optional, but allows access to more content such as large videos, talk to @botfather
* - session_file
- secrets/anon
- optional, records the telegram login session for future usage, '.session' will be appended to the provided value.
* - join_channels
- True
- disables the initial setup with channel_invites config, useful if you have a lot and get stuck
* - channel_invites
- {}
- (JSON string) private channel invite links (format: t.me/joinchat/HASH OR t.me/+HASH) and (optional but important to avoid hanging for minutes on startup) channel id (format: CHANNEL_ID taken from a post url like https://t.me/c/CHANNEL_ID/1), the telegram account will join any new channels on setup
TwitterApiArchiver
------------------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - bearer_token
- None
- [deprecated: see bearer_tokens] twitter API bearer_token which is enough for archiving, if not provided you will need consumer_key, consumer_secret, access_token, access_secret
* - bearer_tokens
- []
- a list of twitter API bearer_token which is enough for archiving, if not provided you will need consumer_key, consumer_secret, access_token, access_secret, if provided you can still add those for better rate limits. CSV of bearer tokens if provided via the command line
* - consumer_key
- None
- twitter API consumer_key
* - consumer_secret
- None
- twitter API consumer_secret
* - access_token
- None
- twitter API access_token
* - access_secret
- None
- twitter API access_secret
VkArchiver
----------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - username
- None
- valid VKontakte username
* - password
- None
- valid VKontakte password
* - session_file
- secrets/vk_config.v2.json
- valid VKontakte password
YoutubeDLArchiver
-----------------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - facebook_cookie
- None
- optional facebook cookie to have more access to content, from browser, looks like 'cookie: datr= xxxx'
* - subtitles
- True
- download subtitles if available
* - comments
- False
- download all comments if available, may lead to large metadata
* - livestreams
- False
- if set, will download live streams, otherwise will skip them; see --max-filesize for more control
* - live_from_start
- False
- if set, will download live streams from their earliest available moment, otherwise starts now.
* - proxy
-
- http/socks (https seems to not work atm) proxy to use for the webdriver, eg https://proxy- user:password@proxy-ip:port
* - end_means_success
- True
- if True, any archived content will mean a 'success', if False this archiver will not return a 'success' stage; this is useful for cases when the yt-dlp will archive a video but ignore other types of content like images or text only pages that the subsequent archivers can retrieve.
* - allow_playlist
- False
- If True will also download playlists, set to False if the expectation is to download a single video.
* - max_downloads
- inf
- Use to limit the number of videos to download when a channel or long page is being extracted. 'inf' means no limit.
* - cookies_from_browser
- None
- optional browser for ytdl to extract cookies from, can be one of: brave, chrome, chromium, edge, firefox, opera, safari, vivaldi, whale
* - cookie_file
- None
- optional cookie file to use for Youtube, see instructions here on how to export from your browser: https://github.com/yt-dlp/yt- dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp
AAApiDb
-------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - api_endpoint
- None
- API endpoint where calls are made to
* - api_token
- None
- API Bearer token.
* - public
- False
- whether the URL should be publicly available via the API
* - author_id
- None
- which email to assign as author
* - group_id
- None
- which group of users have access to the archive in case public=false as author
* - allow_rearchive
- True
- if False then the API database will be queried prior to any archiving operations and stop if the link has already been archived
* - store_results
- True
- when set, will send the results to the API database.
* - tags
- []
- what tags to add to the archived URL
AtlosDb
-------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - api_token
- None
- An Atlos API token. For more information, see https://docs.atlos.org/technical/api/
* - atlos_url
- https://platform.atlos.org
- The URL of your Atlos instance (e.g., https://platform.atlos.org), without a trailing slash.
CSVDb
-----
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - csv_file
- db.csv
- CSV file name
HashEnricher
------------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - algorithm
- SHA-256
- hash algorithm to use
* - chunksize
- 16000000
- number of bytes to use when reading files in chunks (if this value is too large you will run out of RAM), default is 16MB
ScreenshotEnricher
------------------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - width
- 1280
- width of the screenshots
* - height
- 720
- height of the screenshots
* - timeout
- 60
- timeout for taking the screenshot
* - sleep_before_screenshot
- 4
- seconds to wait for the pages to load before taking screenshot
* - http_proxy
-
- http proxy to use for the webdriver, eg http://proxy-user:password@proxy-ip:port
* - save_to_pdf
- False
- save the page as pdf along with the screenshot. PDF saving options can be adjusted with the 'print_options' parameter
* - print_options
- {}
- options to pass to the pdf printer
SSLEnricher
-----------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - skip_when_nothing_archived
- True
- if true, will skip enriching when no media is archived
ThumbnailEnricher
-----------------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - thumbnails_per_minute
- 60
- how many thumbnails to generate per minute of video, can be limited by max_thumbnails
* - max_thumbnails
- 16
- limit the number of thumbnails to generate per video, 0 means no limit
TimestampingEnricher
--------------------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - tsa_urls
- ['http://timestamp.digicert.com', 'http://timestamp.identrust.com', 'http://timestamp.globalsign.com/tsa/r6advanced1', 'http://tss.accv.es:8318/tsa']
- List of RFC3161 Time Stamp Authorities to use, separate with commas if passed via the command line.
WaczArchiverEnricher
--------------------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - profile
- None
- browsertrix-profile (for profile generation see https://github.com/webrecorder/browsertrix- crawler#creating-and-using-browser-profiles).
* - docker_commands
- None
- if a custom docker invocation is needed
* - timeout
- 120
- timeout for WACZ generation in seconds
* - extract_media
- False
- If enabled all the images/videos/audio present in the WACZ archive will be extracted into separate Media and appear in the html report. The .wacz file will be kept untouched.
* - extract_screenshot
- True
- If enabled the screenshot captured by browsertrix will be extracted into separate Media and appear in the html report. The .wacz file will be kept untouched.
* - socks_proxy_host
- None
- SOCKS proxy host for browsertrix-crawler, use in combination with socks_proxy_port. eg: user:password@host
* - socks_proxy_port
- None
- SOCKS proxy port for browsertrix-crawler, use in combination with socks_proxy_host. eg 1234
* - proxy_server
- None
- SOCKS server proxy URL, in development
WaybackArchiverEnricher
-----------------------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - timeout
- 15
- seconds to wait for successful archive confirmation from wayback, if more than this passes the result contains the job_id so the status can later be checked manually.
* - if_not_archived_within
- None
- only tell wayback to archive if no archive is available before the number of seconds specified, use None to ignore this option. For more information: https://docs.google.com/document/d/1N sv52MvSjbLb2PCpHlat0gkzw0EvtSgpKHu4mk0MnrA
* - key
- None
- wayback API key. to get credentials visit https://archive.org/account/s3.php
* - secret
- None
- wayback API secret. to get credentials visit https://archive.org/account/s3.php
* - proxy_http
- None
- http proxy to use for wayback requests, eg http://proxy-user:password@proxy-ip:port
* - proxy_https
- None
- https proxy to use for wayback requests, eg https://proxy-user:password@proxy-ip:port
WhisperEnricher
---------------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - api_endpoint
- None
- WhisperApi api endpoint, eg: https://whisperbox- api.com/api/v1, a deployment of https://github.com/bellingcat/whisperbox- transcribe.
* - api_key
- None
- WhisperApi api key for authentication
* - include_srt
- False
- Whether to include a subtitle SRT (SubRip Subtitle file) for the video (can be used in video players).
* - timeout
- 90
- How many seconds to wait at most for a successful job completion.
* - action
- translate
- which Whisper operation to execute
AtlosFeeder
-----------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - api_token
- None
- An Atlos API token. For more information, see https://docs.atlos.org/technical/api/
* - atlos_url
- https://platform.atlos.org
- The URL of your Atlos instance (e.g., https://platform.atlos.org), without a trailing slash.
CLIFeeder
---------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - urls
- None
- URL(s) to archive, either a single URL or a list of urls, should not come from config.yaml
GsheetsFeeder
-------------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - sheet
- None
- name of the sheet to archive
* - sheet_id
- None
- (alternative to sheet name) the id of the sheet to archive
* - header
- 1
- index of the header row (starts at 1)
* - service_account
- secrets/service_account.json
- service account JSON file path
* - columns
- {'url': 'link', 'status': 'archive status', 'folder': 'destination folder', 'archive': 'archive location', 'date': 'archive date', 'thumbnail': 'thumbnail', 'timestamp': 'upload timestamp', 'title': 'upload title', 'text': 'text content', 'screenshot': 'screenshot', 'hash': 'hash', 'pdq_hash': 'perceptual hashes', 'wacz': 'wacz', 'replaywebpage': 'replaywebpage'}
- names of columns in the google sheet (stringified JSON object)
* - allow_worksheets
- set()
- (CSV) only worksheets whose name is included in allow are included (overrides worksheet_block), leave empty so all are allowed
* - block_worksheets
- set()
- (CSV) explicitly block some worksheets from being processed
* - use_sheet_names_in_stored_paths
- True
- if True the stored files path will include 'workbook_name/worksheet_name/...'
HtmlFormatter
-------------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - detect_thumbnails
- True
- if true will group by thumbnails generated by thumbnail enricher by id 'thumbnail_00'
AtlosStorage
------------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - path_generator
- url
- how to store the file in terms of directory structure: 'flat' sets to root; 'url' creates a directory based on the provided URL; 'random' creates a random directory.
* - filename_generator
- random
- how to name stored files: 'random' creates a random string; 'static' uses a replicable strategy such as a hash.
* - api_token
- None
- An Atlos API token. For more information, see https://docs.atlos.org/technical/api/
* - atlos_url
- https://platform.atlos.org
- The URL of your Atlos instance (e.g., https://platform.atlos.org), without a trailing slash.
GDriveStorage
-------------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - path_generator
- url
- how to store the file in terms of directory structure: 'flat' sets to root; 'url' creates a directory based on the provided URL; 'random' creates a random directory.
* - filename_generator
- random
- how to name stored files: 'random' creates a random string; 'static' uses a replicable strategy such as a hash.
* - root_folder_id
- None
- root google drive folder ID to use as storage, found in URL: 'https://drive.google.com/drive/folders/FOLDER_ID'
* - oauth_token
- None
- JSON filename with Google Drive OAuth token: check auto-archiver repository scripts folder for create_update_gdrive_oauth_token.py. NOTE: storage used will count towards owner of GDrive folder, therefore it is best to use oauth_token_filename over service_account.
* - service_account
- secrets/service_account.json
- service account JSON file path, same as used for Google Sheets. NOTE: storage used will count towards the developer account.
LocalStorage
------------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - path_generator
- url
- how to store the file in terms of directory structure: 'flat' sets to root; 'url' creates a directory based on the provided URL; 'random' creates a random directory.
* - filename_generator
- random
- how to name stored files: 'random' creates a random string; 'static' uses a replicable strategy such as a hash.
* - save_to
- ./archived
- folder where to save archived content
* - save_absolute
- False
- whether the path to the stored file is absolute or relative in the output result inc. formatters (WARN: leaks the file structure)
S3Storage
---------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - path_generator
- url
- how to store the file in terms of directory structure: 'flat' sets to root; 'url' creates a directory based on the provided URL; 'random' creates a random directory.
* - filename_generator
- random
- how to name stored files: 'random' creates a random string; 'static' uses a replicable strategy such as a hash.
* - bucket
- None
- S3 bucket name
* - region
- None
- S3 region name
* - key
- None
- S3 API key
* - secret
- None
- S3 API secret
* - random_no_duplicate
- False
- if set, it will override `path_generator`, `filename_generator` and `folder`. It will check if the file already exists and if so it will not upload it again. Creates a new root folder path `no-dups/`
* - endpoint_url
- https://{region}.digitaloceanspaces.com
- S3 bucket endpoint, {region} are inserted at runtime
* - cdn_url
- https://{bucket}.{region}.cdn.digitaloceanspaces.com/{key}
- S3 CDN url, {bucket}, {region} and {key} are inserted at runtime
* - private
- False
- if true S3 files will not be readable online
Storage
-------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - path_generator
- url
- how to store the file in terms of directory structure: 'flat' sets to root; 'url' creates a directory based on the provided URL; 'random' creates a random directory.
* - filename_generator
- random
- how to name stored files: 'random' creates a random string; 'static' uses a replicable strategy such as a hash.
Gsheets
-------
The following table lists all configuration options for this component:
.. list-table:: Configuration Options
:header-rows: 1
:widths: 25 20 55
* - **Key**
- **Default**
- **Description**
* - sheet
- None
- name of the sheet to archive
* - sheet_id
- None
- (alternative to sheet name) the id of the sheet to archive
* - header
- 1
- index of the header row (starts at 1)
* - service_account
- secrets/service_account.json
- service account JSON file path
* - columns
- {'url': 'link', 'status': 'archive status', 'folder': 'destination folder', 'archive': 'archive location', 'date': 'archive date', 'thumbnail': 'thumbnail', 'timestamp': 'upload timestamp', 'title': 'upload title', 'text': 'text content', 'screenshot': 'screenshot', 'hash': 'hash', 'pdq_hash': 'perceptual hashes', 'wacz': 'wacz', 'replaywebpage': 'replaywebpage'}
- names of columns in the google sheet (stringified JSON object)

BIN
docs/source/bc.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

59
docs/source/conf.py
View File

@@ -1,50 +1,64 @@
# Configuration file for the Sphinx documentation builder.
# https://www.sphinx-doc.org/en/master/usage/configuration.html
import sys
import os
from importlib.metadata import metadata
from datetime import datetime
sys.path.append(os.path.abspath('../scripts'))
from scripts import generate_module_docs
from auto_archiver.version import __version__
# -- Project Hooks -----------------------------------------------------------
# convert the module __manifest__.py files into markdown files
generate_module_docs()
# -- Project information -----------------------------------------------------
from importlib.metadata import metadata
package_metadata = metadata("auto-archiver")
project = package_metadata["name"]
authors = package_metadata["authors"]
copyright = str(datetime.now().year)
author = "Bellingcat"
release = package_metadata["version"]
language = 'en'
# -- General configuration ---------------------------------------------------
extensions = [
"autoapi.extension", # Generate API documentation from docstrings
"myst_parser", # Markdown support
'sphinxcontrib.mermaid', # Mermaid diagrams
"autoapi.extension", # Generate API documentation from docstrings
"sphinxcontrib.mermaid", # Mermaid diagrams
"sphinx.ext.viewcode", # Source code links
"sphinx_copybutton",
"sphinx.ext.napoleon", # Google-style and NumPy-style docstrings
# "sphinx.ext.autodoc", # Include custom docstrings
"sphinx.ext.autosectionlabel",
# 'sphinx.ext.autosummary', # Summarize module/class/function docs
]
templates_path = ['_templates']
exclude_patterns = []
exclude_patterns = ["_build", "Thumbs.db", ".DS_Store", ""]
# -- AutoAPI Configuration ---------------------------------------------------
autoapi_type = 'python'
autoapi_dirs = ["../../src"]
autoapi_dirs = ["../../src/auto_archiver/core/", "../../src/auto_archiver/utils/"]
# get all the modules and add them to the autoapi_dirs
autoapi_dirs.extend([f"../../src/auto_archiver/modules/{m}" for m in os.listdir("../../src/auto_archiver/modules")])
autodoc_typehints = "signature" # Include type hints in the signature
autoapi_ignore = [] # Ignore specific modules
autoapi_ignore = ["*/version.py", ] # Ignore specific modules
autoapi_keep_files = True # Option to retain intermediate JSON files for debugging
autoapi_add_toctree_entry = True # Include API docs in the TOC
autoapi_template_dir = None # Use default templates
autoapi_python_use_implicit_namespaces = True
autoapi_template_dir = "../_templates/autoapi"
autoapi_options = [
"members",
"undoc-members",
"show-inheritance",
"show-module-summary",
"imported-members",
]
# -- Markdown Support --------------------------------------------------------
myst_enable_extensions = [
"colon_fence", # ::: fences
"deflist", # Definition lists
"html_admonition", # HTML-style admonitions
"html_image", # Inline HTML images
@@ -53,12 +67,27 @@ myst_enable_extensions = [
"linkify", # Auto-detect links
"substitution", # Text substitutions
]
myst_heading_anchors = 2
myst_fence_as_directive = ["mermaid"]
source_suffix = {
".rst": "restructuredtext",
".md": "markdown",
}
# -- Options for HTML output -------------------------------------------------
html_theme = 'furo'
# html_static_path = ['_static']
html_theme = 'sphinx_book_theme'
html_static_path = ["../_static"]
html_css_files = ["custom.css"]
html_title = f"Auto Archiver v{__version__}"
html_logo = "bc.png"
html_theme_options = {
"repository_url": "https://github.com/bellingcat/auto-archiver",
"use_repository_button": True,
}
copybutton_prompt_text = r">>> |\.\.\."
copybutton_prompt_is_regexp = True
copybutton_only_copy_prompt_lines = False

34
docs/source/configurations.rst
View File

@@ -1,34 +0,0 @@
Configurations
==============
This section of the documentation provides guidelines for configuring the tool.
File Reference
--------------
Below is the content of the `example.orchestration.yaml` file:
.. raw:: html
<details>
<summary>View example.orchestration.yaml</summary>
.. literalinclude:: ../../example.orchestration.yaml
:language: yaml
:caption: example.orchestration.yaml
.. raw:: html
</details>
Configs
-------
This section of the documentation will show the custom configurations for the individual steps of the tool.
.. include:: _auto/configs.rst

2
docs/source/contributing.md Normal file
View File

@@ -0,0 +1,2 @@
```{include} ../../CONTRIBUTING.md
```

28
docs/source/core_modules.md Normal file
View File

@@ -0,0 +1,28 @@
# Module Documentation
These pages describe the core modules that come with Auto Archiver and provide the main functionality for archiving websites on the internet. There are five core module types:
1. Feeders - these 'feed' information (the URLs) from various sources to the Auto Archiver for processing
2. Extractors - these 'extract' the page data for a given URL that is fed in by a feeder
3. Enrichers - these 'enrich' the data extracted in the previous step with additional information
4. Storage - these 'store' the data in a persistent location (on disk, Google Drive etc.)
5. Databases - these 'store' the status of the entire archiving process in a log file or database.
```{include} modules/autogen/module_list.md
```
```{toctree}
:maxdepth: 1
:caption: Core Modules
:hidden:
modules/config_cheatsheet
modules/feeder
modules/extractor
modules/enricher
modules/storage
modules/database
modules/formatter
```

6
docs/source/developer_guidelines.rst
View File

@@ -1,6 +0,0 @@
Developer Guidelines
====================
This section of the documentation provides guidelines for developers who want to modify or contribute to the tool.

52
docs/source/development/creating_modules.md Normal file
View File

@@ -0,0 +1,52 @@
# Creating Your Own Modules
Modules are what's used to extend Auto Archiver to process different websites or media, and/or transform the data in a way that suits your needs. In most cases, the [Core Modules](../core_modules.md) should be sufficient for every day use, but the most common use-cases for making your own Modules include:
1. Extracting data from a website which doesn't work with the current core extractors.
2. Enriching or altering the data before saving with additional information that the core enrichers do not offer.
3. Storing your data in a different format/location from what the core storage providers offer.
## Setting up the folder structure
1. First, decide what type of module you wish to create. Check the types of modules on the [](../core_modules.md) page to decide what type you need. (Note: a module can be more than one type, more on that below)
2. Create a new python package (a folder) with the name of your module (in this tutorial, we'll call it `awesome_extractor`).
3. Create the `__manifest__.py` and an the `awesome_extractor.py` files in this folder.
When done, you should have a module structure as follows:
```
.
├── awesome_extractor
│ ├── __manifest__.py
│ └── awesome_extractor.py
```
Check out the [core modules](https://github.com/bellingcat/auto-archiver/tree/main/src/auto_archiver/modules) in the Auto Archiver repository for examples of the folder structure for real-world modules.
## Populating the Manifest File
The manifest file is where you define the core information of your module. It is a python dict containing important information, here's an example file:
```{include} ../../../tests/data/test_modules/example_module/__manifest__.py
:name: __manifest__.py
:literal:
:parser: python
```
## Creating the Python Code
The next step is to create your module code. First, create a class which should subclass the base module types from `auto_archiver.core`, here's an example class for the `awesome_extractor` module which is an `extractor`:
```{code-block} python
:filename: awesome_extractor.py
from auto_archiver.core import Extractor, Metadata
def AwesomeExtractor(Extractor):
def download(self, item: Metadata) -> Metadata | False:
url = item.get_url()
# download the content and create the metadata object
metadata = ...
return metadata
```

35
docs/source/development/developer_guidelines.md Normal file
View File

@@ -0,0 +1,35 @@
# Developer Guidelines
This section of the documentation provides guidelines for developers who want to modify or contribute to the tool.
## Developer Install
1. Clone the project using `git clone https://github.com/bellingcat/auto-archiver.git`
2. Install poetry using `curl -sSL https://install.python-poetry.org | python3 -` ([other installation methods](https://python-poetry.org/docs/#installation))
3. Install dependencies with `poetry install`
## Running
4. Run the code with `poetry run auto-archiver [my args]`
```{note}
Add the plugin [poetry-shell-plugin](https://github.com/python-poetry/poetry-plugin-shell) and run `poetry shell` to activate the virtual environment.
This allows you to run the auto-archiver without the `poetry run` prefix.
```
### Optional Development Packages
Install development packages (used for unit tests etc.) using:
`poetry install -with dev`
```{toctree}
:hidden:
creating_modules
docker_development
testing
docs
release
settings_page
```

5
docs/source/development/docker_development.md Normal file
View File

@@ -0,0 +1,5 @@
## Docker development
working with docker locally:
* `docker compose up` to build the first time and run a local image with the settings in `secrets/orchestration.yaml`
* To modify/pass additional command line args, use `docker compose run auto-archiver --config secrets/orchestration.yaml [OTHER ARGUMENTS]`
* To rebuild after code changes, just pass the `--build` flag, e.g. `docker compose up --build`

38
docs/source/development/docs.md Normal file
View File

@@ -0,0 +1,38 @@
### Building the Docs
The documentation is built using [Sphinx](https://www.sphinx-doc.org/en/master/) and [AutoAPI](https://sphinx-autoapi.readthedocs.io/en/latest/) and hosted on ReadTheDocs.
To build the documentation locally, run the following commands:
**Install required dependencies:**
- Install the docs group of dependencies:
```shell
# only the docs dependencies
poetry install --only docs
# or for all dependencies
poetry install
```
- Either use [poetry-plugin-shell](https://github.com/python-poetry/poetry-plugin-shell) to activate the virtual environment: `poetry shell`
- Or prepend the following commands with `poetry run`
**Create the documentation:**
- Build the documentation:
```shell
# Using makefile (Linux/macOS):
make -C docs html
# or using sphinx directly (Windows/Linux/macOS):
sphinx-build -b html docs/source docs/_build/html
```
- If you make significant changes and want a fresh build run: `make -C docs clean` to remove the old build files.
**Viewing the documentation:**
```shell
# to open the documentation in your browser.
open docs/_build/html/index.html
# or run autobuild to automatically update the documentation when you make changes
sphinx-autobuild docs/source docs/_build/html
```

33
docs/source/development/release.md Normal file
View File

@@ -0,0 +1,33 @@
# Release Process
```{note} This is a work in progress.
```
### Update the project version
Update the version number in the project file: [pyproject.toml](../../pyproject.toml) following SemVer:
```toml
[project]
name = "auto-archiver"
version = "0.1.1"
```
Then commit and push the changes.
* The package version is automatically updated in PyPi using the workflow [python-publish.yml](../../.github/workflows/python-publish.yml)
* A Docker image is automatically pushed with the git tag to dockerhub using the workflow [docker-publish.yml](../../.github/workflows/docker-publish.yml)
### Create the release on Git
The release needs a git tag which should match the project version number, prefixed with a 'v'. For example, if the project version is `0.1.1`, the git tag should be `v0.1.1`.
This can be done the usual way, or created within the Github UI when you create the release.
Go to GitHub releases > new release > create the release with the new tag and the release notes.
manual release to docker hub
* `docker image tag auto-archiver bellingcat/auto-archiver:latest`
* `docker push bellingcat/auto-archiver`
### Building the Settings Page
The Settings page is built as part of the python-publish workflow and packaged within the app.

31
docs/source/development/settings_page.md Normal file
View File

@@ -0,0 +1,31 @@
# Configuration Editor
The [configuration editor](../installation/config_editor.md), is an easy-to-use UI for users to edit their auto-archiver settings.
The single-file app is built using React and vite. To get started developing the package, follow these steps:
1. Make sure you have Node v22 installed.
```{note} Tip: if you don't have node installed:
Use `nvm` to manage your node installations. Use:
`curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash` to install `nvm` and then `nvm i 22` to install Node v22
```
2. Generate the `schema.json` file for the currently installed modules using `python scripts/generate_settings_schema.py`
3. Go to the settings folder `cd scripts/settings/` and build your environment with `npm i`
4. Run a development version of the page with `npm run dev` and then open localhost:5173.
5. Build a release version of the page with `npm run build`
A release version creates a single-file app called `dist/index.html`. This file should be copied to `docs/source/installation/settings_base.html` so that it can be integrated into the sphinx docs.
```{note}
The single-file app dist/index.html does not include any `<html>` or `<head>` tags as it is designed to be built into a RTD docs page. Edit `index.html` in the settings folder if you wish to modify the built page.
```
## Readthedocs Integration
The configuration editor is built as part of the RTD deployment (see `.readthedocs.yaml` file). This command is run every time RTD is built:
`cd scripts/settings && npm install && npm run build && yes | cp dist/index.html ../../docs/source/installation/settings_base.html && cd ../..`

21
docs/source/development/testing.md Normal file
View File

@@ -0,0 +1,21 @@
# Testing
`pytest` is used for testing. There are two main types of tests:
1. 'core' tests which should be run on every change
2. 'download' tests which hit the network. These tests will do things like make API calls (e.g. Twitter, Bluesky etc.) and should be run regularly to make sure that APIs have not changed.
## Running Tests
1. Make sure you've installed the dev dependencies with `pytest install --with dev`
2. Tests can be run as follows:
```
#### Command prefix of 'poetry run' removed here for simplicity
# run core tests
pytest -ra -v -m "not download"
# run download tests
pytest -ra -v -m "download"
# run all tests
pytest -ra -v
```

79
docs/source/example.orchestration.yaml Normal file
View File

@@ -0,0 +1,79 @@
# Auto Archiver Configuration
# Steps are the modules that will be run in the order they are defined
steps:
feeders:
- cli_feeder
extractors:
- generic_extractor
- telegram_extractor
enrichers:
- thumbnail_enricher
- meta_enricher
- pdq_hash_enricher
- ssl_enricher
- hash_enricher
databases:
- console_db
- csv_db
storages:
- local_storage
formatters:
- html_formatter
# Global configuration
# Authentication
# a dictionary of authentication information that can be used by extractors to login to website.
# you can use a comma separated list for multiple domains on the same line (common usecase: x.com,twitter.com)
# Common login 'types' are username/password, cookie, api key/token.
# There are two special keys for using cookies, they are: cookies_file and cookies_from_browser.
# Some Examples:
# facebook.com:
# username: "my_username"
# password: "my_password"
# or for a site that uses an API key:
# twitter.com,x.com:
# api_key
# api_secret
# youtube.com:
# cookie: "login_cookie=value ; other_cookie=123" # multiple 'key=value' pairs should be separated by ;
authentication: {}
# Logging settings for your project. See the logging settings with --help
logging:
level: INFO
# These are the global configurations that are used by the modules
file:
rotation:
local_storage:
path_generator: flat
filename_generator: static
save_to: ./local_archive
save_absolute: false
html_formatter:
detect_thumbnails: true
thumbnail_enricher:
thumbnails_per_minute: 60
max_thumbnails: 16
generic_extractor:
subtitles: true
comments: false
livestreams: false
live_from_start: false
proxy: ''
end_means_success: true
allow_playlist: false
max_downloads: inf
csv_db:
csv_file: db.csv
ssl_enricher:
skip_when_nothing_archived: true
hash_enricher:
algorithm: SHA-256
chunksize: 16000000

30
docs/source/flow_overview.md Normal file
View File

@@ -0,0 +1,30 @@
# Archiving Overview
The archiver archives web pages using the following workflow
1. **Feeder** gets the links (from a spreadsheet, from the console, ...)
2. **Extractor** tries to extract content from the given link (e.g. videos from youtube, images from Twitter...)
3. **Enricher** adds more info to the content (hashes, thumbnails, ...)
4. **Formatter** creates a report from all the archived content (HTML, PDF, ...)
5. **Database** knows what's been archived and also stores the archive result (spreadsheet, CSV, or just the console)
Each step in the workflow is handled by 'modules' that interact with the data in different ways. For example, the Twitter Extractor Module would extract information from the Twitter website. The Screenshot Enricher Module will take screenshots of the given page. See the [core modules page](core_modules.md) for an overview of all the modules that are available.
Auto-archiver must have at least one module defined for each step of the workflow. This is done by setting the [configuration](installation/configurations.md) for your auto-archiver instance.
Here's the complete workflow that the auto-archiver goes through:
```mermaid
graph TD
s((start)) --> F(fa:fa-table Feeder)
F -->|get and clean URL| D1{fa:fa-database Database}
D1 -->|is already archived| e((end))
D1 -->|not yet archived| a(fa:fa-download Archivers)
a -->|got media| E(fa:fa-chart-line Enrichers)
E --> S[fa:fa-box-archive Storages]
E --> Fo(fa:fa-code Formatter)
Fo --> S
Fo -->|update database| D2(fa:fa-database Database)
D2 --> e
```

12
docs/source/how_to.md Normal file
View File

@@ -0,0 +1,12 @@
# How-To Guides
The follow pages contain helpful how-to guides for common use cases of the Auto Archiver.
---
```{toctree}
:maxdepth: 1
:glob:
how_to/*
```

110
docs/source/how_to/authentication_how_to.md Normal file
View File

@@ -0,0 +1,110 @@
# Logging in to sites
This how-to guide shows you how you can use various authentication methods to allow you to login to a site you are trying to archive. This is useful for websites that require a user to be logged in to browse them, or for sites that restrict bots.
In this How-To, we will authenticate on use Twitter/X.com using cookies, and on XXXX using username/password.
## Using cookies to authenticate on Twitter/X
It can be useful to archive tweets after logging in, since some tweets are only visible to authenticated users. One case is Tweets marked as 'Sensitive'.
Take this tweet as an example: [https://x.com/SozinhoRamalho/status/1876710769913450647](https://x.com/SozinhoRamalho/status/1876710769913450647)
This tweet has been marked as sensitive, so a normal run of Auto Archiver without a logged in session will fail to extract the tweet:
```{code-block} console
:emphasize-lines: 3,4,5,6
>>> auto-archiver https://x.com/SozinhoRamalho/status/1876710769913450647 ✭ ✱
...
ERROR: [twitter] 1876710769913450647: NSFW tweet requires authentication. Use --cookies,
--cookies-from-browser, --username and --password, --netrc-cmd, or --netrc (twitter) to
provide account credentials. See https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp
for how to manually pass cookies
[twitter] 1876710769913450647: Downloading guest token
[twitter] 1876710769913450647: Downloading GraphQL JSON
2025-02-20 15:06:13.362 | ERROR | auto_archiver.modules.generic_extractor.generic_extractor:download_for_extractor:248 - Error downloading metadata for post: NSFW tweet requires authentication. Use --cookies, --cookies-from-browser, --username and --password, --netrc-cmd, or --netrc (twitter) to provide account credentials. See https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp for how to manually pass cookies
[generic] Extracting URL: https://x.com/SozinhoRamalho/status/1876710769913450647
[generic] 1876710769913450647: Downloading webpage
WARNING: [generic] Falling back on generic information extractor
[generic] 1876710769913450647: Extracting information
ERROR: Unsupported URL: https://x.com/SozinhoRamalho/status/1876710769913450647
2025-02-20 15:06:13.744 | INFO | auto_archiver.core.orchestrator:archive:483 - Trying extractor telegram_extractor for https://x.com/SozinhoRamalho/status/1876710769913450647
2025-02-20 15:06:13.744 | SUCCESS | auto_archiver.modules.console_db.console_db:done:23 - DONE Metadata(status='nothing archived', metadata={'_processed_at': datetime.datetime(2025, 2, 20, 15, 6, 12, 473979, tzinfo=datetime.timezone.utc), 'url': 'https://x.com/SozinhoRamalho/status/1876710769913450647'}, media=[])
...
```
To get round this limitation, we can use **cookies** (information about a logged in user) to mimic being logged in to Twitter. There are two ways to pass cookies to Auto Archiver. One is from a file, and the other is from a browser profile on your computer.
In this tutorial, we will export the Twitter cookies from our browser and add them to Auto Archiver
**1. Installing a cookie exporter extension**
First, we need to install an extension in our browser to export the cookies for a certain site. The [FAQ on yt-dlp](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp) provides some suggestions: Get [cookies.txt LOCALLY](https://chromewebstore.google.com/detail/get-cookiestxt-locally/cclelndahbckbenkjhflpdbgdldlbecc) for Chrome or [cookies.txt](https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/) for Firefox.
**2. Export the cookies**
```{note} See the note [here](../installation/authentication.md#recommendations-for-authentication) on why you shouldn't use your own personal account for archiving.
```
Once the extension is installed in your preferred browser, login to Twitter in this browser, and then activate the extension and export the cookies. You can choose to export all your cookies for your browser, or just cookies for this specific site. In the image below, we're only exporting cookies for Twitter/x.com:
![extract cookies](extract_cookies.png)
**3. Adding the cookies file to Auto Archiver**
You now will have a file called `cookies.txt` (tip: name it `twitter_cookies.txt` if you only exported cookies for Twitter), which needs to be added to Auto Archiver.
Do this by going into your Auto Archiver configuration file, and editing the `authentication` section. We will add the `cookies_file` option for the site `x.com,twitter.com`.
```{note} For websites that have multiple URLs (like x.com and twitter.com) you can 'reuse' the same login information without duplicating it using a comma separated list of domain names.
```
I've saved my `twitter_cookies.txt` file in a `secrets` folder, so here's how my authentication section looks now:
```{code} yaml
:caption: orchestration.yaml
...
authentication:
x.com,twitter.com:
cookies_file: secrets/twitter_cookies.txt
...
```
**4. Re-run your archiving with the cookies enabled**
Now, the next time we re-run Auto Archiver, the cookies from our logged-in session will be used by Auto Archiver, and restricted/sensitive tweets can be downloaded!
```{code} console
>>> auto-archiver https://x.com/SozinhoRamalho/status/1876710769913450647 ✭ ✱ ◼
...
2025-02-20 15:27:46.785 | WARNING | auto_archiver.modules.console_db.console_db:started:13 - STARTED Metadata(status='no archiver', metadata={'_processed_at': datetime.datetime(2025, 2, 20, 15, 27, 46, 785304, tzinfo=datetime.timezone.utc), 'url': 'https://x.com/SozinhoRamalho/status/1876710769913450647'}, media=[])
2025-02-20 15:27:46.785 | INFO | auto_archiver.core.orchestrator:archive:483 - Trying extractor generic_extractor for https://x.com/SozinhoRamalho/status/1876710769913450647
[twitter] Extracting URL: https://x.com/SozinhoRamalho/status/1876710769913450647
...
2025-02-20 15:27:53.134 | INFO | auto_archiver.modules.local_storage.local_storage:upload:26 - ./local_archive/https-x-com-sozinhoramalho-status-1876710769913450647/06e8bacf27ac4bb983bf6280.html
2025-02-20 15:27:53.135 | SUCCESS | auto_archiver.modules.console_db.console_db:done:23 - DONE Metadata(status='yt-dlp_Twitter: success',
metadata={'_processed_at': datetime.datetime(2025, 2, 20, 15, 27, 48, 564738, tzinfo=datetime.timezone.utc), 'url':
'https://x.com/SozinhoRamalho/status/1876710769913450647', 'title': 'ignore tweet, testing sensitivity warning nudity https://t.co/t3u0hQsSB1',
...
```
### Finishing Touches
You've now successfully exported your cookies from a logged-in session in your browser, and used them to authenticate with Twitter and download a sensitive tweet. Congratulations!
Finally,Some important things to remember:
1. It's best not to use your own personal account for archiving. [Here's why](../installation/authentication.md#recommendations-for-authentication).
2. Cookies can be short-lived, so may need updating. Sometimes, a website session may 'expire' or a website may force you to login again. In these instances, you'll need to repeat the export step (step 2) after logging in again to update your cookies.
## Authenticating on XXXX site with username/password
```{note} This section is still under construction 🚧
```

BIN
docs/source/how_to/extract_cookies.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 944 KiB

159
docs/source/how_to/gsheets_setup.md Normal file
View File

@@ -0,0 +1,159 @@
# Using Google Sheets
This guide explains how to set up Google Sheets to process URLs automatically and then store the archiving status back into the Google sheet. It is broadly split into 3 steps:
1. Setting up your Google Sheet
2. Setting up a service account so Auto Archiver can access the sheet
3. Setting the Auto Archiver settings
### 1. Setting up your Google Sheet
Any Google sheet must have at least *one* column, with the name 'link' (you can change this name afterwards). This is the column with the URLs that you want the Auto Archiver to archive.
Your sheet can have many other columns that the Auto Archiver can use, and you can also include any additional columns for your own personal use. The order of the columns does not matter, the naming just needs to be correctly assigned to its corresponding value in the configuration file.
We recommend copying [this template Google Sheet](https://docs.google.com/spreadsheets/d/1NJZo_XZUBKTI1Ghlgi4nTPVvCfb0HXAs6j5tNGas72k/edit?usp=sharing) as a starting point for your project, as this matches the default column names.
Here's an overview of all the columns, and what a complete sheet would look like.
**Inputs:**
These are processed by the Gsheet Feeder and passed to the Auto Archiver.
* **Link** *(required)*: the URL of the post that is to be archived
* **Destination folder**: custom folder for archived file (regardless of storage)
**Outputs:**
These are updated by the Gsheet DB module during the archiving process.
Note the required columns are only required if you are using the Gsheet DB module as well as the feeder.
* **Archive status** *(required)*: Status of archive operation
* **Archive location**: URL of archived post
* **Archive date**: Date archived
* **Thumbnail**: Embeds a thumbnail for the post in the spreadsheet
* **Timestamp**: Timestamp of original post
* **Title**: Post title
* **Text**: Post text
* **Screenshot**: Link to screenshot of post
* **Hash**: Hash of archived HTML file (which contains hashes of post media) - for checksums/verification
* **Perceptual Hash**: Perceptual hashes of found images - these can be used for de-duplication of content
* **WACZ**: Link to a WACZ web archive of post
* **ReplayWebpage**: Link to a ReplayWebpage viewer of the WACZ archive
For example, this is a spreadsheet configured with all of the columns for the auto archiver and a few URLs to archive.
In this example the Ghseet Feeder and Gsheet DB are being used, and the archive is in progress.
(Note that the column names are not case sensitive.)
![A screenshot of a Google Spreadsheet with column headers defined as above, and several Youtube and Twitter URLs in the "Link" column](../../demo-before.png)
We'll change the name of the 'Destination Folder' column in step 3.
## 2. Setting up your Service Account
Once your Google Sheet is set up, you need to create what's called a 'service account' that will allow the Auto Archiver to access it.
To do this, follow the steps in [this guide](https://gspread.readthedocs.io/en/latest/oauth2.html) all the way up until step 8. You should have downloaded a file called `service_account.json` and shared the Google Sheet with the log 'client_email' email address in this file.
Once you've downloaded the file, save it to `secrets/service_account.json`
## 3. Setting up the configuration file
Now that you've set up your Google sheet, and you've set up the service account so Auto Archiver can access the sheet, the final step is to set your configuration.
First, make sure you have `gsheet_feeder_db` set in the `steps.feeders` section of your config. If you wish to store the results of the archiving process back in your Google sheet, make sure to also set the `ghseet_db` settig in the `steps.databases` section. Here's how this might look:
```{code} yaml
steps:
feeders:
- gsheet_feeder_db
...
databases:
- gsheet_feeder_db # optional, if you also want to store the results in the Google sheet and tract the status of active archivals.
...
```
Next, set up the `gsheet_feeder_db` configuration settings in the 'Configurations' part of the config `orchestration.yaml` file. Open up the file, and set the `gsheet_feeder_db.sheet` setting or the `gsheet_feeder_db.sheet_id` setting. The `sheet` should be the name of your sheet, as it shows in the top left of the sheet.
For example, the sheet [here](https://docs.google.com/spreadsheets/d/1NJZo_XZUBKTI1Ghlgi4nTPVvCfb0HXAs6j5tNGas72k/edit?gid=0#gid=0) is called 'Public Auto Archiver template'.
Here's how this might look:
```{code} yaml
...
gsheet_feeder_db:
sheet: 'My Awesome Sheet'
...
```
You can also pass these settings directly on the command line without having to edit the file, here'a an example of how to do that (using docker):
`docker run --rm -v $PWD/secrets:/app/secrets -v $PWD/local_archive:/app/local_archive bellingcat/auto-archiver:dockerize --gsheet_feeder_db.sheet "My Awesome Sheet 2"`.
Here, the sheet name has been overridden/specified in the command line invocation.
### 3a. (Optional) Changing the column names
In step 1, we said we would change the name of the 'Destination Folder'. Perhaps you don't like this name, or already have a sheet with a different name. In our example here, we want to name this column 'Save Folder'. To do this, we need to edit the `ghseet_feeder_db.column` setting in the configuration file.
For more information on this setting, see the [Gsheet Feeder Database docs](../modules/autogen/feeder/gsheet_feeder_db.md#configuration-options). We will first copy the default settings from the Gsheet Feeder docs for the 'column' settings, and then edit the 'Destination Folder' section to rename it 'Save Folder'. Our final configuration section looks like:
```{code} yaml
...
gsheet_feeder_db:
sheet: 'My Awesome Sheet'
header: 1
service_account: secrets/service_account.json
columns:
url: link
status: archive status
folder: save folder # <-- note how this value has been changed
archive: archive location
date: archive date
thumbnail: thumbnail
timestamp: upload timestamp
title: upload title
text: text content
screenshot: screenshot
hash: hash
pdq_hash: perceptual hashes
wacz: wacz
replaywebpage: replaywebpage
```
## 4. Running the Auto Archiver
### Feeding the URLs to the Auto Archiver
The URLs to be archived should be added to the Google Sheet, and optionally a folder value. Leave all the other configured columns empty (but you may add additional columns for your own use, as long as they don't conflict with the column names mapped in the configuration file).
The Auto Archiver will archive any URLs which have an empty 'status' column
### Viewing the Results after archiving
With the `ghseet_feeder_db` installed, once you start running the Auto Archiver, it will update the "Archive status" column.
The status will be set to "Archive in progress" once the archival starts. If the archival is stopped during a run, either manually or because an error is raised the status value should be cleared.
![A screenshot of a Google Spreadsheet with column headers defined as above, and several Youtube and Twitter URLs in the "Link" column. The auto archiver has added "archive in progress" to one of the status columns.](../../demo-progress.png)
The links are downloaded and archived, and the spreadsheet is updated to the following:
![A screenshot of a Google Spreadsheet with videos archived and metadata added per the description of the columns above.](../../demo-after.png)
Note that the first row is skipped, as it is assumed to be a header row (`--gsheet_feeder_db.header=1` and you can change it if you use more rows above). Rows with an empty URL column, or a non-empty archive column are also skipped. All sheets in the document will be checked.
The "archive location" link contains the path of the archived file, in local storage, S3, or in Google Drive.
![The archive result for a link in the demo sheet.](../../demo-archive.png)
### Troubleshooting
**Hanging Archival in progress status**
Occasionally system crashes or other unexpected events can cause the Auto Archiver to exit without cleaning up the status value.
If you are sure that all archival processes have stopped but you still see "Archive in progress" in the status column, you can manually clear the status column to allow the Auto Archiver to retry that archival on the next run.
**Nothing archived status**
Sometimes this means the tool is genuinely unable to extract the content at this point in time, but sometimes it can be resolved with different configurations.
Try:
- Turning on additional 'extractor' types in the configuration file (this can appear as 'no archiver' in the status column).
- Changing credentials or refreshing session files for extractors which require them
- Check if the extractors can accept any additional configurations such as adding a cookie file.

71
docs/source/how_to/logging.md Normal file
View File

@@ -0,0 +1,71 @@
# Keeping Logs
Auto Archiver's logs can be helpful for debugging problematic archiving processes. This guide shows you how to use the logs to
## Setting up logging
Logging settings can be set on the command line or using the orchestration config file ([learn more](../installation/configuration)). A special `logging` section defines the logging options.
#### Enabling or Disabling Logging
Logging to the console is enabled by default. If you want to globally disable Auto Archiver's logging, then you can set `enabled: false` in your `logging` config:
```{code} yaml
...
logging:
enabled: false
...
```
```{note}
This will disable all logs from Auto Archiver, but it does not disable logs for other tools that the Auto Archiver uses (for example: yt-dlp, firefox or ffmpeg). These logs will still appear in your console.
```
#### Logging Level
There are 7 logging levels in total, with 4 commonly used levels. They are: `DEBUG`, `INFO`, `WARNING` and `ERROR`.
Change the warning level by setting the value in your orchestration config file:
```{code} yaml
:caption: orchestration.yaml
...
logging:
level: DEBUG # or INFO / WARNING / ERROR
...
```
For normal usage, it is recommended to use the `INFO` level, or if you prefer quieter logs with less information, you can use the `WARNING` level. If you encounter issues with the archiving, then it's recommended to enable the `DEBUG` level.
```{note} To learn about all logging levels, see the [loguru documentation](https://loguru.readthedocs.io/en/stable/api/logger.html)
```
### Logging to a file
As default, auto-archiver will log to the console. But if you wish to store your logs for future reference, or you are running the auto-archiver from within code a implementation, then you may with to enable file logging. This can be done by setting the `file:` config value in the logging settings.
**Rotation:** For file logging, you can choose to 'rotate' your log files (creating new log files) so they do not get too large. Change this by setting the 'rotation' option in your logging settings. For a full list of rotation options, see the [loguru docs](https://loguru.readthedocs.io/en/stable/overview.html#easier-file-logging-with-rotation-retention-compression).
```{code} yaml
:caption: orchestration.yaml
logging:
...
file: /my/log/file.log
rotation: 1 day
```
### Full logging example
The below example logs only `WARNING` logs to the console and to the file `/my/file.log`, rotating that file once per week:
```{code} yaml
:caption: orchestration.yaml
logging:
level: WARNING
file: /my/file.log
rotation: 1 week
```

146
docs/source/how_to/new_config_format.md Normal file
View File

@@ -0,0 +1,146 @@
# Upgrading from v0.12
```{note} This how-to is only relevant for people who used Auto Archiver before February 2025 (versions prior to 0.13).
If you are new to Auto Archiver, then you are already using the latest configuration format and this how-to is not relevant for you.
```
Versions 0.13+ of Auto Archiver has breaking changes in the configuration format, which means earlier configuration formats will not work without slight modifications.
## How do I know if I need to update my configuration format?
There are two simple ways to check if you need to update your format:
1. When you try and run auto-archiver using your existing configuration file, you get an error about no feeders or formatters being configured, like:
```{code} console
AssertionError: No feeders were configured. Make sure to set at least one feeder in
your configuration file or on the command line (using --feeders)
```
2. Within your configuration file, you have a `feeder:` option. This is the old format. An example old format:
```{code} yaml
steps:
feeder: cli_feeder
...
```
The next two sections outline the two methods you have for updating your file.
## 1. Manually edit the configuration file and change the values.
This is recommended if you want to keep all your old settings. Follow the steps below to change the relevant settings:
#### a) Feeder & Formatter Steps Settings
The feeder and formatter settings have been changed from a single string to a list.
- `steps.feeder (string)` → `steps.feeders (list)`
- `steps.formatter (string)` → `steps.formatters (list)`
Example:
```{code} yaml
steps:
feeder: cli_feeder
...
formatter: html_formatter
# the above should be changed to:
steps:
feeders:
- cli_feeder
...
formatters:
- html_formatter
```
```{note} Auto Archiver still only supports one feeder and formatter, but from v0.13 onwards they must be added to the configuration file as a list.
```
#### b) Extractor (formerly Archiver) Steps Settings
With v0.13 of Auto Archiver, `archivers` have been renamed to `extractors` to better reflect what they actually do - extract information from a URL. Change the configuration by renaming:
- `steps.archivers` → `steps.extractors`
The names of the actual modules have also changed, so for any extractor modules you have enabled, you will need to rename the `archiver` part to `extractor`. Some examples:
- `telethon_archiver` → `telethon_extractor`
- `wacz_archiver_enricher` → `wacz_extractor_enricher`
- `wayback_archiver_enricher` → `wayback_extractor_enricher`
- `vk_archiver` → `vk_extractor`
#### c) Module Renaming
The `youtube_archiver` has been renamed to `generic_extractor` as it is considered the default/fallback extractor. Read more about the [generic extractor](../modules/autogen/extractor/generic_extractor.md).
The `atlos` modules have been merged into one, as have the `gsheets` feeder and database.
- `atlos_feeder` → `atlos_feeder_db_storage`
- `atlos_storage` → `atlos_feeder_db_storage`
- `atlos_db` → `atlos_feeder_db_storage`
- `gsheet_feeder` → `gsheet_feeder_db`
- `gsheet_db` → `gsheet_feeder_db`
Example:
```{code} yaml
steps:
feeders:
- gsheet_feeder_db # formerly gsheet_feeder
...
extractors: # formerly 'archivers'
- telethon_extractor # formerly telethon_archiver
- generic_extractor # formerly youtube_archiver
- vk_extractor # formerly vk_archiver
databases:
- gsheet_feeder_db # formerly gsheet_db
...
```
```{note}
Don't forget to also rename the configuration settings. For example:
```{code} yaml
gsheet_feeder_db: # formerly gsheet_feeder
service_account: secrets/service_account.json
sheet: My Google Sheet
...
```
#### d) Redundant / Obsolete Modules
With v0.13 of Auto Archiver, the following modules have been removed and their features have been built in to the generic_extractor. You should remove them from the 'steps' section of your configuration file:
* `twitter_archiver` - use the `generic_extractor` for general extraction, or the `twitter_api_extractor` for API access.
* `tiktok_archiver` - use the `generic_extractor` to extract TikTok videos.
## 2. Auto-generate a new config, then copy over your settings.
Using this method, you can have Auto Archiver auto-generate a configuration file for you, then you can copy over the desired settings from your old config file. This is probably the easiest method and quickest to setup, but it may require some trial and error as you copy over your settings.
First, move your existing `orchestration.yaml` file to a different folder or rename it.
Then, you can generate a `simple` or `full` config using:
```{code} console
>>> # generate a simple config
>>> auto-archiver
>>> # config will be written to orchestration.yaml
>>>
>>> # generate a full config
>>> auto-archiver --mode=full
>>>
```
After this, copy over any settings from your old config to the new config.

17
docs/source/index.md Normal file
View File

@@ -0,0 +1,17 @@
```{include} ../../README.md
```
```{toctree}
:maxdepth: 2
:hidden:
:caption: Contents:
Overview <self>
installation/setup
core_modules.md
how_to
contributing
development/developer_guidelines
autoapi/index.rst
```

26
docs/source/index.rst
View File

@@ -1,26 +0,0 @@
.. auto-archiver documentation master file, created by
sphinx-quickstart on Sun Jan 12 20:35:50 2025.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Auto Archiver documentation
===========================
.. note::
This is a work in progress.
.. include:: ../../README.md
:parser: myst
.. toctree::
:maxdepth: 1
:hidden:
:caption: Contents:
user_guidelines
developer_guidelines
configurations

72
docs/source/installation/authentication.md Normal file
View File

@@ -0,0 +1,72 @@
# Authentication
The Authentication framework for auto-archiver allows you to add login details for various websites in a flexible way, directly from the configuration file.
There are two main use cases for authentication:
* Some websites require some kind of authentication in order to view the content. Examples include Facebook, Telegram etc.
* Some websites use anti-bot systems to block bot-like tools from accessing the website. Adding real login information to auto-archiver can sometimes bypass this.
## The Authentication Config
You can save your authentication information directly inside your orchestration config file, or as a separate file (for security/multi-deploy purposes). Whether storing your settings inside the orchestration file, or as a separate file, the configuration format is the same. Currently, auto-archiver supports the following authentication types:
**Username & Password:**
- `username`: str - the username to use for login
- `password`: str - the password to use for login
**API**
- `api_key`: str - the API key to use for login
- `api_secret`: str - the API secret to use for login
**Cookies**
- `cookie`: str - a cookie string to use for login (specific to this site)
- `cookies_from_browser`: str - load cookies from this browser, for this site only.
- `cookies_file`: str - load cookies from this file, for this site only.
```{note}
The Username & Password, and API settings only work with the Generic Extractor. Other modules (like the screenshot enricher) can only use the `cookies` options. Furthermore, many sites can still detect bots and block username/password logins. Twitter/X and YouTube are two prominent ones that block username/password logging.
One of the 'Cookies' options is recommended for the most robust archiving.
```
```{code} yaml
authentication:
# optional file to load authentication information from, for security or multi-system deploy purposes
load_from_file: path/to/authentication/file.txt
# optional setting to load cookies from the named browser on the system, for **ALL** websites
cookies_from_browser: firefox
# optional setting to load cookies from a cookies.txt/cookies.jar file, for **ALL** websites. See note below on extracting these
cookies_file: path/to/cookies.jar
mysite.com:
username: myusername
password: 123
facebook.com:
cookie: single_cookie
othersite.com:
api_key: 123
api_secret: 1234
```
### Recommendations for authentication
1. **Store authentication information separately:**
The authentication part of your configuration contains sensitive information. You should make efforts not to share this with others. For extra security, use the `load_from_file` option to keep your authentication settings out of your configuration file, ideally in a different folder.
2. **Don't use your own personal credentials**
Depending on the website you are extracting information from, there may be rules (Terms of Service) that prohibit you from scraping or extracting information using a bot. If you use your own personal account, there's a possibility it might get blocked/disabled. It's recommended to set up a separate, 'throwaway' account. In that way, if it gets blocked you can easily create another one to continue your archiving.
### How to create a cookies.jar or pass cookies directly to auto-archiver
auto-archiver uses yt-dlp's powerful cookies features under the hood. For instructions on how to extract a cookies.jar (or cookies.txt) file directly from your browser, see the FAQ in the [yt-dlp documentation](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp)
```{note} For developers:
For information on how to access and use authentication settings from within your module, see the `{generic_extractor}` for an example, or view the [`auth_for_site()` function in BaseModule](../autoapi/core/base_module/index.rst)
```

6
docs/source/installation/config_cheatsheet.md Normal file
View File

@@ -0,0 +1,6 @@
# Configuration Cheat Sheet
Below is a list of all configurations for the core modules in Auto Archiver
```{include} ../modules/autogen/configs_cheatsheet.md
```

5
docs/source/installation/config_editor.md Normal file
View File

@@ -0,0 +1,5 @@
# Configuration Editor
```{raw} html
:file: settings.html
```

103
docs/source/installation/configurations.md Normal file
View File

@@ -0,0 +1,103 @@
# Configuration
The recommended way to configure auto-archiver for first-time users is to [run the Auto Archiver](setup.md#running) and have it auto-generate a default configuration for you. Then, if needed, you can edit the configuration file using one of the following methods.
## 1. Configuration file
The configuration file is typically called `orchestration.yaml` and stored in the `secrets` folder on your desktop. The configuration file contains all the settings for your entire Auto Archiver workflow in one easy-to-find place.
If you want to have Auto Archiver run with the recommended 'basic' setup,
### Advanced Configuration
The structure of orchestration file is split into 2 parts: `steps` (what [steps](../flow_overview.md) to use) and `configurations` (settings for individual modules).
A default `orchestration.yaml` will be created for you the first time you run auto-archiver (without any arguments). Here's what it looks like:
<details>
<summary>View exampleorchestration.yaml</summary>
```{literalinclude} ../example.orchestration.yaml
:language: yaml
:caption: orchestration.yaml
```
</details>
## 2. Command Line configuration
You can run auto-archiver directly from the command line, without the need for a configuration file, command line arguments are parsed using the format `module_name.config_value`. For example, a config value of `api_key` in the `instagram_extractor` module would be passed on the command line with the flag `--instagram_extractor.api_key=API_KEY`.
The command line arguments are useful for testing or editing config values and enabling/disabling modules on the fly. When you are happy with your settings, you can store them back in your configuration file by passing the `-s/--store` flag on the command line.
```bash
auto-archiver --instagram_extractor.api_key=123 --other_module.setting --store
# will store the new settings into the configuration file (default: orchestration.yaml)
```
```{note} Arguments passed on the command line override those saved in your settings file. Save them to your config file using the -s or --store flag
```
## Seeing all Configuration Options
View the configurable settings for the core modules on the individual doc pages for each [](../core_modules.md).
You can also view all settings available for the modules you have on your system using the `--help` flag in auto-archiver.
```{code-block} console
:caption: Example output when using the --help flag with auto-archiver
$ auto-archiver --help
...
Positional Arguments:
urls URL(s) to archive, either a single URL or a list of urls, should not come from config.yaml
Options:
--help, -h show a full help message and exit
--version show program's version number and exit
--config CONFIG_FILE the filename of the YAML configuration file (defaults to 'config.yaml')
--mode {simple,full} the mode to run the archiver in
-s, --store, --no-store
Store the created config in the config file
--module_paths MODULE_PATHS [MODULE_PATHS ...]
additional paths to search for modules
--feeders STEPS.FEEDERS [STEPS.FEEDERS ...]
the feeders to use
--enrichers STEPS.ENRICHERS [STEPS.ENRICHERS ...]
the enrichers to use
--extractors STEPS.EXTRACTORS [STEPS.EXTRACTORS ...]
the extractors to use
--databases STEPS.DATABASES [STEPS.DATABASES ...]
the databases to use
--storages STEPS.STORAGES [STEPS.STORAGES ...]
the storages to use
--formatters STEPS.FORMATTERS [STEPS.FORMATTERS ...]
the formatter to use
--authentication AUTHENTICATION
A dictionary of sites and their authentication methods (token, username etc.) that extractors can use to log into a website. If passing this on the command line, use a JSON string. You may
also pass a path to a valid JSON/YAML file which will be parsed.
--logging.level {INFO,DEBUG,ERROR,WARNING}
the logging level to use
--logging.file LOGGING.FILE
the logging file to write to
--logging.rotation LOGGING.ROTATION
the logging rotation to use
Wayback Machine Enricher:
Submits the current URL to the Wayback Machine for archiving and returns either a job ID or the...
--wayback_extractor_enricher.timeout TIMEOUT
seconds to wait for successful archive confirmation from wayback, if more than this passes the result contains the job_id so the status can later be checked manually.
--wayback_extractor_enricher.if_not_archived_within IF_NOT_ARCHIVED_WITHIN
only tell wayback to archive if no archive is available before the number of seconds specified, use None to ignore this option. For more information:
https://docs.google.com/document/d/1Nsv52MvSjbLb2PCpHlat0gkzw0EvtSgpKHu4mk0MnrA
--wayback_extractor_enricher.key KEY
wayback API key. to get credentials visit https://archive.org/account/s3.php
--wayback_extractor_enricher.secret SECRET
wayback API secret. to get credentials visit https://archive.org/account/s3.php
--wayback_extractor_enricher.proxy_http PROXY_HTTP
http proxy to use for wayback requests, eg http://proxy-user:password@proxy-ip:port
--wayback_extractor_enricher.proxy_https PROXY_HTTPS
https proxy to use for wayback requests, eg https://proxy-user:password@proxy-ip:port
```

56
docs/source/installation/installation.md Normal file
View File

@@ -0,0 +1,56 @@
# Installation
There are 3 main ways to use the auto-archiver. We recommend the 'docker' method for most uses. This installs all the requirements in one command.
1. Easiest (recommended): [via docker](#installing-with-docker)
2. Local Install: [using pip](#installing-locally-with-pip)
3. Developer Install: [see the developer guidelines](../development/developer_guidelines)
## 1. Installing with Docker
[![dockeri.co](https://dockerico.blankenship.io/image/bellingcat/auto-archiver)](https://hub.docker.com/r/bellingcat/auto-archiver)
Docker works like a virtual machine running inside your computer, making installation simple. You'll need to first set up Docker, and then download the Auto Archiver 'image':
**a) Download and install docker**
Go to the [Docker website](https://docs.docker.com/get-docker/) and download right version for your operating system.
**b) Pull the Auto Archiver docker image**
Open your command line terminal, and copy-paste / type:
```bash
docker pull bellingcat/auto-archiver
```
This will download the docker image, which may take a while.
That's it, all done! You're now ready to set up [your configuration file](configurations.md). Or, if you want to use the recommended defaults, then you can [run Auto Archiver immediately](setup.md#running-a-docker-install).
------------
## 2. Installing Locally with Pip
1. Make sure you have python 3.10 or higher installed
2. Install the package with your preferred package manager: `pip/pipenv/conda install auto-archiver` or `poetry add auto-archiver`
3. Test it's installed with `auto-archiver --help`
4. Install other local dependency requirements (for example `ffmpeg`, `firefox`)
After this, you're ready to set up your [your configuration file](configurations.md), or if you want to use the recommended defaults, then you can [run Auto Archiver immediately](setup.md#running-a-local-install).
### Installing Local Requirements
If using the local installation method, you will also need to install the following dependencies locally:
1.[ffmpeg](https://www.ffmpeg.org/) - for handling of downloaded videos
2. [firefox](https://www.mozilla.org/en-US/firefox/new/) and [geckodriver](https://github.com/mozilla/geckodriver/releases) on a path folder like `/usr/local/bin` - for taking webpage screenshots with the screenshot enricher
3. (optional) [fonts-noto](https://fonts.google.com/noto) to deal with multiple unicode characters during selenium/geckodriver's screenshots: `sudo apt install fonts-noto -y`.
4. [Browsertrix Crawler docker image](https://hub.docker.com/r/webrecorder/browsertrix-crawler) for the WACZ enricher/archiver
## Developer Install
[See the developer guidelines](../development/developer_guidelines)

14
docs/source/installation/requirements.md Normal file
View File

@@ -0,0 +1,14 @@
# Requirements
Using the Auto Archiver is very simple, but ideally you have some familiarity with using the command line to run programs. ([Command line crash course](https://developer.mozilla.org/en-US/docs/Learn_web_development/Getting_started/Environment_setup/Command_line)).
### System Requirements
* Auto Archiver works on any Windows, macOS and Linux computer
* If you're using the **local install** method, then you should make sure to have python3.10+ installed
### Storage Requirements
By default, Auto Archiver uses your local computer storage for any downloaded media (videos, images etc.). If you're downloading large files, this may take up a lot of your local computer's space (more than 5GB of space).
If your storage space is limited, then you may want to set up an [alternative storage method](../modules/storage.md) for your media.

48685
docs/source/installation/settings.html Normal file
View File

File diff suppressed because one or more lines are too long

77
docs/source/installation/setup.md Normal file
View File

@@ -0,0 +1,77 @@
# Getting Started
```{toctree}
:maxdepth: 1
:hidden:
installation.md
configurations.md
config_editor.md
authentication.md
requirements.md
config_cheatsheet.md
```
## Getting Started
To get started with Auto Archiver, there are 3 main steps you need to complete.
1. [Install Auto Archiver](installation.md)
2. [Setup up your configuration](configurations.md) (if you are ok with the default settings, you can skip this step)
3. Run the archiving process<a id="running"></a>
The way you run the Auto Archiver depends on how you installed it (docker install or local install)
### Running a Docker Install
If you installed Auto Archiver using docker, open up your terminal, and copy-paste / type the following command:
```bash
docker run --rm -v $PWD/secrets:/app/secrets -v $PWD/local_archive:/app/local_archive bellingcat/auto-archiver
```
breaking this command down:
1. `docker run` tells docker to start a new container (an instance of the image)
2. `--rm` makes sure this container is removed after execution (less garbage locally)
3. `-v $PWD/secrets:/app/secrets` - your secrets folder with settings
1. `-v` is a volume flag which means a folder that you have on your computer will be connected to a folder inside the docker container
2. `$PWD/secrets` points to a `secrets/` folder in your current working directory (where your console points to), we use this folder as a best practice to hold all the secrets/tokens/passwords/... you use
3. `/app/secrets` points to the path the docker container where this image can be found
4. `-v $PWD/local_archive:/app/local_archive` - (optional) if you use local_storage
1. `-v` same as above, this is a volume instruction
2. `$PWD/local_archive` is a folder `local_archive/` in case you want to archive locally and have the files accessible outside docker
3. `/app/local_archive` is a folder inside docker that you can reference in your orchestration.yml file
### Example invocations
The invocations below will run the auto-archiver Docker image using a configuration file that you have specified
```bash
# Have auto-archiver run with the default settings, generating a settings file in ./secrets/orchestration.yaml
docker run --rm -v $PWD/secrets:/app/secrets -v $PWD/local_archive:/app/local_archive bellingcat/auto-archiver
# uses the same configuration, but with the `gsheet_feeder`, a header on row 2 and with some different column names
# notice that columns is a dictionary so you need to pass it as JSON and it will override only the values provided
docker run --rm -v $PWD/secrets:/app/secrets -v $PWD/local_archive:/app/local_archive bellingcat/auto-archiver --feeders=gsheet_feeder --gsheet_feeder.sheet="use it on another sheets doc" --gsheet_feeder.header=2 --gsheet_feeder.columns='{"url": "link"}'
# Runs auto-archiver for the first time, but in 'full' mode, enabling all modules to get a full settings file
docker run --rm -v $PWD/secrets:/app/secrets -v $PWD/local_archive:/app/local_archive bellingcat/auto-archiver --mode full
```
------------
### Running a Local Install
### Example invocations
Once all your [local requirements](#installing-local-requirements) are correctly installed, the
```bash
# all the configurations come from ./secrets/orchestration.yaml
auto-archiver --config secrets/orchestration.yaml
# uses the same configurations but for another google docs sheet
# with a header on row 2 and with some different column names
# notice that columns is a dictionary so you need to pass it as JSON and it will override only the values provided
auto-archiver --config secrets/orchestration.yaml --gsheet_feeder.sheet="use it on another sheets doc" --gsheet_feeder.header=2 --gsheet_feeder.columns='{"url": "link"}'
# all the configurations come from orchestration.yaml and specifies that s3 files should be private
auto-archiver --config secrets/orchestration.yaml --s3_storage.private=1
```

15
docs/source/modules/database.md Normal file
View File

@@ -0,0 +1,15 @@
# Database Modules
Database modules are used to store the status and results of the extraction and enrichment processes somewhere. The database modules are responsible for creating and managing entires for each item that has been processed.
The default (enabled) databases are the CSV Database and the Console Database.
```{include} autogen/database.md
```
```{toctree}
:maxdepth: 1
:hidden:
:glob:
autogen/database/*
```

14
docs/source/modules/enricher.md Normal file
View File

@@ -0,0 +1,14 @@
# Enricher Modules
Enricher modules are used to add additional information to the items that have been extracted. Common enrichment tasks include adding metadata to items, such as the hash of the item, a screenshot of the webpage when the item was extracted, or general metadata like the date and time the item was extracted.
```{include} autogen/enricher.md
```
```{toctree}
:maxdepth: 1
:hidden:
:glob:
autogen/enricher/*
```

18
docs/source/modules/extractor.md Normal file
View File

@@ -0,0 +1,18 @@
# Extractor Modules
Extractor modules are used to extract the content of a given URL. Typically, one extractor will work for one website or platform (e.g. a Telegram extractor or an Instagram), however, there are several wide-ranging extractors which work for a wide range of websites.
Extractors that are able to extract content from a wide range of websites include:
1. Generic Extractor: parses videos and images on sites using the powerful yt-dlp library.
2. Wayback Machine Extractor: sends pages to the Wayback machine for archiving, and stores the link.
3. WACZ Extractor: runs a web browser to 'browse' the URL and save a copy of the page in WACZ format.
```{include} autogen/extractor.md
```
```{toctree}
:maxdepth: 1
:hidden:
:glob:
autogen/extractor/*
```

20
docs/source/modules/feeder.md Normal file
View File

@@ -0,0 +1,20 @@
# Feeder Modules
Feeder modules are used to feed URLs into the Auto Archiver for processing. Feeders can take these URLs from a variety of sources, such as a file, a database, or the command line.
The default feeder is the command line feeder (`cli_feeder`), which allows you to input URLs directly into `auto-archiver` from the command line.
Command line feeder usage:
```{code} bash
auto-archiver [options] -- URL1 URL2 ...
```
```{include} autogen/feeder.md
```
```{toctree}
:maxdepth: 1
:glob:
:hidden:
autogen/feeder/*
```

13
docs/source/modules/formatter.md Normal file
View File

@@ -0,0 +1,13 @@
# Formatter Modules
Formatter modules are used to format the data extracted from a URL into a specific format. Currently the most widely-used formatter is the HTML formatter, which formats the data into an easily viewable HTML page.
```{include} autogen/formatter.md
```
```{toctree}
:maxdepth: 1
:hidden:
:glob:
autogen/formatter/*
```

15
docs/source/modules/storage.md Normal file
View File

@@ -0,0 +1,15 @@
# Storage Modules
Storage modules are used to store the data extracted from a URL in a persistent location. This can be on your local hard disk, or on a remote server (e.g. S3 or Google Drive).
The default is to store the files downloaded (e.g. images, videos) in a local directory.
```{include} autogen/storage.md
```
```{toctree}
:maxdepth: 1
:hidden:
:glob:
autogen/storage/*
```

16
docs/source/overview.md Normal file
View File

@@ -0,0 +1,16 @@
```{include} ../../README.md
```
```{toctree}
:maxdepth: 2
:hidden:
:caption: Contents:
Overview <self>
installation/installation.rst
core_modules.md
how_to
development/developer_guidelines
autoapi/index.rst
```

11
docs/source/user_guidelines.rst
View File

@@ -1,11 +0,0 @@
User Guidelines
===============
This section of the documentation provides guidelines for users who want to use the tool,
without needing to modify the code.
To see the developer guidelines, see :ref:`developer_guidelines`.
.. note::
This is a work in progress.

156
example.orchestration.yaml
View File

@@ -1,156 +0,0 @@
steps:
# only 1 feeder allowed
feeder: gsheet_feeder # defaults to cli_feeder
archivers: # order matters, uncomment to activate
- bluesky_archiver
# - vk_archiver
# - telethon_archiver
# - telegram_archiver
# - twitter_archiver
# - twitter_api_archiver
# - instagram_api_archiver
# - instagram_tbot_archiver
# - instagram_archiver
# - tiktok_archiver
- youtubedl_archiver
# - wayback_archiver_enricher
# - wacz_archiver_enricher
enrichers:
- hash_enricher
# - meta_enricher
# - metadata_enricher
# - screenshot_enricher
# - pdq_hash_enricher
# - ssl_enricher
# - timestamping_enricher
# - whisper_enricher
# - thumbnail_enricher
# - wayback_archiver_enricher
# - wacz_archiver_enricher
# - pdq_hash_enricher # if you want to calculate hashes for thumbnails, include this after thumbnail_enricher
formatter: html_formatter # defaults to mute_formatter
storages:
- local_storage
# - s3_storage
# - gdrive_storage
databases:
- console_db
# - csv_db
# - gsheet_db
# - mongo_db
configurations:
gsheet_feeder:
sheet: "your sheet name"
header: 1
service_account: "secrets/service_account.json"
# allow_worksheets: "only parse this worksheet"
# block_worksheets: "blocked sheet 1,blocked sheet 2"
use_sheet_names_in_stored_paths: false
columns:
url: link
status: archive status
folder: destination folder
archive: archive location
date: archive date
thumbnail: thumbnail
timestamp: upload timestamp
title: upload title
text: textual content
screenshot: screenshot
hash: hash
pdq_hash: perceptual hashes
wacz: wacz
replaywebpage: replaywebpage
instagram_tbot_archiver:
api_id: "TELEGRAM_BOT_API_ID"
api_hash: "TELEGRAM_BOT_API_HASH"
# session_file: "secrets/anon"
telethon_archiver:
api_id: "TELEGRAM_BOT_API_ID"
api_hash: "TELEGRAM_BOT_API_HASH"
# session_file: "secrets/anon"
join_channels: false
channel_invites: # if you want to archive from private channels
- invite: https://t.me/+123456789
id: 0000000001
- invite: https://t.me/+123456788
id: 0000000002
twitter_api_archiver:
# either bearer_token only
bearer_token: "TWITTER_BEARER_TOKEN"
# OR all of the below
# consumer_key: ""
# consumer_secret: ""
# access_token: ""
# access_secret: ""
instagram_archiver:
username: "INSTAGRAM_USERNAME"
password: "INSTAGRAM_PASSWORD"
# session_file: "secrets/instaloader.session"
vk_archiver:
username: "or phone number"
password: "vk pass"
session_file: "secrets/vk_config.v2.json"
youtubedl_archiver:
subtitles: true
# use one of the following two methods to authenticate in youtube - either provide a cookies file or use the cookies of the given browser
# for more information, see https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp
# cookie_file: "secrets/youtube_cookies.txt"
# cookies_from_browser: firefox
# proxy: socks5://proxy-user:password@proxy-ip:port
screenshot_enricher:
width: 1280
height: 2300
# to save as pdf, uncomment the following lines and adjust the print options
# save_to_pdf: true
# print_options:
# for all options see https://www.selenium.dev/selenium/docs/api/py/webdriver/selenium.webdriver.common.print_page_options.html
# background: true
# orientation: "portrait"
# scale: 1
# page_width: 8.5in
# page_height: 11in
# margin_top: 0.4in
# margin_bottom: 0.4in
# margin_left: 0.4in
# margin_right: 0.4in
# page_ranges: ""
# shrink_to_fit: true
wayback_archiver_enricher:
timeout: 10
key: "wayback key"
secret: "wayback secret"
hash_enricher:
algorithm: "SHA3-512" # can also be SHA-256
wacz_archiver_enricher:
profile: secrets/profile.tar.gz
local_storage:
save_to: "./local_archive"
save_absolute: true
filename_generator: static
path_generator: flat
s3_storage:
bucket: your-bucket-name
region: reg1
key: S3_KEY
secret: S3_SECRET
endpoint_url: "https://{region}.digitaloceanspaces.com"
cdn_url: "https://{bucket}.{region}.cdn.digitaloceanspaces.com/{key}"
# if private:true S3 urls will not be readable online
private: false
# with 'random' you can generate a random UUID for the URL instead of a predictable path, useful to still have public but unlisted files, alternative is 'default' or not omitted from config
key_path: random
gdrive_storage:
path_generator: url
filename_generator: random
root_folder_id: folder_id_from_url
oauth_token: secrets/gd-token.json # needs to be generated with scripts/create_update_gdrive_oauth_token.py
service_account: "secrets/service_account.json"
csv_db:
csv_file: "./local_archive/db.csv"

507
poetry.lock generated
View File

@@ -1,5 +1,24 @@
# This file is automatically @generated by Poetry 2.0.1 and should not be changed by hand.
[[package]]
name = "accessible-pygments"
version = "0.0.5"
description = "A collection of accessible pygments styles"
optional = false
python-versions = ">=3.9"
groups = ["docs"]
files = [
{file = "accessible_pygments-0.0.5-py3-none-any.whl", hash = "sha256:88ae3211e68a1d0b011504b2ffc1691feafce124b845bd072ab6f9f66f34d4b7"},
{file = "accessible_pygments-0.0.5.tar.gz", hash = "sha256:40918d3e6a2b619ad424cb91e556bd3bd8865443d9f22f1dcdf79e33c8046872"},
]
[package.dependencies]
pygments = ">=1.5"
[package.extras]
dev = ["pillow", "pkginfo (>=1.10)", "playwright", "pre-commit", "setuptools", "twine (>=5.0)"]
tests = ["hypothesis", "pytest"]
[[package]]
name = "alabaster"
version = "1.0.0"
@@ -84,14 +103,14 @@ tests-mypy = ["mypy (>=1.11.1)", "pytest-mypy-plugins"]
[[package]]
name = "authlib"
version = "1.4.0"
version = "1.5.1"
description = "The ultimate Python library in building OAuth and OpenID Connect servers and clients."
optional = false
python-versions = ">=3.9"
groups = ["main"]
files = [
{file = "Authlib-1.4.0-py2.py3-none-any.whl", hash = "sha256:4bb20b978c8b636222b549317c1815e1fe62234fc1c5efe8855d84aebf3a74e3"},
{file = "authlib-1.4.0.tar.gz", hash = "sha256:1c1e6608b5ed3624aeeee136ca7f8c120d6f51f731aa152b153d54741840e1f2"},
{file = "authlib-1.5.1-py2.py3-none-any.whl", hash = "sha256:8408861cbd9b4ea2ff759b00b6f02fd7d81ac5a56d0b2b22c08606c6049aae11"},
{file = "authlib-1.5.1.tar.gz", hash = "sha256:5cbc85ecb0667312c1cdc2f9095680bb735883b123fb509fde1e65b1c5df972e"},
]
[package.dependencies]
@@ -115,33 +134,34 @@ tomli = {version = "*", markers = "python_version < \"3.11\""}
[[package]]
name = "babel"
version = "2.16.0"
version = "2.17.0"
description = "Internationalization utilities"
optional = false
python-versions = ">=3.8"
groups = ["docs"]
files = [
{file = "babel-2.16.0-py3-none-any.whl", hash = "sha256:368b5b98b37c06b7daf6696391c3240c938b37767d4584413e8438c5c435fa8b"},
{file = "babel-2.16.0.tar.gz", hash = "sha256:d1f3554ca26605fe173f3de0c65f750f5a42f924499bf134de6423582298e316"},
{file = "babel-2.17.0-py3-none-any.whl", hash = "sha256:4d0b53093fdfb4b21c92b5213dba5a1b23885afa8383709427046b21c366e5f2"},
{file = "babel-2.17.0.tar.gz", hash = "sha256:0c54cffb19f690cdcc52a3b50bcbf71e07a808d1c80d549f2459b9d2cf0afb9d"},
]
[package.extras]
dev = ["freezegun (>=1.0,<2.0)", "pytest (>=6.0)", "pytest-cov"]
dev = ["backports.zoneinfo", "freezegun (>=1.0,<2.0)", "jinja2 (>=3.0)", "pytest (>=6.0)", "pytest-cov", "pytz", "setuptools", "tzdata"]
[[package]]
name = "beautifulsoup4"
version = "4.12.3"
version = "4.13.3"
description = "Screen-scraping library"
optional = false
python-versions = ">=3.6.0"
python-versions = ">=3.7.0"
groups = ["main", "docs"]
files = [
{file = "beautifulsoup4-4.12.3-py3-none-any.whl", hash = "sha256:b80878c9f40111313e55da8ba20bdba06d8fa3969fc68304167741bbf9e082ed"},
{file = "beautifulsoup4-4.12.3.tar.gz", hash = "sha256:74e3d1928edc070d21748185c46e3fb33490f22f52a3addee9aee0f4f7781051"},
{file = "beautifulsoup4-4.13.3-py3-none-any.whl", hash = "sha256:99045d7d3f08f91f0d656bc9b7efbae189426cd913d830294a15eefa0ea4df16"},
{file = "beautifulsoup4-4.13.3.tar.gz", hash = "sha256:1bd32405dacc920b42b83ba01644747ed77456a65760e285fbc47633ceddaf8b"},
]
[package.dependencies]
soupsieve = ">1.2"
typing-extensions = ">=4.0.0"
[package.extras]
cchardet = ["cchardet"]
@@ -152,18 +172,18 @@ lxml = ["lxml"]
[[package]]
name = "boto3"
version = "1.36.6"
version = "1.37.8"
description = "The AWS SDK for Python"
optional = false
python-versions = ">=3.8"
groups = ["main"]
files = [
{file = "boto3-1.36.6-py3-none-any.whl", hash = "sha256:6d473f0f340d02b4e9ad5b8e68786a09728101a8b950231b89ebdaf72b6dca21"},
{file = "boto3-1.36.6.tar.gz", hash = "sha256:b36feae061dc0793cf311468956a0a9e99215ce38bc99a1a4e55a5b105f16297"},
{file = "boto3-1.37.8-py3-none-any.whl", hash = "sha256:b9f506e08c9f54687d6c073ef1c550a24a62cc2d1e0bc7cda9f13112a38818bf"},
{file = "boto3-1.37.8.tar.gz", hash = "sha256:9448f4a079189e19c3253cfdc5b8ef6dc51a3b82431e8347a51f4c1b2d9dab42"},
]
[package.dependencies]
botocore = ">=1.36.6,<1.37.0"
botocore = ">=1.37.8,<1.38.0"
jmespath = ">=0.7.1,<2.0.0"
s3transfer = ">=0.11.0,<0.12.0"
@@ -172,14 +192,14 @@ crt = ["botocore[crt] (>=1.21.0,<2.0a0)"]
[[package]]
name = "botocore"
version = "1.36.6"
version = "1.37.8"
description = "Low-level, data-driven core of boto 3."
optional = false
python-versions = ">=3.8"
groups = ["main"]
files = [
{file = "botocore-1.36.6-py3-none-any.whl", hash = "sha256:f77bbbb03fb420e260174650fb5c0cc142ec20a96967734eed2b0ef24334ef34"},
{file = "botocore-1.36.6.tar.gz", hash = "sha256:4864c53d638da191a34daf3ede3ff1371a3719d952cc0c6bd24ce2836a38dd77"},
{file = "botocore-1.37.8-py3-none-any.whl", hash = "sha256:a6c94f33de12f4b10b10684019e554c980469b8394c6d82448a738cbd8452cef"},
{file = "botocore-1.37.8.tar.gz", hash = "sha256:b5825e08dd3e25642aa22a0d7d92bf81fef1ef857117e4155f923bbccf5aba63"},
]
[package.dependencies]
@@ -188,7 +208,7 @@ python-dateutil = ">=2.1,<3.0.0"
urllib3 = {version = ">=1.25.4,<2.2.0 || >2.2.0,<3", markers = "python_version >= \"3.10\""}
[package.extras]
crt = ["awscrt (==0.23.4)"]
crt = ["awscrt (==0.23.8)"]
[[package]]
name = "brotli"
@@ -343,26 +363,26 @@ beautifulsoup4 = "*"
[[package]]
name = "cachetools"
version = "5.5.1"
version = "5.5.2"
description = "Extensible memoizing collections and decorators"
optional = false
python-versions = ">=3.7"
groups = ["main"]
files = [
{file = "cachetools-5.5.1-py3-none-any.whl", hash = "sha256:b76651fdc3b24ead3c648bbdeeb940c1b04d365b38b4af66788f9ec4a81d42bb"},
{file = "cachetools-5.5.1.tar.gz", hash = "sha256:70f238fbba50383ef62e55c6aff6d9673175fe59f7c6782c7a0b9e38f4a9df95"},
{file = "cachetools-5.5.2-py3-none-any.whl", hash = "sha256:d26a22bcc62eb95c3beabd9f1ee5e820d3d2704fe2967cbe350e20c8ffcd3f0a"},
{file = "cachetools-5.5.2.tar.gz", hash = "sha256:1a661caa9175d26759571b2e19580f9d6393969e5dfca11fdb1f947a23e640d4"},
]
[[package]]
name = "certifi"
version = "2024.12.14"
version = "2025.1.31"
description = "Python package for providing Mozilla's CA Bundle."
optional = false
python-versions = ">=3.6"
groups = ["main", "docs"]
files = [
{file = "certifi-2024.12.14-py3-none-any.whl", hash = "sha256:1275f7a45be9464efc1173084eaa30f866fe2e47d389406136d332ed4967ec56"},
{file = "certifi-2024.12.14.tar.gz", hash = "sha256:b650d30f370c2b724812bee08008be0c4163b163ddaec3f2546c1caf65f191db"},
{file = "certifi-2025.1.31-py3-none-any.whl", hash = "sha256:ca78db4565a652026a4db2bcdf68f2fb589ea80d0be70e03929ed730746b84fe"},
{file = "certifi-2025.1.31.tar.gz", hash = "sha256:3d5da6925056f6f18f119200434a4780a94263f10d1c21d032a6f6b2baa20651"},
]
[[package]]
@@ -655,26 +675,26 @@ typing-inspect = ">=0.4.0,<1"
[[package]]
name = "dateparser"
version = "1.2.0"
version = "1.2.1"
description = "Date parsing library designed to parse dates from HTML pages"
optional = false
python-versions = ">=3.7"
python-versions = ">=3.8"
groups = ["main"]
files = [
{file = "dateparser-1.2.0-py2.py3-none-any.whl", hash = "sha256:0b21ad96534e562920a0083e97fd45fa959882d4162acc358705144520a35830"},
{file = "dateparser-1.2.0.tar.gz", hash = "sha256:7975b43a4222283e0ae15be7b4999d08c9a70e2d378ac87385b1ccf2cffbbb30"},
{file = "dateparser-1.2.1-py3-none-any.whl", hash = "sha256:bdcac262a467e6260030040748ad7c10d6bacd4f3b9cdb4cfd2251939174508c"},
{file = "dateparser-1.2.1.tar.gz", hash = "sha256:7e4919aeb48481dbfc01ac9683c8e20bfe95bb715a38c1e9f6af889f4f30ccc3"},
]
[package.dependencies]
python-dateutil = "*"
pytz = "*"
regex = "<2019.02.19 || >2019.02.19,<2021.8.27 || >2021.8.27"
tzlocal = "*"
python-dateutil = ">=2.7.0"
pytz = ">=2024.2"
regex = ">=2015.06.24,<2019.02.19 || >2019.02.19,<2021.8.27 || >2021.8.27"
tzlocal = ">=0.2"
[package.extras]
calendars = ["convertdate", "hijri-converter"]
fasttext = ["fasttext"]
langdetect = ["langdetect"]
calendars = ["convertdate (>=2.2.1)", "hijridate"]
fasttext = ["fasttext (>=0.9.1)", "numpy (>=1.19.3,<2)"]
langdetect = ["langdetect (>=1.0.0)"]
[[package]]
name = "docutils"
@@ -722,24 +742,6 @@ future = "*"
[package.extras]
dev = ["Sphinx (==2.1.0)", "future (==0.17.1)", "numpy (==1.16.4)", "pytest (==4.6.1)", "pytest-mock (==1.10.4)", "tox (==3.12.1)"]
[[package]]
name = "furo"
version = "2024.8.6"
description = "A clean customisable Sphinx documentation theme."
optional = false
python-versions = ">=3.8"
groups = ["docs"]
files = [
{file = "furo-2024.8.6-py3-none-any.whl", hash = "sha256:6cd97c58b47813d3619e63e9081169880fbe331f0ca883c871ff1f3f11814f5c"},
{file = "furo-2024.8.6.tar.gz", hash = "sha256:b63e4cee8abfc3136d3bc03a3d45a76a850bada4d6374d24c1716b0e01394a01"},
]
[package.dependencies]
beautifulsoup4 = "*"
pygments = ">=2.7"
sphinx = ">=6.0,<9.0"
sphinx-basic-ng = ">=1.0.0.beta2"
[[package]]
name = "future"
version = "1.0.0"
@@ -754,14 +756,14 @@ files = [
[[package]]
name = "google-api-core"
version = "2.24.0"
version = "2.24.1"
description = "Google API client core library"
optional = false
python-versions = ">=3.7"
groups = ["main"]
files = [
{file = "google_api_core-2.24.0-py3-none-any.whl", hash = "sha256:10d82ac0fca69c82a25b3efdeefccf6f28e02ebb97925a8cce8edbfe379929d9"},
{file = "google_api_core-2.24.0.tar.gz", hash = "sha256:e255640547a597a4da010876d333208ddac417d60add22b6851a0c66a831fcaf"},
{file = "google_api_core-2.24.1-py3-none-any.whl", hash = "sha256:bc78d608f5a5bf853b80bd70a795f703294de656c096c0968320830a4bc280f1"},
{file = "google_api_core-2.24.1.tar.gz", hash = "sha256:f8b36f5456ab0dd99a1b693a40a31d1e7757beea380ad1b38faaf8941eae9d8a"},
]
[package.dependencies]
@@ -779,14 +781,14 @@ grpcio-gcp = ["grpcio-gcp (>=0.2.2,<1.0.dev0)"]
[[package]]
name = "google-api-python-client"
version = "2.159.0"
version = "2.163.0"
description = "Google API Client Library for Python"
optional = false
python-versions = ">=3.7"
groups = ["main"]
files = [
{file = "google_api_python_client-2.159.0-py2.py3-none-any.whl", hash = "sha256:baef0bb631a60a0bd7c0bf12a5499e3a40cd4388484de7ee55c1950bf820a0cf"},
{file = "google_api_python_client-2.159.0.tar.gz", hash = "sha256:55197f430f25c907394b44fa078545ffef89d33fd4dca501b7db9f0d8e224bd6"},
{file = "google_api_python_client-2.163.0-py2.py3-none-any.whl", hash = "sha256:080e8bc0669cb4c1fb8efb8da2f5b91a2625d8f0e7796cfad978f33f7016c6c4"},
{file = "google_api_python_client-2.163.0.tar.gz", hash = "sha256:88dee87553a2d82176e2224648bf89272d536c8f04dcdda37ef0a71473886dd7"},
]
[package.dependencies]
@@ -858,14 +860,14 @@ tool = ["click (>=6.0.0)"]
[[package]]
name = "googleapis-common-protos"
version = "1.66.0"
version = "1.69.1"
description = "Common protobufs used in Google APIs"
optional = false
python-versions = ">=3.7"
groups = ["main"]
files = [
{file = "googleapis_common_protos-1.66.0-py2.py3-none-any.whl", hash = "sha256:d7abcd75fabb2e0ec9f74466401f6c119a0b498e27370e9be4c94cb7e382b8ed"},
{file = "googleapis_common_protos-1.66.0.tar.gz", hash = "sha256:c3e7b33d15fdca5374cc0a7346dd92ffa847425cc4ea941d970f13680052ec8c"},
{file = "googleapis_common_protos-1.69.1-py2.py3-none-any.whl", hash = "sha256:4077f27a6900d5946ee5a369fab9c8ded4c0ef1c6e880458ea2f70c14f7b70d5"},
{file = "googleapis_common_protos-1.69.1.tar.gz", hash = "sha256:e20d2d8dda87da6fe7340afbbdf4f0bcb4c8fae7e6cadf55926c31f946b0b9b1"},
]
[package.dependencies]
@@ -876,14 +878,14 @@ grpc = ["grpcio (>=1.44.0,<2.0.0.dev0)"]
[[package]]
name = "gspread"
version = "6.1.4"
version = "6.2.0"
description = "Google Spreadsheets Python API"
optional = false
python-versions = ">=3.8"
groups = ["main"]
files = [
{file = "gspread-6.1.4-py3-none-any.whl", hash = "sha256:c34781c426031a243ad154952b16f21ac56a5af90687885fbee3d1fba5280dcd"},
{file = "gspread-6.1.4.tar.gz", hash = "sha256:b8eec27de7cadb338bb1b9f14a9be168372dee8965c0da32121816b5050ac1de"},
{file = "gspread-6.2.0-py3-none-any.whl", hash = "sha256:7fa1a11e1ecacc6c5946fa016be05941baca8540404314f59aec963dd8ae5db3"},
{file = "gspread-6.2.0.tar.gz", hash = "sha256:bc3d02d1c39e0b40bfc8035b4fec407aa71a17f343fc81cc7e3f75bfa6555de6"},
]
[package.dependencies]
@@ -976,14 +978,14 @@ browser-cookie3 = ["browser_cookie3 (>=0.19.1)"]
[[package]]
name = "jinja2"
version = "3.1.5"
version = "3.1.6"
description = "A very fast and expressive template engine."
optional = false
python-versions = ">=3.7"
groups = ["main", "docs"]
files = [
{file = "jinja2-3.1.5-py3-none-any.whl", hash = "sha256:aba0f4dc9ed8013c424088f68a5c226f7d6097ed89b246d7749c2ec4175c6adb"},
{file = "jinja2-3.1.5.tar.gz", hash = "sha256:8fefff8dc3034e27bb80d67c671eb8a9bc424c0ef4c0826edbff304cceff43bb"},
{file = "jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67"},
{file = "jinja2-3.1.6.tar.gz", hash = "sha256:0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d"},
]
[package.dependencies]
@@ -1019,6 +1021,27 @@ files = [
[package.dependencies]
attrs = ">=19.2.0"
[[package]]
name = "linkify-it-py"
version = "2.0.3"
description = "Links recognition library with FULL unicode support."
optional = false
python-versions = ">=3.7"
groups = ["docs"]
files = [
{file = "linkify-it-py-2.0.3.tar.gz", hash = "sha256:68cda27e162e9215c17d786649d1da0021a451bdc436ef9e0fa0ba5234b9b048"},
{file = "linkify_it_py-2.0.3-py3-none-any.whl", hash = "sha256:6bcbc417b0ac14323382aef5c5192c0075bf8a9d6b41820a2b66371eac6b6d79"},
]
[package.dependencies]
uc-micro-py = "*"
[package.extras]
benchmark = ["pytest", "pytest-benchmark"]
dev = ["black", "flake8", "isort", "pre-commit", "pyproject-flake8"]
doc = ["myst-parser", "sphinx", "sphinx-book-theme"]
test = ["coverage", "pytest", "pytest-cov"]
[[package]]
name = "loguru"
version = "0.7.3"
@@ -1136,14 +1159,14 @@ files = [
[[package]]
name = "marshmallow"
version = "3.26.0"
version = "3.26.1"
description = "A lightweight library for converting complex datatypes to and from native Python datatypes."
optional = false
python-versions = ">=3.9"
groups = ["main"]
files = [
{file = "marshmallow-3.26.0-py3-none-any.whl", hash = "sha256:1287bca04e6a5f4094822ac153c03da5e214a0a60bcd557b140f3e66991b8ca1"},
{file = "marshmallow-3.26.0.tar.gz", hash = "sha256:eb36762a1cc76d7abf831e18a3a1b26d3d481bbc74581b8e532a3d3a8115e1cb"},
{file = "marshmallow-3.26.1-py3-none-any.whl", hash = "sha256:3350409f20a70a7e4e11a27661187b77cdcaeb20abca41c1454fe33636bea09c"},
{file = "marshmallow-3.26.1.tar.gz", hash = "sha256:e6d8affb6cb61d39d26402096dc0aee12d5a26d490a121f118d2e81dc0719dc6"},
]
[package.dependencies]
@@ -1212,14 +1235,14 @@ files = [
[[package]]
name = "myst-parser"
version = "4.0.0"
version = "4.0.1"
description = "An extended [CommonMark](https://spec.commonmark.org/) compliant parser,"
optional = false
python-versions = ">=3.10"
groups = ["docs"]
files = [
{file = "myst_parser-4.0.0-py3-none-any.whl", hash = "sha256:b9317997552424448c6096c2558872fdb6f81d3ecb3a40ce84a7518798f3f28d"},
{file = "myst_parser-4.0.0.tar.gz", hash = "sha256:851c9dfb44e36e56d15d05e72f02b80da21a9e0d07cba96baf5e2d476bb91531"},
{file = "myst_parser-4.0.1-py3-none-any.whl", hash = "sha256:9134e88959ec3b5780aedf8a99680ea242869d012e8821db3126d427edc9c95d"},
{file = "myst_parser-4.0.1.tar.gz", hash = "sha256:5cfea715e4f3574138aecbf7d54132296bfd72bb614d31168f48c477a830a7c4"},
]
[package.dependencies]
@@ -1231,10 +1254,10 @@ pyyaml = "*"
sphinx = ">=7,<9"
[package.extras]
code-style = ["pre-commit (>=3.0,<4.0)"]
code-style = ["pre-commit (>=4.0,<5.0)"]
linkify = ["linkify-it-py (>=2.0,<3.0)"]
rtd = ["ipython", "sphinx (>=7)", "sphinx-autodoc2 (>=0.5.0,<0.6.0)", "sphinx-book-theme (>=1.1,<2.0)", "sphinx-copybutton", "sphinx-design", "sphinx-pyscript", "sphinx-tippy (>=0.4.3)", "sphinx-togglebutton", "sphinxext-opengraph (>=0.9.0,<0.10.0)", "sphinxext-rediraffe (>=0.2.7,<0.3.0)"]
testing = ["beautifulsoup4", "coverage[toml]", "defusedxml", "pytest (>=8,<9)", "pytest-cov", "pytest-param-files (>=0.6.0,<0.7.0)", "pytest-regressions", "sphinx-pytest"]
testing = ["beautifulsoup4", "coverage[toml]", "defusedxml", "pygments (<2.19)", "pytest (>=8,<9)", "pytest-cov", "pytest-param-files (>=0.6.0,<0.7.0)", "pytest-regressions", "sphinx-pytest"]
testing-docutils = ["pygments", "pytest (>=8,<9)", "pytest-param-files (>=0.6.0,<0.7.0)"]
[[package]]
@@ -1345,18 +1368,14 @@ description = "TLS (SSL) sockets, key generation, encryption, decryption, signin
optional = false
python-versions = "*"
groups = ["main"]
files = []
develop = false
files = [
{file = "oscrypto-1.3.0-py2.py3-none-any.whl", hash = "sha256:2b2f1d2d42ec152ca90ccb5682f3e051fb55986e1b170ebde472b133713e7085"},
{file = "oscrypto-1.3.0.tar.gz", hash = "sha256:6f5fef59cb5b3708321db7cca56aed8ad7e662853351e7991fcf60ec606d47a4"},
]
[package.dependencies]
asn1crypto = ">=1.5.1"
[package.source]
type = "git"
url = "https://github.com/wbond/oscrypto.git"
reference = "d5f3437ed24257895ae1edd9e503cfb352e635a8"
resolved_reference = "d5f3437ed24257895ae1edd9e503cfb352e635a8"
[[package]]
name = "outcome"
version = "1.3.0.post0"
@@ -1512,14 +1531,14 @@ testing = ["pytest", "pytest-benchmark"]
[[package]]
name = "proto-plus"
version = "1.25.0"
description = "Beautiful, Pythonic protocol buffers."
version = "1.26.0"
description = "Beautiful, Pythonic protocol buffers"
optional = false
python-versions = ">=3.7"
groups = ["main"]
files = [
{file = "proto_plus-1.25.0-py3-none-any.whl", hash = "sha256:c91fc4a65074ade8e458e95ef8bac34d4008daa7cce4a12d6707066fca648961"},
{file = "proto_plus-1.25.0.tar.gz", hash = "sha256:fbb17f57f7bd05a68b7707e745e26528b0b3c34e378db91eef93912c54982d91"},
{file = "proto_plus-1.26.0-py3-none-any.whl", hash = "sha256:bf2dfaa3da281fc3187d12d224c707cb57214fb2c22ba854eb0c105a3fb2d4d7"},
{file = "proto_plus-1.26.0.tar.gz", hash = "sha256:6e93d5f5ca267b54300880fff156b6a3386b3fa3f43b1da62e680fc0c586ef22"},
]
[package.dependencies]
@@ -1653,6 +1672,35 @@ files = [
{file = "pycryptodomex-3.21.0.tar.gz", hash = "sha256:222d0bd05381dd25c32dd6065c071ebf084212ab79bab4599ba9e6a3e0009e6c"},
]
[[package]]
name = "pydata-sphinx-theme"
version = "0.15.4"
description = "Bootstrap-based Sphinx theme from the PyData community"
optional = false
python-versions = ">=3.9"
groups = ["docs"]
files = [
{file = "pydata_sphinx_theme-0.15.4-py3-none-any.whl", hash = "sha256:2136ad0e9500d0949f96167e63f3e298620040aea8f9c74621959eda5d4cf8e6"},
{file = "pydata_sphinx_theme-0.15.4.tar.gz", hash = "sha256:7762ec0ac59df3acecf49fd2f889e1b4565dbce8b88b2e29ee06fdd90645a06d"},
]
[package.dependencies]
accessible-pygments = "*"
Babel = "*"
beautifulsoup4 = "*"
docutils = "!=0.17.0"
packaging = "*"
pygments = ">=2.7"
sphinx = ">=5"
typing-extensions = "*"
[package.extras]
a11y = ["pytest-playwright"]
dev = ["pandoc", "pre-commit", "pydata-sphinx-theme[doc,test]", "pyyaml", "sphinx-theme-builder[cli]", "tox"]
doc = ["ablog (>=0.11.8)", "colorama", "graphviz", "ipykernel", "ipyleaflet", "ipywidgets", "jupyter_sphinx", "jupyterlite-sphinx", "linkify-it-py", "matplotlib", "myst-parser", "nbsphinx", "numpy", "numpydoc", "pandas", "plotly", "rich", "sphinx-autoapi (>=3.0.0)", "sphinx-copybutton", "sphinx-design", "sphinx-favicon (>=1.0.1)", "sphinx-sitemap", "sphinx-togglebutton", "sphinxcontrib-youtube (>=1.4.1)", "sphinxext-rediraffe", "xarray"]
i18n = ["Babel", "jinja2"]
test = ["pytest", "pytest-cov", "pytest-regressions", "sphinx[test]"]
[[package]]
name = "pygments"
version = "2.19.1"
@@ -1729,14 +1777,14 @@ files = [
[[package]]
name = "pytest"
version = "8.3.4"
version = "8.3.5"
description = "pytest: simple powerful testing with Python"
optional = false
python-versions = ">=3.8"
groups = ["dev"]
files = [
{file = "pytest-8.3.4-py3-none-any.whl", hash = "sha256:50e16d954148559c9a74109af1eaf0c945ba2d8f30f0a3d3335edde19788b6f6"},
{file = "pytest-8.3.4.tar.gz", hash = "sha256:965370d062bce11e73868e0335abac31b4d3de0e82f4007408d242b4f8610761"},
{file = "pytest-8.3.5-py3-none-any.whl", hash = "sha256:c69214aa47deac29fad6c2a4f590b9c4a9fdb16a403176fe154b79c0b4d4d820"},
{file = "pytest-8.3.5.tar.gz", hash = "sha256:f4efe70cc14e511565ac476b57c279e12a855b11f48f212af1080ef2263d3845"},
]
[package.dependencies]
@@ -1768,6 +1816,24 @@ loguru = "*"
[package.extras]
test = ["pytest", "pytest-cov"]
[[package]]
name = "pytest-mock"
version = "3.14.0"
description = "Thin-wrapper around the mock package for easier use with pytest"
optional = false
python-versions = ">=3.8"
groups = ["dev"]
files = [
{file = "pytest-mock-3.14.0.tar.gz", hash = "sha256:2719255a1efeceadbc056d6bf3df3d1c5015530fb40cf347c0f9afac88410bd0"},
{file = "pytest_mock-3.14.0-py3-none-any.whl", hash = "sha256:0b72c38033392a5f4621342fe11e9219ac11ec9d375f8e2a0c164539e0d70f6f"},
]
[package.dependencies]
pytest = ">=6.2.5"
[package.extras]
dev = ["pre-commit", "pytest-asyncio", "tox"]
[[package]]
name = "python-dateutil"
version = "2.9.0.post0"
@@ -1820,14 +1886,14 @@ requests = ">=2.28"
[[package]]
name = "pytz"
version = "2024.2"
version = "2025.1"
description = "World timezone definitions, modern and historical"
optional = false
python-versions = "*"
groups = ["main"]
files = [
{file = "pytz-2024.2-py2.py3-none-any.whl", hash = "sha256:31c7c1817eb7fae7ca4b8c7ee50c72f93aa2dd863de768e1ef4245d426aa0725"},
{file = "pytz-2024.2.tar.gz", hash = "sha256:2aa355083c50a0f93fa581709deac0c9ad65cca8a9e9beac660adcbd493c798a"},
{file = "pytz-2025.1-py2.py3-none-any.whl", hash = "sha256:89dd22dca55b46eac6eda23b2d72721bf1bdfef212645d81513ef5d03038de57"},
{file = "pytz-2025.1.tar.gz", hash = "sha256:c2db42be2a2518b28e65f9207c4d05e6ff547d1efa4086469ef855e4ab70178e"},
]
[[package]]
@@ -2076,14 +2142,14 @@ jupyter = ["ipywidgets (>=7.5.1,<9)"]
[[package]]
name = "rich-argparse"
version = "1.6.0"
version = "1.7.0"
description = "Rich help formatters for argparse and optparse"
optional = false
python-versions = ">=3.8"
groups = ["main"]
files = [
{file = "rich_argparse-1.6.0-py3-none-any.whl", hash = "sha256:fbe70a1d821b3f2fa8958cddf0cae131870a6e9faa04ab52b409cb1eda809bd7"},
{file = "rich_argparse-1.6.0.tar.gz", hash = "sha256:092083c30da186f25bcdff8b1d47fdfb571288510fb051e0488a72cc3128de13"},
{file = "rich_argparse-1.7.0-py3-none-any.whl", hash = "sha256:b8ec8943588e9731967f4f97b735b03dc127c416f480a083060433a97baf2fd3"},
{file = "rich_argparse-1.7.0.tar.gz", hash = "sha256:f31d809c465ee43f367d599ccaf88b73bc2c4d75d74ed43f2d538838c53544ba"},
]
[package.dependencies]
@@ -2182,32 +2248,32 @@ files = [
[[package]]
name = "s3transfer"
version = "0.11.2"
version = "0.11.4"
description = "An Amazon S3 Transfer Manager"
optional = false
python-versions = ">=3.8"
groups = ["main"]
files = [
{file = "s3transfer-0.11.2-py3-none-any.whl", hash = "sha256:be6ecb39fadd986ef1701097771f87e4d2f821f27f6071c872143884d2950fbc"},
{file = "s3transfer-0.11.2.tar.gz", hash = "sha256:3b39185cb72f5acc77db1a58b6e25b977f28d20496b6e58d6813d75f464d632f"},
{file = "s3transfer-0.11.4-py3-none-any.whl", hash = "sha256:ac265fa68318763a03bf2dc4f39d5cbd6a9e178d81cc9483ad27da33637e320d"},
{file = "s3transfer-0.11.4.tar.gz", hash = "sha256:559f161658e1cf0a911f45940552c696735f5c74e64362e515f333ebed87d679"},
]
[package.dependencies]
botocore = ">=1.36.0,<2.0a.0"
botocore = ">=1.37.4,<2.0a.0"
[package.extras]
crt = ["botocore[crt] (>=1.36.0,<2.0a.0)"]
crt = ["botocore[crt] (>=1.37.4,<2.0a.0)"]
[[package]]
name = "selenium"
version = "4.28.1"
version = "4.29.0"
description = "Official Python bindings for Selenium WebDriver"
optional = false
python-versions = ">=3.9"
groups = ["main"]
files = [
{file = "selenium-4.28.1-py3-none-any.whl", hash = "sha256:4238847e45e24e4472cfcf3554427512c7aab9443396435b1623ef406fff1cc1"},
{file = "selenium-4.28.1.tar.gz", hash = "sha256:0072d08670d7ec32db901bd0107695a330cecac9f196e3afb3fa8163026e022a"},
{file = "selenium-4.29.0-py3-none-any.whl", hash = "sha256:ce5d26f1ddc1111641113653af33694c13947dd36c2df09cdd33f554351d372e"},
{file = "selenium-4.29.0.tar.gz", hash = "sha256:3a62f7ec33e669364a6c0562a701deb69745b569c50d55f1a912bf8eb33358ba"},
]
[package.dependencies]
@@ -2316,24 +2382,24 @@ test = ["cython (>=3.0)", "defusedxml (>=0.7.1)", "pytest (>=8.0)", "setuptools
[[package]]
name = "sphinx-autoapi"
version = "3.4.0"
version = "3.6.0"
description = "Sphinx API documentation generator"
optional = false
python-versions = ">=3.8"
python-versions = ">=3.9"
groups = ["docs"]
files = [
{file = "sphinx_autoapi-3.4.0-py3-none-any.whl", hash = "sha256:4027fef2875a22c5f2a57107c71641d82f6166bf55beb407a47aaf3ef14e7b92"},
{file = "sphinx_autoapi-3.4.0.tar.gz", hash = "sha256:e6d5371f9411bbb9fca358c00a9e57aef3ac94cbfc5df4bab285946462f69e0c"},
{file = "sphinx_autoapi-3.6.0-py3-none-any.whl", hash = "sha256:f3b66714493cab140b0e896d33ce7137654a16ac1edb6563edcbd47bf975f711"},
{file = "sphinx_autoapi-3.6.0.tar.gz", hash = "sha256:c685f274e41d0842ae7e199460c322c4bd7fec816ccc2da8d806094b4f64af06"},
]
[package.dependencies]
astroid = [
{version = ">=2.7", markers = "python_version < \"3.12\""},
{version = ">=3.0.0a1", markers = "python_version >= \"3.12\""},
{version = ">=3", markers = "python_version >= \"3.12\""},
]
Jinja2 = "*"
PyYAML = "*"
sphinx = ">=6.1.0"
sphinx = ">=7.4.0"
[[package]]
name = "sphinx-autobuild"
@@ -2359,22 +2425,25 @@ websockets = ">=11"
test = ["httpx", "pytest (>=6)"]
[[package]]
name = "sphinx-basic-ng"
version = "1.0.0b2"
description = "A modern skeleton for Sphinx themes."
name = "sphinx-book-theme"
version = "1.1.4"
description = "A clean book theme for scientific explanations and documentation with Sphinx"
optional = false
python-versions = ">=3.7"
python-versions = ">=3.9"
groups = ["docs"]
files = [
{file = "sphinx_basic_ng-1.0.0b2-py3-none-any.whl", hash = "sha256:eb09aedbabfb650607e9b4b68c9d240b90b1e1be221d6ad71d61c52e29f7932b"},
{file = "sphinx_basic_ng-1.0.0b2.tar.gz", hash = "sha256:9ec55a47c90c8c002b5960c57492ec3021f5193cb26cebc2dc4ea226848651c9"},
{file = "sphinx_book_theme-1.1.4-py3-none-any.whl", hash = "sha256:843b3f5c8684640f4a2d01abd298beb66452d1b2394cd9ef5be5ebd5640ea0e1"},
{file = "sphinx_book_theme-1.1.4.tar.gz", hash = "sha256:73efe28af871d0a89bd05856d300e61edce0d5b2fbb7984e84454be0fedfe9ed"},
]
[package.dependencies]
sphinx = ">=4.0"
pydata-sphinx-theme = "0.15.4"
sphinx = ">=6.1"
[package.extras]
docs = ["furo", "ipython", "myst-parser", "sphinx-copybutton", "sphinx-inline-tabs"]
code-style = ["pre-commit"]
doc = ["ablog", "folium", "ipywidgets", "matplotlib", "myst-nb", "nbclient", "numpy", "numpydoc", "pandas", "plotly", "sphinx-copybutton", "sphinx-design", "sphinx-examples", "sphinx-tabs", "sphinx-thebe", "sphinx-togglebutton", "sphinxcontrib-bibtex", "sphinxcontrib-youtube", "sphinxext-opengraph"]
test = ["beautifulsoup4", "coverage", "defusedxml", "myst-nb", "pytest", "pytest-cov", "pytest-regressions", "sphinx_thebe"]
[[package]]
name = "sphinx-copybutton"
@@ -2516,14 +2585,14 @@ test = ["pytest"]
[[package]]
name = "starlette"
version = "0.45.3"
version = "0.46.0"
description = "The little ASGI library that shines."
optional = false
python-versions = ">=3.9"
groups = ["docs"]
files = [
{file = "starlette-0.45.3-py3-none-any.whl", hash = "sha256:dfb6d332576f136ec740296c7e8bb8c8a7125044e7c6da30744718880cdd059d"},
{file = "starlette-0.45.3.tar.gz", hash = "sha256:2cbcba2a75806f8a41c722141486f37c28e30a0921c5f6fe4346cb0dcee1302f"},
{file = "starlette-0.46.0-py3-none-any.whl", hash = "sha256:913f0798bd90ba90a9156383bcf1350a17d6259451d0d8ee27fc0cf2db609038"},
{file = "starlette-0.46.0.tar.gz", hash = "sha256:b359e4567456b28d473d0193f34c0de0ed49710d75ef183a74a5ce0499324f50"},
]
[package.dependencies]
@@ -2534,14 +2603,14 @@ full = ["httpx (>=0.27.0,<0.29.0)", "itsdangerous", "jinja2", "python-multipart
[[package]]
name = "telethon"
version = "1.38.1"
version = "1.39.0"
description = "Full-featured Telegram client library for Python 3"
optional = false
python-versions = ">=3.5"
groups = ["main"]
files = [
{file = "Telethon-1.38.1-py3-none-any.whl", hash = "sha256:30c187017501bfb982b8af5659f864dda4108f77ea49cfce61e8f6fdb8a18d6e"},
{file = "Telethon-1.38.1.tar.gz", hash = "sha256:f9866c1e37197a0894e0c02aa56a6359bffb14a585e88e18e3e819df4fda399a"},
{file = "Telethon-1.39.0-py3-none-any.whl", hash = "sha256:aa9f394b94be144799a6f6a93ab463867bc7c63503ede9631751940a98f6c703"},
{file = "telethon-1.39.0.tar.gz", hash = "sha256:35d4795d8c91deac515fb0bcb3723866b924de1c724e1d5c230460e96f284a63"},
]
[package.dependencies]
@@ -2630,14 +2699,14 @@ telegram = ["requests"]
[[package]]
name = "trio"
version = "0.28.0"
version = "0.29.0"
description = "A friendly Python library for async concurrency and I/O"
optional = false
python-versions = ">=3.9"
groups = ["main"]
files = [
{file = "trio-0.28.0-py3-none-any.whl", hash = "sha256:56d58977acc1635735a96581ec70513cc781b8b6decd299c487d3be2a721cd94"},
{file = "trio-0.28.0.tar.gz", hash = "sha256:4e547896fe9e8a5658e54e4c7c5fa1db748cbbbaa7c965e7d40505b928c73c05"},
{file = "trio-0.29.0-py3-none-any.whl", hash = "sha256:d8c463f1a9cc776ff63e331aba44c125f423a5a13c684307e828d930e625ba66"},
{file = "trio-0.29.0.tar.gz", hash = "sha256:ea0d3967159fc130acb6939a0be0e558e364fee26b5deeecc893a6b08c361bdf"},
]
[package.dependencies]
@@ -2651,18 +2720,19 @@ sortedcontainers = "*"
[[package]]
name = "trio-websocket"
version = "0.11.1"
version = "0.12.2"
description = "WebSocket library for Trio"
optional = false
python-versions = ">=3.7"
python-versions = ">=3.8"
groups = ["main"]
files = [
{file = "trio-websocket-0.11.1.tar.gz", hash = "sha256:18c11793647703c158b1f6e62de638acada927344d534e3c7628eedcb746839f"},
{file = "trio_websocket-0.11.1-py3-none-any.whl", hash = "sha256:520d046b0d030cf970b8b2b2e00c4c2245b3807853ecd44214acd33d74581638"},
{file = "trio_websocket-0.12.2-py3-none-any.whl", hash = "sha256:df605665f1db533f4a386c94525870851096a223adcb97f72a07e8b4beba45b6"},
{file = "trio_websocket-0.12.2.tar.gz", hash = "sha256:22c72c436f3d1e264d0910a3951934798dcc5b00ae56fc4ee079d46c7cf20fae"},
]
[package.dependencies]
exceptiongroup = {version = "*", markers = "python_version < \"3.11\""}
outcome = ">=1.2.0"
trio = ">=0.11"
wsproto = ">=0.14"
@@ -2729,14 +2799,14 @@ files = [
[[package]]
name = "tzlocal"
version = "5.2"
version = "5.3.1"
description = "tzinfo object for the local timezone"
optional = false
python-versions = ">=3.8"
python-versions = ">=3.9"
groups = ["main"]
files = [
{file = "tzlocal-5.2-py3-none-any.whl", hash = "sha256:49816ef2fe65ea8ac19d19aa7a1ae0551c834303d5014c6d5a62e4cbda8047b8"},
{file = "tzlocal-5.2.tar.gz", hash = "sha256:8d399205578f1a9342816409cc1e46a93ebd5755e39ea2d85334bea911bf0e6e"},
{file = "tzlocal-5.3.1-py3-none-any.whl", hash = "sha256:eb1a66c3ef5847adf7a834f1be0800581b683b5608e74f86ecbcef8ab91bb85d"},
{file = "tzlocal-5.3.1.tar.gz", hash = "sha256:cceffc7edecefea1f595541dbd6e990cb1ea3d19bf01b2809f362a03dd7921fd"},
]
[package.dependencies]
@@ -2745,6 +2815,21 @@ tzdata = {version = "*", markers = "platform_system == \"Windows\""}
[package.extras]
devenv = ["check-manifest", "pytest (>=4.3)", "pytest-cov", "pytest-mock (>=3.3)", "zest.releaser"]
[[package]]
name = "uc-micro-py"
version = "1.0.3"
description = "Micro subset of unicode data files for linkify-it-py projects."
optional = false
python-versions = ">=3.7"
groups = ["docs"]
files = [
{file = "uc-micro-py-1.0.3.tar.gz", hash = "sha256:d321b92cff673ec58027c04015fcaa8bb1e005478643ff4a500882eaab88c48a"},
{file = "uc_micro_py-1.0.3-py3-none-any.whl", hash = "sha256:db1dffff340817673d7b466ec86114a9dc0e9d4d9b5ba229d9d60e5c12600cd5"},
]
[package.extras]
test = ["coverage", "pytest", "pytest-cov"]
[[package]]
name = "uritemplate"
version = "4.1.1"
@@ -2967,81 +3052,81 @@ test = ["websockets"]
[[package]]
name = "websockets"
version = "14.2"
version = "15.0.1"
description = "An implementation of the WebSocket Protocol (RFC 6455 & 7692)"
optional = false
python-versions = ">=3.9"
groups = ["main", "docs"]
files = [
{file = "websockets-14.2-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:e8179f95323b9ab1c11723e5d91a89403903f7b001828161b480a7810b334885"},
{file = "websockets-14.2-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:0d8c3e2cdb38f31d8bd7d9d28908005f6fa9def3324edb9bf336d7e4266fd397"},
{file = "websockets-14.2-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:714a9b682deb4339d39ffa674f7b674230227d981a37d5d174a4a83e3978a610"},
{file = "websockets-14.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f2e53c72052f2596fb792a7acd9704cbc549bf70fcde8a99e899311455974ca3"},
{file = "websockets-14.2-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:e3fbd68850c837e57373d95c8fe352203a512b6e49eaae4c2f4088ef8cf21980"},
{file = "websockets-14.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4b27ece32f63150c268593d5fdb82819584831a83a3f5809b7521df0685cd5d8"},
{file = "websockets-14.2-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:4daa0faea5424d8713142b33825fff03c736f781690d90652d2c8b053345b0e7"},
{file = "websockets-14.2-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:bc63cee8596a6ec84d9753fd0fcfa0452ee12f317afe4beae6b157f0070c6c7f"},
{file = "websockets-14.2-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:7a570862c325af2111343cc9b0257b7119b904823c675b22d4ac547163088d0d"},
{file = "websockets-14.2-cp310-cp310-win32.whl", hash = "sha256:75862126b3d2d505e895893e3deac0a9339ce750bd27b4ba515f008b5acf832d"},
{file = "websockets-14.2-cp310-cp310-win_amd64.whl", hash = "sha256:cc45afb9c9b2dc0852d5c8b5321759cf825f82a31bfaf506b65bf4668c96f8b2"},
{file = "websockets-14.2-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:3bdc8c692c866ce5fefcaf07d2b55c91d6922ac397e031ef9b774e5b9ea42166"},
{file = "websockets-14.2-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:c93215fac5dadc63e51bcc6dceca72e72267c11def401d6668622b47675b097f"},
{file = "websockets-14.2-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:1c9b6535c0e2cf8a6bf938064fb754aaceb1e6a4a51a80d884cd5db569886910"},
{file = "websockets-14.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0a52a6d7cf6938e04e9dceb949d35fbdf58ac14deea26e685ab6368e73744e4c"},
{file = "websockets-14.2-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:9f05702e93203a6ff5226e21d9b40c037761b2cfb637187c9802c10f58e40473"},
{file = "websockets-14.2-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:22441c81a6748a53bfcb98951d58d1af0661ab47a536af08920d129b4d1c3473"},
{file = "websockets-14.2-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:efd9b868d78b194790e6236d9cbc46d68aba4b75b22497eb4ab64fa640c3af56"},
{file = "websockets-14.2-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:1a5a20d5843886d34ff8c57424cc65a1deda4375729cbca4cb6b3353f3ce4142"},
{file = "websockets-14.2-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:34277a29f5303d54ec6468fb525d99c99938607bc96b8d72d675dee2b9f5bf1d"},
{file = "websockets-14.2-cp311-cp311-win32.whl", hash = "sha256:02687db35dbc7d25fd541a602b5f8e451a238ffa033030b172ff86a93cb5dc2a"},
{file = "websockets-14.2-cp311-cp311-win_amd64.whl", hash = "sha256:862e9967b46c07d4dcd2532e9e8e3c2825e004ffbf91a5ef9dde519ee2effb0b"},
{file = "websockets-14.2-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:1f20522e624d7ffbdbe259c6b6a65d73c895045f76a93719aa10cd93b3de100c"},
{file = "websockets-14.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:647b573f7d3ada919fd60e64d533409a79dcf1ea21daeb4542d1d996519ca967"},
{file = "websockets-14.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:6af99a38e49f66be5a64b1e890208ad026cda49355661549c507152113049990"},
{file = "websockets-14.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:091ab63dfc8cea748cc22c1db2814eadb77ccbf82829bac6b2fbe3401d548eda"},
{file = "websockets-14.2-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:b374e8953ad477d17e4851cdc66d83fdc2db88d9e73abf755c94510ebddceb95"},
{file = "websockets-14.2-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a39d7eceeea35db85b85e1169011bb4321c32e673920ae9c1b6e0978590012a3"},
{file = "websockets-14.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:0a6f3efd47ffd0d12080594f434faf1cd2549b31e54870b8470b28cc1d3817d9"},
{file = "websockets-14.2-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:065ce275e7c4ffb42cb738dd6b20726ac26ac9ad0a2a48e33ca632351a737267"},
{file = "websockets-14.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e9d0e53530ba7b8b5e389c02282f9d2aa47581514bd6049d3a7cffe1385cf5fe"},
{file = "websockets-14.2-cp312-cp312-win32.whl", hash = "sha256:20e6dd0984d7ca3037afcb4494e48c74ffb51e8013cac71cf607fffe11df7205"},
{file = "websockets-14.2-cp312-cp312-win_amd64.whl", hash = "sha256:44bba1a956c2c9d268bdcdf234d5e5ff4c9b6dc3e300545cbe99af59dda9dcce"},
{file = "websockets-14.2-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:6f1372e511c7409a542291bce92d6c83320e02c9cf392223272287ce55bc224e"},
{file = "websockets-14.2-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:4da98b72009836179bb596a92297b1a61bb5a830c0e483a7d0766d45070a08ad"},
{file = "websockets-14.2-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:f8a86a269759026d2bde227652b87be79f8a734e582debf64c9d302faa1e9f03"},
{file = "websockets-14.2-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:86cf1aaeca909bf6815ea714d5c5736c8d6dd3a13770e885aafe062ecbd04f1f"},
{file = "websockets-14.2-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:a9b0f6c3ba3b1240f602ebb3971d45b02cc12bd1845466dd783496b3b05783a5"},
{file = "websockets-14.2-cp313-cp313-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:669c3e101c246aa85bc8534e495952e2ca208bd87994650b90a23d745902db9a"},
{file = "websockets-14.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:eabdb28b972f3729348e632ab08f2a7b616c7e53d5414c12108c29972e655b20"},
{file = "websockets-14.2-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:2066dc4cbcc19f32c12a5a0e8cc1b7ac734e5b64ac0a325ff8353451c4b15ef2"},
{file = "websockets-14.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:ab95d357cd471df61873dadf66dd05dd4709cae001dd6342edafc8dc6382f307"},
{file = "websockets-14.2-cp313-cp313-win32.whl", hash = "sha256:a9e72fb63e5f3feacdcf5b4ff53199ec8c18d66e325c34ee4c551ca748623bbc"},
{file = "websockets-14.2-cp313-cp313-win_amd64.whl", hash = "sha256:b439ea828c4ba99bb3176dc8d9b933392a2413c0f6b149fdcba48393f573377f"},
{file = "websockets-14.2-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:7cd5706caec1686c5d233bc76243ff64b1c0dc445339bd538f30547e787c11fe"},
{file = "websockets-14.2-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:ec607328ce95a2f12b595f7ae4c5d71bf502212bddcea528290b35c286932b12"},
{file = "websockets-14.2-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:da85651270c6bfb630136423037dd4975199e5d4114cae6d3066641adcc9d1c7"},
{file = "websockets-14.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c3ecadc7ce90accf39903815697917643f5b7cfb73c96702318a096c00aa71f5"},
{file = "websockets-14.2-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:1979bee04af6a78608024bad6dfcc0cc930ce819f9e10342a29a05b5320355d0"},
{file = "websockets-14.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2dddacad58e2614a24938a50b85969d56f88e620e3f897b7d80ac0d8a5800258"},
{file = "websockets-14.2-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:89a71173caaf75fa71a09a5f614f450ba3ec84ad9fca47cb2422a860676716f0"},
{file = "websockets-14.2-cp39-cp39-musllinux_1_2_i686.whl", hash = "sha256:6af6a4b26eea4fc06c6818a6b962a952441e0e39548b44773502761ded8cc1d4"},
{file = "websockets-14.2-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:80c8efa38957f20bba0117b48737993643204645e9ec45512579132508477cfc"},
{file = "websockets-14.2-cp39-cp39-win32.whl", hash = "sha256:2e20c5f517e2163d76e2729104abc42639c41cf91f7b1839295be43302713661"},
{file = "websockets-14.2-cp39-cp39-win_amd64.whl", hash = "sha256:b4c8cef610e8d7c70dea92e62b6814a8cd24fbd01d7103cc89308d2bfe1659ef"},
{file = "websockets-14.2-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:d7d9cafbccba46e768be8a8ad4635fa3eae1ffac4c6e7cb4eb276ba41297ed29"},
{file = "websockets-14.2-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:c76193c1c044bd1e9b3316dcc34b174bbf9664598791e6fb606d8d29000e070c"},
{file = "websockets-14.2-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:fd475a974d5352390baf865309fe37dec6831aafc3014ffac1eea99e84e83fc2"},
{file = "websockets-14.2-pp310-pypy310_pp73-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:2c6c0097a41968b2e2b54ed3424739aab0b762ca92af2379f152c1aef0187e1c"},
{file = "websockets-14.2-pp310-pypy310_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6d7ff794c8b36bc402f2e07c0b2ceb4a2424147ed4785ff03e2a7af03711d60a"},
{file = "websockets-14.2-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:dec254fcabc7bd488dab64846f588fc5b6fe0d78f641180030f8ea27b76d72c3"},
{file = "websockets-14.2-pp39-pypy39_pp73-macosx_10_15_x86_64.whl", hash = "sha256:bbe03eb853e17fd5b15448328b4ec7fb2407d45fb0245036d06a3af251f8e48f"},
{file = "websockets-14.2-pp39-pypy39_pp73-macosx_11_0_arm64.whl", hash = "sha256:a3c4aa3428b904d5404a0ed85f3644d37e2cb25996b7f096d77caeb0e96a3b42"},
{file = "websockets-14.2-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:577a4cebf1ceaf0b65ffc42c54856214165fb8ceeba3935852fc33f6b0c55e7f"},
{file = "websockets-14.2-pp39-pypy39_pp73-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ad1c1d02357b7665e700eca43a31d52814ad9ad9b89b58118bdabc365454b574"},
{file = "websockets-14.2-pp39-pypy39_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f390024a47d904613577df83ba700bd189eedc09c57af0a904e5c39624621270"},
{file = "websockets-14.2-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:3c1426c021c38cf92b453cdf371228d3430acd775edee6bac5a4d577efc72365"},
{file = "websockets-14.2-py3-none-any.whl", hash = "sha256:7a6ceec4ea84469f15cf15807a747e9efe57e369c384fa86e022b3bea679b79b"},
{file = "websockets-14.2.tar.gz", hash = "sha256:5059ed9c54945efb321f097084b4c7e52c246f2c869815876a69d1efc4ad6eb5"},
{file = "websockets-15.0.1-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:d63efaa0cd96cf0c5fe4d581521d9fa87744540d4bc999ae6e08595a1014b45b"},
{file = "websockets-15.0.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:ac60e3b188ec7574cb761b08d50fcedf9d77f1530352db4eef1707fe9dee7205"},
{file = "websockets-15.0.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:5756779642579d902eed757b21b0164cd6fe338506a8083eb58af5c372e39d9a"},
{file = "websockets-15.0.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0fdfe3e2a29e4db3659dbd5bbf04560cea53dd9610273917799f1cde46aa725e"},
{file = "websockets-15.0.1-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:4c2529b320eb9e35af0fa3016c187dffb84a3ecc572bcee7c3ce302bfeba52bf"},
{file = "websockets-15.0.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ac1e5c9054fe23226fb11e05a6e630837f074174c4c2f0fe442996112a6de4fb"},
{file = "websockets-15.0.1-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:5df592cd503496351d6dc14f7cdad49f268d8e618f80dce0cd5a36b93c3fc08d"},
{file = "websockets-15.0.1-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:0a34631031a8f05657e8e90903e656959234f3a04552259458aac0b0f9ae6fd9"},
{file = "websockets-15.0.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:3d00075aa65772e7ce9e990cab3ff1de702aa09be3940d1dc88d5abf1ab8a09c"},
{file = "websockets-15.0.1-cp310-cp310-win32.whl", hash = "sha256:1234d4ef35db82f5446dca8e35a7da7964d02c127b095e172e54397fb6a6c256"},
{file = "websockets-15.0.1-cp310-cp310-win_amd64.whl", hash = "sha256:39c1fec2c11dc8d89bba6b2bf1556af381611a173ac2b511cf7231622058af41"},
{file = "websockets-15.0.1-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:823c248b690b2fd9303ba00c4f66cd5e2d8c3ba4aa968b2779be9532a4dad431"},
{file = "websockets-15.0.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:678999709e68425ae2593acf2e3ebcbcf2e69885a5ee78f9eb80e6e371f1bf57"},
{file = "websockets-15.0.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:d50fd1ee42388dcfb2b3676132c78116490976f1300da28eb629272d5d93e905"},
{file = "websockets-15.0.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d99e5546bf73dbad5bf3547174cd6cb8ba7273062a23808ffea025ecb1cf8562"},
{file = "websockets-15.0.1-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:66dd88c918e3287efc22409d426c8f729688d89a0c587c88971a0faa2c2f3792"},
{file = "websockets-15.0.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8dd8327c795b3e3f219760fa603dcae1dcc148172290a8ab15158cf85a953413"},
{file = "websockets-15.0.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:8fdc51055e6ff4adeb88d58a11042ec9a5eae317a0a53d12c062c8a8865909e8"},
{file = "websockets-15.0.1-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:693f0192126df6c2327cce3baa7c06f2a117575e32ab2308f7f8216c29d9e2e3"},
{file = "websockets-15.0.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:54479983bd5fb469c38f2f5c7e3a24f9a4e70594cd68cd1fa6b9340dadaff7cf"},
{file = "websockets-15.0.1-cp311-cp311-win32.whl", hash = "sha256:16b6c1b3e57799b9d38427dda63edcbe4926352c47cf88588c0be4ace18dac85"},
{file = "websockets-15.0.1-cp311-cp311-win_amd64.whl", hash = "sha256:27ccee0071a0e75d22cb35849b1db43f2ecd3e161041ac1ee9d2352ddf72f065"},
{file = "websockets-15.0.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:3e90baa811a5d73f3ca0bcbf32064d663ed81318ab225ee4f427ad4e26e5aff3"},
{file = "websockets-15.0.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:592f1a9fe869c778694f0aa806ba0374e97648ab57936f092fd9d87f8bc03665"},
{file = "websockets-15.0.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:0701bc3cfcb9164d04a14b149fd74be7347a530ad3bbf15ab2c678a2cd3dd9a2"},
{file = "websockets-15.0.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e8b56bdcdb4505c8078cb6c7157d9811a85790f2f2b3632c7d1462ab5783d215"},
{file = "websockets-15.0.1-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:0af68c55afbd5f07986df82831c7bff04846928ea8d1fd7f30052638788bc9b5"},
{file = "websockets-15.0.1-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:64dee438fed052b52e4f98f76c5790513235efaa1ef7f3f2192c392cd7c91b65"},
{file = "websockets-15.0.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:d5f6b181bb38171a8ad1d6aa58a67a6aa9d4b38d0f8c5f496b9e42561dfc62fe"},
{file = "websockets-15.0.1-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:5d54b09eba2bada6011aea5375542a157637b91029687eb4fdb2dab11059c1b4"},
{file = "websockets-15.0.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:3be571a8b5afed347da347bfcf27ba12b069d9d7f42cb8c7028b5e98bbb12597"},
{file = "websockets-15.0.1-cp312-cp312-win32.whl", hash = "sha256:c338ffa0520bdb12fbc527265235639fb76e7bc7faafbb93f6ba80d9c06578a9"},
{file = "websockets-15.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:fcd5cf9e305d7b8338754470cf69cf81f420459dbae8a3b40cee57417f4614a7"},
{file = "websockets-15.0.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:ee443ef070bb3b6ed74514f5efaa37a252af57c90eb33b956d35c8e9c10a1931"},
{file = "websockets-15.0.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:5a939de6b7b4e18ca683218320fc67ea886038265fd1ed30173f5ce3f8e85675"},
{file = "websockets-15.0.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:746ee8dba912cd6fc889a8147168991d50ed70447bf18bcda7039f7d2e3d9151"},
{file = "websockets-15.0.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:595b6c3969023ecf9041b2936ac3827e4623bfa3ccf007575f04c5a6aa318c22"},
{file = "websockets-15.0.1-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:3c714d2fc58b5ca3e285461a4cc0c9a66bd0e24c5da9911e30158286c9b5be7f"},
{file = "websockets-15.0.1-cp313-cp313-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0f3c1e2ab208db911594ae5b4f79addeb3501604a165019dd221c0bdcabe4db8"},
{file = "websockets-15.0.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:229cf1d3ca6c1804400b0a9790dc66528e08a6a1feec0d5040e8b9eb14422375"},
{file = "websockets-15.0.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:756c56e867a90fb00177d530dca4b097dd753cde348448a1012ed6c5131f8b7d"},
{file = "websockets-15.0.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:558d023b3df0bffe50a04e710bc87742de35060580a293c2a984299ed83bc4e4"},
{file = "websockets-15.0.1-cp313-cp313-win32.whl", hash = "sha256:ba9e56e8ceeeedb2e080147ba85ffcd5cd0711b89576b83784d8605a7df455fa"},
{file = "websockets-15.0.1-cp313-cp313-win_amd64.whl", hash = "sha256:e09473f095a819042ecb2ab9465aee615bd9c2028e4ef7d933600a8401c79561"},
{file = "websockets-15.0.1-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:5f4c04ead5aed67c8a1a20491d54cdfba5884507a48dd798ecaf13c74c4489f5"},
{file = "websockets-15.0.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:abdc0c6c8c648b4805c5eacd131910d2a7f6455dfd3becab248ef108e89ab16a"},
{file = "websockets-15.0.1-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:a625e06551975f4b7ea7102bc43895b90742746797e2e14b70ed61c43a90f09b"},
{file = "websockets-15.0.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d591f8de75824cbb7acad4e05d2d710484f15f29d4a915092675ad3456f11770"},
{file = "websockets-15.0.1-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:47819cea040f31d670cc8d324bb6435c6f133b8c7a19ec3d61634e62f8d8f9eb"},
{file = "websockets-15.0.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ac017dd64572e5c3bd01939121e4d16cf30e5d7e110a119399cf3133b63ad054"},
{file = "websockets-15.0.1-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:4a9fac8e469d04ce6c25bb2610dc535235bd4aa14996b4e6dbebf5e007eba5ee"},
{file = "websockets-15.0.1-cp39-cp39-musllinux_1_2_i686.whl", hash = "sha256:363c6f671b761efcb30608d24925a382497c12c506b51661883c3e22337265ed"},
{file = "websockets-15.0.1-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:2034693ad3097d5355bfdacfffcbd3ef5694f9718ab7f29c29689a9eae841880"},
{file = "websockets-15.0.1-cp39-cp39-win32.whl", hash = "sha256:3b1ac0d3e594bf121308112697cf4b32be538fb1444468fb0a6ae4feebc83411"},
{file = "websockets-15.0.1-cp39-cp39-win_amd64.whl", hash = "sha256:b7643a03db5c95c799b89b31c036d5f27eeb4d259c798e878d6937d71832b1e4"},
{file = "websockets-15.0.1-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:0c9e74d766f2818bb95f84c25be4dea09841ac0f734d1966f415e4edfc4ef1c3"},
{file = "websockets-15.0.1-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:1009ee0c7739c08a0cd59de430d6de452a55e42d6b522de7aa15e6f67db0b8e1"},
{file = "websockets-15.0.1-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:76d1f20b1c7a2fa82367e04982e708723ba0e7b8d43aa643d3dcd404d74f1475"},
{file = "websockets-15.0.1-pp310-pypy310_pp73-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:f29d80eb9a9263b8d109135351caf568cc3f80b9928bccde535c235de55c22d9"},
{file = "websockets-15.0.1-pp310-pypy310_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b359ed09954d7c18bbc1680f380c7301f92c60bf924171629c5db97febb12f04"},
{file = "websockets-15.0.1-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:cad21560da69f4ce7658ca2cb83138fb4cf695a2ba3e475e0559e05991aa8122"},
{file = "websockets-15.0.1-pp39-pypy39_pp73-macosx_10_15_x86_64.whl", hash = "sha256:7f493881579c90fc262d9cdbaa05a6b54b3811c2f300766748db79f098db9940"},
{file = "websockets-15.0.1-pp39-pypy39_pp73-macosx_11_0_arm64.whl", hash = "sha256:47b099e1f4fbc95b701b6e85768e1fcdaf1630f3cbe4765fa216596f12310e2e"},
{file = "websockets-15.0.1-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:67f2b6de947f8c757db2db9c71527933ad0019737ec374a8a6be9a956786aaf9"},
{file = "websockets-15.0.1-pp39-pypy39_pp73-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:d08eb4c2b7d6c41da6ca0600c077e93f5adcfd979cd777d747e9ee624556da4b"},
{file = "websockets-15.0.1-pp39-pypy39_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4b826973a4a2ae47ba357e4e82fa44a463b8f168e1ca775ac64521442b19e87f"},
{file = "websockets-15.0.1-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:21c1fa28a6a7e3cbdc171c694398b6df4744613ce9b36b1a498e816787e28123"},
{file = "websockets-15.0.1-py3-none-any.whl", hash = "sha256:f7a866fbc1e97b5c617ee4116daaa09b722101d4a3c170c787450ba409f9736f"},
{file = "websockets-15.0.1.tar.gz", hash = "sha256:82544de02076bafba038ce055ee6412d68da13ab47f0c60cab827346de828dee"},
]
[[package]]
@@ -3077,14 +3162,14 @@ h11 = ">=0.9.0,<1"
[[package]]
name = "yt-dlp"
version = "2025.1.26"
version = "2025.2.19"
description = "A feature-rich command-line audio/video downloader"
optional = false
python-versions = ">=3.9"
groups = ["main"]
files = [
{file = "yt_dlp-2025.1.26-py3-none-any.whl", hash = "sha256:3e76bd896b9f96601021ca192ca0fbdd195e3c3dcc28302a3a34c9bc4979da7b"},
{file = "yt_dlp-2025.1.26.tar.gz", hash = "sha256:1c9738266921ad43c568ad01ac3362fb7c7af549276fbec92bd72f140da16240"},
{file = "yt_dlp-2025.2.19-py3-none-any.whl", hash = "sha256:3ed218eaeece55e9d715afd41abc450dc406ee63bf79355169dfde312d38fdb8"},
{file = "yt_dlp-2025.2.19.tar.gz", hash = "sha256:f33ca76df2e4db31880f2fe408d44f5058d9f135015b13e50610dfbe78245bea"},
]
[package.extras]
@@ -3100,4 +3185,4 @@ test = ["pytest (>=8.1,<9.0)", "pytest-rerunfailures (>=14.0,<15.0)"]
[metadata]
lock-version = "2.1"
python-versions = ">=3.10,<3.13"
content-hash = "9ca114395e73af8982abbccc25b385bbca62e50ba7cca8239e52e5c1227cb4b0"
content-hash = "2d0a953383901fe12e97f6f56a76a9d8008788695425792eedbf739a18585188"

7
pyproject.toml
View File

@@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api"
[project]
name = "auto-archiver"
version = "0.13.0"
version = "0.13.5"
description = "Automatically archive links to videos, images, and social media content from Google Sheets (and more)."
requires-python = ">=3.10,<3.13"
@@ -22,7 +22,6 @@ classifiers = [
]
dependencies = [
"oscrypto @ git+https://github.com/wbond/oscrypto.git@d5f3437ed24257895ae1edd9e503cfb352e635a8",
"gspread (>=0.0.0)",
"beautifulsoup4 (>=0.0.0)",
"bs4 (>=0.0.0)",
@@ -64,6 +63,7 @@ dependencies = [
pytest = "^8.3.4"
autopep8 = "^2.3.1"
pytest-loguru = "^0.4.0"
pytest-mock = "^3.14.0"
[tool.poetry.group.docs.dependencies]
sphinx = "^8.1.3"
@@ -72,7 +72,8 @@ sphinxcontrib-mermaid = "^1.0.0"
sphinx-autobuild = "^2024.10.3"
sphinx-copybutton = "^0.5.2"
myst-parser = "^4.0.0"
furo = "^2024.8.6"
sphinx-book-theme = "^1.1.3"
linkify-it-py = "^2.0.3"
[project.scripts]

52
scripts/generate_settings_schema.py Normal file
View File

@@ -0,0 +1,52 @@
import json
import os
import io
from ruamel.yaml import YAML
from auto_archiver.core.module import ModuleFactory
from auto_archiver.core.consts import MODULE_TYPES
from auto_archiver.core.config import EMPTY_CONFIG
class SchemaEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, set):
return list(obj)
return json.JSONEncoder.default(self, obj)
# Get available modules
module_factory = ModuleFactory()
available_modules = module_factory.available_modules()
modules_by_type = {}
# Categorize modules by type
for module in available_modules:
for type in module.manifest.get('type', []):
modules_by_type.setdefault(type, []).append(module)
all_modules_ordered_by_type = sorted(available_modules, key=lambda x: (MODULE_TYPES.index(x.type[0]), not x.requires_setup))
yaml: YAML = YAML()
config_string = io.BytesIO()
yaml.dump(EMPTY_CONFIG, config_string)
config_string = config_string.getvalue().decode('utf-8')
output_schema = {
'modules': dict((module.name,
{
'name': module.name,
'display_name': module.display_name,
'manifest': module.manifest,
'configs': module.configs or None
}
) for module in all_modules_ordered_by_type),
'steps': dict((f"{module_type}s", [module.name for module in modules_by_type[module_type]]) for module_type in MODULE_TYPES),
'configs': [m.name for m in all_modules_ordered_by_type if m.configs],
'module_types': MODULE_TYPES,
'empty_config': config_string
}
current_file_dir = os.path.dirname(os.path.abspath(__file__))
output_file = os.path.join(current_file_dir, 'settings/src/schema.json')
with open(output_file, 'w') as file:
json.dump(output_schema, file, indent=4, cls=SchemaEncoder)

24
scripts/settings/.gitignore vendored Normal file
View File

@@ -0,0 +1,24 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*
node_modules
dist
dist-ssr
*.local
# Editor directories and files
.vscode/*
!.vscode/extensions.json
.idea
.DS_Store
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?

3
scripts/settings/index.html Normal file
View File

@@ -0,0 +1,3 @@
<div id="root"></div>
<script type="module" src="/src/main.tsx"></script>

3743
scripts/settings/package-lock.json generated Normal file
View File

File diff suppressed because it is too large Load Diff

31
scripts/settings/package.json Normal file
View File

@@ -0,0 +1,31 @@
{
"name": "material-ui-vite-ts",
"private": true,
"version": "5.0.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "vite build",
"preview": "vite preview"
},
"dependencies": {
"@dnd-kit/core": "^6.3.1",
"@dnd-kit/sortable": "^10.0.0",
"@emotion/react": "latest",
"@emotion/styled": "latest",
"@mui/icons-material": "latest",
"@mui/material": "latest",
"react": "19.0.0",
"react-dom": "19.0.0",
"react-markdown": "^10.0.0",
"yaml": "^2.7.0"
},
"devDependencies": {
"@types/react": "latest",
"@types/react-dom": "latest",
"@vitejs/plugin-react": "latest",
"typescript": "latest",
"vite": "latest",
"vite-plugin-singlefile": "^2.1.0"
}
}

450
scripts/settings/src/App.tsx Normal file
View File

@@ -0,0 +1,450 @@
import * as React from 'react';
import { useEffect, useState, useRef } from 'react';
import Container from '@mui/material/Container';
import Typography from '@mui/material/Typography';
import Box from '@mui/material/Box';
import FileUploadIcon from '@mui/icons-material/FileUpload';
//
import {
DndContext,
closestCenter,
KeyboardSensor,
PointerSensor,
useSensor,
useSensors,
DragOverlay
} from "@dnd-kit/core";
import {
arrayMove,
SortableContext,
sortableKeyboardCoordinates,
rectSortingStrategy
} from "@dnd-kit/sortable";
import type { DragStartEvent, DragEndEvent, UniqueIdentifier } from "@dnd-kit/core";
import { Module } from './types';
import { modules, steps, module_types, empty_config } from './schema.json';
import {
Stack,
Button,
} from '@mui/material';
import Grid from '@mui/material/Grid2';
import { parseDocument, Document, YAMLSeq, YAMLMap, Scalar } from 'yaml'
import StepCard from './StepCard';
function FileDrop({ setYamlFile }: { setYamlFile: React.Dispatch<React.SetStateAction<Document>> }) {
const [showError, setShowError] = useState(false);
const [label, setLabel] = useState(<>Drag and drop your orchestration.yaml file here, or click to select a file.</>);
const wrapperRef = useRef(null);
function openYAMLFile(event: any) {
let file = event.target.files[0];
if (file.type.indexOf('yaml') === -1) {
setShowError(true);
setLabel(<>Invalid type, only YAML files are accepted.</>)
return;
}
let reader = new FileReader();
reader.onload = function (e) {
let contents = e.target ? e.target.result : '';
try {
let document = parseDocument(contents as string);
if (document.errors.length > 0) {
// not a valid yaml file
setShowError(true);
setLabel(<>Invalid file. Make sure your Orchestration is a valid YAML file with a 'steps' section in it.</>)
return;
} else {
setShowError(false);
setLabel(<>File loaded successfully.</>)
}
// do some basic validation of 'steps'
let steps = document.get('steps');
if (!steps) {
setShowError(true);
setLabel(<>Invalid file. Your orchestration file must have a 'steps' section in it.</>)
return;
}
const replacements = {
feeder: 'feeders',
formatter: 'formatters',
archivers: 'extractors',
};
let error = false;
for (let stepType of Object.keys(replacements)) {
if (steps.get(stepType) !== undefined) {
setShowError(true);
setLabel(<>Invalid file. Your orchestration file appears to be in the old (v0.12) format with a '{stepType}' section.<br/>You should manually update your orchestration file first (hint: {stepType} → {replacements[stepType]})</>);
error = true;
return;
}
};
setYamlFile(document);
} catch (e) {
console.error(e);
}
}
reader.readAsText(file);
}
return (
<>
<div
style={{
position: 'relative',
width: '100%',
border: 'dashed',
borderRadius:'5px',
textAlign: 'center',
borderWidth: '1px',
padding: '20px' }}
onDragEnter={(e) => {
e.currentTarget.style.backgroundColor = 'var(--mui-palette-LinearProgress-infoBg)';
}}
onDragLeave={(e) => {
e.currentTarget.style.backgroundColor = '';
}}
onDrop={(e) => {
e.currentTarget.style.backgroundColor = '';
}}
>
<FileUploadIcon style={{ fontSize: 50 }} />
<input style={{
opacity: 0,
position: 'absolute',
top: 0,
left: 0,
width: '100%',
height: '100%',
cursor: 'pointer',
}}
type="file" id="file"
accept=".yaml"
onChange={openYAMLFile} />
<Typography variant="body1" color={showError ? 'error' : ''} >
{label}
</Typography>
</div>
</>
);
}
function ModuleTypes({ stepType, setEnabledModules, enabledModules, configValues }: { stepType: string, setEnabledModules: any, enabledModules: any, configValues: any }) {
const [showError, setShowError] = useState<boolean>(false);
const [activeId, setActiveId] = useState<UniqueIdentifier>();
const [items, setItems] = useState<string[]>([]);
useEffect(() => {
setItems(enabledModules[stepType].map(([name, enabled]: [string, boolean]) => name));
}
, [enabledModules]);
const toggleModule = (event: any) => {
// make sure that 'feeder' and 'formatter' types only have one value
let name = event.target.id;
let checked = event.target.checked;
if (stepType === 'feeders' || stepType === 'formatters') {
// check how many modules of this type are enabled
const checkedModules = enabledModules[stepType].filter(([m, enabled]: [string, boolean]) => {
return (m !== name && enabled) || (checked && m === name)
});
if (checkedModules.length > 1) {
setShowError(true);
} else {
setShowError(false);
}
} else {
setShowError(false);
}
let newEnabledModules = { ...enabledModules };
newEnabledModules[stepType] = enabledModules[stepType].map(([m, enabled]: [string, boolean]) => {
return (m === name) ? [m, checked] : [m, enabled];
});
setEnabledModules(newEnabledModules);
}
const sensors = useSensors(
useSensor(PointerSensor),
useSensor(KeyboardSensor, {
coordinateGetter: sortableKeyboardCoordinates
})
);
const handleDragStart = (event: DragStartEvent) => {
setActiveId(event.active.id);
};
const handleDragEnd = (event: DragEndEvent) => {
setActiveId(undefined);
const { active, over } = event;
if (active.id !== over?.id) {
const oldIndex = items.indexOf(active.id as string);
const newIndex = items.indexOf(over?.id as string);
let newArray = arrayMove(items, oldIndex, newIndex);
// set it also on steps
let newEnabledModules = { ...enabledModules };
newEnabledModules[stepType] = enabledModules[stepType].sort((a, b) => {
return newArray.indexOf(a[0]) - newArray.indexOf(b[0]);
})
setEnabledModules(newEnabledModules);
}
};
return (
<>
<Box sx={{ my: 4 }}>
<Typography id={stepType} variant="h6" style={{ textTransform: 'capitalize' }} >
{stepType}
</Typography>
<Typography variant="body1" >
Select the <a href="<a href={`https://auto-archiver.readthedocs.io/en/latest/modules/${stepType.slice(0,-1)}.html`}" target="_blank">{stepType}</a> you wish to enable. Drag to reorder.
</Typography>
</Box>
{showError ? <Typography variant="body1" color="error" >Only one {stepType.slice(0,-1)} can be enabled at a time.</Typography> : null}
<DndContext
sensors={sensors}
collisionDetection={closestCenter}
onDragEnd={handleDragEnd}
onDragStart={handleDragStart}
>
<Grid container spacing={1} key={stepType}>
<SortableContext items={items} strategy={rectSortingStrategy}>
{items.map((name: string) => {
let m: Module = modules[name];
return (
<StepCard key={name} type={stepType} module={m} toggleModule={toggleModule} enabledModules={enabledModules} configValues={configValues} />
);
})}
<DragOverlay>
{activeId ? (
<div
style={{
width: "100%",
height: "100%",
backgroundColor: "grey",
opacity: 0.1,
}}
></div>
) : null}
</DragOverlay>
</SortableContext>
</Grid>
</DndContext>
</>
);
}
export default function App() {
const [yamlFile, setYamlFile] = useState<Document>(new Document());
const [enabledModules, setEnabledModules] = useState<{}>(Object.fromEntries(Object.keys(steps).map(type => [type, steps[type].map((name: string) => [name, false])])));
const [configValues, setConfigValues] = useState<{
[key: string]: {
[key: string
]: any
}
}>(
Object.keys(modules).reduce((acc, module) => {
acc[module] = {};
return acc;
}, {})
);
const saveSettings = function (copy: boolean = false) {
// edit the yamlFile
// generate the steps config
let stepsConfig = enabledModules;
let finalYamlFile: Document = null;
if (!yamlFile || yamlFile.contents == null) {
// create the yaml file from
finalYamlFile = parseDocument(empty_config as string);
} else {
finalYamlFile = yamlFile;
}
// set the steps
module_types.forEach((type: string) => {
let stepType = type + 's';
let existingSteps = finalYamlFile.getIn(['steps', stepType]) as YAMLSeq;
stepsConfig[stepType].forEach(([name, enabled]: [string, boolean]) => {
let index = existingSteps.items.findIndex((item) => {
return (item.value || item) === name
});
let stepItem = finalYamlFile.getIn(['steps', stepType], true) as YAMLSeq;
if (enabled && index === -1) {
finalYamlFile.addIn(['steps', stepType], name);
stepItem.commentBefore = stepItem.commentBefore?.replace("\n - " + name, '');
stepItem.comment = stepItem.comment?.replace("\n - " + name, '');
} else if (!enabled && index !== -1) {
// set the value to empty and add a comment before with the commented value
finalYamlFile.deleteIn(['steps', stepType, index]);
stepItem.commentBefore += "\n - " + name;
finalYamlFile.setIn(['steps', stepType], stepItem);
}
});
// sort the items
existingSteps.items.sort((a: Scalar | string, b: Scalar | string) => {
return (stepsConfig[stepType].findIndex((val: [string, boolean]) => {return val[0] === (a.value || a)}) -
stepsConfig[stepType].findIndex((val: [string, boolean]) => {return val[0] === (b.value || b)}))
});
existingSteps.flow = existingSteps.items.length ? false : true;
});
// set all other settings
// loop through each item that isn't 'steps' in the finalYamlFile and check if it exists in configValues
Object.keys(configValues).forEach((module_name: string) => {
// get an existing key
let existingConfig = finalYamlFile.get(module_name, true) as YAMLMap;
if (existingConfig) {
Object.keys(configValues[module_name]).forEach((config_name: string) => {
let existingConfigYAML = existingConfig.get(config_name, true) as Scalar;
if (existingConfigYAML) {
existingConfigYAML.value = configValues[module_name][config_name];
existingConfig.set(config_name, existingConfigYAML);
} else {
existingConfig.set(config_name, configValues[module_name][config_name]);
}
});
finalYamlFile.set(module_name, existingConfig);
} else {
if (configValues[module_name] && Object.keys(configValues[module_name]).length > 0) {
finalYamlFile.set(module_name, configValues[module_name]);
}
}
});
if (copy) {
navigator.clipboard.writeText(String(finalYamlFile)).then(() => {
alert("Settings copied to clipboard.");
});
} else {
// offer the file for download
const blob = new Blob([String(finalYamlFile)], { type: 'application/x-yaml' });
const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url;
a.download = 'orchestration.yaml';
a.click();
}
}
useEffect(() => {
// load the configs, and set the default values if they exist
let newConfigValues = {};
Object.keys(modules).map((module: string) => {
let m = modules[module];
let configs = m.configs;
if (!configs) {
return;
}
newConfigValues[module] = {};
Object.keys(configs).map((config: string) => {
let config_args = configs[config];
if (config_args.default !== undefined) {
newConfigValues[module][config] = config_args.default;
}
});
})
setConfigValues(newConfigValues);
}, []);
useEffect(() => {
if (!yamlFile || yamlFile.contents == null) {
return;
}
let settings = yamlFile.toJS();
// make a deep copy of settings
let stepSettings = settings['steps'];
let newEnabledModules = Object.fromEntries(Object.keys(steps).map((type: string) => {
return [type, steps[type].map((name: string) => {
return [name, stepSettings[type].indexOf(name) !== -1];
}).sort((a, b) => {
let aIndex = stepSettings[type].indexOf(a[0]);
let bIndex = stepSettings[type].indexOf(b[0]);
if (aIndex === -1 && bIndex === -1) {
return a - b;
}
if (bIndex === -1) {
return -1;
}
if (aIndex === -1) {
return 1;
}
return aIndex - bIndex;
})];
}).sort((a, b) => {
return module_types.indexOf(a[0]) - module_types.indexOf(b[0]);
}));
setEnabledModules(newEnabledModules);
// set the config values
let newConfigValues = settings;
delete newConfigValues['steps'];
setConfigValues(Object.keys(modules).reduce((acc, module) => {
acc[module] = newConfigValues[module] || {};
return acc;
}, {}));
}, [yamlFile]);
return (
<Container maxWidth="lg">
<Box sx={{ my: 4 }}>
<Box sx={{ my: 4 }}>
<Typography variant="h5" >
1. Select your orchestration.yaml settings file.
</Typography>
<Typography variant="body1">Or skip this step to start from scratch</Typography>
<FileDrop setYamlFile={setYamlFile} />
</Box>
<Box sx={{ my: 4 }}>
<Typography variant="h5" >
2. Choose the Modules you wish to enable/disable
</Typography>
{Object.keys(steps).map((stepType: string) => {
return (
<Box key={stepType} sx={{ my: 4 }}>
<ModuleTypes stepType={stepType} setEnabledModules={setEnabledModules} enabledModules={enabledModules} configValues={configValues} />
</Box>
);
})}
</Box>
<Box sx={{ my: 4 }}>
<Typography variant="h5" >
3. Configure your Enabled Modules
</Typography>
<Typography variant="body1" >
Next to each module you've enabled, you can click 'Configure' to set the module's settings.
</Typography>
</Box>
<Box sx={{ my: 4 }}>
<Typography variant="h5" >
4. Save your settings
</Typography>
<Stack direction="row" spacing={2} sx={{ my: 2 }}>
<Button variant="contained" color="primary" onClick={() => saveSettings(true)}>Copy Settings to Clipboard</Button>
<Button variant="contained" color="primary" onClick={() => saveSettings()}>Save Settings to File</Button>
</Stack>
</Box>
</Box>
</Container>
);
}

258
scripts/settings/src/StepCard.tsx Normal file
View File

@@ -0,0 +1,258 @@
import { useState } from "react";
import { useSortable } from "@dnd-kit/sortable";
import ReactMarkdown from 'react-markdown';
import { CSS } from "@dnd-kit/utilities";
import {
Card,
CardActions,
CardHeader,
Button,
Dialog,
DialogTitle,
DialogContent,
Box,
IconButton,
Checkbox,
Select,
MenuItem,
FormControl,
FormControlLabel,
FormHelperText,
TextField,
Stack,
Typography,
InputAdornment,
} from '@mui/material';
import Grid from '@mui/material/Grid2';
import DragIndicatorIcon from '@mui/icons-material/DragIndicator';
import Visibility from '@mui/icons-material/Visibility';
import VisibilityOff from '@mui/icons-material/VisibilityOff';
import HelpIconOutlined from '@mui/icons-material/HelpOutline';
import { Module, Config } from "./types";
// adds 'capitalize' method to String prototype
declare global {
interface String {
capitalize(): string;
}
}
String.prototype.capitalize = function (this: string) {
return this.charAt(0).toUpperCase() + this.slice(1);
};
const StepCard = ({
type,
module,
toggleModule,
enabledModules,
configValues
}: {
type: string,
module: Module,
toggleModule: any,
enabledModules: any,
configValues: any
}) => {
const {
attributes,
listeners,
setNodeRef,
transform,
transition,
isDragging
} = useSortable({ id: module.name });
const style = {
...Card.style,
transform: CSS.Transform.toString(transform),
transition,
zIndex: isDragging ? "100" : "auto",
opacity: isDragging ? 0.3 : 1
};
let name = module.name;
const [helpOpen, setHelpOpen] = useState(false);
const [configOpen, setConfigOpen] = useState(false);
const enabled = enabledModules[type].find((m: any) => m[0] === name)[1];
return (
<Grid ref={setNodeRef} size={{ xs: 6, sm: 4, md: 3 }} style={style}>
<Card >
<CardHeader
title={
<FormControlLabel
style={{paddingRight: '0 !important'}}
control={<Checkbox title="Check to enable this module" sx={{paddingTop:0, paddingBottom:0}} id={name} onClick={toggleModule} checked={enabled} />}
label={module.display_name} />
}
/>
<CardActions>
<Box sx={{ justifyContent: 'space-between', display: 'flex', width: '100%' }}>
<Box>
<IconButton title="Module information" size="small" onClick={() => setHelpOpen(true)}>
<HelpIconOutlined />
</IconButton>
{enabled && module.configs && name != 'cli_feeder' ? (
<Button size="small" onClick={() => setConfigOpen(true)}>Configure</Button>
) : null}
</Box>
<IconButton size="small" title="Drag to reorder" sx={{ cursor: 'grab' }} {...listeners} {...attributes}>
<DragIndicatorIcon/>
</IconButton>
</Box>
</CardActions>
</Card>
<Dialog
open={helpOpen}
onClose={() => setHelpOpen(false)}
maxWidth="lg"
>
<DialogTitle>
{module.display_name}
</DialogTitle>
<DialogContent>
<ReactMarkdown>
{module.manifest.description.split("\n").map((line: string) => line.trim()).join("\n")}
</ReactMarkdown>
</DialogContent>
</Dialog>
{module.configs && name != 'cli_feeder' && <ConfigPanel module={module} open={configOpen} setOpen={setConfigOpen} configValues={configValues} />}
</Grid>
)
}
function ConfigField({ config_value, module, configValues }: { config_value: any, module: Module, configValues: any }) {
const [showPassword, setShowPassword] = useState(false);
const handleClickShowPassword = () => setShowPassword((show) => !show);
const handleMouseDownPassword = (event: React.MouseEvent<HTMLButtonElement>) => {
event.preventDefault();
};
const handleMouseUpPassword = (event: React.MouseEvent<HTMLButtonElement>) => {
event.preventDefault();
};
function setConfigValue(config: any, value: any) {
configValues[module.name][config] = value;
}
const config_args: Config = module.configs[config_value];
const config_name: string = config_value.replace(/_/g, " ");
const config_display_name = config_name.capitalize();
const value = configValues[module.name][config_value] || config_args.default;
const config_value_lower = config_value.toLowerCase();
const is_password = config_value_lower.includes('password') ||
config_value_lower.includes('secret') ||
config_value_lower.includes('token') ||
config_value_lower.includes('key') ||
config_value_lower.includes('api_hash') ||
config_args.type === 'password';
const text_input_type = is_password ? 'password' : (config_args.type === 'int' ? 'number' : 'text');
return (
<Box>
<Typography variant='body1' style={{ fontWeight: 'bold' }}>{config_display_name} {config_args.required && (`(required)`)} </Typography>
<FormControl size="small">
{config_args.type === 'bool' ?
<FormControlLabel control={
<Checkbox defaultChecked={value} size="small" id={`${module}.${config_value}`}
onChange={(e) => {
setConfigValue(config_value, e.target.checked);
}}
/>} label={config_args.help.capitalize()}
/>
:
(
config_args.choices !== undefined ?
<Select size="small" id={`${module}.${config_value}`}
defaultValue={config_args.default}
value={value}
onChange={(e) => {
setConfigValue(config_value, e.target.value);
}}
>
{config_args.choices.map((choice: any) => {
return (
<MenuItem key={`${module}.${config_value}.${choice}`}
value={choice}>{choice}</MenuItem>
);
})}
</Select>
:
(config_args.type === 'json_loader' ?
<TextField multiline size="small" id={`${module}.${config_value}`} defaultValue={JSON.stringify(value, null, 2)} rows={6} onChange={
(e) => {
try {
let val = JSON.parse(e.target.value);
setConfigValue(config_value, val);
} catch (e) {
console.log(e);
}
}
} />
:
<TextField size="small" id={`${module}.${config_value}`} defaultValue={value} type={showPassword ? 'text' : text_input_type}
onChange={(e) => {
setConfigValue(config_value, e.target.value);
}}
required={config_args.required}
slotProps={ is_password ? {
input: { endAdornment: (
<InputAdornment position="end">
<IconButton
aria-label="toggle password visibility"
onClick={handleClickShowPassword}
onMouseDown={handleMouseDownPassword}
onMouseUp={handleMouseUpPassword}
>
{showPassword ? <VisibilityOff /> : <Visibility />}
</IconButton>
</InputAdornment>
)}
} : {}}
/>
)
)
}
{config_args.type !== 'bool' && (
<FormHelperText >{config_args.help.capitalize()}</FormHelperText>
)}
</FormControl>
</Box>
)
}
function ConfigPanel({ module, open, setOpen, configValues }: { module: Module, open: boolean, setOpen: any, configValues: any }) {
return (
<>
<Dialog
open={open}
onClose={() => setOpen(false)}
maxWidth="lg"
>
<DialogTitle>
{module.display_name}
</DialogTitle>
<DialogContent>
<Stack direction="column" spacing={1}>
{Object.keys(module.configs).map((config_value: any) => {
return (
<ConfigField key={config_value} config_value={config_value} module={module} configValues={configValues} />
);
})}
</Stack>
</DialogContent>
</Dialog>
</>
);
}
export default StepCard;

44
scripts/settings/src/main.tsx Normal file
View File

@@ -0,0 +1,44 @@
import * as React from 'react';
import * as ReactDOM from 'react-dom/client';
import { ThemeProvider } from '@mui/material/styles';
import { CssBaseline } from '@mui/material';
import App from './App';
import { createTheme } from '@mui/material/styles';
import { red } from '@mui/material/colors';
import { useState, useEffect } from 'react';
function RootApp() {
const [mode, setMode] = useState('light');
useEffect(() => {
setMode(window.localStorage.getItem('theme') || 'light');
}, []);
var observer = new MutationObserver(function(mutations) {
setMode(window.localStorage.getItem('theme') || 'light');
})
observer.observe(document.documentElement, {attributes: true, attributeFilter: ['data-theme']});
// A custom theme for this app
const theme = createTheme({
palette: {
mode: mode == 'light' ? 'light' : 'dark',
},
cssVariables: true
});
return (
<ThemeProvider theme={theme}>
<CssBaseline />
<App />
</ThemeProvider>
);
}
ReactDOM.createRoot(document.getElementById('root')!).render(
<React.StrictMode>
<RootApp />
</React.StrictMode>,
);

2118
scripts/settings/src/schema.json Normal file
View File

File diff suppressed because it is too large Load Diff

21
scripts/settings/src/types.d.ts vendored Normal file
View File

@@ -0,0 +1,21 @@
export interface Config {
name: string;
description: string;
type: string?;
default: any;
help: string;
choices: string[];
required: boolean;
}
interface Manifest {
description: string;
}
export interface Module {
name: string;
description: string;
configs: { [key: string]: Config };
manifest: Manifest;
display_name: string;
}

21
scripts/settings/tsconfig.json Normal file
View File

@@ -0,0 +1,21 @@
{
"compilerOptions": {
"target": "ESNext",
"useDefineForClassFields": true,
"lib": ["DOM", "DOM.Iterable", "ESNext"],
"allowJs": false,
"skipLibCheck": true,
"esModuleInterop": false,
"allowSyntheticDefaultImports": true,
"strict": true,
"forceConsistentCasingInFileNames": true,
"module": "ESNext",
"moduleResolution": "Node",
"resolveJsonModule": true,
"isolatedModules": true,
"noEmit": true,
"jsx": "react-jsx"
},
"include": ["src"],
"references": [{ "path": "./tsconfig.node.json" }]
}

9
scripts/settings/tsconfig.node.json Normal file
View File

@@ -0,0 +1,9 @@
{
"compilerOptions": {
"composite": true,
"module": "ESNext",
"moduleResolution": "Node",
"allowSyntheticDefaultImports": true
},
"include": ["vite.config.ts"]
}

12
scripts/settings/vite.config.ts Normal file
View File

@@ -0,0 +1,12 @@
import { defineConfig } from 'vite';
import react from '@vitejs/plugin-react';
import { viteSingleFile } from "vite-plugin-singlefile"
// https://vite.dev/config/
export default defineConfig({
plugins: [react(), viteSingleFile()],
build: {
minify: false,
sourcemap: true,
}
});

2
src/auto_archiver/__main__.py
View File

@@ -3,7 +3,7 @@ from auto_archiver.core.orchestrator import ArchivingOrchestrator
import sys
def main():
ArchivingOrchestrator().run(sys.argv[1:])
for _ in ArchivingOrchestrator()._command_line_run(sys.argv[1:]): pass
if __name__ == "__main__":
main()

2
src/auto_archiver/core/__init__.py
View File

@@ -3,7 +3,7 @@
"""
from .metadata import Metadata
from .media import Media
from .module import BaseModule
from .base_module import BaseModule
# cannot import ArchivingOrchestrator/Config to avoid circular dep
# from .orchestrator import ArchivingOrchestrator

100
src/auto_archiver/core/base_module.py
View File

@@ -1,13 +1,18 @@
from urllib.parse import urlparse
from typing import Mapping, Any
from __future__ import annotations
from typing import Mapping, Any, Type, TYPE_CHECKING
from abc import ABC
from copy import deepcopy, copy
from tempfile import TemporaryDirectory
from auto_archiver.utils import url as UrlUtil
from auto_archiver.core.consts import MODULE_TYPES as CONF_MODULE_TYPES
from loguru import logger
if TYPE_CHECKING:
from .module import ModuleFactory
class BaseModule(ABC):
"""
@@ -17,41 +22,24 @@ class BaseModule(ABC):
however modules can have a .setup() method to run any setup code
(e.g. logging in to a site, spinning up a browser etc.)
See BaseModule.MODULE_TYPES for the types of modules you can create, noting that
See consts.MODULE_TYPES for the types of modules you can create, noting that
a subclass can be of multiple types. For example, a module that extracts data from
a website and stores it in a database would be both an 'extractor' and a 'database' module.
Each module is a python package, and should have a __manifest__.py file in the
same directory as the module file. The __manifest__.py specifies the module information
like name, author, version, dependencies etc. See BaseModule._DEFAULT_MANIFEST for the
like name, author, version, dependencies etc. See DEFAULT_MANIFEST for the
default manifest structure.
"""
MODULE_TYPES = [
'feeder',
'extractor',
'enricher',
'database',
'storage',
'formatter'
]
_DEFAULT_MANIFEST = {
'name': '', # the display name of the module
'author': 'Bellingcat', # creator of the module, leave this as Bellingcat or set your own name!
'type': [], # the type of the module, can be one or more of BaseModule.MODULE_TYPES
'requires_setup': True, # whether or not this module requires additional setup such as setting API Keys or installing additional softare
'description': '', # a description of the module
'dependencies': {}, # external dependencies, e.g. python packages or binaries, in dictionary format
'entry_point': '', # the entry point for the module, in the format 'module_name::ClassName'. This can be left blank to use the default entry point of module_name::ModuleName
'version': '1.0', # the version of the module
'configs': {} # any configuration options this module has, these will be exposed to the user in the config file or via the command line
}
MODULE_TYPES = CONF_MODULE_TYPES
# NOTE: these here are declard as class variables, but they are overridden by the instance variables in the __init__ method
config: Mapping[str, Any]
authentication: Mapping[str, Mapping[str, str]]
name: str
module_factory: ModuleFactory
# this is set by the orchestrator prior to archiving
tmp_dir: TemporaryDirectory = None
@@ -62,14 +50,6 @@ class BaseModule(ABC):
def config_setup(self, config: dict):
authentication = config.get('authentication', {})
# extract out concatenated sites
for key, val in copy(authentication).items():
if "," in key:
for site in key.split(","):
authentication[site] = val
del authentication[key]
# this is important. Each instance is given its own deepcopied config, so modules cannot
# change values to affect other modules
config = deepcopy(config)
@@ -89,21 +69,28 @@ class BaseModule(ABC):
Returns the authentication information for a given site. This is used to authenticate
with a site before extracting data. The site should be the domain of the site, e.g. 'twitter.com'
extract_cookies: bool - whether or not to extract cookies from the given browser and return the
cookie jar (disabling can speed up) processing if you don't actually need the cookies jar
:param site: the domain of the site to get authentication information for
:param extract_cookies: whether or not to extract cookies from the given browser/file and return the cookie jar (disabling can speed up processing if you don't actually need the cookies jar).
Currently, the dict can have keys of the following types:
- username: str - the username to use for login
- password: str - the password to use for login
- api_key: str - the API key to use for login
- api_secret: str - the API secret to use for login
- cookie: str - a cookie string to use for login (specific to this site)
- cookies_jar: YoutubeDLCookieJar | http.cookiejar.MozillaCookieJar - a cookie jar compatible with requests (e.g. `requests.get(cookies=cookie_jar)`)
:returns: authdict dict of login information for the given site
**Global options:**\n
* cookies_from_browser: str - the name of the browser to extract cookies from (e.g. 'chrome', 'firefox' - uses ytdlp under the hood to extract\n
* cookies_file: str - the path to a cookies file to use for login\n
**Currently, the sites dict can have keys of the following types:**\n
* username: str - the username to use for login\n
* password: str - the password to use for login\n
* api_key: str - the API key to use for login\n
* api_secret: str - the API secret to use for login\n
* cookie: str - a cookie string to use for login (specific to this site)\n
* cookies_file: str - the path to a cookies file to use for login (specific to this site)\n
* cookies_from_browser: str - the name of the browser to extract cookies from (specitic for this site)\n
"""
# TODO: think about if/how we can deal with sites that have multiple domains (main one is x.com/twitter.com)
# for now the user must enter them both, like "x.com,twitter.com" in their config. Maybe we just hard-code?
site = UrlUtil.domain_for_url(site)
site = UrlUtil.domain_for_url(site).lstrip("www.")
# add the 'www' version of the site to the list of sites to check
authdict = {}
@@ -118,8 +105,8 @@ class BaseModule(ABC):
for key in self.authentication.keys():
if key in site or site in key:
logger.debug(f"Could not find exact authentication information for site '{site}'. \
did find information for '{key}' which is close, is this what you meant? \
If so, edit your authentication settings to make sure it exactly matches.")
did find information for '{key}' which is close, is this what you meant? \
If so, edit your authentication settings to make sure it exactly matches.")
def get_ytdlp_cookiejar(args):
import yt_dlp
@@ -130,16 +117,29 @@ class BaseModule(ABC):
ytdlp_opts = getattr(parse_options(args), 'ydl_opts')
return yt_dlp.YoutubeDL(ytdlp_opts).cookiejar
# get the cookies jar, prefer the browser cookies than the file
if 'cookies_from_browser' in self.authentication:
get_cookiejar_options = None
# order of priority:
# 1. cookies_from_browser setting in site config
# 2. cookies_file setting in site config
# 3. cookies_from_browser setting in global config
# 4. cookies_file setting in global config
if 'cookies_from_browser' in authdict:
get_cookiejar_options = ['--cookies-from-browser', authdict['cookies_from_browser']]
elif 'cookies_file' in authdict:
get_cookiejar_options = ['--cookies', authdict['cookies_file']]
elif 'cookies_from_browser' in self.authentication:
authdict['cookies_from_browser'] = self.authentication['cookies_from_browser']
if extract_cookies:
authdict['cookies_jar'] = get_ytdlp_cookiejar(['--cookies-from-browser', self.authentication['cookies_from_browser']])
get_cookiejar_options = ['--cookies-from-browser', self.authentication['cookies_from_browser']]
elif 'cookies_file' in self.authentication:
authdict['cookies_file'] = self.authentication['cookies_file']
if extract_cookies:
authdict['cookies_jar'] = get_ytdlp_cookiejar(['--cookies', self.authentication['cookies_file']])
get_cookiejar_options = ['--cookies', self.authentication['cookies_file']]
if get_cookiejar_options:
authdict['cookies_jar'] = get_ytdlp_cookiejar(get_cookiejar_options)
return authdict
def repr(self):

83
src/auto_archiver/core/config.py
View File

@@ -7,21 +7,23 @@ flexible setup in various environments.
import argparse
from ruamel.yaml import YAML, CommentedMap, add_representer
import json
from loguru import logger
from copy import deepcopy
from .module import BaseModule
from auto_archiver.core.consts import MODULE_TYPES
from typing import Any, List, Type, Tuple
_yaml: YAML = YAML()
DEFAULT_CONFIG_FILE = "secrets/orchestration.yaml"
EMPTY_CONFIG = _yaml.load("""
# Auto Archiver Configuration
# Steps are the modules that will be run in the order they are defined
steps:""" + "".join([f"\n {module}s: []" for module in BaseModule.MODULE_TYPES]) + \
# Steps are the modules that will be run in the order they are defined
steps:""" + "".join([f"\n {module}s: []" for module in MODULE_TYPES]) + \
"""
# Global configuration
@@ -48,9 +50,61 @@ authentication: {}
logging:
level: INFO
""")
# note: 'logging' is explicitly added above in order to better format the config file
# Arg Parse Actions/Classes
class AuthenticationJsonParseAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
try:
auth_dict = json.loads(values)
setattr(namespace, self.dest, auth_dict)
except json.JSONDecodeError as e:
raise argparse.ArgumentTypeError(f"Invalid JSON input for argument '{self.dest}': {e}")
def load_from_file(path):
try:
with open(path, 'r') as f:
try:
auth_dict = json.load(f)
except json.JSONDecodeError:
f.seek(0)
# maybe it's yaml, try that
auth_dict = _yaml.load(f)
if auth_dict.get('authentication'):
auth_dict = auth_dict['authentication']
auth_dict['load_from_file'] = path
return auth_dict
except:
return None
if isinstance(auth_dict, dict) and auth_dict.get('from_file'):
auth_dict = load_from_file(auth_dict['from_file'])
elif isinstance(auth_dict, str):
# if it's a string
auth_dict = load_from_file(auth_dict)
if not isinstance(auth_dict, dict):
raise argparse.ArgumentTypeError("Authentication must be a dictionary of site names and their authentication methods")
global_options = ['cookies_from_browser', 'cookies_file', 'load_from_file']
for key, auth in auth_dict.items():
if key in global_options:
continue
if not isinstance(key, str) or not isinstance(auth, dict):
raise argparse.ArgumentTypeError(f"Authentication must be a dictionary of site names and their authentication methods. Valid global configs are {global_options}")
setattr(namespace, self.dest, auth_dict)
class UniqueAppendAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
for value in values:
if value not in getattr(namespace, self.dest):
getattr(namespace, self.dest).append(value)
class DefaultValidatingParser(argparse.ArgumentParser):
def error(self, message):
@@ -81,6 +135,7 @@ class DefaultValidatingParser(argparse.ArgumentParser):
return super().parse_known_args(args, namespace)
# Config Utils
def to_dot_notation(yaml_conf: CommentedMap | dict) -> dict:
dotdict = {}
@@ -128,6 +183,11 @@ def merge_dicts(dotdict: dict, yaml_dict: CommentedMap) -> CommentedMap:
yaml_subdict[key] = value
continue
if key == 'steps':
for module_type, modules in value.items():
# overwrite the 'steps' from the config file with the ones from the CLI
yaml_subdict[key][module_type] = modules
if is_dict_type(value):
update_dict(value, yaml_subdict[key])
elif is_list_type(value):
@@ -136,7 +196,6 @@ def merge_dicts(dotdict: dict, yaml_dict: CommentedMap) -> CommentedMap:
yaml_subdict[key] = value
update_dict(from_dot_notation(dotdict), yaml_dict)
return yaml_dict
def read_yaml(yaml_filename: str) -> CommentedMap:
@@ -148,8 +207,8 @@ def read_yaml(yaml_filename: str) -> CommentedMap:
pass
if not config:
config = EMPTY_CONFIG
config = deepcopy(EMPTY_CONFIG)
return config
# TODO: make this tidier/find a way to notify of which keys should not be stored
@@ -158,6 +217,14 @@ def read_yaml(yaml_filename: str) -> CommentedMap:
def store_yaml(config: CommentedMap, yaml_filename: str) -> None:
config_to_save = deepcopy(config)
auth_dict = config_to_save.get("authentication", {})
if auth_dict and auth_dict.get('load_from_file'):
# remove all other values from the config, don't want to store it in the config file
auth_dict = {"load_from_file": auth_dict["load_from_file"]}
config_to_save.pop('urls', None)
with open(yaml_filename, "w", encoding="utf-8") as outf:
_yaml.dump(config_to_save, outf)
_yaml.dump(config_to_save, outf)
def is_valid_config(config: CommentedMap) -> bool:
return config and config != EMPTY_CONFIG

25
src/auto_archiver/core/consts.py Normal file
View File

@@ -0,0 +1,25 @@
class SetupError(ValueError):
pass
MODULE_TYPES = [
'feeder',
'extractor',
'enricher',
'database',
'storage',
'formatter'
]
MANIFEST_FILE = "__manifest__.py"
DEFAULT_MANIFEST = {
'name': '', # the display name of the module
'author': 'Bellingcat', # creator of the module, leave this as Bellingcat or set your own name!
'type': [], # the type of the module, can be one or more of MODULE_TYPES
'requires_setup': True, # whether or not this module requires additional setup such as setting API Keys or installing additional software
'description': '', # a description of the module
'dependencies': {}, # external dependencies, e.g. python packages or binaries, in dictionary format
'entry_point': '', # the entry point for the module, in the format 'module_name::ClassName'. This can be left blank to use the default entry point of module_name::ModuleName
'version': '1.0', # the version of the module
'configs': {} # any configuration options this module has, these will be exposed to the user in the config file or via the command line
}

10
src/auto_archiver/core/database.py
View File

@@ -1,3 +1,8 @@
"""
Database module for the auto-archiver that defines the interface for implementing database modules
in the media archiving framework.
"""
from __future__ import annotations
from abc import abstractmethod
from typing import Union
@@ -5,6 +10,11 @@ from typing import Union
from auto_archiver.core import Metadata, BaseModule
class Database(BaseModule):
"""
Base class for implementing database modules in the media archiving framework.
Subclasses must implement the `fetch` and `done` methods to define platform-specific behavior.
"""
def started(self, item: Metadata) -> None:
"""signals the DB that the given item archival has started"""

15
src/auto_archiver/core/enricher.py
View File

@@ -1,5 +1,5 @@
"""
Enrichers are modular components that enhance archived content by adding
Base module for Enrichers – modular components that enhance archived content by adding
context, metadata, or additional processing.
These add additional information to the context, such as screenshots, hashes, and metadata.
@@ -13,7 +13,16 @@ from abc import abstractmethod
from auto_archiver.core import Metadata, BaseModule
class Enricher(BaseModule):
"""Base classes and utilities for enrichers in the Auto-Archiver system."""
"""Base classes and utilities for enrichers in the Auto Archiver system.
Enricher modules must implement the `enrich` method to define their behavior.
"""
@abstractmethod
def enrich(self, to_enrich: Metadata) -> None: pass
def enrich(self, to_enrich: Metadata) -> None:
"""
Enriches a Metadata object with additional information or context.
Takes the metadata object to enrich as an argument and modifies it in place, returning None.
"""
pass

21
src/auto_archiver/core/extractor.py
View File

@@ -29,14 +29,24 @@ class Extractor(BaseModule):
valid_url: re.Pattern = None
def cleanup(self) -> None:
# called when extractors are done, or upon errors, cleanup any resources
"""
Called when extractors are done, or upon errors, cleanup any resources
"""
pass
def sanitize_url(self, url: str) -> str:
# used to clean unnecessary URL parameters OR unfurl redirect links
"""
Used to clean unnecessary URL parameters OR unfurl redirect links
"""
return url
def match_link(self, url: str) -> re.Match:
"""
Returns a match object if the given URL matches the valid_url pattern or False/None if not.
Normally used in the `suitable` method to check if the URL is supported by this extractor.
"""
return self.valid_url.match(url)
def suitable(self, url: str) -> bool:
@@ -71,7 +81,8 @@ class Extractor(BaseModule):
if len(to_filename) > 64:
to_filename = to_filename[-64:]
to_filename = os.path.join(self.tmp_dir, to_filename)
if verbose: logger.debug(f"downloading {url[0:50]=} {to_filename=}")
if verbose:
logger.debug(f"downloading {url[0:50]=} {to_filename=}")
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36'
}
@@ -80,8 +91,8 @@ class Extractor(BaseModule):
d.raise_for_status()
# get mimetype from the response headers
if not Path(to_filename).suffix:
content_type = d.headers.get('Content-Type')
if not mimetypes.guess_type(to_filename)[0]:
content_type = d.headers.get('Content-Type') or self._guess_file_type(url)
extension = mimetypes.guess_extension(content_type)
if extension:
to_filename += extension

18
src/auto_archiver/core/feeder.py
View File

@@ -1,3 +1,7 @@
"""
The feeder base module defines the interface for implementing feeders in the media archiving framework.
"""
from __future__ import annotations
from abc import abstractmethod
from auto_archiver.core import Metadata
@@ -5,5 +9,17 @@ from auto_archiver.core import BaseModule
class Feeder(BaseModule):
"""
Base class for implementing feeders in the media archiving framework.
Subclasses must implement the `__iter__` method to define platform-specific behavior.
"""
@abstractmethod
def __iter__(self) -> Metadata: return None
def __iter__(self) -> Metadata:
"""
Returns an iterator (use `yield`) over the items to be archived.
These should be instances of Metadata, typically created with Metadata().set_url(url).
"""
return None

17
src/auto_archiver/core/formatter.py
View File

@@ -1,9 +1,24 @@
"""
Base module for formatters – modular components that format metadata into media objects for storage.
The most commonly used formatter is the HTML formatter, which takes metadata and formats it into an HTML file for storage.
"""
from __future__ import annotations
from abc import abstractmethod
from auto_archiver.core import Metadata, Media, BaseModule
class Formatter(BaseModule):
"""
Base class for implementing formatters in the media archiving framework.
Subclasses must implement the `format` method to define their behavior.
"""
@abstractmethod
def format(self, item: Metadata) -> Media: return None
def format(self, item: Metadata) -> Media:
"""
Formats a Metadata object into a user-viewable format (e.g. HTML) and stores it if needed.
"""
return None

13
src/auto_archiver/core/media.py
View File

@@ -6,7 +6,7 @@ nested media retrieval, and type validation.
from __future__ import annotations
import os
import traceback
from typing import Any, List
from typing import Any, List, Iterator
from dataclasses import dataclass, field
from dataclasses_json import dataclass_json, config
import mimetypes
@@ -21,14 +21,13 @@ class Media:
Represents a media file with associated properties and storage details.
Attributes:
- filename: The file path of the media.
- key: An optional identifier for the media.
- filename: The file path of the media as saved locally (temporarily, before uploading to the storage).
- urls: A list of URLs where the media is stored or accessible.
- properties: Additional metadata or transformations for the media.
- _mimetype: The media's mimetype (e.g., image/jpeg, video/mp4).
"""
filename: str
key: str = None
_key: str = None
urls: List[str] = field(default_factory=list)
properties: dict = field(default_factory=dict)
_mimetype: str = None # eg: image/jpeg
@@ -47,7 +46,7 @@ class Media:
for any_media in self.all_inner_media(include_self=True):
s.store(any_media, url, metadata=metadata)
def all_inner_media(self, include_self=False):
def all_inner_media(self, include_self=False) -> Iterator[Media]:
"""Retrieves all media, including nested media within properties or transformations on original media.
This function returns a generator for all the inner media.
@@ -67,6 +66,10 @@ class Media:
# checks if the media is already stored in the given storage
return len(self.urls) > 0 and len(self.urls) == len(in_storage.config["steps"]["storages"])
@property
def key(self) -> str:
return self._key
def set(self, key: str, value: Any) -> Media:
self.properties[key] = value
return self

216
src/auto_archiver/core/module.py
View File

@@ -6,7 +6,7 @@ by handling user configuration, validating the steps properties, and implementin
from __future__ import annotations
from dataclasses import dataclass
from typing import List
from typing import List, TYPE_CHECKING
import shutil
import ast
import copy
@@ -16,99 +16,116 @@ import os
from os.path import join
from loguru import logger
import auto_archiver
from .base_module import BaseModule
from auto_archiver.core.consts import DEFAULT_MANIFEST, MANIFEST_FILE
_LAZY_LOADED_MODULES = {}
MANIFEST_FILE = "__manifest__.py"
if TYPE_CHECKING:
from .base_module import BaseModule
def setup_paths(paths: list[str]) -> None:
"""
Sets up the paths for the modules to be loaded from
This is necessary for the modules to be imported correctly
"""
for path in paths:
# check path exists, if it doesn't, log a warning
if not os.path.exists(path):
logger.warning(f"Path '{path}' does not exist. Skipping...")
continue
HAS_SETUP_PATHS = False
# see odoo/module/module.py -> initialize_sys_path
if path not in auto_archiver.modules.__path__:
auto_archiver.modules.__path__.append(path)
class ModuleFactory:
# sort based on the length of the path, so that the longest path is last in the list
auto_archiver.modules.__path__ = sorted(auto_archiver.modules.__path__, key=len, reverse=True)
def __init__(self):
self._lazy_modules = {}
def get_module(module_name: str, config: dict) -> BaseModule:
"""
Gets and sets up a module using the provided config
This will actually load and instantiate the module, and load all its dependencies (i.e. not lazy)
"""
return get_module_lazy(module_name).load(config)
def setup_paths(self, paths: list[str]) -> None:
"""
Sets up the paths for the modules to be loaded from
This is necessary for the modules to be imported correctly
"""
global HAS_SETUP_PATHS
def get_module_lazy(module_name: str, suppress_warnings: bool = False) -> LazyBaseModule:
"""
Lazily loads a module, returning a LazyBaseModule
This has all the information about the module, but does not load the module itself or its dependencies
To load an actual module, call .setup() on a lazy module
"""
if module_name in _LAZY_LOADED_MODULES:
return _LAZY_LOADED_MODULES[module_name]
available = available_modules(limit_to_modules=[module_name], suppress_warnings=suppress_warnings)
if not available:
raise IndexError(f"Module '{module_name}' not found. Are you sure it's installed/exists?")
return available[0]
def available_modules(with_manifest: bool=False, limit_to_modules: List[str]= [], suppress_warnings: bool = False) -> List[LazyBaseModule]:
# search through all valid 'modules' paths. Default is 'modules' in the current directory
# see odoo/modules/module.py -> get_modules
def is_really_module(module_path):
if os.path.isfile(join(module_path, MANIFEST_FILE)):
return True
all_modules = []
for module_folder in auto_archiver.modules.__path__:
# walk through each module in module_folder and check if it has a valid manifest
try:
possible_modules = os.listdir(module_folder)
except FileNotFoundError:
logger.warning(f"Module folder {module_folder} does not exist")
continue
for possible_module in possible_modules:
if limit_to_modules and possible_module not in limit_to_modules:
for path in paths:
# check path exists, if it doesn't, log a warning
if not os.path.exists(path):
logger.warning(f"Path '{path}' does not exist. Skipping...")
continue
possible_module_path = join(module_folder, possible_module)
if not is_really_module(possible_module_path):
# see odoo/module/module.py -> initialize_sys_path
if path not in auto_archiver.modules.__path__:
if HAS_SETUP_PATHS == True:
logger.warning(f"You are attempting to re-initialise the module paths with: '{path}' for a 2nd time. \
This could lead to unexpected behaviour. It is recommended to only use a single modules path. \
If you wish to load modules from different paths then load a 2nd python interpreter (e.g. using multiprocessing).")
auto_archiver.modules.__path__.append(path)
# sort based on the length of the path, so that the longest path is last in the list
auto_archiver.modules.__path__ = sorted(auto_archiver.modules.__path__, key=len, reverse=True)
HAS_SETUP_PATHS = True
def get_module(self, module_name: str, config: dict) -> BaseModule:
"""
Gets and sets up a module using the provided config
This will actually load and instantiate the module, and load all its dependencies (i.e. not lazy)
"""
return self.get_module_lazy(module_name).load(config)
def get_module_lazy(self, module_name: str, suppress_warnings: bool = False) -> LazyBaseModule:
"""
Lazily loads a module, returning a LazyBaseModule
This has all the information about the module, but does not load the module itself or its dependencies
To load an actual module, call .setup() on a lazy module
"""
if module_name in self._lazy_modules:
return self._lazy_modules[module_name]
available = self.available_modules(limit_to_modules=[module_name], suppress_warnings=suppress_warnings)
if not available:
message = f"Module '{module_name}' not found. Are you sure it's installed/exists?"
if 'archiver' in module_name:
message += f" Did you mean {module_name.replace('archiver', 'extractor')}?"
raise IndexError(message)
return available[0]
def available_modules(self, limit_to_modules: List[str]= [], suppress_warnings: bool = False) -> List[LazyBaseModule]:
# search through all valid 'modules' paths. Default is 'modules' in the current directory
# see odoo/modules/module.py -> get_modules
def is_really_module(module_path):
if os.path.isfile(join(module_path, MANIFEST_FILE)):
return True
all_modules = []
for module_folder in auto_archiver.modules.__path__:
# walk through each module in module_folder and check if it has a valid manifest
try:
possible_modules = os.listdir(module_folder)
except FileNotFoundError:
logger.warning(f"Module folder {module_folder} does not exist")
continue
if _LAZY_LOADED_MODULES.get(possible_module):
continue
lazy_module = LazyBaseModule(possible_module, possible_module_path)
_LAZY_LOADED_MODULES[possible_module] = lazy_module
for possible_module in possible_modules:
if limit_to_modules and possible_module not in limit_to_modules:
continue
all_modules.append(lazy_module)
if not suppress_warnings:
for module in limit_to_modules:
if not any(module == m.name for m in all_modules):
logger.warning(f"Module '{module}' not found. Are you sure it's installed?")
possible_module_path = join(module_folder, possible_module)
if not is_really_module(possible_module_path):
continue
if self._lazy_modules.get(possible_module):
continue
lazy_module = LazyBaseModule(possible_module, possible_module_path, factory=self)
return all_modules
self._lazy_modules[possible_module] = lazy_module
all_modules.append(lazy_module)
if not suppress_warnings:
for module in limit_to_modules:
if not any(module == m.name for m in all_modules):
logger.warning(f"Module '{module}' not found. Are you sure it's installed?")
return all_modules
@dataclass
class LazyBaseModule:
@@ -120,17 +137,22 @@ class LazyBaseModule:
"""
name: str
type: list
description: str
path: str
module_factory: ModuleFactory
_manifest: dict = None
_instance: BaseModule = None
_entry_point: str = None
def __init__(self, module_name, path):
def __init__(self, module_name, path, factory: ModuleFactory):
self.name = module_name
self.path = path
self.module_factory = factory
@property
def type(self):
return self.manifest['type']
@property
def entry_point(self):
@@ -161,16 +183,15 @@ class LazyBaseModule:
return self._manifest
# print(f"Loading manifest for module {module_path}")
# load the manifest file
manifest = copy.deepcopy(BaseModule._DEFAULT_MANIFEST)
manifest = copy.deepcopy(DEFAULT_MANIFEST)
with open(join(self.path, MANIFEST_FILE)) as f:
try:
manifest.update(ast.literal_eval(f.read()))
except (ValueError, TypeError, SyntaxError, MemoryError, RecursionError) as e:
logger.error(f"Error loading manifest from file {self.path}/{MANIFEST_FILE}: {e}")
raise ValueError(f"Error loading manifest from file {self.path}/{MANIFEST_FILE}: {e}")
self._manifest = manifest
self.type = manifest['type']
self._entry_point = manifest['entry_point']
self.description = manifest['description']
self.version = manifest['version']
@@ -189,13 +210,14 @@ class LazyBaseModule:
# clear out any empty strings that a user may have erroneously added
continue
if not check(dep):
logger.error(f"Module '{self.name}' requires external dependency '{dep}' which is not available/setup. Have you installed the required dependencies for the '{self.name}' module? See the README for more information.")
logger.error(f"Module '{self.name}' requires external dependency '{dep}' which is not available/setup. \
Have you installed the required dependencies for the '{self.name}' module? See the README for more information.")
exit(1)
def check_python_dep(dep):
# first check if it's a module:
try:
m = get_module_lazy(dep, suppress_warnings=True)
m = self.module_factory.get_module_lazy(dep, suppress_warnings=True)
try:
# we must now load this module and set it up with the config
m.load(config)
@@ -230,19 +252,21 @@ class LazyBaseModule:
__import__(f'{qualname}.{file_name}', fromlist=[self.entry_point])
# finally, get the class instance
instance: BaseModule = getattr(sys.modules[sub_qualname], class_name)()
if not getattr(instance, 'name', None):
instance.name = self.name
if not getattr(instance, 'display_name', None):
instance.display_name = self.display_name
self._instance = instance
# set the name, display name and module factory
instance.name = self.name
instance.display_name = self.display_name
instance.module_factory = self.module_factory
# merge the default config with the user config
default_config = dict((k, v['default']) for k, v in self.configs.items() if v.get('default'))
default_config = dict((k, v['default']) for k, v in self.configs.items() if 'default' in v)
config[self.name] = default_config | config.get(self.name, {})
instance.config_setup(config)
instance.setup()
# save the instance for future easy loading
self._instance = instance
return instance
def __repr__(self):

407
src/auto_archiver/core/orchestrator.py
View File

@@ -5,88 +5,61 @@
"""
from __future__ import annotations
from typing import Generator, Union, List, Type
from urllib.parse import urlparse
from ipaddress import ip_address
from typing import Generator, Union, List, Type, TYPE_CHECKING
import argparse
import os
import sys
import json
from tempfile import TemporaryDirectory
import traceback
from copy import copy
from rich_argparse import RichHelpFormatter
from loguru import logger
import requests
from .metadata import Metadata, Media
from auto_archiver.version import __version__
from .config import _yaml, read_yaml, store_yaml, to_dot_notation, merge_dicts, EMPTY_CONFIG, DefaultValidatingParser
from .module import available_modules, LazyBaseModule, get_module, setup_paths
from .config import read_yaml, store_yaml, to_dot_notation, merge_dicts, is_valid_config, \
DefaultValidatingParser, UniqueAppendAction, AuthenticationJsonParseAction, DEFAULT_CONFIG_FILE
from .module import ModuleFactory, LazyBaseModule
from . import validators, Feeder, Extractor, Database, Storage, Formatter, Enricher
from .module import BaseModule
from .consts import MODULE_TYPES, SetupError
from auto_archiver.utils.url import check_url_or_raise
from loguru import logger
DEFAULT_CONFIG_FILE = "orchestration.yaml"
class JsonParseAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
try:
setattr(namespace, self.dest, json.loads(values))
except json.JSONDecodeError as e:
raise argparse.ArgumentTypeError(f"Invalid JSON input for argument '{self.dest}': {e}")
class AuthenticationJsonParseAction(JsonParseAction):
def __call__(self, parser, namespace, values, option_string=None):
super().__call__(parser, namespace, values, option_string)
auth_dict = getattr(namespace, self.dest)
if isinstance(auth_dict, str):
# if it's a string
try:
with open(auth_dict, 'r') as f:
try:
auth_dict = json.load(f)
except json.JSONDecodeError:
# maybe it's yaml, try that
auth_dict = _yaml.load(f)
except:
pass
if not isinstance(auth_dict, dict):
raise argparse.ArgumentTypeError("Authentication must be a dictionary of site names and their authentication methods")
for site, auth in auth_dict.items():
if not isinstance(site, str) or not isinstance(auth, dict):
raise argparse.ArgumentTypeError("Authentication must be a dictionary of site names and their authentication methods")
setattr(namespace, self.dest, auth_dict)
class UniqueAppendAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
if not hasattr(namespace, self.dest):
setattr(namespace, self.dest, [])
for value in values:
if value not in getattr(namespace, self.dest):
getattr(namespace, self.dest).append(value)
if TYPE_CHECKING:
from .base_module import BaseModule
from .module import LazyBaseModule
class ArchivingOrchestrator:
# instance variables
module_factory: ModuleFactory
setup_finished: bool
logger_id: int
# instance variables, used for convenience to access modules by step
feeders: List[Type[Feeder]]
extractors: List[Type[Extractor]]
enrichers: List[Type[Enricher]]
databases: List[Type[Database]]
storages: List[Type[Storage]]
formatters: List[Type[Formatter]]
def __init__(self):
self.module_factory = ModuleFactory()
self.setup_finished = False
self.logger_id = None
def setup_basic_parser(self):
parser = argparse.ArgumentParser(
prog="auto-archiver",
add_help=False,
description="""
prog="auto-archiver",
add_help=False,
description="""
Auto Archiver is a CLI tool to archive media/metadata from online URLs;
it can read URLs from many sources (Google Sheets, Command Line, ...); and write results to many destinations too (CSV, Google Sheets, MongoDB, ...)!
""",
epilog="Check the code at https://github.com/bellingcat/auto-archiver",
formatter_class=RichHelpFormatter,
epilog="Check the code at https://github.com/bellingcat/auto-archiver",
formatter_class=RichHelpFormatter,
)
parser.add_argument('--help', '-h', action='store_true', dest='help', help='show a full help message and exit')
parser.add_argument('--version', action='version', version=__version__)
@@ -98,105 +71,136 @@ class ArchivingOrchestrator:
self.basic_parser = parser
return parser
def check_steps(self, config):
for module_type in MODULE_TYPES:
if not config['steps'].get(f"{module_type}s", []):
if module_type == 'feeder' or module_type == 'formatter' and config['steps'].get(f"{module_type}"):
raise SetupError(f"It appears you have '{module_type}' set under 'steps' in your configuration file, but as of version 0.13.0 of Auto Archiver, you must use '{module_type}s'. Change this in your configuration file and try again. \
Here's how that would look: \n\nsteps:\n {module_type}s:\n - [your_{module_type}_name_here]\n {'extractors:...' if module_type == 'feeder' else '...'}\n")
if module_type == 'extractor' and config['steps'].get('archivers'):
raise SetupError(f"As of version 0.13.0 of Auto Archiver, the 'archivers' step name has been changed to 'extractors'. Change this in your configuration file and try again. \
Here's how that would look: \n\nsteps:\n extractors:\n - [your_extractor_name_here]\n enrichers:...\n")
raise SetupError(f"No {module_type}s were configured. Make sure to set at least one {module_type} in your configuration file or on the command line (using --{module_type}s)")
def setup_complete_parser(self, basic_config: dict, yaml_config: dict, unused_args: list[str]) -> None:
# modules parser to get the overridden 'steps' values
modules_parser = argparse.ArgumentParser(
add_help=False,
)
self.add_modules_args(modules_parser)
cli_modules, unused_args = modules_parser.parse_known_args(unused_args)
for module_type in MODULE_TYPES:
yaml_config['steps'][f"{module_type}s"] = getattr(cli_modules, f"{module_type}s", []) or yaml_config['steps'].get(f"{module_type}s", [])
parser = DefaultValidatingParser(
add_help=False,
)
self.add_additional_args(parser)
# merge command line module args (--feeders, --enrichers etc.) and add them to the config
# check what mode we're in
# if we have a config file, use that to decide which modules to load
# if simple, we'll load just the modules that has requires_setup = False
# if full, we'll load all modules
# TODO: BUG** - basic_config won't have steps in it, since these args aren't added to 'basic_parser'
# but should we add them? Or should we just add them to the 'complete' parser?
if yaml_config != EMPTY_CONFIG:
if is_valid_config(yaml_config):
self.check_steps(yaml_config)
# only load the modules enabled in config
# TODO: if some steps are empty (e.g. 'feeders' is empty), should we default to the 'simple' ones? Or only if they are ALL empty?
enabled_modules = []
# first loads the modules from the config file, then from the command line
for config in [yaml_config['steps'], basic_config.__dict__]:
for module_type in BaseModule.MODULE_TYPES:
enabled_modules.extend(config.get(f"{module_type}s", []))
for module_type in MODULE_TYPES:
enabled_modules.extend(yaml_config['steps'].get(f"{module_type}s", []))
# clear out duplicates, but keep the order
enabled_modules = list(dict.fromkeys(enabled_modules))
avail_modules = available_modules(with_manifest=True, limit_to_modules=enabled_modules, suppress_warnings=True)
self.add_module_args(avail_modules, parser)
avail_modules = self.module_factory.available_modules(limit_to_modules=enabled_modules, suppress_warnings=True)
self.add_individual_module_args(avail_modules, parser)
elif basic_config.mode == 'simple':
simple_modules = [module for module in available_modules(with_manifest=True) if not module.requires_setup]
self.add_module_args(simple_modules, parser)
simple_modules = [module for module in self.module_factory.available_modules() if not module.requires_setup]
self.add_individual_module_args(simple_modules, parser)
# for simple mode, we use the cli_feeder and any modules that don't require setup
yaml_config['steps']['feeders'] = ['cli_feeder']
# add them to the config
for module in simple_modules:
for module_type in module.type:
yaml_config['steps'].setdefault(f"{module_type}s", []).append(module.name)
else:
# load all modules, they're not using the 'simple' mode
self.add_module_args(available_modules(with_manifest=True), parser)
all_modules = self.module_factory.available_modules()
# add all the modules to the steps
for module in all_modules:
for module_type in module.type:
yaml_config['steps'].setdefault(f"{module_type}s", []).append(module.name)
self.add_individual_module_args(all_modules, parser)
parser.set_defaults(**to_dot_notation(yaml_config))
# reload the parser with the new arguments, now that we have them
parsed, unknown = parser.parse_known_args(unused_args)
# merge the new config with the old one
self.config = merge_dicts(vars(parsed), yaml_config)
config = merge_dicts(vars(parsed), yaml_config)
# set up the authentication dict as needed
config = self.setup_authentication(config)
# clean out args from the base_parser that we don't want in the config
for key in vars(basic_config):
self.config.pop(key, None)
config.pop(key, None)
# setup the logging
self.setup_logging()
self.setup_logging(config)
if unknown:
logger.warning(f"Ignoring unknown/unused arguments: {unknown}\nPerhaps you don't have this module enabled?")
if (self.config != yaml_config and basic_config.store) or not os.path.isfile(basic_config.config_file):
if (config != yaml_config and basic_config.store) or not os.path.isfile(basic_config.config_file):
logger.info(f"Storing configuration file to {basic_config.config_file}")
store_yaml(self.config, basic_config.config_file)
return self.config
store_yaml(config, basic_config.config_file)
return config
def add_modules_args(self, parser: argparse.ArgumentParser = None):
if not parser:
parser = self.parser
# Module loading from the command line
for module_type in MODULE_TYPES:
parser.add_argument(f'--{module_type}s', dest=f'{module_type}s', nargs='+', help=f'the {module_type}s to use', default=[], action=UniqueAppendAction)
def add_additional_args(self, parser: argparse.ArgumentParser = None):
if not parser:
parser = self.parser
# allow passing URLs directly on the command line
parser.add_argument('urls', nargs='*', default=[], help='URL(s) to archive, either a single URL or a list of urls, should not come from config.yaml')
parser.add_argument('--feeders', dest='steps.feeders', nargs='+', default=['cli_feeder'], help='the feeders to use', action=UniqueAppendAction)
parser.add_argument('--enrichers', dest='steps.enrichers', nargs='+', help='the enrichers to use', action=UniqueAppendAction)
parser.add_argument('--extractors', dest='steps.extractors', nargs='+', help='the extractors to use', action=UniqueAppendAction)
parser.add_argument('--databases', dest='steps.databases', nargs='+', help='the databases to use', action=UniqueAppendAction)
parser.add_argument('--storages', dest='steps.storages', nargs='+', help='the storages to use', action=UniqueAppendAction)
parser.add_argument('--formatters', dest='steps.formatters', nargs='+', help='the formatter to use', action=UniqueAppendAction)
parser.add_argument('--authentication', dest='authentication', help='A dictionary of sites and their authentication methods \
(token, username etc.) that extractors can use to log into \
a website. If passing this on the command line, use a JSON string. \
You may also pass a path to a valid JSON/YAML file which will be parsed.',\
You may also pass a path to a valid JSON/YAML file which will be parsed.',
default={},
nargs="?",
action=AuthenticationJsonParseAction)
# logging arguments
parser.add_argument('--logging.level', action='store', dest='logging.level', choices=['INFO', 'DEBUG', 'ERROR', 'WARNING'], help='the logging level to use', default='INFO', type=str.upper)
parser.add_argument('--logging.file', action='store', dest='logging.file', help='the logging file to write to', default=None)
parser.add_argument('--logging.rotation', action='store', dest='logging.rotation', help='the logging rotation to use', default=None)
def add_module_args(self, modules: list[LazyBaseModule] = None, parser: argparse.ArgumentParser = None) -> None:
def add_individual_module_args(self, modules: list[LazyBaseModule] = None, parser: argparse.ArgumentParser = None) -> None:
if not modules:
modules = available_modules(with_manifest=True)
module: LazyBaseModule
modules = self.module_factory.available_modules()
for module in modules:
if module.name == 'cli_feeder':
# special case. For the CLI feeder, allow passing URLs directly on the command line without setting --cli_feeder.urls=
parser.add_argument('urls', nargs='*', default=[], help='URL(s) to archive, either a single URL or a list of urls, should not come from config.yaml')
continue
if not module.configs:
# this module has no configs, don't show anything in the help
# (TODO: do we want to show something about this module though, like a description?)
@@ -224,21 +228,35 @@ class ArchivingOrchestrator:
arg.should_store = should_store
def show_help(self, basic_config: dict):
# for the help message, we want to load *all* possible modules and show the help
# add configs as arg parser arguments
# for the help message, we want to load manifests from *all* possible modules and show their help/settings
# add configs as arg parser arguments
self.add_modules_args(self.basic_parser)
self.add_additional_args(self.basic_parser)
self.add_module_args(parser=self.basic_parser)
self.add_individual_module_args(parser=self.basic_parser)
self.basic_parser.print_help()
self.basic_parser.exit()
def setup_logging(self):
def setup_logging(self, config):
logging_config = config['logging']
if logging_config.get('enabled', True) is False:
# disabled logging settings, they're set on a higher level
logger.disable('auto_archiver')
return
# setup loguru logging
logger.remove(0) # remove the default logger
logging_config = self.config['logging']
logger.add(sys.stderr, level=logging_config['level'])
if log_file := logging_config['file']:
logger.add(log_file) if not logging_config['rotation'] else logger.add(log_file, rotation=logging_config['rotation'])
try:
logger.remove(0) # remove the default logger
except ValueError:
pass
# add other logging info
if self.logger_id is None: # note - need direct comparison to None since need to consider falsy value 0
self.logger_id = logger.add(sys.stderr, level=logging_config['level'])
if log_file := logging_config['file']:
logger.add(log_file) if not logging_config['rotation'] else logger.add(log_file, rotation=logging_config['rotation'])
def install_modules(self, modules_by_type):
"""
@@ -246,59 +264,38 @@ class ArchivingOrchestrator:
orchestrator's attributes (self.feeders, self.extractors etc.). If no modules of a certain type
are loaded, the program will exit with an error message.
"""
invalid_modules = []
for module_type in BaseModule.MODULE_TYPES:
for module_type in MODULE_TYPES:
step_items = []
modules_to_load = modules_by_type[f"{module_type}s"]
assert modules_to_load, f"No {module_type}s were configured. Make sure to set at least one {module_type} in your configuration file or on the command line (using --{module_type}s)"
if not modules_to_load:
raise SetupError(f"No {module_type}s were configured. Make sure to set at least one {module_type} in your configuration file or on the command line (using --{module_type}s)")
def check_steps_ok():
if not len(step_items):
logger.error(f"NO {module_type.upper()}S LOADED. Please check your configuration and try again.")
if len(modules_to_load):
logger.error(f"Tried to load the following modules, but none were available: {modules_to_load}")
exit()
logger.error(f"Unable to load any {module_type}s. Tried the following, but none were available: {modules_to_load}")
raise SetupError(f"NO {module_type.upper()}S LOADED. Please check your configuration and try again.")
if (module_type == 'feeder' or module_type == 'formatter') and len(step_items) > 1:
logger.error(f"Only one {module_type} is allowed, found {len(step_items)} {module_type}s. Please remove one of the following from your configuration file: {modules_to_load}")
exit()
raise SetupError(f"Only one {module_type} is allowed, found {len(step_items)} {module_type}s. Please remove one of the following from your configuration file: {modules_to_load}")
for module in modules_to_load:
if module == 'cli_feeder':
# pseudo module, don't load it
urls = self.config['urls']
if not urls:
logger.error("No URLs provided. Please provide at least one URL via the command line, or set up an alternative feeder. Use --help for more information.")
exit()
# cli_feeder is a pseudo module, it just takes the command line args
def feed(self) -> Generator[Metadata]:
for url in urls:
logger.debug(f"Processing URL: '{url}'")
yield Metadata().set_url(url)
pseudo_module = type('CLIFeeder', (Feeder,), {
'name': 'cli_feeder',
'display_name': 'CLI Feeder',
'__iter__': feed
})()
pseudo_module.__iter__ = feed
step_items.append(pseudo_module)
continue
if module in invalid_modules:
continue
loaded_module = None
try:
loaded_module: BaseModule = get_module(module, self.config)
loaded_module: BaseModule = self.module_factory.get_module(module, self.config)
except (KeyboardInterrupt, Exception) as e:
logger.error(f"Error during setup of modules: {e}\n{traceback.format_exc()}")
if module_type == 'extractor' and loaded_module.name == module:
if loaded_module and module_type == 'extractor':
loaded_module.cleanup()
exit()
raise e
if not loaded_module:
invalid_modules.append(module)
@@ -308,48 +305,107 @@ class ArchivingOrchestrator:
check_steps_ok()
setattr(self, f"{module_type}s", step_items)
def load_config(self, config_file: str) -> dict:
if not os.path.exists(config_file) and config_file != DEFAULT_CONFIG_FILE:
logger.error(f"The configuration file {config_file} was not found. Make sure the file exists and try again, or run without the --config file to use the default settings.")
exit()
raise FileNotFoundError(f"Configuration file {config_file} not found")
return read_yaml(config_file)
def setup_config(self, args: list) -> dict:
"""
Sets up the configuration file, merging the default config with the user's config
This function should only ever be run once.
"""
def run(self, args: list) -> None:
self.setup_basic_parser()
# parse the known arguments for now (basically, we want the config file)
basic_config, unused_args = self.basic_parser.parse_known_args(args)
# setup any custom module paths, so they'll show in the help and for arg parsing
setup_paths(basic_config.module_paths)
self.module_factory.setup_paths(basic_config.module_paths)
# if help flag was called, then show the help
if basic_config.help:
self.show_help(basic_config)
# merge command line --feeder etc. args with what's in the yaml config
yaml_config = self.load_config(basic_config.config_file)
self.setup_complete_parser(basic_config, yaml_config, unused_args)
return self.setup_complete_parser(basic_config, yaml_config, unused_args)
def check_for_updates(self):
response = requests.get("https://pypi.org/pypi/auto-archiver/json").json()
latest_version = response['info']['version']
# check version compared to current version
if latest_version != __version__:
if os.environ.get('RUNNING_IN_DOCKER'):
update_cmd = "`docker pull bellingcat/auto-archiver:latest`"
else:
update_cmd = "`pip install --upgrade auto-archiver`"
logger.warning("")
logger.warning("********* IMPORTANT: UPDATE AVAILABLE ********")
logger.warning(f"A new version of auto-archiver is available (v{latest_version}, you have {__version__})")
logger.warning(f"Make sure to update to the latest version using: {update_cmd}")
logger.warning("")
def setup(self, args: list):
"""
Function to configure all setup of the orchestrator: setup configs and load modules.
This method should only ever be called once
"""
self.check_for_updates()
if self.setup_finished:
logger.warning("The `setup_config()` function should only ever be run once. \
If you need to re-run the setup, please re-instantiate a new instance of the orchestrator. \
For code implementatations, you should call .setup_config() once then you may call .feed() \
multiple times to archive multiple URLs.")
return
self.setup_basic_parser()
self.config = self.setup_config(args)
logger.info(f"======== Welcome to the AUTO ARCHIVER ({__version__}) ==========")
self.install_modules(self.config['steps'])
# log out the modules that were loaded
for module_type in BaseModule.MODULE_TYPES:
for module_type in MODULE_TYPES:
logger.info(f"{module_type.upper()}S: " + ", ".join(m.display_name for m in getattr(self, f"{module_type}s")))
self.setup_finished = True
for _ in self.feed():
pass
def _command_line_run(self, args: list) -> Generator[Metadata]:
"""
This is the main entry point for the orchestrator, when run from the command line.
def cleanup(self)->None:
:param args: list of arguments to pass to the orchestrator - these are the command line args
You should not call this method from code implementations.
This method sets up the configuration, loads the modules, and runs the feed.
If you wish to make code invocations yourself, you should use the 'setup' and 'feed' methods separately.
To test configurations, without loading any modules you can also first call 'setup_configs'
"""
try:
self.setup(args)
return self.feed()
except Exception as e:
logger.error(e)
exit(1)
def cleanup(self) -> None:
logger.info("Cleaning up")
for e in self.extractors:
e.cleanup()
def feed(self) -> Generator[Metadata]:
url_count = 0
for feeder in self.feeders:
for item in feeder:
@@ -393,7 +449,6 @@ class ArchivingOrchestrator:
m.tmp_dir = None
tmp_dir.cleanup()
def archive(self, result: Metadata) -> Union[Metadata, None]:
"""
Runs the archiving process for a single URL
@@ -407,8 +462,8 @@ class ArchivingOrchestrator:
original_url = result.get_url().strip()
try:
self.assert_valid_url(original_url)
except AssertionError as e:
check_url_or_raise(original_url)
except ValueError as e:
logger.error(f"Error archiving URL {original_url}: {e}")
raise e
@@ -440,13 +495,13 @@ class ArchivingOrchestrator:
try:
result.merge(a.download(result))
if result.is_success(): break
except Exception as e:
except Exception as e:
logger.error(f"ERROR archiver {a.name}: {e}: {traceback.format_exc()}")
# 4 - call enrichers to work with archived content
for e in self.enrichers:
try: e.enrich(result)
except Exception as exc:
except Exception as exc:
logger.error(f"ERROR enricher {e.name}: {exc}: {traceback.format_exc()}")
# 5 - store all downloaded/generated media
@@ -468,30 +523,30 @@ class ArchivingOrchestrator:
logger.error(f"ERROR database {d.name}: {e}: {traceback.format_exc()}")
return result
def assert_valid_url(self, url: str) -> bool:
def setup_authentication(self, config: dict) -> dict:
"""
Blocks localhost, private, reserved, and link-local IPs and all non-http/https schemes.
Setup authentication for all modules that require it
Split up strings into multiple sites if they are comma separated
"""
assert url.startswith("http://") or url.startswith("https://"), f"Invalid URL scheme"
authentication = config.get('authentication', {})
# extract out concatenated sites
for key, val in copy(authentication).items():
if "," in key:
for site in key.split(","):
site = site.strip()
authentication[site] = val
del authentication[key]
parsed = urlparse(url)
assert parsed.scheme in ["http", "https"], f"Invalid URL scheme"
assert parsed.hostname, f"Invalid URL hostname"
assert parsed.hostname != "localhost", f"Invalid URL"
try: # special rules for IP addresses
ip = ip_address(parsed.hostname)
except ValueError: pass
else:
assert ip.is_global, f"Invalid IP used"
assert not ip.is_reserved, f"Invalid IP used"
assert not ip.is_link_local, f"Invalid IP used"
assert not ip.is_private, f"Invalid IP used"
config['authentication'] = authentication
return config
# Helper Properties
@property
def all_modules(self) -> List[Type[BaseModule]]:
return self.feeders + self.extractors + self.enrichers + self.databases + self.storages + self.formatters
return self.feeders + self.extractors + self.enrichers + self.databases + self.storages + self.formatters

75
src/auto_archiver/core/storage.py
View File

@@ -1,7 +1,29 @@
"""
Base module for Storage modules – modular components that store media objects in various locations.
If you are looking to implement a new storage module, you should subclass the `Storage` class and
implement the `get_cdn_url` and `uploadf` methods.
Your module **must** also have two config variables 'path_generator' and 'filename_generator' which
determine how the key is generated for the media object. The 'path_generator' and 'filename_generator'
variables can be set to one of the following values:
- 'flat': A flat structure with no subfolders
- 'url': A structure based on the URL of the media object
- 'random': A random structure
The 'filename_generator' variable can be set to one of the following values:
- 'random': A random string
- 'static': A replicable strategy such as a hash
If you don't want to use this naming convention, you can override the `set_key` method in your subclass.
"""
from __future__ import annotations
from abc import abstractmethod
from typing import IO
import os
import platform
from loguru import logger
from slugify import slugify
@@ -10,56 +32,85 @@ from auto_archiver.utils.misc import random_str
from auto_archiver.core import Media, BaseModule, Metadata
from auto_archiver.modules.hash_enricher.hash_enricher import HashEnricher
from auto_archiver.core.module import get_module
class Storage(BaseModule):
"""
Base class for implementing storage modules in the media archiving framework.
Subclasses must implement the `get_cdn_url` and `uploadf` methods to define their behavior.
"""
def store(self, media: Media, url: str, metadata: Metadata=None) -> None:
if media.is_stored(in_storage=self):
logger.debug(f"{media.key} already stored, skipping")
return
self.set_key(media, url, metadata)
self.upload(media, metadata=metadata)
media.add_url(self.get_cdn_url(media))
@abstractmethod
def get_cdn_url(self, media: Media) -> str: pass
def get_cdn_url(self, media: Media) -> str:
"""
Returns the URL of the media object stored in the CDN.
"""
pass
@abstractmethod
def uploadf(self, file: IO[bytes], key: str, **kwargs: dict) -> bool: pass
def uploadf(self, file: IO[bytes], key: str, **kwargs: dict) -> bool:
"""
Uploads (or saves) a file to the storage service/location.
This method should not be called directly, but instead through the 'store' method,
which sets up the media for storage.
"""
pass
def upload(self, media: Media, **kwargs) -> bool:
"""
Uploads a media object to the storage service.
This method should not be called directly, but instead be called through the 'store' method,
which sets up the media for storage.
"""
logger.debug(f'[{self.__class__.__name__}] storing file {media.filename} with key {media.key}')
with open(media.filename, 'rb') as f:
return self.uploadf(f, media, **kwargs)
def set_key(self, media: Media, url, metadata: Metadata) -> None:
def set_key(self, media: Media, url: str, metadata: Metadata) -> None:
"""takes the media and optionally item info and generates a key"""
if media.key is not None and len(media.key) > 0: return
if media.key is not None and len(media.key) > 0:
# media key is already set
return
folder = metadata.get_context('folder', '')
filename, ext = os.path.splitext(media.filename)
# Handle path_generator logic
path_generator = self.config.get("path_generator", "url")
path_generator = self.path_generator
if path_generator == "flat":
path = ""
filename = slugify(filename) # Ensure filename is slugified
elif path_generator == "url":
path = slugify(url)
path = slugify(url)[:70]
elif path_generator == "random":
path = self.config.get("random_path", random_str(24), True)
path = random_str(24)
else:
raise ValueError(f"Invalid path_generator: {path_generator}")
# Handle filename_generator logic
filename_generator = self.config.get("filename_generator", "random")
filename_generator = self.filename_generator
if filename_generator == "random":
filename = random_str(24)
elif filename_generator == "static":
# load the hash_enricher module
he = get_module(HashEnricher, self.config)
he = self.module_factory.get_module("hash_enricher", self.config)
hd = he.calculate_hash(media.filename)
filename = hd[:24]
else:
raise ValueError(f"Invalid filename_generator: {filename_generator}")
key = os.path.join(folder, path, f"{filename}{ext}")
media.key = os.path.join(folder, path, f"{filename}{ext}")
media._key = key

6
src/auto_archiver/core/validators.py
View File

@@ -1,6 +1,7 @@
# used as validators for config values. Should raise an exception if the value is invalid.
from pathlib import Path
import argparse
import json
def example_validator(value):
if "example" not in value:
@@ -16,4 +17,7 @@ def positive_number(value):
def valid_file(value):
if not Path(value).is_file():
raise argparse.ArgumentTypeError(f"File '{value}' does not exist.")
return value
return value
def json_loader(cli_val):
return json.loads(cli_val)

10
src/auto_archiver/modules/api_db/__manifest__.py
View File

@@ -1,5 +1,5 @@
{
"name": "Auto-Archiver API Database",
"name": "Auto Archiver API Database",
"type": ["database"],
"entry_point": "api_db::AAApiDb",
"requires_setup": True,
@@ -24,9 +24,9 @@
"help": "which group of users have access to the archive in case public=false as author",
},
"use_api_cache": {
"default": True,
"default": False,
"type": "bool",
"help": "if False then the API database will be queried prior to any archiving operations and stop if the link has already been archived",
"help": "if True then the API database will be queried prior to any archiving operations and stop if the link has already been archived",
},
"store_results": {
"default": True,
@@ -39,7 +39,7 @@
},
},
"description": """
Provides integration with the Auto-Archiver API for querying and storing archival data.
Provides integration with the Auto Archiver API for querying and storing archival data.
### Features
- **API Integration**: Supports querying for existing archives and submitting results.
@@ -49,6 +49,6 @@
- **Optional Storage**: Archives results conditionally based on configuration.
### Setup
Requires access to an Auto-Archiver API instance and a valid API token.
Requires access to an Auto Archiver API instance and a valid API token.
""",
}

1
src/auto_archiver/modules/atlos_db/__init__.py
View File

@@ -1 +0,0 @@
from atlos_db import AtlosDb

36
src/auto_archiver/modules/atlos_db/__manifest__.py
View File

@@ -1,36 +0,0 @@
{
"name": "Atlos Database",
"type": ["database"],
"entry_point": "atlos_db::AtlosDb",
"requires_setup": True,
"dependencies":
{"python": ["loguru",
""],
"bin": [""]},
"configs": {
"api_token": {
"default": None,
"help": "An Atlos API token. For more information, see https://docs.atlos.org/technical/api/",
},
"atlos_url": {
"default": "https://platform.atlos.org",
"help": "The URL of your Atlos instance (e.g., https://platform.atlos.org), without a trailing slash.",
"type": "str"
},
},
"description": """
Handles integration with the Atlos platform for managing archival results.
### Features
- Outputs archival results to the Atlos API for storage and tracking.
- Updates failure status with error details when archiving fails.
- Processes and formats metadata, including ISO formatting for datetime fields.
- Skips processing for items without an Atlos ID.
### Setup
Required configs:
- atlos_url: Base URL for the Atlos API.
- api_token: Authentication token for API access.
"""
,
}

66
src/auto_archiver/modules/atlos_db/atlos_db.py
View File

@@ -1,66 +0,0 @@
from typing import Union
import requests
from loguru import logger
from auto_archiver.core import Database
from auto_archiver.core import Metadata
class AtlosDb(Database):
"""
Outputs results to Atlos
"""
def failed(self, item: Metadata, reason: str) -> None:
"""Update DB accordingly for failure"""
# If the item has no Atlos ID, there's nothing for us to do
if not item.metadata.get("atlos_id"):
logger.info(f"Item {item.get_url()} has no Atlos ID, skipping")
return
requests.post(
f"{self.atlos_url}/api/v2/source_material/metadata/{item.metadata['atlos_id']}/auto_archiver",
headers={"Authorization": f"Bearer {self.api_token}"},
json={"metadata": {"processed": True, "status": "error", "error": reason}},
).raise_for_status()
logger.info(
f"Stored failure for {item.get_url()} (ID {item.metadata['atlos_id']}) on Atlos: {reason}"
)
def fetch(self, item: Metadata) -> Union[Metadata, bool]:
"""check and fetch if the given item has been archived already, each
database should handle its own caching, and configuration mechanisms"""
return False
def _process_metadata(self, item: Metadata) -> dict:
"""Process metadata for storage on Atlos. Will convert any datetime
objects to ISO format."""
return {
k: v.isoformat() if hasattr(v, "isoformat") else v
for k, v in item.metadata.items()
}
def done(self, item: Metadata, cached: bool = False) -> None:
"""archival result ready - should be saved to DB"""
if not item.metadata.get("atlos_id"):
logger.info(f"Item {item.get_url()} has no Atlos ID, skipping")
return
requests.post(
f"{self.atlos_url}/api/v2/source_material/metadata/{item.metadata['atlos_id']}/auto_archiver",
headers={"Authorization": f"Bearer {self.api_token}"},
json={
"metadata": dict(
processed=True,
status="success",
results=self._process_metadata(item),
)
},
).raise_for_status()
logger.info(
f"Stored success for {item.get_url()} (ID {item.metadata['atlos_id']}) on Atlos"
)

13
src/auto_archiver/modules/atlos_db/base_configs.py
View File

@@ -1,13 +0,0 @@
def get_atlos_config_options():
return {
"api_token": {
"default": None,
"help": "An Atlos API token. For more information, see https://docs.atlos.org/technical/api/",
"type": str
},
"atlos_url": {
"default": "https://platform.atlos.org",
"help": "The URL of your Atlos instance (e.g., https://platform.atlos.org), without a trailing slash.",
"type": str
},
}

1
src/auto_archiver/modules/atlos_feeder/__init__.py
View File

@@ -1 +0,0 @@
from .atlos_feeder import AtlosFeeder

34
src/auto_archiver/modules/atlos_feeder/__manifest__.py
View File

@@ -1,34 +0,0 @@
{
"name": "Atlos Feeder",
"type": ["feeder"],
"requires_setup": True,
"dependencies": {
"python": ["loguru", "requests"],
},
"configs": {
"api_token": {
"type": "str",
"required": True,
"help": "An Atlos API token. For more information, see https://docs.atlos.org/technical/api/",
},
"atlos_url": {
"default": "https://platform.atlos.org",
"help": "The URL of your Atlos instance (e.g., https://platform.atlos.org), without a trailing slash.",
"type": "str"
},
},
"description": """
AtlosFeeder: A feeder module that integrates with the Atlos API to fetch source material URLs for archival.
### Features
- Connects to the Atlos API to retrieve a list of source material URLs.
- Filters source materials based on visibility, processing status, and metadata.
- Converts filtered source materials into `Metadata` objects with the relevant `atlos_id` and URL.
- Iterates through paginated results using a cursor for efficient API interaction.
### Notes
- Requires an Atlos API endpoint and a valid API token for authentication.
- Ensures only unprocessed, visible, and ready-to-archive URLs are returned.
- Handles pagination transparently when retrieving data from the Atlos API.
"""
}

42
src/auto_archiver/modules/atlos_feeder/atlos_feeder.py
View File

@@ -1,42 +0,0 @@
import requests
from loguru import logger
from auto_archiver.core import Feeder
from auto_archiver.core import Metadata
class AtlosFeeder(Feeder):
def __iter__(self) -> Metadata:
# Get all the urls from the Atlos API
count = 0
cursor = None
while True:
response = requests.get(
f"{self.atlos_url}/api/v2/source_material",
headers={"Authorization": f"Bearer {self.api_token}"},
params={"cursor": cursor},
)
data = response.json()
response.raise_for_status()
cursor = data["next"]
for item in data["results"]:
if (
item["source_url"] not in [None, ""]
and (
item["metadata"]
.get("auto_archiver", {})
.get("processed", False)
!= True
)
and item["visibility"] == "visible"
and item["status"] not in ["processing", "pending"]
):
yield Metadata().set_url(item["source_url"]).set(
"atlos_id", item["id"]
)
count += 1
if len(data["results"]) == 0 or cursor is None:
break

1
src/auto_archiver/modules/atlos_feeder_db_storage/__init__.py Normal file
View File

@@ -0,0 +1 @@
from .atlos_feeder_db_storage import AtlosFeederDbStorage

Some files were not shown because too many files have changed in this diff Show More