Start fleshing out the docs more - rearrange, separate out modules section, move files over to md (from rst)

docs/source/conf.py

@@ -17,6 +17,7 @@ extensions = [
     'sphinxcontrib.mermaid',        # Mermaid diagrams
     "sphinx.ext.viewcode",          # Source code links
     "sphinx.ext.napoleon",          # Google-style and NumPy-style docstrings
+    "sphinx.ext.autosectionlabel",
     # "sphinx.ext.autodoc",           # Include custom docstrings
     # 'sphinx.ext.autosummary',       # Summarize module/class/function docs
 ]
@@ -27,12 +28,13 @@ exclude_patterns = []
 
 # -- AutoAPI Configuration ---------------------------------------------------
 autoapi_type = 'python'
-autoapi_dirs = ["../../src"]
+autoapi_dirs = ["../../src/auto_archiver/core/", "../../src/auto_archiver/utils/", "../../src/auto_archiver/modules/"]
 autodoc_typehints = "signature"     # Include type hints in the signature
-autoapi_ignore = []                 # Ignore specific modules
-autoapi_keep_files = True          # Option to retain intermediate JSON files for debugging
+autoapi_ignore = ["*/version.py", ]                 # Ignore specific modules
+autoapi_keep_files = False          # Option to retain intermediate JSON files for debugging
 autoapi_add_toctree_entry = True    # Include API docs in the TOC
-autoapi_template_dir = None         # Use default templates
+autoapi_python_use_implicit_namespaces = True
+autoapi_template_dir = "../_templates/autoapi"
 autoapi_options = [
     "members",
     "undoc-members",
@@ -53,6 +55,8 @@ myst_enable_extensions = [
     "linkify",              # Auto-detect links
     "substitution",         # Text substitutions
 ]
+myst_heading_anchors = 2
+
 source_suffix = {
     ".rst": "restructuredtext",
     ".md": "markdown",

docs/source/core_modules.rst

@@ -0,0 +1,11 @@
+Core Modules
+============
+
+These pages are intended for developers of the `auto-archiver` package, and include documentation on the core classes and functions used by the auto-archiver
+
+.. toctree::
+   :titlesonly:
+
+   {% for page in pages|selectattr("is_top_level_object") %}
+   {{ page.include_path }}
+   {% endfor %}

docs/source/developer_guidelines.rst

@@ -1,6 +0,0 @@
-
-Developer Guidelines
-====================
-
-This section of the documentation provides guidelines for developers who want to modify or contribute to the tool.
-

docs/source/development/developer_guidelines.md

@@ -0,0 +1,34 @@
+
+# Developer Guidelines
+
+This section of the documentation provides guidelines for developers who want to modify or contribute to the tool.
+
+
+## Developer Install
+
+1. Clone the project using `git clone https://github.com/bellingcat/auto-archiver.git` 
+2. Install poetry using `curl -sSL https://install.python-poetry.org | python3 -` ([other installation methods](https://python-poetry.org/docs/#installation))
+3. Install dependencies with `poetry install`
+
+## Running 
+4. Run the code with `poetry run auto-archiver [my args]`
+
+```{note}
+Add the plugin [poetry-shell-plugin](https://github.com/python-poetry/poetry-plugin-shell) and run `poetry shell` to activate the virtual environment.
+This allows you to run the auto-archiver without the `poetry run` prefix.
+```
+
+### Optional Development Packages
+
+Install development packages (used for unit tests etc.) using:
+`poetry install -with dev`
+
+
+```{toctree}
+:hidden:
+
+docker_development
+testing
+docs
+release
+```

docs/source/development/docker_development.md

@@ -0,0 +1,5 @@
+## Docker development
+working with docker locally:
+  * `docker compose up` to build the first time and run a local image with the settings in `secrets/orchestration.yaml`
+  * To modify/pass additional command line args, use `docker compose run auto-archiver --config secrets/orchestration.yaml [OTHER ARGUMENTS]`
+  * To rebuild after code changes, just pass the `--build` flag, e.g. `docker compose up --build`

docs/source/development/docs.md

@@ -0,0 +1,38 @@
+
+### Building the Docs
+
+The documentation is built using [Sphinx](https://www.sphinx-doc.org/en/master/) and [AutoAPI](https://sphinx-autoapi.readthedocs.io/en/latest/) and hosted on ReadTheDocs.
+To build the documentation locally, run the following commands:
+
+**Install required dependencies:**
+- Install the docs group of dependencies: 
+```shell
+# only the docs dependencies
+poetry install --only docs
+
+# or for all dependencies 
+poetry install
+```
+- Either use [poetry-plugin-shell](https://github.com/python-poetry/poetry-plugin-shell) to activate the virtual environment: `poetry shell`
+- Or prepend the following commands with `poetry run`
+
+**Create the documentation:**
+- Build the documentation: 
+```
+# Using makefile (Linux/macOS):
+make -C docs html
+
+# or using sphinx directly (Windows/Linux/macOS):
+sphinx-build -b html docs/source docs/_build/html
+```
+- If you make significant changes and want a fresh build run: `make -C docs clean` to remove the old build files.
+
+**Viewing the documentation:**
+```shell
+# to open the documentation in your browser.
+open docs/_build/html/index.html
+
+# or run autobuild to automatically update the documentation when you make changes
+sphinx-autobuild docs/source docs/_build/html
+```
+

docs/source/development/release.md

@@ -0,0 +1,15 @@
+# Release Process
+
+```{note} This is a work in progress.
+```
+
+1. Update the version number in [version.py](src/auto_archiver/version.py)
+2. Go to github releases > new release > use `vx.y.z` for matching version notation
+   1. package is automatically updated in pypi
+   2. docker image is automatically pushed to dockerhup
+
+
+
+manual release to docker hub
+  * `docker image tag auto-archiver bellingcat/auto-archiver:latest`
+  * `docker push bellingcat/auto-archiver`

docs/source/development/testing.md

@@ -0,0 +1,13 @@
+### Testing
+
+Tests are split using `pytest.mark` into 'core' and 'download' tests. Download tests will hit the network and make API calls (e.g. Twitter, Bluesky etc.) and should be run regularly to make sure that APIs have not changed.
+
+Tests can be run as follows:
+```
+# run core tests
+pytest -ra -v -m "not download" # or poetry run pytest -ra -v -m "not download"
+# run download tests
+pytest -ra -v -m "download" # or poetry run pytest -ra -v -m "download"
+# run all tests
+pytest -ra -v # or poetry run pytest -ra -v
+```

docs/source/index.rst

@@ -1,26 +1,18 @@
-.. auto-archiver documentation master file, created by
-   sphinx-quickstart on Sun Jan 12 20:35:50 2025.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
-
-Auto Archiver documentation
-===========================
-
-.. note::
-
-   This is a work in progress.
-
 
 .. include:: ../../README.md
    :parser: myst
 
 
 .. toctree::
-   :maxdepth: 1
+
+   :maxdepth: 2
    :hidden:
    :caption: Contents:
 
+   Overview <self>
    user_guidelines
-   developer_guidelines
-   configurations
+   installation/installation
+   development/developer_guidelines
+   autoapi/index
+   core_modules
 

docs/source/installation/configurations.rst

@@ -2,6 +2,13 @@
 Configurations
 ==============
 
+
+```{toctree}
+:hidden:
+
+configurations
+```
+
 This section of the documentation provides guidelines for configuring the tool.
 
 File Reference

docs/source/installation/installation.md

@@ -0,0 +1,54 @@
+# Installing Auto Archiver
+
+
+There are 3  main ways to use the auto-archiver:
+1. Easiest: [via docker](#installing-with-docker)
+2. Local Install: [using pip](#local-installing-with-pip)
+3. Developer Install: [see the developer guidelines](../development/developer_guidelines)
+
+
+But **you always need a configuration/orchestration file**, which is where you'll configure where/what/how to archive. Make sure you read [orchestration](#orchestration).
+
+
+## Installing with Docker
+
+[![dockeri.co](https://dockerico.blankenship.io/image/bellingcat/auto-archiver)](https://hub.docker.com/r/bellingcat/auto-archiver)
+
+Docker works like a virtual machine running inside your computer, it isolates everything and makes installation simple. Since it is an isolated environment when you need to pass it your orchestration file or get downloaded media out of docker you will need to connect folders on your machine with folders inside docker with the `-v` volume flag.
+
+
+1. install [docker](https://docs.docker.com/get-docker/)
+2. pull the auto-archiver docker [image](https://hub.docker.com/r/bellingcat/auto-archiver) with `docker pull bellingcat/auto-archiver`
+3. run the docker image locally in a container: `docker run --rm -v $PWD/secrets:/app/secrets -v $PWD/local_archive:/app/local_archive bellingcat/auto-archiver --config secrets/orchestration.yaml` breaking this command down:
+   1. `docker run` tells docker to start a new container (an instance of the image)
+   2. `--rm` makes sure this container is removed after execution (less garbage locally)
+   3. `-v $PWD/secrets:/app/secrets` - your secrets folder
+      1. `-v` is a volume flag which means a folder that you have on your computer will be connected to a folder inside the docker container
+      2. `$PWD/secrets` points to a `secrets/` folder in your current working directory (where your console points to), we use this folder as a best practice to hold all the secrets/tokens/passwords/... you use
+      3. `/app/secrets` points to the path the docker container where this image can be found
+   4.  `-v $PWD/local_archive:/app/local_archive` - (optional) if you use local_storage
+       1.  `-v` same as above, this is a volume instruction
+       2.  `$PWD/local_archive` is a folder `local_archive/` in case you want to archive locally and have the files accessible outside docker
+       3.  `/app/local_archive` is a folder inside docker that you can reference in your orchestration.yml file 
+
+## Local installing with Pip
+
+1. make sure you have python 3.10 or higher installed
+2. install the package with your preferred package manager: `pip/pipenv/conda install auto-archiver` or `poetry add auto-archiver`
+3. test it's installed with `auto-archiver --help`
+4. run it with your orchestration file and pass any flags you want in the command line `auto-archiver --config secrets/orchestration.yaml` if your orchestration file is inside a `secrets/`, which we advise
+
+### Installing Local Requirements
+
+If using the local installation method, you will also need to install the following dependencies locally:
+
+1. [ffmpeg](https://www.ffmpeg.org/) must also be installed locally for this tool to work. 
+2. [firefox](https://www.mozilla.org/en-US/firefox/new/) and [geckodriver](https://github.com/mozilla/geckodriver/releases) on a path folder like `/usr/local/bin`. 
+3. (optional) [fonts-noto](https://fonts.google.com/noto) to deal with multiple unicode characters during selenium/geckodriver's screenshots: `sudo apt install fonts-noto -y`.
+
+
+
+
+## Developer Install
+
+[See the developer guidelines](../development/developer_guidelines)

docs/source/user_guidelines.md

@@ -0,0 +1,12 @@
+
+# User Guidelines
+
+
+This section of the documentation provides guidelines for users who want to use the tool,
+without needing to modify the code.
+To see the developer guidelines, see the [](development/developer_guidelines)
+
+```{note}
+This is a work in progress.
+```
+    

docs/source/user_guidelines.rst

@@ -1,11 +0,0 @@
-
-User Guidelines
-===============
-
-This section of the documentation provides guidelines for users who want to use the tool,
-without needing to modify the code.
-To see the developer guidelines, see :ref:`developer_guidelines`.
-
-.. note::
-
-    This is a work in progress.