Start fleshing out the docs more - rearrange, separate out modules section, move files over to md (from rst)

docs/source/development/developer_guidelines.md

@@ -0,0 +1,34 @@
+
+# Developer Guidelines
+
+This section of the documentation provides guidelines for developers who want to modify or contribute to the tool.
+
+
+## Developer Install
+
+1. Clone the project using `git clone https://github.com/bellingcat/auto-archiver.git` 
+2. Install poetry using `curl -sSL https://install.python-poetry.org | python3 -` ([other installation methods](https://python-poetry.org/docs/#installation))
+3. Install dependencies with `poetry install`
+
+## Running 
+4. Run the code with `poetry run auto-archiver [my args]`
+
+```{note}
+Add the plugin [poetry-shell-plugin](https://github.com/python-poetry/poetry-plugin-shell) and run `poetry shell` to activate the virtual environment.
+This allows you to run the auto-archiver without the `poetry run` prefix.
+```
+
+### Optional Development Packages
+
+Install development packages (used for unit tests etc.) using:
+`poetry install -with dev`
+
+
+```{toctree}
+:hidden:
+
+docker_development
+testing
+docs
+release
+```

docs/source/development/docker_development.md

@@ -0,0 +1,5 @@
+## Docker development
+working with docker locally:
+  * `docker compose up` to build the first time and run a local image with the settings in `secrets/orchestration.yaml`
+  * To modify/pass additional command line args, use `docker compose run auto-archiver --config secrets/orchestration.yaml [OTHER ARGUMENTS]`
+  * To rebuild after code changes, just pass the `--build` flag, e.g. `docker compose up --build`

docs/source/development/docs.md

@@ -0,0 +1,38 @@
+
+### Building the Docs
+
+The documentation is built using [Sphinx](https://www.sphinx-doc.org/en/master/) and [AutoAPI](https://sphinx-autoapi.readthedocs.io/en/latest/) and hosted on ReadTheDocs.
+To build the documentation locally, run the following commands:
+
+**Install required dependencies:**
+- Install the docs group of dependencies: 
+```shell
+# only the docs dependencies
+poetry install --only docs
+
+# or for all dependencies 
+poetry install
+```
+- Either use [poetry-plugin-shell](https://github.com/python-poetry/poetry-plugin-shell) to activate the virtual environment: `poetry shell`
+- Or prepend the following commands with `poetry run`
+
+**Create the documentation:**
+- Build the documentation: 
+```
+# Using makefile (Linux/macOS):
+make -C docs html
+
+# or using sphinx directly (Windows/Linux/macOS):
+sphinx-build -b html docs/source docs/_build/html
+```
+- If you make significant changes and want a fresh build run: `make -C docs clean` to remove the old build files.
+
+**Viewing the documentation:**
+```shell
+# to open the documentation in your browser.
+open docs/_build/html/index.html
+
+# or run autobuild to automatically update the documentation when you make changes
+sphinx-autobuild docs/source docs/_build/html
+```
+

docs/source/development/release.md

@@ -0,0 +1,15 @@
+# Release Process
+
+```{note} This is a work in progress.
+```
+
+1. Update the version number in [version.py](src/auto_archiver/version.py)
+2. Go to github releases > new release > use `vx.y.z` for matching version notation
+   1. package is automatically updated in pypi
+   2. docker image is automatically pushed to dockerhup
+
+
+
+manual release to docker hub
+  * `docker image tag auto-archiver bellingcat/auto-archiver:latest`
+  * `docker push bellingcat/auto-archiver`

docs/source/development/testing.md

@@ -0,0 +1,13 @@
+### Testing
+
+Tests are split using `pytest.mark` into 'core' and 'download' tests. Download tests will hit the network and make API calls (e.g. Twitter, Bluesky etc.) and should be run regularly to make sure that APIs have not changed.
+
+Tests can be run as follows:
+```
+# run core tests
+pytest -ra -v -m "not download" # or poetry run pytest -ra -v -m "not download"
+# run download tests
+pytest -ra -v -m "download" # or poetry run pytest -ra -v -m "download"
+# run all tests
+pytest -ra -v # or poetry run pytest -ra -v
+```