70 Commits

Author SHA1 Message Date
msramalho
8a4a314cf9 ruff python version to dev version 2026-02-23 12:32:24 +00:00
msramalho
75e8b788ae revert ruff workflow changes 2026-02-23 12:31:20 +00:00
dependabot[bot]
bf4c196cc2 Bump the actions group with 5 updates
Bumps the actions group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [actions/checkout](https://github.com/actions/checkout) | `4` | `6` |
| [docker/login-action](https://github.com/docker/login-action) | `3.4.0` | `3.7.0` |
| [docker/metadata-action](https://github.com/docker/metadata-action) | `5.7.0` | `5.10.0` |
| [actions/setup-python](https://github.com/actions/setup-python) | `5` | `6` |
| [actions/cache](https://github.com/actions/cache) | `4` | `5` |


Updates `actions/checkout` from 4 to 6
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v6)

Updates `docker/login-action` from 3.4.0 to 3.7.0
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](74a5d14239...c94ce9fb46)

Updates `docker/metadata-action` from 5.7.0 to 5.10.0
- [Release notes](https://github.com/docker/metadata-action/releases)
- [Commits](902fa8ec7d...c299e40c65)

Updates `actions/setup-python` from 5 to 6
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v5...v6)

Updates `actions/cache` from 4 to 5
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: docker/login-action
  dependency-version: 3.7.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: docker/metadata-action
  dependency-version: 5.10.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: actions/setup-python
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: actions/cache
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-02-01 20:17:43 +00:00
msramalho
c2c9718f73 make python api tests work on gh when no env is set 2025-06-30 02:20:51 +01:00
msramalho
499c272260 dependabot switch to monthly 2025-06-10 16:37:52 +01:00
msramalho
b3183510ea installs ffmpeg in GH actions 2025-06-07 20:03:26 +01:00
erinhmclark
9c2b506189 update runner os to matrix os. 2025-03-31 12:00:24 +01:00
erinhmclark
8940580638 Add poetry cache clear, and small code change 2025-03-31 11:41:26 +01:00
erinhmclark
c2821d7c83 Fix poetry install deletion 2025-03-31 11:25:51 +01:00
erinhmclark
1edfdae03e Update download tests to match cache process. 2025-03-31 11:17:40 +01:00
erinhmclark
6c7f6af4b4 Add cache action with key to OS, py version and lockfile hash, and install packages from source. 2025-03-31 11:11:56 +01:00
Patrick Robertson
2ec494b4b9 Revert downloads CI tests changes
It wasn't properly being triggered after the core tests. this reverts so that the download tests just run whatever
2025-03-28 17:55:58 +04:00
Patrick Robertson
48fbfc3b86 Merge pull request #280 from bellingcat/download-tests
Download tests
2025-03-28 13:33:30 +04:00
Patrick Robertson
46a51cce11 Fix up tests-download to properly run once core tests completed 2025-03-27 15:18:58 +04:00
dependabot[bot]
1612fef59b Bump the actions group with 3 updates
Bumps the actions group with 3 updates: [actions/checkout](https://github.com/actions/checkout), [docker/login-action](https://github.com/docker/login-action) and [docker/metadata-action](https://github.com/docker/metadata-action).


Updates `actions/checkout` from 3 to 4
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v3...v4)

Updates `docker/login-action` from 3.3.0 to 3.4.0
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](9780b0c442...74a5d14239)

Updates `docker/metadata-action` from 5.6.1 to 5.7.0
- [Release notes](https://github.com/docker/metadata-action/releases)
- [Commits](369eb591f4...902fa8ec7d)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: docker/login-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: docker/metadata-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-26 12:09:25 +00:00
Patrick Robertson
e9ee4d67ba Re-add 'ubuntu-latest' - now that we're rid of tsp_client 2025-03-26 15:29:36 +04:00
Patrick Robertson
4cfa6455c7 Only make the downloads action run if the core action was successful 2025-03-26 15:07:57 +04:00
Patrick Robertson
ace97ac7fd Don't run ruff on non-python file changes 2025-03-24 18:00:14 +04:00
Patrick Robertson
ad373ae733 Add explicit dependabots for pip/poetry, GH actiona and npm 2025-03-24 17:57:53 +04:00
Miguel Sozinho Ramalho
7b454baa02 Create dependabot.yml 2025-03-24 10:49:36 +00:00
erinhmclark
72f48f0147 Fix merge conflicts. 2025-03-14 12:11:24 +00:00
erinhmclark
4af3cd7b2a Revert ruff to separate commands. 2025-03-13 21:47:09 +00:00
erinhmclark
c7c24fbaf2 Update style_guide.md to clarify pre-commit setup, add Docker commands to Makefile and merge ruff actions. 2025-03-13 20:26:29 +00:00
erinhmclark
85abe1837a Ruff format with defaults. 2025-03-10 18:44:54 +00:00
erinhmclark
b386ae6287 Add poetry.lock and pyproject.toml paths to trigger tests. 2025-03-07 18:01:08 +00:00
erinhmclark
4949e9bcd2 Update docker-publish.yaml to use docker registry. 2025-03-07 15:34:56 +00:00
erinhmclark
3877b538be Update docker-publish.yaml to use docker registry. 2025-03-07 15:31:25 +00:00
erinhmclark
87ab98c270 Update docker/build-push version 2025-03-07 15:14:20 +00:00
erinhmclark
7bbf0da0d1 Add cache-from and cache-to to docker-publish.yaml. 2025-03-07 15:09:10 +00:00
Miguel Sozinho Ramalho
5614af3f63 removes fixed oscrypto dependency, it blocked pypi publishing (#195)
* disables tests on ubuntu-latest

* drops fixed oscrypto version for git commit

* version bump
2025-02-14 10:51:56 +00:00
Patrick Robertson
e8138eac1c Add ubuntu-latest to the matrix of test runners (#181)
* Don't clutter logs with info about generic dropin

* Add ubuntu-latest to unit tests

This is currently failing due to an issue with oscrypto and newer openssl https://github.com/wbond/oscrypto/issues/78#issuecomment-1756317472

* fix oscrypto version for ubuntu 24 compatibility (boto3 too see #180)

---------

Co-authored-by: msramalho <19508417+msramalho@users.noreply.github.com>
2025-01-24 14:03:55 +00:00
Patrick Robertson
6388983815 Merge branch 'main' into youtubedlp-rewrite 2025-01-21 16:43:14 +01:00
Patrick Robertson
5aa717452e Quick test that the app actually runs in core tests 2025-01-17 17:02:54 +01:00
Patrick Robertson
59eb8f7520 Add TWITTER_BEARER_TOKEN to env for running download tests 2025-01-17 12:04:40 +01:00
Patrick Robertson
394bcd8d47 Further refactoring of youtubedl_archiver->base_archiver
* Keep twitter_api_archiver
* Remove unit tests for obsolete archivers
* Guess filename of media using the 'Content-Type' header
* Add mechanism to run 'expensive' tests last (see conftest.py) and also flag expensive tests to fail straight off (pytest.mark.incremental)
2025-01-17 11:56:08 +01:00
erinhmclark
33686ea851 Update versions for GH Actions and Geckodriver. 2025-01-15 17:35:42 +00:00
Patrick Robertson
8f17a235f3 Switch to ubuntu-22.04 for CI tests
An issue with oscrypto means it currently does not work on 24.04. Ref: https://github.com/wbond/oscrypto/issues/78#issuecomment-2565688091
2025-01-14 10:24:14 +01:00
Patrick Robertson
ab2eb3c7f5 Add dev dependencies to poetry 2025-01-13 20:42:08 +01:00
Patrick Robertson
bdfedfcf61 Merge branch 'main' into feat/unittest 2025-01-13 19:50:47 +01:00
erinhmclark
84ee1b422f Update and restrict versions of Poetry and Python. 2025-01-13 17:42:51 +00:00
Patrick Robertson
b9aea99de8 Prettify pytest output 2025-01-13 18:41:24 +01:00
Patrick Robertson
9b596e59d6 Run expensive download tests once per week, on a month at 2:35pm
(time is offset from the hour to alleviate high load on Github
2025-01-13 18:33:02 +01:00
Patrick Robertson
bbef80de4c Add unit tests for html_formatter, csv_db 2025-01-13 17:58:10 +01:00
Patrick Robertson
2353f9d6a5 Separate CI for download tests and core tests 2025-01-13 16:27:46 +01:00
erinhmclark
4a63ca7753 Update PyPi workflow to read python version from pyproject.toml. 2025-01-12 12:15:56 +00:00
erinhmclark
dba8f46016 Replaced comments for python-publish.yaml workflow. 2025-01-12 12:15:56 +00:00
erinhmclark
50e8c93477 Updated workflow for python-publish.yaml to use poetry (untested), and cleanup of pipenv files. 2025-01-12 12:15:56 +00:00
Patrick Robertson
9dc4eb35de Switch to pytest and use vcr for request storing 2025-01-08 11:25:13 +01:00
Patrick Robertson
1bd017000e Add Github CI test workflow 2024-12-31 15:20:33 +01:00
Miguel Sozinho Ramalho
7a21ae96af V0.9.0 - closes several open issues: new enrichers and bug fixes (#133)
* clean orchestrator code, add archiver cleanup logic

* improves documentation for database.py

* telethon archivers isolate sessions into copied files

* closes #127

* closes #125

* closes #84

* meta enricher applies to all media

* closes #61 adds subtitles and comments

* minor update

* minor fixes to yt-dlp subtitles and comments

* closes #17 but logic is imperfect.

* closes #85 ssl enhancer

* minimifies html, JS refactor for preview of certificates

* closes #91 adds freetsa timestamp authority

* version bump

* simplify download_url method

* skip ssl if nothing archived

* html preview improvements

* adds retrying lib

* manual download archiver improvements

* meta only runs when relevant data available

* new metadata convenience method

* html template improvements

* removes debug message

* does not close #91 yet, will need a few more certificate chaing logging

* adds verbosity config

* new instagram api archiver

* adds proxy support we

* adds proxy/end support and bug fix for yt-dlp

* proxy support for webdriver

* adds socks proxy to wacz_enricher

* refactor recursivity in inner media and display

* infinite recursive display

* foolproofing timestamping authortities

* version to 0.9.0

* minor fixes from code-review
2024-02-20 18:05:29 +00:00