Commit Graph

51 Commits

Author SHA1 Message Date
msramalho
4941823565 fix growing volume size in wacz_enricher 2024-02-21 17:25:55 +00:00
Miguel Sozinho Ramalho
7a21ae96af V0.9.0 - closes several open issues: new enrichers and bug fixes (#133)
* clean orchestrator code, add archiver cleanup logic

* improves documentation for database.py

* telethon archivers isolate sessions into copied files

* closes #127

* closes #125

* closes #84

* meta enricher applies to all media

* closes #61 adds subtitles and comments

* minor update

* minor fixes to yt-dlp subtitles and comments

* closes #17 but logic is imperfect.

* closes #85 ssl enhancer

* minimifies html, JS refactor for preview of certificates

* closes #91 adds freetsa timestamp authority

* version bump

* simplify download_url method

* skip ssl if nothing archived

* html preview improvements

* adds retrying lib

* manual download archiver improvements

* meta only runs when relevant data available

* new metadata convenience method

* html template improvements

* removes debug message

* does not close #91 yet, will need a few more certificate chaing logging

* adds verbosity config

* new instagram api archiver

* adds proxy support we

* adds proxy/end support and bug fix for yt-dlp

* proxy support for webdriver

* adds socks proxy to wacz_enricher

* refactor recursivity in inner media and display

* infinite recursive display

* foolproofing timestamping authortities

* version to 0.9.0

* minor fixes from code-review
2024-02-20 18:05:29 +00:00
Miguel Sozinho Ramalho
e6b6b83007 0.8.0 new features and dependency updates (#119)
* wacz can extract_screenshot only

* new meta enricher

* twitter api can use multiple authentication tokens in sequence

* cleanup non-dup logic

* meta info on archive duration

* minor html report update

* updated dependencies

* new version
2023-12-20 14:13:22 +00:00
Miguel Sozinho Ramalho
3e56ef137d reduce s3 duplicating while keeping random urls via hash (#112) 2023-12-12 19:12:03 +00:00
Kai
9eb39943c7 Extract text in wacz_enricher (#110) 2023-12-05 22:24:12 +00:00
msramalho
b157f9a6b1 renaming variable 2023-09-15 19:52:47 +01:00
msramalho
ea38a604bb fixes #96 by not assigning to self.prop 2023-09-15 19:35:35 +01:00
Kai
f7839a99cc Add configs for path to write and read wacz archives (#93)
Co-authored-by: msramalho <19508417+msramalho@users.noreply.github.com>
2023-09-14 17:49:37 +01:00
Miguel Sozinho Ramalho
3ae25e51e7 adds flexibile setup for wacz in docker (#94) 2023-09-12 20:07:21 +01:00
msramalho
0dd45d90f1 fix: docker+wacz troubles 2023-09-08 15:09:50 +01:00
msramalho
b17b5953dd closes #59 2023-08-17 18:11:58 +01:00
msramalho
810a31b1f0 fix: whisper handle error http code 2023-08-08 18:06:48 +01:00
msramalho
8b15d733b1 adds whisper endpoints 2023-08-05 14:03:57 +01:00
msramalho
a1742b5565 fixing whisper enricher 2023-08-05 13:57:09 +01:00
msramalho
60a1f3a27a minor fixes 2023-07-31 16:08:48 +01:00
msramalho
bd231488ff parameter fix 2023-07-28 13:10:06 +01:00
msramalho
139bdec051 excludes files from perceptual hash 2023-07-28 12:51:24 +01:00
msramalho
f15a70f859 missing hash_enricher import 2023-07-28 12:51:04 +01:00
msramalho
1695954c98 new metadata enricher 2023-07-28 12:46:30 +01:00
msramalho
aa71c85a98 improving ignored content from waczs 2023-07-28 12:19:14 +01:00
msramalho
7a5c9c65bd detects duplicates before storing, eg: wacz getting media already fetched by another archiver 2023-07-28 10:51:48 +01:00
msramalho
e3a0003a47 adding WACZ screenshots 2023-07-27 21:36:25 +01:00
msramalho
e8f44b652e minor improvements 2023-07-27 15:42:23 +01:00
msramalho
dd034da844 feat: WACZ enricher can now be probed for media, and used as an archiver OR enricher 2023-07-27 15:42:10 +01:00
msramalho
a0971fc601 final code review changes 2023-06-26 17:32:19 +01:00
msramalho
3bbfdf6eba fix: excluding screenshots 2023-06-26 17:27:49 +01:00
msramalho
b4f86d0e8d refactor to hash all images and save hex string 2023-06-26 17:06:30 +01:00
Emiel de Heij
9fc09c724b add module for perceptual hashing with pdq 2023-06-26 15:25:55 +02:00
Logan Williams
c47da0a46f Fix issue with profiles in browsertrix 2023-05-11 15:08:27 +02:00
msramalho
e11be449e8 fix: delete completed whisper tasks 2023-05-10 18:57:17 +01:00
Logan Williams
ac82764ffc Working, but some cleanup still necessary 2023-05-09 19:34:16 +02:00
Logan Williams
0fae7d96fb Detect running in docker container in WACZ enricher 2023-05-09 19:34:16 +02:00
msramalho
875e1de589 feat: re-enable HASH on gsheet 2023-05-09 11:17:44 +01:00
msramalho
8f3d4e05c3 fixing bug in whisper wnericher 2023-05-04 09:36:10 +01:00
msramalho
5b0bff612e whisper transcripts to content 2023-05-02 19:05:32 +01:00
msramalho
5fdaa6c739 whisper improvements 2023-04-18 19:28:36 +01:00
msramalho
3d389ee05b add url info 2023-04-18 19:14:47 +01:00
msramalho
69bcfea2eb to_json fix 2023-04-18 18:48:51 +01:00
msramalho
2e2e695444 whisper enricher 2023-03-23 18:50:37 +00:00
msramalho
493055a8d9 cleanup 2023-03-23 18:50:30 +00:00
msramalho
906ed0f6e0 creating global context and refactoring tmp_dir logic 2023-03-23 11:17:38 +00:00
msramalho
0654e8c5c6 hash calculation in chunks to avoid exhausting RAM 2023-03-10 11:34:29 +00:00
msramalho
0e3c427371 Bump version to v0.4.3 for release 2023-02-27 10:30:06 +01:00
msramalho
cd81cae559 auth wall for WACZ 2023-02-20 16:08:45 +00:00
msramalho
5505255ea3 url auth wall detect 2023-02-17 15:45:58 +00:00
msramalho
e758bd076b test 2023-02-02 12:43:23 +00:00
msramalho
9bcca427a0 wacz in gsheets 2023-02-02 12:41:06 +00:00
msramalho
39bfde2026 thumbnails bug fix 2023-02-01 00:35:48 +00:00
msramalho
d1e4dde3f6 fixing imports 2023-01-27 00:19:58 +00:00
msramalho
b763fc4188 final naming cleanup + new feeders/dbs 2023-01-21 19:44:12 +00:00