Commit Graph

34 Commits

Author SHA1 Message Date
msramalho
ce4d7ac649 WIP refactor logging 2025-06-21 15:54:51 +01:00
msramalho
33cca734d9 original_url changes still constitute empty result 2025-06-17 16:06:25 +01:00
erinhmclark
e7fa88f1c7 Implementing ruff suggestions. 2025-03-10 21:45:30 +00:00
erinhmclark
ca44a40b88 Ruff fix on src. 2025-03-10 19:03:45 +00:00
erinhmclark
85abe1837a Ruff format with defaults. 2025-03-10 18:44:54 +00:00
msramalho
7309cd32e7 fix: context to be updated on Metadata.merge 2025-02-11 12:51:17 +00:00
Patrick Robertson
c25d5cae84 Remove ArchivingContext completely
Context for a specific url/item is now passed around via the metadata (metadata.set_context('key', 'val') and metadata.get_context('key', default='something')
The only other thing that was passed around in ArchivingContext was the storage info, which is already accessible now via self.config
2025-01-30 17:50:54 +01:00
erinhmclark
d3eec5d90f Basic docs structure for RTD 2025-01-15 21:45:29 +00:00
Patrick Robertson
306df62a98 Fix all instances of utcnow() 2025-01-14 17:51:41 +01:00
Patrick Robertson
4e13a09a87 Fix deprecation warning about utcnow 2025-01-14 11:01:40 +01:00
R. Miles McCain
f603400d0d Add direct Atlos integration (#137)
* Add Atlos feeder

* Add Atlos db

* Add Atlos storage

* Fix Atlos storages

* Fix Atlos feeder

* Only include URLs in Atlos feeder once they're processed

* Remove print

* Add Atlos documentation to README

* Formatting fixes

* Don't archive existing material

* avoid KeyError in atlos_db

* version bump

---------

Co-authored-by: msramalho <19508417+msramalho@users.noreply.github.com>
2024-04-15 19:25:17 +01:00
Miguel Sozinho Ramalho
7a21ae96af V0.9.0 - closes several open issues: new enrichers and bug fixes (#133)
* clean orchestrator code, add archiver cleanup logic

* improves documentation for database.py

* telethon archivers isolate sessions into copied files

* closes #127

* closes #125

* closes #84

* meta enricher applies to all media

* closes #61 adds subtitles and comments

* minor update

* minor fixes to yt-dlp subtitles and comments

* closes #17 but logic is imperfect.

* closes #85 ssl enhancer

* minimifies html, JS refactor for preview of certificates

* closes #91 adds freetsa timestamp authority

* version bump

* simplify download_url method

* skip ssl if nothing archived

* html preview improvements

* adds retrying lib

* manual download archiver improvements

* meta only runs when relevant data available

* new metadata convenience method

* html template improvements

* removes debug message

* does not close #91 yet, will need a few more certificate chaing logging

* adds verbosity config

* new instagram api archiver

* adds proxy support we

* adds proxy/end support and bug fix for yt-dlp

* proxy support for webdriver

* adds socks proxy to wacz_enricher

* refactor recursivity in inner media and display

* infinite recursive display

* foolproofing timestamping authortities

* version to 0.9.0

* minor fixes from code-review
2024-02-20 18:05:29 +00:00
msramalho
499832d146 fix datetime parsing 2023-12-13 18:41:48 +00:00
Miguel Sozinho Ramalho
a786d4bb0e chooses most complete result from api (#116) 2023-12-13 11:26:46 +00:00
Miguel Sozinho Ramalho
6f36e92e02 enables api_db cache queries if configured with new option (#113) 2023-12-12 19:20:26 +00:00
msramalho
bd231488ff parameter fix 2023-07-28 13:10:06 +01:00
msramalho
7a5c9c65bd detects duplicates before storing, eg: wacz getting media already fetched by another archiver 2023-07-28 10:51:48 +01:00
msramalho
3dd3775cbd removes rearchiving logic 2023-07-27 20:14:50 +01:00
msramalho
0cba2c25c6 get all media method 2023-06-26 17:28:19 +01:00
msramalho
0a91863212 typing fixes 2023-05-24 11:18:39 +01:00
msramalho
9d44f4b207 content append instead of replace 2023-05-02 19:06:00 +01:00
msramalho
8c22a9df72 fixes "url-not-found" 2023-05-02 14:30:07 +01:00
msramalho
493055a8d9 cleanup 2023-03-23 18:50:30 +00:00
msramalho
6f6eb2db7a Archiving Context refactor complete 2023-03-23 14:28:45 +00:00
msramalho
906ed0f6e0 creating global context and refactoring tmp_dir logic 2023-03-23 11:17:38 +00:00
msramalho
aa5430451e instagram archiver via telegram bot 2023-02-17 15:46:29 +00:00
msramalho
4854929a1d thumbnail and bot token 2023-02-02 13:49:56 +00:00
msramalho
d1e4dde3f6 fixing imports 2023-01-27 00:19:58 +00:00
msramalho
ac000d5943 cleanup 2023-01-27 00:03:30 +00:00
msramalho
2508bb8a1b cleanup + rearchivable logic 2023-01-26 23:01:34 +00:00
msramalho
9dd8afed8c minor improvements 2023-01-22 23:15:54 +00:00
msramalho
746f6a333e further cleanup 2023-01-21 19:57:54 +00:00
msramalho
b763fc4188 final naming cleanup + new feeders/dbs 2023-01-21 19:44:12 +00:00
msramalho
753039240f pyproject 2023-01-21 19:01:02 +00:00