erinhmclark
8d894066f2
Merge branch 'load_modules' into add_module_tests
...
# Conflicts:
# src/auto_archiver/modules/gsheet_feeder/gsheet_feeder.py
# src/auto_archiver/utils/misc.py
2025-02-10 19:00:05 +00:00
msramalho
ab6cf52533
fixes bad hash initialization
2025-02-10 16:45:28 +00:00
erinhmclark
c4bb667cec
Merge branch 'load_modules' into add_module_tests
...
# Conflicts:
# src/auto_archiver/modules/s3_storage/s3_storage.py
# src/auto_archiver/utils/gsheet.py
# src/auto_archiver/utils/misc.py
2025-02-10 16:17:08 +00:00
erinhmclark
f311621e58
Small fixes.
...
Add timestamp helper method.
2025-02-10 15:57:42 +00:00
msramalho
15abf686b1
decouples s3_storage from hash_enricher
2025-02-10 15:48:54 +00:00
Patrick Robertson
63aba6ad39
Fix sphinx-autoapi imports
2025-02-07 21:54:49 +01:00
erinhmclark
266c7a14e6
Context related fixes, some more tests.
2025-02-06 16:53:00 +00:00
Patrick Robertson
0633e17998
Close the facebook 'login' window if it's there - to allow for proper screenshots
2025-02-04 14:18:46 +01:00
Patrick Robertson
72b5ea9ab6
Restore headless arg
2025-02-03 17:40:40 +01:00
Patrick Robertson
c574b694ed
Set up screenshot enricher to use authentication/cookies
2025-02-03 17:25:59 +01:00
Patrick Robertson
b7d9145f6c
Further tidyups + refactoring for new structure
...
* Add implementation tests for orchestrator + logging tests
* Standardise method/class vars for extractors to see if they are suitable
* Fix bugs with removing default loguru logger (allows further customisation)
* Fix bug loading required fields from file
*
2025-01-30 13:21:10 +01:00
Patrick Robertson
3d37c494aa
Tidy ups + unit tests:
...
1. Allow loading modules from --module_paths=/extra/path/here
2. Improved unit tests for module loading
3. Further small tidy ups/clean ups
2025-01-29 18:42:49 +01:00
erinhmclark
e1a9373336
Refactoring for new config setup
2025-01-27 19:03:02 +00:00
erinhmclark
96b35a272c
Rm gsheet references in utils
2025-01-24 18:51:15 +00:00
erinhmclark
dd402b456f
Fix and add types to manifest
2025-01-24 18:50:11 +00:00
erinhmclark
1942e8b819
Gsheets utility revert
2025-01-24 13:34:30 +00:00
erinhmclark
024fe58377
fix config parsing in manifests, remove module level configs
2025-01-24 13:33:12 +00:00
erinhmclark
0453d95f56
fix config parsing in manifests
2025-01-24 13:24:54 +00:00
Patrick Robertson
6388983815
Merge branch 'main' into youtubedlp-rewrite
2025-01-21 16:43:14 +01:00
Patrick Robertson
4bb4ebdf82
Further cleanup, abstracts 'dropins' out into generic files
2025-01-21 16:36:45 +01:00
Patrick Robertson
fd2e7f973b
Further tidy-ups, also adds some ytdlp utils to 'utils'
2025-01-20 16:31:28 +01:00
erinhmclark
d3eec5d90f
Basic docs structure for RTD
2025-01-15 21:45:29 +00:00
Patrick Robertson
663c8ad93a
Add 'print_pdf' option to the screenshot enricher. Fixes #132
2024-12-20 07:14:03 +01:00
R. Miles McCain
f603400d0d
Add direct Atlos integration ( #137 )
...
* Add Atlos feeder
* Add Atlos db
* Add Atlos storage
* Fix Atlos storages
* Fix Atlos feeder
* Only include URLs in Atlos feeder once they're processed
* Remove print
* Add Atlos documentation to README
* Formatting fixes
* Don't archive existing material
* avoid KeyError in atlos_db
* version bump
---------
Co-authored-by: msramalho <19508417+msramalho@users.noreply.github.com >
2024-04-15 19:25:17 +01:00
Miguel Sozinho Ramalho
7a21ae96af
V0.9.0 - closes several open issues: new enrichers and bug fixes ( #133 )
...
* clean orchestrator code, add archiver cleanup logic
* improves documentation for database.py
* telethon archivers isolate sessions into copied files
* closes #127
* closes #125
* closes #84
* meta enricher applies to all media
* closes #61 adds subtitles and comments
* minor update
* minor fixes to yt-dlp subtitles and comments
* closes #17 but logic is imperfect.
* closes #85 ssl enhancer
* minimifies html, JS refactor for preview of certificates
* closes #91 adds freetsa timestamp authority
* version bump
* simplify download_url method
* skip ssl if nothing archived
* html preview improvements
* adds retrying lib
* manual download archiver improvements
* meta only runs when relevant data available
* new metadata convenience method
* html template improvements
* removes debug message
* does not close #91 yet, will need a few more certificate chaing logging
* adds verbosity config
* new instagram api archiver
* adds proxy support we
* adds proxy/end support and bug fix for yt-dlp
* proxy support for webdriver
* adds socks proxy to wacz_enricher
* refactor recursivity in inner media and display
* infinite recursive display
* foolproofing timestamping authortities
* version to 0.9.0
* minor fixes from code-review
2024-02-20 18:05:29 +00:00
Miguel Sozinho Ramalho
3e56ef137d
reduce s3 duplicating while keeping random urls via hash ( #112 )
2023-12-12 19:12:03 +00:00
Galen Reich
381940f5a8
Fix Selenium headless invokation ( #106 )
...
Co-authored-by: msramalho <19508417+msramalho@users.noreply.github.com >
2023-11-13 11:56:35 +01:00
msramalho
ceb717ea65
exclude vk emojis
2023-08-17 18:11:26 +01:00
msramalho
6e4fb76940
exclude ok resource images from wacz enricher
2023-08-09 11:26:46 +01:00
msramalho
60a1f3a27a
minor fixes
2023-07-31 16:08:48 +01:00
msramalho
fb197f1064
excluding telegram embeds
2023-07-28 12:57:15 +01:00
msramalho
aa71c85a98
improving ignored content from waczs
2023-07-28 12:19:14 +01:00
msramalho
fc93ebaba0
cleanup
2023-07-28 10:49:39 +01:00
msramalho
59551b3b20
minor improvements: finding best twitter image quality
2023-07-27 21:36:15 +01:00
msramalho
f086d89111
new escape message
2023-07-27 20:14:59 +01:00
msramalho
dd034da844
feat: WACZ enricher can now be probed for media, and used as an archiver OR enricher
2023-07-27 15:42:10 +01:00
msramalho
888ad8f004
fix: twitter hack videos extension detection
2023-07-26 16:12:56 +01:00
msramalho
7c0b05b276
new column
2023-06-26 17:27:57 +01:00
msramalho
613b1f1e50
properly overwrite configs
2023-05-19 12:35:19 +01:00
msramalho
a655b3c987
gsheet accepts ID too
2023-05-19 12:17:34 +01:00
msramalho
45b982ec38
fix: max chars on sheets cell
2023-05-10 18:57:33 +01:00
msramalho
ae3e607705
fix: depreacating thumbnail_index
2023-05-09 11:29:05 +01:00
msramalho
c1a60fde8a
fix: deprecates duration column
2023-05-09 11:26:19 +01:00
msramalho
23894fad51
normalize columns
2023-02-20 16:08:35 +00:00
msramalho
aa5430451e
instagram archiver via telegram bot
2023-02-17 15:46:29 +00:00
msramalho
5505255ea3
url auth wall detect
2023-02-17 15:45:58 +00:00
msramalho
5b0593ce82
arg parse fix
2023-02-02 11:00:24 +00:00
msramalho
d1e4dde3f6
fixing imports
2023-01-27 00:19:58 +00:00
msramalho
746f6a333e
further cleanup
2023-01-21 19:57:54 +00:00
msramalho
753039240f
pyproject
2023-01-21 19:01:02 +00:00
