Tristan Lee
|
289a47d7b1
|
tested telegram transformers and implemented vk transformers
|
2022-06-23 15:06:10 -05:00 |
|
Tristan Lee
|
a2a7882f1c
|
fixed Gettr and Bitchute info transformers, added missing or incorrect TelegramTransformer fields, added Telegram mentions to the transformer.
|
2022-06-13 13:42:33 -05:00 |
|
Tristan Lee
|
f4072183be
|
added transformer for Gettr
|
2022-05-20 02:22:34 -05:00 |
|
Tristan Lee
|
591f1986e8
|
added Rumble transformers and test
|
2022-05-19 19:40:48 -05:00 |
|
Tristan Lee
|
e2094522c9
|
updated Bitchute transformer and addewd test
|
2022-05-19 18:13:50 -05:00 |
|
Tristan Lee
|
f0414a4f4d
|
updated transformer tests
|
2022-05-19 16:34:19 -05:00 |
|
Tristan Lee
|
282f33eff3
|
implemented deferred media archiving for all scrapers, and implemented tests for them. Refactored archiving methods of Instagram and Gettr scrapers to be able to use default archiving method
|
2022-04-01 01:30:49 -05:00 |
|
Logan Williams
|
94cf6c3d84
|
TelegramTelethonScraper: Use channel_id when channel has been previously encountered
|
2022-03-31 16:37:54 +02:00 |
|
Logan Williams
|
7f87b03de5
|
Add option to clear registered scrapers, necessary for tests
|
2022-03-31 16:17:35 +02:00 |
|
Tristan Lee
|
b7871b060d
|
added capability to scrape Gab group posts
|
2022-03-30 09:11:07 -05:00 |
|
Tristan Lee
|
1f99e52436
|
refactored Gab scraper to use gabber instead of garc
|
2022-03-30 08:05:10 -05:00 |
|
Tristan Lee
|
67d1abf024
|
added methods for extracting channel profile metadata, and tests
|
2022-03-28 21:11:34 -05:00 |
|
Tristan Lee
|
16870d7daa
|
implemented methods for extracting profile metadata (still need to test)
|
2022-03-28 20:16:59 -05:00 |
|
Logan Williams
|
fa516da763
|
Rename TransformedResult to the clearer Post
|
2022-03-22 11:41:55 +01:00 |
|
Logan Williams
|
571b019137
|
Fix tests for Twitter transformer
|
2022-03-22 11:33:27 +01:00 |
|
Tristan Lee
|
e287fd03d9
|
merged scraper into main and fixed minor merge conflict
|
2022-03-15 09:12:12 -05:00 |
|
Tristan Lee
|
750f0cc887
|
added scraper for Instagram
|
2022-03-14 10:28:10 -05:00 |
|
Logan Williams
|
fd4b617743
|
Add TwitterTransformer test
|
2022-03-14 13:39:10 +01:00 |
|
Tristan Lee
|
965bf1e2dc
|
added youtube scraper, moved from official youtube-dl repo to using yt-dlp because download speed for youtube videos is much better
|
2022-03-11 17:19:52 -06:00 |
|
Tristan Lee
|
821c39004b
|
incorporated vkontakte scraper
|
2022-03-10 22:32:39 -06:00 |
|
Tristan Lee
|
5783206ad8
|
implemented method to reset database, to enable the 'contoller' fixture scope to be shared across the whole package, which will enable the transformer tests to be run without re-running the scrapers
|
2022-03-10 10:20:49 -06:00 |
|
Tristan Lee
|
6cf3b8842d
|
renamed 'archive_media' and 'media' to avoid name collision, changed scope of test fixture controller to 'function' so that db is fresh for each executed test
|
2022-03-09 13:19:35 -06:00 |
|
Tristan Lee
|
739e1d8484
|
added capability of running scraper without archiving media, and implemented prototype Telethon scraper for Telegram
|
2022-03-09 12:12:01 -06:00 |
|
Tristan Lee
|
cd5f68e9e5
|
added basic unit tests
|
2022-03-04 12:36:09 -06:00 |
|