bellingcat/cisticola
1
0
Fork 0
mirror of https://github.com/bellingcat/cisticola.git synced 2026-06-08 03:18:34 +03:00
Code Issues Packages Projects Releases Wiki Activity
177 Commits 14 Branches 4 Tags
f3997ff6ae1f28a1aca8ed59c665457908e7ed62
Commit Graph

177 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Logan Williams
f3997ff6ae Catch errors in Bitchute channel profile scraper; add multi index on posts forwarded from/channel 2022-08-01 09:58:52 +00:00
Logan Williams
3a04fb51d4 Merge branch 'main' of https://github.com/bellingcat/cisticola into main 2022-07-28 09:18:34 +00:00
Logan Williams
fee216386b Fix issue with chronological media archiving 2022-07-28 09:18:07 +00:00
Logan Williams
d05584a09f Minor bug fixes; helper tool for Telethon sessions 2022-07-28 08:42:59 +00:00
Logan Williams
ee24367caa Add features for running archive-media simultaneously 2022-07-20 09:26:47 +00:00
Logan Williams
fbb846b8d6 Fix two small bugs with media archiving 2022-07-05 13:30:39 +02:00
Logan Williams
b99958a894 Merge pull request #58 from bellingcat/media-etl 
Media ETL
 2022-07-05 11:51:08 +02:00
Logan Williams
51e5ca1f04 Use smaller batches for now 2022-07-05 09:48:57 +00:00
Logan Williams
6149c4279d Add some more fields to media DB, fix bugs in testing 2022-07-05 11:11:43 +02:00
Logan Williams
4ddd8d6b63 Only select untransformed media; simplify insert function 2022-07-05 10:03:38 +02:00
Logan Williams
9948af2c4a Media archiving ETL working for Telegram 2022-07-05 10:03:36 +02:00
Logan Williams
c24babb081 Fix bugs in Gettr/Rumble transformers, avoid offset in batch requests 2022-07-04 14:30:40 +00:00
Logan Williams
ed4723ed1e Fix merge error 2022-06-30 11:04:41 +00:00
Logan Williams
589ac3ba5b Merge branch 'main' of https://github.com/bellingcat/cisticola into main 2022-06-24 10:34:25 +00:00
Logan Williams
7215469a74 Correct logs for transforming posts 2022-06-24 10:32:27 +00:00
Logan Williams
fe0f4f9e2c Merge pull request #62 from bellingcat/other-transformer-fixes 
Fixed broken channel_info transformers, added Telegram post transformer fields
 2022-06-24 11:00:50 +02:00
Tristan Lee
289a47d7b1 tested telegram transformers and implemented vk transformers 2022-06-23 15:06:10 -05:00
Tristan Lee
bb2e2806e6 got post transformers and channel_info transformers working for Rumble, Bitchute, Gettr 2022-06-21 19:05:41 -05:00
Tristan Lee
619fe42a31 got transformers for Bitchute, Rumble, and Gettr working for all raw_posts. 2022-06-20 21:45:41 -05:00
Tristan Lee
a2a7882f1c fixed Gettr and Bitchute info transformers, added missing or incorrect TelegramTransformer fields, added Telegram mentions to the transformer. 2022-06-13 13:42:33 -05:00
Logan Williams
6e962de244 Don't scrape channel info unless specifically scraping channel info 2022-06-10 08:41:45 +00:00
Logan Williams
6183972b1a Merge branch 'more-channel-info-transformers' of https://github.com/bellingcat/cisticola into main 2022-06-10 08:07:02 +00:00
Logan Williams
d83a13e0cc Merge branch 'main' of https://github.com/bellingcat/cisticola into main 2022-06-09 14:11:52 +00:00
Logan Williams
fba3a661c7 Sleep after every gsheet API call 2022-06-09 14:11:29 +00:00
Logan Williams
6294ea7ea7 Increment TelegramTelethonScraper version 2022-06-09 15:30:56 +02:00
Logan Williams
92d4839b5e Revise Telethon scraper to use the same client connection 2022-06-09 10:01:27 +02:00
Logan Williams
9a30ecb243 Stop overwriting media when a large file is detected 2022-06-08 17:01:28 +02:00
Logan Williams
708d952937 Merge branch 'main' of https://github.com/bellingcat/cisticola into main 2022-06-08 08:11:34 +00:00
Logan Williams
bf96f248c1 Merge pull request #57 from bellingcat/synchronization-improvements 
Google sheet channel synchronization improvements
 2022-06-08 10:08:23 +02:00
Logan Williams
143b20fc56 Update Pipfile to load pyexiftool from PyPi 
Co-authored-by: Tristan Lee <tristan@bellingcat.com>
 2022-06-08 10:08:18 +02:00
Logan Williams
f932034ab2 Add a bit more logging 2022-06-07 13:14:48 +02:00
Logan Williams
39358c7f23 Update platform ID and screenname when synchronizing with gsheet; highlight dupes 2022-06-06 16:36:39 +02:00
Logan Williams
4d22838a94 Fixes to Pipfile; easier spacy setup 2022-06-06 16:35:52 +02:00
Tristan Lee
f4072183be added transformer for Gettr 2022-05-20 02:22:34 -05:00
Tristan Lee
591f1986e8 added Rumble transformers and test 2022-05-19 19:40:48 -05:00
Tristan Lee
e2094522c9 updated Bitchute transformer and addewd test 2022-05-19 18:13:50 -05:00
Tristan Lee
f0414a4f4d updated transformer tests 2022-05-19 16:34:19 -05:00
Tristan Lee
424c063ef2 Merge pull request #55 from bellingcat/channel-info-transformers 
Transformers for raw channel info
 2022-05-18 06:58:18 -07:00
Logan Williams
c279ced73d Minor bug fixes from testing 2022-05-18 09:29:53 +01:00
Logan Williams
7c8147bb2a Add CLI for channel info transform 2022-05-18 09:20:33 +01:00
Logan Williams
6145fd0b6b Add Telegram transformer for channel info 2022-05-18 09:17:49 +01:00
Tristan Lee
317da2c9d4 Merge pull request #54 from bellingcat/transformers 
Functional Telegram transformer
 2022-05-17 10:18:12 -07:00
Logan Williams
9869612b67 Merge pull request #50 from bellingcat/odysee-refactor 
Implemented Polyphemus refactoring changes into Odysee scraper
 2022-05-16 12:25:35 +01:00
Logan Williams
7f55b721dd Bug fixes in transformers 2022-05-13 15:39:01 +00:00
Logan Williams
34da733e7c Add date_transformed; refinements to telethon transformer 2022-05-12 13:03:30 +00:00
Logan Williams
4493618801 Synchronizing channels will update other info for existing channels 2022-05-12 13:02:14 +00:00
Logan Williams
ab482443db Merge branch 'main' of https://github.com/bellingcat/cisticola into transformers 2022-04-16 13:55:23 +00:00
Logan Williams
8535a87def Ad hoc changes to transformers 2022-04-16 13:46:26 +00:00
Logan Williams
3b8b03283a Add logging to transform 2022-04-14 11:32:15 +00:00
Logan Williams
428af3575f Merge branch 'transformers' of https://github.com/bellingcat/cisticola into main 2022-04-14 11:31:34 +00:00