bellingcat/snscrape
1
0
Fork 0
mirror of https://github.com/bellingcat/snscrape.git synced 2026-06-10 19:38:29 +03:00
Code Issues Packages Projects Releases Wiki Activity
346 Commits 4 Branches 10 Tags
ff5e2d61ee83a6f10c2b166b7e449936dcc95670
Commit Graph

346 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
JustAnotherArchivist
ff5e2d61ee Update search API parameters 2023-01-13 03:01:48 +00:00
JustAnotherArchivist
129ad3fc34 Add --max-empty-pages option to stop long (potentially infinite) empty pagination 
Fixes #636
 2023-01-13 02:35:48 +00:00
JustAnotherArchivist
7de8d734e9 Override TLS ciphers to get past Twitter's new fingerprinting 
Fixes #647
 2023-01-13 02:25:39 +00:00
JustAnotherArchivist
ceb06664f0 Clarify descriptions of issue templates 2023-01-11 22:52:52 +00:00
JustAnotherArchivist
996cf882cc Expose status code for non-200 Twitter responses 2023-01-11 20:01:05 +00:00
JustAnotherArchivist
e449d5cdbe Expose individual error messages when all request retries fail 2023-01-11 20:01:05 +00:00
JustAnotherArchivist
cbdaee6864 Merge pull request #343 from TheTechRobo/master 
Add issue templates for snscrape
 2022-12-19 23:25:17 +00:00
JustAnotherArchivist
a3bee057b1 Merge pull request #615 from engkimo/fix-return-twitter-place-ids 
Add returning Twitter Place IDs
 2022-12-19 22:57:40 +00:00
JustAnotherArchivist
6f9a0e6534 Merge pull request #590 from caseyho/UnifiedCardApp_no_category 
Handle tweets that contain card info with no category
 2022-12-19 22:55:36 +00:00
engkimo
4ff4af13cf Add returning Twitter Place IDs 2022-12-06 11:23:01 +09:00
JustAnotherArchivist
e09aea70e7 Fix Twitter username length limit 
Although 15 characters is the official, current limit, there are accounts with longer usernames. 20 is the longest observed example, but it's unclear what the true limit is.
 2022-12-03 06:36:52 +00:00
Casey Ho
aa325fa1a5 Handle UnifiedCardApp with no category 2022-11-14 17:38:03 -08:00
JustAnotherArchivist
46a603053c Handle users with extensions but no label 
Fixes #559
 2022-10-16 21:13:46 +00:00
JustAnotherArchivist
59abeaf04c Make newsletter card images optional 
Fixes #546
 2022-09-04 15:04:20 +00:00
JustAnotherArchivist
e13033fea0 Fix AttributeError on certain videos included from other platforms 2022-08-24 15:53:21 +00:00
JustAnotherArchivist
9294c26ffa Make PeriscopeBroadcastCard.thumbnailUrl optional to handle tweets without a thumbnail 
Fixes #507
 2022-08-21 01:58:41 +00:00
JustAnotherArchivist
d6bce5b1d6 Merge pull request #518 from hgrsd/fix/vkontakte-photo-scrape 
fix(vkontakte): update photo detection
 2022-08-21 01:49:59 +00:00
JustAnotherArchivist
2c7a85a620 Add warning on unknown page_info types 2022-08-21 01:40:49 +00:00
JustAnotherArchivist
ff18f6f771 Fix video extraction on Weibo 
Fixes #509
 2022-08-21 01:40:31 +00:00
JustAnotherArchivist
da3d870e10 Drop app icons when Twitter didn't actually include them in the response 
Fixes #470
 2022-08-13 21:17:55 +00:00
hgrsd
279d1cf4a1 fix(vkontakte): update photo detection 2022-07-16 18:27:02 +01:00
JustAnotherArchivist
d72b51953f Fix missing r prefix on string with regex backslashes 2022-06-24 23:12:50 +00:00
JustAnotherArchivist
d5b406bc1b Update API parameters to what Twitter currently uses 
The `count` reduction does not affect anything as Twitter ignores that parameter now. Cf. #481
 2022-06-23 19:50:17 +00:00
JustAnotherArchivist
50899c01f3 Fix crash on malformed guest token cache file 
Fixes #494
 2022-06-16 17:12:04 +00:00
JustAnotherArchivist
bcad6923c2 Rename Tweet.content to rawContent and User.description to renderedDescription for consistency 
Closes #479
 2022-06-14 00:35:02 +00:00
JustAnotherArchivist
0d361685ff Fix AttributeError crash on scrapers using the default CLI constructor 
Introduced by 267b7d0e

Fixes #483
 2022-06-01 17:35:38 +00:00
JustAnotherArchivist
530f4fa122 Fix KeyErrors on display_url and expanded_url for certain users with broken profile links 
Fixes #480
 2022-05-29 17:23:43 +00:00
JustAnotherArchivist
dc6bc9bf9d Refactor how links on Twitter are handled 
All links in text (tweets, profile descriptions, and profile links) are now represented by TextLink objects, which contain all relevant information: the displayed text (if available), the URL, the short t.co URL, and the indices in the text at which it appears.

Closes #478
 2022-05-29 07:16:04 +00:00
JustAnotherArchivist
01cf6a09b3 Fix type of description URL objects 2022-05-29 05:08:23 +00:00
JustAnotherArchivist
ef7c4fad3e Fix AttributeError for DescriptionURL on from-import 2022-05-29 05:08:23 +00:00
JustAnotherArchivist
faeffe2603 Merge pull request #474 from GeraniumKF/GeraniumKF-reddit-since-crash 
Fix crash using --since with Reddit
 2022-05-23 23:06:16 +00:00
Geranium
e3bdc02a7c Reddit: deprecate 'created' property for 'date' 
This fixes a crash when using --since with the Reddit scraper,
as the CLI code expects items to have a date property.
 2022-05-23 23:31:44 +01:00
JustAnotherArchivist
ed3ea944d1 Fix newsletter issue cards without an issue description 
Fixes #456
 2022-04-16 19:44:36 +00:00
JustAnotherArchivist
e7a6d38a5f Add support for community_details cards 2022-04-15 20:07:01 +00:00
JustAnotherArchivist
6c50eee31b Fix proxies not being applied correctly due to missing merge with environment settings 
Fixes #447
 2022-04-15 19:23:54 +00:00
JustAnotherArchivist
5103a33afa Fix t.co card URL replacement on retweets 
Fixes #411
 2022-04-15 03:18:45 +00:00
JustAnotherArchivist
247bd82d79 Refactor to tweetId variable 2022-04-15 03:14:29 +00:00
JustAnotherArchivist
5fc67f2bcf Add support for 'message me' cards 2022-04-15 02:52:37 +00:00
JustAnotherArchivist
65e7d8bd24 Fix warning on card URL translation to include the tweet ID 2022-04-15 02:52:03 +00:00
JustAnotherArchivist
3870282a42 Fix broadcast and event card crashes 2022-04-12 20:53:38 +00:00
JustAnotherArchivist
7c0fcdec43 Fix Periscope card crashes 2022-04-12 18:29:51 +00:00
JustAnotherArchivist
9af1f19034 Properly support all card types 
Fixes #407
 2022-04-12 18:11:26 +00:00
JustAnotherArchivist
5fc3c0e290 Fix crash in locals dumping on module-less frames 2022-04-12 18:03:36 +00:00
JustAnotherArchivist
5d156c6a15 Detect and raise error on redirect from GraphQL endpoint to login 
#165
 2022-04-03 02:34:30 +00:00
JustAnotherArchivist
694657ef80 Fix broken exception references 2022-03-09 01:01:47 +00:00
JustAnotherArchivist
1ab0f4fccb Fix missing quoted tweet reference in certain buggy cases 2022-03-07 22:16:58 +00:00
JustAnotherArchivist
3a92b5bf0d Add log message for guest token file deletion 2022-02-26 19:32:55 +00:00
JustAnotherArchivist
2480b173f4 Fix crash on race condition in CLI guest token manager resets 
Fixes #414
 2022-02-26 19:31:08 +00:00
JustAnotherArchivist
77bbb9f61f Remove useless pass 2022-02-20 18:54:51 +00:00
JustAnotherArchivist
57a624c618 Merge pull request #410 from AccentuSoft/master 
Fix Vkontakte-user module crash on users with millions of followers
 2022-02-18 06:01:35 +00:00