bellingcat/snscrape
1
0
Fork 0
mirror of https://github.com/bellingcat/snscrape.git synced 2026-06-11 20:08:29 +03:00
Code Issues Packages Projects Releases Wiki Activity
388 Commits 4 Branches 10 Tags
4e70306f9976cf946e3366ed036d7f7347d378c6
Commit Graph

183 Commits

Author SHA1 Message Date
JustAnotherArchivist
4e70306f99 Deprecate Entity type 
There is no meaningful distinction from Items, and it complicates the integration of scrapers for user searches
 2023-02-21 21:24:00 +00:00
JustAnotherArchivist
7327a01397 Refactor module-level deprecation code 2023-02-21 21:23:12 +00:00
JustAnotherArchivist
880a0a7f55 Handle TweetUnavailable results 
Fixes #433
 2023-02-21 20:16:23 +00:00
JustAnotherArchivist
57b126c656 Add support for scraping Twitter Communities 
Closes #614
 2023-02-21 20:15:57 +00:00
JustAnotherArchivist
82f64a6472 Remove dead code 2023-02-21 06:22:13 +00:00
JustAnotherArchivist
6a6b02cb28 Handle tombstones 
Closes #392
Fixes #603
 2023-02-21 04:23:47 +00:00
JustAnotherArchivist
3d6cd63a00 Fix more logger typos 2023-02-21 04:23:47 +00:00
JustAnotherArchivist
9a2f1524c2 Remove dead code 2023-02-21 04:23:47 +00:00
JustAnotherArchivist
b5694e01a2 Fix logger typo 2023-02-21 04:23:47 +00:00
JustAnotherArchivist
280b972f22 Fix extraction of tweets behind 'offensive' replies button 2023-02-21 04:23:47 +00:00
JustAnotherArchivist
c65e36a094 Bump GraphQL endpoints 2023-02-19 06:21:40 +00:00
JustAnotherArchivist
fe5d90b748 Fix tweets behind 'Show more replies' button getting missed 
Fixes #572
 2023-02-19 03:29:39 +00:00
quentinwolf
0933a30e37 change fullUrl to use 'orig' instead of 'large' 
Changing fullUrl from '&name=large' to '&name=orig'  since large is capped at half the resolution of orig which may not be ideal for scraping/archiving.

Large images are  2048px x 1365px
Original images are up to 4096px × 2730px

Alternatively one could add largeUrl as an alternative to download the Large image and utillze fullUrl as above to download the original image for those that do wish to save either versions, but I feel there is no reason for saving the middle-resolution image.
 2023-02-13 16:45:44 -07:00
JustAnotherArchivist
d60ce38b6a Make (most) consistency errors in unified cards non-fatal 
Fixes #703
 2023-02-10 02:39:06 +00:00
JustAnotherArchivist
b515a66b93 Fix crash in recursive tweet scraping 
Introduced by 3e297c9a

Fixes #684
 2023-01-19 16:18:15 +00:00
JustAnotherArchivist
d0fb9ab8a9 Log TLS connection details for debugging 2023-01-16 02:39:05 +00:00
JustAnotherArchivist
5d3f27bc2b Fix title-less BroadcastCard crash 2023-01-15 16:36:04 +00:00
JustAnotherArchivist
b7cb270b6e Fix crash on empty user objects 2023-01-15 12:31:28 +00:00
JustAnotherArchivist
564a5eca77 Fix crash on unavailable users in cards 2023-01-13 09:12:16 +00:00
JustAnotherArchivist
bf0e720b5a Fix crash on empty tweet entries in timelines 
Fixes #620
 2023-01-13 09:01:15 +00:00
JustAnotherArchivist
27374285a2 Fix crash on missing source label data 
This data had been announced in mid-November to disappear but was still always returned by the API until very recently.
 2023-01-13 08:32:02 +00:00
JustAnotherArchivist
238bdcd560 Reduce warnings about duplicate users on cards 2023-01-13 08:28:52 +00:00
JustAnotherArchivist
e846a6a4cd Fix KeyError in card user handling 2023-01-13 08:06:57 +00:00
JustAnotherArchivist
cbeb65d5c9 Fix KeyError crash on some tweets with AmplifyCards 
Fixes #601
 2023-01-13 07:57:31 +00:00
JustAnotherArchivist
3e19f8f84b Add support for image_collection_website unified cards 2023-01-13 07:36:53 +00:00
JustAnotherArchivist
28f5a45825 Fix empty page counter not getting reset on results 2023-01-13 06:59:51 +00:00
JustAnotherArchivist
2196bdf3e8 Extract vibe 2023-01-13 04:09:00 +00:00
JustAnotherArchivist
faf09b2f5e Extract tweet view counts 
Closes #629
 2023-01-13 04:00:50 +00:00
JustAnotherArchivist
3e297c9a42 Update GraphQL API parameters 2023-01-13 04:00:31 +00:00
JustAnotherArchivist
a0414d92cf Extract alt text for media on Twitter 
Closes #588
 2023-01-13 03:13:10 +00:00
JustAnotherArchivist
ff5e2d61ee Update search API parameters 2023-01-13 03:01:48 +00:00
JustAnotherArchivist
129ad3fc34 Add --max-empty-pages option to stop long (potentially infinite) empty pagination 
Fixes #636
 2023-01-13 02:35:48 +00:00
JustAnotherArchivist
7de8d734e9 Override TLS ciphers to get past Twitter's new fingerprinting 
Fixes #647
 2023-01-13 02:25:39 +00:00
JustAnotherArchivist
996cf882cc Expose status code for non-200 Twitter responses 2023-01-11 20:01:05 +00:00
JustAnotherArchivist
a3bee057b1 Merge pull request #615 from engkimo/fix-return-twitter-place-ids 
Add returning Twitter Place IDs
 2022-12-19 22:57:40 +00:00
JustAnotherArchivist
6f9a0e6534 Merge pull request #590 from caseyho/UnifiedCardApp_no_category 
Handle tweets that contain card info with no category
 2022-12-19 22:55:36 +00:00
engkimo
4ff4af13cf Add returning Twitter Place IDs 2022-12-06 11:23:01 +09:00
JustAnotherArchivist
e09aea70e7 Fix Twitter username length limit 
Although 15 characters is the official, current limit, there are accounts with longer usernames. 20 is the longest observed example, but it's unclear what the true limit is.
 2022-12-03 06:36:52 +00:00
Casey Ho
aa325fa1a5 Handle UnifiedCardApp with no category 2022-11-14 17:38:03 -08:00
JustAnotherArchivist
46a603053c Handle users with extensions but no label 
Fixes #559
 2022-10-16 21:13:46 +00:00
JustAnotherArchivist
59abeaf04c Make newsletter card images optional 
Fixes #546
 2022-09-04 15:04:20 +00:00
JustAnotherArchivist
9294c26ffa Make PeriscopeBroadcastCard.thumbnailUrl optional to handle tweets without a thumbnail 
Fixes #507
 2022-08-21 01:58:41 +00:00
JustAnotherArchivist
da3d870e10 Drop app icons when Twitter didn't actually include them in the response 
Fixes #470
 2022-08-13 21:17:55 +00:00
JustAnotherArchivist
d5b406bc1b Update API parameters to what Twitter currently uses 
The `count` reduction does not affect anything as Twitter ignores that parameter now. Cf. #481
 2022-06-23 19:50:17 +00:00
JustAnotherArchivist
50899c01f3 Fix crash on malformed guest token cache file 
Fixes #494
 2022-06-16 17:12:04 +00:00
JustAnotherArchivist
bcad6923c2 Rename Tweet.content to rawContent and User.description to renderedDescription for consistency 
Closes #479
 2022-06-14 00:35:02 +00:00
JustAnotherArchivist
530f4fa122 Fix KeyErrors on display_url and expanded_url for certain users with broken profile links 
Fixes #480
 2022-05-29 17:23:43 +00:00
JustAnotherArchivist
dc6bc9bf9d Refactor how links on Twitter are handled 
All links in text (tweets, profile descriptions, and profile links) are now represented by TextLink objects, which contain all relevant information: the displayed text (if available), the URL, the short t.co URL, and the indices in the text at which it appears.

Closes #478
 2022-05-29 07:16:04 +00:00
JustAnotherArchivist
01cf6a09b3 Fix type of description URL objects 2022-05-29 05:08:23 +00:00
JustAnotherArchivist
ef7c4fad3e Fix AttributeError for DescriptionURL on from-import 2022-05-29 05:08:23 +00:00