JustAnotherArchivist
247bd82d79
Refactor to tweetId variable
2022-04-15 03:14:29 +00:00
JustAnotherArchivist
5fc67f2bcf
Add support for 'message me' cards
2022-04-15 02:52:37 +00:00
JustAnotherArchivist
65e7d8bd24
Fix warning on card URL translation to include the tweet ID
2022-04-15 02:52:03 +00:00
JustAnotherArchivist
3870282a42
Fix broadcast and event card crashes
2022-04-12 20:53:38 +00:00
JustAnotherArchivist
7c0fcdec43
Fix Periscope card crashes
2022-04-12 18:29:51 +00:00
JustAnotherArchivist
9af1f19034
Properly support all card types
...
Fixes #407
2022-04-12 18:11:26 +00:00
JustAnotherArchivist
694657ef80
Fix broken exception references
2022-03-09 01:01:47 +00:00
JustAnotherArchivist
1ab0f4fccb
Fix missing quoted tweet reference in certain buggy cases
2022-03-07 22:16:58 +00:00
JustAnotherArchivist
3a92b5bf0d
Add log message for guest token file deletion
2022-02-26 19:32:55 +00:00
JustAnotherArchivist
2480b173f4
Fix crash on race condition in CLI guest token manager resets
...
Fixes #414
2022-02-26 19:31:08 +00:00
JustAnotherArchivist
77bbb9f61f
Remove useless pass
2022-02-20 18:54:51 +00:00
JustAnotherArchivist
2f9c0457df
Convert t.co card URLs to unshortened when possible
2022-02-17 01:50:15 +00:00
JustAnotherArchivist
878f2a3c7a
Handle cards without descriptions and thumbnails
...
Fixes #407
2022-02-17 01:49:32 +00:00
JustAnotherArchivist
25ee014e29
Extract cards
2022-02-16 02:59:21 +00:00
JustAnotherArchivist
a192dc6236
Handle TweetWithVisibilityResults
...
Fixes #400
2022-02-14 18:08:59 +00:00
JustAnotherArchivist
a7242f340b
Remove obsolete TODO
...
There is no retweetedTweetRef in Twitter's JS.
2022-02-14 18:08:29 +00:00
JustAnotherArchivist
359cc25cdf
Fix crash on entity attribute when scraping suspended users
...
Fixes #396
2022-02-10 04:22:59 +00:00
JustAnotherArchivist
01799a7391
Detect when CLI guest token from file has expired
2022-02-08 19:38:45 +00:00
JustAnotherArchivist
b0753c34ed
Fix forgotten method name changes in 7d939c11
...
Fixes #393
2022-02-08 15:35:49 +00:00
JustAnotherArchivist
7f78fa0bc0
Recurse through all tweets encountered, not only ones with a positive replyCount
...
Fixes #266
2022-02-07 18:13:56 +00:00
JustAnotherArchivist
9235890f9a
Fix KeyError crash on attempting to scrape inexistent tweet ID
2022-02-07 04:04:21 +00:00
JustAnotherArchivist
7d939c110c
Port profile and tweet scrapers to GraphQL API
...
Fixes #367
2022-02-07 03:49:14 +00:00
JustAnotherArchivist
8e95e9a9a7
Fix crash on places without a bounding box
...
Fixes #374
2022-02-07 00:38:22 +00:00
JustAnotherArchivist
560c78c5cf
Make all optional scraper arguments keyword-only and fix Mastodon argument style to conform with the other scrapers
...
Cf. #376
2022-01-30 00:21:18 +00:00
JustAnotherArchivist
107c3c71c2
Remove unnecessary f-strings
...
Cf. #370
2022-01-28 21:22:13 +00:00
JustAnotherArchivist
eebdfc1c55
Refactor username vs ID mess
...
Closes #354
2022-01-12 22:36:26 +00:00
JustAnotherArchivist
e6076353c8
Fix user ID being a string instead of an int on the entity
2022-01-12 22:35:50 +00:00
JustAnotherArchivist
deb2659dd6
Prefix CLI-related methods with an underscore
...
Closes #355
2022-01-12 21:07:10 +00:00
JustAnotherArchivist
853848ed5d
ScrollDirection is not part of the public API
2022-01-05 19:43:19 +00:00
JustAnotherArchivist
0b4abdc43f
Fix baseUrl on tweet scrapes
2022-01-05 02:39:54 +00:00
JustAnotherArchivist
267b7d0e32
Rename CLI classmethods
2022-01-05 02:27:09 +00:00
JustAnotherArchivist
acb7f10a4f
Cache Twitter tokens on disk from the CLI for reuse between scrapes
...
Closes #339
2022-01-05 02:20:40 +00:00
JustAnotherArchivist
f189ab4241
Prefix all private API names with an underscore
...
Cf. #328
2022-01-03 17:51:23 +00:00
JustAnotherArchivist
0336ce13ed
Add support for fetching a guest token from the API
2021-12-23 04:26:50 +00:00
JustAnotherArchivist
193d4f80d6
Fix user agent in API headers staying constant
2021-12-23 04:25:23 +00:00
JustAnotherArchivist
8540045658
Fix typo
2021-12-15 16:36:28 +00:00
JustAnotherArchivist
7fdc8bcb53
Randomise user agent when the guest token can't be found
2021-12-14 20:04:46 +00:00
JustAnotherArchivist
4b3c6aefe7
Add default values to user and tweet scrapers for a more untuitive usage
2021-12-12 04:57:16 +00:00
JustAnotherArchivist
525cd71225
Retry guest token retrieval
...
Fixes #325 (hopefully)
2021-12-12 00:10:59 +00:00
JustAnotherArchivist
72abff9e5c
Reuse guest tokens across scrapes
...
Cf. #326
2021-12-11 23:18:42 +00:00
JustAnotherArchivist
a6b6f3faaa
Throw an error on empty arguments
...
Fixes #290
2021-10-10 17:43:27 +00:00
JustAnotherArchivist
5e829e2541
Refactor class instantiation to remove the need to repeat 'retries' everywhere
2021-09-30 09:58:10 +00:00
JustAnotherArchivist
e5e0da25a0
Remove unused imports
2021-09-30 09:24:18 +00:00
JustAnotherArchivist
e382891642
Fix Twitter trends not having a str representation
2021-09-21 21:40:50 +00:00
JustAnotherArchivist
e5f4389464
Add Twitter trend scraper
...
Due to restrictions on Twitter's side, it is not possible to get trends from a custom location as that would require using an account and/or their API.
Closes #206
2021-09-21 21:28:41 +00:00
JustAnotherArchivist
d91f971f51
Refactor user label implementation and add support for bot accounts
...
Closes #281
2021-09-21 19:39:40 +00:00
JustAnotherArchivist
5fc2562642
Add user label support on entity retrieval
2021-09-19 03:32:35 +00:00
JustAnotherArchivist
2825bd0a73
Remove accidental empty line
2021-09-19 03:31:56 +00:00
Ed Summers
9831f2a4a0
missing ext
...
While doing some long term data collection I found some user objects
that lack the key 'ext'. This would cause an exception unless it's
checked for before trying to dig out results.
2021-09-16 13:31:47 -04:00
Ed Summers
a11eef6b06
User label url
...
Each label also has a URL which is used for learning more about the
label. While there are more label descriptions than label URLs the URLs
do seem to group language variants of the same label. For example
https://help.twitter.com/rules-and-policies/state-affiliated-china is
used for all of the following label descriptions:
* Média affilié à un État, Chine
* China state-affiliated media
* 中国官方媒体
* Çin devletine bağlı medya
* China government official
In some analysis contexts it could be useful to group these together.
2021-09-16 13:04:57 -04:00
