JustAnotherArchivist
7b967ff82a
Twitter reverted their guest token change ( 90f9598e)
2020-07-08 22:07:18 +00:00
JustAnotherArchivist
90f9598ecc
Adjust to Twitter's new method of handing out guest tokens
...
Fixes #64
2020-06-24 21:22:58 +00:00
JustAnotherArchivist
7b3c7deb28
Catch login redirects on Instagram
2020-05-30 00:56:34 +00:00
JustAnotherArchivist
040a11656c
Update README
2020-05-30 00:53:52 +00:00
JustAnotherArchivist
1459245258
Consistently raise ScraperException on fatal errors
2020-05-30 00:53:49 +00:00
JustAnotherArchivist
dbe4c5ce55
Remove Google+ module
...
Google+ was mostly shut down in early 2019. What remained (Google+ for G Suite) was renamed to Google Currents and is for internal communication only (and therefore out of scope for snscrape).
2020-05-30 00:35:06 +00:00
JustAnotherArchivist
80491ecc2c
Remove Gab module
...
Since Gab's move to a fork of Mastodon in July 2019, the module had been broken, and a new module would better be written from scratch as the platform changed entirely.
2020-05-30 00:23:33 +00:00
JustAnotherArchivist
1a71b58101
Add support for Telegram
...
Closes #50
2020-05-29 23:44:01 +00:00
JustAnotherArchivist
0ce37a69d4
Log exception details on crashes
2020-05-29 22:29:23 +00:00
JustAnotherArchivist
722bfd5f7c
Handle Twitter tombstones
...
Fixes #63
2020-05-29 22:12:37 +00:00
JustAnotherArchivist
b6cc3180d9
Force TwitterThreadScraper and TwitterListMembersScraper to fetch the old design
2020-03-04 00:40:49 +00:00
JustAnotherArchivist
613395d1c2
Port TwitterSearchScraper to redesign
...
Fixes #57
2020-03-04 00:40:49 +00:00
JustAnotherArchivist
82a87b7b5a
Merge pull request #53 from JackDallas/add-more-insta-fields
...
Add more fields to the instagram scraper
2020-02-09 23:48:59 +00:00
Jack Dallas
9568028bf9
Update changed fields
2020-02-07 11:30:16 +00:00
JustAnotherArchivist
6df351772e
Fix crash in Facebook scraper on link-less entries
2020-02-05 16:15:10 +00:00
JustAnotherArchivist
541173b0c8
Merge pull request #54 from jodizzle/fix/vkontakte-user
...
Fix vkontakte-user: pagination returns JSON now, and handle some unscrapable profiles.
2020-02-05 14:56:12 +00:00
Jody Leonard
b6772d3778
vkontakte-user: Handle additional un-scrapeable profile case
2019-10-31 16:01:29 -04:00
Jody Leonard
20ea117a2c
Fix vkontakte-user pagination
2019-10-30 22:29:49 -04:00
JackDallas
ff54c350bc
Add more fields to the instagram scraper
2019-08-30 12:43:02 +01:00
JustAnotherArchivist
e6aae35304
Use setuptools_scm for versioning through git tags
2019-07-01 17:41:18 +00:00
JustAnotherArchivist
b698a201f5
Update scraper list
2019-07-01 16:05:21 +00:00
JustAnotherArchivist
7fe72cf708
Add a note about reporting issues with proper debugging information
2019-07-01 16:01:11 +00:00
JustAnotherArchivist
4651cde447
Refactor CLI logging and add --dump-locals for better debugging
2019-07-01 15:46:10 +00:00
JustAnotherArchivist
c99cc4b5d3
Remove existing logging handlers
2019-07-01 15:42:06 +00:00
JustAnotherArchivist
628074d6fc
Print contents when ignoring a link-less entry
2019-07-01 01:35:00 +00:00
JustAnotherArchivist
64b293bd9e
Add support for media sets
...
Closes #48
2019-07-01 01:34:17 +00:00
JustAnotherArchivist
180f4dfeb7
Add support for photo.php URLs
...
Fixes #42
2019-06-30 18:36:39 +00:00
JustAnotherArchivist
6d6e3fa16c
Fix crash on (some?) inexistent groups
2019-06-30 18:36:30 +00:00
JustAnotherArchivist
5f7e6936c1
Add support for Facebook groups
...
Closes #47
2019-06-30 17:16:09 +00:00
JustAnotherArchivist
e2c05c9e0c
Split common code off into FacebookCommonScraper and refactor odd link detection in preparation of group scraping
2019-06-30 16:28:33 +00:00
JustAnotherArchivist
14e11b28d2
Add support for Twitter lists
...
Closes #46
2019-06-30 14:39:29 +00:00
JustAnotherArchivist
1a07b3b7e8
Add support for Twitter threads
2019-06-30 02:11:46 +00:00
JustAnotherArchivist
4d8cc7bdb9
Extract outlinks from Facebook
2019-06-27 15:29:05 +00:00
JustAnotherArchivist
eec83f181e
Check HTTP status code before attempting parsing
2019-06-27 15:25:26 +00:00
JustAnotherArchivist
fae7432c64
Log details about failed JSON parsing
2019-06-27 15:25:08 +00:00
JustAnotherArchivist
757818474d
Add tweet ID and username fields to Tweet items
2019-06-23 11:48:54 +00:00
JustAnotherArchivist
e6c934c0b8
Retrieve as many posts at once as possible for Instagram hashtags
2019-06-21 09:56:12 +00:00
JustAnotherArchivist
d2315feec1
Add support for Instagram locations
2019-06-21 09:55:30 +00:00
JustAnotherArchivist
765ceeeb10
More complete and more readable exception dump
2019-06-18 14:25:38 +00:00
JustAnotherArchivist
731a2e8c8b
Check that Instagram returned valid JSON, take 2
...
Fixes #22
2019-06-10 15:03:15 +00:00
JustAnotherArchivist
7d1916292c
Twitter: stop recursion based on whether the server returns the same position instead of detecting an empty feed
...
Fixes #37
2019-06-10 14:38:25 +00:00
JustAnotherArchivist
0d509c4ba0
Check that Instagram returned valid JSON ( fixes #22 )
2019-05-30 15:04:05 +00:00
JustAnotherArchivist
907a003a59
Fix crash when Twitter search produces no results ( fixes #41 )
2019-05-24 11:51:50 +00:00
JustAnotherArchivist
8ada279b57
Add warning if Twitter module gets no results
2019-05-24 11:50:39 +00:00
JustAnotherArchivist
900eae54a6
Ignore branded content link on Facebook silently
2019-05-24 11:49:44 +00:00
JustAnotherArchivist
7989af27b5
Handle tweets by temporarily blocked accounts (which show up in the search results but don't have a date or content)
2019-05-21 22:37:43 +00:00
JustAnotherArchivist
e528ca3f26
Dump locals only for snscrape modules ( closes #39 )
2019-05-18 01:08:49 +00:00
JustAnotherArchivist
32a427dac3
Fix pagination on Twitter ( fixes #40 )
2019-05-18 01:08:00 +00:00
JustAnotherArchivist
7001983556
Skip timeline entries that don't have a link ( fixes #36 )
2019-05-16 23:17:46 +00:00
JustAnotherArchivist
64438afc92
Work around tweet URLs that don't have a data-expanded-url attribute ( fixes #38 )
2019-05-16 22:51:22 +00:00
