JustAnotherArchivist
eec83f181e
Check HTTP status code before attempting parsing
2019-06-27 15:25:26 +00:00
JustAnotherArchivist
fae7432c64
Log details about failed JSON parsing
2019-06-27 15:25:08 +00:00
JustAnotherArchivist
757818474d
Add tweet ID and username fields to Tweet items
2019-06-23 11:48:54 +00:00
JustAnotherArchivist
e6c934c0b8
Retrieve as many posts at once as possible for Instagram hashtags
2019-06-21 09:56:12 +00:00
JustAnotherArchivist
d2315feec1
Add support for Instagram locations
2019-06-21 09:55:30 +00:00
JustAnotherArchivist
765ceeeb10
More complete and more readable exception dump
2019-06-18 14:25:38 +00:00
JustAnotherArchivist
731a2e8c8b
Check that Instagram returned valid JSON, take 2
...
Fixes #22
2019-06-10 15:03:15 +00:00
JustAnotherArchivist
7d1916292c
Twitter: stop recursion based on whether the server returns the same position instead of detecting an empty feed
...
Fixes #37
2019-06-10 14:38:25 +00:00
JustAnotherArchivist
0d509c4ba0
Check that Instagram returned valid JSON ( fixes #22 )
2019-05-30 15:04:05 +00:00
JustAnotherArchivist
907a003a59
Fix crash when Twitter search produces no results ( fixes #41 )
2019-05-24 11:51:50 +00:00
JustAnotherArchivist
8ada279b57
Add warning if Twitter module gets no results
2019-05-24 11:50:39 +00:00
JustAnotherArchivist
900eae54a6
Ignore branded content link on Facebook silently
2019-05-24 11:49:44 +00:00
JustAnotherArchivist
7989af27b5
Handle tweets by temporarily blocked accounts (which show up in the search results but don't have a date or content)
2019-05-21 22:37:43 +00:00
JustAnotherArchivist
e528ca3f26
Dump locals only for snscrape modules ( closes #39 )
2019-05-18 01:08:49 +00:00
JustAnotherArchivist
32a427dac3
Fix pagination on Twitter ( fixes #40 )
2019-05-18 01:08:00 +00:00
JustAnotherArchivist
7001983556
Skip timeline entries that don't have a link ( fixes #36 )
2019-05-16 23:17:46 +00:00
JustAnotherArchivist
64438afc92
Work around tweet URLs that don't have a data-expanded-url attribute ( fixes #38 )
2019-05-16 22:51:22 +00:00
JustAnotherArchivist
9e6538556a
Dump also the deeper frames, not just the get_items one
2019-05-16 22:48:35 +00:00
JustAnotherArchivist
9c8bbf051c
Fix order of processing in Twitter module for more useful locals dump output
2019-05-16 22:22:53 +00:00
JustAnotherArchivist
c6a11298ac
Fix missing linebreak in locals dump output
2019-05-16 22:22:21 +00:00
JustAnotherArchivist
02cbf6ddf6
Dump locals to a temporary file in case of an exception
2019-05-16 18:29:30 +00:00
JustAnotherArchivist
3817aa59d4
Add support for extracting links from tweets (including cards)
...
Both the t.co and the original URLs can be extracted. Note that card links are always t.co since Twitter's HTML does not include the original URL for those.
2019-05-16 16:42:52 +00:00
JustAnotherArchivist
46a51008f8
Fix Instagram signature calculation
2019-05-16 16:19:51 +00:00
JustAnotherArchivist
f91979eb32
Add --max-position option to twitter-search scraper as a workaround for pagination stopping early ( #37 )
...
The value needs to be of the format 'TWEET-<seenID>-<newestID>' where <seenID> is the last result that was returned by a previous scrape and <newestID> is the first result returned by the initial scrape.
2019-05-10 17:30:15 +00:00
JustAnotherArchivist
85fff319bc
Disable Twitter's spelling correction
...
src=typd means "this is what was typed in and could be incorrect". src=spxr is "no, I really mean that". src=sprv appears to be an alias of spxr that is no longer used.
2019-05-10 16:43:59 +00:00
JustAnotherArchivist
6b145526b7
Update README with new modules
2019-04-21 23:10:32 +02:00
JustAnotherArchivist
abf31764b1
Version 0.2.0
2019-04-21 23:03:21 +02:00
JustAnotherArchivist
64693f74bb
Update Instagram query hash
2019-04-19 01:47:38 +02:00
JustAnotherArchivist
a7d08ed51c
Remove leftover debugging print
2019-04-19 01:40:29 +02:00
JustAnotherArchivist
f48ca7726e
Add support for Gab
2019-04-19 00:40:43 +02:00
JustAnotherArchivist
78c295f7e0
Add support for VKontakte ( fixes #13 )
2019-04-18 18:39:21 +02:00
JustAnotherArchivist
a5aca1a14f
Add support for Instagram hashtags ( fixes #29 )
2019-04-18 16:14:54 +02:00
JustAnotherArchivist
96f7d871c1
Ignore Scraper subclasses which don't set a name
2019-04-18 16:14:26 +02:00
JustAnotherArchivist
b5dfd37949
Support unix timestamps in --since
2019-04-18 16:01:35 +02:00
JustAnotherArchivist
b511397791
Add --since option to return only results newer than a certain date ( fixes #19 )
2019-04-18 15:12:29 +02:00
JustAnotherArchivist
536fcb3303
Return proper items from scrapers including clean URLs ( fixes #9 and #10 )
2019-04-18 14:44:21 +02:00
JustAnotherArchivist
f8d812f799
Include permalink.php, events, and notes ( fixes #32 )
2019-04-18 04:22:47 +02:00
JustAnotherArchivist
c2cebd9166
Accept-Language header to get an English response unconditionally
2019-04-18 03:58:37 +02:00
JustAnotherArchivist
73bc99596f
Treat Twitter responses without a Content-Type header as invalid ( fixes #21 )
2019-04-18 02:24:35 +02:00
JustAnotherArchivist
8458c12218
Rewrite link extraction on Facebook ( fixes #17 )
...
Facebook's returned HTML has a large number of inconsistencies; some (most) pages include a <link rel="canonical" /> but some don't, for example. This was at the root of the failing post extraction for some Facebook pages (#17 ). The previous link extraction technique was also quite poor for other reasons though. The new method uses the relevant CSS classes instead. Despite probably being the result of a CSS minimiser or similar, these seem to be quite stable: they haven't changed in the past two years (but the more readable ones have!).
2019-04-18 02:14:21 +02:00
JustAnotherArchivist
b59c7e8d8f
Merge pull request #28 from peterk/master
...
Adds socks proxy support (via requests)
2019-03-11 13:32:07 +01:00
Peter Krantz
3ceb849d98
Adds socks proxy support (via requests)
2019-01-10 22:54:42 +01:00
JustAnotherArchivist
f5ee1f7ac5
Merge pull request #26 from ludios/avoid-twitter-bans
...
twitter: randomize user agent to avoid Twitter's (IP, UA)-keyed bans
2018-12-25 02:19:17 +01:00
Ivan Kozik
1984110f78
twitter: randomize user agent to avoid Twitter's (IP, UA)-keyed bans
2018-12-24 08:03:33 +00:00
JustAnotherArchivist
c5a5dcb92c
snscrape is now on PyPI
2018-10-09 17:26:03 +02:00
JustAnotherArchivist
cfb1c9a2aa
Version 0.1.3
2018-10-01 03:26:22 +02:00
JustAnotherArchivist
d0d3c8b2a6
Better log output for temporary failures ( fixes #2 )
2018-10-01 03:24:29 +02:00
JustAnotherArchivist
4d0350e541
Disable "quality filter" on Twitter ( fixes #3 )
2018-10-01 02:51:33 +02:00
JustAnotherArchivist
d17aa15bcb
Version 0.1.2
2018-09-11 12:44:07 +02:00
JustAnotherArchivist
d1ef280d6e
Fix snscrape.modules not getting installed
2018-09-11 12:43:10 +02:00
