Commit Graph

18 Commits

Author SHA1 Message Date
JustAnotherArchivist
5070953feb Skip private fields and properties on dataclass-to-JSON conversion 2022-01-06 02:08:48 +00:00
JustAnotherArchivist
267b7d0e32 Rename CLI classmethods 2022-01-05 02:27:09 +00:00
JustAnotherArchivist
1f1c1bd8af Fix docstring style 2021-12-14 20:05:51 +00:00
JustAnotherArchivist
a6b6f3faaa Throw an error on empty arguments
Fixes #290
2021-10-10 17:43:27 +00:00
JustAnotherArchivist
5e829e2541 Refactor class instantiation to remove the need to repeat 'retries' everywhere 2021-09-30 09:58:10 +00:00
JustAnotherArchivist
821326bcfb Fix a few f-strings 2021-09-30 09:23:56 +00:00
JustAnotherArchivist
ed117e8891 Log response status code and redirects 2021-08-29 18:26:00 +00:00
JustAnotherArchivist
460be9d581 Add _type attribute on all JSON objects, remove separate attribute on Twitter media 2021-05-22 18:14:54 +00:00
JustAnotherArchivist
fdc33d0dba Include properties in JSON representation
This fixes the lack of the profile URL on Twitter users because it's generated using the username rather than set explicitly as a field.
2020-11-05 05:55:26 +00:00
JustAnotherArchivist
16ebe8bf48 Introduce dedicated IntWithGranularity type and deprecate the direct *Granularity fields 2020-10-16 18:20:47 +00:00
JustAnotherArchivist
1bbe25647a Refactor deprecated properties 2020-10-16 18:11:52 +00:00
JustAnotherArchivist
bd53e729a0 Replace named tuples with dataclasses and move JSON conversion logic to the base classes
Named tuples were never really adequate for this since the order aspect of them doesn't make sense.
Further, named tuples don't support multiple inheritance. This meant that the objects returned by get_items() were not actually Items, for example. Since Python 3.9, such named tuples cannot be created anymore.

Fixes #111
2020-10-15 23:44:28 +00:00
JustAnotherArchivist
ffd9289edc Reduce the logging level of retryable retrieval errors from WARNING to INFO
There is no real need to report these as WARNINGs as snscrape tries and in most cases manages to recover. Without --verbose, snscrape's output can be confusing (see #76). If the retries fail as well, snscrape will still log that as an ERROR and crash loudly.
2020-10-11 22:29:27 +00:00
JustAnotherArchivist
800cfd5be0 Add support for disabling following redirects 2020-09-13 00:52:26 +00:00
JustAnotherArchivist
1a2e367a87 Cache entities 2020-09-01 02:34:21 +00:00
JustAnotherArchivist
dd25fd0526 Add support for extracting the entity behind a scrape
Closes #11

Backwards incompatibility: snscrape.modules.twitter.Account is now called User. However, this was previously only used on the list member scraper, which has been broken for a while since the list member list is no longer publicly accessible.

For compatibility reasons, the CLI does not output the entity by default; the new option --with-entity enables it.
2020-08-24 01:38:27 +00:00
JustAnotherArchivist
d0d3c8b2a6 Better log output for temporary failures (fixes #2) 2018-10-01 03:24:29 +02:00
JustAnotherArchivist
6b6ae3d33b Rename from socialmediascraper to snscrape 2018-08-21 22:54:14 +02:00