bellingcat/cisticola
1
0
Fork 0
mirror of https://github.com/bellingcat/cisticola.git synced 2026-06-12 13:28:34 +03:00
Code Issues Packages Projects Releases Wiki Activity

updated documentation, minor fixes like excluding very long cookiestring from docs

Browse Source
This commit is contained in:
Tristan Lee
2023-08-03 01:59:30 -05:00
parent e2142966e7
commit b8ddc400f3
30 changed files with 64 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
docs/source/about.rst Normal file
View File

@@ -0,0 +1,23 @@
About Cisticola
===============
The *cisticola* application enables users to easily collect, process, and analyze large-scale data from several social media platforms.
Definitions
-----------
- *Platform*: a social media website, for example Telegram, YouTube, or Rumble.
- *Channel*: an account or group on a platform, for example Twitter users, Telegram private chat groups, YouTube channels, and Gab groups.
- *Post*: a single item created by a channel, for example a Telegram message, a Tweet, or a YouTube video. Posts can contain one or more media attachments.
- *Media*: a file uploaded to a platform by a channel as part of a post.
Components
----------
Cisticola has many components
- :py:mod:`cisticola.base`: contains Object Relational Mapping (ORM) dataclasses that imperatively map to pre-defined SQL tables
- :py:mod:`cisticola.scraper`: contains platform-specific modules for scraping raw data from platforms. For example, the :py:mod:`cisticola.scraper.bitchute` module extracts raw data from Bitchute.
- :py:mod:`cisticola.transformer`: contains platform-specific modules for converting raw data into a standardized, cross-platform format.
The data extracted by scrapers varies by platform, but typically includes media files attached to posts.
Separating the "scraping" and "transforming" steps is useful because it ensures that no data is thrown away during the transormation. There may be some fields in the raw data that aren't included in the transformed format, but could be found to be useful in the future.