bellingcat/cisticola
1
0
Fork 0
mirror of https://github.com/bellingcat/cisticola.git synced 2026-06-08 03:18:34 +03:00
Code Issues Packages Projects Releases Wiki Activity
86 Commits 14 Branches 4 Tags
0099558c6895c62f34bb634ed4ba3ddff61fae06
Go to file
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE
Logan Williams 0099558c68 Merge pull request #26 from bellingcat/deferred-media-archiving 
Implemented deferred media archiving for all scrapers
2022-04-02 14:15:35 +02:00
cisticola
ensured that before being scraped, all channels are added to the database, preventing channel.platform_id from being null.
2022-04-01 17:03:02 -05:00
docs
refactored Gab scraper to use gabber instead of garc
2022-03-30 08:05:10 -05:00
tests
implemented deferred media archiving for all scrapers, and implemented tests for them. Refactored archiving methods of Instagram and Gettr scrapers to be able to use default archiving method
2022-04-01 01:30:49 -05:00
.gitignore
Add option to clear registered scrapers, necessary for tests
2022-03-31 16:17:35 +02:00
app.py
Catch exceptions in get_posts so that archiving continues despites errors
2022-03-31 20:27:18 +02:00
Pipfile
implemented deferred media archiving for all scrapers, and implemented tests for them. Refactored archiving methods of Instagram and Gettr scrapers to be able to use default archiving method
2022-04-01 01:30:49 -05:00
Pipfile.lock
merged main
2022-04-01 02:05:25 -05:00
pytest.ini
implemented deferred media archiving for all scrapers, and implemented tests for them. Refactored archiving methods of Instagram and Gettr scrapers to be able to use default archiving method
2022-04-01 01:30:49 -05:00
README.md
Add documentation generation with Sphinx
2022-02-21 17:52:38 +01:00
russian_telegram_ingest.py
renamed 'archive_media' and 'media' to avoid name collision, changed scope of test fixture controller to 'function' so that db is fresh for each executed test
2022-03-09 13:19:35 -06:00

README.md

Cisticola

Cisticola, the bird

Description
Coordinates scrapers and interfaces with database
open-source-researchpythonscraperservice
Readme 1 MiB
Languages
Python 99.8%
Shell 0.2%