mirror of
https://github.com/bellingcat/tiktok-hashtag-analysis.git
synced 2026-06-08 03:18:31 +03:00
merged changes
This commit is contained in:
Tristan Lee
140
README.md
140
README.md
@@ -2,62 +2,64 @@
|
||||
The tool helps to download posts and videos from TikTok for a given set of hashtags. It uses the [tiktok-scraper](https://github.com/drawrowfly/tiktok-scraper) Node package to download the posts and videos.
|
||||
|
||||
## Pre-requisites
|
||||
1. Make sure you have Python 3.6 or a later version installed
|
||||
2. Install the [Pipenv](https://pipenv.pypa.io/en/latest/) Python package using the command:
|
||||
1. Make sure you have Python 3.6 or a later version installed.
|
||||
2. Download and install TikTok scraper: https://github.com/drawrowfly/tiktok-scraper
|
||||
3. (Optional) create and activate a virtual environment for this tool, for example by executing the following command, which creates the `env` virtual environment:
|
||||
|
||||
`pip3 install pipenv`
|
||||
`python3 -m venv env`
|
||||
|
||||
3. Install the dependencies of this tool using the command:
|
||||
|
||||
`pipenv install`
|
||||
4. Install the Python package dependencies for this tool by executing the command:
|
||||
|
||||
3. Download and install [TikTok scraper](https://github.com/drawrowfly/tiktok-scraper)
|
||||
`pip install -r requirements.txt`
|
||||
|
||||
### Options for running run_downloader.py
|
||||
## About the tool
|
||||
### Command-line arguments
|
||||
```
|
||||
$ python run_downloader.py -h
|
||||
usage: run_downloader.py [-h] [-t [T [T ...]]] [-f F] [-p] [-v]
|
||||
|
||||
$ python run_downloader.py -h
|
||||
usage: run_downloader.py [-h] [-t [T [T ...]]] [-f F] [-p] [-v]
|
||||
Download the tiktoks for the requested hashtags
|
||||
|
||||
Download the tiktoks for the requested hashtags
|
||||
optional arguments:
|
||||
-h, --help show this help message and exit
|
||||
-t [T [T ...]] List of hashtags
|
||||
-f F File name with the list of hashtags
|
||||
-p Download posts
|
||||
-v Download videos
|
||||
```
|
||||
|
||||
optional arguments:
|
||||
-h, --help show this help message and exit
|
||||
-t [T [T ...]] List of hashtags
|
||||
-f F File name with the list of hashtags
|
||||
-p Download posts
|
||||
-v Download videos
|
||||
### Structure of output data
|
||||
```
|
||||
$ tree ../data
|
||||
../data
|
||||
├── ids
|
||||
│ └── post_ids.json
|
||||
├── log
|
||||
│ └── log.json
|
||||
├── london
|
||||
│ └── posts
|
||||
│ └── data.json
|
||||
├── newyork
|
||||
│ └── posts
|
||||
│ └── data.json
|
||||
└── paris
|
||||
└── posts
|
||||
└── data.json
|
||||
```
|
||||
|
||||
### Data organization
|
||||
|
||||
$ tree ../data
|
||||
../data
|
||||
├── ids
|
||||
│ └── post_ids.json
|
||||
├── log
|
||||
│ └── log.json
|
||||
├── london
|
||||
│ └── posts
|
||||
│ └── data.json
|
||||
├── newyork
|
||||
│ └── posts
|
||||
│ └── data.json
|
||||
└── paris
|
||||
└── posts
|
||||
└── data.json
|
||||
|
||||
The `data` folder contains all the downloaded data as shown in the picture above.
|
||||
1. the `log` folder contains log.json which records the total number of downloaded posts and videos for the hashtags against the time stamp of when the script is run.
|
||||
2. the `ids` folder contains two files `post_ids.json` and `video_ids.json` that records the ids of the downloaded posts and videos for each hashtag.
|
||||
3. Each hashtag has a folder with two subfolders `posts` and `videos` that store posts and videos respectively. The posts are stored in the `data.json` file in the `posts` folder, and videos are stored as the `.mp4` files in the `videos` folder.
|
||||
The `data` folder contains all the downloaded data as shown in the tree diagram above.
|
||||
- The `log` folder contains the `log.json` file, which records the total number of downloaded posts and videos for the hashtags against the timestamp of when the script was run.
|
||||
- The `ids` folder contains two files `post_ids.json` and `video_ids.json` that record the ids of the downloaded posts and videos for each hashtag.
|
||||
- Each hashtag has a folder with two subfolders `posts` and `videos` that store posts and videos respectively. The posts are stored in the `data.json` file in the `posts` folder, and videos are stored as the `.mp4` files in the `videos` folder.
|
||||
|
||||
|
||||
|
||||
### Post download
|
||||
Run the run_downloader.py with the following option:
|
||||
## How to use
|
||||
### Post downloading
|
||||
Running the `run_downloader.py` script with the following options will scrape posts containing the hashtags `#london`, `#paris`, or `#newyork`:
|
||||
|
||||
python3 run_downloader.py -t london paris newyork -p
|
||||
|
||||
which will produce an output similar to the following log:
|
||||
and will produce an output similar to the following log:
|
||||
|
||||
$ python3 run_downloader.py -t london paris newyork -p
|
||||
['london', 'paris', 'newyork']
|
||||
@@ -75,29 +77,53 @@ which will produce an output similar to the following log:
|
||||
Total posts for the hashtag newyork are: 941
|
||||
Successfully logged 2864 entries!!!!
|
||||
|
||||
1. The `-t` option allows to type in a space-separated list of hashtags as a command line argument.
|
||||
2. The `-p` option specifies the download posts option.
|
||||
- The `-t` flag allows a space-separated list of hashtags to be specified as a command line argument
|
||||
- The `-p` flag specifies that posts, not videos, will be downloaded
|
||||
|
||||
|
||||
### Video download
|
||||
### Video downloading
|
||||
Running the `run_downloader.py` script with the following options will scrape trending videos containing the hashtags `#london`, `#paris`, or `#newyork`:
|
||||
` python3 run_downloader.py -t london -v`
|
||||
|
||||
1. The `-t` option allows to type in a space-separated list of hashtags as a command line argument.
|
||||
2. The `-v` option is for downloading the videos
|
||||
The above code download all the trending videos for the hashtag london. Note that video downloading is a time and data rate consuming task, as a result we strongly recommend using one hashtag at a time to avoid complications.
|
||||
- The `-t` flag allows a space-separated list of hashtags to be specified as a command line argument
|
||||
- The `-v` flag specifies that videos, not posts, will be downloaded
|
||||
|
||||
Note that video downloading is a time and data rate consuming task, as a result we strongly recommend using one hashtag at a time when using the `-v` flag to avoid complications.
|
||||
|
||||
## Analyzing results
|
||||
### Top n hashtag occurrences
|
||||
In the analytics folder, the file `hashtag_frequencies.py` will plot the frequencies of top occurring hashtags in a given set of posts.
|
||||
Assume we want to plot the graph of top 20 occurring hashtags in the downloaded posts of the hashtag london.
|
||||
The script `hashtag_frequencies.py` analyzes the frequencies of top occurring hashtags in a given set of posts.
|
||||
|
||||
1. Plotting the saving the image as a png file: ` python3 hashtag_frequencies.py -p ../data/london/posts/data.json 20`
|
||||
```
|
||||
python hashtag_frequencies.py --help
|
||||
usage: hashtag_frequencies.py [-h] [-p] [-d] input_file n
|
||||
|
||||
<img width="1390" alt="Screenshot 2022-02-25 at 19 45 40" src="https://user-images.githubusercontent.com/72805812/155770710-0d167bbb-4c44-44d2-ba1c-fa57026afea8.png">
|
||||
positional arguments:
|
||||
input_file The json hashtag file name
|
||||
n The number of top n occurrences
|
||||
|
||||
The figure above shows the top 20 occurring hashtags among all the posts downloaded for the hashtag london. Clearly, the highest occurrence will be of the hashtag london as the file `data/london/posts/data.json` contain all the posts with hashtag london.
|
||||
optional arguments:
|
||||
-h, --help show this help message and exit
|
||||
-p, --plot Plot the occurrences
|
||||
-d, --print List top n hashtags
|
||||
```
|
||||
|
||||
2. Printing the result in the shell: ` python3 hashtag_frequencies.py -d ../data/london/posts/data.json 20`
|
||||
Assume we want to analyze the top 20 occurring hashtags in the downloaded posts of the `#london` hashtag.
|
||||
|
||||
- The results can be plotted and saved as a PNG file by executing the following command:
|
||||
|
||||
`python3 hashtag_frequencies.py -p ../data/london/posts/data.json 20`
|
||||
|
||||
which will produce a figure similar to that shown below:
|
||||
|
||||
|
||||
|
||||
Clearly, the highest occurrence will be of the `#london` hashtag, as all posts in the file `data/london/posts/data.json` contain the hashtag `#london`.
|
||||
|
||||
- The results can be displayed in tabular form by executing the following command:
|
||||
|
||||
`python3 hashtag_frequencies.py -d ../data/london/posts/data.json 20`
|
||||
|
||||
which will produce a terminal output similar to the following:
|
||||
```
|
||||
Rank Hashtag Occurrences Frequency (Occurrences/Total-Posts(total_posts))
|
||||
0 london 962 1.0
|
||||
@@ -122,4 +148,4 @@ The figure above shows the top 20 occurring hashtags among all the posts downloa
|
||||
19 america 20 0.02079002079002079
|
||||
```
|
||||
|
||||
The same result of 1 is printed in the shell. The last column shows the ratio of the occurrence to the total posts.
|
||||
The `Frequency` column shows the ratio of the occurrence to the total number of downloaded posts.
|
||||
Reference in New Issue
Block a user