21 Commits

Author SHA1 Message Date
Miguel Sozinho Ramalho
f8c25a3cef Update README.md 2023-12-22 13:31:11 +00:00
Richard Mwewa
d73ffd8acb Update __init__.py 2023-12-03 21:32:33 +02:00
Richard Mwewa
1a95af33ea Update pyproject.toml 2023-12-03 21:32:08 +02:00
Richard Mwewa
1dfefd9ef1 Update README.md 2023-12-03 21:31:47 +02:00
Richard Mwewa
7d5894224b Delete RPST GUI/RPST/ApiHandler.vb 2023-12-03 18:57:08 +00:00
Richard Mwewa
981fbfcac1 Add files via upload 2023-12-03 18:55:21 +00:00
Richard Mwewa
145d33ef9e Add files via upload 2023-12-03 18:54:23 +00:00
Richard Mwewa
c544ede53b Delete RPST GUI/RPST directory 2023-12-03 20:50:29 +02:00
Richard Mwewa
1c2d114b0e Delete rpst/rpst.py 2023-12-03 20:49:36 +02:00
Richard Mwewa
5dbc752056 Merge pull request #21 from bellingcat/dev
Dev
2023-12-03 20:49:18 +02:00
rly0nheart
4ae45320c7 Resolve conflict 2023-12-03 20:47:27 +02:00
rly0nheart
c25d3942ec Added setup project for easy installation. Scraping more than 100 posts. Fully Async. Major code improvements and optimisations 2023-12-03 18:37:13 +00:00
rly0nheart
6611cc2023 Update 2023-12-03 18:35:46 +00:00
rly0nheart
9abc351ffa Added setup project for easy installation. Scraping more than 100 posts. Fully Async. Major code improvements and optimisations 2023-12-03 18:32:10 +00:00
rly0nheart
f92efe640e Update rpst 2023-12-03 18:14:51 +02:00
rly0nheart
68253a6986 Update README.md 2023-12-03 18:12:04 +02:00
Richard Mwewa
1cf78c3609 Update README.md 2023-12-03 17:52:16 +02:00
rly0nheart
ff905764cf Scraping more than 100 posts. Code refactor and optimisation. Improved the file writer and update checker functions 2023-12-03 17:39:32 +02:00
Richard Mwewa
76a6dec671 Merge pull request #20 from bellingcat/galen-endpoint-fix
Add subdomain to Reddit endpoint to avoid redirect
2023-11-30 15:59:07 +02:00
Galen Reich
10fa0688a5 Add explicit www. to api endpoint 2023-11-30 11:32:45 +00:00
Galen Reich
ad547b4eaf Add explicit www. to reddit endpoint 2023-11-30 11:32:05 +00:00
54 changed files with 1976 additions and 6012 deletions

View File

@@ -1,42 +1,47 @@
# Note
> ## Use [Knew Karma](https://github.com/bellingcat/knewkarma) for more advanced and improved features.
## Note
> Use [Knew Karma](https://pypi.org/project/knewkarma) for more advanced and improved features.
![rpst](https://github.com/bellingcat/reddit-post-scraping-tool/assets/74001397/b9ec50b2-d2cb-419f-b8f0-d170b0630875)
# RPST (Reddit Post Scraping Tool)
Retrieve **Reddit** posts that contain the specified keyword from a specified subreddit.
[![Upload Python Package](https://github.com/bellingcat/reddit-post-scraping-tool/actions/workflows/python-publish.yml/badge.svg)](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/python-publish.yml) [![CodeQL](https://github.com/bellingcat/reddit-post-scraping-tool/actions/workflows/codeql.yml/badge.svg)](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/codeql.yml) ![.Net](https://img.shields.io/badge/.NET-5C2D91?style=flat&logo=.net&logoColor=white) ![Python](https://img.shields.io/badge/python-3670A0?style=flat&logo=python&logoColor=ffdd54)
Retrieve **Reddit** posts that contain the specified **keyword** from a specified **subreddit**.
[![.Net](https://img.shields.io/badge/Visual%20Basic%20.NET-5C2D91?style=flat&logo=.net&logoColor=white)](https://github.com/search?q=repo%3Abellingcat%2Freddit-post-scraping-tool++language%3A%22Visual+Basic+.NET%22&type=code) [![Python](https://img.shields.io/badge/Python-3670A0?style=flat&logo=python&logoColor=ffdd54)](https://github.com/search?q=repo%3Abellingcat%2Freddit-post-scraping-tool++language%3APython&type=code) [![Docker](https://img.shields.io/badge/Dockefile-%230db7ed.svg?style=flat&logo=docker&logoColor=white)](https://github.com/search?q=repo%3Abellingcat%2Freddit-post-scraping-tool++language%3ADockerfile&type=code) [![PyPI - Version](https://img.shields.io/pypi/v/reddit-post-scraping-tool?style=flat&logo=pypi&logoColor=ffdd54&label=PyPI&labelColor=3670A0&color=3670A0)](https://pypi.org/project/reddit-post-scraping-tool) [![BuyMeACoffee](https://img.shields.io/badge/Buy%20Me%20a%20Coffee-ffdd00?style=flat&logo=buy-me-a-coffee&logoColor=black)](https://buymeacoffee.com/_rly0nheart)
# ✅ Features
## *GUI*
- [x] Dark mode (*Right-click>Settings>Dark Mode*).
- [x] Saves results to a JSON/CSV file (*Right-click>Settings>Save posts>to JSON/to CSV*).
- [x] Logs errors to a file.
- [x] In-App feature to check for Updates.
## *CLI*
- [x] Saves results to JSON (*specifiy* `--json`).
- [x] Saves results to CSV (*specify* `--csv`).
- [x] Automatically checks for new updates, and notifies user if updates were found.
# 📃 TODO
## *GUI*
- [ ] Make it installable with a setup.exe/setup.msi file.
# 🖥️ Tested environments
## *GUI*
- [x] Microsoft Windows 11
## *CLI*
- [x] Android Termux
- [x] Microsoft Windows 11
- [x] Ubuntu 22.04 - latest versions
# 📖 Wiki
[Refer to the Wiki](https://github.com/bellingcat/reddit-post-scraping-tool/wiki) for installation instructions, in addition to all other documentation.
# 📖 Documentation
[Refer to the Wiki](https://github.com/bellingcat/reddit-post-scraping-tool/wiki) for installation instructions, in
addition to all other documentation.
# 🖼️ Screenshots
You can view a collection of screenshots for both the *CLI* and *GUI* [here](https://github.com/bellingcat/reddit-post-scraping-tool/tree/master/images)
***
<a href="https://www.buymeacoffee.com/_rly0nheart"><img src="https://img.buymeacoffee.com/button-api/?text=Buy me a coffee&emoji=&slug=_rly0nheart&button_colour=40DCA5&font_colour=ffffff&font_family=Comic&outline_colour=000000&coffee_colour=FFDD00" /></a>
![me](https://github.com/bellingcat/reddit-post-scraping-tool/assets/74001397/21e0bb33-7a84-45d6-92ba-00e40891ba31)
[![me](https://github.com/bellingcat/knewkarma/assets/74001397/efd19c7e-9840-4969-b33c-04087e73e4da)](https://about.me/rly0nheart)

View File

@@ -5,6 +5,8 @@ VisualStudioVersion = 17.4.33213.308
MinimumVisualStudioVersion = 10.0.40219.1
Project("{778DAE3C-4631-46EA-AA77-85C1314464D9}") = "RPST", "RPST\RPST.vbproj", "{46C2541E-6F65-461A-A479-F65D445C36EA}"
EndProject
Project("{54435603-DBB4-11D2-8724-00A0C9A8B90C}") = "RPSTSetup", "RPSTSetup\RPSTSetup.vdproj", "{7D89A26E-2D54-4BB7-B9C4-E1382E657DEA}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
@@ -15,6 +17,8 @@ Global
{46C2541E-6F65-461A-A479-F65D445C36EA}.Debug|Any CPU.Build.0 = Debug|Any CPU
{46C2541E-6F65-461A-A479-F65D445C36EA}.Release|Any CPU.ActiveCfg = Release|Any CPU
{46C2541E-6F65-461A-A479-F65D445C36EA}.Release|Any CPU.Build.0 = Release|Any CPU
{7D89A26E-2D54-4BB7-B9C4-E1382E657DEA}.Debug|Any CPU.ActiveCfg = Debug
{7D89A26E-2D54-4BB7-B9C4-E1382E657DEA}.Release|Any CPU.ActiveCfg = Release
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE

View File

@@ -15,17 +15,46 @@ Public Class ApiHandler
''' Asyncrosnously scrape Reddit data.
''' </summary>
''' <returns>Json object containing scraped data.</returns>
Public Async Function ScrapeRedditAsync(subreddit As String, listing As String, limit As Integer, timeframe As String) As Task(Of JObject)
Dim ApiEndpoint As String = $"https://reddit.com/r/{subreddit}/{listing}.json?limit={limit}&t={timeframe}"
Return Await GetJObjectFromEndpointAsync(endpoint:=ApiEndpoint)
Public Async Function AsyncGetPosts(subreddit As String, listing As String, limit As Integer, timeframe As String) As Task(Of JArray)
Dim PostsEndpoint As String = $"https://www.reddit.com/r/{subreddit}/{listing}.json?limit={limit}&t={timeframe}"
Return Await PaginatedPosts(endpoint:=PostsEndpoint, limit:=limit)
End Function
''' <summary>
''' Retrieves posts in a paginated manner until the specified limit is reached.
''' </summary>
''' <param name="endpoint">The API endpoint for retrieving posts.</param>
''' <param name="limit">The limit on the number of posts to retrieve.</param>
''' <returns>A Task(Of JArray) representing the asynchronous operation, which upon completion returns a JArray of posts.</returns>
Private Async Function PaginatedPosts(endpoint As String, limit As Integer) As Task(Of JArray)
Dim allPosts As New JArray()
Dim lastPostId As String = ""
Dim useAfter As Boolean = limit > 100
While allPosts.Count < limit
Dim endpointWithAfter As String = If(useAfter And Not String.IsNullOrEmpty(lastPostId), $"{endpoint}&after={lastPostId}", endpoint)
Dim postsData As JObject = Await AsyncGetData(endpoint:=endpointWithAfter)
Dim postsChildren As JArray = postsData("data")("children")
If postsChildren.Count = 0 Then
Exit While
End If
allPosts.Merge(postsChildren)
lastPostId = postsChildren.Last("data")("id").ToString()
End While
Return allPosts
End Function
''' <summary>
''' Asyncrosnously gets remote version information from the repository release page.
''' </summary>
''' <returns>Json object containing update data.</returns>
Public Async Function CheckUpdatesAsync() As Task(Of JObject)
Return Await GetJObjectFromEndpointAsync(endpoint:=UpdatesEndpoint)
Return Await AsyncGetData(endpoint:=UpdatesEndpoint)
End Function
''' <summary>
@@ -33,7 +62,7 @@ Public Class ApiHandler
''' </summary>
''' <param name="endpoint">The URL endpoint to retrieve data from.</param>
''' <returns>A JObject containing the retrieved data.</returns>
Private Async Function GetJObjectFromEndpointAsync(endpoint As String) As Task(Of JObject)
Private Async Function AsyncGetData(endpoint As String) As Task(Of JObject)
Try
Using httpClient As New HttpClient()
httpClient.DefaultRequestHeaders.Add("User-Agent", Headers)

View File

@@ -0,0 +1,115 @@
Imports Newtonsoft.Json.Linq
Public Class PostsProcessor
Private Shared ReadOnly ApiHandler As New ApiHandler()
''' <summary>
''' Checks if the given Reddit post contains the given keyword in its text.
''' </summary>
''' <param name="post">The Reddit post to check.</param>
''' <param name="keyword">The keyword to check for.</param>
''' <returns>True if the post contains the keyword, False otherwise.</returns>
Public Shared Function PostContainsKeyword(post As JObject, keyword As String) As Boolean
Return post("data")("selftext").ToString.ToLower(Globalization.CultureInfo.InvariantCulture).Contains(keyword.ToLower(System.Globalization.CultureInfo.InvariantCulture))
End Function
''' <summary>
''' Collects user inputs, fetches Reddit posts based on the inputs, checks if posts contain the keyword, and saves posts to a JSON file if necessary.
''' </summary>
''' <param name="JSONToolStripMenuItem">Indicates whether to save the posts to a JSON file.</param>
''' <remarks>
''' This function initializes the DataGridView, iterates over each post, adds the posts containing the keyword to the DataGridView and updates the UI.
''' It also shows a message if the keyword was not found in any of the posts or if the inputs are empty.
''' </remarks>
Public Shared Async Sub ProcessRedditPosts(settings)
' Collect inputs from the user.
Dim inputs = Utilities.CollectInputs()
If inputs.HasValue Then
' Fetch Reddit posts based on the inputs.
Dim processor As New PostsProcessor()
Dim posts As JArray = Await ApiHandler.AsyncGetPosts(subreddit:=inputs.Value.Subreddit, listing:=inputs.Value.Listing, limit:=inputs.Value.Limit, timeframe:=inputs.Value.Timeframe)
Dim totalPosts As Integer = 0
Dim keywordFound As Boolean = False
Dim foundPosts As Integer = 0
Dim foundPostsList As New JArray
PostsWindow.DataGridViewPosts.Rows.Clear()
PostsWindow.DataGridViewPosts.Columns.Clear()
PostsWindow.DataGridViewPosts.Columns.Add("PostCount", "Index")
PostsWindow.DataGridViewPosts.Columns.Add("PostAuthor", "Author")
PostsWindow.DataGridViewPosts.Columns.Add("PostID", "ID")
PostsWindow.DataGridViewPosts.Columns.Add("PostTitle", "Title")
PostsWindow.DataGridViewPosts.Columns.Add("PostText", "Text")
PostsWindow.DataGridViewPosts.Columns.Add("PostSubreddit", "Subreddit")
PostsWindow.DataGridViewPosts.Columns.Add("SubredditVisibility", "Subreddit Type")
PostsWindow.DataGridViewPosts.Columns.Add("PostThumbnail", "Thumbnail")
PostsWindow.DataGridViewPosts.Columns.Add("PostIsNSFW", "NSFW")
PostsWindow.DataGridViewPosts.Columns.Add("PostIsGilded", "Is Gilded")
PostsWindow.DataGridViewPosts.Columns.Add("PostUpvotes", "Upvotes")
PostsWindow.DataGridViewPosts.Columns.Add("PostUpvoteRatio", "Upvote Ratio")
PostsWindow.DataGridViewPosts.Columns.Add("PostDownvotes", "Downvotes")
PostsWindow.DataGridViewPosts.Columns.Add("PostIsCrosspostable", "↪️ Is Shareable")
PostsWindow.DataGridViewPosts.Columns.Add("PostScore", "Score")
PostsWindow.DataGridViewPosts.Columns.Add("PostCategory", "Category")
PostsWindow.DataGridViewPosts.Columns.Add("PostDomain", "Domain")
PostsWindow.DataGridViewPosts.Columns.Add("PostPermalink", "Permalink")
PostsWindow.DataGridViewPosts.Columns.Add("PostCreatedAt", "Created At")
' Iterate over each post.
For Each post In posts
totalPosts += 1
' Check if the post contains the keyword
If PostsProcessor.PostContainsKeyword(post, inputs.Value.Keyword.ToLower(Globalization.CultureInfo.InvariantCulture)) Then
foundPosts += 1
foundPostsList.Add(post)
PostsWindow.DataGridViewPosts.Rows.Add(totalPosts,
post("data")("author"),
post("data")("id"),
post("data")("title"),
post("data")("selftext"),
post("data")("subreddit_name_prefixed"),
post("data")("subreddit_type"),
post("data")("thumbnail"),
post("data")("over_18"),
post("data")("gilded"),
post("data")("ups"),
post("data")("upvote_ratio"),
post("data")("downs"),
post("data")("is_crosspostable"),
post("data")("score"),
post("data")("category"),
post("data")("domain"),
post("data")("permalink"),
post("data")("created"))
PostsWindow.Text = $"Showing {foundPosts}/{inputs.Value.Limit} {inputs.Value.Listing} posts containing the word {inputs.Value.Keyword}, from r/{inputs.Value.Subreddit}"
PostsWindow.Show()
keywordFound = True
End If
Next
' Check if the keyword was found in any posts
If Not keywordFound Then
MessageBox.Show($"Keyword `{inputs.Value.Keyword}` was not found in any of the " + posts("data")("children").Count.ToString(Globalization.CultureInfo.InvariantCulture) _
+ $" {inputs.Value.Listing} posts from r/{inputs.Value.Subreddit}", "Not Found", MessageBoxButtons.OK, MessageBoxIcon.Warning)
End If
If settings.SaveToJson Then
' Save posts to a JSON file if SaveToJson is True.
Utilities.SavePostsToJson(posts:=foundPostsList)
End If
If settings.SaveToCsv Then
' Save posts to a CSV file if SaveToCsv is True.
Utilities.SavePostsToCSV(posts:=foundPostsList)
End If
Else
End If
End Sub
End Class

View File

@@ -13,7 +13,11 @@ Public Class SettingsManager
Public Property SaveToJson As Boolean
Public Property SaveToCsv As Boolean
Private ReadOnly settingsFilePath As String = Path.Combine(Environment.CurrentDirectory, "config.json")
Private ReadOnly settingsFilePath As String = Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData),
"RPST",
"settings.json"
)
''' <summary>
''' Loads application settings from the 'settings.json' file.
@@ -31,9 +35,9 @@ Public Class SettingsManager
SaveToJson = settings.SaveToJson
SaveToCsv = settings.SaveToCsv
FormMain.DarkModeToolStripMenuItem.Checked = settings.DarkMode
FormMain.ToJSONToolStripMenuItem.Checked = settings.SaveToJson
FormMain.ToCSVToolStripMenuItem.Checked = settings.SaveToCsv
MainWindow.DarkModeToolStripMenuItem.Checked = settings.DarkMode
MainWindow.ToJSONToolStripMenuItem.Checked = settings.SaveToJson
MainWindow.ToCSVToolStripMenuItem.Checked = settings.SaveToCsv
Else
' Settings file does not exist
' Create a new file with default settings 'False'
@@ -41,15 +45,19 @@ Public Class SettingsManager
Dim jsonOutput = JsonSerializer.Serialize(defaultSettings)
File.WriteAllText(settingsFilePath, jsonOutput)
DarkMode = False
SaveToJson = False
SaveToCsv = False
MainWindow.ToJSONToolStripMenuItem.Checked = False
MainWindow.ToCSVToolStripMenuItem.Checked = False
FormMain.ToJSONToolStripMenuItem.Checked = False
FormMain.ToCSVToolStripMenuItem.Checked = False
FormMain.DarkModeToolStripMenuItem.Checked = False
If Utilities.IsSystemDarkTheme() Then
DarkMode = True
MainWindow.DarkModeToolStripMenuItem.Checked = True
Else
DarkMode = False
MainWindow.DarkModeToolStripMenuItem.Checked = False
End If
End If
End Sub
@@ -94,10 +102,10 @@ Public Class SettingsManager
Dim settings As Dictionary(Of String, Object) = GetSettings()
' Apply the SaveToJson setting to the menu item checkbox
FormMain.ToJSONToolStripMenuItem.Checked = CBool(settings("SaveToJson"))
MainWindow.ToJSONToolStripMenuItem.Checked = CBool(settings("SaveToJson"))
' Apply the SaveToCsv setting to the menu item checkbox
FormMain.ToCSVToolStripMenuItem.Checked = CBool(settings("SaveToCsv"))
MainWindow.ToCSVToolStripMenuItem.Checked = CBool(settings("SaveToCsv"))
' Apply the color scheme based on the Dark Mode setting
ApplyColorScheme(isDarkMode:=CBool(settings("DarkMode")))
@@ -137,40 +145,40 @@ Public Class SettingsManager
End If
' Applying Main Form colors
FormMain.BackColor = colorMap("MainBackground")
FormMain.TextBoxKeyword.BackColor = colorMap("TextBoxBackground")
FormMain.TextBoxSubreddit.BackColor = colorMap("TextBoxBackground")
FormMain.NumericUpDownLimit.BackColor = colorMap("TextBoxBackground")
FormMain.ComboBoxListing.BackColor = colorMap("TextBoxBackground")
FormMain.ComboBoxTimeframe.BackColor = colorMap("TextBoxBackground")
FormMain.TextBoxKeyword.ForeColor = colorMap("Foreground")
FormMain.TextBoxSubreddit.ForeColor = colorMap("Foreground")
FormMain.NumericUpDownLimit.ForeColor = colorMap("Foreground")
FormMain.ComboBoxListing.ForeColor = colorMap("Foreground")
FormMain.ComboBoxTimeframe.ForeColor = colorMap("Foreground")
FormMain.LabelKeyword.ForeColor = colorMap("Foreground")
FormMain.LabelSubreddit.ForeColor = colorMap("Foreground")
FormMain.LabelLimit.ForeColor = colorMap("Foreground")
FormMain.LabelListing.ForeColor = colorMap("Foreground")
FormMain.LabelTimeframe.ForeColor = colorMap("Foreground")
MainWindow.BackColor = colorMap("MainBackground")
MainWindow.TextBoxKeyword.BackColor = colorMap("TextBoxBackground")
MainWindow.TextBoxSubreddit.BackColor = colorMap("TextBoxBackground")
MainWindow.NumericUpDownLimit.BackColor = colorMap("TextBoxBackground")
MainWindow.ComboBoxListing.BackColor = colorMap("TextBoxBackground")
MainWindow.ComboBoxTimeframe.BackColor = colorMap("TextBoxBackground")
MainWindow.TextBoxKeyword.ForeColor = colorMap("Foreground")
MainWindow.TextBoxSubreddit.ForeColor = colorMap("Foreground")
MainWindow.NumericUpDownLimit.ForeColor = colorMap("Foreground")
MainWindow.ComboBoxListing.ForeColor = colorMap("Foreground")
MainWindow.ComboBoxTimeframe.ForeColor = colorMap("Foreground")
MainWindow.LabelKeyword.ForeColor = colorMap("Foreground")
MainWindow.LabelSubreddit.ForeColor = colorMap("Foreground")
MainWindow.LabelLimit.ForeColor = colorMap("Foreground")
MainWindow.LabelListing.ForeColor = colorMap("Foreground")
MainWindow.LabelTimeframe.ForeColor = colorMap("Foreground")
' Applying Right-Click Menu colors
FormMain.SettingsToolStripMenuItem.BackColor = colorMap("MenuBackground")
FormMain.DarkModeToolStripMenuItem.BackColor = colorMap("MenuBackground")
FormMain.SavePostsToolStripMenuItem.BackColor = colorMap("MenuBackground")
FormMain.ToJSONToolStripMenuItem.BackColor = colorMap("MenuBackground")
FormMain.ToCSVToolStripMenuItem.BackColor = colorMap("MenuBackground")
FormMain.AboutToolStripMenuItem.BackColor = colorMap("MenuBackground")
FormMain.CheckForUpdatesToolStripMenuItem.BackColor = colorMap("MenuBackground")
FormMain.QuitToolStripMenuItem.BackColor = colorMap("MenuBackground")
FormMain.SettingsToolStripMenuItem.ForeColor = colorMap("Foreground")
FormMain.DarkModeToolStripMenuItem.ForeColor = colorMap("Foreground")
FormMain.SavePostsToolStripMenuItem.ForeColor = colorMap("Foreground")
FormMain.ToJSONToolStripMenuItem.ForeColor = colorMap("Foreground")
FormMain.ToCSVToolStripMenuItem.ForeColor = colorMap("Foreground")
FormMain.AboutToolStripMenuItem.ForeColor = colorMap("Foreground")
FormMain.CheckForUpdatesToolStripMenuItem.ForeColor = colorMap("Foreground")
FormMain.QuitToolStripMenuItem.ForeColor = colorMap("Foreground")
MainWindow.SettingsToolStripMenuItem.BackColor = colorMap("MenuBackground")
MainWindow.DarkModeToolStripMenuItem.BackColor = colorMap("MenuBackground")
MainWindow.SavePostsToolStripMenuItem.BackColor = colorMap("MenuBackground")
MainWindow.ToJSONToolStripMenuItem.BackColor = colorMap("MenuBackground")
MainWindow.ToCSVToolStripMenuItem.BackColor = colorMap("MenuBackground")
MainWindow.AboutToolStripMenuItem.BackColor = colorMap("MenuBackground")
MainWindow.CheckForUpdatesToolStripMenuItem.BackColor = colorMap("MenuBackground")
MainWindow.QuitToolStripMenuItem.BackColor = colorMap("MenuBackground")
MainWindow.SettingsToolStripMenuItem.ForeColor = colorMap("Foreground")
MainWindow.DarkModeToolStripMenuItem.ForeColor = colorMap("Foreground")
MainWindow.SavePostsToolStripMenuItem.ForeColor = colorMap("Foreground")
MainWindow.ToJSONToolStripMenuItem.ForeColor = colorMap("Foreground")
MainWindow.ToCSVToolStripMenuItem.ForeColor = colorMap("Foreground")
MainWindow.AboutToolStripMenuItem.ForeColor = colorMap("Foreground")
MainWindow.CheckForUpdatesToolStripMenuItem.ForeColor = colorMap("Foreground")
MainWindow.QuitToolStripMenuItem.ForeColor = colorMap("Foreground")
' Applying About Box colors
AboutBox.BackColor = colorMap("AboutBackground")
@@ -186,9 +194,9 @@ Public Class SettingsManager
' Updating Dark Mode Text
If isDarkMode Then
FormMain.DarkModeToolStripMenuItem.Text = "Dark Mode: Enabled"
MainWindow.DarkModeToolStripMenuItem.Text = "Dark Mode: Disable"
Else
FormMain.DarkModeToolStripMenuItem.Text = "Dark Mode: Disabled"
MainWindow.DarkModeToolStripMenuItem.Text = "Dark Mode: Enable"
End If
End Sub

View File

@@ -1,8 +1,28 @@
Imports System.IO
Imports Microsoft.Win32
Imports Newtonsoft.Json
Imports Newtonsoft.Json.Linq
Public Class Utilities
''' <summary>
''' Determines if the Windows system theme is in dark mode.
''' </summary>
''' <returns>
''' True if the dark mode is enabled, otherwise false.
''' </returns>
Public Shared Function IsSystemDarkTheme() As Boolean
Dim registryKey As RegistryKey = Registry.CurrentUser.OpenSubKey(
"Software\Microsoft\Windows\CurrentVersion\Themes\Personalize"
)
If registryKey IsNot Nothing Then
Dim appsUseLightTheme As Object = registryKey.GetValue("AppsUseLightTheme")
Return appsUseLightTheme IsNot Nothing AndAlso CType(appsUseLightTheme, Integer) = 0
Else
Return False
End If
End Function
''' <summary>
''' Shows the license notice in a messagebox.
''' </summary>
@@ -10,9 +30,7 @@ Public Class Utilities
''' Result of the Dialog (Yes/No).
''' </returns>
Public Shared Function LicenseAgreement()
Dim result As DialogResult = MessageBox.Show($"MIT License
{My.Application.Info.Copyright}
Dim result As DialogResult = MessageBox.Show($"{My.Application.Info.Copyright}
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the ""Software""), to deal
@@ -29,7 +47,7 @@ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.", "License Agreement", MessageBoxButtons.OKCancel, MessageBoxIcon.Information)
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.", "MIT License", MessageBoxButtons.OK, MessageBoxIcon.Information)
Return result
End Function
@@ -44,7 +62,10 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SO
''' If the directories already exist, the function will not perform any actions.
''' </remarks>
Public Shared Sub PathFinder()
Dim directoryPath As String = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData), "RPST", "logs")
Dim directoryPath As String = Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData),
"RPST", "logs"
)
If Not Directory.Exists(directoryPath) Then
Directory.CreateDirectory(directoryPath)
@@ -57,8 +78,8 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SO
''' </summary>
''' <returns>
''' Tuple containing:
''' Keyword (String) - Keyword entered by user in theFormMain.
''' Subreddit (String) - Subreddit entered by user in theFormMain.
''' Keyword (String) - Keyword entered by user in theMainWindow.
''' Subreddit (String) - Subreddit entered by user in theMainWindow.
''' Listing (String) - Listing chosen by user in the StartForm, defaults to 'top' if none is selected.
''' Limit (Integer) - Limit entered by user in the StartForm, defaults to 10 if the entered value is over 100.
''' Timeframe (String) - Timeframe chosen by user in the StartForm, defaults to 'all' if none is selected.
@@ -67,12 +88,12 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SO
''' If keyword or subreddit are empty, Displays a warning and returns nothing.
''' </remarks>
Public Shared Function CollectInputs() As (Keyword As String, Subreddit As String, Listing As String, Limit As Integer, Timeframe As String)?
Dim keyword As String = FormMain.TextBoxKeyword.Text.Trim()
Dim subreddit As String = FormMain.TextBoxSubreddit.Text.Trim()
Dim keyword As String = MainWindow.TextBoxKeyword.Text.Trim()
Dim subreddit As String = MainWindow.TextBoxSubreddit.Text.Trim()
' Convert the Listing and Subreddit to lowercase using InvariantCulture.
Dim listing As String = If(String.IsNullOrEmpty(FormMain.ComboBoxListing.Text), "top", FormMain.ComboBoxListing.Text.ToLower(Globalization.CultureInfo.InvariantCulture).Trim())
Dim timeframe As String = If(String.IsNullOrEmpty(FormMain.ComboBoxTimeframe.Text), "all", FormMain.ComboBoxTimeframe.Text.ToLower(Globalization.CultureInfo.InvariantCulture).Trim())
Dim limit As Integer = FormMain.NumericUpDownLimit.Value
Dim listing As String = If(String.IsNullOrEmpty(MainWindow.ComboBoxListing.Text), "top", MainWindow.ComboBoxListing.Text.ToLower(Globalization.CultureInfo.InvariantCulture).Trim())
Dim timeframe As String = If(String.IsNullOrEmpty(MainWindow.ComboBoxTimeframe.Text), "all", MainWindow.ComboBoxTimeframe.Text.ToLower(Globalization.CultureInfo.InvariantCulture).Trim())
Dim limit As Integer = MainWindow.NumericUpDownLimit.Value
' Validate inputs.
If String.IsNullOrEmpty(keyword) AndAlso String.IsNullOrEmpty(subreddit) Then
@@ -149,38 +170,4 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SO
MessageBox.Show($"Posts saved to {fileName}", "Saved", MessageBoxButtons.OK, MessageBoxIcon.Information)
End If
End Sub
''' <summary>
''' Checks if the "launch.log" file exists in the directory: C:\Users\<username>\AppData\Roaming\RPSTl\logs.
''' </summary>
''' <remarks>
''' If the file doesn't exist, it shows a MessageBox with the License Agreement Notice with buttons Yes and No.
''' If the user clicks on the Yes button, it creates one the launch.log file, otherwise assume the user did not agree to the License and close the program.
''' The launc.log file is used to determine whether the program has been run before.
''' </remarks>
Public Shared Sub LogFirstTimeLaunch()
Dim filePath As String = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData), "RPST", "logs", "launch.log")
Dim textToWrite As String = $"
{My.Application.Info.AssemblyName}
-------------------------
User: {Environment.UserName}
Host: {Environment.MachineName}
OS: {Environment.OSVersion}
x64: {Environment.Is64BitOperatingSystem}
First launched on: {DateTime.Now}"
If Not File.Exists(filePath) Then
Dim result As DialogResult = LicenseAgreement()
If result = DialogResult.OK Then
File.WriteAllText(filePath, textToWrite)
Else
FormMain.Close()
End If
End If
End Sub
End Class

View File

@@ -1,69 +0,0 @@
Imports Newtonsoft.Json.Linq
Public Class DataGridViewHandler
''' <summary>
''' Initializes the DataGridView by clearing any existing data and setting up the necessary columns.
''' </summary>
''' <param name="dataGridView">The DataGridView to be initialized.</param>
Public Shared Sub AddColumn(dataGridView As DataGridView)
''' <summary>
''' Clear the Columns and Rows before adding Items to them.
''' <summary>
dataGridView.Rows.Clear()
dataGridView.Columns.Clear()
dataGridView.Columns.Add("PostCount", "🔢 Index")
dataGridView.Columns.Add("PostAuthor", "👤 Author")
dataGridView.Columns.Add("PostID", "🆔 ID")
dataGridView.Columns.Add("PostText", "📝 Text")
dataGridView.Columns.Add("PostSubreddit", "🫂 Subreddit")
dataGridView.Columns.Add("SubredditVisibility", "🫣 Visibility")
dataGridView.Columns.Add("PostThumbnail", "🖼️ Thumbnail")
dataGridView.Columns.Add("PostIsNSFW", "🔞 NSFW")
dataGridView.Columns.Add("PostIsGilded", "🥇 Gilded")
dataGridView.Columns.Add("PostUpvotes", "⬆️ Upvotes")
dataGridView.Columns.Add("PostUpvoteRatio", "📊 Upvote Ratio")
dataGridView.Columns.Add("PostDownvotes", "⬇️ Downvotes")
dataGridView.Columns.Add("PostAwards", "🏆 Awards")
dataGridView.Columns.Add("PostTopAward", "🏆 Top Award")
dataGridView.Columns.Add("PostIsCrosspostable", "↪️ Is cross-postable?")
dataGridView.Columns.Add("PostScore", "📈 Score")
dataGridView.Columns.Add("PostCategory", "🟢 Category")
dataGridView.Columns.Add("PostDomain", "🌐 Domain")
dataGridView.Columns.Add("PostPermalink", "🔗 Permalink")
dataGridView.Columns.Add("PostCreatedAt", "📅 Created At")
dataGridView.Columns.Add("PostApprovedAt", "📅 Approved At")
dataGridView.Columns.Add("PostApprovedBy", "👤 Approved By")
End Sub
Public Shared Sub AddRow(dataGridView As DataGridView, post As JObject, postNumber As Integer)
''' <summary>
''' Adds a row to the DataGridView based on the data from a Reddit post.
''' </summary>
''' <param name="dataGridView">The DataGridView to which the row will be added.</param>
''' <param name="post">A JObject representing the Reddit post.</param>
''' <param name="postNumber">The number of the post.</param>
dataGridView.Rows.Add(postNumber,
post("data")("author"),
post("data")("id"),
post("data")("selftext"),
post("data")("subreddit_name_prefixed"),
post("data")("subreddit_type"),
post("data")("thumbnail"),
post("data")("over_18"),
post("data")("gilded"),
post("data")("ups"),
post("data")("upvote_ratio"),
post("data")("downs"),
post("data")("total_awards_received"),
post("data")("top_awarded_type"),
post("data")("is_crosspostable"),
post("data")("score"),
post("data")("category"),
post("data")("domain"),
post("data")("permalink"),
post("data")("created"),
post("data")("approved_at_utc"),
post("data")("approved_by"))
End Sub
End Class

View File

@@ -1,58 +0,0 @@
<Global.Microsoft.VisualBasic.CompilerServices.DesignerGenerated()>
Partial Class FormPosts
Inherits System.Windows.Forms.Form
'Form overrides dispose to clean up the component list.
<System.Diagnostics.DebuggerNonUserCode()>
Protected Overrides Sub Dispose(ByVal disposing As Boolean)
Try
If disposing AndAlso components IsNot Nothing Then
components.Dispose()
End If
Finally
MyBase.Dispose(disposing)
End Try
End Sub
'Required by the Windows Form Designer
Private components As System.ComponentModel.IContainer
'NOTE: The following procedure is required by the Windows Form Designer
'It can be modified using the Windows Form Designer.
'Do not modify it using the code editor.
<System.Diagnostics.DebuggerStepThrough()>
Private Sub InitializeComponent()
Dim resources As ComponentModel.ComponentResourceManager = New ComponentModel.ComponentResourceManager(GetType(FormPosts))
DataGridViewPosts = New DataGridView()
CType(DataGridViewPosts, ComponentModel.ISupportInitialize).BeginInit()
SuspendLayout()
'
' DataGridViewPosts
'
DataGridViewPosts.BackgroundColor = Color.Gainsboro
DataGridViewPosts.ColumnHeadersHeightSizeMode = DataGridViewColumnHeadersHeightSizeMode.AutoSize
DataGridViewPosts.Dock = DockStyle.Fill
DataGridViewPosts.Location = New Point(0, 0)
DataGridViewPosts.Name = "DataGridViewPosts"
DataGridViewPosts.ReadOnly = True
DataGridViewPosts.RowHeadersVisible = False
DataGridViewPosts.RowTemplate.Height = 25
DataGridViewPosts.Size = New Size(501, 365)
DataGridViewPosts.TabIndex = 3
'
' FormPosts
'
AutoScaleDimensions = New SizeF(7F, 15F)
AutoScaleMode = AutoScaleMode.Font
ClientSize = New Size(501, 365)
Controls.Add(DataGridViewPosts)
Icon = CType(resources.GetObject("$this.Icon"), Icon)
Name = "FormPosts"
StartPosition = FormStartPosition.CenterScreen
Text = "Posts"
CType(DataGridViewPosts, ComponentModel.ISupportInitialize).EndInit()
ResumeLayout(False)
End Sub
Friend WithEvents DataGridViewPosts As DataGridView
End Class

View File

@@ -1,5 +0,0 @@
Public Class FormPosts
Private Sub FormResults_Load(sender As Object, e As EventArgs) Handles MyBase.Load
Me.Text = $"{Me.Text} - {FormMain.TextBoxKeyword.Text}, r/{FormMain.TextBoxSubreddit.Text}, {FormMain.NumericUpDownLimit.Text}, {FormMain.ComboBoxListing.Text}, {FormMain.ComboBoxTimeframe.Text}"
End Sub
End Class

19
RPST GUI/RPST/LICENSE.rtf Normal file
View File

@@ -0,0 +1,19 @@
{\rtf1\ansi\ansicpg1252\cocoartf2577
\cocoatextscaling0\cocoaplatform0{\fonttbl\f0\fswiss\fcharset0 Helvetica;}
{\colortbl;\red255\green255\blue255;}
{\*\expandedcolortbl;;}
\paperw11900\paperh16840\margl1440\margr1440\vieww13560\viewh17700\viewkind0
\pard\tx0\tx566\tx1133\tx1700\tx2267\tx2834\tx3401\tx3968\tx4535\tx5102\tx5669\tx6236\tx6803\pardirnatural\partightenfactor0
{
{\header The MIT License (MIT)}
The MIT License (MIT)\par
\fs20\li0\fi0 Copyright (c) 2023 Richard Mwewa
\par\par
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
\par\par
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
\par\par
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\par
}
}

View File

@@ -33,15 +33,10 @@ Namespace My
<Global.System.Diagnostics.DebuggerStepThroughAttribute()> _
Protected Overrides Sub OnCreateMainForm()
Me.MainForm = Global.RPST.FormMain
Me.MainForm = Global.RPST.MainWindow
End Sub
<Global.System.Diagnostics.DebuggerStepThroughAttribute()> _
Protected Overrides Sub OnCreateSplashScreen()
Me.SplashScreen = Global.RPST.SplashScreen
End Sub
<Global.System.Diagnostics.DebuggerStepThroughAttribute()> _
<Global.System.Diagnostics.DebuggerStepThroughAttribute()>
Protected Overrides Function OnInitialize(ByVal commandLineArgs As System.Collections.ObjectModel.ReadOnlyCollection(Of String)) As Boolean
Me.MinimumSplashScreenDisplayTime = 2000
Return MyBase.OnInitialize(commandLineArgs)

View File

@@ -1,12 +1,12 @@
<?xml version="1.0" encoding="utf-16"?>
<MyApplicationData xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<MySubMain>true</MySubMain>
<MainForm>Form1</MainForm>
<MainForm>MainWindow</MainForm>
<SingleInstance>false</SingleInstance>
<ShutdownMode>0</ShutdownMode>
<EnableVisualStyles>true</EnableVisualStyles>
<AuthenticationMode>0</AuthenticationMode>
<SaveMySettingsOnExit>true</SaveMySettingsOnExit>
<SplashScreen>SplashScreen</SplashScreen>
<SplashScreen></SplashScreen>
<MinimumSplashScreenDisplayTime>2000</MinimumSplashScreenDisplayTime>
</MyApplicationData>

View File

@@ -1,88 +0,0 @@
Imports Newtonsoft.Json.Linq
Public Class PostsProcessor
Private ReadOnly ApiHandler As New ApiHandler
''' <summary>
''' Asyncronously fetches Reddit posts based on the given parameters and returns them as a JObject.
''' </summary>
''' <param name="subreddit">The subreddit to fetch posts from.</param>
''' <param name="listing">The type of listing (e.g., "new", "top", etc.).</param>
''' <param name="limit">The maximum number of posts to fetch.</param>
''' <param name="timeframe">The timeframe to consider for the posts (e.g., "day", "week", "month", "year", "all").</param>
''' <returns>A JObject containing the fetched Reddit posts.</returns>
Public Async Function FetchPostsAsync(subreddit As String, listing As String, limit As Integer, timeframe As String) As Task(Of JObject)
Dim posts As JObject = Await ApiHandler.ScrapeRedditAsync(subreddit, listing, limit, timeframe)
Return posts
End Function
''' <summary>
''' Checks if the given Reddit post contains the given keyword in its text.
''' </summary>
''' <param name="post">The Reddit post to check.</param>
''' <param name="keyword">The keyword to check for.</param>
''' <returns>True if the post contains the keyword, False otherwise.</returns>
Public Shared Function PostContainsKeyword(post As JObject, keyword As String) As Boolean
Return post("data")("selftext").ToString.ToLower(Globalization.CultureInfo.InvariantCulture).Contains(keyword.ToLower(System.Globalization.CultureInfo.InvariantCulture))
End Function
''' <summary>
''' Collects user inputs, fetches Reddit posts based on the inputs, checks if posts contain the keyword, and saves posts to a JSON file if necessary.
''' </summary>
''' <param name="JSONToolStripMenuItem">Indicates whether to save the posts to a JSON file.</param>
''' <remarks>
''' This function initializes the DataGridView, iterates over each post, adds the posts containing the keyword to the DataGridView and updates the UI.
''' It also shows a message if the keyword was not found in any of the posts or if the inputs are empty.
''' </remarks>
Public Shared Async Sub ProcessRedditPosts(settings)
' Collect inputs from the user.
Dim inputs = Utilities.CollectInputs()
If inputs.HasValue Then
' Initialize the DataGridView.
DataGridViewHandler.AddColumn(FormPosts.DataGridViewPosts)
' Fetch Reddit posts based on the inputs.
Dim processor As New PostsProcessor()
Dim posts As JObject = Await processor.FetchPostsAsync(subreddit:=inputs.Value.Subreddit, listing:=inputs.Value.Listing, limit:=inputs.Value.Limit, timeframe:=inputs.Value.Timeframe)
Dim totalPosts As Integer = 0
Dim keywordFound As Boolean = False
Dim foundPosts As Integer = 0
Dim foundPostsList As New JArray
' Iterate over each post.
For Each post In posts("data")("children")
totalPosts += 1
' Check if the post contains the keyword
If PostsProcessor.PostContainsKeyword(post, inputs.Value.Keyword.ToLower(Globalization.CultureInfo.InvariantCulture)) Then
foundPosts += 1
foundPostsList.Add(post)
' Add the post to the DataGridView.
DataGridViewHandler.AddRow(FormPosts.DataGridViewPosts, post, totalPosts)
FormPosts.Show()
keywordFound = True
End If
Next
' Check if the keyword was found in any posts
If Not keywordFound Then
MessageBox.Show($"Keyword `{inputs.Value.Keyword}` was not found in any of the " + posts("data")("children").Count.ToString(Globalization.CultureInfo.InvariantCulture) _
+ $" {inputs.Value.Listing} posts from r/{inputs.Value.Subreddit}", "Not Found", MessageBoxButtons.OK, MessageBoxIcon.Warning)
End If
If settings.SaveToJson Then
' Save posts to a JSON file if SaveToJson is True.
Utilities.SavePostsToJson(posts:=foundPostsList)
End If
If settings.SaveToCsv Then
' Save posts to a CSV file if SaveToCsv is True.
Utilities.SavePostsToCSV(posts:=foundPostsList)
End If
Else
End If
End Sub
End Class

View File

@@ -1,43 +1,52 @@
![reddit](https://github.com/bellingcat/reddit-post-scraping-tool/assets/74001397/558d31b8-575d-4ab4-a4cf-ec5c41105d12)
![rpst](https://github.com/bellingcat/reddit-post-scraping-tool/assets/74001397/b9ec50b2-d2cb-419f-b8f0-d170b0630875)
## Note
> Use [Knew Karma](https://github.com/bellingcat/knewkarma) for more advanced and improved features.
# RPST (Reddit Post Scraping Tool)
Retrieve **Reddit** posts that contain the specified keyword from a specified subreddit.
Retrieve **Reddit** posts that contain the specified **keyword** from a specified **subreddit**.
[![Upload Python Package](https://github.com/bellingcat/reddit-post-scraping-tool/actions/workflows/python-publish.yml/badge.svg)](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/python-publish.yml) [![CodeQL](https://github.com/bellingcat/reddit-post-scraping-tool/actions/workflows/codeql.yml/badge.svg)](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/codeql.yml) ![.Net](https://img.shields.io/badge/.NET-5C2D91?style=flat&logo=.net&logoColor=white) ![Python](https://img.shields.io/badge/python-3670A0?style=flat&logo=python&logoColor=ffdd54)
# ✅ Features
## *GUI*
- [x] Dark mode (*Right-click*).
- [x] Saves results to a JSON file (*Right-click*).
- [x] Dark mode (*Right-click>Settings>Dark Mode*).
- [x] Saves results to a JSON/CSV file (*Right-click>Settings>Save posts>to JSON/to CSV*).
- [x] Logs errors to a file.
- [x] In-App feature to check for Updates.
## *CLI*
- [x] Saves results to JSON (*specifiy* `--json`).
- [x] Saves results to CSV (*specify* `--csv`).
- [x] Automatically checks for new updates, and notifies user if updates were found.
# 📃 TODO
## *GUI*
- [ ] Make it installable with a setup.exe/setup.msi file.
- [x] Add manual dark mode option, that will be persistent in all sessions.
- [x] Make settings persistent in all sessions.
- [x] Make it save results to a CSV file.
# 🖥️ Tested environments
## *GUI*
- [x] Microsoft Windows 11
## *CLI*
- [x] Android Termux
- [x] Microsoft Windows 11
- [x] Ubuntu 22.04 - latest versions
# 📖 Wiki
[Refer to the Wiki](https://github.com/bellingcat/reddit-post-scraping-tool/wiki) for installation instructions, in addition to all other documentation.
# 📖 Documentation
[Refer to the Wiki](https://github.com/bellingcat/reddit-post-scraping-tool/wiki) for installation instructions, in
addition to all other documentation.
# 🖼️ Screenshots
You can view a collection of screenshots for both the *CLI* and *GUI* [here](https://github.com/bellingcat/reddit-post-scraping-tool/tree/master/images)
***
<a href="https://www.buymeacoffee.com/_rly0nheart"><img src="https://img.buymeacoffee.com/button-api/?text=Buy me a coffee&emoji=&slug=_rly0nheart&button_colour=40DCA5&font_colour=ffffff&font_family=Comic&outline_colour=000000&coffee_colour=FFDD00" /></a>
![me](https://github.com/bellingcat/reddit-post-scraping-tool/assets/74001397/21e0bb33-7a84-45d6-92ba-00e40891ba31)
[![me](https://github.com/bellingcat/knewkarma/assets/74001397/efd19c7e-9840-4969-b33c-04087e73e4da)](https://about.me/rly0nheart)

View File

@@ -6,18 +6,18 @@
<StartupObject>RPST.My.MyApplication</StartupObject>
<UseWindowsForms>true</UseWindowsForms>
<MyType>WindowsForms</MyType>
<ApplicationIcon>icon.ico</ApplicationIcon>
<Company>Bellingcat</Company>
<ApplicationIcon>Resources\icon.ico</ApplicationIcon>
<Company>Richard Mwewa</Company>
<Description>Retrieve Reddit posts that contain the specified keyword from a specified subreddit. </Description>
<Copyright>© 2023 Richard Mwewa. All rights reserved.</Copyright>
<PackageProjectUrl>https://github.com/bellingcat/reddit-post-scraping-tool</PackageProjectUrl>
<PackageReadmeFile>README.md</PackageReadmeFile>
<RepositoryUrl>https://github.com/bellingcat/reddit-post-scraping-tool</RepositoryUrl>
<AssemblyVersion>1.9.1.1</AssemblyVersion>
<FileVersion>1.9.1.1</FileVersion>
<AssemblyVersion>2.0.0.0</AssemblyVersion>
<FileVersion>2.0.0.0</FileVersion>
<PackageLicenseFile>LICENSE</PackageLicenseFile>
<PackageRequireLicenseAcceptance>True</PackageRequireLicenseAcceptance>
<Version>1.9.1</Version>
<Version>2.0.0</Version>
<PackageTags>reddit;scraper;reddit-scraper;osint</PackageTags>
<PackageReleaseNotes></PackageReleaseNotes>
<AnalysisLevel>6.0-recommended</AnalysisLevel>
@@ -27,10 +27,11 @@
<Product>$(AssemblyName) (Reddit Post Scraping Tool)</Product>
<AssemblyName>RPST</AssemblyName>
<Title>Reddit Post Scraping Tool.</Title>
<PackageIcon>icon.png</PackageIcon>
</PropertyGroup>
<ItemGroup>
<Content Include="icon.ico" />
<Content Include="Resources\icon.ico" />
</ItemGroup>
<ItemGroup>
@@ -77,6 +78,10 @@
<Pack>True</Pack>
<PackagePath>\</PackagePath>
</None>
<None Update="Resources\icon.png">
<Pack>True</Pack>
<PackagePath>\</PackagePath>
</None>
</ItemGroup>
</Project>

View File

@@ -1,16 +1,13 @@
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="Current" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<Compile Update="AboutBox.vb">
<Compile Update="Windows\AboutBox.vb">
<SubType>Form</SubType>
</Compile>
<Compile Update="FormMain.vb">
<Compile Update="Windows\MainWindow.vb">
<SubType>Form</SubType>
</Compile>
<Compile Update="FormPosts.vb">
<SubType>Form</SubType>
</Compile>
<Compile Update="SplashScreen.vb">
<Compile Update="Windows\PostsWindow.vb">
<SubType>Form</SubType>
</Compile>
</ItemGroup>

Binary file not shown.

After

Width:  |  Height:  |  Size: 9.4 KiB

View File

Before

Width:  |  Height:  |  Size: 26 KiB

After

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 48 KiB

View File

@@ -1,127 +0,0 @@
<Global.Microsoft.VisualBasic.CompilerServices.DesignerGenerated()> _
Partial Class SplashScreen
Inherits System.Windows.Forms.Form
'Form overrides dispose to clean up the component list.
<System.Diagnostics.DebuggerNonUserCode()> _
Protected Overrides Sub Dispose(ByVal disposing As Boolean)
Try
If disposing AndAlso components IsNot Nothing Then
components.Dispose()
End If
Finally
MyBase.Dispose(disposing)
End Try
End Sub
Friend WithEvents ApplicationTitle As Label
Friend WithEvents Version As Label
Friend WithEvents Copyright As Label
Friend WithEvents MainLayoutPanel As TableLayoutPanel
Friend WithEvents DetailsLayoutPanel As TableLayoutPanel
'Required by the Windows Form Designer
Private components As System.ComponentModel.IContainer
'NOTE: The following procedure is required by the Windows Form Designer
'It can be modified using the Windows Form Designer.
'Do not modify it using the code editor.
<System.Diagnostics.DebuggerStepThrough()> _
Private Sub InitializeComponent()
Dim resources As ComponentModel.ComponentResourceManager = New ComponentModel.ComponentResourceManager(GetType(SplashScreen))
MainLayoutPanel = New TableLayoutPanel()
DetailsLayoutPanel = New TableLayoutPanel()
Copyright = New Label()
Version = New Label()
ApplicationTitle = New Label()
MainLayoutPanel.SuspendLayout()
DetailsLayoutPanel.SuspendLayout()
SuspendLayout()
'
' MainLayoutPanel
'
MainLayoutPanel.BackColor = Color.White
MainLayoutPanel.BackgroundImage = CType(resources.GetObject("MainLayoutPanel.BackgroundImage"), Image)
MainLayoutPanel.BackgroundImageLayout = ImageLayout.Stretch
MainLayoutPanel.ColumnCount = 2
MainLayoutPanel.ColumnStyles.Add(New ColumnStyle(SizeType.Absolute, 270F))
MainLayoutPanel.ColumnStyles.Add(New ColumnStyle(SizeType.Absolute, 73F))
MainLayoutPanel.Controls.Add(DetailsLayoutPanel, 1, 1)
MainLayoutPanel.Controls.Add(ApplicationTitle, 1, 0)
MainLayoutPanel.Dock = DockStyle.Fill
MainLayoutPanel.Location = New Point(0, 0)
MainLayoutPanel.Name = "MainLayoutPanel"
MainLayoutPanel.RowStyles.Add(New RowStyle(SizeType.Absolute, 223F))
MainLayoutPanel.RowStyles.Add(New RowStyle(SizeType.Absolute, 33F))
MainLayoutPanel.Size = New Size(483, 313)
MainLayoutPanel.TabIndex = 0
'
' DetailsLayoutPanel
'
DetailsLayoutPanel.Anchor = AnchorStyles.None
DetailsLayoutPanel.BackColor = Color.Transparent
DetailsLayoutPanel.ColumnCount = 1
DetailsLayoutPanel.ColumnStyles.Add(New ColumnStyle(SizeType.Absolute, 142F))
DetailsLayoutPanel.ColumnStyles.Add(New ColumnStyle(SizeType.Absolute, 142F))
DetailsLayoutPanel.Controls.Add(Copyright, 0, 1)
DetailsLayoutPanel.Controls.Add(Version, 0, 0)
DetailsLayoutPanel.Location = New Point(273, 226)
DetailsLayoutPanel.Name = "DetailsLayoutPanel"
DetailsLayoutPanel.RowCount = 2
DetailsLayoutPanel.RowStyles.Add(New RowStyle(SizeType.Percent, 61.70213F))
DetailsLayoutPanel.RowStyles.Add(New RowStyle(SizeType.Percent, 38.29787F))
DetailsLayoutPanel.Size = New Size(207, 84)
DetailsLayoutPanel.TabIndex = 1
'
' Copyright
'
Copyright.Anchor = AnchorStyles.None
Copyright.BackColor = Color.Transparent
Copyright.Font = New Font("Segoe UI", 8.25F, FontStyle.Regular, GraphicsUnit.Point)
Copyright.Location = New Point(3, 51)
Copyright.Name = "Copyright"
Copyright.Size = New Size(201, 33)
Copyright.TabIndex = 2
Copyright.Text = "Copyright"
'
' Version
'
Version.Anchor = AnchorStyles.None
Version.BackColor = Color.Transparent
Version.Font = New Font("Segoe UI Semibold", 9F, FontStyle.Bold Or FontStyle.Underline, GraphicsUnit.Point)
Version.Location = New Point(3, 4)
Version.Name = "Version"
Version.Size = New Size(201, 43)
Version.TabIndex = 1
Version.Text = "Version"
'
' ApplicationTitle
'
ApplicationTitle.Anchor = AnchorStyles.None
ApplicationTitle.BackColor = Color.Transparent
ApplicationTitle.Font = New Font("Segoe UI", 18F, FontStyle.Bold, GraphicsUnit.Point)
ApplicationTitle.Location = New Point(273, 0)
ApplicationTitle.Name = "ApplicationTitle"
ApplicationTitle.Size = New Size(207, 223)
ApplicationTitle.TabIndex = 0
ApplicationTitle.Text = "Reddit Post Scraping Tool."
ApplicationTitle.TextAlign = ContentAlignment.BottomLeft
'
' SplashScreen
'
AutoScaleDimensions = New SizeF(7F, 15F)
AutoScaleMode = AutoScaleMode.Font
ClientSize = New Size(483, 313)
ControlBox = False
Controls.Add(MainLayoutPanel)
FormBorderStyle = FormBorderStyle.FixedSingle
MaximizeBox = False
MinimizeBox = False
Name = "SplashScreen"
ShowInTaskbar = False
StartPosition = FormStartPosition.CenterScreen
MainLayoutPanel.ResumeLayout(False)
DetailsLayoutPanel.ResumeLayout(False)
ResumeLayout(False)
End Sub
End Class

File diff suppressed because it is too large Load Diff

View File

@@ -1,9 +0,0 @@
Public NotInheritable Class SplashScreen
Private Sub SplashScreen_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles MyBase.Load
' Version info
Version.Text = $"Version {My.Application.Info.Version}"
'Copyright info
Copyright.Text = My.Application.Info.Copyright
End Sub
End Class

View File

@@ -71,7 +71,6 @@ from a specified subreddit. "
Shell("cmd /c start mailto:rly0nheart@duck.com")
End Sub
''' <summary>
''' Handles the Click event for ButtonOK event.
''' </summary>

View File

@@ -1,5 +1,5 @@
<Global.Microsoft.VisualBasic.CompilerServices.DesignerGenerated()>
Partial Class FormMain
Partial Class MainWindow
Inherits System.Windows.Forms.Form
'Form overrides dispose to clean up the component list.
@@ -23,7 +23,7 @@ Partial Class FormMain
<System.Diagnostics.DebuggerStepThrough()>
Private Sub InitializeComponent()
components = New ComponentModel.Container()
Dim resources As ComponentModel.ComponentResourceManager = New ComponentModel.ComponentResourceManager(GetType(FormMain))
Dim resources As System.ComponentModel.ComponentResourceManager = New System.ComponentModel.ComponentResourceManager(GetType(MainWindow))
TextBoxKeyword = New TextBox()
TextBoxSubreddit = New TextBox()
ButtonSearch = New Button()
@@ -165,14 +165,14 @@ Partial Class FormMain
'
ContextMenuStripRightClick.Items.AddRange(New ToolStripItem() {SettingsToolStripMenuItem, AboutToolStripMenuItem, CheckForUpdatesToolStripMenuItem, QuitToolStripMenuItem})
ContextMenuStripRightClick.Name = "ContextMenuStrip1"
ContextMenuStripRightClick.Size = New Size(181, 114)
ContextMenuStripRightClick.Size = New Size(172, 92)
'
' SettingsToolStripMenuItem
'
SettingsToolStripMenuItem.DropDownItems.AddRange(New ToolStripItem() {DarkModeToolStripMenuItem, SavePostsToolStripMenuItem})
SettingsToolStripMenuItem.Image = CType(resources.GetObject("SettingsToolStripMenuItem.Image"), Image)
SettingsToolStripMenuItem.Name = "SettingsToolStripMenuItem"
SettingsToolStripMenuItem.Size = New Size(180, 22)
SettingsToolStripMenuItem.Size = New Size(171, 22)
SettingsToolStripMenuItem.Text = "Settings"
'
' DarkModeToolStripMenuItem
@@ -215,7 +215,7 @@ Partial Class FormMain
AboutToolStripMenuItem.AutoToolTip = True
AboutToolStripMenuItem.Image = CType(resources.GetObject("AboutToolStripMenuItem.Image"), Image)
AboutToolStripMenuItem.Name = "AboutToolStripMenuItem"
AboutToolStripMenuItem.Size = New Size(180, 22)
AboutToolStripMenuItem.Size = New Size(171, 22)
AboutToolStripMenuItem.Text = "About RPST"
'
' CheckForUpdatesToolStripMenuItem
@@ -223,7 +223,7 @@ Partial Class FormMain
CheckForUpdatesToolStripMenuItem.AutoToolTip = True
CheckForUpdatesToolStripMenuItem.Image = CType(resources.GetObject("CheckForUpdatesToolStripMenuItem.Image"), Image)
CheckForUpdatesToolStripMenuItem.Name = "CheckForUpdatesToolStripMenuItem"
CheckForUpdatesToolStripMenuItem.Size = New Size(180, 22)
CheckForUpdatesToolStripMenuItem.Size = New Size(171, 22)
CheckForUpdatesToolStripMenuItem.Text = "Check for Updates"
'
' QuitToolStripMenuItem
@@ -232,19 +232,20 @@ Partial Class FormMain
QuitToolStripMenuItem.Font = New Font("Segoe UI Semibold", 9F, FontStyle.Bold, GraphicsUnit.Point)
QuitToolStripMenuItem.Image = CType(resources.GetObject("QuitToolStripMenuItem.Image"), Image)
QuitToolStripMenuItem.Name = "QuitToolStripMenuItem"
QuitToolStripMenuItem.Size = New Size(180, 22)
QuitToolStripMenuItem.Size = New Size(171, 22)
QuitToolStripMenuItem.Text = "Quit"
'
' NumericUpDownLimit
'
NumericUpDownLimit.Location = New Point(118, 78)
NumericUpDownLimit.Minimum = New Decimal(New Integer() {5, 0, 0, 0})
NumericUpDownLimit.Maximum = New Decimal(New Integer() {10000, 0, 0, 0})
NumericUpDownLimit.Minimum = New Decimal(New Integer() {200, 0, 0, 0})
NumericUpDownLimit.Name = "NumericUpDownLimit"
NumericUpDownLimit.ReadOnly = True
NumericUpDownLimit.Size = New Size(100, 23)
NumericUpDownLimit.TabIndex = 15
ToolTip.SetToolTip(NumericUpDownLimit, "Number of posts to go through. Default value is `10`.")
NumericUpDownLimit.Value = New Decimal(New Integer() {10, 0, 0, 0})
NumericUpDownLimit.Value = New Decimal(New Integer() {200, 0, 0, 0})
'
' ToolTip
'
@@ -255,7 +256,7 @@ Partial Class FormMain
ToolTip.ToolTipIcon = ToolTipIcon.Info
ToolTip.ToolTipTitle = "Tip"
'
' FormMain
' MainWindow
'
AutoScaleDimensions = New SizeF(7F, 15F)
AutoScaleMode = AutoScaleMode.Font
@@ -276,7 +277,7 @@ Partial Class FormMain
FormBorderStyle = FormBorderStyle.FixedSingle
Icon = CType(resources.GetObject("$this.Icon"), Icon)
MaximizeBox = False
Name = "FormMain"
Name = "MainWindow"
StartPosition = FormStartPosition.CenterScreen
Text = "RPST"
ContextMenuStripRightClick.ResumeLayout(False)

View File

@@ -1,6 +1,6 @@
Imports Newtonsoft.Json.Linq
Public Class FormMain
Public Class MainWindow
ReadOnly settings As New SettingsManager()
ReadOnly ApiHandler As New ApiHandler()
@@ -10,14 +10,14 @@ Public Class FormMain
''' </summary>
''' <param name="sender">The source of the event.</param>
''' <param name="e">An EventArgs that contains the event data.</param>
Private Sub FormMain_Load(sender As Object, e As EventArgs) Handles MyBase.Load
Private Sub MainWindow_Load(sender As Object, e As EventArgs) Handles MyBase.Load
Utilities.PathFinder()
settings.LoadSettings()
settings.ToggleSettings(enabled:=settings.DarkMode, saveTo:="darkmode")
settings.ToggleSettings(enabled:=settings.SaveToJson, saveTo:="json")
settings.ToggleSettings(enabled:=settings.SaveToCsv, saveTo:="csv")
Utilities.PathFinder()
Utilities.LogFirstTimeLaunch()
Me.Text = My.Application.Info.AssemblyName
End Sub

View File

@@ -0,0 +1,203 @@
<Global.Microsoft.VisualBasic.CompilerServices.DesignerGenerated()>
Partial Class PostsWindow
Inherits System.Windows.Forms.Form
'Form overrides dispose to clean up the component list.
<System.Diagnostics.DebuggerNonUserCode()>
Protected Overrides Sub Dispose(ByVal disposing As Boolean)
Try
If disposing AndAlso components IsNot Nothing Then
components.Dispose()
End If
Finally
MyBase.Dispose(disposing)
End Try
End Sub
'Required by the Windows Form Designer
Private components As System.ComponentModel.IContainer
'NOTE: The following procedure is required by the Windows Form Designer
'It can be modified using the Windows Form Designer.
'Do not modify it using the code editor.
<System.Diagnostics.DebuggerStepThrough()>
Private Sub InitializeComponent()
Dim resources As System.ComponentModel.ComponentResourceManager = New System.ComponentModel.ComponentResourceManager(GetType(PostsWindow))
DataGridViewPosts = New DataGridView()
postIndex = New DataGridViewTextBoxColumn()
postAuthor = New DataGridViewTextBoxColumn()
postId = New DataGridViewTextBoxColumn()
postTitle = New DataGridViewTextBoxColumn()
postText = New DataGridViewTextBoxColumn()
postSubreddit = New DataGridViewTextBoxColumn()
postSubredditType = New DataGridViewTextBoxColumn()
postThumbnail = New DataGridViewTextBoxColumn()
postIsNSFW = New DataGridViewTextBoxColumn()
postIsGilded = New DataGridViewTextBoxColumn()
postUpvotes = New DataGridViewTextBoxColumn()
postUpvoteRatio = New DataGridViewTextBoxColumn()
postIsShareable = New DataGridViewTextBoxColumn()
postScore = New DataGridViewTextBoxColumn()
postCategory = New DataGridViewTextBoxColumn()
postDomain = New DataGridViewTextBoxColumn()
postPermalink = New DataGridViewTextBoxColumn()
postCreatedAt = New DataGridViewTextBoxColumn()
CType(DataGridViewPosts, ComponentModel.ISupportInitialize).BeginInit()
SuspendLayout()
'
' DataGridViewPosts
'
DataGridViewPosts.BackgroundColor = Color.Gainsboro
DataGridViewPosts.ColumnHeadersHeightSizeMode = DataGridViewColumnHeadersHeightSizeMode.AutoSize
DataGridViewPosts.Columns.AddRange(New DataGridViewColumn() {postIndex, postAuthor, postId, postTitle, postText, postSubreddit, postSubredditType, postThumbnail, postIsNSFW, postIsGilded, postUpvotes, postUpvoteRatio, postIsShareable, postScore, postCategory, postDomain, postPermalink, postCreatedAt})
DataGridViewPosts.Dock = DockStyle.Fill
DataGridViewPosts.Location = New Point(0, 0)
DataGridViewPosts.Name = "DataGridViewPosts"
DataGridViewPosts.ReadOnly = True
DataGridViewPosts.RowHeadersVisible = False
DataGridViewPosts.RowTemplate.Height = 25
DataGridViewPosts.Size = New Size(501, 365)
DataGridViewPosts.TabIndex = 3
'
' postIndex
'
postIndex.HeaderText = "Index"
postIndex.Name = "postIndex"
postIndex.ReadOnly = True
'
' postAuthor
'
postAuthor.HeaderText = "Author"
postAuthor.Name = "postAuthor"
postAuthor.ReadOnly = True
'
' postId
'
postId.HeaderText = "ID"
postId.Name = "postId"
postId.ReadOnly = True
'
' postTitle
'
postTitle.HeaderText = "Title"
postTitle.Name = "postTitle"
postTitle.ReadOnly = True
'
' postText
'
postText.HeaderText = "Text"
postText.Name = "postText"
postText.ReadOnly = True
'
' postSubreddit
'
postSubreddit.HeaderText = "Subreddit"
postSubreddit.Name = "postSubreddit"
postSubreddit.ReadOnly = True
'
' postSubredditType
'
postSubredditType.HeaderText = "Subreddit Type"
postSubredditType.Name = "postSubredditType"
postSubredditType.ReadOnly = True
'
' postThumbnail
'
postThumbnail.HeaderText = "Thumbnail"
postThumbnail.Name = "postThumbnail"
postThumbnail.ReadOnly = True
'
' postIsNSFW
'
postIsNSFW.HeaderText = "Is NSFW"
postIsNSFW.Name = "postIsNSFW"
postIsNSFW.ReadOnly = True
'
' postIsGilded
'
postIsGilded.HeaderText = "Is Gilded"
postIsGilded.Name = "postIsGilded"
postIsGilded.ReadOnly = True
'
' postUpvotes
'
postUpvotes.HeaderText = "Upvotes"
postUpvotes.Name = "postUpvotes"
postUpvotes.ReadOnly = True
'
' postUpvoteRatio
'
postUpvoteRatio.HeaderText = "Upvote Ratio"
postUpvoteRatio.Name = "postUpvoteRatio"
postUpvoteRatio.ReadOnly = True
'
' postIsShareable
'
postIsShareable.HeaderText = "Is Shareable"
postIsShareable.Name = "postIsShareable"
postIsShareable.ReadOnly = True
'
' postScore
'
postScore.HeaderText = "Score"
postScore.Name = "postScore"
postScore.ReadOnly = True
'
' postCategory
'
postCategory.HeaderText = "Category"
postCategory.Name = "postCategory"
postCategory.ReadOnly = True
'
' postDomain
'
postDomain.HeaderText = "Domain"
postDomain.Name = "postDomain"
postDomain.ReadOnly = True
'
' postPermalink
'
postPermalink.HeaderText = "Permalink"
postPermalink.Name = "postPermalink"
postPermalink.ReadOnly = True
'
' postCreatedAt
'
postCreatedAt.HeaderText = "Created At"
postCreatedAt.Name = "postCreatedAt"
postCreatedAt.ReadOnly = True
'
' PostsWindow
'
AutoScaleDimensions = New SizeF(7F, 15F)
AutoScaleMode = AutoScaleMode.Font
ClientSize = New Size(501, 365)
Controls.Add(DataGridViewPosts)
Icon = CType(resources.GetObject("$this.Icon"), Icon)
Name = "PostsWindow"
StartPosition = FormStartPosition.CenterScreen
Text = "Posts"
CType(DataGridViewPosts, ComponentModel.ISupportInitialize).EndInit()
ResumeLayout(False)
End Sub
Friend WithEvents DataGridViewPosts As DataGridView
Friend WithEvents postIndex As DataGridViewTextBoxColumn
Friend WithEvents postAuthor As DataGridViewTextBoxColumn
Friend WithEvents postId As DataGridViewTextBoxColumn
Friend WithEvents postTitle As DataGridViewTextBoxColumn
Friend WithEvents postText As DataGridViewTextBoxColumn
Friend WithEvents postSubreddit As DataGridViewTextBoxColumn
Friend WithEvents postSubredditType As DataGridViewTextBoxColumn
Friend WithEvents postThumbnail As DataGridViewTextBoxColumn
Friend WithEvents postIsNSFW As DataGridViewTextBoxColumn
Friend WithEvents postIsGilded As DataGridViewTextBoxColumn
Friend WithEvents postUpvotes As DataGridViewTextBoxColumn
Friend WithEvents postUpvoteRatio As DataGridViewTextBoxColumn
Friend WithEvents postIsShareable As DataGridViewTextBoxColumn
Friend WithEvents postScore As DataGridViewTextBoxColumn
Friend WithEvents postCategory As DataGridViewTextBoxColumn
Friend WithEvents postDomain As DataGridViewTextBoxColumn
Friend WithEvents postPermalink As DataGridViewTextBoxColumn
Friend WithEvents postCreatedAt As DataGridViewTextBoxColumn
End Class

View File

@@ -117,6 +117,60 @@
<resheader name="writer">
<value>System.Resources.ResXResourceWriter, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</value>
</resheader>
<metadata name="postIndex.UserAddedColumn" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
<value>True</value>
</metadata>
<metadata name="postAuthor.UserAddedColumn" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
<value>True</value>
</metadata>
<metadata name="postId.UserAddedColumn" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
<value>True</value>
</metadata>
<metadata name="postTitle.UserAddedColumn" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
<value>True</value>
</metadata>
<metadata name="postText.UserAddedColumn" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
<value>True</value>
</metadata>
<metadata name="postSubreddit.UserAddedColumn" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
<value>True</value>
</metadata>
<metadata name="postSubredditType.UserAddedColumn" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
<value>True</value>
</metadata>
<metadata name="postThumbnail.UserAddedColumn" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
<value>True</value>
</metadata>
<metadata name="postIsNSFW.UserAddedColumn" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
<value>True</value>
</metadata>
<metadata name="postIsGilded.UserAddedColumn" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
<value>True</value>
</metadata>
<metadata name="postUpvotes.UserAddedColumn" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
<value>True</value>
</metadata>
<metadata name="postUpvoteRatio.UserAddedColumn" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
<value>True</value>
</metadata>
<metadata name="postIsShareable.UserAddedColumn" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
<value>True</value>
</metadata>
<metadata name="postScore.UserAddedColumn" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
<value>True</value>
</metadata>
<metadata name="postCategory.UserAddedColumn" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
<value>True</value>
</metadata>
<metadata name="postDomain.UserAddedColumn" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
<value>True</value>
</metadata>
<metadata name="postPermalink.UserAddedColumn" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
<value>True</value>
</metadata>
<metadata name="postCreatedAt.UserAddedColumn" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
<value>True</value>
</metadata>
<assembly alias="System.Drawing" name="System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
<data name="$this.Icon" type="System.Drawing.Icon, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
<value>

View File

@@ -0,0 +1,3 @@
Public Class PostsWindow
End Class

View File

@@ -0,0 +1,792 @@
"DeployProject"
{
"VSVersion" = "3:800"
"ProjectType" = "8:{978C614F-708E-4E1A-B201-565925725DBA}"
"IsWebType" = "8:FALSE"
"ProjectName" = "8:RPSTSetup"
"LanguageId" = "3:0"
"CodePage" = "3:1252"
"UILanguageId" = "3:0"
"SccProjectName" = "8:"
"SccLocalPath" = "8:"
"SccAuxPath" = "8:"
"SccProvider" = "8:"
"Hierarchy"
{
"Entry"
{
"MsmKey" = "8:_3E2A5BCE69FF40C19B380CE7DE18F582"
"OwnerKey" = "8:_UNDEFINED"
"MsmSig" = "8:_UNDEFINED"
}
"Entry"
{
"MsmKey" = "8:_68E75ECCCEB74C9DAEE029419E7ACA2B"
"OwnerKey" = "8:_UNDEFINED"
"MsmSig" = "8:_UNDEFINED"
}
"Entry"
{
"MsmKey" = "8:_76F301ADC81F41699FE5F6EFEFECAA11"
"OwnerKey" = "8:_UNDEFINED"
"MsmSig" = "8:_UNDEFINED"
}
}
"Configurations"
{
"Debug"
{
"DisplayName" = "8:Debug"
"IsDebugOnly" = "11:TRUE"
"IsReleaseOnly" = "11:FALSE"
"OutputFilename" = "8:Debug\\RPSTSetup.msi"
"PackageFilesAs" = "3:2"
"PackageFileSize" = "3:-2147483648"
"CabType" = "3:1"
"Compression" = "3:2"
"SignOutput" = "11:FALSE"
"CertificateFile" = "8:"
"PrivateKeyFile" = "8:"
"TimeStampServer" = "8:"
"InstallerBootstrapper" = "3:2"
"BootstrapperCfg:{63ACBE69-63AA-4F98-B2B6-99F9E24495F2}"
{
"Enabled" = "11:TRUE"
"PromptEnabled" = "11:TRUE"
"PrerequisitesLocation" = "2:1"
"Url" = "8:"
"ComponentsUrl" = "8:"
"Items"
{
"{EDC2488A-8267-493A-A98E-7D9C3B36CDF3}:Microsoft.NetCore.CoreRuntime.6.0.x64"
{
"Name" = "8:.NET Runtime 6.0.25 (x64)"
"ProductCode" = "8:Microsoft.NetCore.CoreRuntime.6.0.x64"
}
"{EDC2488A-8267-493A-A98E-7D9C3B36CDF3}:Microsoft.NetCore.CoreRuntime.6.0.x86"
{
"Name" = "8:.NET Runtime 6.0.25 (x86)"
"ProductCode" = "8:Microsoft.NetCore.CoreRuntime.6.0.x86"
}
}
}
}
"Release"
{
"DisplayName" = "8:Release"
"IsDebugOnly" = "11:FALSE"
"IsReleaseOnly" = "11:TRUE"
"OutputFilename" = "8:Release\\RPSTSetup.msi"
"PackageFilesAs" = "3:2"
"PackageFileSize" = "3:-2147483648"
"CabType" = "3:1"
"Compression" = "3:2"
"SignOutput" = "11:FALSE"
"CertificateFile" = "8:"
"PrivateKeyFile" = "8:"
"TimeStampServer" = "8:"
"InstallerBootstrapper" = "3:2"
}
}
"Deployable"
{
"CustomAction"
{
}
"DefaultFeature"
{
"Name" = "8:DefaultFeature"
"Title" = "8:"
"Description" = "8:"
}
"ExternalPersistence"
{
"LaunchCondition"
{
"{A06ECF26-33A3-4562-8140-9B0E340D4F24}:_17213E668C074AD5AD5FC8E06206E69E"
{
"Name" = "8:.NET Core"
"Message" = "8:[VSDNETCOREMSG]"
"AllowLaterVersions" = "11:FALSE"
"InstallUrl" = "8:https://dotnet.microsoft.com/download/dotnet-core/[NetCoreVerMajorDotMinor]"
"IsNETCore" = "11:TRUE"
"Architecture" = "2:0"
"Runtime" = "2:0"
}
}
}
"File"
{
"{1FB2D0AE-D3B9-43D4-B9DD-F88EC61E35DE}:_3E2A5BCE69FF40C19B380CE7DE18F582"
{
"SourcePath" = "8:..\\RPST\\Resources\\icon-small.ico"
"TargetName" = "8:icon-small.ico"
"Tag" = "8:"
"Folder" = "8:_362735941ABB4269A087A0EAC1F3EB41"
"Condition" = "8:"
"Transitive" = "11:FALSE"
"Vital" = "11:TRUE"
"ReadOnly" = "11:TRUE"
"Hidden" = "11:TRUE"
"System" = "11:FALSE"
"Permanent" = "11:FALSE"
"SharedLegacy" = "11:FALSE"
"PackageAs" = "3:1"
"Register" = "3:1"
"Exclude" = "11:FALSE"
"IsDependency" = "11:FALSE"
"IsolateTo" = "8:"
}
"{1FB2D0AE-D3B9-43D4-B9DD-F88EC61E35DE}:_76F301ADC81F41699FE5F6EFEFECAA11"
{
"SourcePath" = "8:..\\RPST\\LICENSE.rtf"
"TargetName" = "8:LICENSE.rtf"
"Tag" = "8:"
"Folder" = "8:_362735941ABB4269A087A0EAC1F3EB41"
"Condition" = "8:"
"Transitive" = "11:FALSE"
"Vital" = "11:TRUE"
"ReadOnly" = "11:TRUE"
"Hidden" = "11:TRUE"
"System" = "11:FALSE"
"Permanent" = "11:FALSE"
"SharedLegacy" = "11:FALSE"
"PackageAs" = "3:1"
"Register" = "3:1"
"Exclude" = "11:FALSE"
"IsDependency" = "11:FALSE"
"IsolateTo" = "8:"
}
}
"FileType"
{
}
"Folder"
{
"{1525181F-901A-416C-8A58-119130FE478E}:_14AD380FC0AA495FB87400E478966DD2"
{
"Name" = "8:#1919"
"AlwaysCreate" = "11:TRUE"
"Condition" = "8:"
"Transitive" = "11:FALSE"
"Property" = "8:ProgramMenuFolder"
"Folders"
{
}
}
"{3C67513D-01DD-4637-8A68-80971EB9504F}:_362735941ABB4269A087A0EAC1F3EB41"
{
"DefaultLocation" = "8:[ProgramFilesFolder][Manufacturer]\\[ProductName]"
"Name" = "8:#1925"
"AlwaysCreate" = "11:FALSE"
"Condition" = "8:"
"Transitive" = "11:FALSE"
"Property" = "8:TARGETDIR"
"Folders"
{
}
}
"{1525181F-901A-416C-8A58-119130FE478E}:_BD020B7E2C9F475083F2EE7493C6CA56"
{
"Name" = "8:#1916"
"AlwaysCreate" = "11:TRUE"
"Condition" = "8:"
"Transitive" = "11:FALSE"
"Property" = "8:DesktopFolder"
"Folders"
{
}
}
}
"LaunchCondition"
{
}
"Locator"
{
}
"MsiBootstrapper"
{
"LangId" = "3:0"
"RequiresElevation" = "11:FALSE"
}
"Product"
{
"Name" = "8:Microsoft Visual Studio"
"ProductName" = "8:RPST (Reddit Post Scraping Tool)"
"ProductCode" = "8:{76E86D3A-14D8-426B-ADB6-C22C3D444E49}"
"PackageCode" = "8:{D33BBB6A-B9C1-4596-A299-931B9FEE6921}"
"UpgradeCode" = "8:{05EF97C6-762C-4254-94FF-53380A799007}"
"AspNetVersion" = "8:4.0.30319.0"
"RestartWWWService" = "11:FALSE"
"RemovePreviousVersions" = "11:TRUE"
"DetectNewerInstalledVersion" = "11:TRUE"
"InstallAllUsers" = "11:FALSE"
"ProductVersion" = "8:2.0.0"
"Manufacturer" = "8:Richard Mwewa"
"ARPHELPTELEPHONE" = "8:"
"ARPHELPLINK" = "8:https://github.com/bellingcat/reddit-post-scraping-tool/wiki"
"Title" = "8:RPST (Reddit Post Scraping Tool)"
"Subject" = "8:"
"ARPCONTACT" = "8:Richard Mwewa"
"Keywords" = "8:"
"ARPCOMMENTS" = "8:Retrieve Reddit posts that contain the specified keyword from a specified subreddit."
"ARPURLINFOABOUT" = "8:https://about.me/rly0nheart"
"ARPPRODUCTICON" = "8:_3E2A5BCE69FF40C19B380CE7DE18F582"
"ARPIconIndex" = "3:0"
"SearchPath" = "8:"
"UseSystemSearchPath" = "11:TRUE"
"TargetPlatform" = "3:0"
"PreBuildEvent" = "8:"
"PostBuildEvent" = "8:"
"RunPostBuildEvent" = "3:0"
}
"Registry"
{
"HKLM"
{
"Keys"
{
"{60EA8692-D2D5-43EB-80DC-7906BF13D6EF}:_49885B940CB4489B9981836DE0A26E07"
{
"Name" = "8:Software"
"Condition" = "8:"
"AlwaysCreate" = "11:FALSE"
"DeleteAtUninstall" = "11:FALSE"
"Transitive" = "11:FALSE"
"Keys"
{
"{60EA8692-D2D5-43EB-80DC-7906BF13D6EF}:_E3E857007AC8478B9D0C9C009F6A8BAF"
{
"Name" = "8:[Manufacturer]"
"Condition" = "8:"
"AlwaysCreate" = "11:FALSE"
"DeleteAtUninstall" = "11:FALSE"
"Transitive" = "11:FALSE"
"Keys"
{
}
"Values"
{
}
}
}
"Values"
{
}
}
}
}
"HKCU"
{
"Keys"
{
"{60EA8692-D2D5-43EB-80DC-7906BF13D6EF}:_17B8420A2F55464D8FD66D76E90FE530"
{
"Name" = "8:Software"
"Condition" = "8:"
"AlwaysCreate" = "11:FALSE"
"DeleteAtUninstall" = "11:FALSE"
"Transitive" = "11:FALSE"
"Keys"
{
"{60EA8692-D2D5-43EB-80DC-7906BF13D6EF}:_247BBD924EA946C0B4F05A907A48C9D5"
{
"Name" = "8:[Manufacturer]"
"Condition" = "8:"
"AlwaysCreate" = "11:FALSE"
"DeleteAtUninstall" = "11:FALSE"
"Transitive" = "11:FALSE"
"Keys"
{
}
"Values"
{
}
}
}
"Values"
{
}
}
}
}
"HKCR"
{
"Keys"
{
}
}
"HKU"
{
"Keys"
{
}
}
"HKPU"
{
"Keys"
{
}
}
}
"Sequences"
{
}
"Shortcut"
{
"{970C0BB2-C7D0-45D7-ABFA-7EC378858BC0}:_03274DE5011840E682E398A86F5A065D"
{
"Name" = "8:RPST (Reddit Post Scraping Tool)"
"Arguments" = "8:"
"Description" = "8:Retrieve Reddit posts that contain the specified keyword from a specified subreddit."
"ShowCmd" = "3:1"
"IconIndex" = "3:0"
"Transitive" = "11:FALSE"
"Target" = "8:_68E75ECCCEB74C9DAEE029419E7ACA2B"
"Folder" = "8:_14AD380FC0AA495FB87400E478966DD2"
"WorkingFolder" = "8:_362735941ABB4269A087A0EAC1F3EB41"
"Icon" = "8:_3E2A5BCE69FF40C19B380CE7DE18F582"
"Feature" = "8:"
}
"{970C0BB2-C7D0-45D7-ABFA-7EC378858BC0}:_6C793AC49B71464FA40CB4D6C361BD84"
{
"Name" = "8:RPST (Reddit Post Scraping Tool)"
"Arguments" = "8:"
"Description" = "8:Retrieve Reddit posts that contain the specified keyword from a specified subreddit."
"ShowCmd" = "3:1"
"IconIndex" = "3:0"
"Transitive" = "11:FALSE"
"Target" = "8:_68E75ECCCEB74C9DAEE029419E7ACA2B"
"Folder" = "8:_BD020B7E2C9F475083F2EE7493C6CA56"
"WorkingFolder" = "8:_362735941ABB4269A087A0EAC1F3EB41"
"Icon" = "8:_3E2A5BCE69FF40C19B380CE7DE18F582"
"Feature" = "8:"
}
}
"UserInterface"
{
"{2479F3F5-0309-486D-8047-8187E2CE5BA0}:_260F17414CD94855B1C4E94459527B0E"
{
"UseDynamicProperties" = "11:FALSE"
"IsDependency" = "11:FALSE"
"SourcePath" = "8:<VsdDialogDir>\\VsdUserInterface.wim"
}
"{DF760B10-853B-4699-99F2-AFF7185B4A62}:_2E6A77A8635449DCAA91FDB47AD57928"
{
"Name" = "8:#1900"
"Sequence" = "3:2"
"Attributes" = "3:1"
"Dialogs"
{
"{688940B3-5CA9-4162-8DEE-2993FA9D8CBC}:_318D351DD1CD404E8A4BD093773AE2F5"
{
"Sequence" = "3:100"
"DisplayName" = "8:License Agreement"
"UseDynamicProperties" = "11:TRUE"
"IsDependency" = "11:FALSE"
"SourcePath" = "8:<VsdDialogDir>\\VsdAdminLicenseDlg.wid"
"Properties"
{
"BannerBitmap"
{
"Name" = "8:BannerBitmap"
"DisplayName" = "8:#1001"
"Description" = "8:#1101"
"Type" = "3:8"
"ContextData" = "8:Bitmap"
"Attributes" = "3:4"
"Setting" = "3:1"
"UsePlugInResources" = "11:TRUE"
}
"EulaText"
{
"Name" = "8:EulaText"
"DisplayName" = "8:#1008"
"Description" = "8:#1108"
"Type" = "3:6"
"ContextData" = "8:"
"Attributes" = "3:0"
"Setting" = "3:2"
"Value" = "8:_76F301ADC81F41699FE5F6EFEFECAA11"
"UsePlugInResources" = "11:TRUE"
}
"Sunken"
{
"Name" = "8:Sunken"
"DisplayName" = "8:#1007"
"Description" = "8:#1107"
"Type" = "3:5"
"ContextData" = "8:4;True=4;False=0"
"Attributes" = "3:0"
"Setting" = "3:0"
"Value" = "3:4"
"DefaultValue" = "3:4"
"UsePlugInResources" = "11:TRUE"
}
}
}
"{688940B3-5CA9-4162-8DEE-2993FA9D8CBC}:_5F8575AA5ECB4174B1C60F9DBFBC9855"
{
"Sequence" = "3:200"
"DisplayName" = "8:Installation Folder"
"UseDynamicProperties" = "11:TRUE"
"IsDependency" = "11:FALSE"
"SourcePath" = "8:<VsdDialogDir>\\VsdAdminFolderDlg.wid"
"Properties"
{
"BannerBitmap"
{
"Name" = "8:BannerBitmap"
"DisplayName" = "8:#1001"
"Description" = "8:#1101"
"Type" = "3:8"
"ContextData" = "8:Bitmap"
"Attributes" = "3:4"
"Setting" = "3:1"
"UsePlugInResources" = "11:TRUE"
}
}
}
"{688940B3-5CA9-4162-8DEE-2993FA9D8CBC}:_7D142C585EE6498DA19A25EEBEDACFF6"
{
"Sequence" = "3:300"
"DisplayName" = "8:Confirm Installation"
"UseDynamicProperties" = "11:TRUE"
"IsDependency" = "11:FALSE"
"SourcePath" = "8:<VsdDialogDir>\\VsdAdminConfirmDlg.wid"
"Properties"
{
"BannerBitmap"
{
"Name" = "8:BannerBitmap"
"DisplayName" = "8:#1001"
"Description" = "8:#1101"
"Type" = "3:8"
"ContextData" = "8:Bitmap"
"Attributes" = "3:4"
"Setting" = "3:1"
"UsePlugInResources" = "11:TRUE"
}
}
}
}
}
"{DF760B10-853B-4699-99F2-AFF7185B4A62}:_521CEA4C4D36467CAF5E9B792033EB4B"
{
"Name" = "8:#1901"
"Sequence" = "3:2"
"Attributes" = "3:2"
"Dialogs"
{
"{688940B3-5CA9-4162-8DEE-2993FA9D8CBC}:_80EEC0D3EB6146438998401A4D904E1A"
{
"Sequence" = "3:100"
"DisplayName" = "8:Progress"
"UseDynamicProperties" = "11:TRUE"
"IsDependency" = "11:FALSE"
"SourcePath" = "8:<VsdDialogDir>\\VsdAdminProgressDlg.wid"
"Properties"
{
"BannerBitmap"
{
"Name" = "8:BannerBitmap"
"DisplayName" = "8:#1001"
"Description" = "8:#1101"
"Type" = "3:8"
"ContextData" = "8:Bitmap"
"Attributes" = "3:4"
"Setting" = "3:1"
"UsePlugInResources" = "11:TRUE"
}
"ShowProgress"
{
"Name" = "8:ShowProgress"
"DisplayName" = "8:#1009"
"Description" = "8:#1109"
"Type" = "3:5"
"ContextData" = "8:1;True=1;False=0"
"Attributes" = "3:0"
"Setting" = "3:0"
"Value" = "3:1"
"DefaultValue" = "3:1"
"UsePlugInResources" = "11:TRUE"
}
}
}
}
}
"{2479F3F5-0309-486D-8047-8187E2CE5BA0}:_5EFCFC798BF24EB68DC201E4505F8489"
{
"UseDynamicProperties" = "11:FALSE"
"IsDependency" = "11:FALSE"
"SourcePath" = "8:<VsdDialogDir>\\VsdBasicDialogs.wim"
}
"{DF760B10-853B-4699-99F2-AFF7185B4A62}:_6BFF4405824D471EA651E90715AAACDF"
{
"Name" = "8:#1901"
"Sequence" = "3:1"
"Attributes" = "3:2"
"Dialogs"
{
"{688940B3-5CA9-4162-8DEE-2993FA9D8CBC}:_D34EDE0B30824A01B0B266B6079C8469"
{
"Sequence" = "3:100"
"DisplayName" = "8:Progress"
"UseDynamicProperties" = "11:TRUE"
"IsDependency" = "11:FALSE"
"SourcePath" = "8:<VsdDialogDir>\\VsdProgressDlg.wid"
"Properties"
{
"BannerBitmap"
{
"Name" = "8:BannerBitmap"
"DisplayName" = "8:#1001"
"Description" = "8:#1101"
"Type" = "3:8"
"ContextData" = "8:Bitmap"
"Attributes" = "3:4"
"Setting" = "3:1"
"UsePlugInResources" = "11:TRUE"
}
"ShowProgress"
{
"Name" = "8:ShowProgress"
"DisplayName" = "8:#1009"
"Description" = "8:#1109"
"Type" = "3:5"
"ContextData" = "8:1;True=1;False=0"
"Attributes" = "3:0"
"Setting" = "3:0"
"Value" = "3:1"
"DefaultValue" = "3:1"
"UsePlugInResources" = "11:TRUE"
}
}
}
}
}
"{DF760B10-853B-4699-99F2-AFF7185B4A62}:_D30397AD6E1547799AC7430D693FA0FC"
{
"Name" = "8:#1902"
"Sequence" = "3:1"
"Attributes" = "3:3"
"Dialogs"
{
"{688940B3-5CA9-4162-8DEE-2993FA9D8CBC}:_71B305AEDF384563B2D30069B0F145DF"
{
"Sequence" = "3:100"
"DisplayName" = "8:Finished"
"UseDynamicProperties" = "11:TRUE"
"IsDependency" = "11:FALSE"
"SourcePath" = "8:<VsdDialogDir>\\VsdFinishedDlg.wid"
"Properties"
{
"BannerBitmap"
{
"Name" = "8:BannerBitmap"
"DisplayName" = "8:#1001"
"Description" = "8:#1101"
"Type" = "3:8"
"ContextData" = "8:Bitmap"
"Attributes" = "3:4"
"Setting" = "3:1"
"UsePlugInResources" = "11:TRUE"
}
"UpdateText"
{
"Name" = "8:UpdateText"
"DisplayName" = "8:#1058"
"Description" = "8:#1158"
"Type" = "3:15"
"ContextData" = "8:"
"Attributes" = "3:0"
"Setting" = "3:1"
"Value" = "8:#1258"
"DefaultValue" = "8:#1258"
"UsePlugInResources" = "11:TRUE"
}
}
}
}
}
"{DF760B10-853B-4699-99F2-AFF7185B4A62}:_EBE4314B289A4ECEB75F0B5D00905DB4"
{
"Name" = "8:#1900"
"Sequence" = "3:1"
"Attributes" = "3:1"
"Dialogs"
{
"{688940B3-5CA9-4162-8DEE-2993FA9D8CBC}:_5A069E8C7FE74313BA59FE93ED3B017D"
{
"Sequence" = "3:200"
"DisplayName" = "8:License Agreement"
"UseDynamicProperties" = "11:TRUE"
"IsDependency" = "11:FALSE"
"SourcePath" = "8:<VsdDialogDir>\\VsdLicenseDlg.wid"
"Properties"
{
"BannerBitmap"
{
"Name" = "8:BannerBitmap"
"DisplayName" = "8:#1001"
"Description" = "8:#1101"
"Type" = "3:8"
"ContextData" = "8:Bitmap"
"Attributes" = "3:4"
"Setting" = "3:1"
"UsePlugInResources" = "11:TRUE"
}
"EulaText"
{
"Name" = "8:EulaText"
"DisplayName" = "8:#1008"
"Description" = "8:#1108"
"Type" = "3:6"
"ContextData" = "8:"
"Attributes" = "3:0"
"Setting" = "3:2"
"Value" = "8:_76F301ADC81F41699FE5F6EFEFECAA11"
"UsePlugInResources" = "11:TRUE"
}
"Sunken"
{
"Name" = "8:Sunken"
"DisplayName" = "8:#1007"
"Description" = "8:#1107"
"Type" = "3:5"
"ContextData" = "8:4;True=4;False=0"
"Attributes" = "3:0"
"Setting" = "3:0"
"Value" = "3:4"
"DefaultValue" = "3:4"
"UsePlugInResources" = "11:TRUE"
}
}
}
"{688940B3-5CA9-4162-8DEE-2993FA9D8CBC}:_69D777B0C6974DD0AFA54BDF51755C81"
{
"Sequence" = "3:400"
"DisplayName" = "8:Confirm Installation"
"UseDynamicProperties" = "11:TRUE"
"IsDependency" = "11:FALSE"
"SourcePath" = "8:<VsdDialogDir>\\VsdConfirmDlg.wid"
"Properties"
{
"BannerBitmap"
{
"Name" = "8:BannerBitmap"
"DisplayName" = "8:#1001"
"Description" = "8:#1101"
"Type" = "3:8"
"ContextData" = "8:Bitmap"
"Attributes" = "3:4"
"Setting" = "3:1"
"UsePlugInResources" = "11:TRUE"
}
}
}
"{688940B3-5CA9-4162-8DEE-2993FA9D8CBC}:_7A7EE4F003334B01BE835D840DF961DF"
{
"Sequence" = "3:300"
"DisplayName" = "8:Installation Folder"
"UseDynamicProperties" = "11:TRUE"
"IsDependency" = "11:FALSE"
"SourcePath" = "8:<VsdDialogDir>\\VsdFolderDlg.wid"
"Properties"
{
"BannerBitmap"
{
"Name" = "8:BannerBitmap"
"DisplayName" = "8:#1001"
"Description" = "8:#1101"
"Type" = "3:8"
"ContextData" = "8:Bitmap"
"Attributes" = "3:4"
"Setting" = "3:1"
"UsePlugInResources" = "11:TRUE"
}
"InstallAllUsersVisible"
{
"Name" = "8:InstallAllUsersVisible"
"DisplayName" = "8:#1059"
"Description" = "8:#1159"
"Type" = "3:5"
"ContextData" = "8:1;True=1;False=0"
"Attributes" = "3:0"
"Setting" = "3:0"
"Value" = "3:1"
"DefaultValue" = "3:1"
"UsePlugInResources" = "11:TRUE"
}
}
}
}
}
"{DF760B10-853B-4699-99F2-AFF7185B4A62}:_FE23A7B7220447F981819CF713DF20E5"
{
"Name" = "8:#1902"
"Sequence" = "3:2"
"Attributes" = "3:3"
"Dialogs"
{
"{688940B3-5CA9-4162-8DEE-2993FA9D8CBC}:_54098FCDE6914AEE8AA27CDB24E77DAF"
{
"Sequence" = "3:100"
"DisplayName" = "8:Finished"
"UseDynamicProperties" = "11:TRUE"
"IsDependency" = "11:FALSE"
"SourcePath" = "8:<VsdDialogDir>\\VsdAdminFinishedDlg.wid"
"Properties"
{
"BannerBitmap"
{
"Name" = "8:BannerBitmap"
"DisplayName" = "8:#1001"
"Description" = "8:#1101"
"Type" = "3:8"
"ContextData" = "8:Bitmap"
"Attributes" = "3:4"
"Setting" = "3:1"
"UsePlugInResources" = "11:TRUE"
}
}
}
}
}
}
"MergeModule"
{
}
"ProjectOutput"
{
"{5259A561-127C-4D43-A0A1-72F10C7B3BF8}:_68E75ECCCEB74C9DAEE029419E7ACA2B"
{
"SourcePath" = "8:..\\RPST\\obj\\Debug\\net6.0-windows\\apphost.exe"
"TargetName" = "8:"
"Tag" = "8:"
"Folder" = "8:_362735941ABB4269A087A0EAC1F3EB41"
"Condition" = "8:"
"Transitive" = "11:FALSE"
"Vital" = "11:TRUE"
"ReadOnly" = "11:TRUE"
"Hidden" = "11:FALSE"
"System" = "11:FALSE"
"Permanent" = "11:FALSE"
"SharedLegacy" = "11:FALSE"
"PackageAs" = "3:1"
"Register" = "3:1"
"Exclude" = "11:FALSE"
"IsDependency" = "11:FALSE"
"IsolateTo" = "8:"
"ProjectOutputGroupRegister" = "3:1"
"OutputConfiguration" = "8:"
"OutputGroupCanonicalName" = "8:PublishItems"
"OutputProjectGuid" = "8:{46C2541E-6F65-461A-A479-F65D445C36EA}"
"ShowKeyOutput" = "11:TRUE"
"ExcludeFilters"
{
}
}
}
}
}

Binary file not shown.

Before

Width:  |  Height:  |  Size: 68 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 27 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 576 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 519 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 823 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 508 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 21 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.0 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1017 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 886 KiB

View File

@@ -7,27 +7,27 @@ packages = ["rpst"]
[project]
name = "reddit-post-scraping-tool"
version = "1.9.1.1"
version = "2.0.1.0"
description = "Retrieve Reddit posts that contain the specified keyword from a specified subreddit."
readme = "README.md"
requires-python = ">=3.8"
license = {file = "LICENSE"}
license = { file = "LICENSE" }
keywords = ["reddit-crawler", "reddit-scraping", "reddit", "reddit-api"]
authors = [{name = "Richard Mwewa", email = "rly0nheart@duck.com"}]
authors = [{ name = "Richard Mwewa", email = "rly0nheart@duck.com" }]
classifiers = [
"Development Status :: 5 - Production/Stable",
"Programming Language :: Python :: 3",
"Programming Language :: Visual Basic",
"Intended Audience :: End Users/Desktop",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
"Natural Language :: English"
"Development Status :: 5 - Production/Stable",
"Programming Language :: Python :: 3",
"Programming Language :: Visual Basic",
"Intended Audience :: End Users/Desktop",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
"Natural Language :: English"
]
dependencies = [
"rich",
"glyphoji",
"requests",
"aiohttp",
"rich-argparse"
]
[project.urls]
@@ -36,4 +36,5 @@ documentation = "https://github.com/bellingcat/reddit-post-scraping-tool/wiki"
repository = "https://github.com/bellingcat/reddit-post-scraping-tool.git"
[project.scripts]
rpst = "rpst.main:run"
rpst = "rpst.scraper:run"
reddit_post_scraping_tool = "rpst.scraper:run"

View File

@@ -1 +1,42 @@
import os
__author__: str = "Richard Mwewa"
__about_author__: str = "https://about/me/rly0nheart"
__version__: str = "2.0.1.0"
__description__: str = f"""
# RPST (Reddit Post Scraping Tool) {__version__}
> Retrieve Reddit posts that contain the specified keyword from a specified subreddit.
"""
__epilog__: str = f"""
# by [{__author__}]({__about_author__})
```
MIT License
Copyright (c) 2023 {__author__}
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
```
"""
# Construct path to the program's directory
PROGRAM_DIRECTORY: str = os.path.expanduser(
os.path.join("~", "reddit_post_scraping_tool")
)

164
rpst/api.py Normal file
View File

@@ -0,0 +1,164 @@
# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #
from typing import Union
import aiohttp
from .coreutils import log
# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #
REDDIT_ENDPOINT: str = "https://www.reddit.com"
PYPI_PROJECT_ENDPOINT: str = "https://pypi.org/pypi/reddit-post-scraping-tool/json"
# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #
async def get_data(session: aiohttp.ClientSession, endpoint: str) -> Union[dict, list]:
"""
Fetches JSON data from a given API endpoint.
:param session: aiohttp session to use for the request.
:param endpoint: The API endpoint to fetch data from.
:return: Returns JSON data as a dictionary or list. Returns an empty dict if fetching fails.
"""
try:
async with session.get(
endpoint,
) as response:
if response.status == 200:
return await response.json()
else:
error_message = await response.json()
log.error(f"An API error occurred: {error_message}")
return {}
except aiohttp.ClientConnectionError as error:
log.error(f"An HTTP error occurred: {error}")
return {}
except Exception as error:
log.critical(f"An unknown error occurred: {error}")
return {}
# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #
async def get_updates(session: aiohttp.ClientSession):
"""
Gets and compares the current program version with the remote version.
Assumes version format: major.minor.patch.prefix
:param session: aiohttp session to use for the request.
"""
from . import __version__
# Make a GET request to PyPI to get the project's latest release.
response: dict = await get_data(endpoint=PYPI_PROJECT_ENDPOINT, session=session)
if response.get("info"):
release: dict = response.get("info")
remote_version: str = release.get("version")
# Splitting the version strings into components
remote_parts: list = remote_version.split(".")
local_parts: list = __version__.split(".")
update_message: str = ""
# Check for differences in version parts
if remote_parts[0] != local_parts[0]:
update_message = (
f"MAJOR update ({remote_version}) available."
f" It might introduce significant changes."
)
elif remote_parts[1] != local_parts[1]:
update_message = (
f"MINOR update ({remote_version}) available."
f" Includes small feature changes/improvements."
)
elif remote_parts[2] != local_parts[2]:
update_message = (
f"PATCH update ({remote_version}) available."
f" Generally for bug fixes and small tweaks."
)
elif (
len(remote_parts) > 3
and len(local_parts) > 3
and remote_parts[3] != local_parts[3]
):
update_message = (
f"BUILD update ({remote_version}) available."
f" Might be for specific builds or special versions."
)
if update_message:
log.info(update_message)
# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #
async def get_posts(
subreddit: str,
listing: str,
timeframe: str,
limit: int,
session: aiohttp.ClientSession,
) -> list:
all_posts = await paginated_posts(
posts_endpoint=f"{REDDIT_ENDPOINT}/r/{subreddit}/{listing}.json?limit={limit}&t={timeframe}",
limit=limit,
session=session,
)
return all_posts[:limit]
# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #
async def paginated_posts(
posts_endpoint: str, limit: int, session: aiohttp.ClientSession
) -> list:
"""
Paginates and retrieves posts until the specified limit is reached.
:param posts_endpoint: API endpoint for retrieving posts.
:param limit: Limit of the number of posts to retrieve.
:param session: aiohttp session to use for the request.
:return: A list of all posts.
"""
all_posts: list = []
last_post_id: str = ""
# Determine whether to use the 'after' parameter
use_after: bool = limit > 100
while len(all_posts) < limit:
# Make the API request with the 'after' parameter if it's provided and the limit is more than 100
if use_after and last_post_id:
endpoint_with_after: str = f"{posts_endpoint}&after={last_post_id}"
else:
endpoint_with_after: str = posts_endpoint
posts_data: dict = await get_data(endpoint=endpoint_with_after, session=session)
posts_children: list = posts_data.get("data", {}).get("children", [])
# If there are no more posts, break out of the loop
if not posts_children:
break
all_posts.extend(posts_children)
# We use the id of the last post in the list to paginate to the next posts
last_post_id: str = all_posts[-1].get("data").get("id")
return all_posts
# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #

103
rpst/base.py Normal file
View File

@@ -0,0 +1,103 @@
# +++++++++++++++++++++++++++++++++++++++++++++++++ #
from dataclasses import dataclass
from typing import List
import aiohttp
from .api import get_posts, get_updates
from .coreutils import timestamp_to_utc
# +++++++++++++++++++++++++++++++++++++++++++++++++ #
@dataclass
class Post:
id: str
thumbnail: str
title: str
text: str
author: str
subreddit: str
subreddit_id: str
subreddit_type: str
upvotes: int
upvote_ratio: float
downvotes: int
gilded: int
is_nsfw: bool
is_shareable: bool
is_edited: bool
comments: int
hide_from_bots: bool
score: float
domain: str
permalink: str
is_locked: bool
is_archived: bool
created_at: str
raw_post: dict
# +++++++++++++++++++++++++++++++++++++++++++++++++ #
async def find_posts(
keyword: str,
subreddit: str,
listing: str,
timeframe: str,
limit: int,
) -> List[Post]:
async with aiohttp.ClientSession() as session:
found_posts_count: int = 0
found_posts_list: list = []
await get_updates(session=session)
raw_posts: list = await get_posts(
subreddit=subreddit,
listing=listing,
timeframe=timeframe,
limit=limit,
session=session,
)
for raw_post in raw_posts:
post_data: dict = raw_post.get("data")
if keyword.lower() in post_data.get(
"selftext"
) or keyword.lower() in post_data.get("title"):
found_posts_count += 1
post = Post(
id=post_data.get("id"),
thumbnail=post_data.get("thumbnail"),
title=post_data.get("title"),
text=post_data.get("selftext"),
author=post_data.get("author"),
subreddit=post_data.get("subreddit"),
subreddit_id=post_data.get("subreddit_id"),
subreddit_type=post_data.get("subreddit_type"),
upvotes=post_data.get("ups"),
upvote_ratio=post_data.get("upvote_ratio"),
downvotes=post_data.get("downs"),
gilded=post_data.get("gilded"),
is_nsfw=post_data.get("over_18"),
is_shareable=post_data.get("is_reddit_media_domain"),
is_edited=post_data.get("edited"),
comments=post_data.get("num_comments"),
hide_from_bots=post_data.get("is_robot_indexable"),
score=post_data.get("score"),
domain=post_data.get("domain"),
permalink=post_data.get("permalink"),
is_locked=post_data.get("locked"),
is_archived=post_data.get("archived"),
created_at=timestamp_to_utc(timestamp=post_data.get("created_utc")),
raw_post=post_data,
)
found_posts_list.append(post)
return found_posts_list
# +++++++++++++++++++++++++++++++++++++++++++++++++ #

170
rpst/coreutils.py Normal file
View File

@@ -0,0 +1,170 @@
# +++++++++++++++++++++++++++++++++++++++++++++++++ #
import argparse
import csv
import json
import logging
import os
from datetime import datetime
from rich.logging import RichHandler
from rich.markdown import Markdown
from rich_argparse import RichHelpFormatter
# +++++++++++++++++++++++++++++++++++++++++++++++++ #
def timestamp_to_utc(timestamp: int) -> str:
"""
Converts a Unix timestamp to a formatted datetime string.
:param timestamp: The Unix timestamp to be converted.
:return: A formatted datetime string in the format "dd MMMM yyyy, hh:mm:ssAM/PM".
"""
utc_from_timestamp: datetime = datetime.utcfromtimestamp(timestamp)
datetime_string: str = utc_from_timestamp.strftime("%d %B %Y, %I:%M:%S%p")
return datetime_string
# +++++++++++++++++++++++++++++++++++++++++++++++++ #
def pathfinder(directories: list[str]):
for directory in directories:
os.makedirs(directory, exist_ok=True)
# +++++++++++++++++++++++++++++++++++++++++++++++++ #
def save_posts(
filename: str,
save_to_dir: str,
posts: list,
save_json: bool = False,
save_csv: bool = False,
):
posts_data: list = [post.__dict__ for post in posts]
if save_json:
json_path = os.path.join(os.path.join(save_to_dir, "json"), f"{filename}.json")
with open(json_path, "w", encoding="utf-8") as json_file:
json.dump(posts_data, json_file, indent=4)
log.info(
f"{os.path.getsize(json_file.name)} bytes written to [link file://{json_file.name}]{json_file.name}"
)
if save_csv:
csv_path = os.path.join(os.path.join(save_to_dir, "csv"), f"{filename}.csv")
with open(csv_path, "w", newline="", encoding="utf-8") as csv_file:
writer = csv.writer(csv_file)
if posts:
writer.writerow(
posts_data[0].keys()
) # header from keys of the first item
for post in posts:
writer.writerow(post.__dict__.values())
log.info(
f"{os.path.getsize(csv_file.name)} bytes written to [link file://{csv_file.name}]{csv_file.name}"
)
# +++++++++++++++++++++++++++++++++++++++++++++++++ #
def create_parser():
"""
Creates and configures an argument parser for the command line arguments.
:return: A configured argparse.ArgumentParser object ready to parse the command line arguments.
"""
from . import __version__, __description__, __epilog__
parser = argparse.ArgumentParser(
description=Markdown(__description__, style="argparse.text"),
epilog=Markdown(__epilog__, style="argparse.text"),
formatter_class=RichHelpFormatter,
)
parser.add_argument(
"keyword",
help="keyword to search for, in posts",
)
parser.add_argument("subreddit", help="subreddit to scrape")
parser.add_argument(
"-l",
"--limit",
help="maximum number of posts to scrape (default: %(default)s)",
default=200,
type=int,
)
parser.add_argument(
"-ls",
"--listing",
default="top",
const="top",
nargs="?",
choices=["best", "controversial", "hot", "new", "rising", "top"],
help="listing of posts to scrape (default: %(default)s)",
)
parser.add_argument(
"-t",
"--timeframe",
default="all",
const="all",
nargs="?",
choices=["hour", "day", "week", "month", "year", "all"],
help="timeframe from which to scrape posts (default: %(default)s)",
)
parser.add_argument(
"-j",
"--json",
help="write found posts to a json file",
action="store_true",
)
parser.add_argument(
"-c",
"--csv",
help="write found posts to a csv file",
action="store_true",
)
parser.add_argument(
"-d",
"--debug",
help="(dev) run rpst in debug mode",
action="store_true",
)
parser.add_argument("-v", "--version", action="version", version=__version__)
return parser
# +++++++++++++++++++++++++++++++++++++++++++++++++ #
def set_loglevel(debug_mode: bool) -> logging.getLogger:
"""
Configure and return a logging object with the specified log level.
:param debug_mode: If True, the log level is set to "NOTSET". Otherwise, it is set to "INFO".
:return: A logging object configured with the specified log level.
"""
logging.basicConfig(
level="DEBUG" if debug_mode else "INFO",
format="%(message)s",
handlers=[
RichHandler(
markup=True, log_time_format="%I:%M:%S%p", show_level=debug_mode
)
],
)
return logging.getLogger("RPST (Reddit Post Scraping Tool)")
# +++++++++++++++++++++++++++++++++++++++++++++++++ #
args: argparse = create_parser().parse_args()
log: logging.getLogger = set_loglevel(debug_mode=args.debug)
# +++++++++++++++++++++++++++++++++++++++++++++++++ #

View File

@@ -1,33 +0,0 @@
from datetime import datetime
from .rpst import get_posts
from .utils import create_parser, set_loglevel, check_updates
def run():
"""
Main entry point for the program. It creates a parser, parses the command line arguments,
checks for updates, gets posts, and handles any exceptions that occur during the execution.
"""
# Create a parser and parse the command line arguments
parser = create_parser()
args = parser.parse_args()
log = set_loglevel(debug_mode=args.debug)
# Record the start time
start_time = datetime.now()
try:
# Check for updates
check_updates(version_tag="1.9.1.1")
# Get posts with the provided/parsed arguments
get_posts(args=args)
except KeyboardInterrupt:
log.warning("User interruption detected ([yellow]Ctrl+C[/]).")
except Exception as e:
log.error(f"An error occurred: [red]{e}[/]")
finally:
log.info(f"Finished in {datetime.now() - start_time} seconds.")

View File

@@ -1,131 +0,0 @@
import argparse
from datetime import datetime
import requests
from glyphoji import glyph
from rich import print
from rich.tree import Tree
from .utils import convert_timestamp_to_datetime, write_post_data
def create_post_branch(post: dict, keyword: str, tree: Tree, args: argparse) -> Tree:
"""
This function extracts relevant data from a Reddit post and adds it in a tree branch structure,
followed by the post's selftext.
:param post: A dictionary containing the data of a Reddit post.
:param keyword: The keyword that is used to find posts, in his case gets uses as the filename.
:param tree: Tree where the post branch will be added.
:param args: A namespace object from argparse.
:returns: The main tree with added post branches.
"""
# Define the data to extract from the post.
post_data = {
# "Author": post["data"]["author"],
f"{glyph.id_button} ID": post["data"]["id"],
f"{glyph.people_hugging} Subreddit": post["data"]["subreddit_name_prefixed"],
f"{glyph.face_with_peeking_eye} Visibility": post["data"]["subreddit_type"],
f"{glyph.framed_picture} Thumbnail": post["data"]["thumbnail"],
f"{glyph.white_question_mark} Gilded": post["data"]["gilded"],
f"{glyph.up_arrow} Upvotes": post["data"]["ups"],
f"{glyph.chart_increasing} Upvote ratio": post["data"]["upvote_ratio"],
f"{glyph.down_arrow} Downvotes": post["data"]["downs"],
f"{glyph.trophy} Awards": post["data"]["total_awards_received"],
f"{glyph.trophy} Top award": post["data"]["top_awarded_type"],
f"{glyph.no_one_under_eighteen} Is NSFW?": post["data"]["over_18"],
f"{glyph.left_arrow_curving_right} Is crosspostable?": post["data"][
"is_crosspostable"
],
f"{glyph.bar_chart} Score": post["data"]["score"],
f"{glyph.card_file_box} Category": post["data"]["category"],
f"{glyph.globe_with_meridians} Domain": post["data"]["domain"],
f"{glyph.calendar} Posted on": convert_timestamp_to_datetime(
post["data"]["created"]
),
f"{glyph.calendar} Approved at": post["data"]["approved_at_utc"],
f"{glyph.bust_in_silhouette} Approved by": post["data"]["approved_by"],
}
# Add the post's branch to the main tree.
post_branch = tree.add(f"{glyph.page_with_curl} {post['data']['title']}")
# Add each piece of extracted data as a branch of the post_branch.
for post_key, post_value in post_data.items():
post_branch.add(f"{post_key}: {post_value}", style="dim")
# This ensures that the post's selftext is also added to the written json/csv file.
post_data[f"{glyph.clipboard} Text"] = post["data"]["selftext"]
write_post_data(
filename=keyword, post_data=post_data, tree_branch=post_branch, args=args
)
post_branch.add(post["data"]["selftext"], style="italic")
return tree
def get_posts(args: argparse):
"""
Scrapes a given subreddit for posts that contain a specified keyword.
The search is limited by the number of posts and timeframe specified.
:param args: Namespace object from argparse.
Expected Object Attributes
--------------------------
- keyword: The keyword to search for in the posts.
- subreddit: The subreddit to scrape.
- listing: The type of posts to scrape. This could be 'hot', 'new', etc.
- timeframe: The timeframe from which to scrape posts. This could be 'day', 'week', etc.
- limit: The maximum number of posts to scrape.
- json: If specified, all found posts will be written to a json file.
"""
keyword = args.keyword
subreddit = args.subreddit
listing = args.listing
timeframe = args.timeframe
limit = args.limit
# Create main result tree.
main_tree = Tree(
f"[bold]{glyph.calendar} {datetime.now()}[/]", guide_style="bold bright_blue"
)
# Start a new session
session = requests.session()
# Set the User-Agent to mimic a Safari browser on a Mac.
session.headers = {
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, "
"like Gecko) Version/14.1.1 Safari/605.1.15"
}
# Send a GET request to the specified subreddit and listing,
# limiting the response by the specified limit and timeframe.
response = session.get(
f"https://reddit.com/r/{subreddit}/{listing}"
f".json?limit={limit}&t={timeframe}"
).json()
# Initialize a counter for the number of posts found that contain the keyword.
found_posts = 0
# Loop through each post in the response
for post_index, post in enumerate(response["data"]["children"], start=1):
# If the keyword is found in the post's selftext or title, increment the counter and process the post.
if (
keyword.lower() in post["data"]["selftext"]
or keyword.lower() in post["data"]["title"]
):
# Create a branch for found post(s) and show post index and post author as the title
found_tree = main_tree.add(
f"{glyph.bust_in_silhouette} #{post_index} by [bold]@{post['data']['author']}[/]"
)
found_posts += 1
create_post_branch(post=post, keyword=keyword, tree=found_tree, args=args)
# Log the number of posts in which the keyword was found
main_tree.add(
f"{glyph.check_mark_button} Keyword ('{keyword}') was found in "
f"{found_posts}/{len(response['data']['children'])} {listing} posts from r/{subreddit}."
)
print(main_tree)

94
rpst/scraper.py Normal file
View File

@@ -0,0 +1,94 @@
# +++++++++++++++++++++++++++++++++++++++++++++++++ #
import asyncio
import os
from datetime import datetime
from rich.pretty import pprint
from . import __version__, PROGRAM_DIRECTORY
from .base import find_posts
from .coreutils import args, log, save_posts, pathfinder
# +++++++++++++++++++++++++++++++++++++++++++++++++ #
def run():
"""Main entry point for rpst or rpst."""
# ------------------------------------- #
keyword: str = args.keyword
subreddit: str = args.subreddit
listing: str = args.listing
limit: int = args.limit
# ------------------------------------- #
start_time = datetime.now()
# ------------------------------------- #
print(
"""
┳┓┏┓┏┓┏┳┓
┣┫┃┃┗┓ ┃
┛┗┣┛┗┛ ┻ """
)
# ------------------------------------- #
try:
log.info(
f"[bold]RPST[/] {__version__} started at {start_time.strftime('%a %b %d %Y, %I:%M:%S%p')}..."
)
found_posts = asyncio.run(
find_posts(
keyword=keyword,
subreddit=subreddit,
listing=listing,
timeframe=args.timeframe,
limit=limit,
),
)
if found_posts:
pprint(
found_posts,
expand_all=True,
)
log.info(
f"'{subreddit}': Found {len(found_posts)}/{limit} {listing} posts containing the keyword ('{keyword}')"
)
if args.json or args.csv:
target_dir: str = os.path.join(PROGRAM_DIRECTORY, subreddit)
pathfinder(
directories=[
os.path.join(target_dir, "csv"),
os.path.join(target_dir, "json"),
]
)
save_posts(
filename=keyword,
save_to_dir=target_dir,
posts=found_posts,
save_json=args.json,
save_csv=args.csv,
)
else:
log.info(
f"'r/{subreddit}': No {listing} posts found that contain the keyword ('{keyword}')"
)
except KeyboardInterrupt:
log.warning("User interruption detected ([yellow]Ctrl+C[/])")
except Exception as error:
log.error(f"An error occurred: [red]{error}[/]")
finally:
log.info(f"Finished in {datetime.now() - start_time} seconds")
# ------------------------------------- #
# +++++++++++++++++++++++++++++++++++++++++++++++++ #

View File

@@ -1,182 +0,0 @@
import os
import csv
import json
import logging
import argparse
from datetime import datetime
import requests
from glyphoji import glyph
from rich import print
from rich.tree import Tree
from rich.markdown import Markdown
from rich.logging import RichHandler
def convert_timestamp_to_datetime(timestamp: int) -> str:
"""
Converts a Unix timestamp to a formatted datetime string.
:param timestamp: The Unix timestamp to be converted.
:return: A formatted datetime string in the format "dd MMMM yyyy, hh:mm:ssAM/PM".
"""
utc_from_timestamp = datetime.utcfromtimestamp(timestamp)
datetime_object = utc_from_timestamp.strftime("%d %B %Y, %I:%M:%S%p")
return datetime_object
def create_parser():
"""
Creates and configures an argument parser for the command line arguments.
:return: A configured argparse.ArgumentParser object ready to parse the command line arguments.
"""
parser = argparse.ArgumentParser(
description="RPST (Reddit Post Scraping Tool) —by Richard Mwewa | https://about.me/rly0nheart",
epilog="Retrieve Reddit posts that contain the specified keyword from a specified subreddit."
)
parser.add_argument(
"-k", "--keyword", help="The keyword to search for in the posts.", required=True
)
parser.add_argument(
"-s", "--subreddit", help="The subreddit to scrape.", required=True
)
parser.add_argument(
"-c",
"--limit",
help="The maximum number of posts to scrape (1-100). (default: %(default)s)",
default=10,
type=int,
choices=range(
1, 101
), # This enforces that the limit must be between 1 and 100 inclusive.
)
parser.add_argument(
"-l",
"--listing",
default="top",
const="top",
nargs="?",
choices=["controversial", "hot", "best", "new", "rising"],
help="The type of posts to scrape (default: %(default)s)",
)
parser.add_argument(
"-t",
"--timeframe",
default="all",
const="all",
nargs="?",
choices=["hour", "day", "week", "month", "year", "all"],
help="The timeframe from which to scrape posts (default: %(default)s)",
)
parser.add_argument(
"--json",
help="Write all found posts to a json file.",
action="store_true",
)
parser.add_argument(
"--csv",
help="Write all found posts to a csv file.",
action="store_true",
)
parser.add_argument(
"-d",
"--debug",
help="run rpst in debug mode",
action="store_true",
)
return parser
def check_updates(version_tag: str):
"""
This function checks if there's a new release of a project on GitHub. If there is, it logs an
information message and prints the release notes.
:param version_tag: A string representing the current version of the project.
"""
# Make a GET request to the GitHub API to get the latest release of the project.
response = requests.get(
"https://api.github.com/repos/bellingcat/reddit-post-scraping-tool/releases/latest"
).json()
# Check if the latest release's tag matches the current version tag.
if response["tag_name"] != version_tag:
# If not, convert the release notes from Markdown to HTML.
raw_release_notes = response["body"]
# Log an info message about the new release.
print(
f"{glyph.up_arrow} A new release of RPST is available ({response['tag_name']}). "
f"Run 'pip install --upgrade reddit-post-scraping-tool' to get the updates."
)
# Print the release notes.
print(Markdown(raw_release_notes))
def set_loglevel(debug_mode: bool) -> logging.getLogger:
"""
Configure and return a logging object with the specified log level.
:param debug_mode: If True, the log level is set to "NOTSET". Otherwise, it is set to "INFO".
:return: A logging object configured with the specified log level.
"""
logging.basicConfig(
level="NOTSET" if debug_mode else "INFO",
format="%(message)s",
handlers=[
RichHandler(markup=True, log_time_format="[%I:%M:%S %p]", show_level=False)
],
)
return logging.getLogger("RPST")
def write_post_data(post_data: dict, filename: str, args, tree_branch: Tree):
"""
Writes post data to a specified JSON or CSV file based on the args provided, and updates
the provided tree with the status.
:param post_data: A dictionary containing post data.
:param filename: The name of the file to which post data will be written.
:param args: A namespace object from argparse containing the output format options (args.json or args.csv).
:param tree_branch: A rich Tree object to which status information will be added.
"""
home_directory = os.path.expanduser("~")
if args.json:
json_file_path = os.path.join(home_directory, f"{filename}.json")
with open(json_file_path, "a", encoding="utf-8") as file:
file.write(json.dumps(post_data, ensure_ascii=False))
file.write("\n") # Separate posts with newline
tree_branch.add(
f"{glyph.page_facing_up} JSON data successfully written/appended to file: "
f"[italic][link file://{json_file_path}]{json_file_path}[/]"
)
else:
tree_branch.add(
f"{glyph.cross_mark_button} JSON data writing operation was skipped. No changes made."
)
if args.csv:
csv_file_path = os.path.join(home_directory, f"{filename}.csv")
with open(csv_file_path, "a", newline="", encoding="utf-8") as csvfile:
writer = csv.DictWriter(csvfile, fieldnames=post_data.keys())
# Write headers if file is empty
if csvfile.tell() == 0:
writer.writeheader()
writer.writerow(post_data)
tree_branch.add(
f"{glyph.page_facing_up} CSV data successfully written/appended to file: "
f"[italic][link file://{csv_file_path}]{csv_file_path}[/]"
)
else:
tree_branch.add(
f"{glyph.cross_mark_button} CSV data writing operation was skipped. No changes made."
)