36 Commits

Author SHA1 Message Date
Richard Mwewa
5468d93976 Add files via upload 2023-08-30 03:26:13 +02:00
Richard Mwewa
d186f6f7da Merge pull request #15 from bellingcat/dev
Dev
2023-08-30 03:25:10 +02:00
Richard Mwewa
b5d52e4bb5 Update main.py
dev 1.8.0.0
2023-08-30 03:22:03 +02:00
Richard Mwewa
a2bdc66a03 Update pyproject.toml
dev 1.8.0.0
2023-08-30 03:21:03 +02:00
Richard Mwewa
bb72360699 Update README.md 2023-08-30 03:19:55 +02:00
Richard Mwewa
adc15555e3 Update README.md 2023-08-30 03:19:22 +02:00
Richard Mwewa
07ad626dd2 Add files via upload
dev 1.8.0.0
2023-08-30 03:16:36 +02:00
Richard Mwewa
2729c984bc Delete RPST GUI directory 2023-08-30 03:15:21 +02:00
Richard Mwewa
f3445b8e06 Add files via upload 2023-08-29 19:26:26 +02:00
Richard Mwewa
358f264bdd Merge pull request #14 from bellingcat/dev
dev 1.7.1.0
2023-08-26 15:52:46 +02:00
Richard Mwewa
e3dda99233 dev 1.7.1.0 2023-08-26 15:44:21 +02:00
Richard Mwewa
7198e0be90 Merge pull request #13 from bellingcat/dev
Dev
2023-08-25 23:13:15 +02:00
Richard Mwewa
f9f0ed5085 Update main.py 2023-08-25 23:12:06 +02:00
Richard Mwewa
b0c53a1511 Update RPST.vbproj 2023-08-25 23:11:18 +02:00
Richard Mwewa
54b9abc4b6 Update pyproject.toml 2023-08-25 23:10:29 +02:00
Richard Mwewa
4ae14ff02e Delete .nomedia 2023-08-25 23:09:36 +02:00
Richard Mwewa
376eeab243 Add files via upload 2023-08-25 23:09:17 +02:00
Richard Mwewa
6d427849d5 Create .nomedia 2023-08-25 23:04:27 +02:00
Richard Mwewa
ad8cb63541 Update README.md 2023-08-25 23:03:34 +02:00
Richard Mwewa
57f8c24cee Update rpst.py 2023-08-25 17:09:42 +02:00
Richard Mwewa
750967c322 Merge pull request #12 from bellingcat/dev
Dev
2023-08-25 15:53:49 +02:00
Richard Mwewa
cfef86cbe3 Update README.md 2023-08-25 15:53:23 +02:00
Richard Mwewa
2a2696403d Update README.md 2023-08-25 15:52:49 +02:00
Richard Mwewa
b0a8d75d8c Update README.md 2023-08-25 15:49:07 +02:00
Richard Mwewa
b31c38f5cc Update README.md 2023-08-25 15:48:19 +02:00
Richard Mwewa
b5b7df868e Update README.md 2023-08-25 15:44:13 +02:00
Richard Mwewa
566f558720 Update utils.py 2023-08-25 15:27:18 +02:00
Richard Mwewa
c3e5ce6441 Update rpst.py 2023-08-25 15:26:42 +02:00
Richard Mwewa
7c164938c9 Update RPST.vbproj 2023-08-25 15:06:31 +02:00
Richard Mwewa
b08c4a147b Create utils.py 2023-08-25 15:04:37 +02:00
Richard Mwewa
8f259b7a40 Update pyproject.toml
1.7.0.0
2023-08-25 14:54:31 +02:00
Richard Mwewa
f117c99cc7 Update and rename __main.py to main.py
1.7.0.0
2023-08-25 14:52:27 +02:00
Richard Mwewa
3a9a87e67c Update and rename __rpst.py to rpst.py
1.7.0.0
2023-08-25 14:51:16 +02:00
Richard Mwewa
cce254e976 Update pyproject.toml 2023-08-14 02:51:49 +02:00
Richard Mwewa
418b2acc4c Update README.md 2023-08-12 05:25:07 +02:00
Richard Mwewa
d26699cc1f Update README.md 2023-08-12 05:24:35 +02:00
30 changed files with 3567 additions and 3141 deletions

View File

@@ -1,33 +1,42 @@
# RPST (Reddit Post Scraping Tool)
Given a subreddit name and a keyword, RPST will return all posts from a specified listing (default is 'top') that contain the provided keyword.
[![Upload Python Package](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/python-publish.yml/badge.svg)](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/python-publish.yml) [![CodeQL](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/codeql.yml/badge.svg)](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/codeql.yml) ![.Net](https://img.shields.io/badge/.NET-5C2D91?style=flat&logo=.net&logoColor=white) ![Python](https://img.shields.io/badge/python-3670A0?style=flat&logo=python&logoColor=ffdd54)
![2023-08-09_04-05](https://github.com/bellingcat/reddit-post-scraping-tool/assets/74001397/d8917a35-3eac-44ce-aa96-1f9685095254)
![2023-08-09_04-05_1](https://github.com/bellingcat/reddit-post-scraping-tool/assets/74001397/d2fe7269-91d4-49ad-87fb-44282c5637a7)
***
[![Upload Python Package](https://github.com/bellingcat/reddit-post-scraping-tool/actions/workflows/python-publish.yml/badge.svg)](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/python-publish.yml) [![CodeQL](https://github.com/bellingcat/reddit-post-scraping-tool/actions/workflows/codeql.yml/badge.svg)](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/codeql.yml) ![.Net](https://img.shields.io/badge/.NET-5C2D91?style=flat&logo=.net&logoColor=white) ![Python](https://img.shields.io/badge/python-3670A0?style=flat&logo=python&logoColor=ffdd54)
# ✅ Features
## GUI
- [x] Dark mode (Right-click)
- [x] Saves results to a JSON (Right-click)
- [x] Logs errors to a file
## *GUI*
- [x] Dark mode (*Right-click*).
- [x] Saves results to a JSON file (*Right-click*).
- [x] Logs errors to a file.
- [x] In-App feature to check for Updates.
## CLI
- [x] Saves results to a JSON (-j/--json)
- [x] Automatically checks for new updates. Notifies user if update were found.
## *CLI*
- [x] Saves results to JSON (*specifiy* `--json`).
- [x] Saves results to CSV (*specify* `--csv`).
- [x] Automatically checks for new updates, and notifies user if updates were found.
# 📃 TODO
## GUI
## *GUI*
- [ ] Make it installable with a setup.exe/setup.msi file.
- [x] Add manual dark mode option, that will be persistent in all sessions
- [ ] Make it save results to a CSV file
- [x] Add manual dark mode option, that will be persistent in all sessions.
- [x] Make settings persistent in all sessions.
- [x] Make it save results to a CSV file.
# 🖥️ Tested environments
## *GUI*
- [x] Microsoft Windows 11
## *CLI*
- [x] Android Termux
- [x] Microsoft Windows 11
- [x] Ubuntu 22.04 - latest versions
# 📖 Wiki
[Refer to the Wiki](https://github.com/rly0nheart/reddit-post-scraping-tool/wiki) for installation instructions, in addition to all other documentation.
[Refer to the Wiki](https://github.com/bellingcat/reddit-post-scraping-tool/wiki) for installation instructions, in addition to all other documentation.
# 😁 Donations
If you like `RPST` and would like to show support, you can Buy A Coffee for the developer using the button below
# 🖼️ Screenshots
You can view a collection of screenshots for both the *CLI* and *GUI* [here](https://github.com/bellingcat/reddit-post-scraping-tool/tree/master/images)
***
<a href="https://www.buymeacoffee.com/_rly0nheart"><img src="https://img.buymeacoffee.com/button-api/?text=Buy me a coffee&emoji=&slug=_rly0nheart&button_colour=40DCA5&font_colour=ffffff&font_family=Comic&outline_colour=000000&coffee_colour=FFDD00" /></a>
<a href="https://www.buymeacoffee.com/_rly0nheart" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/default-orange.png" alt="Buy Me A Coffee" height="41" width="174"></a>
Your support will be much appreciated😊
![me](https://github.com/bellingcat/reddit-post-scraping-tool/assets/74001397/21e0bb33-7a84-45d6-92ba-00e40891ba31)

View File

@@ -26,9 +26,9 @@ Partial Class AboutBox
PictureBoxLogo = New PictureBox()
LabelProgramName = New Label()
LabelProgramDescription = New Label()
LabelVersion = New Label()
LinkLabelReadtheWiki = New LinkLabel()
Panel1 = New Panel()
LinkLabelVersion = New LinkLabel()
LicenseRichTextBox = New RichTextBox()
CType(PictureBoxLogo, ComponentModel.ISupportInitialize).BeginInit()
Panel1.SuspendLayout()
@@ -67,17 +67,6 @@ Partial Class AboutBox
LabelProgramDescription.TabIndex = 4
LabelProgramDescription.Text = "Description"
'
' LabelVersion
'
LabelVersion.AutoSize = True
LabelVersion.Font = New Font("Segoe UI", 9F, FontStyle.Underline, GraphicsUnit.Point)
LabelVersion.ForeColor = SystemColors.ControlText
LabelVersion.Location = New Point(347, 17)
LabelVersion.Name = "LabelVersion"
LabelVersion.Size = New Size(45, 15)
LabelVersion.TabIndex = 5
LabelVersion.Text = "Version"
'
' LinkLabelReadtheWiki
'
LinkLabelReadtheWiki.AutoSize = True
@@ -92,15 +81,25 @@ Partial Class AboutBox
' Panel1
'
Panel1.BackColor = SystemColors.Control
Panel1.Controls.Add(LinkLabelVersion)
Panel1.Controls.Add(LabelProgramDescription)
Panel1.Controls.Add(LabelProgramName)
Panel1.Controls.Add(LinkLabelReadtheWiki)
Panel1.Controls.Add(LabelVersion)
Panel1.Location = New Point(106, 12)
Panel1.Name = "Panel1"
Panel1.Size = New Size(409, 93)
Panel1.TabIndex = 7
'
' LinkLabelVersion
'
LinkLabelVersion.AutoSize = True
LinkLabelVersion.Location = New Point(347, 17)
LinkLabelVersion.Name = "LinkLabelVersion"
LinkLabelVersion.Size = New Size(45, 15)
LinkLabelVersion.TabIndex = 7
LinkLabelVersion.TabStop = True
LinkLabelVersion.Text = "Version"
'
' LicenseRichTextBox
'
LicenseRichTextBox.Font = New Font("Cambria", 9.75F, FontStyle.Regular, GraphicsUnit.Point)
@@ -137,8 +136,8 @@ Partial Class AboutBox
Friend WithEvents PictureBoxLogo As PictureBox
Friend WithEvents LabelProgramName As Label
Friend WithEvents LabelProgramDescription As Label
Friend WithEvents LabelVersion As Label
Friend WithEvents LinkLabelReadtheWiki As LinkLabel
Friend WithEvents Panel1 As Panel
Friend WithEvents LicenseRichTextBox As RichTextBox
Friend WithEvents LinkLabelVersion As LinkLabel
End Class

View File

@@ -18,7 +18,7 @@
<resheader name="reader">System.Resources.ResXResourceReader, System.Windows.Forms, ...</resheader>
<resheader name="writer">System.Resources.ResXResourceWriter, System.Windows.Forms, ...</resheader>
<data name="Name1"><value>this is my long string</value><comment>this is a comment</comment></data>
<data name="Color1" type="System.Drawing.Color, System.Drawing"">Blue</data>
<data name="Color1" type="System.Drawing.Color, System.Drawing">Blue</data>
<data name="Bitmap1" mimetype="application/x-microsoft.net.object.binary.base64">
<value>[base64 mime encoded serialized .NET Framework object]</value>
</data>

View File

@@ -31,13 +31,13 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SO
''' <param name="e">The event data.</param>
Private Sub AboutBox_Load(sender As Object, e As EventArgs) Handles MyBase.Load
settings.LoadSettings()
settings.ToggleDarkMode(settings.DarkMode)
settings.ToggleSettings(settings.DarkMode, "darkmode")
LabelProgramName.Text = My.Application.Info.ProductName
LabelProgramDescription.Text = "Given a subreddit name and a keyword,
RPST returns all top posts (by default)
that contain the specified keyword."
LabelVersion.Text = $"v{My.Application.Info.Version}"
LinkLabelVersion.Text = $"v{My.Application.Info.Version}"
LicenseRichTextBox.Text = LicenseText
End Sub
@@ -50,4 +50,8 @@ that contain the specified keyword."
Private Sub LinkLabelReadtheWiki_LinkClicked(sender As Object, e As LinkLabelLinkClickedEventArgs) Handles LinkLabelReadtheWiki.LinkClicked
Shell("cmd /c start https://github.com/bellingcat/reddit-post-scraping-tool/wiki")
End Sub
Private Sub LinkLabelVersion_LinkClicked(sender As Object, e As LinkLabelLinkClickedEventArgs) Handles LinkLabelVersion.LinkClicked
Shell($"cmd /c start https://github.com/bellingcat/reddit-post-scraping-tool/releases/tag/{My.Application.Info.Version}")
End Sub
End Class

View File

@@ -6,32 +6,34 @@ Public Class DataGridViewHandler
''' </summary>
''' <param name="dataGridView">The DataGridView to be initialized.</param>
Public Shared Sub AddColumn(dataGridView As DataGridView)
' Clear the Columns and Rows before adding Items to them
''' <summary>
''' Clear the Columns and Rows before adding Items to them.
''' <summary>
dataGridView.Rows.Clear()
dataGridView.Columns.Clear()
dataGridView.Columns.Add("PostCount", "INDEX")
dataGridView.Columns.Add("PostAuthor", "AUTHOR")
dataGridView.Columns.Add("PostID", "ID")
dataGridView.Columns.Add("PostText", "TEXT")
dataGridView.Columns.Add("PostSubreddit", "SUBREDDIT")
dataGridView.Columns.Add("SubredditVisibility", "VISIBILITY")
dataGridView.Columns.Add("PostThumbnail", "THUMBNAIL")
dataGridView.Columns.Add("PostIsNSFW", "NSFW")
dataGridView.Columns.Add("PostIsGilded", "GILDED")
dataGridView.Columns.Add("PostUpvotes", "UPVOTES")
dataGridView.Columns.Add("PostUpvoteRatio", "UPVOTE RATIO")
dataGridView.Columns.Add("PostDownvotes", "DOWNVOTES")
dataGridView.Columns.Add("PostAwards", "AWARDS")
dataGridView.Columns.Add("PostTopAward", "TOP AWARD")
dataGridView.Columns.Add("PostIsCrosspostable", "IS CROSS-POSTABLE?")
dataGridView.Columns.Add("PostScore", "SCORE")
dataGridView.Columns.Add("PostCategory", "CATEGORY")
dataGridView.Columns.Add("PostDomain", "DOMAIN")
dataGridView.Columns.Add("PostPermalink", "PERMALINK")
dataGridView.Columns.Add("PostCreatedAt", "CREATED AT")
dataGridView.Columns.Add("PostApprovedAt", "APPROVED ATt")
dataGridView.Columns.Add("PostApprovedBy", "APPROVED BY")
dataGridView.Columns.Add("PostCount", "🔢 Index")
dataGridView.Columns.Add("PostAuthor", "👤 Author")
dataGridView.Columns.Add("PostID", "🆔 ID")
dataGridView.Columns.Add("PostText", "📝 Text")
dataGridView.Columns.Add("PostSubreddit", "🫂 Subreddit")
dataGridView.Columns.Add("SubredditVisibility", "🫣 Visibility")
dataGridView.Columns.Add("PostThumbnail", "🖼️ Thumbnail")
dataGridView.Columns.Add("PostIsNSFW", "🔞 NSFW")
dataGridView.Columns.Add("PostIsGilded", "🥇 Gilded")
dataGridView.Columns.Add("PostUpvotes", "⬆️ Upvotes")
dataGridView.Columns.Add("PostUpvoteRatio", "📊 Upvote Ratio")
dataGridView.Columns.Add("PostDownvotes", "⬇️ Downvotes")
dataGridView.Columns.Add("PostAwards", "🏆 Awards")
dataGridView.Columns.Add("PostTopAward", "🏆 Top Award")
dataGridView.Columns.Add("PostIsCrosspostable", "↪️ Is cross-postable?")
dataGridView.Columns.Add("PostScore", "📈 Score")
dataGridView.Columns.Add("PostCategory", "🟢 Category")
dataGridView.Columns.Add("PostDomain", "🌐 Domain")
dataGridView.Columns.Add("PostPermalink", "🔗 Permalink")
dataGridView.Columns.Add("PostCreatedAt", "📅 Created At")
dataGridView.Columns.Add("PostApprovedAt", "📅 Approved At")
dataGridView.Columns.Add("PostApprovedBy", "👤 Approved By")
End Sub
Public Shared Sub AddRow(dataGridView As DataGridView, post As JObject, postNumber As Integer)

View File

@@ -61,7 +61,7 @@ Partial Class DeveloperBox
GreetingLabel.TabIndex = 3
GreetingLabel.Text = "👋🏾Hello, I'm Ritchie"
'
' DeveloperForm
' DeveloperBox
'
AutoScaleDimensions = New SizeF(7F, 15F)
AutoScaleMode = AutoScaleMode.Font
@@ -73,7 +73,7 @@ Partial Class DeveloperBox
FormBorderStyle = FormBorderStyle.FixedSingle
MaximizeBox = False
MinimizeBox = False
Name = "DeveloperForm"
Name = "DeveloperBox"
ShowIcon = False
ShowInTaskbar = False
StartPosition = FormStartPosition.CenterParent

File diff suppressed because it is too large Load Diff

View File

@@ -35,14 +35,15 @@ Partial Class FormMain
LabelListing = New Label()
LabelTimeframe = New Label()
ContextMenuStripRightClick = New ContextMenuStrip(components)
ToolStripMenuItemDarkMode = New ToolStripMenuItem()
ToolStripMenuItemSavePosts = New ToolStripMenuItem()
ToolStripMenuItemtoJSON = New ToolStripMenuItem()
ToolStripMenuItemtoCSV = New ToolStripMenuItem()
ToolStripMenuItemAbout = New ToolStripMenuItem()
ToolStripMenuItemDeveloper = New ToolStripMenuItem()
ToolStripMenuItemCheckUpdates = New ToolStripMenuItem()
ToolStripMenuItemQuit = New ToolStripMenuItem()
SettingsToolStripMenuItem = New ToolStripMenuItem()
DarkModeToolStripMenuItem = New ToolStripMenuItem()
SavePostsToolStripMenuItem = New ToolStripMenuItem()
ToJSONToolStripMenuItem = New ToolStripMenuItem()
ToCSVToolStripMenuItem = New ToolStripMenuItem()
AboutToolStripMenuItem = New ToolStripMenuItem()
DeveloperToolStripMenuItem = New ToolStripMenuItem()
CheckForUpdatesToolStripMenuItem = New ToolStripMenuItem()
QuitToolStripMenuItem = New ToolStripMenuItem()
NumericUpDownLimit = New NumericUpDown()
ToolTip = New ToolTip(components)
ContextMenuStripRightClick.SuspendLayout()
@@ -55,19 +56,19 @@ Partial Class FormMain
TextBoxKeyword.ForeColor = SystemColors.WindowText
TextBoxKeyword.Location = New Point(118, 20)
TextBoxKeyword.Name = "TextBoxKeyword"
TextBoxKeyword.PlaceholderText = "Keyword"
TextBoxKeyword.PlaceholderText = "*Keyword"
TextBoxKeyword.Size = New Size(100, 23)
TextBoxKeyword.TabIndex = 0
ToolTip.SetToolTip(TextBoxKeyword, "Enter the keyword you want to search for.")
ToolTip.SetToolTip(TextBoxKeyword, "[required] The keyword to search for.")
'
' TextBoxSubreddit
'
TextBoxSubreddit.Location = New Point(118, 49)
TextBoxSubreddit.Name = "TextBoxSubreddit"
TextBoxSubreddit.PlaceholderText = "Subreddit"
TextBoxSubreddit.PlaceholderText = "*Subreddit"
TextBoxSubreddit.Size = New Size(100, 23)
TextBoxSubreddit.TabIndex = 4
ToolTip.SetToolTip(TextBoxSubreddit, "Provide the subreddit to search in.")
ToolTip.SetToolTip(TextBoxSubreddit, "[required] The subreddit to search in.")
'
' ButtonScrape
'
@@ -76,7 +77,7 @@ Partial Class FormMain
ButtonScrape.Size = New Size(51, 28)
ButtonScrape.TabIndex = 6
ButtonScrape.Text = "Scrape"
ToolTip.SetToolTip(ButtonScrape, "You can also just hit ENTER to start scraping.")
ToolTip.SetToolTip(ButtonScrape, "Hitting ENTER will also start the scraping process.")
ButtonScrape.UseVisualStyleBackColor = True
'
' ComboBoxTimeframe
@@ -91,7 +92,7 @@ Partial Class FormMain
ComboBoxTimeframe.Size = New Size(100, 23)
ComboBoxTimeframe.TabIndex = 8
ComboBoxTimeframe.Text = "All"
ToolTip.SetToolTip(ComboBoxTimeframe, "Select the time period for the posts. Default value is `All`.")
ToolTip.SetToolTip(ComboBoxTimeframe, "The time period for the posts. Default value is `All`.")
'
' ComboBoxListing
'
@@ -105,7 +106,7 @@ Partial Class FormMain
ComboBoxListing.Size = New Size(100, 23)
ComboBoxListing.TabIndex = 9
ComboBoxListing.Text = "Top"
ToolTip.SetToolTip(ComboBoxListing, "Choose the type of post listings. Default value is `Top`.")
ToolTip.SetToolTip(ComboBoxListing, "The type of post listings. Default value is `Top`.")
'
' LabelKeyword
'
@@ -123,7 +124,7 @@ Partial Class FormMain
LabelSubreddit.AutoEllipsis = True
LabelSubreddit.Font = New Font("Segoe UI Semibold", 9F, FontStyle.Bold Or FontStyle.Underline, GraphicsUnit.Point)
LabelSubreddit.ForeColor = Color.Black
LabelSubreddit.Location = New Point(19, 52)
LabelSubreddit.Location = New Point(19, 51)
LabelSubreddit.Name = "LabelSubreddit"
LabelSubreddit.Size = New Size(71, 23)
LabelSubreddit.TabIndex = 11
@@ -134,7 +135,7 @@ Partial Class FormMain
LabelLimit.AutoEllipsis = True
LabelLimit.Font = New Font("Segoe UI Semibold", 9F, FontStyle.Bold Or FontStyle.Underline, GraphicsUnit.Point)
LabelLimit.ForeColor = Color.Black
LabelLimit.Location = New Point(19, 75)
LabelLimit.Location = New Point(19, 80)
LabelLimit.Name = "LabelLimit"
LabelLimit.Size = New Size(56, 23)
LabelLimit.TabIndex = 12
@@ -145,7 +146,7 @@ Partial Class FormMain
LabelListing.AutoEllipsis = True
LabelListing.Font = New Font("Segoe UI Semibold", 9F, FontStyle.Bold Or FontStyle.Underline, GraphicsUnit.Point)
LabelListing.ForeColor = Color.Black
LabelListing.Location = New Point(19, 107)
LabelListing.Location = New Point(19, 108)
LabelListing.Name = "LabelListing"
LabelListing.Size = New Size(56, 23)
LabelListing.TabIndex = 13
@@ -156,7 +157,7 @@ Partial Class FormMain
LabelTimeframe.AutoEllipsis = True
LabelTimeframe.Font = New Font("Segoe UI Semibold", 9F, FontStyle.Bold Or FontStyle.Underline, GraphicsUnit.Point)
LabelTimeframe.ForeColor = Color.Black
LabelTimeframe.Location = New Point(19, 136)
LabelTimeframe.Location = New Point(19, 137)
LabelTimeframe.Name = "LabelTimeframe"
LabelTimeframe.Size = New Size(81, 23)
LabelTimeframe.TabIndex = 14
@@ -164,78 +165,85 @@ Partial Class FormMain
'
' ContextMenuStripRightClick
'
ContextMenuStripRightClick.Items.AddRange(New ToolStripItem() {ToolStripMenuItemDarkMode, ToolStripMenuItemSavePosts, ToolStripMenuItemAbout, ToolStripMenuItemDeveloper, ToolStripMenuItemCheckUpdates, ToolStripMenuItemQuit})
ContextMenuStripRightClick.Items.AddRange(New ToolStripItem() {AboutToolStripMenuItem, DeveloperToolStripMenuItem, CheckForUpdatesToolStripMenuItem, SettingsToolStripMenuItem, QuitToolStripMenuItem})
ContextMenuStripRightClick.Name = "ContextMenuStrip1"
ContextMenuStripRightClick.Size = New Size(154, 136)
ContextMenuStripRightClick.Size = New Size(172, 114)
'
' ToolStripMenuItemDarkMode
' SettingsToolStripMenuItem
'
ToolStripMenuItemDarkMode.AutoToolTip = True
ToolStripMenuItemDarkMode.CheckOnClick = True
ToolStripMenuItemDarkMode.Image = CType(resources.GetObject("ToolStripMenuItemDarkMode.Image"), Image)
ToolStripMenuItemDarkMode.Name = "ToolStripMenuItemDarkMode"
ToolStripMenuItemDarkMode.Size = New Size(153, 22)
ToolStripMenuItemDarkMode.Text = "Dark Mode"
SettingsToolStripMenuItem.DropDownItems.AddRange(New ToolStripItem() {DarkModeToolStripMenuItem, SavePostsToolStripMenuItem})
SettingsToolStripMenuItem.Image = CType(resources.GetObject("SettingsToolStripMenuItem.Image"), Image)
SettingsToolStripMenuItem.Name = "SettingsToolStripMenuItem"
SettingsToolStripMenuItem.Size = New Size(171, 22)
SettingsToolStripMenuItem.Text = "Settings"
'
' ToolStripMenuItemSavePosts
' DarkModeToolStripMenuItem
'
ToolStripMenuItemSavePosts.AutoToolTip = True
ToolStripMenuItemSavePosts.DropDownItems.AddRange(New ToolStripItem() {ToolStripMenuItemtoJSON, ToolStripMenuItemtoCSV})
ToolStripMenuItemSavePosts.Image = CType(resources.GetObject("ToolStripMenuItemSavePosts.Image"), Image)
ToolStripMenuItemSavePosts.Name = "ToolStripMenuItemSavePosts"
ToolStripMenuItemSavePosts.Size = New Size(153, 22)
ToolStripMenuItemSavePosts.Text = "Save Posts"
ToolStripMenuItemSavePosts.ToolTipText = "Save found posts to..."
DarkModeToolStripMenuItem.CheckOnClick = True
DarkModeToolStripMenuItem.Image = CType(resources.GetObject("DarkModeToolStripMenuItem.Image"), Image)
DarkModeToolStripMenuItem.Name = "DarkModeToolStripMenuItem"
DarkModeToolStripMenuItem.Size = New Size(180, 22)
DarkModeToolStripMenuItem.Text = "Dark Mode"
'
' ToolStripMenuItemtoJSON
' SavePostsToolStripMenuItem
'
ToolStripMenuItemtoJSON.AutoToolTip = True
ToolStripMenuItemtoJSON.CheckOnClick = True
ToolStripMenuItemtoJSON.Image = CType(resources.GetObject("ToolStripMenuItemtoJSON.Image"), Image)
ToolStripMenuItemtoJSON.Name = "ToolStripMenuItemtoJSON"
ToolStripMenuItemtoJSON.Size = New Size(116, 22)
ToolStripMenuItemtoJSON.Text = "to JSON"
SavePostsToolStripMenuItem.AutoToolTip = True
SavePostsToolStripMenuItem.DropDownItems.AddRange(New ToolStripItem() {ToJSONToolStripMenuItem, ToCSVToolStripMenuItem})
SavePostsToolStripMenuItem.Image = CType(resources.GetObject("SavePostsToolStripMenuItem.Image"), Image)
SavePostsToolStripMenuItem.Name = "SavePostsToolStripMenuItem"
SavePostsToolStripMenuItem.Size = New Size(180, 22)
SavePostsToolStripMenuItem.Text = "Save posts"
'
' ToolStripMenuItemtoCSV
' ToJSONToolStripMenuItem
'
ToolStripMenuItemtoCSV.AutoToolTip = True
ToolStripMenuItemtoCSV.Enabled = False
ToolStripMenuItemtoCSV.Image = CType(resources.GetObject("ToolStripMenuItemtoCSV.Image"), Image)
ToolStripMenuItemtoCSV.Name = "ToolStripMenuItemtoCSV"
ToolStripMenuItemtoCSV.Size = New Size(116, 22)
ToolStripMenuItemtoCSV.Text = "to CSV"
ToJSONToolStripMenuItem.AutoToolTip = True
ToJSONToolStripMenuItem.CheckOnClick = True
ToJSONToolStripMenuItem.Image = CType(resources.GetObject("ToJSONToolStripMenuItem.Image"), Image)
ToJSONToolStripMenuItem.Name = "ToJSONToolStripMenuItem"
ToJSONToolStripMenuItem.Size = New Size(180, 22)
ToJSONToolStripMenuItem.Text = "to JSON"
'
' ToolStripMenuItemAbout
' ToCSVToolStripMenuItem
'
ToolStripMenuItemAbout.AutoToolTip = True
ToolStripMenuItemAbout.Image = CType(resources.GetObject("ToolStripMenuItemAbout.Image"), Image)
ToolStripMenuItemAbout.Name = "ToolStripMenuItemAbout"
ToolStripMenuItemAbout.Size = New Size(153, 22)
ToolStripMenuItemAbout.Text = "About"
ToCSVToolStripMenuItem.AutoToolTip = True
ToCSVToolStripMenuItem.CheckOnClick = True
ToCSVToolStripMenuItem.Image = CType(resources.GetObject("ToCSVToolStripMenuItem.Image"), Image)
ToCSVToolStripMenuItem.Name = "ToCSVToolStripMenuItem"
ToCSVToolStripMenuItem.Size = New Size(180, 22)
ToCSVToolStripMenuItem.Text = "to CSV"
'
' ToolStripMenuItemDeveloper
' AboutToolStripMenuItem
'
ToolStripMenuItemDeveloper.AutoToolTip = True
ToolStripMenuItemDeveloper.Image = CType(resources.GetObject("ToolStripMenuItemDeveloper.Image"), Image)
ToolStripMenuItemDeveloper.Name = "ToolStripMenuItemDeveloper"
ToolStripMenuItemDeveloper.Size = New Size(153, 22)
ToolStripMenuItemDeveloper.Text = "Developer"
AboutToolStripMenuItem.AutoToolTip = True
AboutToolStripMenuItem.Image = CType(resources.GetObject("AboutToolStripMenuItem.Image"), Image)
AboutToolStripMenuItem.Name = "AboutToolStripMenuItem"
AboutToolStripMenuItem.Size = New Size(171, 22)
AboutToolStripMenuItem.Text = "About"
'
' ToolStripMenuItemCheckUpdates
' DeveloperToolStripMenuItem
'
ToolStripMenuItemCheckUpdates.AutoToolTip = True
ToolStripMenuItemCheckUpdates.Image = CType(resources.GetObject("ToolStripMenuItemCheckUpdates.Image"), Image)
ToolStripMenuItemCheckUpdates.Name = "ToolStripMenuItemCheckUpdates"
ToolStripMenuItemCheckUpdates.Size = New Size(153, 22)
ToolStripMenuItemCheckUpdates.Text = "Check Updates"
DeveloperToolStripMenuItem.AutoToolTip = True
DeveloperToolStripMenuItem.Image = CType(resources.GetObject("DeveloperToolStripMenuItem.Image"), Image)
DeveloperToolStripMenuItem.Name = "DeveloperToolStripMenuItem"
DeveloperToolStripMenuItem.Size = New Size(171, 22)
DeveloperToolStripMenuItem.Text = "Developer"
'
' ToolStripMenuItemQuit
' CheckForUpdatesToolStripMenuItem
'
ToolStripMenuItemQuit.AutoToolTip = True
ToolStripMenuItemQuit.Image = CType(resources.GetObject("ToolStripMenuItemQuit.Image"), Image)
ToolStripMenuItemQuit.Name = "ToolStripMenuItemQuit"
ToolStripMenuItemQuit.Size = New Size(153, 22)
ToolStripMenuItemQuit.Text = "Quit"
CheckForUpdatesToolStripMenuItem.AutoToolTip = True
CheckForUpdatesToolStripMenuItem.Image = CType(resources.GetObject("CheckForUpdatesToolStripMenuItem.Image"), Image)
CheckForUpdatesToolStripMenuItem.Name = "CheckForUpdatesToolStripMenuItem"
CheckForUpdatesToolStripMenuItem.Size = New Size(171, 22)
CheckForUpdatesToolStripMenuItem.Text = "Check for Updates"
'
' QuitToolStripMenuItem
'
QuitToolStripMenuItem.AutoToolTip = True
QuitToolStripMenuItem.Font = New Font("Segoe UI Semibold", 9F, FontStyle.Bold, GraphicsUnit.Point)
QuitToolStripMenuItem.Image = CType(resources.GetObject("QuitToolStripMenuItem.Image"), Image)
QuitToolStripMenuItem.Name = "QuitToolStripMenuItem"
QuitToolStripMenuItem.Size = New Size(171, 22)
QuitToolStripMenuItem.Text = "Quit"
'
' NumericUpDownLimit
'
@@ -245,7 +253,7 @@ Partial Class FormMain
NumericUpDownLimit.ReadOnly = True
NumericUpDownLimit.Size = New Size(100, 23)
NumericUpDownLimit.TabIndex = 15
ToolTip.SetToolTip(NumericUpDownLimit, "Set how many posts you want to go through. Default value is `10`.")
ToolTip.SetToolTip(NumericUpDownLimit, "Number of posts to go through. Default value is `10`.")
NumericUpDownLimit.Value = New Decimal(New Integer() {10, 0, 0, 0})
'
' ToolTip
@@ -298,14 +306,16 @@ Partial Class FormMain
Friend WithEvents LabelListing As Label
Friend WithEvents LabelTimeframe As Label
Friend WithEvents ContextMenuStripRightClick As ContextMenuStrip
Friend WithEvents ToolStripMenuItemSavePosts As ToolStripMenuItem
Friend WithEvents ToolStripMenuItemtoJSON As ToolStripMenuItem
Friend WithEvents ToolStripMenuItemtoCSV As ToolStripMenuItem
Friend WithEvents SavePostsToolStripMenuItem As ToolStripMenuItem
Friend WithEvents ToJSONToolStripMenuItem As ToolStripMenuItem
Friend WithEvents ToCSVToolStripMenuItem As ToolStripMenuItem
Friend WithEvents NumericUpDownLimit As NumericUpDown
Friend WithEvents ToolStripMenuItemDarkMode As ToolStripMenuItem
Friend WithEvents ToolStripMenuItemAbout As ToolStripMenuItem
Friend WithEvents ToolStripMenuItemDeveloper As ToolStripMenuItem
Friend WithEvents ToolStripMenuItemCheckUpdates As ToolStripMenuItem
Friend WithEvents ToolStripMenuItemQuit As ToolStripMenuItem
Friend WithEvents AboutToolStripMenuItem As ToolStripMenuItem
Friend WithEvents DeveloperToolStripMenuItem As ToolStripMenuItem
Friend WithEvents CheckForUpdatesToolStripMenuItem As ToolStripMenuItem
Friend WithEvents QuitToolStripMenuItem As ToolStripMenuItem
Friend WithEvents ToolTip As ToolTip
Friend WithEvents SettingsToolStripMenuItem As ToolStripMenuItem
Friend WithEvents DarkModeToolStripMenuItem As ToolStripMenuItem
Friend WithEvents SaveFoundPostsToolStripMenuItem As ToolStripMenuItem
End Class

File diff suppressed because it is too large Load Diff

View File

@@ -1,7 +1,4 @@
Imports System.IO
Imports System.Windows.Forms.VisualStyles.VisualStyleElement
Imports Newtonsoft.Json
Imports Newtonsoft.Json.Linq
Imports Newtonsoft.Json.Linq
Public Class FormMain
ReadOnly settings As New SettingsManager()
@@ -14,32 +11,25 @@ Public Class FormMain
''' <param name="sender">The source of the event.</param>
''' <param name="e">An EventArgs that contains the event data.</param>
Private Sub FormMain_Load(sender As Object, e As EventArgs) Handles MyBase.Load
settings.LoadSettings()
settings.ToggleDarkMode(settings.DarkMode)
settings.ToggleSettings(settings.DarkMode, "darkmode")
settings.ToggleSettings(settings.SaveToJson, "json")
settings.ToggleSettings(settings.SaveToCsv, "csv")
Utilities.PathFinder()
Utilities.LogFirstTimeLaunch()
Me.Text = My.Application.Info.AssemblyName
End Sub
''' <summary>
''' Event handler for the 'Dark Mode' checkbox change event.
''' It toggles the dark mode of the application based on the checkbox status.
''' </summary>
''' <param name="sender">The source of the event.</param>
''' <param name="e">An EventArgs that contains the event data.</param>
Private Sub DarkModeToolStripMenuItem_CheckedChanged(sender As Object, e As EventArgs) Handles ToolStripMenuItemDarkMode.CheckedChanged
settings.ToggleDarkMode(ToolStripMenuItemDarkMode.Checked)
End Sub
''' <summary>
''' Event handler for the 'About' menu item click.
''' It shows the 'About' box.
''' </summary>
''' <param name="sender">The source of the event.</param>
''' <param name="e">An EventArgs that contains the event data.</param>
Private Sub ToolStripMenuItemAbout_Click(sender As Object, e As EventArgs) Handles ToolStripMenuItemAbout.Click
Private Sub ToolStripMenuItemAbout_Click(sender As Object, e As EventArgs) Handles AboutToolStripMenuItem.Click
AboutBox.Show()
End Sub
@@ -50,7 +40,7 @@ Public Class FormMain
''' </summary>
''' <param name="sender">The source of the event.</param>
''' <param name="e">An EventArgs that contains the event data.</param>
Private Sub ToolStripMenuItemDeveloper_Click(sender As Object, e As EventArgs) Handles ToolStripMenuItemDeveloper.Click
Private Sub ToolStripMenuItemDeveloper_Click(sender As Object, e As EventArgs) Handles DeveloperToolStripMenuItem.Click
DeveloperBox.ShowDialog()
End Sub
@@ -61,7 +51,7 @@ Public Class FormMain
''' </summary>
''' <param name="sender">The source of the event.</param>
''' <param name="e">An EventArgs that contains the event data.</param>
Private Sub ToolStripMenuItemCheckUpdates_Click(sender As Object, e As EventArgs) Handles ToolStripMenuItemCheckUpdates.Click
Private Sub ToolStripMenuItemCheckUpdates_Click(sender As Object, e As EventArgs) Handles CheckForUpdatesToolStripMenuItem.Click
Dim data As JObject = ApiHandler.CheckUpdates()
If data("tag_name").ToString = My.Application.Info.Version.ToString Then
MessageBox.Show($"You're running the latest version v{My.Application.Info.Version} of {Me.Text}. Check again soon! :)", $"{Me.Text} v{My.Application.Info.Version}", MessageBoxButtons.OK, MessageBoxIcon.Information)
@@ -83,7 +73,7 @@ Public Class FormMain
''' </summary>
''' <param name="sender">The source of the event.</param>
''' <param name="e">An EventArgs that contains the event data.</param>
Private Sub ToolStripMenuItemQuit_Click(sender As Object, e As EventArgs) Handles ToolStripMenuItemQuit.Click
Private Sub ToolStripMenuItemQuit_Click(sender As Object, e As EventArgs) Handles QuitToolStripMenuItem.Click
Dim result As DialogResult = MessageBox.Show("This will close the program, continue?", "Quit", MessageBoxButtons.YesNo, MessageBoxIcon.Question)
If result = DialogResult.Yes Then
Me.Close()
@@ -99,7 +89,8 @@ Public Class FormMain
''' <param name="sender">The sender of the event.</param>
''' <param name="e">The EventArgs instance containing the event data.</param>
Private Sub ButtonScrape_Click(sender As Object, e As EventArgs) Handles ButtonScrape.Click
Utilities.ProcessRedditPosts(ToolStripMenuItemtoJSON)
settings.LoadSettings()
PostsProcessor.ProcessRedditPosts(settings:=settings)
End Sub
@@ -110,11 +101,13 @@ Public Class FormMain
''' <param name="sender">The source of the event.</param>
''' <param name="e">The <see cref="KeyEventArgs"/> instance containing the event data.</param>
Private Sub TextBoxKeyword_KeyDown(sender As Object, e As KeyEventArgs) Handles TextBoxKeyword.KeyDown
settings.LoadSettings()
' Check if the Enter key is pressed
If e.KeyCode = Keys.Enter Then
' Prevent the beep sound that usually comes with the Enter key in a single-line TextBox
e.SuppressKeyPress = True
Utilities.ProcessRedditPosts(ToolStripMenuItemtoJSON)
PostsProcessor.ProcessRedditPosts(settings:=settings)
End If
End Sub
@@ -126,11 +119,13 @@ Public Class FormMain
''' <param name="sender">The source of the event.</param>
''' <param name="e">The <see cref="KeyEventArgs"/> instance containing the event data.</param>
Private Sub TextBoxSubreddit_KeyDown(sender As Object, e As KeyEventArgs) Handles TextBoxSubreddit.KeyDown
settings.LoadSettings()
' Check if the Enter key is pressed
If e.KeyCode = Keys.Enter Then
' Prevent the beep sound that usually comes with the Enter key in a single-line TextBox
e.SuppressKeyPress = True
Utilities.ProcessRedditPosts(ToolStripMenuItemtoJSON)
PostsProcessor.ProcessRedditPosts(settings:=settings)
End If
End Sub
@@ -142,11 +137,13 @@ Public Class FormMain
''' <param name="sender">The source of the event.</param>
''' <param name="e">The <see cref="KeyEventArgs"/> instance containing the event data.</param>
Private Sub NumericUpDownLimit_KeyDown(sender As Object, e As KeyEventArgs) Handles NumericUpDownLimit.KeyDown
settings.LoadSettings()
' Check if the Enter key is pressed
If e.KeyCode = Keys.Enter Then
' Prevent the beep sound that usually comes with the Enter key in a single-line TextBox
e.SuppressKeyPress = True
Utilities.ProcessRedditPosts(ToolStripMenuItemtoJSON)
PostsProcessor.ProcessRedditPosts(settings:=settings)
End If
End Sub
@@ -158,11 +155,13 @@ Public Class FormMain
''' <param name="sender">The source of the event.</param>
''' <param name="e">The <see cref="KeyEventArgs"/> instance containing the event data.</param>
Private Sub ComboBoxListing_KeyDown(sender As Object, e As KeyEventArgs) Handles ComboBoxListing.KeyDown
settings.LoadSettings()
' Check if the Enter key is pressed
If e.KeyCode = Keys.Enter Then
' Prevent the beep sound that usually comes with the Enter key in a single-line TextBox
e.SuppressKeyPress = True
Utilities.ProcessRedditPosts(ToolStripMenuItemtoJSON)
PostsProcessor.ProcessRedditPosts(settings:=settings)
End If
End Sub
@@ -174,11 +173,43 @@ Public Class FormMain
''' <param name="sender">The source of the event.</param>
''' <param name="e">The <see cref="KeyEventArgs"/> instance containing the event data.</param>
Private Sub ComboBoxTimeframe_KeyDown(sender As Object, e As KeyEventArgs) Handles ComboBoxTimeframe.KeyDown
settings.LoadSettings()
' Check if the Enter key is pressed
If e.KeyCode = Keys.Enter Then
' Prevent the beep sound that usually comes with the Enter key in a single-line TextBox
e.SuppressKeyPress = True
Utilities.ProcessRedditPosts(ToolStripMenuItemtoJSON)
PostsProcessor.ProcessRedditPosts(settings:=settings)
End If
End Sub
''' <summary>
''' Event handler for the 'Dark Mode' checkbox change event.
''' It toggles the dark mode of the application based on the checkbox status.
''' </summary>
''' <param name="sender">The source of the event.</param>
''' <param name="e">An EventArgs that contains the event data.</param>
Private Sub ToolStripMenuItemDarkMode_CheckedChanged(sender As Object, e As EventArgs) Handles DarkModeToolStripMenuItem.CheckedChanged
settings.ToggleSettings(DarkModeToolStripMenuItem.Checked, "darkmode")
End Sub
''' <summary>
''' Event handler for the 'to CSV' checkbox change event.
''' It toggles the dark mode of the application based on the checkbox status.
''' </summary>
''' <param name="sender">The source of the event.</param>
''' <param name="e">An EventArgs that contains the event data.</param>
Private Sub ToCSVToolStripMenuItem_CheckedChanged(sender As Object, e As EventArgs) Handles ToCSVToolStripMenuItem.CheckedChanged
settings.ToggleSettings(ToCSVToolStripMenuItem.Checked, "csv")
End Sub
''' <summary>
''' Event handler for the 'to JSON' checkbox change event.
''' It toggles the dark mode of the application based on the checkbox status.
''' </summary>
''' <param name="sender">The source of the event.</param>
''' <param name="e">An EventArgs that contains the event data.</param>
Private Sub ToJSONToolStripMenuItem_CheckedChanged(sender As Object, e As EventArgs) Handles ToJSONToolStripMenuItem.CheckedChanged
settings.ToggleSettings(ToJSONToolStripMenuItem.Checked, "json")
End Sub
End Class

View File

@@ -26,4 +26,63 @@ Public Class PostsProcessor
Return post("data")("selftext").ToString.ToLower(Globalization.CultureInfo.InvariantCulture).Contains(keyword.ToLower(System.Globalization.CultureInfo.InvariantCulture))
End Function
''' <summary>
''' Collects user inputs, fetches Reddit posts based on the inputs, checks if posts contain the keyword, and saves posts to a JSON file if necessary.
''' </summary>
''' <param name="JSONToolStripMenuItem">Indicates whether to save the posts to a JSON file.</param>
''' <remarks>
''' This function initializes the DataGridView, iterates over each post, adds the posts containing the keyword to the DataGridView and updates the UI.
''' It also shows a message if the keyword was not found in any of the posts or if the inputs are empty.
''' </remarks>
Public Shared Sub ProcessRedditPosts(settings)
' Collect inputs from the user.
Dim inputs = Utilities.CollectInputs()
If inputs.HasValue Then
' Initialize the DataGridView.
DataGridViewHandler.AddColumn(FormPosts.DataGridViewPosts)
' Fetch Reddit posts based on the inputs.
Dim processor As New PostsProcessor()
Dim posts As JObject = processor.FetchPosts(inputs.Value.Subreddit, inputs.Value.Listing, inputs.Value.Limit, inputs.Value.Timeframe)
Dim totalPosts As Integer = 0
Dim keywordFound As Boolean = False
Dim foundPosts As Integer = 0
Dim foundPostsList As New JArray
' Iterate over each post.
For Each post In posts("data")("children")
totalPosts += 1
' Check if the post contains the keyword
If PostsProcessor.PostContainsKeyword(post, inputs.Value.Keyword.ToLower(Globalization.CultureInfo.InvariantCulture)) Then
foundPosts += 1
foundPostsList.Add(post)
' Add the post to the DataGridView.
DataGridViewHandler.AddRow(FormPosts.DataGridViewPosts, post, totalPosts)
FormPosts.Show()
keywordFound = True
End If
Next
' Check if the keyword was found in any posts
If Not keywordFound Then
MessageBox.Show($"Keyword `{inputs.Value.Keyword}` was not found in any of the " + posts("data")("children").Count.ToString(Globalization.CultureInfo.InvariantCulture) _
+ $" {inputs.Value.Listing} posts from r/{inputs.Value.Subreddit}", "Not Found", MessageBoxButtons.OK, MessageBoxIcon.Warning)
End If
If settings.SaveToJson Then
' Save posts to a JSON file if SaveToJson is True.
Utilities.SavePostsToJson(foundPostsList)
End If
If settings.SaveToCsv Then
' Save posts to a CSV file if SaveToCsv is True.
Utilities.SavePostsToCSV(foundPostsList)
End If
Else
End If
End Sub
End Class

View File

@@ -1,33 +1,42 @@
# RPST (Reddit Post Scraping Tool)
Given a subreddit name and a keyword, RPST will return all posts from a specified listing (default is 'top') that contain the provided keyword.
[![Upload Python Package](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/python-publish.yml/badge.svg)](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/python-publish.yml) [![CodeQL](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/codeql.yml/badge.svg)](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/codeql.yml) ![.Net](https://img.shields.io/badge/.NET-5C2D91?style=flat&logo=.net&logoColor=white) ![Python](https://img.shields.io/badge/python-3670A0?style=flat&logo=python&logoColor=ffdd54)
![2023-08-09_04-05](https://github.com/bellingcat/reddit-post-scraping-tool/assets/74001397/d8917a35-3eac-44ce-aa96-1f9685095254)
![2023-08-09_04-05_1](https://github.com/bellingcat/reddit-post-scraping-tool/assets/74001397/d2fe7269-91d4-49ad-87fb-44282c5637a7)
***
[![Upload Python Package](https://github.com/bellingcat/reddit-post-scraping-tool/actions/workflows/python-publish.yml/badge.svg)](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/python-publish.yml) [![CodeQL](https://github.com/bellingcat/reddit-post-scraping-tool/actions/workflows/codeql.yml/badge.svg)](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/codeql.yml) ![.Net](https://img.shields.io/badge/.NET-5C2D91?style=flat&logo=.net&logoColor=white) ![Python](https://img.shields.io/badge/python-3670A0?style=flat&logo=python&logoColor=ffdd54)
# ✅ Features
## GUI
- [x] Dark mode (Right-click)
- [x] Saves results to a JSON (Right-click)
- [x] Logs errors to a file
## *GUI*
- [x] Dark mode (*Right-click*).
- [x] Saves results to a JSON file (*Right-click*).
- [x] Logs errors to a file.
- [x] In-App feature to check for Updates.
## CLI
- [x] Saves results to a JSON (-j/--json)
- [x] Automatically checks for new updates. Notifies user if update were found.
## *CLI*
- [x] Saves results to JSON (*specifiy* `--json`).
- [x] Saves results to CSV (*specify* `--csv`).
- [x] Automatically checks for new updates, and notifies user if updates were found.
# 📃 TODO
## GUI
## *GUI*
- [ ] Make it installable with a setup.exe/setup.msi file.
- [x] Add manual dark mode option, that will be persistent in all sessions
- [ ] Make it save results to a CSV file
- [x] Add manual dark mode option, that will be persistent in all sessions.
- [x] Make settings persistent in all sessions.
- [x] Make it save results to a CSV file.
# 🖥️ Tested environments
## *GUI*
- [x] Microsoft Windows 11
## *CLI*
- [x] Android Termux
- [x] Microsoft Windows 11
- [x] Ubuntu 22.04 - latest versions
# 📖 Wiki
[Refer to the Wiki](https://github.com/rly0nheart/reddit-post-scraping-tool/wiki) for installation instructions, in addition to all other documentation.
[Refer to the Wiki](https://github.com/bellingcat/reddit-post-scraping-tool/wiki) for installation instructions, in addition to all other documentation.
# 😁 Donations
If you like `RPST` and would like to show support, you can Buy A Coffee for the developer using the button below
# 🖼️ Screenshots
You can view a collection of screenshots for both the *CLI* and *GUI* [here](https://github.com/bellingcat/reddit-post-scraping-tool/tree/master/images)
***
<a href="https://www.buymeacoffee.com/_rly0nheart"><img src="https://img.buymeacoffee.com/button-api/?text=Buy me a coffee&emoji=&slug=_rly0nheart&button_colour=40DCA5&font_colour=ffffff&font_family=Comic&outline_colour=000000&coffee_colour=FFDD00" /></a>
<a href="https://www.buymeacoffee.com/_rly0nheart" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/default-orange.png" alt="Buy Me A Coffee" height="41" width="174"></a>
Your support will be much appreciated😊
![me](https://github.com/bellingcat/reddit-post-scraping-tool/assets/74001397/21e0bb33-7a84-45d6-92ba-00e40891ba31)

View File

@@ -13,11 +13,11 @@
<PackageProjectUrl>https://github.com/bellingcat/reddit-post-scraping-tool</PackageProjectUrl>
<PackageReadmeFile>README.md</PackageReadmeFile>
<RepositoryUrl>https://github.com/bellingcat/reddit-post-scraping-tool</RepositoryUrl>
<AssemblyVersion>1.6.2.0</AssemblyVersion>
<FileVersion>1.6.2.0</FileVersion>
<AssemblyVersion>1.8.0.0</AssemblyVersion>
<FileVersion>1.8.0.0</FileVersion>
<PackageLicenseFile>LICENSE</PackageLicenseFile>
<PackageRequireLicenseAcceptance>True</PackageRequireLicenseAcceptance>
<Version>1.6.2</Version>
<Version>1.8.0</Version>
<PackageTags>reddit;scraper;reddit-scraper;osint</PackageTags>
<PackageReleaseNotes></PackageReleaseNotes>
<AnalysisLevel>6.0-recommended</AnalysisLevel>
@@ -39,7 +39,7 @@
</ItemGroup>
<ItemGroup>
<PackageReference Include="Newtonsoft.Json" Version="13.0.2" />
<PackageReference Include="Newtonsoft.Json" Version="13.0.3" />
</ItemGroup>
<ItemGroup>
@@ -78,4 +78,4 @@
</None>
</ItemGroup>
</Project>
</Project>

View File

@@ -10,8 +10,10 @@ Public Class SettingsManager
''' Indicates whether the dark mode is enabled or disabled.
''' </summary>
Public Property DarkMode As Boolean
Public Property SaveToJson As Boolean
Public Property SaveToCsv As Boolean
Private ReadOnly settingsFilePath As String = Path.Combine(Environment.CurrentDirectory, "settings.json")
Private ReadOnly settingsFilePath As String = Path.Combine(Environment.CurrentDirectory, "config.json")
''' <summary>
''' Loads application settings from the 'settings.json' file.
@@ -23,34 +25,55 @@ Public Class SettingsManager
If File.Exists(settingsFilePath) Then
Dim json As String = File.ReadAllText(settingsFilePath)
Dim options As New JsonSerializerOptions With {.PropertyNameCaseInsensitive = True}
Dim settings = Text.Json.JsonSerializer.Deserialize(Of SettingsManager)(json, options)
Me.DarkMode = settings.DarkMode
FormMain.ToolStripMenuItemDarkMode.Checked = settings.DarkMode
Dim settings = JsonSerializer.Deserialize(Of SettingsManager)(json, options)
DarkMode = settings.DarkMode
SaveToJson = settings.SaveToJson
SaveToCsv = settings.SaveToCsv
FormMain.DarkModeToolStripMenuItem.Checked = settings.DarkMode
FormMain.ToJSONToolStripMenuItem.Checked = settings.SaveToJson
FormMain.ToCSVToolStripMenuItem.Checked = settings.SaveToCsv
Else
' Settings file does not exist
' Create a new file with default settings 'False'
Dim defaultSettings = New SettingsManager With {.DarkMode = False}
Dim jsonOutput = Text.Json.JsonSerializer.Serialize(defaultSettings)
Dim defaultSettings = New SettingsManager With {.DarkMode = False, .SaveToCsv = False, .SaveToJson = False}
Dim jsonOutput = JsonSerializer.Serialize(defaultSettings)
File.WriteAllText(settingsFilePath, jsonOutput)
Me.DarkMode = False
FormMain.ToolStripMenuItemDarkMode.Checked = False
DarkMode = False
SaveToJson = False
SaveToCsv = False
FormMain.ToJSONToolStripMenuItem.Checked = False
FormMain.ToCSVToolStripMenuItem.Checked = False
FormMain.DarkModeToolStripMenuItem.Checked = False
End If
End Sub
''' <summary>
''' Toggles the Dark Mode setting on or off based on the provided parameter.
''' Retrieves application settings from a JSON file.
''' </summary>
''' <param name="enabled">A Boolean indicating if Dark Mode should be enabled or not.</param>
Public Sub ToggleDarkMode(enabled As Boolean)
Dim json As String = File.ReadAllText(settingsFilePath)
Dim options As New JsonSerializerOptions With {.PropertyNameCaseInsensitive = True}
Dim settings As SettingsManager = JsonSerializer.Deserialize(Of SettingsManager)(json, options)
settings.DarkMode = enabled
SaveSettings(settings)
ApplyTheme()
End Sub
''' <returns>A Dictionary containing the names and values of all settings.
''' If the settings file doesn't exist, returns a Dictionary with default values.</returns>
Private Function GetSettings() As Dictionary(Of String, Object)
Dim settings As New Dictionary(Of String, Object)
If File.Exists(settingsFilePath) Then
' Read and parse the JSON settings file.
Dim json As String = File.ReadAllText(settingsFilePath)
Dim jObject As JObject = JObject.Parse(json)
' Loop through each property in the JObject and add it to the settings Dictionary.
For Each item As JProperty In jObject.Properties()
settings.Add(item.Name, item.Value.ToObject(Of Object)())
Next
Else
End If
Return settings
End Function
''' <summary>
''' Saves the provided settings to the 'settings.json' file.
@@ -63,12 +86,20 @@ Public Class SettingsManager
''' <summary>
''' Applies the visual theme based on the Dark Mode setting.
''' If Dark Mode is enabled, a dark theme is applied. If it's disabled, a light theme is set.
''' Applies the current settings to the application's interface. This includes
''' toggling SaveToJson, SaveToCsv, and applying the visual theme based on the Dark Mode setting.
''' </summary>
Public Sub ApplyTheme()
Dim DarkMode As Boolean = GetDarkMode()
If DarkMode Then
Public Sub ApplySettings()
' Retrieve the current settings
Dim settings As Dictionary(Of String, Object) = GetSettings()
' Apply the SaveToJson setting to the menu item checkbox
FormMain.ToJSONToolStripMenuItem.Checked = CBool(settings("SaveToJson"))
' Apply the SaveToCsv setting to the menu item checkbox
FormMain.ToCSVToolStripMenuItem.Checked = CBool(settings("SaveToCsv"))
If CBool(settings("DarkMode")) Then
' Enable dark mode for the Main form
' Background colours (I know 'Colours'/'Colors'😆)
FormMain.BackColor = ColorTranslator.FromHtml("#FF121212")
@@ -94,23 +125,25 @@ Public Class SettingsManager
' Enable dark mode on 'Right Click Menu' items
' Background colours
FormMain.ToolStripMenuItemDarkMode.BackColor = ColorTranslator.FromHtml("#FF121212")
FormMain.ToolStripMenuItemSavePosts.BackColor = ColorTranslator.FromHtml("#FF121212")
FormMain.ToolStripMenuItemtoJSON.BackColor = ColorTranslator.FromHtml("#FF121212")
FormMain.ToolStripMenuItemtoCSV.BackColor = ColorTranslator.FromHtml("#FF121212")
FormMain.ToolStripMenuItemAbout.BackColor = ColorTranslator.FromHtml("#FF121212")
FormMain.ToolStripMenuItemDeveloper.BackColor = ColorTranslator.FromHtml("#FF121212")
FormMain.ToolStripMenuItemCheckUpdates.BackColor = ColorTranslator.FromHtml("#FF121212")
FormMain.ToolStripMenuItemQuit.BackColor = ColorTranslator.FromHtml("#FF121212")
FormMain.SettingsToolStripMenuItem.BackColor = ColorTranslator.FromHtml("#FF121212")
FormMain.DarkModeToolStripMenuItem.BackColor = ColorTranslator.FromHtml("#FF121212")
FormMain.SavePostsToolStripMenuItem.BackColor = ColorTranslator.FromHtml("#FF121212")
FormMain.ToJSONToolStripMenuItem.BackColor = ColorTranslator.FromHtml("#FF121212")
FormMain.ToCSVToolStripMenuItem.BackColor = ColorTranslator.FromHtml("#FF121212")
FormMain.AboutToolStripMenuItem.BackColor = ColorTranslator.FromHtml("#FF121212")
FormMain.DeveloperToolStripMenuItem.BackColor = ColorTranslator.FromHtml("#FF121212")
FormMain.CheckForUpdatesToolStripMenuItem.BackColor = ColorTranslator.FromHtml("#FF121212")
FormMain.QuitToolStripMenuItem.BackColor = ColorTranslator.FromHtml("#FF121212")
' Foreground colours
FormMain.ToolStripMenuItemDarkMode.ForeColor = SystemColors.Control
FormMain.ToolStripMenuItemSavePosts.ForeColor = SystemColors.Control
FormMain.ToolStripMenuItemtoJSON.ForeColor = SystemColors.Control
FormMain.ToolStripMenuItemtoCSV.ForeColor = SystemColors.Control
FormMain.ToolStripMenuItemAbout.ForeColor = SystemColors.Control
FormMain.ToolStripMenuItemDeveloper.ForeColor = SystemColors.Control
FormMain.ToolStripMenuItemCheckUpdates.ForeColor = SystemColors.Control
FormMain.ToolStripMenuItemQuit.ForeColor = SystemColors.Control
FormMain.SettingsToolStripMenuItem.ForeColor = SystemColors.Control
FormMain.DarkModeToolStripMenuItem.ForeColor = SystemColors.Control
FormMain.SavePostsToolStripMenuItem.ForeColor = SystemColors.Control
FormMain.ToJSONToolStripMenuItem.ForeColor = SystemColors.Control
FormMain.ToCSVToolStripMenuItem.ForeColor = SystemColors.Control
FormMain.AboutToolStripMenuItem.ForeColor = SystemColors.Control
FormMain.DeveloperToolStripMenuItem.ForeColor = SystemColors.Control
FormMain.CheckForUpdatesToolStripMenuItem.ForeColor = SystemColors.Control
FormMain.QuitToolStripMenuItem.ForeColor = SystemColors.Control
' Enable dark mode for the About box
@@ -123,10 +156,10 @@ Public Class SettingsManager
AboutBox.LicenseRichTextBox.ForeColor = SystemColors.Control
AboutBox.LabelProgramName.ForeColor = SystemColors.Control
AboutBox.LabelProgramDescription.ForeColor = SystemColors.Control
AboutBox.LabelVersion.ForeColor = SystemColors.Control
AboutBox.LinkLabelVersion.ForeColor = SystemColors.Control
' If dark mode is enabled, set the 'Dark Mode' text value to 'Light mode'
FormMain.ToolStripMenuItemDarkMode.Text = "Light Mode"
FormMain.DarkModeToolStripMenuItem.Text = "Dark Mode: Enabled"
Else
' Disable dark mode for the Main Form
' Background colours
@@ -152,23 +185,25 @@ Public Class SettingsManager
' Disable dark mode on 'Right Click Menu' items
' Background colours
FormMain.ToolStripMenuItemDarkMode.BackColor = Color.Gainsboro
FormMain.ToolStripMenuItemSavePosts.BackColor = Color.Gainsboro
FormMain.ToolStripMenuItemtoJSON.BackColor = Color.Gainsboro
FormMain.ToolStripMenuItemtoCSV.BackColor = Color.Gainsboro
FormMain.ToolStripMenuItemAbout.BackColor = Color.Gainsboro
FormMain.ToolStripMenuItemDeveloper.BackColor = Color.Gainsboro
FormMain.ToolStripMenuItemCheckUpdates.BackColor = Color.Gainsboro
FormMain.ToolStripMenuItemQuit.BackColor = Color.Gainsboro
FormMain.SettingsToolStripMenuItem.BackColor = Color.Gainsboro
FormMain.DarkModeToolStripMenuItem.BackColor = Color.Gainsboro
FormMain.SavePostsToolStripMenuItem.BackColor = Color.Gainsboro
FormMain.ToJSONToolStripMenuItem.BackColor = Color.Gainsboro
FormMain.ToCSVToolStripMenuItem.BackColor = Color.Gainsboro
FormMain.AboutToolStripMenuItem.BackColor = Color.Gainsboro
FormMain.DeveloperToolStripMenuItem.BackColor = Color.Gainsboro
FormMain.CheckForUpdatesToolStripMenuItem.BackColor = Color.Gainsboro
FormMain.QuitToolStripMenuItem.BackColor = Color.Gainsboro
' Foreground colours
FormMain.ToolStripMenuItemDarkMode.ForeColor = Color.Black
FormMain.ToolStripMenuItemSavePosts.ForeColor = Color.Black
FormMain.ToolStripMenuItemtoJSON.ForeColor = Color.Black
FormMain.ToolStripMenuItemtoCSV.ForeColor = Color.Black
FormMain.ToolStripMenuItemAbout.ForeColor = Color.Black
FormMain.ToolStripMenuItemDeveloper.ForeColor = Color.Black
FormMain.ToolStripMenuItemCheckUpdates.ForeColor = Color.Black
FormMain.ToolStripMenuItemQuit.ForeColor = Color.Black
FormMain.SettingsToolStripMenuItem.ForeColor = Color.Black
FormMain.DarkModeToolStripMenuItem.ForeColor = Color.Black
FormMain.SavePostsToolStripMenuItem.ForeColor = Color.Black
FormMain.ToJSONToolStripMenuItem.ForeColor = Color.Black
FormMain.ToCSVToolStripMenuItem.ForeColor = Color.Black
FormMain.AboutToolStripMenuItem.ForeColor = Color.Black
FormMain.DeveloperToolStripMenuItem.ForeColor = Color.Black
FormMain.CheckForUpdatesToolStripMenuItem.ForeColor = Color.Black
FormMain.QuitToolStripMenuItem.ForeColor = Color.Black
' Disable dark mode for the About box
' Background colours
@@ -181,26 +216,39 @@ Public Class SettingsManager
AboutBox.Panel1.ForeColor = SystemColors.WindowText
AboutBox.LabelProgramName.ForeColor = SystemColors.WindowText
AboutBox.LabelProgramDescription.ForeColor = SystemColors.WindowText
AboutBox.LabelVersion.ForeColor = SystemColors.WindowText
AboutBox.LinkLabelVersion.ForeColor = SystemColors.WindowText
' If dark mode is disabled, set the 'Light Mode' text value to 'Dark Mode'
FormMain.ToolStripMenuItemDarkMode.Text = "Dark Mode"
FormMain.DarkModeToolStripMenuItem.Text = "Dark Mode: Disabled"
End If
End Sub
''' <summary>
''' Retrieves the Dark Mode setting value from 'settings.json'.
''' If the settings file doesn't exist, defaults to returning 'False' (Dark Mode off).
''' Toggles specific settings on or off based on the provided parameters.
''' </summary>
''' <returns>A Boolean indicating if Dark Mode is enabled or not.</returns>
Private Function GetDarkMode() As Boolean
If File.Exists(settingsFilePath) Then
Dim json As String = File.ReadAllText(settingsFilePath)
Dim settings As JObject = JObject.Parse(json)
Return settings(NameOf(DarkMode)).ToObject(Of Boolean)()
''' <param name="enabled">A Boolean indicating if the setting option should be enabled or not.</param>
''' <param name="saveTo">A String specifying the type of setting to toggle ('json', 'csv', or 'darkmode').</param>
Public Sub ToggleSettings(enabled As Boolean, saveTo As String)
' Read the existing settings from the settings file
Dim json As String = File.ReadAllText(settingsFilePath)
Dim options As New JsonSerializerOptions With {.PropertyNameCaseInsensitive = True}
Dim settings As SettingsManager = JsonSerializer.Deserialize(Of SettingsManager)(json, options)
' Update the settings based on the specified saveTo parameter
If saveTo.ToLower(Globalization.CultureInfo.InvariantCulture) = "json" Then
settings.SaveToJson = enabled
ElseIf saveTo.ToLower(Globalization.CultureInfo.InvariantCulture) = "csv" Then
settings.SaveToCsv = enabled
ElseIf saveTo.ToLower(Globalization.CultureInfo.InvariantCulture) = "darkmode" Then
settings.DarkMode = enabled
Else
Return False
' Handle unexpected value of saveTo (if needed)
End If
End Function
' Save the updated settings back to the settings file
SaveSettings(settings)
' Apply the updated settings to the application
ApplySettings()
End Sub
End Class

View File

@@ -3,54 +3,6 @@ Imports Newtonsoft.Json
Imports Newtonsoft.Json.Linq
Public Class Utilities
''' <summary>
''' Collects user inputs, fetches Reddit posts based on the inputs, checks if posts contain the keyword, and saves posts to a JSON file if necessary.
''' </summary>
''' <param name="JSONToolStripMenuItem">Indicates whether to save the posts to a JSON file.</param>
''' <remarks>
''' This function initializes the DataGridView, iterates over each post, adds the posts containing the keyword to the DataGridView and updates the UI.
''' It also shows a message if the keyword was not found in any of the posts or if the inputs are empty.
''' </remarks>
Public Shared Sub ProcessRedditPosts(JSONToolStripMenuItem As ToolStripMenuItem)
' Collect inputs from the user
Dim inputs = CollectInputs()
If inputs.HasValue Then
' Initialize the DataGridView
DataGridViewHandler.AddColumn(FormPosts.DataGridViewPosts)
' Fetch Reddit posts based on the inputs
Dim processor As New PostsProcessor()
Dim posts As JObject = processor.FetchPosts(inputs.Value.Subreddit, inputs.Value.Listing, inputs.Value.Limit, inputs.Value.Timeframe)
Dim totalPosts As Integer = 0
Dim keywordFound As Boolean = False
' Iterate over each post
For Each post In posts("data")("children")
totalPosts += 1
' Check if the post contains the keyword
If PostsProcessor.PostContainsKeyword(post, inputs.Value.Keyword.ToLower(Globalization.CultureInfo.InvariantCulture)) Then
' Add the post to the DataGridView
DataGridViewHandler.AddRow(FormPosts.DataGridViewPosts, post, totalPosts)
FormPosts.Show()
keywordFound = True
End If
Next
' Check if the keyword was found in any posts
If Not keywordFound Then
MessageBox.Show($"Keyword `{inputs.Value.Keyword}` was not found in any of the " + posts("data")("children").Count.ToString(Globalization.CultureInfo.InvariantCulture) _
+ $" {inputs.Value.Listing} posts from r/{inputs.Value.Subreddit}", "Not Found", MessageBoxButtons.OK, MessageBoxIcon.Warning)
End If
If JSONToolStripMenuItem.Checked Then
' Save posts to a JSON file if the JSONToolStripMenuItem is checked
Utilities.SavePostsToJson(posts("data"))
End If
Else
End If
End Sub
''' <summary>
''' Checks for the existence of the 'logs' directory under the 'RPST' directory within the user's AppData\Roaming folder.
@@ -87,14 +39,18 @@ Public Class Utilities
Public Shared Function CollectInputs() As (Keyword As String, Subreddit As String, Listing As String, Limit As Integer, Timeframe As String)?
Dim keyword As String = FormMain.TextBoxKeyword.Text.Trim()
Dim subreddit As String = FormMain.TextBoxSubreddit.Text.Trim()
' Convert the Listing and Subreddit to lowercase using InvariantCulture
''' <summary>
''' Convert the Listing and Subreddit to lowercase using InvariantCulture.
''' <summary>
Dim listing As String = If(String.IsNullOrEmpty(FormMain.ComboBoxListing.Text), "top", FormMain.ComboBoxListing.Text.ToLower(Globalization.CultureInfo.InvariantCulture).Trim())
Dim timeframe As String = If(String.IsNullOrEmpty(FormMain.ComboBoxTimeframe.Text), "all", FormMain.ComboBoxTimeframe.Text.ToLower(Globalization.CultureInfo.InvariantCulture).Trim())
Dim limit As Integer = FormMain.NumericUpDownLimit.Value
' Validate inputs
''' <summary>
''' Validate inputs.
''' <summary>
If String.IsNullOrEmpty(keyword) AndAlso String.IsNullOrEmpty(subreddit) Then
MessageBox.Show("Keyword and Subreddit fields should not be empty.", "Invalid Inputs", MessageBoxButtons.OK, MessageBoxIcon.Warning)
MessageBox.Show("Keyword and Subreddit should not be empty.", "Invalid Inputs", MessageBoxButtons.OK, MessageBoxIcon.Warning)
Return Nothing
ElseIf String.IsNullOrEmpty(keyword) Then
MessageBox.Show("Keyword field should not be empty.", "Invalid Input", MessageBoxButtons.OK, MessageBoxIcon.Warning)
@@ -137,6 +93,42 @@ Public Class Utilities
End Sub
''' <summary>
''' Saves Reddit posts contained in a JArray to a CSV file.
''' </summary>
''' <param name="posts">A JArray containing the Reddit posts to be saved.</param>
''' <remarks>
''' This function displays a SaveFileDialog to allow the user to specify the file name and location.
''' It then iterates through the JArray to write each post's details (totalPosts, title, subreddit, author, score) into the selected CSV file.
''' </remarks>
Public Shared Sub SavePostsToCSV(posts As JArray)
Dim saveFileDialog As New SaveFileDialog With {
.Filter = "CSV files (*.csv)|*.csv",
.Title = "Save posts to CSV"
}
If saveFileDialog.ShowDialog() = DialogResult.OK Then
Dim fileName As String = saveFileDialog.FileName
Using csvWriter As New StreamWriter(fileName)
''' <summary>
''' Write the header.
''' <summary>
csvWriter.WriteLine("Index,Author,ID,Subreddit,Visibility,Thumbnail,NSFW,Gilded,Upvotes,Upvote Ratio,Downvotes,Award,Top Award,Is cross-postable?,Score,Category,Text,Domain,Permalink,Created At,Approved At,Approved By")
Dim postCount As Integer = 0
For Each post In posts
postCount += 1
csvWriter.WriteLine($"{postCount},{post("data")("author")},{post("data")("id")},{post("data")("subreddit_name_prefixed")},{post("data")("subreddit_type")},{post("data")("thumbnail")},{post("data")("over_18")},{post("data")("gilded")},{post("data")("ups")},{post("data")("upvote_ratio")},{post("data")("downs")},{post("data")("total_awards_received")},{post("data")("top_awarded_type")},{post("data")("is_crosspostable")},{post("data")("score")},{post("data")("category")},{post("data")("selftext")},{post("data")("domain")},{post("data")("permalink")},{post("data")("created")},{post("data")("approved_at_utc")},{post("data")("approved_by")}")
Next
End Using
MessageBox.Show($"Posts saved to {fileName}", "Success", MessageBoxButtons.OK, MessageBoxIcon.Information)
End If
End Sub
''' <summary>
''' Shows the license notice in a messagebox.
''' </summary>

BIN
images/2023-08-08_07-04.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 68 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 75 KiB

BIN
images/2023-08-08_07-12.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 27 KiB

BIN
images/2023-08-25_15-30.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 576 KiB

BIN
images/2023-08-25_15-31.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 519 KiB

BIN
images/2023-08-25_15-35.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 823 KiB

BIN
images/2023-08-25_15-39.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 508 KiB

BIN
images/2023-08-30_03-11.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 15 KiB

BIN
images/2023-08-30_03-12.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 21 KiB

View File

@@ -7,18 +7,18 @@ packages = ["rpst"]
[project]
name = "reddit-post-scraping-tool"
version = "1.6.2.0"
version = "1.8.0.0"
description = "Given a subreddit name and a keyword, RPST returns all top (by default) posts that contain the specified keyword."
readme = "README.md"
requires-python = ">=3.8"
license = {file = "LICENSE"}
keywords = ["osint", "reddit-crawler", "reddit-scraping", "reddit"]
keywords = ["reddit-crawler", "reddit-scraping", "reddit", "reddit-api"]
authors = [{name = "Richard Mwewa", email = "rly0nheart@duck.com"}]
classifiers = [
"Development Status :: 5 - Production/Stable",
"Programming Language :: Python :: 3",
"Programming Language :: Visual Basic",
"Intended Audience :: Information Technology",
"Intended Audience :: End Users/Desktop",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
"Natural Language :: English"
@@ -26,6 +26,7 @@ classifiers = [
dependencies = [
"rich",
"glyphoji",
"requests",
]
@@ -35,4 +36,4 @@ documentation = "https://github.com/bellingcat/reddit-post-scraping-tool/wiki"
repository = "https://github.com/bellingcat/reddit-post-scraping-tool.git"
[project.scripts]
rpst = "rpst.__main:run"
rpst = "rpst.main:run"

View File

@@ -1,263 +0,0 @@
import os
import json
import logging
import argparse
from datetime import datetime
import requests
from rich.tree import Tree
from rich import print as xprint
from rich.markdown import Markdown
from rich.logging import RichHandler
def convert_timestamp_to_datetime(timestamp: int) -> str:
"""
Converts a Unix timestamp to a formatted datetime string.
:param timestamp: The Unix timestamp to be converted.
:return: A formatted datetime string in the format "dd MMMM yyyy, hh:mm:ssAM/PM".
"""
utc_from_timestamp = datetime.utcfromtimestamp(timestamp)
datetime_object = utc_from_timestamp.strftime("%d %B %Y, %I:%M:%S%p")
return datetime_object
def write_post_data(post_data: dict, filename: str) -> str:
"""
Writes post data to a specified JSON file.
:param post_data: A dictionary containing post data.
:param filename: The name of the file to which post data will be written.
:returns: A string representation of the file path.
"""
home_directory = os.path.expanduser("~")
file_path = os.path.join(home_directory, f"{filename}.json")
# Write the data to a JSON file
with open(file_path, "a") as file:
file.write(json.dumps(post_data))
file.write("\n") # write a newline to separate posts.
return file.name
def check_updates(version_tag: str):
"""
This function checks if there's a new release of a project on GitHub. If there is, it logs an
information message and prints the release notes.
:param version_tag: A string representing the current version of the project.
"""
# Make a GET request to the GitHub API to get the latest release of the project.
response = requests.get(
"https://api.github.com/repos/bellingcat/reddit-post-scraping-tool/releases/latest"
).json()
# Check if the latest release's tag matches the current version tag.
if response["tag_name"] != version_tag:
# If not, convert the release notes from Markdown to HTML.
raw_release_notes = response["body"]
markdown_release_notes = Markdown(raw_release_notes)
# Log an info message about the new release.
log.info(
f"A new release of RPST is available ({response['tag_name']}). "
f"Run 'pip install --upgrade reddit-post-scraping-tool' to get the updates."
)
# Print the release notes.
xprint(markdown_release_notes)
def create_post_branch(post: dict, keyword: str, output: bool, tree: Tree) -> Tree:
"""
This function extracts relevant data from a Reddit post and adds it in a tree branch structure,
followed by the post's selftext.
:param post: A dictionary containing the data of a Reddit post.
:param keyword: The keyword that is used to find posts, in his case gets uses as the filename.
:param output: If specified, all found posts will be written to a json file.
:param tree: Tree where the post branch will be added.
:returns: The main tree with added post branches.
"""
# Define the data to extract from the post.
post_data = {
# "Author": post["data"]["author"],
"ID": post["data"]["id"],
"Subreddit": post["data"]["subreddit_name_prefixed"],
"Visibility": post["data"]["subreddit_type"],
"Thumbnail": post["data"]["thumbnail"],
"Gilded": post["data"]["gilded"],
"Upvotes": post["data"]["ups"],
"Upvote ratio": post["data"]["upvote_ratio"],
"Downvotes": post["data"]["downs"],
"Awards": post["data"]["total_awards_received"],
"Top award": post["data"]["top_awarded_type"],
"Is NSFW?": post["data"]["over_18"],
"Is crosspostable?": post["data"]["is_crosspostable"],
"Score": post["data"]["score"],
"Category": post["data"]["category"],
"Domain": post["data"]["domain"],
"Posted on": convert_timestamp_to_datetime(post["data"]["created"]),
"Approved at": post["data"]["approved_at_utc"],
"Approved by": post["data"]["approved_by"],
}
# Add the post's branch to the main tree.
post_branch = tree.add(f":scroll: {post['data']['title']}")
# Add each piece of extracted data as a branch of the post_branch.
for post_key, post_value in post_data.items():
post_branch.add(f"{post_key}: {post_value}", style="dim")
# If -j/--json is passed, write found posts to a json file.
if output:
# This ensures that the post's selftext is also added to the written json file.
post_data["Text"] = post["data"]["selftext"]
output_file = write_post_data(filename=keyword, post_data=post_data)
tree.add(
f":page_facing_up: Post data written/appended to "
f"[italic][link file://{output_file}]{output_file}[/]"
)
post_branch.add(post["data"]["selftext"], style="italic")
return tree
def get_posts(arguments: argparse):
"""
Scrapes a given subreddit for posts that contain a specified keyword.
The search is limited by the number of posts and timeframe specified.
:param arguments: Namespace object from argparse.
Expected Object Attributes
--------------------------
- keyword: The keyword to search for in the posts.
- subreddit: The subreddit to scrape.
- listing: The type of posts to scrape. This could be 'hot', 'new', etc.
- timeframe: The timeframe from which to scrape posts. This could be 'day', 'week', etc.
- limit: The maximum number of posts to scrape.
- json: If specified, all found posts will be written to a json file.
"""
keyword = arguments.keyword
subreddit = arguments.subreddit
listing = arguments.listing
timeframe = arguments.timeframe
limit = arguments.limit
json_output = arguments.json
# Create main result tree.
main_tree = Tree(f"[bold]{datetime.now()}[/]", guide_style="bold bright_blue")
# Start a new session
session = requests.session()
# Set the User-Agent to mimic a Safari browser on a Mac.
session.headers = {
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, "
"like Gecko) Version/14.1.1 Safari/605.1.15"
}
# Send a GET request to the specified subreddit and listing,
# limiting the response by the specified limit and timeframe.
response = session.get(
f"https://reddit.com/r/{subreddit}/{listing}"
f".json?limit={limit}&t={timeframe}"
).json()
# Initialize a counter for the number of posts found that contain the keyword.
found_posts = 0
# Loop through each post in the response
for post_index, post in enumerate(response["data"]["children"], start=1):
# If the keyword is found in the post's selftext or title, increment the counter and process the post.
if (
keyword.lower() in post["data"]["selftext"]
or keyword.lower() in post["data"]["title"]
):
# Create a branch for found post(s) and show post index and post author as the title
found_tree = main_tree.add(
f":bust_in_silhouette: #{post_index} by [bold]@{post['data']['author']}[/]"
)
found_posts += 1
create_post_branch(
post=post,
keyword=keyword,
output=json_output,
tree=found_tree,
)
# Log the number of posts in which the keyword was found
main_tree.add(
f"Keyword ('{keyword}') was found in {found_posts}/{len(response['data']['children'])} "
f"{listing} posts from r/{subreddit}."
)
xprint(main_tree)
def create_parser():
"""
Creates and configures an argument parser for the command line arguments.
:return: A configured argparse.ArgumentParser object ready to parse the command line arguments.
"""
parser = argparse.ArgumentParser(
description="RPST (Reddit Post Scraping Tool) —by Richard Mwewa | https://about.me/rly0nheart",
epilog="Given a subreddit name and a keyword, "
"RPST returns all top (by default) posts that contain the specified keyword.",
)
parser.add_argument(
"-k", "--keyword", help="The keyword to search for in the posts.", required=True
)
parser.add_argument(
"-s", "--subreddit", help="The subreddit to scrape.", required=True
)
parser.add_argument(
"-c",
"--limit",
help="The maximum number of posts to scrape (1-100). (default: %(default)s)",
default=10,
type=int,
choices=range(
1, 101
), # This enforces that the limit must be between 1 and 100 inclusive.
)
parser.add_argument(
"-l",
"--listing",
default="top",
const="top",
nargs="?",
choices=["controversial", "hot", "best", "new", "rising"],
help="The type of posts to scrape (default: %(default)s)",
)
parser.add_argument(
"-t",
"--timeframe",
default="all",
const="all",
nargs="?",
choices=["hour", "day", "week", "month", "year", "all"],
help="The timeframe from which to scrape posts (default: %(default)s)",
)
parser.add_argument(
"-j",
"--json",
help="Write all found posts to a json file.",
action="store_true",
)
return parser
logging.basicConfig(
level="NOTSET",
format="%(message)s",
handlers=[
RichHandler(markup=True, log_time_format="[%H:%M:%S%p]", show_level=False)
],
)
log = logging.getLogger("rich")

View File

@@ -1,5 +1,7 @@
from datetime import datetime
from rpst.__rpst import log, get_posts, check_updates, create_parser
from .rpst import get_posts
from .utils import create_parser, set_loglevel, check_updates
def run():
@@ -10,20 +12,22 @@ def run():
# Create a parser and parse the command line arguments
parser = create_parser()
arguments = parser.parse_args()
args = parser.parse_args()
log = set_loglevel(debug_mode=args.debug)
# Record the start time
start_time = datetime.now()
try:
# Check for updates
check_updates(version_tag="1.6.2.0")
check_updates(version_tag="1.8.0.0")
# Get posts with the provided/parsed arguments
get_posts(arguments=arguments)
get_posts(args=args)
except KeyboardInterrupt:
log.warning("User interruption detected.")
log.warning("User interruption detected ([yellow]Ctrl+C[/]).")
except Exception as e:
log.error(f"An error occurred: {e}")
log.error(f"An error occurred: [red]{e}[/]")
finally:
log.info(f'Finished in {datetime.now() - start_time} seconds.')
log.info(f"Finished in {datetime.now() - start_time} seconds.")

131
rpst/rpst.py Normal file
View File

@@ -0,0 +1,131 @@
import argparse
from datetime import datetime
import requests
from glyphoji import glyph
from rich import print
from rich.tree import Tree
from .utils import convert_timestamp_to_datetime, write_post_data
def create_post_branch(post: dict, keyword: str, tree: Tree, args: argparse) -> Tree:
"""
This function extracts relevant data from a Reddit post and adds it in a tree branch structure,
followed by the post's selftext.
:param post: A dictionary containing the data of a Reddit post.
:param keyword: The keyword that is used to find posts, in his case gets uses as the filename.
:param tree: Tree where the post branch will be added.
:param args: A namespace object from argparse.
:returns: The main tree with added post branches.
"""
# Define the data to extract from the post.
post_data = {
# "Author": post["data"]["author"],
f"{glyph.id_button} ID": post["data"]["id"],
f"{glyph.people_hugging} Subreddit": post["data"]["subreddit_name_prefixed"],
f"{glyph.face_with_peeking_eye} Visibility": post["data"]["subreddit_type"],
f"{glyph.framed_picture} Thumbnail": post["data"]["thumbnail"],
f"{glyph.white_question_mark} Gilded": post["data"]["gilded"],
f"{glyph.up_arrow} Upvotes": post["data"]["ups"],
f"{glyph.chart_increasing} Upvote ratio": post["data"]["upvote_ratio"],
f"{glyph.down_arrow} Downvotes": post["data"]["downs"],
f"{glyph.trophy} Awards": post["data"]["total_awards_received"],
f"{glyph.trophy} Top award": post["data"]["top_awarded_type"],
f"{glyph.no_one_under_eighteen} Is NSFW?": post["data"]["over_18"],
f"{glyph.left_arrow_curving_right} Is crosspostable?": post["data"][
"is_crosspostable"
],
f"{glyph.bar_chart} Score": post["data"]["score"],
f"{glyph.card_file_box} Category": post["data"]["category"],
f"{glyph.globe_with_meridians} Domain": post["data"]["domain"],
f"{glyph.calendar} Posted on": convert_timestamp_to_datetime(
post["data"]["created"]
),
f"{glyph.calendar} Approved at": post["data"]["approved_at_utc"],
f"{glyph.bust_in_silhouette} Approved by": post["data"]["approved_by"],
}
# Add the post's branch to the main tree.
post_branch = tree.add(f"{glyph.page_with_curl} {post['data']['title']}")
# Add each piece of extracted data as a branch of the post_branch.
for post_key, post_value in post_data.items():
post_branch.add(f"{post_key}: {post_value}", style="dim")
# This ensures that the post's selftext is also added to the written json/csv file.
post_data[f"{glyph.clipboard} Text"] = post["data"]["selftext"]
write_post_data(
filename=keyword, post_data=post_data, tree_branch=post_branch, args=args
)
post_branch.add(post["data"]["selftext"], style="italic")
return tree
def get_posts(args: argparse):
"""
Scrapes a given subreddit for posts that contain a specified keyword.
The search is limited by the number of posts and timeframe specified.
:param args: Namespace object from argparse.
Expected Object Attributes
--------------------------
- keyword: The keyword to search for in the posts.
- subreddit: The subreddit to scrape.
- listing: The type of posts to scrape. This could be 'hot', 'new', etc.
- timeframe: The timeframe from which to scrape posts. This could be 'day', 'week', etc.
- limit: The maximum number of posts to scrape.
- json: If specified, all found posts will be written to a json file.
"""
keyword = args.keyword
subreddit = args.subreddit
listing = args.listing
timeframe = args.timeframe
limit = args.limit
# Create main result tree.
main_tree = Tree(
f"[bold]{glyph.calendar} {datetime.now()}[/]", guide_style="bold bright_blue"
)
# Start a new session
session = requests.session()
# Set the User-Agent to mimic a Safari browser on a Mac.
session.headers = {
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, "
"like Gecko) Version/14.1.1 Safari/605.1.15"
}
# Send a GET request to the specified subreddit and listing,
# limiting the response by the specified limit and timeframe.
response = session.get(
f"https://reddit.com/r/{subreddit}/{listing}"
f".json?limit={limit}&t={timeframe}"
).json()
# Initialize a counter for the number of posts found that contain the keyword.
found_posts = 0
# Loop through each post in the response
for post_index, post in enumerate(response["data"]["children"], start=1):
# If the keyword is found in the post's selftext or title, increment the counter and process the post.
if (
keyword.lower() in post["data"]["selftext"]
or keyword.lower() in post["data"]["title"]
):
# Create a branch for found post(s) and show post index and post author as the title
found_tree = main_tree.add(
f"{glyph.bust_in_silhouette} #{post_index} by [bold]@{post['data']['author']}[/]"
)
found_posts += 1
create_post_branch(post=post, keyword=keyword, tree=found_tree, args=args)
# Log the number of posts in which the keyword was found
main_tree.add(
f"{glyph.check_mark_button} Keyword ('{keyword}') was found in "
f"{found_posts}/{len(response['data']['children'])} {listing} posts from r/{subreddit}."
)
print(main_tree)

183
rpst/utils.py Normal file
View File

@@ -0,0 +1,183 @@
import os
import csv
import json
import logging
import argparse
from datetime import datetime
import requests
from glyphoji import glyph
from rich import print
from rich.tree import Tree
from rich.markdown import Markdown
from rich.logging import RichHandler
def convert_timestamp_to_datetime(timestamp: int) -> str:
"""
Converts a Unix timestamp to a formatted datetime string.
:param timestamp: The Unix timestamp to be converted.
:return: A formatted datetime string in the format "dd MMMM yyyy, hh:mm:ssAM/PM".
"""
utc_from_timestamp = datetime.utcfromtimestamp(timestamp)
datetime_object = utc_from_timestamp.strftime("%d %B %Y, %I:%M:%S%p")
return datetime_object
def create_parser():
"""
Creates and configures an argument parser for the command line arguments.
:return: A configured argparse.ArgumentParser object ready to parse the command line arguments.
"""
parser = argparse.ArgumentParser(
description="RPST (Reddit Post Scraping Tool) —by Richard Mwewa | https://about.me/rly0nheart",
epilog="Given a subreddit name and a keyword, "
"RPST returns all top (by default) posts that contain the specified keyword.",
)
parser.add_argument(
"-k", "--keyword", help="The keyword to search for in the posts.", required=True
)
parser.add_argument(
"-s", "--subreddit", help="The subreddit to scrape.", required=True
)
parser.add_argument(
"-c",
"--limit",
help="The maximum number of posts to scrape (1-100). (default: %(default)s)",
default=10,
type=int,
choices=range(
1, 101
), # This enforces that the limit must be between 1 and 100 inclusive.
)
parser.add_argument(
"-l",
"--listing",
default="top",
const="top",
nargs="?",
choices=["controversial", "hot", "best", "new", "rising"],
help="The type of posts to scrape (default: %(default)s)",
)
parser.add_argument(
"-t",
"--timeframe",
default="all",
const="all",
nargs="?",
choices=["hour", "day", "week", "month", "year", "all"],
help="The timeframe from which to scrape posts (default: %(default)s)",
)
parser.add_argument(
"--json",
help="Write all found posts to a json file.",
action="store_true",
)
parser.add_argument(
"--csv",
help="Write all found posts to a csv file.",
action="store_true",
)
parser.add_argument(
"-d",
"--debug",
help="run rpst in debug mode",
action="store_true",
)
return parser
def check_updates(version_tag: str):
"""
This function checks if there's a new release of a project on GitHub. If there is, it logs an
information message and prints the release notes.
:param version_tag: A string representing the current version of the project.
"""
# Make a GET request to the GitHub API to get the latest release of the project.
response = requests.get(
"https://api.github.com/repos/bellingcat/reddit-post-scraping-tool/releases/latest"
).json()
# Check if the latest release's tag matches the current version tag.
if response["tag_name"] != version_tag:
# If not, convert the release notes from Markdown to HTML.
raw_release_notes = response["body"]
# Log an info message about the new release.
print(
f"{glyph.up_arrow} A new release of RPST is available ({response['tag_name']}). "
f"Run 'pip install --upgrade reddit-post-scraping-tool' to get the updates."
)
# Print the release notes.
print(Markdown(raw_release_notes))
def set_loglevel(debug_mode: bool) -> logging.getLogger:
"""
Configure and return a logging object with the specified log level.
:param debug_mode: If True, the log level is set to "NOTSET". Otherwise, it is set to "INFO".
:return: A logging object configured with the specified log level.
"""
logging.basicConfig(
level="NOTSET" if debug_mode else "INFO",
format="%(message)s",
handlers=[
RichHandler(markup=True, log_time_format="[%I:%M:%S %p]", show_level=False)
],
)
return logging.getLogger("RPST")
def write_post_data(post_data: dict, filename: str, args, tree_branch: Tree):
"""
Writes post data to a specified JSON or CSV file based on the args provided, and updates
the provided tree with the status.
:param post_data: A dictionary containing post data.
:param filename: The name of the file to which post data will be written.
:param args: A namespace object from argparse containing the output format options (args.json or args.csv).
:param tree_branch: A rich Tree object to which status information will be added.
"""
home_directory = os.path.expanduser("~")
if args.json:
json_file_path = os.path.join(home_directory, f"{filename}.json")
with open(json_file_path, "a", encoding="utf-8") as file:
file.write(json.dumps(post_data, ensure_ascii=False))
file.write("\n") # Separate posts with newline
tree_branch.add(
f"{glyph.page_facing_up} JSON data successfully written/appended to file: "
f"[italic][link file://{json_file_path}]{json_file_path}[/]"
)
else:
tree_branch.add(
f"{glyph.cross_mark_button} JSON data writing operation was skipped. No changes made."
)
if args.csv:
csv_file_path = os.path.join(home_directory, f"{filename}.csv")
with open(csv_file_path, "a", newline="", encoding="utf-8") as csvfile:
writer = csv.DictWriter(csvfile, fieldnames=post_data.keys())
# Write headers if file is empty
if csvfile.tell() == 0:
writer.writeheader()
writer.writerow(post_data)
tree_branch.add(
f"{glyph.page_facing_up} CSV data successfully written/appended to file: "
f"[italic][link file://{csv_file_path}]{csv_file_path}[/]"
)
else:
tree_branch.add(
f"{glyph.cross_mark_button} CSV data writing operation was skipped. No changes made."
)