Compare commits
51 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5468d93976 | ||
|
|
d186f6f7da | ||
|
|
b5d52e4bb5 | ||
|
|
a2bdc66a03 | ||
|
|
bb72360699 | ||
|
|
adc15555e3 | ||
|
|
07ad626dd2 | ||
|
|
2729c984bc | ||
|
|
f3445b8e06 | ||
|
|
358f264bdd | ||
|
|
e3dda99233 | ||
|
|
7198e0be90 | ||
|
|
f9f0ed5085 | ||
|
|
b0c53a1511 | ||
|
|
54b9abc4b6 | ||
|
|
4ae14ff02e | ||
|
|
376eeab243 | ||
|
|
6d427849d5 | ||
|
|
ad8cb63541 | ||
|
|
57f8c24cee | ||
|
|
750967c322 | ||
|
|
cfef86cbe3 | ||
|
|
2a2696403d | ||
|
|
b0a8d75d8c | ||
|
|
b31c38f5cc | ||
|
|
b5b7df868e | ||
|
|
566f558720 | ||
|
|
c3e5ce6441 | ||
|
|
7c164938c9 | ||
|
|
b08c4a147b | ||
|
|
8f259b7a40 | ||
|
|
f117c99cc7 | ||
|
|
3a9a87e67c | ||
|
|
cce254e976 | ||
|
|
418b2acc4c | ||
|
|
d26699cc1f | ||
|
|
9efb1cea4a | ||
|
|
ba6eeb38a6 | ||
|
|
2053c0f0bc | ||
|
|
8bef73001c | ||
|
|
c9d9628326 | ||
|
|
2f1619b4c5 | ||
|
|
33db66dbc3 | ||
|
|
bbbdab906d | ||
|
|
74264224a5 | ||
|
|
ce75d40f76 | ||
|
|
406e34c4bb | ||
|
|
38140ea2be | ||
|
|
4c3d3a688f | ||
|
|
a03b649904 | ||
|
|
aa3b506a96 |
49
README.md
@@ -1,33 +1,42 @@
|
||||
# RPST (Reddit Post Scraping Tool)
|
||||
Given a subreddit name and a keyword, RPST will return all posts from a specified listing (default is 'top') that contain the provided keyword.
|
||||
|
||||
[](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/python-publish.yml) [](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/codeql.yml)  
|
||||

|
||||

|
||||
***
|
||||
[](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/python-publish.yml) [](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/codeql.yml)  
|
||||
|
||||
# ✅ Features
|
||||
## GUI
|
||||
- [x] Dark mode (Right-click)
|
||||
- [x] Saves results to a JSON (Right-click)
|
||||
- [x] Logs errors to a file
|
||||
## *GUI*
|
||||
- [x] Dark mode (*Right-click*).
|
||||
- [x] Saves results to a JSON file (*Right-click*).
|
||||
- [x] Logs errors to a file.
|
||||
- [x] In-App feature to check for Updates.
|
||||
|
||||
## CLI
|
||||
- [x] Saves results to a JSON (-j/--json)
|
||||
- [x] Automatically checks for new updates. Notifies user if update were found.
|
||||
## *CLI*
|
||||
- [x] Saves results to JSON (*specifiy* `--json`).
|
||||
- [x] Saves results to CSV (*specify* `--csv`).
|
||||
- [x] Automatically checks for new updates, and notifies user if updates were found.
|
||||
|
||||
# 📃 TODO
|
||||
## GUI
|
||||
## *GUI*
|
||||
- [ ] Make it installable with a setup.exe/setup.msi file.
|
||||
- [x] Add manual dark mode option, that will be persistent in all sessions
|
||||
- [ ] Make it save results to a CSV file
|
||||
- [x] Add manual dark mode option, that will be persistent in all sessions.
|
||||
- [x] Make settings persistent in all sessions.
|
||||
- [x] Make it save results to a CSV file.
|
||||
|
||||
# 🖥️ Tested environments
|
||||
## *GUI*
|
||||
- [x] Microsoft Windows 11
|
||||
|
||||
## *CLI*
|
||||
- [x] Android Termux
|
||||
- [x] Microsoft Windows 11
|
||||
- [x] Ubuntu 22.04 - latest versions
|
||||
|
||||
# 📖 Wiki
|
||||
[Refer to the Wiki](https://github.com/rly0nheart/reddit-post-scraping-tool/wiki) for installation instructions, in addition to all other documentation.
|
||||
[Refer to the Wiki](https://github.com/bellingcat/reddit-post-scraping-tool/wiki) for installation instructions, in addition to all other documentation.
|
||||
|
||||
# 😁 Donations
|
||||
If you like `RPST` and would like to show support, you can Buy A Coffee for the developer using the button below
|
||||
# 🖼️ Screenshots
|
||||
You can view a collection of screenshots for both the *CLI* and *GUI* [here](https://github.com/bellingcat/reddit-post-scraping-tool/tree/master/images)
|
||||
***
|
||||
<a href="https://www.buymeacoffee.com/_rly0nheart"><img src="https://img.buymeacoffee.com/button-api/?text=Buy me a coffee&emoji=&slug=_rly0nheart&button_colour=40DCA5&font_colour=ffffff&font_family=Comic&outline_colour=000000&coffee_colour=FFDD00" /></a>
|
||||
|
||||
<a href="https://www.buymeacoffee.com/_rly0nheart" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/default-orange.png" alt="Buy Me A Coffee" height="41" width="174"></a>
|
||||
|
||||
Your support will be much appreciated😊
|
||||

|
||||
|
||||
27
RPST GUI/RPST/AboutBox.Designer.vb
generated
@@ -26,9 +26,9 @@ Partial Class AboutBox
|
||||
PictureBoxLogo = New PictureBox()
|
||||
LabelProgramName = New Label()
|
||||
LabelProgramDescription = New Label()
|
||||
LabelVersion = New Label()
|
||||
LinkLabelReadtheWiki = New LinkLabel()
|
||||
Panel1 = New Panel()
|
||||
LinkLabelVersion = New LinkLabel()
|
||||
LicenseRichTextBox = New RichTextBox()
|
||||
CType(PictureBoxLogo, ComponentModel.ISupportInitialize).BeginInit()
|
||||
Panel1.SuspendLayout()
|
||||
@@ -67,17 +67,6 @@ Partial Class AboutBox
|
||||
LabelProgramDescription.TabIndex = 4
|
||||
LabelProgramDescription.Text = "Description"
|
||||
'
|
||||
' LabelVersion
|
||||
'
|
||||
LabelVersion.AutoSize = True
|
||||
LabelVersion.Font = New Font("Segoe UI", 9F, FontStyle.Underline, GraphicsUnit.Point)
|
||||
LabelVersion.ForeColor = SystemColors.ControlText
|
||||
LabelVersion.Location = New Point(347, 17)
|
||||
LabelVersion.Name = "LabelVersion"
|
||||
LabelVersion.Size = New Size(45, 15)
|
||||
LabelVersion.TabIndex = 5
|
||||
LabelVersion.Text = "Version"
|
||||
'
|
||||
' LinkLabelReadtheWiki
|
||||
'
|
||||
LinkLabelReadtheWiki.AutoSize = True
|
||||
@@ -92,15 +81,25 @@ Partial Class AboutBox
|
||||
' Panel1
|
||||
'
|
||||
Panel1.BackColor = SystemColors.Control
|
||||
Panel1.Controls.Add(LinkLabelVersion)
|
||||
Panel1.Controls.Add(LabelProgramDescription)
|
||||
Panel1.Controls.Add(LabelProgramName)
|
||||
Panel1.Controls.Add(LinkLabelReadtheWiki)
|
||||
Panel1.Controls.Add(LabelVersion)
|
||||
Panel1.Location = New Point(106, 12)
|
||||
Panel1.Name = "Panel1"
|
||||
Panel1.Size = New Size(409, 93)
|
||||
Panel1.TabIndex = 7
|
||||
'
|
||||
' LinkLabelVersion
|
||||
'
|
||||
LinkLabelVersion.AutoSize = True
|
||||
LinkLabelVersion.Location = New Point(347, 17)
|
||||
LinkLabelVersion.Name = "LinkLabelVersion"
|
||||
LinkLabelVersion.Size = New Size(45, 15)
|
||||
LinkLabelVersion.TabIndex = 7
|
||||
LinkLabelVersion.TabStop = True
|
||||
LinkLabelVersion.Text = "Version"
|
||||
'
|
||||
' LicenseRichTextBox
|
||||
'
|
||||
LicenseRichTextBox.Font = New Font("Cambria", 9.75F, FontStyle.Regular, GraphicsUnit.Point)
|
||||
@@ -137,8 +136,8 @@ Partial Class AboutBox
|
||||
Friend WithEvents PictureBoxLogo As PictureBox
|
||||
Friend WithEvents LabelProgramName As Label
|
||||
Friend WithEvents LabelProgramDescription As Label
|
||||
Friend WithEvents LabelVersion As Label
|
||||
Friend WithEvents LinkLabelReadtheWiki As LinkLabel
|
||||
Friend WithEvents Panel1 As Panel
|
||||
Friend WithEvents LicenseRichTextBox As RichTextBox
|
||||
Friend WithEvents LinkLabelVersion As LinkLabel
|
||||
End Class
|
||||
|
||||
@@ -18,7 +18,7 @@
|
||||
<resheader name="reader">System.Resources.ResXResourceReader, System.Windows.Forms, ...</resheader>
|
||||
<resheader name="writer">System.Resources.ResXResourceWriter, System.Windows.Forms, ...</resheader>
|
||||
<data name="Name1"><value>this is my long string</value><comment>this is a comment</comment></data>
|
||||
<data name="Color1" type="System.Drawing.Color, System.Drawing"">Blue</data>
|
||||
<data name="Color1" type="System.Drawing.Color, System.Drawing">Blue</data>
|
||||
<data name="Bitmap1" mimetype="application/x-microsoft.net.object.binary.base64">
|
||||
<value>[base64 mime encoded serialized .NET Framework object]</value>
|
||||
</data>
|
||||
|
||||
@@ -31,13 +31,13 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SO
|
||||
''' <param name="e">The event data.</param>
|
||||
Private Sub AboutBox_Load(sender As Object, e As EventArgs) Handles MyBase.Load
|
||||
settings.LoadSettings()
|
||||
settings.ToggleDarkMode(settings.DarkMode)
|
||||
settings.ToggleSettings(settings.DarkMode, "darkmode")
|
||||
|
||||
LabelProgramName.Text = My.Application.Info.ProductName
|
||||
LabelProgramDescription.Text = "Given a subreddit name and a keyword,
|
||||
RPST returns all top posts (by default)
|
||||
that contain the specified keyword."
|
||||
LabelVersion.Text = $"v{My.Application.Info.Version}"
|
||||
LinkLabelVersion.Text = $"v{My.Application.Info.Version}"
|
||||
LicenseRichTextBox.Text = LicenseText
|
||||
End Sub
|
||||
|
||||
@@ -50,4 +50,8 @@ that contain the specified keyword."
|
||||
Private Sub LinkLabelReadtheWiki_LinkClicked(sender As Object, e As LinkLabelLinkClickedEventArgs) Handles LinkLabelReadtheWiki.LinkClicked
|
||||
Shell("cmd /c start https://github.com/bellingcat/reddit-post-scraping-tool/wiki")
|
||||
End Sub
|
||||
|
||||
Private Sub LinkLabelVersion_LinkClicked(sender As Object, e As LinkLabelLinkClickedEventArgs) Handles LinkLabelVersion.LinkClicked
|
||||
Shell($"cmd /c start https://github.com/bellingcat/reddit-post-scraping-tool/releases/tag/{My.Application.Info.Version}")
|
||||
End Sub
|
||||
End Class
|
||||
|
||||
@@ -6,32 +6,34 @@ Public Class DataGridViewHandler
|
||||
''' </summary>
|
||||
''' <param name="dataGridView">The DataGridView to be initialized.</param>
|
||||
Public Shared Sub AddColumn(dataGridView As DataGridView)
|
||||
' Clear the Columns and Rows before adding Items to them
|
||||
''' <summary>
|
||||
''' Clear the Columns and Rows before adding Items to them.
|
||||
''' <summary>
|
||||
dataGridView.Rows.Clear()
|
||||
dataGridView.Columns.Clear()
|
||||
|
||||
dataGridView.Columns.Add("PostCount", "INDEX")
|
||||
dataGridView.Columns.Add("PostAuthor", "AUTHOR")
|
||||
dataGridView.Columns.Add("PostID", "ID")
|
||||
dataGridView.Columns.Add("PostText", "TEXT")
|
||||
dataGridView.Columns.Add("PostSubreddit", "SUBREDDIT")
|
||||
dataGridView.Columns.Add("SubredditVisibility", "VISIBILITY")
|
||||
dataGridView.Columns.Add("PostThumbnail", "THUMBNAIL")
|
||||
dataGridView.Columns.Add("PostIsNSFW", "NSFW")
|
||||
dataGridView.Columns.Add("PostIsGilded", "GILDED")
|
||||
dataGridView.Columns.Add("PostUpvotes", "UPVOTES")
|
||||
dataGridView.Columns.Add("PostUpvoteRatio", "UPVOTE RATIO")
|
||||
dataGridView.Columns.Add("PostDownvotes", "DOWNVOTES")
|
||||
dataGridView.Columns.Add("PostAwards", "AWARDS")
|
||||
dataGridView.Columns.Add("PostTopAward", "TOP AWARD")
|
||||
dataGridView.Columns.Add("PostIsCrosspostable", "IS CROSS-POSTABLE?")
|
||||
dataGridView.Columns.Add("PostScore", "SCORE")
|
||||
dataGridView.Columns.Add("PostCategory", "CATEGORY")
|
||||
dataGridView.Columns.Add("PostDomain", "DOMAIN")
|
||||
dataGridView.Columns.Add("PostPermalink", "PERMALINK")
|
||||
dataGridView.Columns.Add("PostCreatedAt", "CREATED AT")
|
||||
dataGridView.Columns.Add("PostApprovedAt", "APPROVED ATt")
|
||||
dataGridView.Columns.Add("PostApprovedBy", "APPROVED BY")
|
||||
dataGridView.Columns.Add("PostCount", "🔢 Index")
|
||||
dataGridView.Columns.Add("PostAuthor", "👤 Author")
|
||||
dataGridView.Columns.Add("PostID", "🆔 ID")
|
||||
dataGridView.Columns.Add("PostText", "📝 Text")
|
||||
dataGridView.Columns.Add("PostSubreddit", "🫂 Subreddit")
|
||||
dataGridView.Columns.Add("SubredditVisibility", "🫣 Visibility")
|
||||
dataGridView.Columns.Add("PostThumbnail", "🖼️ Thumbnail")
|
||||
dataGridView.Columns.Add("PostIsNSFW", "🔞 NSFW")
|
||||
dataGridView.Columns.Add("PostIsGilded", "🥇 Gilded")
|
||||
dataGridView.Columns.Add("PostUpvotes", "⬆️ Upvotes")
|
||||
dataGridView.Columns.Add("PostUpvoteRatio", "📊 Upvote Ratio")
|
||||
dataGridView.Columns.Add("PostDownvotes", "⬇️ Downvotes")
|
||||
dataGridView.Columns.Add("PostAwards", "🏆 Awards")
|
||||
dataGridView.Columns.Add("PostTopAward", "🏆 Top Award")
|
||||
dataGridView.Columns.Add("PostIsCrosspostable", "↪️ Is cross-postable?")
|
||||
dataGridView.Columns.Add("PostScore", "📈 Score")
|
||||
dataGridView.Columns.Add("PostCategory", "🟢 Category")
|
||||
dataGridView.Columns.Add("PostDomain", "🌐 Domain")
|
||||
dataGridView.Columns.Add("PostPermalink", "🔗 Permalink")
|
||||
dataGridView.Columns.Add("PostCreatedAt", "📅 Created At")
|
||||
dataGridView.Columns.Add("PostApprovedAt", "📅 Approved At")
|
||||
dataGridView.Columns.Add("PostApprovedBy", "👤 Approved By")
|
||||
End Sub
|
||||
|
||||
Public Shared Sub AddRow(dataGridView As DataGridView, post As JObject, postNumber As Integer)
|
||||
|
||||
4
RPST GUI/RPST/DeveloperBox.Designer.vb
generated
@@ -61,7 +61,7 @@ Partial Class DeveloperBox
|
||||
GreetingLabel.TabIndex = 3
|
||||
GreetingLabel.Text = "👋🏾Hello, I'm Ritchie"
|
||||
'
|
||||
' DeveloperForm
|
||||
' DeveloperBox
|
||||
'
|
||||
AutoScaleDimensions = New SizeF(7F, 15F)
|
||||
AutoScaleMode = AutoScaleMode.Font
|
||||
@@ -73,7 +73,7 @@ Partial Class DeveloperBox
|
||||
FormBorderStyle = FormBorderStyle.FixedSingle
|
||||
MaximizeBox = False
|
||||
MinimizeBox = False
|
||||
Name = "DeveloperForm"
|
||||
Name = "DeveloperBox"
|
||||
ShowIcon = False
|
||||
ShowInTaskbar = False
|
||||
StartPosition = FormStartPosition.CenterParent
|
||||
|
||||
176
RPST GUI/RPST/FormMain.Designer.vb
generated
@@ -35,14 +35,15 @@ Partial Class FormMain
|
||||
LabelListing = New Label()
|
||||
LabelTimeframe = New Label()
|
||||
ContextMenuStripRightClick = New ContextMenuStrip(components)
|
||||
ToolStripMenuItemDarkMode = New ToolStripMenuItem()
|
||||
ToolStripMenuItemSavePosts = New ToolStripMenuItem()
|
||||
ToolStripMenuItemtoJSON = New ToolStripMenuItem()
|
||||
ToolStripMenuItemtoCSV = New ToolStripMenuItem()
|
||||
ToolStripMenuItemAbout = New ToolStripMenuItem()
|
||||
ToolStripMenuItemDeveloper = New ToolStripMenuItem()
|
||||
ToolStripMenuItemCheckUpdates = New ToolStripMenuItem()
|
||||
ToolStripMenuItemQuit = New ToolStripMenuItem()
|
||||
SettingsToolStripMenuItem = New ToolStripMenuItem()
|
||||
DarkModeToolStripMenuItem = New ToolStripMenuItem()
|
||||
SavePostsToolStripMenuItem = New ToolStripMenuItem()
|
||||
ToJSONToolStripMenuItem = New ToolStripMenuItem()
|
||||
ToCSVToolStripMenuItem = New ToolStripMenuItem()
|
||||
AboutToolStripMenuItem = New ToolStripMenuItem()
|
||||
DeveloperToolStripMenuItem = New ToolStripMenuItem()
|
||||
CheckForUpdatesToolStripMenuItem = New ToolStripMenuItem()
|
||||
QuitToolStripMenuItem = New ToolStripMenuItem()
|
||||
NumericUpDownLimit = New NumericUpDown()
|
||||
ToolTip = New ToolTip(components)
|
||||
ContextMenuStripRightClick.SuspendLayout()
|
||||
@@ -55,19 +56,19 @@ Partial Class FormMain
|
||||
TextBoxKeyword.ForeColor = SystemColors.WindowText
|
||||
TextBoxKeyword.Location = New Point(118, 20)
|
||||
TextBoxKeyword.Name = "TextBoxKeyword"
|
||||
TextBoxKeyword.PlaceholderText = "Keyword"
|
||||
TextBoxKeyword.PlaceholderText = "*Keyword"
|
||||
TextBoxKeyword.Size = New Size(100, 23)
|
||||
TextBoxKeyword.TabIndex = 0
|
||||
ToolTip.SetToolTip(TextBoxKeyword, "Enter the keyword you want to search for.")
|
||||
ToolTip.SetToolTip(TextBoxKeyword, "[required] The keyword to search for.")
|
||||
'
|
||||
' TextBoxSubreddit
|
||||
'
|
||||
TextBoxSubreddit.Location = New Point(118, 49)
|
||||
TextBoxSubreddit.Name = "TextBoxSubreddit"
|
||||
TextBoxSubreddit.PlaceholderText = "Subreddit"
|
||||
TextBoxSubreddit.PlaceholderText = "*Subreddit"
|
||||
TextBoxSubreddit.Size = New Size(100, 23)
|
||||
TextBoxSubreddit.TabIndex = 4
|
||||
ToolTip.SetToolTip(TextBoxSubreddit, "Provide the subreddit to search in.")
|
||||
ToolTip.SetToolTip(TextBoxSubreddit, "[required] The subreddit to search in.")
|
||||
'
|
||||
' ButtonScrape
|
||||
'
|
||||
@@ -76,7 +77,7 @@ Partial Class FormMain
|
||||
ButtonScrape.Size = New Size(51, 28)
|
||||
ButtonScrape.TabIndex = 6
|
||||
ButtonScrape.Text = "Scrape"
|
||||
ToolTip.SetToolTip(ButtonScrape, "You can also just hit ENTER to start scraping.")
|
||||
ToolTip.SetToolTip(ButtonScrape, "Hitting ENTER will also start the scraping process.")
|
||||
ButtonScrape.UseVisualStyleBackColor = True
|
||||
'
|
||||
' ComboBoxTimeframe
|
||||
@@ -91,7 +92,7 @@ Partial Class FormMain
|
||||
ComboBoxTimeframe.Size = New Size(100, 23)
|
||||
ComboBoxTimeframe.TabIndex = 8
|
||||
ComboBoxTimeframe.Text = "All"
|
||||
ToolTip.SetToolTip(ComboBoxTimeframe, "Select the time period for the posts. Default value is `All`.")
|
||||
ToolTip.SetToolTip(ComboBoxTimeframe, "The time period for the posts. Default value is `All`.")
|
||||
'
|
||||
' ComboBoxListing
|
||||
'
|
||||
@@ -105,7 +106,7 @@ Partial Class FormMain
|
||||
ComboBoxListing.Size = New Size(100, 23)
|
||||
ComboBoxListing.TabIndex = 9
|
||||
ComboBoxListing.Text = "Top"
|
||||
ToolTip.SetToolTip(ComboBoxListing, "Choose the type of post listings. Default value is `Top`.")
|
||||
ToolTip.SetToolTip(ComboBoxListing, "The type of post listings. Default value is `Top`.")
|
||||
'
|
||||
' LabelKeyword
|
||||
'
|
||||
@@ -123,7 +124,7 @@ Partial Class FormMain
|
||||
LabelSubreddit.AutoEllipsis = True
|
||||
LabelSubreddit.Font = New Font("Segoe UI Semibold", 9F, FontStyle.Bold Or FontStyle.Underline, GraphicsUnit.Point)
|
||||
LabelSubreddit.ForeColor = Color.Black
|
||||
LabelSubreddit.Location = New Point(19, 52)
|
||||
LabelSubreddit.Location = New Point(19, 51)
|
||||
LabelSubreddit.Name = "LabelSubreddit"
|
||||
LabelSubreddit.Size = New Size(71, 23)
|
||||
LabelSubreddit.TabIndex = 11
|
||||
@@ -134,7 +135,7 @@ Partial Class FormMain
|
||||
LabelLimit.AutoEllipsis = True
|
||||
LabelLimit.Font = New Font("Segoe UI Semibold", 9F, FontStyle.Bold Or FontStyle.Underline, GraphicsUnit.Point)
|
||||
LabelLimit.ForeColor = Color.Black
|
||||
LabelLimit.Location = New Point(19, 75)
|
||||
LabelLimit.Location = New Point(19, 80)
|
||||
LabelLimit.Name = "LabelLimit"
|
||||
LabelLimit.Size = New Size(56, 23)
|
||||
LabelLimit.TabIndex = 12
|
||||
@@ -145,7 +146,7 @@ Partial Class FormMain
|
||||
LabelListing.AutoEllipsis = True
|
||||
LabelListing.Font = New Font("Segoe UI Semibold", 9F, FontStyle.Bold Or FontStyle.Underline, GraphicsUnit.Point)
|
||||
LabelListing.ForeColor = Color.Black
|
||||
LabelListing.Location = New Point(19, 107)
|
||||
LabelListing.Location = New Point(19, 108)
|
||||
LabelListing.Name = "LabelListing"
|
||||
LabelListing.Size = New Size(56, 23)
|
||||
LabelListing.TabIndex = 13
|
||||
@@ -156,7 +157,7 @@ Partial Class FormMain
|
||||
LabelTimeframe.AutoEllipsis = True
|
||||
LabelTimeframe.Font = New Font("Segoe UI Semibold", 9F, FontStyle.Bold Or FontStyle.Underline, GraphicsUnit.Point)
|
||||
LabelTimeframe.ForeColor = Color.Black
|
||||
LabelTimeframe.Location = New Point(19, 136)
|
||||
LabelTimeframe.Location = New Point(19, 137)
|
||||
LabelTimeframe.Name = "LabelTimeframe"
|
||||
LabelTimeframe.Size = New Size(81, 23)
|
||||
LabelTimeframe.TabIndex = 14
|
||||
@@ -164,78 +165,85 @@ Partial Class FormMain
|
||||
'
|
||||
' ContextMenuStripRightClick
|
||||
'
|
||||
ContextMenuStripRightClick.Items.AddRange(New ToolStripItem() {ToolStripMenuItemDarkMode, ToolStripMenuItemSavePosts, ToolStripMenuItemAbout, ToolStripMenuItemDeveloper, ToolStripMenuItemCheckUpdates, ToolStripMenuItemQuit})
|
||||
ContextMenuStripRightClick.Items.AddRange(New ToolStripItem() {AboutToolStripMenuItem, DeveloperToolStripMenuItem, CheckForUpdatesToolStripMenuItem, SettingsToolStripMenuItem, QuitToolStripMenuItem})
|
||||
ContextMenuStripRightClick.Name = "ContextMenuStrip1"
|
||||
ContextMenuStripRightClick.Size = New Size(154, 136)
|
||||
ContextMenuStripRightClick.Size = New Size(172, 114)
|
||||
'
|
||||
' ToolStripMenuItemDarkMode
|
||||
' SettingsToolStripMenuItem
|
||||
'
|
||||
ToolStripMenuItemDarkMode.AutoToolTip = True
|
||||
ToolStripMenuItemDarkMode.CheckOnClick = True
|
||||
ToolStripMenuItemDarkMode.Image = CType(resources.GetObject("ToolStripMenuItemDarkMode.Image"), Image)
|
||||
ToolStripMenuItemDarkMode.Name = "ToolStripMenuItemDarkMode"
|
||||
ToolStripMenuItemDarkMode.Size = New Size(153, 22)
|
||||
ToolStripMenuItemDarkMode.Text = "Dark Mode"
|
||||
SettingsToolStripMenuItem.DropDownItems.AddRange(New ToolStripItem() {DarkModeToolStripMenuItem, SavePostsToolStripMenuItem})
|
||||
SettingsToolStripMenuItem.Image = CType(resources.GetObject("SettingsToolStripMenuItem.Image"), Image)
|
||||
SettingsToolStripMenuItem.Name = "SettingsToolStripMenuItem"
|
||||
SettingsToolStripMenuItem.Size = New Size(171, 22)
|
||||
SettingsToolStripMenuItem.Text = "Settings"
|
||||
'
|
||||
' ToolStripMenuItemSavePosts
|
||||
' DarkModeToolStripMenuItem
|
||||
'
|
||||
ToolStripMenuItemSavePosts.AutoToolTip = True
|
||||
ToolStripMenuItemSavePosts.DropDownItems.AddRange(New ToolStripItem() {ToolStripMenuItemtoJSON, ToolStripMenuItemtoCSV})
|
||||
ToolStripMenuItemSavePosts.Image = CType(resources.GetObject("ToolStripMenuItemSavePosts.Image"), Image)
|
||||
ToolStripMenuItemSavePosts.Name = "ToolStripMenuItemSavePosts"
|
||||
ToolStripMenuItemSavePosts.Size = New Size(153, 22)
|
||||
ToolStripMenuItemSavePosts.Text = "Save Posts"
|
||||
ToolStripMenuItemSavePosts.ToolTipText = "Save found posts to..."
|
||||
DarkModeToolStripMenuItem.CheckOnClick = True
|
||||
DarkModeToolStripMenuItem.Image = CType(resources.GetObject("DarkModeToolStripMenuItem.Image"), Image)
|
||||
DarkModeToolStripMenuItem.Name = "DarkModeToolStripMenuItem"
|
||||
DarkModeToolStripMenuItem.Size = New Size(180, 22)
|
||||
DarkModeToolStripMenuItem.Text = "Dark Mode"
|
||||
'
|
||||
' ToolStripMenuItemtoJSON
|
||||
' SavePostsToolStripMenuItem
|
||||
'
|
||||
ToolStripMenuItemtoJSON.AutoToolTip = True
|
||||
ToolStripMenuItemtoJSON.CheckOnClick = True
|
||||
ToolStripMenuItemtoJSON.Image = CType(resources.GetObject("ToolStripMenuItemtoJSON.Image"), Image)
|
||||
ToolStripMenuItemtoJSON.Name = "ToolStripMenuItemtoJSON"
|
||||
ToolStripMenuItemtoJSON.Size = New Size(116, 22)
|
||||
ToolStripMenuItemtoJSON.Text = "to JSON"
|
||||
SavePostsToolStripMenuItem.AutoToolTip = True
|
||||
SavePostsToolStripMenuItem.DropDownItems.AddRange(New ToolStripItem() {ToJSONToolStripMenuItem, ToCSVToolStripMenuItem})
|
||||
SavePostsToolStripMenuItem.Image = CType(resources.GetObject("SavePostsToolStripMenuItem.Image"), Image)
|
||||
SavePostsToolStripMenuItem.Name = "SavePostsToolStripMenuItem"
|
||||
SavePostsToolStripMenuItem.Size = New Size(180, 22)
|
||||
SavePostsToolStripMenuItem.Text = "Save posts"
|
||||
'
|
||||
' ToolStripMenuItemtoCSV
|
||||
' ToJSONToolStripMenuItem
|
||||
'
|
||||
ToolStripMenuItemtoCSV.AutoToolTip = True
|
||||
ToolStripMenuItemtoCSV.Enabled = False
|
||||
ToolStripMenuItemtoCSV.Image = CType(resources.GetObject("ToolStripMenuItemtoCSV.Image"), Image)
|
||||
ToolStripMenuItemtoCSV.Name = "ToolStripMenuItemtoCSV"
|
||||
ToolStripMenuItemtoCSV.Size = New Size(116, 22)
|
||||
ToolStripMenuItemtoCSV.Text = "to CSV"
|
||||
ToJSONToolStripMenuItem.AutoToolTip = True
|
||||
ToJSONToolStripMenuItem.CheckOnClick = True
|
||||
ToJSONToolStripMenuItem.Image = CType(resources.GetObject("ToJSONToolStripMenuItem.Image"), Image)
|
||||
ToJSONToolStripMenuItem.Name = "ToJSONToolStripMenuItem"
|
||||
ToJSONToolStripMenuItem.Size = New Size(180, 22)
|
||||
ToJSONToolStripMenuItem.Text = "to JSON"
|
||||
'
|
||||
' ToolStripMenuItemAbout
|
||||
' ToCSVToolStripMenuItem
|
||||
'
|
||||
ToolStripMenuItemAbout.AutoToolTip = True
|
||||
ToolStripMenuItemAbout.Image = CType(resources.GetObject("ToolStripMenuItemAbout.Image"), Image)
|
||||
ToolStripMenuItemAbout.Name = "ToolStripMenuItemAbout"
|
||||
ToolStripMenuItemAbout.Size = New Size(153, 22)
|
||||
ToolStripMenuItemAbout.Text = "About"
|
||||
ToCSVToolStripMenuItem.AutoToolTip = True
|
||||
ToCSVToolStripMenuItem.CheckOnClick = True
|
||||
ToCSVToolStripMenuItem.Image = CType(resources.GetObject("ToCSVToolStripMenuItem.Image"), Image)
|
||||
ToCSVToolStripMenuItem.Name = "ToCSVToolStripMenuItem"
|
||||
ToCSVToolStripMenuItem.Size = New Size(180, 22)
|
||||
ToCSVToolStripMenuItem.Text = "to CSV"
|
||||
'
|
||||
' ToolStripMenuItemDeveloper
|
||||
' AboutToolStripMenuItem
|
||||
'
|
||||
ToolStripMenuItemDeveloper.AutoToolTip = True
|
||||
ToolStripMenuItemDeveloper.Image = CType(resources.GetObject("ToolStripMenuItemDeveloper.Image"), Image)
|
||||
ToolStripMenuItemDeveloper.Name = "ToolStripMenuItemDeveloper"
|
||||
ToolStripMenuItemDeveloper.Size = New Size(153, 22)
|
||||
ToolStripMenuItemDeveloper.Text = "Developer"
|
||||
AboutToolStripMenuItem.AutoToolTip = True
|
||||
AboutToolStripMenuItem.Image = CType(resources.GetObject("AboutToolStripMenuItem.Image"), Image)
|
||||
AboutToolStripMenuItem.Name = "AboutToolStripMenuItem"
|
||||
AboutToolStripMenuItem.Size = New Size(171, 22)
|
||||
AboutToolStripMenuItem.Text = "About"
|
||||
'
|
||||
' ToolStripMenuItemCheckUpdates
|
||||
' DeveloperToolStripMenuItem
|
||||
'
|
||||
ToolStripMenuItemCheckUpdates.AutoToolTip = True
|
||||
ToolStripMenuItemCheckUpdates.Image = CType(resources.GetObject("ToolStripMenuItemCheckUpdates.Image"), Image)
|
||||
ToolStripMenuItemCheckUpdates.Name = "ToolStripMenuItemCheckUpdates"
|
||||
ToolStripMenuItemCheckUpdates.Size = New Size(153, 22)
|
||||
ToolStripMenuItemCheckUpdates.Text = "Check Updates"
|
||||
DeveloperToolStripMenuItem.AutoToolTip = True
|
||||
DeveloperToolStripMenuItem.Image = CType(resources.GetObject("DeveloperToolStripMenuItem.Image"), Image)
|
||||
DeveloperToolStripMenuItem.Name = "DeveloperToolStripMenuItem"
|
||||
DeveloperToolStripMenuItem.Size = New Size(171, 22)
|
||||
DeveloperToolStripMenuItem.Text = "Developer"
|
||||
'
|
||||
' ToolStripMenuItemQuit
|
||||
' CheckForUpdatesToolStripMenuItem
|
||||
'
|
||||
ToolStripMenuItemQuit.AutoToolTip = True
|
||||
ToolStripMenuItemQuit.Image = CType(resources.GetObject("ToolStripMenuItemQuit.Image"), Image)
|
||||
ToolStripMenuItemQuit.Name = "ToolStripMenuItemQuit"
|
||||
ToolStripMenuItemQuit.Size = New Size(153, 22)
|
||||
ToolStripMenuItemQuit.Text = "Quit"
|
||||
CheckForUpdatesToolStripMenuItem.AutoToolTip = True
|
||||
CheckForUpdatesToolStripMenuItem.Image = CType(resources.GetObject("CheckForUpdatesToolStripMenuItem.Image"), Image)
|
||||
CheckForUpdatesToolStripMenuItem.Name = "CheckForUpdatesToolStripMenuItem"
|
||||
CheckForUpdatesToolStripMenuItem.Size = New Size(171, 22)
|
||||
CheckForUpdatesToolStripMenuItem.Text = "Check for Updates"
|
||||
'
|
||||
' QuitToolStripMenuItem
|
||||
'
|
||||
QuitToolStripMenuItem.AutoToolTip = True
|
||||
QuitToolStripMenuItem.Font = New Font("Segoe UI Semibold", 9F, FontStyle.Bold, GraphicsUnit.Point)
|
||||
QuitToolStripMenuItem.Image = CType(resources.GetObject("QuitToolStripMenuItem.Image"), Image)
|
||||
QuitToolStripMenuItem.Name = "QuitToolStripMenuItem"
|
||||
QuitToolStripMenuItem.Size = New Size(171, 22)
|
||||
QuitToolStripMenuItem.Text = "Quit"
|
||||
'
|
||||
' NumericUpDownLimit
|
||||
'
|
||||
@@ -245,7 +253,7 @@ Partial Class FormMain
|
||||
NumericUpDownLimit.ReadOnly = True
|
||||
NumericUpDownLimit.Size = New Size(100, 23)
|
||||
NumericUpDownLimit.TabIndex = 15
|
||||
ToolTip.SetToolTip(NumericUpDownLimit, "Set how many posts you want to go through. Default value is `10`.")
|
||||
ToolTip.SetToolTip(NumericUpDownLimit, "Number of posts to go through. Default value is `10`.")
|
||||
NumericUpDownLimit.Value = New Decimal(New Integer() {10, 0, 0, 0})
|
||||
'
|
||||
' ToolTip
|
||||
@@ -298,14 +306,16 @@ Partial Class FormMain
|
||||
Friend WithEvents LabelListing As Label
|
||||
Friend WithEvents LabelTimeframe As Label
|
||||
Friend WithEvents ContextMenuStripRightClick As ContextMenuStrip
|
||||
Friend WithEvents ToolStripMenuItemSavePosts As ToolStripMenuItem
|
||||
Friend WithEvents ToolStripMenuItemtoJSON As ToolStripMenuItem
|
||||
Friend WithEvents ToolStripMenuItemtoCSV As ToolStripMenuItem
|
||||
Friend WithEvents SavePostsToolStripMenuItem As ToolStripMenuItem
|
||||
Friend WithEvents ToJSONToolStripMenuItem As ToolStripMenuItem
|
||||
Friend WithEvents ToCSVToolStripMenuItem As ToolStripMenuItem
|
||||
Friend WithEvents NumericUpDownLimit As NumericUpDown
|
||||
Friend WithEvents ToolStripMenuItemDarkMode As ToolStripMenuItem
|
||||
Friend WithEvents ToolStripMenuItemAbout As ToolStripMenuItem
|
||||
Friend WithEvents ToolStripMenuItemDeveloper As ToolStripMenuItem
|
||||
Friend WithEvents ToolStripMenuItemCheckUpdates As ToolStripMenuItem
|
||||
Friend WithEvents ToolStripMenuItemQuit As ToolStripMenuItem
|
||||
Friend WithEvents AboutToolStripMenuItem As ToolStripMenuItem
|
||||
Friend WithEvents DeveloperToolStripMenuItem As ToolStripMenuItem
|
||||
Friend WithEvents CheckForUpdatesToolStripMenuItem As ToolStripMenuItem
|
||||
Friend WithEvents QuitToolStripMenuItem As ToolStripMenuItem
|
||||
Friend WithEvents ToolTip As ToolTip
|
||||
Friend WithEvents SettingsToolStripMenuItem As ToolStripMenuItem
|
||||
Friend WithEvents DarkModeToolStripMenuItem As ToolStripMenuItem
|
||||
Friend WithEvents SaveFoundPostsToolStripMenuItem As ToolStripMenuItem
|
||||
End Class
|
||||
|
||||
@@ -1,7 +1,4 @@
|
||||
Imports System.IO
|
||||
Imports System.Windows.Forms.VisualStyles.VisualStyleElement
|
||||
Imports Newtonsoft.Json
|
||||
Imports Newtonsoft.Json.Linq
|
||||
Imports Newtonsoft.Json.Linq
|
||||
|
||||
Public Class FormMain
|
||||
ReadOnly settings As New SettingsManager()
|
||||
@@ -14,32 +11,25 @@ Public Class FormMain
|
||||
''' <param name="sender">The source of the event.</param>
|
||||
''' <param name="e">An EventArgs that contains the event data.</param>
|
||||
Private Sub FormMain_Load(sender As Object, e As EventArgs) Handles MyBase.Load
|
||||
|
||||
settings.LoadSettings()
|
||||
settings.ToggleDarkMode(settings.DarkMode)
|
||||
settings.ToggleSettings(settings.DarkMode, "darkmode")
|
||||
settings.ToggleSettings(settings.SaveToJson, "json")
|
||||
settings.ToggleSettings(settings.SaveToCsv, "csv")
|
||||
|
||||
Utilities.PathFinder()
|
||||
Utilities.LogFirstTimeLaunch()
|
||||
Me.Text = My.Application.Info.AssemblyName
|
||||
End Sub
|
||||
|
||||
|
||||
''' <summary>
|
||||
''' Event handler for the 'Dark Mode' checkbox change event.
|
||||
''' It toggles the dark mode of the application based on the checkbox status.
|
||||
''' </summary>
|
||||
''' <param name="sender">The source of the event.</param>
|
||||
''' <param name="e">An EventArgs that contains the event data.</param>
|
||||
Private Sub DarkModeToolStripMenuItem_CheckedChanged(sender As Object, e As EventArgs) Handles ToolStripMenuItemDarkMode.CheckedChanged
|
||||
settings.ToggleDarkMode(ToolStripMenuItemDarkMode.Checked)
|
||||
End Sub
|
||||
|
||||
|
||||
''' <summary>
|
||||
''' Event handler for the 'About' menu item click.
|
||||
''' It shows the 'About' box.
|
||||
''' </summary>
|
||||
''' <param name="sender">The source of the event.</param>
|
||||
''' <param name="e">An EventArgs that contains the event data.</param>
|
||||
Private Sub ToolStripMenuItemAbout_Click(sender As Object, e As EventArgs) Handles ToolStripMenuItemAbout.Click
|
||||
Private Sub ToolStripMenuItemAbout_Click(sender As Object, e As EventArgs) Handles AboutToolStripMenuItem.Click
|
||||
AboutBox.Show()
|
||||
End Sub
|
||||
|
||||
@@ -50,7 +40,7 @@ Public Class FormMain
|
||||
''' </summary>
|
||||
''' <param name="sender">The source of the event.</param>
|
||||
''' <param name="e">An EventArgs that contains the event data.</param>
|
||||
Private Sub ToolStripMenuItemDeveloper_Click(sender As Object, e As EventArgs) Handles ToolStripMenuItemDeveloper.Click
|
||||
Private Sub ToolStripMenuItemDeveloper_Click(sender As Object, e As EventArgs) Handles DeveloperToolStripMenuItem.Click
|
||||
DeveloperBox.ShowDialog()
|
||||
End Sub
|
||||
|
||||
@@ -61,7 +51,7 @@ Public Class FormMain
|
||||
''' </summary>
|
||||
''' <param name="sender">The source of the event.</param>
|
||||
''' <param name="e">An EventArgs that contains the event data.</param>
|
||||
Private Sub ToolStripMenuItemCheckUpdates_Click(sender As Object, e As EventArgs) Handles ToolStripMenuItemCheckUpdates.Click
|
||||
Private Sub ToolStripMenuItemCheckUpdates_Click(sender As Object, e As EventArgs) Handles CheckForUpdatesToolStripMenuItem.Click
|
||||
Dim data As JObject = ApiHandler.CheckUpdates()
|
||||
If data("tag_name").ToString = My.Application.Info.Version.ToString Then
|
||||
MessageBox.Show($"You're running the latest version v{My.Application.Info.Version} of {Me.Text}. Check again soon! :)", $"{Me.Text} v{My.Application.Info.Version}", MessageBoxButtons.OK, MessageBoxIcon.Information)
|
||||
@@ -83,7 +73,7 @@ Public Class FormMain
|
||||
''' </summary>
|
||||
''' <param name="sender">The source of the event.</param>
|
||||
''' <param name="e">An EventArgs that contains the event data.</param>
|
||||
Private Sub ToolStripMenuItemQuit_Click(sender As Object, e As EventArgs) Handles ToolStripMenuItemQuit.Click
|
||||
Private Sub ToolStripMenuItemQuit_Click(sender As Object, e As EventArgs) Handles QuitToolStripMenuItem.Click
|
||||
Dim result As DialogResult = MessageBox.Show("This will close the program, continue?", "Quit", MessageBoxButtons.YesNo, MessageBoxIcon.Question)
|
||||
If result = DialogResult.Yes Then
|
||||
Me.Close()
|
||||
@@ -99,7 +89,8 @@ Public Class FormMain
|
||||
''' <param name="sender">The sender of the event.</param>
|
||||
''' <param name="e">The EventArgs instance containing the event data.</param>
|
||||
Private Sub ButtonScrape_Click(sender As Object, e As EventArgs) Handles ButtonScrape.Click
|
||||
Utilities.ProcessRedditPosts(ToolStripMenuItemtoJSON)
|
||||
settings.LoadSettings()
|
||||
PostsProcessor.ProcessRedditPosts(settings:=settings)
|
||||
End Sub
|
||||
|
||||
|
||||
@@ -110,11 +101,13 @@ Public Class FormMain
|
||||
''' <param name="sender">The source of the event.</param>
|
||||
''' <param name="e">The <see cref="KeyEventArgs"/> instance containing the event data.</param>
|
||||
Private Sub TextBoxKeyword_KeyDown(sender As Object, e As KeyEventArgs) Handles TextBoxKeyword.KeyDown
|
||||
settings.LoadSettings()
|
||||
|
||||
' Check if the Enter key is pressed
|
||||
If e.KeyCode = Keys.Enter Then
|
||||
' Prevent the beep sound that usually comes with the Enter key in a single-line TextBox
|
||||
e.SuppressKeyPress = True
|
||||
Utilities.ProcessRedditPosts(ToolStripMenuItemtoJSON)
|
||||
PostsProcessor.ProcessRedditPosts(settings:=settings)
|
||||
End If
|
||||
End Sub
|
||||
|
||||
@@ -126,11 +119,13 @@ Public Class FormMain
|
||||
''' <param name="sender">The source of the event.</param>
|
||||
''' <param name="e">The <see cref="KeyEventArgs"/> instance containing the event data.</param>
|
||||
Private Sub TextBoxSubreddit_KeyDown(sender As Object, e As KeyEventArgs) Handles TextBoxSubreddit.KeyDown
|
||||
settings.LoadSettings()
|
||||
|
||||
' Check if the Enter key is pressed
|
||||
If e.KeyCode = Keys.Enter Then
|
||||
' Prevent the beep sound that usually comes with the Enter key in a single-line TextBox
|
||||
e.SuppressKeyPress = True
|
||||
Utilities.ProcessRedditPosts(ToolStripMenuItemtoJSON)
|
||||
PostsProcessor.ProcessRedditPosts(settings:=settings)
|
||||
End If
|
||||
End Sub
|
||||
|
||||
@@ -142,11 +137,13 @@ Public Class FormMain
|
||||
''' <param name="sender">The source of the event.</param>
|
||||
''' <param name="e">The <see cref="KeyEventArgs"/> instance containing the event data.</param>
|
||||
Private Sub NumericUpDownLimit_KeyDown(sender As Object, e As KeyEventArgs) Handles NumericUpDownLimit.KeyDown
|
||||
settings.LoadSettings()
|
||||
|
||||
' Check if the Enter key is pressed
|
||||
If e.KeyCode = Keys.Enter Then
|
||||
' Prevent the beep sound that usually comes with the Enter key in a single-line TextBox
|
||||
e.SuppressKeyPress = True
|
||||
Utilities.ProcessRedditPosts(ToolStripMenuItemtoJSON)
|
||||
PostsProcessor.ProcessRedditPosts(settings:=settings)
|
||||
End If
|
||||
End Sub
|
||||
|
||||
@@ -158,11 +155,13 @@ Public Class FormMain
|
||||
''' <param name="sender">The source of the event.</param>
|
||||
''' <param name="e">The <see cref="KeyEventArgs"/> instance containing the event data.</param>
|
||||
Private Sub ComboBoxListing_KeyDown(sender As Object, e As KeyEventArgs) Handles ComboBoxListing.KeyDown
|
||||
settings.LoadSettings()
|
||||
|
||||
' Check if the Enter key is pressed
|
||||
If e.KeyCode = Keys.Enter Then
|
||||
' Prevent the beep sound that usually comes with the Enter key in a single-line TextBox
|
||||
e.SuppressKeyPress = True
|
||||
Utilities.ProcessRedditPosts(ToolStripMenuItemtoJSON)
|
||||
PostsProcessor.ProcessRedditPosts(settings:=settings)
|
||||
End If
|
||||
End Sub
|
||||
|
||||
@@ -174,11 +173,43 @@ Public Class FormMain
|
||||
''' <param name="sender">The source of the event.</param>
|
||||
''' <param name="e">The <see cref="KeyEventArgs"/> instance containing the event data.</param>
|
||||
Private Sub ComboBoxTimeframe_KeyDown(sender As Object, e As KeyEventArgs) Handles ComboBoxTimeframe.KeyDown
|
||||
settings.LoadSettings()
|
||||
|
||||
' Check if the Enter key is pressed
|
||||
If e.KeyCode = Keys.Enter Then
|
||||
' Prevent the beep sound that usually comes with the Enter key in a single-line TextBox
|
||||
e.SuppressKeyPress = True
|
||||
Utilities.ProcessRedditPosts(ToolStripMenuItemtoJSON)
|
||||
PostsProcessor.ProcessRedditPosts(settings:=settings)
|
||||
End If
|
||||
End Sub
|
||||
|
||||
''' <summary>
|
||||
''' Event handler for the 'Dark Mode' checkbox change event.
|
||||
''' It toggles the dark mode of the application based on the checkbox status.
|
||||
''' </summary>
|
||||
''' <param name="sender">The source of the event.</param>
|
||||
''' <param name="e">An EventArgs that contains the event data.</param>
|
||||
Private Sub ToolStripMenuItemDarkMode_CheckedChanged(sender As Object, e As EventArgs) Handles DarkModeToolStripMenuItem.CheckedChanged
|
||||
settings.ToggleSettings(DarkModeToolStripMenuItem.Checked, "darkmode")
|
||||
End Sub
|
||||
|
||||
''' <summary>
|
||||
''' Event handler for the 'to CSV' checkbox change event.
|
||||
''' It toggles the dark mode of the application based on the checkbox status.
|
||||
''' </summary>
|
||||
''' <param name="sender">The source of the event.</param>
|
||||
''' <param name="e">An EventArgs that contains the event data.</param>
|
||||
Private Sub ToCSVToolStripMenuItem_CheckedChanged(sender As Object, e As EventArgs) Handles ToCSVToolStripMenuItem.CheckedChanged
|
||||
settings.ToggleSettings(ToCSVToolStripMenuItem.Checked, "csv")
|
||||
End Sub
|
||||
|
||||
''' <summary>
|
||||
''' Event handler for the 'to JSON' checkbox change event.
|
||||
''' It toggles the dark mode of the application based on the checkbox status.
|
||||
''' </summary>
|
||||
''' <param name="sender">The source of the event.</param>
|
||||
''' <param name="e">An EventArgs that contains the event data.</param>
|
||||
Private Sub ToJSONToolStripMenuItem_CheckedChanged(sender As Object, e As EventArgs) Handles ToJSONToolStripMenuItem.CheckedChanged
|
||||
settings.ToggleSettings(ToJSONToolStripMenuItem.Checked, "json")
|
||||
End Sub
|
||||
End Class
|
||||
|
||||
@@ -26,4 +26,63 @@ Public Class PostsProcessor
|
||||
Return post("data")("selftext").ToString.ToLower(Globalization.CultureInfo.InvariantCulture).Contains(keyword.ToLower(System.Globalization.CultureInfo.InvariantCulture))
|
||||
End Function
|
||||
|
||||
|
||||
''' <summary>
|
||||
''' Collects user inputs, fetches Reddit posts based on the inputs, checks if posts contain the keyword, and saves posts to a JSON file if necessary.
|
||||
''' </summary>
|
||||
''' <param name="JSONToolStripMenuItem">Indicates whether to save the posts to a JSON file.</param>
|
||||
''' <remarks>
|
||||
''' This function initializes the DataGridView, iterates over each post, adds the posts containing the keyword to the DataGridView and updates the UI.
|
||||
''' It also shows a message if the keyword was not found in any of the posts or if the inputs are empty.
|
||||
''' </remarks>
|
||||
Public Shared Sub ProcessRedditPosts(settings)
|
||||
' Collect inputs from the user.
|
||||
Dim inputs = Utilities.CollectInputs()
|
||||
|
||||
If inputs.HasValue Then
|
||||
' Initialize the DataGridView.
|
||||
DataGridViewHandler.AddColumn(FormPosts.DataGridViewPosts)
|
||||
|
||||
' Fetch Reddit posts based on the inputs.
|
||||
Dim processor As New PostsProcessor()
|
||||
Dim posts As JObject = processor.FetchPosts(inputs.Value.Subreddit, inputs.Value.Listing, inputs.Value.Limit, inputs.Value.Timeframe)
|
||||
Dim totalPosts As Integer = 0
|
||||
Dim keywordFound As Boolean = False
|
||||
Dim foundPosts As Integer = 0
|
||||
Dim foundPostsList As New JArray
|
||||
|
||||
' Iterate over each post.
|
||||
For Each post In posts("data")("children")
|
||||
totalPosts += 1
|
||||
' Check if the post contains the keyword
|
||||
If PostsProcessor.PostContainsKeyword(post, inputs.Value.Keyword.ToLower(Globalization.CultureInfo.InvariantCulture)) Then
|
||||
foundPosts += 1
|
||||
foundPostsList.Add(post)
|
||||
' Add the post to the DataGridView.
|
||||
DataGridViewHandler.AddRow(FormPosts.DataGridViewPosts, post, totalPosts)
|
||||
FormPosts.Show()
|
||||
keywordFound = True
|
||||
End If
|
||||
Next
|
||||
|
||||
' Check if the keyword was found in any posts
|
||||
If Not keywordFound Then
|
||||
MessageBox.Show($"Keyword `{inputs.Value.Keyword}` was not found in any of the " + posts("data")("children").Count.ToString(Globalization.CultureInfo.InvariantCulture) _
|
||||
+ $" {inputs.Value.Listing} posts from r/{inputs.Value.Subreddit}", "Not Found", MessageBoxButtons.OK, MessageBoxIcon.Warning)
|
||||
End If
|
||||
|
||||
|
||||
If settings.SaveToJson Then
|
||||
' Save posts to a JSON file if SaveToJson is True.
|
||||
Utilities.SavePostsToJson(foundPostsList)
|
||||
End If
|
||||
|
||||
If settings.SaveToCsv Then
|
||||
' Save posts to a CSV file if SaveToCsv is True.
|
||||
Utilities.SavePostsToCSV(foundPostsList)
|
||||
End If
|
||||
Else
|
||||
End If
|
||||
End Sub
|
||||
|
||||
End Class
|
||||
|
||||
@@ -1,33 +1,42 @@
|
||||
# RPST (Reddit Post Scraping Tool)
|
||||
Given a subreddit name and a keyword, RPST will return all posts from a specified listing (default is 'top') that contain the provided keyword.
|
||||
|
||||
[](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/python-publish.yml) [](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/codeql.yml)  
|
||||

|
||||

|
||||
***
|
||||
[](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/python-publish.yml) [](https://github.com/rly0nheart/reddit-post-scraping-tool/actions/workflows/codeql.yml)  
|
||||
|
||||
# ✅ Features
|
||||
## GUI
|
||||
- [x] Dark mode (Right-click)
|
||||
- [x] Saves results to a JSON (Right-click)
|
||||
- [x] Logs errors to a file
|
||||
## *GUI*
|
||||
- [x] Dark mode (*Right-click*).
|
||||
- [x] Saves results to a JSON file (*Right-click*).
|
||||
- [x] Logs errors to a file.
|
||||
- [x] In-App feature to check for Updates.
|
||||
|
||||
## CLI
|
||||
- [x] Saves results to a JSON (-j/--json)
|
||||
- [x] Automatically checks for new updates. Notifies user if update were found.
|
||||
## *CLI*
|
||||
- [x] Saves results to JSON (*specifiy* `--json`).
|
||||
- [x] Saves results to CSV (*specify* `--csv`).
|
||||
- [x] Automatically checks for new updates, and notifies user if updates were found.
|
||||
|
||||
# 📃 TODO
|
||||
## GUI
|
||||
## *GUI*
|
||||
- [ ] Make it installable with a setup.exe/setup.msi file.
|
||||
- [x] Add manual dark mode option, that will be persistent in all sessions
|
||||
- [ ] Make it save results to a CSV file
|
||||
- [x] Add manual dark mode option, that will be persistent in all sessions.
|
||||
- [x] Make settings persistent in all sessions.
|
||||
- [x] Make it save results to a CSV file.
|
||||
|
||||
# 🖥️ Tested environments
|
||||
## *GUI*
|
||||
- [x] Microsoft Windows 11
|
||||
|
||||
## *CLI*
|
||||
- [x] Android Termux
|
||||
- [x] Microsoft Windows 11
|
||||
- [x] Ubuntu 22.04 - latest versions
|
||||
|
||||
# 📖 Wiki
|
||||
[Refer to the Wiki](https://github.com/rly0nheart/reddit-post-scraping-tool/wiki) for installation instructions, in addition to all other documentation.
|
||||
[Refer to the Wiki](https://github.com/bellingcat/reddit-post-scraping-tool/wiki) for installation instructions, in addition to all other documentation.
|
||||
|
||||
# 😁 Donations
|
||||
If you like `RPST` and would like to show support, you can Buy A Coffee for the developer using the button below
|
||||
# 🖼️ Screenshots
|
||||
You can view a collection of screenshots for both the *CLI* and *GUI* [here](https://github.com/bellingcat/reddit-post-scraping-tool/tree/master/images)
|
||||
***
|
||||
<a href="https://www.buymeacoffee.com/_rly0nheart"><img src="https://img.buymeacoffee.com/button-api/?text=Buy me a coffee&emoji=&slug=_rly0nheart&button_colour=40DCA5&font_colour=ffffff&font_family=Comic&outline_colour=000000&coffee_colour=FFDD00" /></a>
|
||||
|
||||
<a href="https://www.buymeacoffee.com/_rly0nheart" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/default-orange.png" alt="Buy Me A Coffee" height="41" width="174"></a>
|
||||
|
||||
Your support will be much appreciated😊
|
||||

|
||||
|
||||
@@ -13,11 +13,11 @@
|
||||
<PackageProjectUrl>https://github.com/bellingcat/reddit-post-scraping-tool</PackageProjectUrl>
|
||||
<PackageReadmeFile>README.md</PackageReadmeFile>
|
||||
<RepositoryUrl>https://github.com/bellingcat/reddit-post-scraping-tool</RepositoryUrl>
|
||||
<AssemblyVersion>1.6.0.0</AssemblyVersion>
|
||||
<FileVersion>1.6.0.0</FileVersion>
|
||||
<AssemblyVersion>1.8.0.0</AssemblyVersion>
|
||||
<FileVersion>1.8.0.0</FileVersion>
|
||||
<PackageLicenseFile>LICENSE</PackageLicenseFile>
|
||||
<PackageRequireLicenseAcceptance>True</PackageRequireLicenseAcceptance>
|
||||
<Version>1.6.0</Version>
|
||||
<Version>1.8.0</Version>
|
||||
<PackageTags>reddit;scraper;reddit-scraper;osint</PackageTags>
|
||||
<PackageReleaseNotes></PackageReleaseNotes>
|
||||
<AnalysisLevel>6.0-recommended</AnalysisLevel>
|
||||
@@ -39,7 +39,7 @@
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Newtonsoft.Json" Version="13.0.2" />
|
||||
<PackageReference Include="Newtonsoft.Json" Version="13.0.3" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
@@ -78,4 +78,4 @@
|
||||
</None>
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
</Project>
|
||||
|
||||
@@ -10,8 +10,10 @@ Public Class SettingsManager
|
||||
''' Indicates whether the dark mode is enabled or disabled.
|
||||
''' </summary>
|
||||
Public Property DarkMode As Boolean
|
||||
Public Property SaveToJson As Boolean
|
||||
Public Property SaveToCsv As Boolean
|
||||
|
||||
Private ReadOnly settingsFilePath As String = Path.Combine(Environment.CurrentDirectory, "settings.json")
|
||||
Private ReadOnly settingsFilePath As String = Path.Combine(Environment.CurrentDirectory, "config.json")
|
||||
|
||||
''' <summary>
|
||||
''' Loads application settings from the 'settings.json' file.
|
||||
@@ -23,34 +25,55 @@ Public Class SettingsManager
|
||||
If File.Exists(settingsFilePath) Then
|
||||
Dim json As String = File.ReadAllText(settingsFilePath)
|
||||
Dim options As New JsonSerializerOptions With {.PropertyNameCaseInsensitive = True}
|
||||
Dim settings = Text.Json.JsonSerializer.Deserialize(Of SettingsManager)(json, options)
|
||||
Me.DarkMode = settings.DarkMode
|
||||
FormMain.ToolStripMenuItemDarkMode.Checked = settings.DarkMode
|
||||
Dim settings = JsonSerializer.Deserialize(Of SettingsManager)(json, options)
|
||||
|
||||
DarkMode = settings.DarkMode
|
||||
SaveToJson = settings.SaveToJson
|
||||
SaveToCsv = settings.SaveToCsv
|
||||
|
||||
FormMain.DarkModeToolStripMenuItem.Checked = settings.DarkMode
|
||||
FormMain.ToJSONToolStripMenuItem.Checked = settings.SaveToJson
|
||||
FormMain.ToCSVToolStripMenuItem.Checked = settings.SaveToCsv
|
||||
Else
|
||||
' Settings file does not exist
|
||||
' Create a new file with default settings 'False'
|
||||
Dim defaultSettings = New SettingsManager With {.DarkMode = False}
|
||||
Dim jsonOutput = Text.Json.JsonSerializer.Serialize(defaultSettings)
|
||||
Dim defaultSettings = New SettingsManager With {.DarkMode = False, .SaveToCsv = False, .SaveToJson = False}
|
||||
Dim jsonOutput = JsonSerializer.Serialize(defaultSettings)
|
||||
File.WriteAllText(settingsFilePath, jsonOutput)
|
||||
|
||||
Me.DarkMode = False
|
||||
FormMain.ToolStripMenuItemDarkMode.Checked = False
|
||||
DarkMode = False
|
||||
SaveToJson = False
|
||||
SaveToCsv = False
|
||||
|
||||
FormMain.ToJSONToolStripMenuItem.Checked = False
|
||||
FormMain.ToCSVToolStripMenuItem.Checked = False
|
||||
FormMain.DarkModeToolStripMenuItem.Checked = False
|
||||
|
||||
|
||||
End If
|
||||
End Sub
|
||||
|
||||
|
||||
''' <summary>
|
||||
''' Toggles the Dark Mode setting on or off based on the provided parameter.
|
||||
''' Retrieves application settings from a JSON file.
|
||||
''' </summary>
|
||||
''' <param name="enabled">A Boolean indicating if Dark Mode should be enabled or not.</param>
|
||||
Public Sub ToggleDarkMode(enabled As Boolean)
|
||||
Dim json As String = File.ReadAllText(settingsFilePath)
|
||||
Dim options As New JsonSerializerOptions With {.PropertyNameCaseInsensitive = True}
|
||||
Dim settings As SettingsManager = JsonSerializer.Deserialize(Of SettingsManager)(json, options)
|
||||
settings.DarkMode = enabled
|
||||
SaveSettings(settings)
|
||||
ApplyTheme()
|
||||
End Sub
|
||||
''' <returns>A Dictionary containing the names and values of all settings.
|
||||
''' If the settings file doesn't exist, returns a Dictionary with default values.</returns>
|
||||
Private Function GetSettings() As Dictionary(Of String, Object)
|
||||
Dim settings As New Dictionary(Of String, Object)
|
||||
If File.Exists(settingsFilePath) Then
|
||||
' Read and parse the JSON settings file.
|
||||
Dim json As String = File.ReadAllText(settingsFilePath)
|
||||
Dim jObject As JObject = JObject.Parse(json)
|
||||
|
||||
' Loop through each property in the JObject and add it to the settings Dictionary.
|
||||
For Each item As JProperty In jObject.Properties()
|
||||
settings.Add(item.Name, item.Value.ToObject(Of Object)())
|
||||
Next
|
||||
Else
|
||||
End If
|
||||
Return settings
|
||||
End Function
|
||||
|
||||
''' <summary>
|
||||
''' Saves the provided settings to the 'settings.json' file.
|
||||
@@ -63,12 +86,20 @@ Public Class SettingsManager
|
||||
|
||||
|
||||
''' <summary>
|
||||
''' Applies the visual theme based on the Dark Mode setting.
|
||||
''' If Dark Mode is enabled, a dark theme is applied. If it's disabled, a light theme is set.
|
||||
''' Applies the current settings to the application's interface. This includes
|
||||
''' toggling SaveToJson, SaveToCsv, and applying the visual theme based on the Dark Mode setting.
|
||||
''' </summary>
|
||||
Public Sub ApplyTheme()
|
||||
Dim DarkMode As Boolean = GetDarkMode()
|
||||
If DarkMode Then
|
||||
Public Sub ApplySettings()
|
||||
' Retrieve the current settings
|
||||
Dim settings As Dictionary(Of String, Object) = GetSettings()
|
||||
|
||||
' Apply the SaveToJson setting to the menu item checkbox
|
||||
FormMain.ToJSONToolStripMenuItem.Checked = CBool(settings("SaveToJson"))
|
||||
|
||||
' Apply the SaveToCsv setting to the menu item checkbox
|
||||
FormMain.ToCSVToolStripMenuItem.Checked = CBool(settings("SaveToCsv"))
|
||||
|
||||
If CBool(settings("DarkMode")) Then
|
||||
' Enable dark mode for the Main form
|
||||
' Background colours (I know 'Colours'/'Colors'😆)
|
||||
FormMain.BackColor = ColorTranslator.FromHtml("#FF121212")
|
||||
@@ -94,23 +125,25 @@ Public Class SettingsManager
|
||||
|
||||
' Enable dark mode on 'Right Click Menu' items
|
||||
' Background colours
|
||||
FormMain.ToolStripMenuItemDarkMode.BackColor = ColorTranslator.FromHtml("#FF121212")
|
||||
FormMain.ToolStripMenuItemSavePosts.BackColor = ColorTranslator.FromHtml("#FF121212")
|
||||
FormMain.ToolStripMenuItemtoJSON.BackColor = ColorTranslator.FromHtml("#FF121212")
|
||||
FormMain.ToolStripMenuItemtoCSV.BackColor = ColorTranslator.FromHtml("#FF121212")
|
||||
FormMain.ToolStripMenuItemAbout.BackColor = ColorTranslator.FromHtml("#FF121212")
|
||||
FormMain.ToolStripMenuItemDeveloper.BackColor = ColorTranslator.FromHtml("#FF121212")
|
||||
FormMain.ToolStripMenuItemCheckUpdates.BackColor = ColorTranslator.FromHtml("#FF121212")
|
||||
FormMain.ToolStripMenuItemQuit.BackColor = ColorTranslator.FromHtml("#FF121212")
|
||||
FormMain.SettingsToolStripMenuItem.BackColor = ColorTranslator.FromHtml("#FF121212")
|
||||
FormMain.DarkModeToolStripMenuItem.BackColor = ColorTranslator.FromHtml("#FF121212")
|
||||
FormMain.SavePostsToolStripMenuItem.BackColor = ColorTranslator.FromHtml("#FF121212")
|
||||
FormMain.ToJSONToolStripMenuItem.BackColor = ColorTranslator.FromHtml("#FF121212")
|
||||
FormMain.ToCSVToolStripMenuItem.BackColor = ColorTranslator.FromHtml("#FF121212")
|
||||
FormMain.AboutToolStripMenuItem.BackColor = ColorTranslator.FromHtml("#FF121212")
|
||||
FormMain.DeveloperToolStripMenuItem.BackColor = ColorTranslator.FromHtml("#FF121212")
|
||||
FormMain.CheckForUpdatesToolStripMenuItem.BackColor = ColorTranslator.FromHtml("#FF121212")
|
||||
FormMain.QuitToolStripMenuItem.BackColor = ColorTranslator.FromHtml("#FF121212")
|
||||
' Foreground colours
|
||||
FormMain.ToolStripMenuItemDarkMode.ForeColor = SystemColors.Control
|
||||
FormMain.ToolStripMenuItemSavePosts.ForeColor = SystemColors.Control
|
||||
FormMain.ToolStripMenuItemtoJSON.ForeColor = SystemColors.Control
|
||||
FormMain.ToolStripMenuItemtoCSV.ForeColor = SystemColors.Control
|
||||
FormMain.ToolStripMenuItemAbout.ForeColor = SystemColors.Control
|
||||
FormMain.ToolStripMenuItemDeveloper.ForeColor = SystemColors.Control
|
||||
FormMain.ToolStripMenuItemCheckUpdates.ForeColor = SystemColors.Control
|
||||
FormMain.ToolStripMenuItemQuit.ForeColor = SystemColors.Control
|
||||
FormMain.SettingsToolStripMenuItem.ForeColor = SystemColors.Control
|
||||
FormMain.DarkModeToolStripMenuItem.ForeColor = SystemColors.Control
|
||||
FormMain.SavePostsToolStripMenuItem.ForeColor = SystemColors.Control
|
||||
FormMain.ToJSONToolStripMenuItem.ForeColor = SystemColors.Control
|
||||
FormMain.ToCSVToolStripMenuItem.ForeColor = SystemColors.Control
|
||||
FormMain.AboutToolStripMenuItem.ForeColor = SystemColors.Control
|
||||
FormMain.DeveloperToolStripMenuItem.ForeColor = SystemColors.Control
|
||||
FormMain.CheckForUpdatesToolStripMenuItem.ForeColor = SystemColors.Control
|
||||
FormMain.QuitToolStripMenuItem.ForeColor = SystemColors.Control
|
||||
|
||||
|
||||
' Enable dark mode for the About box
|
||||
@@ -123,10 +156,10 @@ Public Class SettingsManager
|
||||
AboutBox.LicenseRichTextBox.ForeColor = SystemColors.Control
|
||||
AboutBox.LabelProgramName.ForeColor = SystemColors.Control
|
||||
AboutBox.LabelProgramDescription.ForeColor = SystemColors.Control
|
||||
AboutBox.LabelVersion.ForeColor = SystemColors.Control
|
||||
AboutBox.LinkLabelVersion.ForeColor = SystemColors.Control
|
||||
|
||||
' If dark mode is enabled, set the 'Dark Mode' text value to 'Light mode'
|
||||
FormMain.ToolStripMenuItemDarkMode.Text = "Light Mode"
|
||||
FormMain.DarkModeToolStripMenuItem.Text = "Dark Mode: Enabled"
|
||||
Else
|
||||
' Disable dark mode for the Main Form
|
||||
' Background colours
|
||||
@@ -152,23 +185,25 @@ Public Class SettingsManager
|
||||
|
||||
' Disable dark mode on 'Right Click Menu' items
|
||||
' Background colours
|
||||
FormMain.ToolStripMenuItemDarkMode.BackColor = Color.Gainsboro
|
||||
FormMain.ToolStripMenuItemSavePosts.BackColor = Color.Gainsboro
|
||||
FormMain.ToolStripMenuItemtoJSON.BackColor = Color.Gainsboro
|
||||
FormMain.ToolStripMenuItemtoCSV.BackColor = Color.Gainsboro
|
||||
FormMain.ToolStripMenuItemAbout.BackColor = Color.Gainsboro
|
||||
FormMain.ToolStripMenuItemDeveloper.BackColor = Color.Gainsboro
|
||||
FormMain.ToolStripMenuItemCheckUpdates.BackColor = Color.Gainsboro
|
||||
FormMain.ToolStripMenuItemQuit.BackColor = Color.Gainsboro
|
||||
FormMain.SettingsToolStripMenuItem.BackColor = Color.Gainsboro
|
||||
FormMain.DarkModeToolStripMenuItem.BackColor = Color.Gainsboro
|
||||
FormMain.SavePostsToolStripMenuItem.BackColor = Color.Gainsboro
|
||||
FormMain.ToJSONToolStripMenuItem.BackColor = Color.Gainsboro
|
||||
FormMain.ToCSVToolStripMenuItem.BackColor = Color.Gainsboro
|
||||
FormMain.AboutToolStripMenuItem.BackColor = Color.Gainsboro
|
||||
FormMain.DeveloperToolStripMenuItem.BackColor = Color.Gainsboro
|
||||
FormMain.CheckForUpdatesToolStripMenuItem.BackColor = Color.Gainsboro
|
||||
FormMain.QuitToolStripMenuItem.BackColor = Color.Gainsboro
|
||||
' Foreground colours
|
||||
FormMain.ToolStripMenuItemDarkMode.ForeColor = Color.Black
|
||||
FormMain.ToolStripMenuItemSavePosts.ForeColor = Color.Black
|
||||
FormMain.ToolStripMenuItemtoJSON.ForeColor = Color.Black
|
||||
FormMain.ToolStripMenuItemtoCSV.ForeColor = Color.Black
|
||||
FormMain.ToolStripMenuItemAbout.ForeColor = Color.Black
|
||||
FormMain.ToolStripMenuItemDeveloper.ForeColor = Color.Black
|
||||
FormMain.ToolStripMenuItemCheckUpdates.ForeColor = Color.Black
|
||||
FormMain.ToolStripMenuItemQuit.ForeColor = Color.Black
|
||||
FormMain.SettingsToolStripMenuItem.ForeColor = Color.Black
|
||||
FormMain.DarkModeToolStripMenuItem.ForeColor = Color.Black
|
||||
FormMain.SavePostsToolStripMenuItem.ForeColor = Color.Black
|
||||
FormMain.ToJSONToolStripMenuItem.ForeColor = Color.Black
|
||||
FormMain.ToCSVToolStripMenuItem.ForeColor = Color.Black
|
||||
FormMain.AboutToolStripMenuItem.ForeColor = Color.Black
|
||||
FormMain.DeveloperToolStripMenuItem.ForeColor = Color.Black
|
||||
FormMain.CheckForUpdatesToolStripMenuItem.ForeColor = Color.Black
|
||||
FormMain.QuitToolStripMenuItem.ForeColor = Color.Black
|
||||
|
||||
' Disable dark mode for the About box
|
||||
' Background colours
|
||||
@@ -181,26 +216,39 @@ Public Class SettingsManager
|
||||
AboutBox.Panel1.ForeColor = SystemColors.WindowText
|
||||
AboutBox.LabelProgramName.ForeColor = SystemColors.WindowText
|
||||
AboutBox.LabelProgramDescription.ForeColor = SystemColors.WindowText
|
||||
AboutBox.LabelVersion.ForeColor = SystemColors.WindowText
|
||||
AboutBox.LinkLabelVersion.ForeColor = SystemColors.WindowText
|
||||
|
||||
' If dark mode is disabled, set the 'Light Mode' text value to 'Dark Mode'
|
||||
FormMain.ToolStripMenuItemDarkMode.Text = "Dark Mode"
|
||||
FormMain.DarkModeToolStripMenuItem.Text = "Dark Mode: Disabled"
|
||||
End If
|
||||
End Sub
|
||||
|
||||
|
||||
''' <summary>
|
||||
''' Retrieves the Dark Mode setting value from 'settings.json'.
|
||||
''' If the settings file doesn't exist, defaults to returning 'False' (Dark Mode off).
|
||||
''' Toggles specific settings on or off based on the provided parameters.
|
||||
''' </summary>
|
||||
''' <returns>A Boolean indicating if Dark Mode is enabled or not.</returns>
|
||||
Private Function GetDarkMode() As Boolean
|
||||
If File.Exists(settingsFilePath) Then
|
||||
Dim json As String = File.ReadAllText(settingsFilePath)
|
||||
Dim settings As JObject = JObject.Parse(json)
|
||||
Return settings(NameOf(DarkMode)).ToObject(Of Boolean)()
|
||||
''' <param name="enabled">A Boolean indicating if the setting option should be enabled or not.</param>
|
||||
''' <param name="saveTo">A String specifying the type of setting to toggle ('json', 'csv', or 'darkmode').</param>
|
||||
Public Sub ToggleSettings(enabled As Boolean, saveTo As String)
|
||||
' Read the existing settings from the settings file
|
||||
Dim json As String = File.ReadAllText(settingsFilePath)
|
||||
Dim options As New JsonSerializerOptions With {.PropertyNameCaseInsensitive = True}
|
||||
Dim settings As SettingsManager = JsonSerializer.Deserialize(Of SettingsManager)(json, options)
|
||||
|
||||
' Update the settings based on the specified saveTo parameter
|
||||
If saveTo.ToLower(Globalization.CultureInfo.InvariantCulture) = "json" Then
|
||||
settings.SaveToJson = enabled
|
||||
ElseIf saveTo.ToLower(Globalization.CultureInfo.InvariantCulture) = "csv" Then
|
||||
settings.SaveToCsv = enabled
|
||||
ElseIf saveTo.ToLower(Globalization.CultureInfo.InvariantCulture) = "darkmode" Then
|
||||
settings.DarkMode = enabled
|
||||
Else
|
||||
Return False
|
||||
' Handle unexpected value of saveTo (if needed)
|
||||
End If
|
||||
End Function
|
||||
|
||||
' Save the updated settings back to the settings file
|
||||
SaveSettings(settings)
|
||||
' Apply the updated settings to the application
|
||||
ApplySettings()
|
||||
End Sub
|
||||
End Class
|
||||
@@ -3,54 +3,6 @@ Imports Newtonsoft.Json
|
||||
Imports Newtonsoft.Json.Linq
|
||||
|
||||
Public Class Utilities
|
||||
''' <summary>
|
||||
''' Collects user inputs, fetches Reddit posts based on the inputs, checks if posts contain the keyword, and saves posts to a JSON file if necessary.
|
||||
''' </summary>
|
||||
''' <param name="JSONToolStripMenuItem">Indicates whether to save the posts to a JSON file.</param>
|
||||
''' <remarks>
|
||||
''' This function initializes the DataGridView, iterates over each post, adds the posts containing the keyword to the DataGridView and updates the UI.
|
||||
''' It also shows a message if the keyword was not found in any of the posts or if the inputs are empty.
|
||||
''' </remarks>
|
||||
Public Shared Sub ProcessRedditPosts(JSONToolStripMenuItem As ToolStripMenuItem)
|
||||
' Collect inputs from the user
|
||||
Dim inputs = CollectInputs()
|
||||
|
||||
If inputs.HasValue Then
|
||||
' Initialize the DataGridView
|
||||
DataGridViewHandler.AddColumn(FormPosts.DataGridViewPosts)
|
||||
|
||||
' Fetch Reddit posts based on the inputs
|
||||
Dim processor As New PostsProcessor()
|
||||
Dim posts As JObject = processor.FetchPosts(inputs.Value.Subreddit, inputs.Value.Listing, inputs.Value.Limit, inputs.Value.Timeframe)
|
||||
Dim totalPosts As Integer = 0
|
||||
Dim keywordFound As Boolean = False
|
||||
|
||||
' Iterate over each post
|
||||
For Each post In posts("data")("children")
|
||||
totalPosts += 1
|
||||
' Check if the post contains the keyword
|
||||
If PostsProcessor.PostContainsKeyword(post, inputs.Value.Keyword.ToLower(Globalization.CultureInfo.InvariantCulture)) Then
|
||||
' Add the post to the DataGridView
|
||||
DataGridViewHandler.AddRow(FormPosts.DataGridViewPosts, post, totalPosts)
|
||||
FormPosts.Show()
|
||||
keywordFound = True
|
||||
End If
|
||||
Next
|
||||
|
||||
' Check if the keyword was found in any posts
|
||||
If Not keywordFound Then
|
||||
MessageBox.Show($"Keyword `{inputs.Value.Keyword}` was not found in any of the " + posts("data")("children").Count.ToString(Globalization.CultureInfo.InvariantCulture) _
|
||||
+ $" {inputs.Value.Listing} posts from r/{inputs.Value.Subreddit}", "Not Found", MessageBoxButtons.OK, MessageBoxIcon.Warning)
|
||||
End If
|
||||
|
||||
If JSONToolStripMenuItem.Checked Then
|
||||
' Save posts to a JSON file if the JSONToolStripMenuItem is checked
|
||||
Utilities.SavePostsToJson(posts("data"))
|
||||
End If
|
||||
Else
|
||||
End If
|
||||
End Sub
|
||||
|
||||
|
||||
''' <summary>
|
||||
''' Checks for the existence of the 'logs' directory under the 'RPST' directory within the user's AppData\Roaming folder.
|
||||
@@ -87,14 +39,18 @@ Public Class Utilities
|
||||
Public Shared Function CollectInputs() As (Keyword As String, Subreddit As String, Listing As String, Limit As Integer, Timeframe As String)?
|
||||
Dim keyword As String = FormMain.TextBoxKeyword.Text.Trim()
|
||||
Dim subreddit As String = FormMain.TextBoxSubreddit.Text.Trim()
|
||||
' Convert the Listing and Subreddit to lowercase using InvariantCulture
|
||||
''' <summary>
|
||||
''' Convert the Listing and Subreddit to lowercase using InvariantCulture.
|
||||
''' <summary>
|
||||
Dim listing As String = If(String.IsNullOrEmpty(FormMain.ComboBoxListing.Text), "top", FormMain.ComboBoxListing.Text.ToLower(Globalization.CultureInfo.InvariantCulture).Trim())
|
||||
Dim timeframe As String = If(String.IsNullOrEmpty(FormMain.ComboBoxTimeframe.Text), "all", FormMain.ComboBoxTimeframe.Text.ToLower(Globalization.CultureInfo.InvariantCulture).Trim())
|
||||
Dim limit As Integer = FormMain.NumericUpDownLimit.Value
|
||||
|
||||
' Validate inputs
|
||||
''' <summary>
|
||||
''' Validate inputs.
|
||||
''' <summary>
|
||||
If String.IsNullOrEmpty(keyword) AndAlso String.IsNullOrEmpty(subreddit) Then
|
||||
MessageBox.Show("Keyword and Subreddit fields should not be empty.", "Invalid Inputs", MessageBoxButtons.OK, MessageBoxIcon.Warning)
|
||||
MessageBox.Show("Keyword and Subreddit should not be empty.", "Invalid Inputs", MessageBoxButtons.OK, MessageBoxIcon.Warning)
|
||||
Return Nothing
|
||||
ElseIf String.IsNullOrEmpty(keyword) Then
|
||||
MessageBox.Show("Keyword field should not be empty.", "Invalid Input", MessageBoxButtons.OK, MessageBoxIcon.Warning)
|
||||
@@ -137,6 +93,42 @@ Public Class Utilities
|
||||
End Sub
|
||||
|
||||
|
||||
''' <summary>
|
||||
''' Saves Reddit posts contained in a JArray to a CSV file.
|
||||
''' </summary>
|
||||
''' <param name="posts">A JArray containing the Reddit posts to be saved.</param>
|
||||
''' <remarks>
|
||||
''' This function displays a SaveFileDialog to allow the user to specify the file name and location.
|
||||
''' It then iterates through the JArray to write each post's details (totalPosts, title, subreddit, author, score) into the selected CSV file.
|
||||
''' </remarks>
|
||||
Public Shared Sub SavePostsToCSV(posts As JArray)
|
||||
Dim saveFileDialog As New SaveFileDialog With {
|
||||
.Filter = "CSV files (*.csv)|*.csv",
|
||||
.Title = "Save posts to CSV"
|
||||
}
|
||||
|
||||
If saveFileDialog.ShowDialog() = DialogResult.OK Then
|
||||
Dim fileName As String = saveFileDialog.FileName
|
||||
Using csvWriter As New StreamWriter(fileName)
|
||||
''' <summary>
|
||||
''' Write the header.
|
||||
''' <summary>
|
||||
csvWriter.WriteLine("Index,Author,ID,Subreddit,Visibility,Thumbnail,NSFW,Gilded,Upvotes,Upvote Ratio,Downvotes,Award,Top Award,Is cross-postable?,Score,Category,Text,Domain,Permalink,Created At,Approved At,Approved By")
|
||||
|
||||
Dim postCount As Integer = 0
|
||||
For Each post In posts
|
||||
postCount += 1
|
||||
csvWriter.WriteLine($"{postCount},{post("data")("author")},{post("data")("id")},{post("data")("subreddit_name_prefixed")},{post("data")("subreddit_type")},{post("data")("thumbnail")},{post("data")("over_18")},{post("data")("gilded")},{post("data")("ups")},{post("data")("upvote_ratio")},{post("data")("downs")},{post("data")("total_awards_received")},{post("data")("top_awarded_type")},{post("data")("is_crosspostable")},{post("data")("score")},{post("data")("category")},{post("data")("selftext")},{post("data")("domain")},{post("data")("permalink")},{post("data")("created")},{post("data")("approved_at_utc")},{post("data")("approved_by")}")
|
||||
Next
|
||||
End Using
|
||||
|
||||
MessageBox.Show($"Posts saved to {fileName}", "Success", MessageBoxButtons.OK, MessageBoxIcon.Information)
|
||||
End If
|
||||
End Sub
|
||||
|
||||
|
||||
|
||||
|
||||
''' <summary>
|
||||
''' Shows the license notice in a messagebox.
|
||||
''' </summary>
|
||||
|
||||
BIN
images/2023-08-08_07-04.png
Normal file
|
After Width: | Height: | Size: 68 KiB |
BIN
images/2023-08-08_07-04_1.png
Normal file
|
After Width: | Height: | Size: 75 KiB |
BIN
images/2023-08-08_07-12.png
Normal file
|
After Width: | Height: | Size: 32 KiB |
BIN
images/2023-08-08_07-12_1.png
Normal file
|
After Width: | Height: | Size: 27 KiB |
BIN
images/2023-08-25_15-30.png
Normal file
|
After Width: | Height: | Size: 576 KiB |
BIN
images/2023-08-25_15-31.png
Normal file
|
After Width: | Height: | Size: 519 KiB |
BIN
images/2023-08-25_15-35.png
Normal file
|
After Width: | Height: | Size: 823 KiB |
BIN
images/2023-08-25_15-39.png
Normal file
|
After Width: | Height: | Size: 508 KiB |
BIN
images/2023-08-30_03-11.png
Normal file
|
After Width: | Height: | Size: 15 KiB |
BIN
images/2023-08-30_03-12.png
Normal file
|
After Width: | Height: | Size: 21 KiB |
@@ -7,18 +7,18 @@ packages = ["rpst"]
|
||||
|
||||
[project]
|
||||
name = "reddit-post-scraping-tool"
|
||||
version = "1.6.0.0"
|
||||
version = "1.8.0.0"
|
||||
description = "Given a subreddit name and a keyword, RPST returns all top (by default) posts that contain the specified keyword."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.8"
|
||||
license = {file = "LICENSE"}
|
||||
keywords = ["osint", "reddit-crawler", "reddit-scraping", "reddit"]
|
||||
keywords = ["reddit-crawler", "reddit-scraping", "reddit", "reddit-api"]
|
||||
authors = [{name = "Richard Mwewa", email = "rly0nheart@duck.com"}]
|
||||
classifiers = [
|
||||
"Development Status :: 5 - Production/Stable",
|
||||
"Programming Language :: Python :: 3",
|
||||
"Programming Language :: Visual Basic",
|
||||
"Intended Audience :: Information Technology",
|
||||
"Intended Audience :: End Users/Desktop",
|
||||
"License :: OSI Approved :: MIT License",
|
||||
"Operating System :: OS Independent",
|
||||
"Natural Language :: English"
|
||||
@@ -26,6 +26,7 @@ classifiers = [
|
||||
|
||||
dependencies = [
|
||||
"rich",
|
||||
"glyphoji",
|
||||
"requests",
|
||||
]
|
||||
|
||||
@@ -35,4 +36,4 @@ documentation = "https://github.com/bellingcat/reddit-post-scraping-tool/wiki"
|
||||
repository = "https://github.com/bellingcat/reddit-post-scraping-tool.git"
|
||||
|
||||
[project.scripts]
|
||||
rpst = "rpst.__main:run"
|
||||
rpst = "rpst.main:run"
|
||||
|
||||
1
rpst/__init__.py
Normal file
@@ -0,0 +1 @@
|
||||
|
||||
201
rpst/__rpst_.py
@@ -1,201 +0,0 @@
|
||||
import json
|
||||
import logging
|
||||
import argparse
|
||||
import requests
|
||||
from rich.tree import Tree
|
||||
from rich import print as xprint
|
||||
from rich.markdown import Markdown
|
||||
from rich.logging import RichHandler
|
||||
|
||||
|
||||
def write_post_data(post_data: dict, filename: str):
|
||||
"""
|
||||
Writes post data to a specified JSON file.
|
||||
|
||||
:param post_data: A dictionary containing post data.
|
||||
:param filename: The name of the file to which post data will be written.
|
||||
"""
|
||||
# Write the data to a JSON file
|
||||
with open(filename + ".json", 'a') as file:
|
||||
file.write(json.dumps(post_data))
|
||||
file.write('\n') # write a newline to separate posts
|
||||
|
||||
log.info(f"Post data written to '{file.name}'")
|
||||
|
||||
|
||||
def check_updates(version_tag: str):
|
||||
"""
|
||||
This function checks if there's a new release of a project on GitHub. If there is, it logs an
|
||||
information message and prints the release notes.
|
||||
|
||||
:param version_tag: A string representing the current version of the project.
|
||||
"""
|
||||
|
||||
# Make a GET request to the GitHub API to get the latest release of the project
|
||||
response = requests.get("https://api.github.com/repos/bellingcat/reddit-post-scraping-tool/releases/latest").json()
|
||||
|
||||
# Check if the latest release's tag matches the current version tag
|
||||
if response['tag_name'] != version_tag:
|
||||
|
||||
# If not, convert the release notes from Markdown to HTML
|
||||
raw_release_notes = response['body']
|
||||
markdown_release_notes = Markdown(raw_release_notes)
|
||||
|
||||
# Log an info message about the new release
|
||||
log.info(
|
||||
f"A new release of RPST is available ({response['tag_name']}). "
|
||||
f"Run 'pip install --upgrade reddit-post-scraping-tool' to get the updates."
|
||||
)
|
||||
|
||||
# Print the release notes
|
||||
xprint(markdown_release_notes)
|
||||
|
||||
|
||||
def format_post_data(post: dict, keyword: str, output: bool):
|
||||
"""
|
||||
This function extracts relevant data from a Reddit post and displays it in a tree structure,
|
||||
followed by the post's selftext.
|
||||
|
||||
:param post: A dictionary containing the data of a Reddit post.
|
||||
:param keyword: The keyword that is used to find posts, in his case gets uses as the filename.
|
||||
:param output: If specified, all found posts will be written to a json file.
|
||||
"""
|
||||
# Define the data to extract from the post
|
||||
post_data = {
|
||||
'Author': post['data']['author'],
|
||||
'ID': post['data']['id'],
|
||||
'Subreddit': post["data"]["subreddit_name_prefixed"],
|
||||
'Visibility': post['data']['subreddit_type'],
|
||||
'Thumbnail': post["data"]["thumbnail"],
|
||||
'NSFW': post['data']['over_18'],
|
||||
'Gilded': post['data']['gilded'],
|
||||
'Upvotes': post["data"]["ups"],
|
||||
'Upvote ratio': post["data"]["upvote_ratio"],
|
||||
'Downvotes': post["data"]["downs"],
|
||||
'Awards': post["data"]["total_awards_received"],
|
||||
'Top award': post['data']['top_awarded_type'],
|
||||
'Is crosspostable?': post['data']['is_crosspostable'],
|
||||
'Score': post["data"]["score"],
|
||||
'Category': post['data']['category'],
|
||||
'Domain': post["data"]["domain"],
|
||||
'Created': post['data']['created'],
|
||||
'Approved at': post['data']['approved_at_utc'],
|
||||
'Approved by': post['data']['approved_by'],
|
||||
}
|
||||
if output:
|
||||
write_post_data(filename=keyword, post_data=post_data)
|
||||
# Create a tree structure with the post's title as the root
|
||||
post_tree = Tree("\n" + post['data']['title'])
|
||||
|
||||
# Add each piece of extracted data as a branch of the tree
|
||||
for post_key, post_value in post_data.items():
|
||||
post_tree.add(f"{post_key}: {post_value}")
|
||||
|
||||
# Print the tree structure
|
||||
xprint(post_tree)
|
||||
|
||||
# Print the post's selftext
|
||||
print(post['data']['selftext'] + "\n")
|
||||
|
||||
|
||||
def get_posts(arguments: argparse):
|
||||
"""
|
||||
Scrapes a given subreddit for posts that contain a specified keyword.
|
||||
The search is limited by the number of posts and timeframe specified. The results are either
|
||||
printed to the console or saved to a specified file, based on the 'output' argument.
|
||||
|
||||
:param arguments: Namespace object from argparse.
|
||||
|
||||
Expected Object Attributes
|
||||
--------------------------
|
||||
- keyword: The keyword to search for in the posts.
|
||||
- subreddit: The subreddit to scrape.
|
||||
- listing: The type of posts to scrape. This could be 'hot', 'new', etc.
|
||||
- timeframe: The timeframe from which to scrape posts. This could be 'day', 'week', etc.
|
||||
- limit: The maximum number of posts to scrape.
|
||||
- json: If specified, all found posts will be written to a json file.
|
||||
|
||||
Also logs the number of posts in which the keyword was found.
|
||||
"""
|
||||
keyword = arguments.keyword
|
||||
subreddit = arguments.subreddit
|
||||
listing = arguments.listing
|
||||
timeframe = arguments.timeframe
|
||||
limit = arguments.limit
|
||||
json_output = arguments.json
|
||||
|
||||
# Start a new session
|
||||
session = requests.session()
|
||||
# Set the User-Agent to mimic a Safari browser on a Mac
|
||||
session.headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, '
|
||||
'like Gecko) Version/14.1.1 Safari/605.1.15'}
|
||||
|
||||
# Send a GET request to the specified subreddit and listing,
|
||||
# limiting the response by the specified limit and timeframe
|
||||
response = session.get(f'https://reddit.com/r/{subreddit}/{listing}'
|
||||
f'.json?limit={limit}&t={timeframe}').json()
|
||||
|
||||
# Initialize a counter for the number of posts found that contain the keyword
|
||||
found_posts = 0
|
||||
|
||||
# Loop through each post in the response
|
||||
for post in response['data']['children']:
|
||||
# If the keyword is found in the post's selftext or title, increment the counter and process the post
|
||||
if keyword.lower() in post['data']['selftext'] or keyword.lower() in post['data']['title']:
|
||||
found_posts += 1
|
||||
format_post_data(post=post, keyword=keyword, output=json_output)
|
||||
|
||||
# Log the number of posts in which the keyword was found
|
||||
log.info(f"Keyword ('{keyword}') was found in {found_posts}/{len(response['data']['children'])} "
|
||||
f"{listing} posts from r/{subreddit}.")
|
||||
|
||||
|
||||
def create_parser():
|
||||
"""
|
||||
Creates and configures an argument parser for the command line arguments.
|
||||
|
||||
:return: A configured argparse.ArgumentParser object ready to parse the command line arguments.
|
||||
"""
|
||||
parser = argparse.ArgumentParser(
|
||||
description='RPST: Reddit Post Scraping Tool —by Richard Mwewa | https://about.me/rly0nheart',
|
||||
epilog='Given a subreddit name and a keyword, '
|
||||
'RPST returns all top (by default) posts that contain the specified keyword.'
|
||||
)
|
||||
|
||||
parser.add_argument('-k', '--keyword', help='The keyword to search for in the posts.', required=True)
|
||||
parser.add_argument('-s', '--subreddit', help='The subreddit to scrape.', required=True)
|
||||
parser.add_argument(
|
||||
'-c', '--limit',
|
||||
help='The maximum number of posts to scrape (1-100). (default: %(default)s)',
|
||||
default=10,
|
||||
type=int,
|
||||
choices=range(1, 101) # This enforces that the limit must be between 1 and 100 inclusive.
|
||||
)
|
||||
parser.add_argument(
|
||||
'-l', '--listing',
|
||||
default='top',
|
||||
const='top',
|
||||
nargs='?',
|
||||
choices=['controversial', 'hot', 'best', 'new', 'rising'],
|
||||
help='The type of posts to scrape (default: %(default)s)'
|
||||
)
|
||||
parser.add_argument(
|
||||
'-t', '--timeframe',
|
||||
default='all',
|
||||
const='all',
|
||||
nargs='?',
|
||||
choices=['hour', 'day', 'week', 'month', 'year', 'all'],
|
||||
help='The timeframe from which to scrape posts (default: %(default)s)'
|
||||
)
|
||||
parser.add_argument(
|
||||
'-j', '--json',
|
||||
help='Write all found posts to a json file.',
|
||||
action='store_true'
|
||||
)
|
||||
|
||||
return parser
|
||||
|
||||
|
||||
logging.basicConfig(level="NOTSET", format="%(message)s",
|
||||
handlers=[RichHandler(markup=True, log_time_format='[%H:%M:%S%p]')])
|
||||
log = logging.getLogger("rich")
|
||||
@@ -1,5 +1,7 @@
|
||||
from datetime import datetime
|
||||
from rpst.__rpst import log, get_posts, check_updates, create_parser
|
||||
|
||||
from .rpst import get_posts
|
||||
from .utils import create_parser, set_loglevel, check_updates
|
||||
|
||||
|
||||
def run():
|
||||
@@ -10,20 +12,22 @@ def run():
|
||||
|
||||
# Create a parser and parse the command line arguments
|
||||
parser = create_parser()
|
||||
arguments = parser.parse_args()
|
||||
args = parser.parse_args()
|
||||
|
||||
log = set_loglevel(debug_mode=args.debug)
|
||||
|
||||
# Record the start time
|
||||
start_time = datetime.now()
|
||||
|
||||
try:
|
||||
# Check for updates
|
||||
check_updates(version_tag="1.6.0.0")
|
||||
check_updates(version_tag="1.8.0.0")
|
||||
|
||||
# Get posts with the provided/parsed arguments
|
||||
get_posts(arguments=arguments)
|
||||
get_posts(args=args)
|
||||
except KeyboardInterrupt:
|
||||
log.warning("User interruption detected.")
|
||||
log.warning("User interruption detected ([yellow]Ctrl+C[/]).")
|
||||
except Exception as e:
|
||||
log.error(f"An error occurred: {e}")
|
||||
log.error(f"An error occurred: [red]{e}[/]")
|
||||
finally:
|
||||
log.info(f'Finished in {datetime.now() - start_time} seconds.')
|
||||
log.info(f"Finished in {datetime.now() - start_time} seconds.")
|
||||
131
rpst/rpst.py
Normal file
@@ -0,0 +1,131 @@
|
||||
import argparse
|
||||
from datetime import datetime
|
||||
|
||||
import requests
|
||||
from glyphoji import glyph
|
||||
from rich import print
|
||||
from rich.tree import Tree
|
||||
|
||||
from .utils import convert_timestamp_to_datetime, write_post_data
|
||||
|
||||
|
||||
def create_post_branch(post: dict, keyword: str, tree: Tree, args: argparse) -> Tree:
|
||||
"""
|
||||
This function extracts relevant data from a Reddit post and adds it in a tree branch structure,
|
||||
followed by the post's selftext.
|
||||
|
||||
:param post: A dictionary containing the data of a Reddit post.
|
||||
:param keyword: The keyword that is used to find posts, in his case gets uses as the filename.
|
||||
:param tree: Tree where the post branch will be added.
|
||||
:param args: A namespace object from argparse.
|
||||
:returns: The main tree with added post branches.
|
||||
"""
|
||||
# Define the data to extract from the post.
|
||||
post_data = {
|
||||
# "Author": post["data"]["author"],
|
||||
f"{glyph.id_button} ID": post["data"]["id"],
|
||||
f"{glyph.people_hugging} Subreddit": post["data"]["subreddit_name_prefixed"],
|
||||
f"{glyph.face_with_peeking_eye} Visibility": post["data"]["subreddit_type"],
|
||||
f"{glyph.framed_picture} Thumbnail": post["data"]["thumbnail"],
|
||||
f"{glyph.white_question_mark} Gilded": post["data"]["gilded"],
|
||||
f"{glyph.up_arrow} Upvotes": post["data"]["ups"],
|
||||
f"{glyph.chart_increasing} Upvote ratio": post["data"]["upvote_ratio"],
|
||||
f"{glyph.down_arrow} Downvotes": post["data"]["downs"],
|
||||
f"{glyph.trophy} Awards": post["data"]["total_awards_received"],
|
||||
f"{glyph.trophy} Top award": post["data"]["top_awarded_type"],
|
||||
f"{glyph.no_one_under_eighteen} Is NSFW?": post["data"]["over_18"],
|
||||
f"{glyph.left_arrow_curving_right} Is crosspostable?": post["data"][
|
||||
"is_crosspostable"
|
||||
],
|
||||
f"{glyph.bar_chart} Score": post["data"]["score"],
|
||||
f"{glyph.card_file_box} Category": post["data"]["category"],
|
||||
f"{glyph.globe_with_meridians} Domain": post["data"]["domain"],
|
||||
f"{glyph.calendar} Posted on": convert_timestamp_to_datetime(
|
||||
post["data"]["created"]
|
||||
),
|
||||
f"{glyph.calendar} Approved at": post["data"]["approved_at_utc"],
|
||||
f"{glyph.bust_in_silhouette} Approved by": post["data"]["approved_by"],
|
||||
}
|
||||
|
||||
# Add the post's branch to the main tree.
|
||||
post_branch = tree.add(f"{glyph.page_with_curl} {post['data']['title']}")
|
||||
|
||||
# Add each piece of extracted data as a branch of the post_branch.
|
||||
for post_key, post_value in post_data.items():
|
||||
post_branch.add(f"{post_key}: {post_value}", style="dim")
|
||||
|
||||
# This ensures that the post's selftext is also added to the written json/csv file.
|
||||
post_data[f"{glyph.clipboard} Text"] = post["data"]["selftext"]
|
||||
write_post_data(
|
||||
filename=keyword, post_data=post_data, tree_branch=post_branch, args=args
|
||||
)
|
||||
post_branch.add(post["data"]["selftext"], style="italic")
|
||||
|
||||
return tree
|
||||
|
||||
|
||||
def get_posts(args: argparse):
|
||||
"""
|
||||
Scrapes a given subreddit for posts that contain a specified keyword.
|
||||
The search is limited by the number of posts and timeframe specified.
|
||||
|
||||
:param args: Namespace object from argparse.
|
||||
|
||||
Expected Object Attributes
|
||||
--------------------------
|
||||
- keyword: The keyword to search for in the posts.
|
||||
- subreddit: The subreddit to scrape.
|
||||
- listing: The type of posts to scrape. This could be 'hot', 'new', etc.
|
||||
- timeframe: The timeframe from which to scrape posts. This could be 'day', 'week', etc.
|
||||
- limit: The maximum number of posts to scrape.
|
||||
- json: If specified, all found posts will be written to a json file.
|
||||
"""
|
||||
keyword = args.keyword
|
||||
subreddit = args.subreddit
|
||||
listing = args.listing
|
||||
timeframe = args.timeframe
|
||||
limit = args.limit
|
||||
|
||||
# Create main result tree.
|
||||
main_tree = Tree(
|
||||
f"[bold]{glyph.calendar} {datetime.now()}[/]", guide_style="bold bright_blue"
|
||||
)
|
||||
|
||||
# Start a new session
|
||||
session = requests.session()
|
||||
# Set the User-Agent to mimic a Safari browser on a Mac.
|
||||
session.headers = {
|
||||
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, "
|
||||
"like Gecko) Version/14.1.1 Safari/605.1.15"
|
||||
}
|
||||
|
||||
# Send a GET request to the specified subreddit and listing,
|
||||
# limiting the response by the specified limit and timeframe.
|
||||
response = session.get(
|
||||
f"https://reddit.com/r/{subreddit}/{listing}"
|
||||
f".json?limit={limit}&t={timeframe}"
|
||||
).json()
|
||||
|
||||
# Initialize a counter for the number of posts found that contain the keyword.
|
||||
found_posts = 0
|
||||
|
||||
# Loop through each post in the response
|
||||
for post_index, post in enumerate(response["data"]["children"], start=1):
|
||||
# If the keyword is found in the post's selftext or title, increment the counter and process the post.
|
||||
if (
|
||||
keyword.lower() in post["data"]["selftext"]
|
||||
or keyword.lower() in post["data"]["title"]
|
||||
):
|
||||
# Create a branch for found post(s) and show post index and post author as the title
|
||||
found_tree = main_tree.add(
|
||||
f"{glyph.bust_in_silhouette} #{post_index} by [bold]@{post['data']['author']}[/]"
|
||||
)
|
||||
found_posts += 1
|
||||
create_post_branch(post=post, keyword=keyword, tree=found_tree, args=args)
|
||||
|
||||
# Log the number of posts in which the keyword was found
|
||||
main_tree.add(
|
||||
f"{glyph.check_mark_button} Keyword ('{keyword}') was found in "
|
||||
f"{found_posts}/{len(response['data']['children'])} {listing} posts from r/{subreddit}."
|
||||
)
|
||||
print(main_tree)
|
||||
183
rpst/utils.py
Normal file
@@ -0,0 +1,183 @@
|
||||
import os
|
||||
import csv
|
||||
import json
|
||||
import logging
|
||||
import argparse
|
||||
from datetime import datetime
|
||||
|
||||
import requests
|
||||
from glyphoji import glyph
|
||||
from rich import print
|
||||
from rich.tree import Tree
|
||||
|
||||
from rich.markdown import Markdown
|
||||
from rich.logging import RichHandler
|
||||
|
||||
|
||||
def convert_timestamp_to_datetime(timestamp: int) -> str:
|
||||
"""
|
||||
Converts a Unix timestamp to a formatted datetime string.
|
||||
|
||||
:param timestamp: The Unix timestamp to be converted.
|
||||
:return: A formatted datetime string in the format "dd MMMM yyyy, hh:mm:ssAM/PM".
|
||||
"""
|
||||
utc_from_timestamp = datetime.utcfromtimestamp(timestamp)
|
||||
datetime_object = utc_from_timestamp.strftime("%d %B %Y, %I:%M:%S%p")
|
||||
return datetime_object
|
||||
|
||||
|
||||
def create_parser():
|
||||
"""
|
||||
Creates and configures an argument parser for the command line arguments.
|
||||
|
||||
:return: A configured argparse.ArgumentParser object ready to parse the command line arguments.
|
||||
"""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="RPST (Reddit Post Scraping Tool) —by Richard Mwewa | https://about.me/rly0nheart",
|
||||
epilog="Given a subreddit name and a keyword, "
|
||||
"RPST returns all top (by default) posts that contain the specified keyword.",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"-k", "--keyword", help="The keyword to search for in the posts.", required=True
|
||||
)
|
||||
parser.add_argument(
|
||||
"-s", "--subreddit", help="The subreddit to scrape.", required=True
|
||||
)
|
||||
parser.add_argument(
|
||||
"-c",
|
||||
"--limit",
|
||||
help="The maximum number of posts to scrape (1-100). (default: %(default)s)",
|
||||
default=10,
|
||||
type=int,
|
||||
choices=range(
|
||||
1, 101
|
||||
), # This enforces that the limit must be between 1 and 100 inclusive.
|
||||
)
|
||||
parser.add_argument(
|
||||
"-l",
|
||||
"--listing",
|
||||
default="top",
|
||||
const="top",
|
||||
nargs="?",
|
||||
choices=["controversial", "hot", "best", "new", "rising"],
|
||||
help="The type of posts to scrape (default: %(default)s)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-t",
|
||||
"--timeframe",
|
||||
default="all",
|
||||
const="all",
|
||||
nargs="?",
|
||||
choices=["hour", "day", "week", "month", "year", "all"],
|
||||
help="The timeframe from which to scrape posts (default: %(default)s)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--json",
|
||||
help="Write all found posts to a json file.",
|
||||
action="store_true",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--csv",
|
||||
help="Write all found posts to a csv file.",
|
||||
action="store_true",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-d",
|
||||
"--debug",
|
||||
help="run rpst in debug mode",
|
||||
action="store_true",
|
||||
)
|
||||
|
||||
return parser
|
||||
|
||||
|
||||
def check_updates(version_tag: str):
|
||||
"""
|
||||
This function checks if there's a new release of a project on GitHub. If there is, it logs an
|
||||
information message and prints the release notes.
|
||||
|
||||
:param version_tag: A string representing the current version of the project.
|
||||
"""
|
||||
|
||||
# Make a GET request to the GitHub API to get the latest release of the project.
|
||||
response = requests.get(
|
||||
"https://api.github.com/repos/bellingcat/reddit-post-scraping-tool/releases/latest"
|
||||
).json()
|
||||
|
||||
# Check if the latest release's tag matches the current version tag.
|
||||
if response["tag_name"] != version_tag:
|
||||
# If not, convert the release notes from Markdown to HTML.
|
||||
raw_release_notes = response["body"]
|
||||
|
||||
# Log an info message about the new release.
|
||||
print(
|
||||
f"{glyph.up_arrow} A new release of RPST is available ({response['tag_name']}). "
|
||||
f"Run 'pip install --upgrade reddit-post-scraping-tool' to get the updates."
|
||||
)
|
||||
|
||||
# Print the release notes.
|
||||
print(Markdown(raw_release_notes))
|
||||
|
||||
|
||||
def set_loglevel(debug_mode: bool) -> logging.getLogger:
|
||||
"""
|
||||
Configure and return a logging object with the specified log level.
|
||||
|
||||
:param debug_mode: If True, the log level is set to "NOTSET". Otherwise, it is set to "INFO".
|
||||
:return: A logging object configured with the specified log level.
|
||||
"""
|
||||
logging.basicConfig(
|
||||
level="NOTSET" if debug_mode else "INFO",
|
||||
format="%(message)s",
|
||||
handlers=[
|
||||
RichHandler(markup=True, log_time_format="[%I:%M:%S %p]", show_level=False)
|
||||
],
|
||||
)
|
||||
return logging.getLogger("RPST")
|
||||
|
||||
|
||||
def write_post_data(post_data: dict, filename: str, args, tree_branch: Tree):
|
||||
"""
|
||||
Writes post data to a specified JSON or CSV file based on the args provided, and updates
|
||||
the provided tree with the status.
|
||||
|
||||
:param post_data: A dictionary containing post data.
|
||||
:param filename: The name of the file to which post data will be written.
|
||||
:param args: A namespace object from argparse containing the output format options (args.json or args.csv).
|
||||
:param tree_branch: A rich Tree object to which status information will be added.
|
||||
"""
|
||||
home_directory = os.path.expanduser("~")
|
||||
|
||||
if args.json:
|
||||
json_file_path = os.path.join(home_directory, f"{filename}.json")
|
||||
with open(json_file_path, "a", encoding="utf-8") as file:
|
||||
file.write(json.dumps(post_data, ensure_ascii=False))
|
||||
file.write("\n") # Separate posts with newline
|
||||
tree_branch.add(
|
||||
f"{glyph.page_facing_up} JSON data successfully written/appended to file: "
|
||||
f"[italic][link file://{json_file_path}]{json_file_path}[/]"
|
||||
)
|
||||
else:
|
||||
tree_branch.add(
|
||||
f"{glyph.cross_mark_button} JSON data writing operation was skipped. No changes made."
|
||||
)
|
||||
|
||||
if args.csv:
|
||||
csv_file_path = os.path.join(home_directory, f"{filename}.csv")
|
||||
with open(csv_file_path, "a", newline="", encoding="utf-8") as csvfile:
|
||||
writer = csv.DictWriter(csvfile, fieldnames=post_data.keys())
|
||||
|
||||
# Write headers if file is empty
|
||||
if csvfile.tell() == 0:
|
||||
writer.writeheader()
|
||||
|
||||
writer.writerow(post_data)
|
||||
tree_branch.add(
|
||||
f"{glyph.page_facing_up} CSV data successfully written/appended to file: "
|
||||
f"[italic][link file://{csv_file_path}]{csv_file_path}[/]"
|
||||
)
|
||||
else:
|
||||
tree_branch.add(
|
||||
f"{glyph.cross_mark_button} CSV data writing operation was skipped. No changes made."
|
||||
)
|
||||