Skip to content
BranchesTags

Repository files navigation

Reddit Intelligence System

Automated Reddit data collection and strategic intelligence reporting system

Transform raw Reddit data into actionable marketing intelligence with automated crawling, analysis, and professional reporting. Fully customizable for any brand or company.

🎯 What This Does

This system automatically:

  1. Crawls Reddit posts from your configured subreddits (weekly, monthly, yearly data)
  2. Analyzes posts for pain points, trending topics, and brand relevance
  3. Generates professional intelligence reports (PDF) with:
    • Subreddit activity rankings and profiles
    • Pain point detection based on your brand's strengths
    • Trending topic analysis
    • Brand relevance scoring (0-100 per post)
    • Karma-building strategies per community
    • Weekly action plan

Output: Consulting-grade PDF reports that tell your team WHERE to post, WHAT to say, and HOW to gain karma.

Fully Customizable: Configure your brand name, strengths, pain points, and target subreddits in brand_config.py

📊 Example Use Cases

This system is designed for any brand to monitor relevant Reddit communities and identify engagement opportunities.

Example Configuration (Cloud Storage Company):

  • Subreddits: kubernetes, MachineLearning, devops, aws, dataengineering
  • Brand Strengths: Cost optimization, data sovereignty, performance
  • Pain Points: High cloud costs, compliance requirements, vendor lock-in
  • Sample Output: 1,000+ posts analyzed, 410 high-engagement opportunities identified

Example Configuration (SaaS Company):

  • Subreddits: startups, SaaS, entrepreneur, smallbusiness
  • Brand Strengths: Ease of use, integrations, customer support
  • Pain Points: Complex onboarding, poor UI/UX, expensive tools

You can configure your own:

  • Industry-specific subreddits (technology, finance, healthcare, gaming, etc.)
  • Brand differentiators and competitive advantages
  • Customer pain points your product solves
  • Custom report branding and messaging

🚀 Quick Start

1. Prerequisites

# Required
- Python 3.8+
- Supabase account (free tier works)
- macOS (for automated scheduling)

# Install dependencies
pip install -r requirements.txt

2. Database Setup

Option A: Use existing Supabase project

Run the migration in your Supabase SQL Editor:

# Copy contents of database/RUN_THIS_IN_SUPABASE.sql to Supabase SQL Editor
# Execute the script

Option B: Fresh Supabase project

# 1. Create new project at https://supabase.com
# 2. Go to SQL Editor
# 3. Paste contents of database/RUN_THIS_IN_SUPABASE.sql
# 4. Execute

3. Configure Brand & Credentials

A) Brand Configuration

# 1. Copy the example brand configuration
cp brand_config.py.example brand_config.py

# 2. Edit brand_config.py with your brand details
nano brand_config.py

Customize these settings in brand_config.py:

  • BRAND_NAME: Your company name
  • BRAND_STRENGTHS: Keywords related to your competitive advantages
  • PAIN_POINT_KEYWORDS: Problems your product solves
  • TARGET_SUBREDDITS: Subreddits relevant to your industry
  • REPORT_TITLE and REPORT_SUBTITLE: Custom report branding

B) Supabase Credentials

IMPORTANT: Never commit credentials to GitHub!

# 1. Copy the example environment file
cp .env.example .env

# 2. Edit .env with your Supabase credentials
nano .env

Add your credentials to .env:

SUPABASE_URL=https://YOUR_PROJECT.supabase.co
SUPABASE_SERVICE_ROLE_KEY=your_service_role_key_here

Where to find these:

  • Go to your Supabase project → Settings → API
  • URL: Project URL
  • Service Role Key: Under "Project API keys" → service_role (keep this secret!)

For LaunchAgents (automated scheduling): Edit the plist files and replace placeholders with your credentials:

# Edit crawler plist (rename from com.yourcompany.* to your actual company)
nano com.yourcompany.redditcrawler.plist

# Edit report plist
nano com.yourcompany.redditreport.plist

# Replace YOUR_SUPABASE_URL_HERE and YOUR_SUPABASE_SERVICE_ROLE_KEY_HERE

4. Run the Crawler

Manual run (test):

python3 local_reddit_crawler.py

Expected output:

📊 r/kubernetes
----------------------------------------
    week: 88 fetched, 5 new, 83 duplicates
   month: 100 fetched, 12 new, 88 duplicates
    year: 100 fetched, 37 new, 63 duplicates
   TOTAL: ✓ 54 new, 234 duplicates

Time: ~5-7 minutes (rate-limited for Reddit compliance)

4. Generate Intelligence Report

python3 generate_intelligence_report.py

Output: [BRAND_SHORT_NAME]_Reddit_Intelligence_YYYYMMDD.pdf (e.g., YC_Reddit_Intelligence_20231115.pdf)

📅 Fully Automated Weekly Intelligence (macOS)

✅ FULLY AUTOMATED - No manual intervention required!

Every Monday, the system automatically:

  1. 9:00 AM: Crawls Reddit for fresh posts
  2. 9:15 AM: Generates intelligence report PDF

Setup LaunchAgents

First, rename the plist files to match your company name:

# 1. Rename plist files (replace 'yourcompany' with your actual company name)
mv com.yourcompany.redditcrawler.plist com.acme.redditcrawler.plist
mv com.yourcompany.redditreport.plist com.acme.redditreport.plist

# 2. Edit the Label inside each plist file to match the new filename
nano com.acme.redditcrawler.plist  # Update <key>Label</key> value
nano com.acme.redditreport.plist    # Update <key>Label</key> value

# 3. Copy both plist files to LaunchAgents
cp com.acme.redditcrawler.plist ~/Library/LaunchAgents/
cp com.acme.redditreport.plist ~/Library/LaunchAgents/

# 4. Load both agents
launchctl load ~/Library/LaunchAgents/com.acme.redditcrawler.plist
launchctl load ~/Library/LaunchAgents/com.acme.redditreport.plist

# 5. Verify both are loaded
launchctl list | grep acme

Expected output:

-	0	com.acme.redditreport
-	0	com.acme.redditcrawler

Automated Schedule

Time Task Duration Output
9:00 AM Crawler fetches posts 5-7 min logs/crawler.log
9:15 AM Report generated ~10 sec [BRAND_SHORT_NAME]_Reddit_Intelligence_YYYYMMDD.pdf

Fresh report every Monday - Just check the folder for the new PDF!

Manual Control

# Test now (don't wait for Monday)
# Replace 'yourcompany' with your actual company name from plist filenames
launchctl start com.yourcompany.redditcrawler
# Wait 7 minutes...
launchctl start com.yourcompany.redditreport

# View logs in real-time
tail -f logs/crawler.log
tail -f logs/report.log

# Disable automation
launchctl unload ~/Library/LaunchAgents/com.yourcompany.redditcrawler.plist
launchctl unload ~/Library/LaunchAgents/com.yourcompany.redditreport.plist

# Re-enable automation
launchctl load ~/Library/LaunchAgents/com.yourcompany.redditcrawler.plist
launchctl load ~/Library/LaunchAgents/com.yourcompany.redditreport.plist

📖 How to Use the Intelligence Report

Report Structure

  1. Executive Summary (Page 1-2)

    • Key findings and metrics
    • Strategic recommendations
    • Top opportunities overview

  2. Subreddit Intelligence (Page 3-5)

    • Activity rankings (🔥 indicators)
    • IC positioning angles per subreddit
    • Karma-building strategies
    • Risk assessments

  3. Trending Topics & Pain Points (Page 6)

    • Hot topics this week
    • Pain point analysis (AWS costs, GDPR, etc.)
    • IC solution fit scoring

  4. Priority Opportunities (Page 7-9)

    • Top 10 posts ranked by IC relevance (0-100)
    • Engagement strategies per post
    • Risk levels and timing windows

  5. Karma-Building Playbook (Page 10)

    • DOs and DON'Ts for Reddit engagement
    • Example comment flows
    • Platform-specific strategies

  6. Weekly Action Plan (Page 11)

    • Monday: 30 min (top 5 posts)
    • Wednesday: 15 min (follow-ups)
    • Friday: 10 min (wrap-up)
    • Total: 55 minutes/week

Brand Relevance Score Explained

Algorithm (0-100 points):

  • Brand Strengths Match (+10 each): Keywords from your BRAND_STRENGTHS configuration
  • Pain Points (+20): Keywords from your PAIN_POINT_KEYWORDS configuration
  • High Engagement (+15): 100+ upvotes OR 50+ comments

Score Interpretation:

  • 80-100: PERFECT FIT - Must engage
  • 60-79: VERY RELEVANT - High priority
  • 40-59: GOOD MATCH - Consider
  • 20-39: MODERATE - Low priority
  • 0-19: WEAK - Skip

Example (for a cloud storage company):

Post: "Our AWS bill is $15k/month - need cheaper S3-compatible storage with GDPR compliance"

Score: 65/100

  • AWS cost pain point (+20)
  • "cheaper" keyword (+10 cost strength)
  • "storage" keyword (+10 storage strength)
  • "GDPR" keyword (+10 sovereignty strength)
  • High engagement (+15)

🛠️ Configuration

All customization is done in brand_config.py. Copy from the example template:

cp brand_config.py.example brand_config.py
nano brand_config.py

Configure Brand Identity

BRAND_NAME = "Your Company"
BRAND_SHORT_NAME = "YC"  # Used for scoring and filenames
BRAND_FOOTER = "Generated for Your Company Marketing Team"

REPORT_TITLE = "Reddit Intelligence Report"
REPORT_SUBTITLE = "Strategic Social Listening Analysis"

Configure Tracked Subreddits

TARGET_SUBREDDITS = [
    "technology",
    "business",
    "entrepreneur",
    # Add your industry-specific subreddits:
    "cloudnative",
    "sysadmin",
    "marketing"
]

Configure Brand Strengths & Pain Points

BRAND_STRENGTHS = {
    'cost': ['cost', 'expensive', 'pricing', 'cheaper'],
    'performance': ['fast', 'speed', 'latency', 'performance'],
    # Add your competitive advantages
}

PAIN_POINT_KEYWORDS = {
    'high_costs': ['expensive', 'cost too much', 'pricing'],
    'complexity': ['complicated', 'complex', 'confusing'],
    # Add pain points your product solves
}

Change Crawl Frequency

Edit com.yourcompany.redditcrawler.plist (your renamed plist file):

<!-- Weekly (Monday 9 AM) -->
<key>StartCalendarInterval</key>
<dict>
    <key>Weekday</key>
    <integer>1</integer>  <!-- 1=Monday -->
    <key>Hour</key>
    <integer>9</integer>
</dict>

<!-- Daily -->
<key>StartCalendarInterval</key>
<dict>
    <key>Hour</key>
    <integer>9</integer>
</dict>

📁 Project Structure

reddit-intelligence/
├── local_reddit_crawler.py              # Main crawler script
├── generate_intelligence_report.py      # Report generator script
├── brand_config.py                      # Your brand config (git-ignored)
├── .env                                 # Your credentials (git-ignored)
├── requirements.txt                     # Python dependencies
├── LICENSE                              # MIT License
│
├── config/                              # Configuration templates
│   ├── brand_config.py.example          # Brand settings template
│   ├── .env.example                     # Credentials template
│   ├── com.yourcompany.redditcrawler.plist  # LaunchAgent template
│   ├── com.yourcompany.redditreport.plist   # LaunchAgent template
│   └── README.md                        # Config setup guide
│
├── database/                            # SQL scripts
│   ├── RUN_THIS_IN_SUPABASE.sql         # Main database setup
│   ├── check_schema.sql                 # Verification queries
│   ├── performance_tracking.sql         # Analytics queries
│   └── README.md                        # Database documentation
│
├── reports/                             # Generated intelligence reports
│   ├── 2024/
│   │   ├── week_46/                     # Nov 11-17, 2024
│   │   ├── week_47/                     # Nov 18-24, 2024
│   │   ├── week_48/                     # Nov 25-Dec 1, 2024
│   │   ├── week_49/                     # Dec 2-8, 2024
│   │   ├── week_50/                     # Dec 9-15, 2024
│   │   ├── week_51/                     # Dec 16-22, 2024
│   │   └── week_52/                     # Dec 23-29, 2024
│   └── README.md                        # Reports guide
│
├── docs/                                # Additional documentation
│   ├── EDGE_FUNCTION_OPTION.md          # Cloud deployment guide
│   └── README.md                        # Docs index
│
├── logs/                                # Crawler logs (auto-created)
│
└── archive/                             # Legacy files (not in git)

Core Files

File Purpose
local_reddit_crawler.py Fetches posts from Reddit API, stores in Supabase
generate_intelligence_report.py Analyzes data, generates PDF reports
brand_config.py Your brand configuration (created from template)
.env Your Supabase credentials (created from template)
requirements.txt Python dependencies (supabase, reportlab, requests)

Key Directories

Directory Purpose
config/ Configuration templates for setup
database/ SQL scripts for Supabase setup
reports/ Weekly intelligence reports organized by calendar week
docs/ Additional guides and documentation
logs/ Crawler execution logs

🔧 Troubleshooting

Crawler Issues

Problem: Reddit API error: 429

  • Solution: Rate limited. Wait 60 seconds and retry. The crawler already includes delays.

Problem: Supabase connection error

  • Solution: Check your service role key in local_reddit_crawler.py line 14.

Problem: Duplicate posts (all duplicates, no new posts)

  • Solution: Normal! This means you already have all recent data. Wait a week for fresh posts.

Report Generation Issues

Problem: No posts found

  • Solution: Run the crawler first to populate data.

Problem: Low brand relevance scores

  • Solution: Adjust keywords in BRAND_STRENGTHS and PAIN_POINT_KEYWORDS in brand_config.py to match your priorities.

Problem: PDF generation errors

  • Solution: pip install reportlab --upgrade

LaunchAgent Issues

Problem: Crawler not running on schedule

# Check status (replace 'yourcompany' with your company name)
launchctl list | grep yourcompany

# Check logs
tail -f logs/reddit_crawler.log

# Reload agent
launchctl unload ~/Library/LaunchAgents/com.yourcompany.redditcrawler.plist
launchctl load ~/Library/LaunchAgents/com.yourcompany.redditcrawler.plist

📊 Database Schema

Main Tables

subreddits - Tracked communities

  • id, name, display_name, crawl_enabled, last_crawled_at

reddit_posts - Crawled posts

  • Post metadata (title, text, author, url)
  • Engagement metrics (score, comments, upvote_ratio)
  • Timestamps (created_utc, created_date)
  • Calculated fields (engagement_score, is_high_engagement)

Useful Queries

Check latest crawl:

SELECT post_subreddit, COUNT(*), MAX(post_created_date)
FROM reddit_posts
GROUP BY post_subreddit
ORDER BY COUNT(*) DESC;

Top posts this week:

SELECT post_title, post_subreddit, post_score, post_num_comments
FROM reddit_posts
WHERE post_created_date > NOW() - INTERVAL '7 days'
ORDER BY post_score DESC
LIMIT 20;

Pain point mentions:

SELECT post_title, post_subreddit, post_score
FROM reddit_posts
WHERE post_title ILIKE '%aws cost%'
   OR post_text ILIKE '%aws cost%'
ORDER BY post_score DESC;

🎯 Best Practices

Data Collection

  • ✅ Run crawler weekly for fresh data
  • ✅ Let it fetch from all 3 time periods (week, month, year)
  • ✅ Check logs after each run
  • ✅ Monitor Supabase database size (free tier: 500MB)

Intelligence Reports

  • ✅ Generate report Monday morning before team engagement
  • ✅ Focus on top 5 opportunities (highest IC relevance)
  • ✅ Share report with marketing/community team
  • ✅ Track which opportunities you engaged with

Reddit Engagement

  • ✅ NO brand mentions in first comment (build credibility first)
  • ✅ Reference specific numbers from posts ("$8k/month resonates...")
  • ✅ Ask genuine follow-up questions
  • ✅ Only mention IC in follow-ups if conversation flows naturally
  • ❌ Never use promotional language ("Check out...", "Try...")
  • ❌ Never hijack promotional posts (beta launches, product announcements)
  • ❌ Never engage in posts older than 24 hours

📈 Performance Tracking

Weekly Metrics to Track

Create a simple spreadsheet:

Week Posts Analyzed Opportunities Engaged Karma Earned IC Mentions Notes
Nov 11 1000 10 3 142 2 Focused on r/kubernetes
Nov 18 1150 12 5 287 3 AWS cost thread went viral

Success Metrics

  • Target: 5-8 engagements/week
  • Time: 55 minutes/week
  • Karma: 150-300 upvotes/week
  • IC Mentions: 2-5/week (in follow-ups only)
  • Negative responses: 0

🔒 Privacy & Compliance

  • ✅ Read-only access (no posting automation)
  • ✅ Respects Reddit rate limits (2-3s delays)
  • ✅ Uses standard Reddit JSON API (public data)
  • ✅ No personal data collection
  • ✅ Complies with Reddit API Terms of Service

Important: Always follow Reddit's User Agreement and use responsibly.

🚧 Roadmap

  • Comment thread analysis (understand discussions, not just posts)
  • Sentiment analysis (positive/negative/neutral detection)
  • Competitor mention tracking
  • Email/Slack notifications for high-priority opportunities
  • Historical trend analysis (topic popularity over time)
  • Web dashboard for exploring data

💡 Tips for Maximum Value

Week 1: Setup & Learn

  • Run crawler manually, verify data
  • Generate first report, understand structure
  • Share with team, get feedback

Week 2-4: Optimize

  • Track which opportunities convert to high karma
  • Refine IC relevance keywords for your use case
  • Experiment with engagement timing

Long-term

  • Build reputation in 2-3 core subreddits
  • Use reports to guide content strategy
  • Track pain points that keep appearing → product roadmap input

🆘 Support

Issues?

  1. Check logs: logs/reddit_crawler.log
  2. Verify database: Supabase dashboard → Table Editor
  3. Test crawler: python3 local_reddit_crawler.py
  4. Check dependencies: pip install -r requirements.txt

Common Questions:

  • Q: Why so many duplicates? A: Normal! It means you already have those posts. Fresh data comes weekly.

  • Q: Can I run this on Windows/Linux? A: Yes, but auto-scheduling (LaunchAgent) is macOS-specific. Use cron for Linux, Task Scheduler for Windows.

  • Q: How do I add more subreddits? A: Edit SUBREDDITS list in local_reddit_crawler.py

  • Q: Can I customize the report design? A: Yes! Edit generate_intelligence_report.py - it uses ReportLab (Python PDF library)

📄 License

MIT License - Use freely for your marketing intelligence needs.

Open Source Reddit Intelligence System

Built with ❤️ and shared with the community.

Last updated: November 2024

About

No description or website provided.

Topics

tech

Resources

Readme
MIT license
Activity
Custom properties

Stars

1 star

Watchers

1 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages