MediaBot

A production-ready Telegram bot for automated media discovery, indexing, queueing, and download management.

Features

Core Functionality

• Automated Crawling: Recursively crawl h5ai directory listings from multiple ISP servers
• Intelligent Indexing: Parse media filenames to extract metadata (title, year, quality, edition, codec, etc.)
• Smart Search: Hybrid search combining FTS5 full-text search with fuzzy matching (RapidFuzz WRatio)
• Resumable Downloads: Support for HTTP range requests with 3-attempt retry logic and exponential backoff
• File Organization: Auto-organize downloads into Movies/Series/Anime folders with metadata-based paths
• TMDB Integration: Optional background enrichment with movie metadata (ratings, descriptions)
• Admin Control: Telegram commands for manual reindexing, status checks, download queue management

Production Features

• FTS Index Sync: Automatic triggers keep search index in sync; rebuild on startup if needed
• Error Isolation: Graceful degradation if one server/download fails; others continue
• Status Monitoring: Real-time system health checks (disk space, database count, active downloads)
• Config Validation: Comprehensive environment variable validation at startup with clear errors
• Logging: Structured logging with time-rotated files (bot/crawler/downloader/search)
• Test Suite: Pytest coverage for parsers, database, search, and file organization

Setup

Requirements

• Python 3.10+
• SQLite3 (included with Python)
• Dependencies listed in requirements.txt

Installation

1. Clone or extract the project:

   git clone https://github.com/yourusername/mediabot.git
   cd mediabot

2. Create virtual environment (optional but recommended):

   python -m venv venv
   venv\Scripts\activate

3. Install dependencies:

   pip install -r requirements.txt

4. Copy and configure environment:

   cp .env.example .env

   Edit .env with your values:

   BOT_TOKEN=<your_telegram_bot_token>
   ADMIN_CHAT_ID=<your_admin_chat_id>
   CATEGORY_URLS=http://server1.local/movies,http://server2.local/series
   TMDB_API_KEY=<optional_tmdb_key>

   See .env.example for all available options.

5. Optional: Install aria2 for faster downloads

   • Windows: winget install aria2
   • Linux: sudo apt-get install aria2 or sudo yum install aria2
   • macOS: brew install aria2

   Then set in .env: ARIA2_PATH=aria2c (or full path if not in PATH)

6. Initialize database:

   python -c "from database.schema import init_db; init_db()"

7. Run the bot:

   python main.py

Usage

Telegram Commands

Command	Description
/start	Display bot help and commands
/help	Show available download patterns
/status	Show system health (media count, disk space, active downloads)
/reindex	Manually trigger media index crawl (choose category)
/queue	Show active downloads and progress
/cancel	Cancel active downloads

Search & Download

Simply send a message with download followed by what you want:

• Movie: download interstellar
• Movie with year: download interstellar 2014
• Movie with quality: download dune 1080p
• Series: download dark season 1
• Series shorthand: download dark s02e03
• Anime: download naruto episode 100
• Direct URL: download https://server.local/path/to/file.mkv

The bot will parse your intent, search the database, and present matching results for you to select.

Download Syntax

Send any message starting with download:

• Movie: download interstellar
• Movie with year: download interstellar 2014
• Movie with quality: download dune 1080p
• Series: download dark season 1
• Series shorthand: download dark s02e03
• Anime: download naruto episode 100
• Complex: download dark season 1 episode 5 720p

The bot will:

1. Parse your intent
2. Search the index using fuzzy matching + FTS
3. Present variants (if same title/year with different qualities) or disambiguation (if different titles)
4. Queue selected download with 3-attempt retry
5. Resume if interrupted
6. Organize into final folder structure

Architecture

Package Structure

mediabot/
├── bot/                      # Telegram bot handlers & keyboards
├── config/                   # Settings & environment validation
├── crawler/                  # h5ai spider, filename parsing
├── database/                 # SQLite schema, queries, FTS index
├── downloader/               # HTTP download manager, file organization
├── metadata/                 # TMDB enrichment (background)
├── parser/                   # Intent parsing for search queries
├── search/                   # FTS + fuzzy search engine
├── utils/                    # Logging, disk checks, status, formatting
├── tests/                    # Pytest test suite
├── main.py                   # Bot startup & initialization
└── requirements.txt          # Python dependencies

Data Flow

[Telegram User]
      ↓
[Bot Handlers] → parse intent → [Search Engine]
      ↓                              ↓
[Database Queries] ← FTS Index ← [Filename Parser]
      ↓                              ↑
[Download Queue] → [HTTP Downloader] → [ISP Servers]
      ↓
[File Organizer] → [Final Folder]

Database Schema

Tables

• media_index: Crawled media with metadata
• media_fts: Full-text search index (synced via triggers)
• download_queue: Active/completed downloads with resume state
• tmdb_cache: Cached TMDB metadata (30-day TTL)

Indexes

• Title, year, quality, category, season (for fast filtering)

Testing

Run the test suite:

pytest tests/ -v

Tests cover:

• Filename parsing edge cases
• Intent parsing robustness
• Database CRUD and FTS sync
• Search ranking
• File organization

Production Readiness

✅ Completed:

• FTS index auto-sync with triggers
• Startup config validation
• Error isolation and recovery
• Disk space monitoring
• Failed download recovery
• Comprehensive logging
• Edge case handling in parsers
• Test suite coverage

Performance Notes

• Search: O(n) fuzzy match + O(log n) FTS lookup, typically < 500ms
• Crawling: Async I/O with connection pooling, limited to 10 concurrent connections per server
• Downloads: Semaphore limits concurrent downloads (default 3), supports resume on network failure
• FTS: Triggers keep index in sync; rebuild < 1s for 10k files

Troubleshooting

Bot won't start

• Check .env for required variables (BOT_TOKEN, ADMIN_CHAT_ID)
• Verify CATEGORY_URLS are set and accessible
• Run python config/settings.py to validate configuration
• Check logs in LOG_PATH

Search returns no results

• Ensure database is indexed: /reindex command or check media count with /status
• Verify FTS index is synced: bot checks at startup and logs warnings if rebuild needed
• Verify CATEGORY_URLS in .env are correct and servers are reachable

Downloads fail repeatedly

• Check disk space: /status should show sufficient free space
• Verify CATEGORY_URLS are reachable and return h5ai directory listings
• Check logs for network errors or server response codes
• If using aria2, verify ARIA2_PATH is correct or set ARIA2_ENABLED=false to use HTTP only

Out of memory during crawl

• Reduce MAX_CRAWLER_DEPTH in .env (default 4)
• Limit CATEGORY_URLS to fewer servers
• Restart bot between crawls

License

This project is provided as-is for educational and personal use. You are free to fork, modify, and use for non-commercial purposes. For commercial use or redistribution, please obtain explicit permission.

Contributing

Contributions are welcome! Please open an issue or submit a pull request for any improvements.

Support

For issues or questions, check logs in LOG_PATH and ensure all environment variables in .env are configured correctly.