Skip to content

danilenzo/cuepoint

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

75 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

cuepoint

CI Python 3.12+ Tests Coverage mypy strict Docker License

Find the electronic music events worth going to — automatically.

Checking event listings manually is tedious — cross-referencing Resident Advisor, SoundCloud, Discogs, and Bandcamp for each artist on each lineup to figure out which events are actually worth attending. Cuepoint automates that entire workflow.

Give it a list of cities and a date range — it fetches every upcoming event from Resident Advisor and venue websites, enriches every artist from three platforms, scores and ranks events against your taste profile, and outputs an interactive report ready to browse before the weekend.

16 cities · 4 APIs · 641 tests · Used in production weekly.

Cuepoint card view


Tech Stack

Core: Python 3.12, FastAPI, httpx (async), SQLite (WAL mode) Data: pandas, BeautifulSoup, lxml Infra: Docker, Docker Compose, Make Testing: pytest (641 tests across 49 files), mypy strict, ruff APIs: GraphQL consumption, REST (Discogs, SoundCloud), web scraping (Bandcamp) Patterns: ETL pipeline, registry pattern, retry with exponential backoff, incremental processing, thread-safe concurrency


Engineering Highlights

  • 641 tests across 49 files — unit, integration, API endpoint, security (XSS, SSRF), concurrency, report template, and end-to-end pipeline coverage
  • Strict type checkingmypy --strict across the entire codebase with full type annotations
  • Zero Selenium — all HTTP via async httpx with retry/backoff/jitter
  • Incremental processing — SHA-256 lineup hashing skips 60-70% of enrichment work on repeat scans
  • Thread-safe concurrency — SQLite WAL + thread-local connections + semaphore rate limiters + ThreadPoolExecutor
  • Production-grade API — rate limiting, pagination, health checks, CSV export, background task execution
  • Clean architecture — frozen dataclasses, decorator-based registry pattern, configurable scoring weights via TOML
  • Retry resilience — all external calls wrapped with exponential backoff, jitter, and Retry-After header support
  • Circuit breaker — tracks SoundCloud 403 failure ratio, trips at 60% of 8+ requests; pipeline continues with Discogs/Bandcamp data rather than stalling
  • Dual SoundCloud auth — OAuth 2.1 when configured, automatic fallback to scraped client_id from JS bundles
  • CI pipeline — lint, typecheck, test, and security audit on every push
  • Scoring feedback loop — Went/Skipped verdicts from the report tune signal weights (clamped multipliers, cold-start gated) and accumulate genre/artist boosts; offline-capable via localStorage queue
  • Glass-design report — Vue 3 single-file report with glassmorphism design tokens, ambient glows, staggered entrance motion, sheet/modal transitions, relevance-ranked card lineups, an event detail view with a "why this matches" panel, and a cards-only mobile layout with bottom action bar

Screenshots

Card view Card view — flyer-forward, discovery signals, strength indicators

Table view Table view — dense multi-source data: pricing, follower counts, genres, promoters

Swagger UI REST API — FastAPI with rate limiting, pagination, CSV export, health checks

CI pipeline CI pipeline — lint, typecheck, test, and security audit on every push


Quick Start

Docker (recommended)

git clone https://github.com/danilenzo/cuepoint.git && cd cuepoint
docker compose up --build
# API at http://localhost:8000
# Swagger UI at http://localhost:8000/docs

Local

pip install -e ".[dev]"

# CLI - scan Berlin events for the next 7 days
python -m cuepoint.event_fetcher --cities berlin --days 7

# API
uvicorn cuepoint.api:app --reload --port 8000

Makefile

make install    # install package + dev tools
make run        # start API server
make test       # run test suite
make lint       # check linting

REST API

Interactive docs at http://localhost:8000/docs (Swagger UI).

Method Path Description
POST /scan Start a background scan for one or more cities
GET /status List all scans with their status
GET /status/{scan_id} Get status of a specific scan
GET /results/{city} Latest results for a city (paginated)
GET /results/{city}/export Download results as CSV
GET /health Readiness check (DB status, version)
GET /cities List available city keys
POST /feedback Record Went/Skipped verdicts (single or batch)
GET /feedback/stats Feedback counts and learned adjustments

Features: pagination, rate limiting (5 scans/60s per IP), persistent SQLite storage, CSV export, health checks for container orchestration.

Example requests
# start a scan
curl -X POST http://localhost:8000/scan \
  -H "Content-Type: application/json" \
  -d '{"cities": ["berlin", "london"], "days": 7}'

# check status
curl http://localhost:8000/status/abc123def456

# get results (page 1, 50 per page)
curl http://localhost:8000/results/berlin?page=1&page_size=50

# export as CSV
curl -O http://localhost:8000/results/berlin/export
Example response
{
  "city": "Berlin",
  "event_count": 42,
  "page": 1,
  "page_size": 50,
  "total_pages": 1,
  "events": [
    {
      "title": "Klubnacht",
      "event_date": "2026-04-18",
      "venue_name": "Berghain",
      "score": 48500.0,
      "lineup_notable": 4,
      "lineup_total": 8,
      "genres": ["Techno"],
      "artists": [
        {
          "name": "Surgeon",
          "sc_followers": 52000,
          "tags": ["techno", "industrial"],
          "rising": false
        }
      ]
    }
  ]
}

Architecture

src/cuepoint/
  event_fetcher.py   -- CLI entry point, pipeline orchestration, parallel runner
  api.py             -- FastAPI REST API with background scan execution
  enrichment.py      -- SC -> Discogs -> Bandcamp -> rising -> save pipeline
  scoring.py         -- filter, sort, scoring formula with configurable weights
  discovery.py       -- rising detection, artist similarity, label affinity
  db.py              -- SQLite storage (WAL mode, thread-safe, indexed)
  sc.py              -- SoundCloud API client (OAuth + fallback)
  discogs.py         -- Discogs REST API client (token auth, 60 req/min)
  bandcamp.py        -- Bandcamp scraper with semaphore rate limiting
  club_scrapers.py   -- @register_club decorator, per-room lineup parsing
  http_utils.py      -- @retry_on_failure with exponential backoff + jitter
  html_creator.py    -- Vue 3 interactive report generator
  gui.py             -- CustomTkinter desktop GUI
  following.py       -- followed artist set, URL matching
  fuzzy_match.py     -- name normalization, Levenshtein distance
  config.py          -- typed accessors from config.toml
tests/               -- 641 tests across 49 files

Pipeline Flow

Resident Advisor (GraphQL)  ──┐
Club websites (HTTP scrape)  ──┼──► Parse lineups
                               │
           ┌───────────────────┘
           ▼
    Enrich each artist (async, cached)
    ├── SoundCloud  → followers, tags
    ├── Discogs     → have/want, styles, labels
    └── Bandcamp    → supporters, tags, releases
           │
           ▼
    Score & rank events
    ├── Platform metrics weighted sum
    ├── Genre match multipliers
    ├── Followed artist bonus
    ├── Rising detection (growth vs baseline)
    ├── Artist similarity (Jaccard overlap)
    └── Shared label + recency bonuses
           │
           ▼
    Output: HTML report / REST API / GUI

Key Design Decisions

  • Thread-safe concurrency — SQLite WAL mode with thread-local connections, per-source rate limiters with locks and semaphores, ThreadPoolExecutor for parallel city scans
  • Incremental scans — lineup hash comparison (SHA-256) via SQLite snapshots skips 60-70% of enrichment work on repeat runs
  • Tiered cache TTLs — 7 days for followed artists, 30 days for others, 14-day soft stale threshold triggers re-enrichment
  • Retry resilience — all external API calls wrapped with exponential backoff + jitter, respects Retry-After on 429s
  • Registry pattern — club scrapers use @register_club("city") decorator for clean extensibility
  • Frozen dataclassScanContext replaces mutable module globals for safe parallel execution

Data Storage

SQLite at cache/cuepoint.db (WAL mode, thread-safe):

Table Purpose TTL
artist_urls Artist ID -> SC/Discogs/BC URLs permanent
artist_cache Full enrichment data per artist 30d (7d for followed)
found_events Events featuring followed artists permanent
artist_metrics_history Baselines for rising detection permanent
scan_events Lineup snapshots for incremental scans overwritten each scan
api_results Latest scan results per city for the API overwritten each scan

Scoring Formula
event_score = sum(artist_scores) + ra_genre_bonus * genre_count

artist_score = (sc_followers * genre_match / sc_weight)
             + (dc_have * genre_match / dc_weight)
             + (bc_supporters * genre_match / bc_weight)
             + followed_bonus
             + rising_bonus
             + similarity_score * similarity_weight
             + shared_label_count * shared_label_bonus
             + dc_ratio * dc_ratio_weight
             + recency_bonus * decay_factor

All weights configurable in config.toml.

Configuration

All settings in config.toml (defaults in config.toml.example):

[general]
days_ahead = 7
max_workers = 3
incremental = true

[cache]
ttl_days = 30
ttl_following_days = 7
stale_days = 14

[scoring]
sc_weight = 10
dc_weight = 5
bc_weight = 8
followed_bonus = 1000000

[genres]
filter = ["Techno", "Drum & Bass", "Drum n Bass"]
Supported Cities

amsterdam, athens, barcelona, berlin, birmingham, bristol, buenos aires, lisbon, london, madrid, osaka, paris, tbilisi, tokyo, warsaw, wuppertal

Club Scrapers
City Club Rooms
Berlin Berghain Berghain, Panorama Bar, Saule, Kantine
Berlin Tresor Tresor, Globus, Aurora Bar
Tbilisi Bassiani MainRoom, SecondRoom
Tbilisi Khidi --
Wuppertal Openground FREIFELD, ANNEX

Club events are deduplicated against the listing platform by venue + date matching.


Testing

make test                           # run all 641 tests
pytest tests/ --cov=src/cuepoint    # with coverage

49 test files covering: config validation, SQLite storage (CRUD + batch ops + migrations), HTTP retry logic (sync + async), SoundCloud auth/circuit breaker, Discogs/Bandcamp API mocking, club scraper parsing, enrichment pipeline, scoring with discovery signals, genre filtering, fuzzy matching, event fetching/parsing, HTML helpers, following detection, payload builders, pipeline stats, FastAPI endpoints (health, pagination, rate limiting, export), security (XSS injection, SSRF prevention), concurrency under load, lazy initialization, and full end-to-end pipeline tests.


Deploy

The project includes a Dockerfile and Procfile for one-click deploy on any container platform. The $PORT env var is respected automatically.

# Railway
railway up

# Fly.io
fly launch

Honest Gaps

Trade-offs and things left undone, stated plainly:

  • No structured logging — loguru plaintext output; would emit JSON in production for log aggregation
  • Mocked HTTP in tests — responses are hand-mocked rather than recorded VCR fixtures, so tests can drift from real API shapes
  • Single-process architecture — parallelism is thread-based within one process; no distributed workers or job queue

License

MIT

About

Multi-source ETL pipeline for electronic music event discovery — scrapes RA, enriches via SoundCloud/Discogs/Bandcamp, scores and ranks events

Topics

Resources

License

Contributing

Stars

1 star

Watchers

0 watching

Forks

Packages

 
 
 

Contributors

Languages