25Hz tracking data pipeline via core.async — compactness and pressing intensity metrics with explicit backpressure monitoring.
Football tracking data arrives at 25 frames per second — 1,500 positional snapshots per minute per match. formation-stream replays StatsBomb 360 freeze frames through a core.async pipeline that computes defensive compactness and pressing intensity per snapshot, with explicit backpressure monitoring that reports buffer fill levels, processing lag, and dropped events. It demonstrates that Clojure's channel abstraction can handle real-time tracking data velocity without losing events — or, when it does lose events, that the system knows exactly how many and why.
| Dimension | Python asyncio | Python threading | core.async |
|---|---|---|---|
| Concurrency model | Single-threaded cooperative. A slow consumer blocks the entire event loop. | OS threads with GIL contention. | Go-blocks on a fixed thread pool. Producer and consumer are fully decoupled. |
| Buffer depth | Queue depth is often implicit and unmonitored (asyncio.Queue exists but is rarely instrumented). |
queue.Queue has a size parameter but overflow handling is ad-hoc. |
Channel buffer size is a first-class parameter you set intentionally. offer! vs >! makes the overflow policy explicit. |
| Backpressure | Must be manually implemented with semaphores or bounded queues. | Must be manually implemented. | Built into the channel abstraction. A full buffer either blocks the producer (>!) or drops the event (offer!) — you choose. |
| Cross-runtime | Python only. | Python only. | The same core.async code runs on the JVM for batch replay and in ClojureScript for browser-based real-time display (Project 5 — dugout). One abstraction, two runtimes. |
| Transducers | No equivalent. Processing is a separate step. | No equivalent. | Channels accept transducers — the enrichment step runs inside the channel transfer, not as a separate consumer. |
┌──────────────────────┐
│ StatsBomb 360 │
│ Freeze Frames │
│ (or synthetic) │
└────────┬─────────────┘
│
▼
┌──────────────────────┐ offer!
│ PRODUCER │─────────────────┐
│ go-loop │ │ (dropped events
│ 40ms intervals │ │ recorded in monitor)
│ (configurable) │ │
└────────┬─────────────┘ │
│ a/offer! │
▼ │
┌──────────────────────┐ │
│ tracking-ch │◄────────────────┘
│ buffer: 100 │
│ absorbs producer │
│ bursts │
└────────┬─────────────┘
│ a/pipe
▼
┌──────────────────────┐
│ enriched-ch │
│ buffer: 50 │
│ xf: enrich-snapshot│
│ (compactness, │
│ pressing, zone) │
└────────┬─────────────┘
│ a/<!
▼
┌──────────────────────┐
│ CONSUMER │
│ go-loop │
│ emit → stdout │
│ record lag → │
│ monitor │
└────────┬─────────────┘
│
▼
┌──────────────────────┐
│ MONITOR │
│ atom-based state │
│ events/sec │
│ avg lag (ms) │
│ dropped count │
│ periodic report │
└──────────────────────┘
What it measures: How tightly packed the defensive team is.
Formula: Mean pairwise Euclidean distance between all defending players.
compactness = (1 / n(n-1)) × Σᵢ Σⱼ d(pᵢ, pⱼ) where i ≠ j
Where d(pᵢ, pⱼ) is the Euclidean distance between defender positions pᵢ and pⱼ on the StatsBomb 120×80 pitch coordinate system.
Interpretation: Lower value = more compact defensive shape. A compact defense closes passing lanes and limits space for the attacking team.
Trade-off note: Convex hull area would be more accurate for measuring the "spread" of a defensive block, but it requires a computational geometry library (e.g., JTS). Mean pairwise distance is a reasonable proxy that avoids external dependencies and is trivially testable.
What it measures: How aggressively the defending team is pressing the ball carrier.
Formula: Mean distance of the nearest N defenders to the ball carrier (actor).
pressing = (1/N) × Σᵢ₌₁ᴺ d(actor, defenderᵢ)
Where N = 3 by default (the Gegenpressing principle requires at least 3 players pressing the ball immediately after loss of possession), and defenders are sorted by distance to the actor.
Interpretation: Lower value = higher pressing intensity. A pressing intensity below ~8.0 on the StatsBomb coordinate system indicates an active Gegenpressing moment.
In a producer-consumer pipeline, backpressure occurs when the producer generates data faster than the consumer can process it. Without explicit handling, this leads to unbounded memory growth, lost data, or a blocked producer.
The producer uses a/offer! instead of a/>! to put events onto the tracking channel:
a/>!(blocking put): Parks the producer go-block until buffer space is available. Safe but causes the producer to fall behind real-time — unacceptable for a 25Hz tracking feed.a/offer!(non-blocking put): Returnsfalseimmediately if the buffer is full. The event is dropped, but the producer maintains real-time pacing.
The monitor namespace maintains an atom with:
- events-processed: Total events that made it through the pipeline
- events-dropped: Total events that
offer!could not place (buffer full) - processing-lags: Rolling window (last 100) of millisecond lag from ingestion to emit
- events-per-sec: Computed from events-processed and wall clock time
A periodic go-loop reports these stats at a configurable interval (default: 5 seconds). In production, this would feed a Grafana dashboard or similar.
When the tracking-ch buffer (100 events) fills:
offer!returnsfalse- The monitor records a drop via
record-drop! - The producer continues at real-time pacing — it does not block
- The dropped event is lost (acceptable for positional data; the next frame arrives in 40ms)
This is an explicit design choice: for tracking data, it is better to lose a frame and maintain timing than to buffer indefinitely and fall behind.
| Channel | Buffer Size | Rationale |
|---|---|---|
tracking-ch |
100 | At 25Hz, 100 events = 4 seconds of data. This absorbs short processing spikes (e.g., GC pauses) without dropping. If the consumer falls behind by more than 4 seconds, the data is stale anyway — dropping is correct. |
enriched-ch |
50 | Smaller than tracking-ch because metric computation is fast (~1ms per snapshot). A large buffer here would mask consumer-side backpressure — if the emit step is slow, we want to see drops at the tracking-ch level, not silently buffer thousands of enriched events. |
The 2:1 ratio between buffers is intentional. The tracking-ch buffer absorbs producer bursts; the enriched-ch buffer absorbs emit I/O variability. If the enriched-ch buffer were as large as tracking-ch, we would never observe backpressure until we had 150 events in flight — by which point we're 6 seconds behind real-time.
- Java 21+
- Leiningen
lein run -- --synthetic true --speed 2.0 --format ednexport STATSBOMB_DATA_PATH="$HOME/data/statsbomb-open-data/data"
lein run -- --match-id 3764760 --speed 1.0 --format json| Flag | Default | Description |
|---|---|---|
--synthetic |
false |
Use synthetic freeze frames (no data files needed) |
--match-id |
— | StatsBomb match ID (required unless --synthetic) |
--speed |
1.0 |
Replay speed multiplier (2.0 = double speed) |
--format |
edn |
Output format: edn or json |
--report-ms |
5000 |
Monitor report interval in milliseconds |
lein test{:event-uuid "a3f1c892-7d4e-4b1a-9c5f-2e8d6b3a1f07", :compactness 18.866, :pressing-intensity 14.038, :ball-zone :zone10}
{:event-uuid "b7e2d541-3c8f-4a2b-8d6e-9f1a5c4b2d03", :compactness 23.860, :pressing-intensity 16.638, :ball-zone :zone09}
{:event-uuid "c5d4e832-1b6a-4f9c-7e3d-8a2b5c1d4e06", :compactness 15.735, :pressing-intensity 9.964, :ball-zone :zone09}
{:event-uuid "d8f3a721-4e5b-4c8d-6a2f-1b9c7e3d5a08", :compactness 22.940, :pressing-intensity 15.870, :ball-zone :zone10}
{:event-uuid "e1c6b943-2d7f-4a5e-9b8c-3f4a6d2e1b09", :compactness 12.154, :pressing-intensity 8.983, :ball-zone :zone09}Monitor report (logged at INFO level):
Pipeline monitor report {:events-processed 100, :events-dropped 0, :events-per-sec "147.1", :avg-lag-ms "0.7"}
This is Project 4 of a five-project Clojure sports data engineering portfolio:
| Project | Repository | Connection |
|---|---|---|
| pitch-pipe | dennisgathu8/pitch-pipe | Zone computation (compute-zone) is reimplemented inline from pitch-pipe's zone logic. When pitch-pipe is published to Clojars, this function will be replaced with a direct dependency. |
| press-logic | dennisgathu8/press-logic | Pressing intensity metric connects directly to press-logic's Gegenpressing rules. The N=3 default for nearest defenders mirrors press-logic's pressing trigger threshold. |
| dugout (Project 5) | Coming soon | formation-stream's enriched output is designed to feed dugout's WebSocket stream for real-time browser-based tactical display. The same core.async channels work in ClojureScript. |
Building formation-stream was the moment backpressure stopped being an abstract concept and became a concrete design decision. The difference between offer! and >! seems small — one drops, one blocks — but at 25Hz the consequences diverge completely. A blocking producer falls behind real-time within seconds; the buffer fills, the go-block parks, and suddenly you're processing frame 200 while the match is at frame 500. With offer!, you lose a frame but you stay synchronized. That frame arrives again in 40 milliseconds — the cost of dropping it is negligible compared to the cost of falling behind. What surprised me most was how the buffer size ratio between tracking-ch (100) and enriched-ch (50) acts as a pressure gauge: when drops start appearing at the tracking channel but the enriched channel is healthy, you know the bottleneck is upstream. When the enriched channel starts backing up, the emit step is too slow. The two-buffer architecture doesn't just decouple producer and consumer — it gives you a diagnostic instrument for understanding where the system is under load.
This project uses data from StatsBomb Open Data, provided free of charge for research and educational purposes. StatsBomb are the originators and owners of this data.
The 360 freeze frame format is documented in the StatsBomb Open Data specification.
MIT