Skip to content

Outbox processor — asyncio batch engine with backpressure #21

Description

@slusset

Parent Epic

Part of #16 — [Epic] Databricks EHR Writeback — Delta Outbox + Canonical Prediction Model

Summary

Build the async batch processor that polls the Delta outbox, dispatches predictions to EHR endpoints with bounded concurrency, and commits results back.

Dependencies

Tasks

  • Asyncio dispatcher with asyncio.Semaphore for configurable concurrency (default 10-20)
  • Batch claim → process → commit loop: claim N records, dispatch concurrently, MERGE results back
  • Retry logic: increment retry_count on failure, route to dead letter after configurable max retries
  • Graceful shutdown: finish in-flight requests before exiting
  • Integration with existing dead_letter.py for dead letter routing
  • Configurable batch size, concurrency, retry limits, lock timeout
  • New module: src/ehr_writeback/outbox/processor.py

Acceptance Criteria

  • Processor respects concurrency semaphore (never exceeds N concurrent calls)
  • Failed records are retried up to max, then dead-lettered
  • Batch results are committed atomically via Delta MERGE
  • Stale locks from crashed processors are reclaimed on next batch
  • Unit tests with mock EHR adapter

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions