Add highly available background task delivery

[zzstoatzz]

closes #21218

this PR replaces process-local background-task delivery with a Docket queue so any Prefect API replica can deliver a deferred task run to any matching TaskWorker.

Implementation

• Prefect publishes the serialized task run to a keyed Docket queue during the existing deferred SCHEDULED transition.
• Each task key has durable scheduled and retry topics. All retry topics share the higher priority; FIFO is preserved within each topic.
• Docket provides queue-wide idempotency, exact topic capacity/backpressure, competing consumers, visibility renewal, abandoned-claim recovery, and atomic release to the retry topic.
• Graceful WebSocket disconnects release the unaccepted delivery immediately. A killed API replica stops renewing visibility, so another replica reclaims the same delivery after the configured timeout.
• One subscription uses a constant multi-stream reader rather than one Redis connection per task key.
• Redis acknowledgements are pipelined off the WebSocket serial path. If the durable ack fails, the stream entry remains and is redelivered at least once.
• Task.map(..., deferred=True) creates runs through one shared client with bounded concurrency (2) instead of opening a client and submitting each mapped run sequentially.

The Prefect adapter is 189 lines and has no Redis client access or Lua. Those details live behind the proposed Docket queue API in chrisguidry/docket#451.

HA and delivery guarantees

Shared Redis is the delivery source of truth, consistent with Prefect existing Docket-backed server services. Publication is awaited before the state transition returns. Claims are at least once: an API crash cannot delete an unacknowledged task, and duplicate execution remains possible if a process fails after the downstream worker accepts work but before Redis records the acknowledgement.

The cross-process reproduction starts replicas subscribed to matching and non-matching task keys, lets replica A claim a run, hard-terminates A, and starts replica B. Replica B reclaims and acknowledges the same task-run delivery; the unrelated-key replica never claims it.

Self-hosted HA uses the existing Docket Redis configuration. There is no separate background-task HA procedure.

Results and Cloud compatibility

Docket owns delivery only. Prefect still owns task execution, task-run state, parameter persistence, return-value persistence, events, and PrefectDistributedFuture.result(). Docket results are not used as Prefect task results.

The TaskWorker WebSocket protocol, task-run REST/state schemas, Task.delay, and result storage are unchanged. The mapped-submission optimization uses the existing task-run create endpoint, so this branch does not require a Prefect Cloud backend change.

The branch is temporarily pinned through an immutable source archive to Docket commit 6d05b3a415b4849e3f3464544cf45cbec36c3f5e. Before merge, that pin should become the released version containing Docket #451. Queue consumers require Redis 6.2+ for XAUTOCLAIM.

Performance

Controlled OSS setup after rebasing onto current main: clean PostgreSQL databases, one API server, two TaskWorkers (limit=100 each), warning-level logging, and 2,000 mapped deferred no-op tasks with persisted parameters/results. Main and this branch used the same Python 3.12 environment, machine, and harness.

revision	submit	submit rate	end to end	completion rate	p95 queue latency
current main (f3e3cd5a01)	44.90s	44.54/s	45.04s	44.40/s	41ms
this PR (6fa611dad7)	32.59s	61.37/s	32.62s	61.32/s	62ms

Against current main this PR reduces wall time 27.6% and raises end-to-end throughput 38.1%. All 2,000 runs completed without failure, and persisted results sampled at indices 0, 1,000, and 1,999 were correct.

An earlier one-worker paced comparison was not a win: bounded concurrent submission can outrun one worker, and the paced comparison was approximately 3% slower than main. The improvement appears when workers can consume the faster documented mapped-submission path; the single-worker caveat is intentional and should remain visible in review.

The CLI benchmark invokes syncing uv run --directory … for every sample. In a controlled Python 3.12 comparison (30 interleaved prefect --version samples each), main's released Docket measured 478.1ms mean / 474.3ms median, the source-archive pin measured 695.3ms / 716.4ms, and the identical PR commit with the identical Docket commit packaged as a local wheel measured 471.7ms / 467.0ms. uv -vv resolved the archive environment on each invocation in about 236ms versus 4ms for both wheel environments. The failing shards therefore measure direct-reference resolution overhead, not installed Prefect startup; Docket is absent from startup sys.modules, and pyinstrument has no Docket startup frames. Replacing the temporary archive pin with the released wheel removes that overhead.

Breaking changes

This removes prefect.server.task_queue, including the technically public TaskQueue and MultiQueue classes and per-key configure_task_key helper. They represented the process-local implementation that caused #21218 and are not retained as a compatibility facade. This is suitable for a minor release, but it is a real import break.

The existing global scheduled/retry queue-size settings remain and now apply to durable Docket topics. server.tasks.scheduling.delivery_visibility_timeout is additive.

Validation

• After rebasing onto current main, 40 delivery, TaskWorker subscription, WebSocket client, background execution, mapping, parameter-storage, and result-path tests pass.
• 1,860 settings tests pass (3 expected skips).
• 82 server/lifespan tests pass (4 known skips).
• Cross-process hard-kill HA reproduction passes against real Redis.
• A fresh 2,000-task controlled OSS comparison against current main completes with zero failures and correct persisted results.
• Controlled 90-sample CLI comparison isolates the benchmark delta to uv resolving the temporary archive reference; replacing only that reference with a wheel matches main.
• Prefect Ruff, formatting, mypy, lockfile, generated-settings, codespell, and inline-code hooks pass. The UI pre-commit wrapper cannot source /nvm.sh in this environment; npm run check passes directly with the installed Node 22 toolchain.
• Docket queue tests pass on Python 3.10, memory Redis, Redis 6.2, and Valkey 8.1. The refreshed Python 3.10 memory CI command passes 872 tests (12 skipped) at 100% statement and branch coverage; formatting, type completeness, Pyright, file-size, and docs gates pass.
• A clean Prefect wheel and sdist resolve the immutable Docket archive and import the queue-backed server successfully in a Git-less image.
• Rebased cleanly onto Prefect main f3e3cd5a01. The 65 intervening commits and the Prefect 3.8.1 release do not replace background-task delivery; the future-timeout and WebSocket User-Agent changes compose with this implementation. Docket main and its latest 0.23.1 release are unchanged since this branch's base.
• Fresh rebased CI has 104 passing checks. The only failures are three CLI startup shards and their aggregate, all measuring the documented temporary source-archive resolution overhead.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
prefect-oss	🟢 Ready	View Preview	Jul 23, 2026, 7:52 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

[codspeed-hq]

Merging this PR will not alter performance

✅ 8 untouched benchmarks
⏩ 1 skipped benchmark¹

---

_{Comparing codex/docket-background-task-ha (6fa611d) with main (f3e3cd5)}

Footnotes

1. 1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports. ↩