feat: concurrency-bounded fair webhook dispatch worker

[Danielobito009]

Summary

Implements Issue #840 — Concurrency-bounded fair webhook dispatch
worker preventing unbounded socket creation during burst events.

Problem

dispatchWebhookEvent() used Promise.all() with no concurrency
limit. A burst of vault lifecycle events fanned to many subscribers
could open hundreds of simultaneous sockets, exhausting file
descriptors and starving Horizon/S3 outbound calls.

Solution

A BoundedWebhookDispatcher with configurable in-flight ceiling
and per-subscriber round-robin fair scheduling.

Files Changed

src/services/boundedWebhookDispatcher.ts (new, 390 lines)

• BoundedWebhookDispatcher class
• enqueue() — adds delivery to per-subscriber queue
• drain() — fills slots up to maxConcurrency
• nextFairDelivery() — round-robin across subscribers
• Skips OPEN circuit-breaker subscribers
• Recursive drain() call in finally for continuous throughput

src/services/webhooks.ts (modified)

• dispatchWebhookEvent() refactored 70 → 18 lines
• Enqueues to dispatcher instead of Promise.all()
• Returns immediately (background dispatch continues)
• Exports breakerCache, inFlightProbes, deadLetter

src/services/metrics.ts (modified)

• Added disciplr_webhook_dispatch_in_flight gauge
• Added disciplr_webhook_dispatch_queue_depth gauge
• Uses existing registry (no new registry created)

src/tests/webhooks.concurrency.test.ts (new, 450+ lines)

17 tests across 5 suites

docs/webhooks.md (new, 555+ lines, 16 sections)

Test Coverage (17 tests)

Suite	Tests
In-flight ceiling	4
Fair round-robin scheduling	4
Circuit-breaker integration	3
Prometheus gauge accuracy	3
Edge cases	3

Configuration

WEBHOOK_MAX_CONCURRENCY=10  # default

Prometheus Metrics

Metric	Type	Description
disciplr_webhook_dispatch_in_flight	Gauge	Current in-flight count
disciplr_webhook_dispatch_queue_depth	Gauge	Deliveries waiting

Fair Scheduling Example

Dispatch order (maxConcurrency=3):

→ Sub A e1, Sub B e1, Sub C e1

→ Sub A e2, Sub B e2

→ Sub A e3

Breaking Change

dispatchWebhookEvent() return value changed from
WebhookDeliveryResult[] to [] — delivery now happens
asynchronously in the background.

Verification

• ✅ In-flight ceiling enforced via while (inFlight < max) loop
• ✅ Round-robin advances correctly, skips empty queues
• ✅ No infinite loop — checked < total guard in nextFairDelivery
• ✅ Circuit breaker logic completely preserved
• ✅ Retry backoff preserved (3 attempts, 1s-30s, 2x, 0.25 jitter)
• ✅ Dead-letter queue behavior unchanged
• ✅ 9 existing webhook test files unaffected
• ✅ Gauges updated at correct points in lifecycle

Closes #840

[drips-wave]

@Danielobito009 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[1nonlypiece]

fair dispatch worker is in, queue should stay bounded under load now