Skip to content

feat(scripts): add exponential backoff retry logic and DLQ for keeper (#76) - #733

Merged
orunganiekan merged 1 commit into
SiLioLabs:masterfrom
Abdullahi-Code9:feat/keeper-retry-dlq
Jul 31, 2026
Merged

feat(scripts): add exponential backoff retry logic and DLQ for keeper (#76)#733
orunganiekan merged 1 commit into
SiLioLabs:masterfrom
Abdullahi-Code9:feat/keeper-retry-dlq

Conversation

@Abdullahi-Code9

Copy link
Copy Markdown
Contributor

Closes #582

📝 Description

This PR introduces robust exponential backoff retry logic and a Dead Letter Queue (DLQ) persistent storage mechanism for the PayFlow keeper service (scripts/keeper.ts), along with a dedicated DLQ replay utility (scripts/replay-dlq.ts).

🚀 Changes Made

1. Exponential Backoff & Smart Error Categorization

  • Retry Helper: Wrapped charge execution in retryWithBackoff() with configurable max retries and exponential delay (baseMs * 2^attempt + jitter).
  • Short-Circuit Permanent Errors: Detects unrecoverable contract errors (NoSubscriptionFound, SubscriptionInactive, ContractPaused, Unauthorized) and bypasses retries immediately to avoid wasting RPC budget.

2. Dead Letter Queue (failed-batches.jsonl)

  • Pushes failed batches to an append-only JSONL file in data/dlq/failed-batches.jsonl upon exhausting all retry attempts.
  • Records full metadata: timestamp, offset, limit, error, tx_xdr, and attempts.

3. DLQ Replay CLI (scripts/replay-dlq.ts)

  • Reads failed-batches.jsonl line-by-line and attempts re-execution of transient failures.
  • Supports --dry-run to preview pending entries without submitting transactions.
  • Rotates resolved entries to replayed-batches.jsonl and moves unrecoverable entries to dead-batches.jsonl.

🧪 Verification & Testing

Type Safety Verification

Verified strict TypeScript types across all scripts:

npx tsc --noEmit

@Abdullahi-Code9
Abdullahi-Code9 force-pushed the feat/keeper-retry-dlq branch from 1bd8570 to 63ccf00 Compare July 31, 2026 12:43
@Abdullahi-Code9
Abdullahi-Code9 force-pushed the feat/keeper-retry-dlq branch from 63ccf00 to 6b34e7d Compare July 31, 2026 13:17
@orunganiekan
orunganiekan merged commit 0a673bc into SiLioLabs:master Jul 31, 2026
0 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

## Issue #76: Implement Keeper with Retry Logic and Dead-Letter Queue

2 participants