[Cross-Layer] Contiguous-Cursor Invariant and Reorg/Gap Recovery for the Soroban Event Indexer

[ogazboiz]

Contributor coordination and questions: https://t.me/+DOylgFv1jyJlNzM0

Task Classification: Reliability and data-integrity defect (event ingestion recovery); not security-relevant
Affected Layers: backend (primary), frontend, contracts
Affected Paths: backend/src/services/eventIndexer.ts, backend/migrations/*, backend/src/services/defaultChecker.ts (consumer), frontend TanStack Query keys and SSE consumer, contracts/* event publish sites
Severity: High
Estimated Window: 48-72 hours

Technical Context & Monorepo Integration Failure

backend/src/services/eventIndexer.ts calls Soroban RPC getEvents with a startLedger and a filter, then advances a single opaque cursor (the RPC pagingToken) across pages until latestLedger is reached. The cursor is treated as sufficient durable state. Two failure classes are undetected:

• Gap: a ledger range between the last confirmed-scanned ledger and the first event of the next page is never verified as scanned. On a crash-restart, if startLedger is derived from a stale in-memory value or clamped to the RPC retention floor, the indexer resumes past a range it never read. Empty ledgers legitimately produce no events, so a jump in event ledger sequence alone cannot distinguish "empty" from "skipped."
• Rollback: Stellar reaches finality through SCP and does not fork like proof-of-work chains, but the RPC source can still present a rolled-back view. Captive-core catchup, a lagging replica swapped for a fresher one, or a node re-sync can surface different event content for ledgers already ingested. The current code re-applies whatever a page returns with no comparison against prior state.

The failure spans layers because a single-layer fix is incomplete. The backend cannot reconcile a suspect range unless each event carries stable identity (ledger_seq, tx_hash, event_index); that identity originates in the contract event envelope and must be preserved through ingestion. The frontend holds derived state (loan status, pool balances) keyed in TanStack Query; after a backend resync rewrites persistent rows, cached client state is stale until the affected keys are invalidated. A backend-only patch leaves the UI serving pre-rollback values over SSE with no re-pull signal.

Concrete drift walkthrough:

1. Indexer confirms ledgers up to 1000; ledger_checkpoints.last_contiguous_ledger = 1000.
2. The backend process restarts. startLedger is computed from an unpersisted counter and clamped to the RPC retention floor 1040.
3. getEvents returns events from ledger 1040 onward; the cursor advances normally.
4. Ledgers 1001..1039 are never scanned. A LoanDefaulted event emitted at ledger 1022 is missed.
5. defaultChecker.ts never observes the default; no NFT claim path fires; the frontend shows the loan as Active indefinitely.
6. No error is raised because the cursor is monotonic and latestLedger is satisfied.

Core Component Invariants & Code Paths

Smart Contract Infrastructure

• Crates loan_manager, lending_pool, multisig_governance, remittance_nft publish via env.events().publish. Add an application-level monotonic identifier to each event topic tuple (for example a per-contract event_seq: u64 bumped in instance storage, alongside the domain topic such as ("loan","defaulted", loan_id)).
• Ledger sequence and transaction hash are supplied by the RPC envelope (event.ledger, event.txHash, and the composite event.id encoding ledger/tx/op/event index); the contract does not set them. The added event_seq gives the backend an ordering key that is stable across a re-read of the same ledger.
• Invariant: every emitted event is uniquely addressable by (ledger_seq, tx_hash, event_index) and carries a contract-local event_seq. Typed errors (LoanError, PoolError, GovernanceError, NftError) are unaffected; no state-transition logic changes. Persistent and instance TTLs remain unchanged.

Backend/API Layer

• New table via backend/migrations (migrate:create), name ledger_checkpoints: columns id, last_contiguous_ledger BIGINT, cursor TEXT, range_start BIGINT, range_end BIGINT, range_digest TEXT, status TEXT (verified|suspect|backfilling), updated_at TIMESTAMPTZ. range_digest is a hash over the ordered (ledger_seq, tx_hash, event_index) set for a checkpointed window.
• indexed_events table gains a UNIQUE (ledger_seq, tx_hash, event_index) constraint to make re-application idempotent under INSERT ... ON CONFLICT DO NOTHING (or DO UPDATE when payload reconciliation is required).
• eventIndexer.ts refactor and invariants:
  • ingestPage(page) writes events and, per closed window, records a checkpoint. The contiguous-cursor invariant: last_contiguous_ledger only advances when every ledger from last_contiguous_ledger + 1 through the page boundary is confirmed scanned against latestLedger, not merely when the cursor moves.
  • detectGap(page) flags status='suspect' and records range_start/range_end when the confirmed scan floor exceeds last_contiguous_ledger + 1.
  • detectReorg(range) re-reads a bounded window, recomputes range_digest, and compares against the stored digest; a mismatch marks the range suspect.
  • backfillRange(start, end) performs a bounded re-read (capped window, refuse ranges below RPC retention and emit a typed IndexerError), re-applies idempotently, recomputes the digest, and advances last_contiguous_ledger only on a match.
  • On any resync, emit an SSE resync frame identifying affected contract ids and ledger range for downstream consumers.
• defaultChecker.ts reads only committed, reconciled rows; it must not act on status='suspect' windows until backfill completes.

Frontend Client

• The SSE consumer handles the new resync frame. On receipt, call queryClient.invalidateQueries for affected keys (['loans', ...], ['pool', poolId], ['events', contractId]) scoped by the reported contract ids, forcing a re-pull rather than trusting cached derived state.
• Invariant: after a resync frame, no rendered view reflects pre-rollback data; Playwright coverage asserts the re-pull. Public config unchanged (NEXT_PUBLIC_*).

Root Configuration & Orchestration

• No root workspace exists; changes remain per-layer. Extend .github/workflows/ci.yml: the migration-check job (service postgres:16) must run the new migration up and down with no residual objects; the backend job runs the reorg/gap integration suite; the frontend job runs the resync Playwright spec.
• docker-compose.yml gains a fixture RPC stub (or recorded fixtures) so integration tests can inject a gap and a rollback deterministically.

Verification & Acceptance Criteria

• Root-initiated compilation per layer: cargo build in contracts/ (overflow-checks on), npm run build and npm run typecheck in backend/ and frontend/.
• Contract unit tests assert every event topic includes event_seq and that identity (ledger_seq, tx_hash, event_index) is resolvable from the RPC envelope in a test harness.
• Backend unit tests: detectGap, detectReorg, backfillRange, digest computation, and ON CONFLICT idempotency (re-apply yields zero net row change).
• Backend integration tests against the fixture RPC: injected skipped range triggers status='suspect', backfill closes it, last_contiguous_ledger advances only after a digest match; injected rollback triggers re-read and reconciliation.
• Migration up/down verified in the migration-check job with no residual objects.
• Frontend end-to-end (Playwright): a resync SSE frame invalidates the correct query keys and the view re-pulls reconciled values.
• PR attachments: ledger_checkpoints state dump before and after an injected gap and an injected rollback; indexer logs showing the suspect to verified transition; CI run links for all three jobs.

Suggested Execution Path

• Phase 1: Contract event identity (6-10h). Add event_seq to publish sites across the four crates; extend contract tests. Deliverable: updated events plus tests. Exit check: cargo test green; identity fields present in emitted topics.
• Phase 2: Checkpoint schema (8-12h). Author the ledger_checkpoints migration and the indexed_events unique constraint. Deliverable: reversible migration. Exit check: migrate:up/migrate:down succeed against postgres:16 with no residual objects.
• Phase 3: Indexer detection and backfill (14-20h). Implement the contiguous-cursor invariant, detectGap, detectReorg, backfillRange, digesting, idempotent re-apply, and the SSE resync emit; gate defaultChecker.ts on non-suspect windows. Deliverable: refactored eventIndexer.ts with unit and integration tests. Exit check: injected gap and rollback both reconcile deterministically.
• Phase 4: Frontend resync (8-12h). Handle the resync frame; invalidate scoped query keys; add Playwright coverage. Deliverable: updated SSE consumer and spec. Exit check: e2e proves re-pull after resync.
• Phase 5: Root CI and container integration (12-18h). Wire the fixture RPC into docker-compose.yml; extend ci.yml jobs; assemble PR state dumps. Deliverable: passing CI across backend, migration-check, frontend. Exit check: all three jobs green with attachments included.

[midenotch]

I've reviewed the issue and understand the requirements. I'd love to work on this and submit a clean PR that meets the acceptance criteria.

[JClark011]

I'd love to work on this. I understand the contiguous cursor and reorg recovery requirements, and I have a clear approach to implementing them.

[apatafamilycompound123-ops]

Hi, I'd like to work on this issue. I've gone through the description and acceptance criteria and I'm confident I can deliver it. I'll make sure every acceptance criterion is met and submit a clean, tested PR. Please assign it to me.

[zeke-codes]

Hello, can i work on this issue?

[Johnsource-hub]

Hi, I'd like to work on this issue. I've gone through the description and acceptance criteria and I'm confident I can deliver it. I'll make sure every acceptance criterion is met and submit a clean, tested PR. Please assign it to me.

[tosin-zoffun]

I'd like to take this one. The core problem is that a monotonic cursor can't tell "empty" from "skipped", the pagingToken proves you consumed pages, not that you scanned every ledger in between, which is exactly how the walkthrough loses ledgers 1001 to 1039 after a restart, so the fix is a real contiguity invariant where last_contiguous_ledger only advances once every ledger since the last checkpoint is confirmed scanned and anything past that floor goes suspect instead of silently moving on. I work across all three layers this touches, Soroban and Rust for the event_seq addition in the four crates, TypeScript for the indexer refactor, and Next.js for the SSE resync handling, which matters since the identity that makes reconciliation possible starts in the contract envelope and has to survive through to the frontend's query invalidation. The checkpoint digests let detectReorg spot a rolled-back RPC view, the unique constraint on (ledger_seq, tx_hash, event_index) makes backfill idempotent and safe to retry, backfillRange stays bounded and refuses ranges below RPC retention with a typed error, and defaultChecker only acts on verified windows so the missed-default case can't happen quietly. Tests follow the issue's list, from the detection and digest unit coverage to the fixture-RPC gap and rollback runs, clean migration up and down, and the Playwright resync spec. The 48 to 72 hour window works for me, I'd put the checkpoint schema up early for review