feat(module-postgres): early warning and mitigation for WAL slot invalidation during snapshot

[Sleepful]

Summary

• Detect replication slot invalidation mid-snapshot and abort early instead of completing a doomed hours-long snapshot
• Log WAL budget consumption with rate and ETA during snapshot — warn at 50% remaining
• Block futile replication retries when slot is lost during snapshot (retrying would repeat the same long snapshot)
• Surface actionable error messages with error code PSYNC_S1146 and docs link
• Expose WAL budget fields in the diagnostics API for dashboard/operator visibility

Problem

When PowerSync runs a full snapshot (triggered by sync rule changes), the replication slot can be silently invalidated if WAL growth exceeds max_slot_wal_keep_size. The operator only discovers this after the snapshot completes and streaming fails — often hours or days later. The fix is simple (increase the limit) but discovery is painful.

Changes

Mid-snapshot slot health check

After each chunk flush in snapshotTable(), query pg_replication_slots to check if the slot is still valid. If wal_status = 'lost', abort immediately with an enriched MissingReplicationSlotError carrying walStatus and phase fields.

The check hits shared memory (~1-2ms per round-trip), negligible next to per-chunk storage flush.

Conditional retry

New shouldRetryReplication() function controls whether WalStreamReplicationJob retries after slot invalidation:

Condition	Action
Slot lost during snapshot	Block retry — would repeat the same long snapshot
Slot lost during streaming	Allow retry — streaming invalidation is often transient
Slot missing	Allow retry — may have been dropped externally
invalidation_reason = rows_removed	Allow retry — not a WAL budget issue

WAL budget reporting

Time-throttled logging (every 2 min) of WAL budget during snapshot:

• Budget remaining (bytes and %)
• Consumption rate (GB/hr) computed from successive samples
• ETA to exhaustion
• Warning at 50% remaining

Computation extracted into exported pure functions (computeWalBudgetReport, formatWalBudgetLine, formatBytes, formatDuration) for testability.

Actionable error messages

Slot invalidation errors now include:

• Error code PSYNC_S1146
• Fix guidance ("Increase max_slot_wal_keep_size on the source database")
• Docs link (docs.powersync.com/self-hosting/troubleshooting/replication-slot-invalidated)
• Observed WAL budget context when available (limit, time to exhaustion)

Applied to both checkSlotHealth() (mid-snapshot) and initSlot() (between replication cycles).

Diagnostics API

Three new optional fields on the SyncRulesStatus connection object:

• wal_status — slot status from pg_replication_slots (PG 13+)
• safe_wal_size — bytes remaining before potential invalidation
• max_slot_wal_keep_size — configured limit in bytes

New optional getSlotWalBudget() on RouteAPI, implemented by PostgresRouteAPIAdapter. Non-Postgres adapters are unchanged.

Bug fix

withMaxWalSize() test helper was ignoring its size parameter and hardcoding '100MB'. Now uses the parameter.

Files changed

File	Change
modules/module-postgres/src/replication/WalStream.ts	MissingReplicationSlotError enrichment, checkSlotHealth(), WAL budget tracking, shouldRetryReplication(), formatWalBudgetContext(), pure functions
modules/module-postgres/src/replication/WalStreamReplicationJob.ts	Conditional retry wiring
modules/module-postgres/src/api/PostgresRouteAPIAdapter.ts	getSlotWalBudget() implementation
packages/service-core/src/api/RouteAPI.ts	SlotWalBudgetInfo interface, optional method
packages/service-core/src/api/diagnostics.ts	WAL budget fields in diagnostics response
packages/service-errors/src/codes.ts	PSYNC_S1146 error code
packages/types/src/definitions.ts	wal_status, safe_wal_size, max_slot_wal_keep_size fields
modules/module-postgres/test/src/wal_stream.test.ts	2 integration tests (slot loss detection + diagnostic context)
modules/module-postgres/test/src/replication_retry.test.ts	4 pure function tests for retry logic
modules/module-postgres/test/src/wal_budget.test.ts	17 pure function tests for budget computation/formatting
modules/module-postgres/test/src/wal_budget_api.test.ts	3 integration tests for diagnostics adapter
modules/module-postgres/test/src/wal_stream_utils.ts	withMaxWalSize() bug fix

Testing

• 26 new tests total (4 retry + 17 budget + 3 diagnostics API + 2 integration)
• All existing tests pass
• Pure function tests run without Postgres in milliseconds
• Integration tests use onSnapshotChunkFlushed hook for deterministic WAL generation (no race conditions)

[changeset-bot]

🦋 Changeset detected

Latest commit: aed41f0

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 19 packages

Name	Type
@powersync/service-types	Patch
@powersync/service-core	Patch
@powersync/service-module-postgres	Patch
@powersync/service-errors	Patch
@powersync/service-schema	Patch
@powersync/service-client	Patch
@powersync/lib-service-postgres	Patch
@powersync/service-module-core	Patch
@powersync/service-module-mongodb-storage	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mssql	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres-storage	Patch
@powersync/service-core-tests	Patch
@powersync/service-image	Patch
test-client	Patch
@powersync/lib-services-framework	Patch
@powersync/service-rsocket-router	Patch
@powersync/lib-service-mongodb	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[Sleepful]

Addressing review feedback

Two new commits addressing all review comments:

Error class refactor:

• walStatus and phase are now required on MissingReplicationSlotError — no more optional fields with guessed defaults
• walStatus is now a WalStatus union type ('reserved' | 'extended' | 'unreserved' | 'lost' | 'missing') instead of string
• Added invalidationReason field (passed through from PG 14+ pg_replication_slots)
• Removed SlotInvalidationContext interface — shouldRetryReplication() now takes the error object directly, eliminating the (e as any) cast and default values in WalStreamReplicationJob
• Extracted MissingReplicationSlotError.ts and wal-budget-utils.ts to reduce WalStream.ts size — barrel exports from new files in replication-index.ts

Cleanup:

• Removed docs URLs from all error messages — kept [PSYNC_S1146] error code only (matching codebase pattern)
• Throttled checkSlotHealth() itself (renamed walBudgetLogIntervalMs → slotHealthCheckIntervalMs, 2-min default). Integration tests override to 0 for per-chunk checking.
• Fixed the misleading "exhausted in" label in formatWalBudgetContext() — was showing the log interval, not an ETA. Simplified to just show the WAL limit.
  Note on initSlot() throw sites: these use phase: 'streaming' (not 'snapshot') because initSlot() runs at replication startup, not mid-snapshot. A lost slot found at startup is a pre-existing condition where retry is appropriate. Only checkSlotHealth() sets phase: 'snapshot' to block retry during active snapshots.

[Sleepful]

fwiw flaky CI test, retried twice to make it pass:

https://github.com/powersync-ja/powersync-service/actions/runs/23939262388/job/69821896741

FAIL  test/src/BinLogListener.test.ts > BinlogListener tests > Multi database events
Error: Timeout while waiting for [1] schema changes.
 ❯ waitForSchemaChanges test/src/BinLogListener.test.ts:478:13
    476|       await vi.waitFor(() => expect(eventHandler.schemaChanges.length)…
    477|     } catch (error) {
    478|       throw new Error(`Timeout while waiting for [${count}] schema cha…
       |             ^
    479|     }
    480|   }