Skip to content

fix: sync database storage incrementally#357

Open
yanhao98 wants to merge 1 commit into
basketikun:mainfrom
yanhao98:fix/incremental-database-storage
Open

fix: sync database storage incrementally#357
yanhao98 wants to merge 1 commit into
basketikun:mainfrom
yanhao98:fix/incremental-database-storage

Conversation

@yanhao98

Copy link
Copy Markdown

Summary

  • replace the database storage backend's delete-all/reinsert-all snapshot writes with incremental synchronization
  • preserve unchanged database rows and IDs
  • update only changed JSON payloads, insert new keys, and delete keys missing from the new snapshot
  • add regression tests covering reordered snapshots, changed rows, inserted rows, removed rows, duplicate-key rollback/redaction, and the auth-key id to key_id mapping

Why

The previous implementation rewrote every account on every save. With PostgreSQL and large token/JSON fields this creates large numbers of dead TOAST tuples. A production table with roughly 2,782 live accounts grew to approximately 109 GB, of which approximately 108 GB was TOAST, while the compacted live relation was only tens of MB.

Closes #355

Test plan

uv run --with pytest pytest -q test/test_database_storage.py

Result:

6 passed

The repository-wide pytest invocation currently also collects integration scripts that require a running server and configured auth; those pre-existing tests fail locally with connection-refused/401 errors and are unrelated to this storage change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Database storage rewrites all rows on every save, causing PostgreSQL TOAST bloat

2 participants