Parallel block ingest with connection-pooled RPC client - #565
Open
emersonian wants to merge 1 commit into
Open
Conversation
emersonian
force-pushed
the
be/sync-speed
branch
from
June 11, 2026 17:49
c7d677e to
b6e0c4f
Compare
This was referenced Jun 11, 2026
ValarDragon
added a commit
to valargroup/lightwalletd
that referenced
this pull request
Jun 12, 2026
…ingest Backport upstream zcash#565: Parallel block ingest with connection-pooled RPC client
BlockIngestor fetched one block at a time and issued a getbestblockhash before each one, so initial sync was bounded by round-trip latency rather than by the backend's throughput. Fetch a window of blocks concurrently (LWD_INGEST_WORKERS, default 8; LWD_INGEST_WINDOW, default 64) and commit them to the cache in strict height order, so cache and reorg semantics are unchanged. Refresh the tip only once we reach the last known tip instead of polling per block. Any error, missing block, or chain mismatch stops the batch and the outer loop re-evaluates from the (possibly rewound) next block, matching the serial path's recovery behaviour. Unit tests (rep != 0) and darkside keep the serial path. Concurrency alone was not enough: NewContextRawRequest set httpReq.Close = true, mirroring btcd's rpcclient, and configured no connection pool, so every request paid for a fresh TCP (and, with TLS, a full handshake) connection and the fetch workers serialized on connection setup rather than on the backend's actual work. Size the transport's idle/total connection pool from the ingest worker count, with headroom for frontend gRPC traffic (LWD_RPC_POOL overrides), and stop closing the connection after each request. Response bodies were already drained before Close, so connections were already eligible for reuse.
emersonian
force-pushed
the
be/sync-speed
branch
from
July 27, 2026 19:45
b6e0c4f to
f18632f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Speeds up initial sync by fetching blocks concurrently instead of one at a time.
This fix allowed us to scale out lightwalletd quickly to new nodes during the 5.0.0 upgrade last week to meet unprecedented demand without slow initial sync times, it is running in production now.