[Bug] Reconcile validators on every block - #89
Merged
Conversation
The trigger list never fired: msgByType/eventByType only carry types that have an entry in MsgHandlers/EventHandlers, and of the six staking messages listed only MsgCreateValidator has one, so validators were re-synced exactly once per validator creation — on pnf mainnet the last full write was block 801233, 62k blocks behind the tip, leaving NodesHub stuck at Unstaked/90 POKT and PNF-12 at Unstaking. Reading the raw block would not fix it either, since the staking end-blocker moves validators in and out of the active set with no message at all, so the trigger list is dropped and the chain re-sync now runs every block; reconcileValidators skips the rows whose values did not change so the steady state stays one ABCI read and zero writes.
Pinning the chain read to the block height makes it the one part of block indexing that depends on the node still serving state at that height, so a pruned height during a from-genesis reindex or a transient RPC error would throw out of the block handler and stall indexing; running every block is what makes catching it safe, since the next block reconciles the same set. Also reports the "Staked in store but missing from chain" close-out refusal once per validator per process instead of once per block, and documents the flat-object invariant jsonFieldEquals depends on.
Wrapping the whole reconcile swallowed bulkCreate failures too, and those write through the block's Postgres transaction shared with every other handler, so an aborted write would surface later as an unrelated error in a sibling handler while the log claimed it was safely retried next block. The tolerance now covers only the height-pinned chain read, which is the part that legitimately fails on a pruned height or a transient RPC error. The missing-from-chain report also forgets a validator once the chain returns it again, so a later incident on the same id still reports instead of being silenced for the life of the process.
oten91
approved these changes
Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The validator chain re-sync ran behind a trigger list that could never fire, so validators were re-synced exactly once per validator creation. It now runs on every block, with a dirty check so an unchanged validator does not open a historical row.
Issue
msgsByType/eventsByTypeonly carry types registered inMsgHandlers/EventHandlers; everything else is dropped intounhandledMsgTypes. Of the six staking messages inVALIDATOR_RECONCILE_MSG_TYPESonlyMsgCreateValidatorhas a handler, and none of the three event types do, sohasStakeChangeevaluatedundefined?.length ?? 0for 8 of 9 triggers.On pnf mainnet the last full validator write was block 801233 — 62k blocks behind the tip. NodesHub sat at
Unstaked / 90 POKTagainstbonded / 6,099,991 POKTon chain, PNF-12 atUnstakingagainstbonded, and 11 of 36 live rows disagreed with the chain. This is what surfaced on the explorer and got reported by PNF.Reading the raw block would not have been enough either: the staking end-blocker moves validators in and out of the active set with no message at all, and that is exactly the transition that froze both rows.
Applications are unaffected — all 14 of their trigger types are registered, and a height-pinned dry run at 863553 showed 125/125 active apps matching the chain exactly.
Type of change
Testing
No unit tests in this repo for the mapping layer. Verified:
yarn run lint— 0 errors.jsonFieldEqualsexercised standalone over 9 cases: key order,""vs absent, null vs absent, real moniker change, commission equal/changed, field added, nil/nil, nil/value.tscwas not run:subql codegendoes not emitsrc/types/proto-interfaces/in this environment, sobuild.shaborts there. Pre-existing, identical before and after this branch; the real build runs in CI.Sanity Checklist