Broker startup does not reconcile Helix InstanceConfig tags on restart, leading to stale broker tags and misrouting

[praveenc7]

Summary

When a Pinot broker is not shut down cleanly and later restarts (often on a different pod/host), it can come up with a different intended/actual tag than what’s stored in Helix InstanceConfig. Today the broker does not reliably converge to the intended tag on restart because the startup tag reconciliation logic only runs for new brokers. As a result, the broker can remain stuck with stale Helix tags and announce under them, this causes misrouting for us, since we rely on Helix tags for broker routing.

In updateInstanceConfigAndBrokerResourceIfNeeded() , tags are only updated from config when the broker has no tags in Helix: This means for restarts (where InstanceConfig already has tags), the broker never reconciles tags with CONFIG_OF_BROKER_INSTANCE_TAGS, so it can stay permanently incorrect until manual intervention/rebuild broker resources.

Example

Before shutdown

• Actual tag: foo
• Tag in Helix InstanceConfig: foo

After restart (pod moved)

• Actual tag: bar
• Tag in Helix InstanceConfig: foo (stale)

Result: broker may announce as foo even though it should be bar, we rely in this logic for routing so at times this results in mis-routing of queries

Proposal

On broker startup:

1. Resolve the broker’s intended/actual tag (from config/env/deployment).
2. Read Helix InstanceConfig for the broker and check the stored tag.
3. If mismatch (InstanceConfig=foo, intended=bar):
   •Update InstanceConfig to bar even if tags are non-empty (not only when empty), and do so before startup.
4. Proceed with announcement only after InstanceConfig reflects bar.

This ensures a broker always starts with the correct tag and announces only for that tag.

[Jackie-Jiang]

This comes with behavior change, and potentially confuses users. Say a broker is added with tag foo, then during operation admin decided to update the tag to bar. Next restart would reset the tag back to foo and cause issue. We can consider introducing a flag to force overriding the tag when they mismatch

[praveenc7]

Thanks @Jackie-Jiang , that's a fair point — forcing the config tag on every restart could unintentionally overwrite intentional runtime changes made by admins.

I'll update the PR to introduce a config flag (something like pinot.broker.tags.force.override.on.startup) that defaults to false to preserve current behavior. When enabled, the broker will reconcile tags from config on restart even if Helix already has tags set.