Skip to content

[Question] Correct procedure for re-IPing a Pulsar cluster with ZooKeeper and BookKeeper #4694

@Warre-Tielens

Description

@Warre-Tielens

Hello,

I’m looking for information on the correct way to re-IP an Apache Pulsar cluster using ZooKeeper and BookKeeper.

Before performing the re-IP, I wanted to make some configuration changes so that the BookieID uses <FQDN>:<port> instead of the default <IP>:<port>. To do this, I set useHostNameAsBookieID=true. However, after restarting the BookKeeper systemd service, the service fails to start and the following error appears in the logs:

2025-12-23T10:22:39,564 - ERROR [main:Main@218] - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: Cookie [5
bookieHost: “<hostname>.dummy.com:3181"
journalDir: “/data/bookkeeper/journal01"
ledgerDirs: "1\t/data/bookkeeper/ledger01"
instanceId: "9a36eaf5-564e-4b5d-9636-24b082125688"
] is not matching with [5
bookieHost: “10.0.0.1:3181"
journalDir: "/data/bookkeeper/journal01"
ledgerDirs: "1\t/data/bookkeeper/ledger01"
instanceId: "9a36eaf5-564e-4b5d-9636-24b082125688"
]

I tried the following steps to resolve the issue:

  • Stopping the Bookkeeper systemd service
  • Moving the Cookie/VERSION files from the journal01 and ledger01 directories to /tmp
  • Deleting the old cookie using:
    • bin/bookkeeper shell cookie_delete 10.0.0.1:3181
  • Running updateBookieInLedger with the old srcBookie and new destBookie
    • bin/bookkeeper shell updateBookieInLedger --srcBookie ’10.0.0.1:3181' --destBookie ‘<hostname>.dummy.com:3181' --updatespersec 5
  • Generating a new cookie and coping the created file to the VERSION files for the journal01 and ledger01 directories.
    • bin/bookkeeper shell cookie_generate -j /data/bookkeeper/journal01 -l /data/bookkeeper/ledger01 -o /tmp/test-cookie <hostname>.dummy.com:3181

None of these steps unfortunately seemed to work. The only way I was able to recover was by clearing the entire journal and ledger directories and letting the node re-sync from the other bookies.

So my main questions are:

  • What is the correct/recommended approach to re-IP Bookkeeper nodes while preserving existing ledger data?
  • Are there any additional steps required when changing useHostNameAsBookieID on an existing cluster like clearing or updating the Cookie?

Some additional context about the environment:
Pulsar version: 3.0.5
Bookkeeper version: 4.16.5
Zookeeper version: 3.8.4
openJDK version: 17.0.17
Running on RHEL8 VM's

Thanks in advance!

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions