[Bug] ESXi → Proxmox migration fails with Ceph RBD as target storage — RBD device not mapped before dd

[oetti77]

Describe the bug

When migrating a VM from ESXi to Proxmox VE with Ceph RBD as the target storage, the migration fails during the disk transfer phase. PegaProx attempts to write to /dev/rbd-pve/<uuid>/vm_nvme/vm-<vmid>-disk-0 but the RBD device is never mapped/created beforehand, so dd fails with "No such file or directory".

Steps to Reproduce

1. Add an ESXi host and a Proxmox cluster with Ceph RBD storage (vm_nvme) to PegaProx
2. Start a V2P migration job, selecting the Ceph RBD pool as target storage
3. PegaProx creates the VM shell successfully and mounts the ESXi datastore via SSHFS
4. Transfer phase begins — PegaProx tries to dd directly to /dev/rbd-pve/<uuid>/vm_nvme/vm-<vmid>-disk-0
5. Migration fails immediately

Relevant log output:

Allocated: vm_nvme:vm-105-disk-0
Target: /dev/rbd-pve/27a9b882-9c9f-498a-9b8e-2faa7c99459e/vm_nvme/vm-105-disk-0
SSH: dd: failed to open '/dev/rbd-pve/27a9b882-9c9f-498a-9b8e-2faa7c99459e/vm_nvme/vm-105-disk-0': No such file or directory
SSHFS: dd: failed to open '/dev/rbd-pve/...': No such file or directory
ALL methods failed (0 of 214748364800 bytes)
FAILED: Transfer failed for test-vm-01.vmdk

Expected behavior

PegaProx should either:

• Create and map the RBD volume (rbd create + rbd map) before attempting the dd transfer, or
• Use qemu-img with native RBD support (rbd:pool/image:conf=...) instead of relying on the kernel block device path

Environment

• PegaProx version: 0.9.12.3
• Source: VMware ESXi 7.x
• Target: Proxmox VE 9.2.2, Ceph RBD storage (20.2.1)
• Related issue: [Bug] xcp-ng -> proxmox with ceph rbd as storage is not working #272 (same root cause, different source hypervisor)

[mkellermann97]

Nailed it — thanks for the precise diagnosis. Confirmed in the code: after pvesm alloc we resolve the target with pvesm path, which hands back the /dev/rbd-pve/<uuid>/… string but never maps the kernel device, so dd opens a path that doesn't exist yet — exactly your "No such file or directory" (and #272 is the same root cause). Fixed on Testing (e1cc752): we now call pvesm activate-volume right after the path resolves, which krbd-maps the RBD image and is a harmless no-op for LVM/file storages. Full disclosure — I don't have a Ceph RBD target in my dev environment, so I verified it by inspection + that it doesn't disturb the non-RBD path, but not against live RBD; once it lands in a build, could you re-run the migration to your vm_nvme pool and confirm? If activate-volume ever isn't enough on your side, your second option (qemu-img's native rbd: path) is the fallback we'd wire in next. opencollective.com/pegaprox if PegaProx is earning its keep. — Marcus

[oetti77]

Follow-up after testing the fix from commit e1cc752 (Testing branch):

Good news: the pvesm activate-volume fix resolves the original "No such file or directory" — the RBD volume is now created and mapped before dd starts. Transfer ran successfully at ~104 MB/s over a 1 Gbit link.

However, two new issues surfaced during testing:

---

Issue 1 — dd output parsing reads first status block instead of last

When dd emits multiple progress lines (via SIGUSR1 signals or at completion), PegaProx reads the first block instead of the final one. In our case the log contained:

42207281152 bytes (42 GB, 39 GiB) copied, 402.997 s, 105 MB/s   ← PegaProx reads this
...
214748364800 bytes (215 GB, 200 GiB) copied, 2059.14 s, 104 MB/s ← actual final result

PegaProx reported only 42 GB transferred and marked the migration as failed, even though all 200 GB were fully written to the RBD volume.

---

Issue 2 — RBD volume is deleted on reported failure, even when transfer was complete

Because of the parsing bug above, PegaProx deleted the RBD volume during cleanup — despite the data being fully written. We had to manually recover:

qm set 105 --scsi0 vm_nvme:200
rbd map vm_nvme/vm-105-disk-0
dd if=/tmp/sshfs-mount/vm.vmdk of=/dev/rbd0 bs=4M status=progress

Suggestion: before deleting the volume on failure, verify actual bytes written vs. expected size — or at minimum keep the volume and mark it as "incomplete" so the operator can decide.

---

Side note: pvesm activate-volume not available on our PVE version

The call to pvesm activate-volume fails on our setup:

ERROR: unknown command 'pvesm activate-volume'

We used rbd map manually as a workaround. May be worth adding a version check or fallback to rbd map directly.

---

Overall the fix is on the right track — just needs the dd output parser to grab the last matching line instead of the first, and a safer cleanup strategy on failure.

[mkellermann97]

Brilliant debugging — you found a real one. Issue 1 is fixed on Testing (38ab1a4): all three transfer methods now read the last bytes copied line from dd instead of the first, so a finished disk is no longer mistaken for a partial one. I checked it against your exact dd log — the old parser scored 39 GiB (false fail), the new one reads the full 200 GiB (pass). That also resolves Issue 2 for your case (no false "incomplete" → no cleanup delete), and on top of it I'm adding a written-vs-expected size guard before any volume removal, so a future misread can never wipe a populated disk again.

On Issue 3 — pvesm activate-volume being an unknown command is a version gap; what does pveversion report on your nodes? I'll wire an rbd map fallback for the releases that don't carry it rather than ship a blind guess. Thanks again for the thorough write-up — opencollective.com/pegaprox if PegaProx has earned a coffee. — Marcus

[oetti77]

Great news, thanks for the quick fixes!

pveversion on our target node:

pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-7-pve)

On this version pvesm activate-volume is indeed not available — rbd map was our workaround. Looking forward to the fallback! ☕

[oetti77]

Follow-up after testing commit 38ab1a4 (Testing branch) — two new issues found:

Good news first: the dd output parser fix works perfectly — SSH OK: 50.00 GB was reported correctly for a 50 GB disk. Issue 1 and 2 from our previous comment are resolved. 🎉

---

Issue 3 (confirmed) — pvesm activate-volume not available on PVE 9.2.2

As mentioned, pvesm activate-volume is an unknown command on our node:

pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-7-pve)

Without this call, the RBD image is never mapped to a kernel block device before dd starts. The dd process reports success (writing 50 GB at 109 MB/s), but the data goes nowhere — the target path /dev/rbd-pve/<uuid>/vm_nvme/vm-106-disk-0 doesn't exist as a block device. Result: empty disk, "not a bootable disk" on first VM start.

Fix suggestion: Fall back to rbd map <pool>/<image> when pvesm activate-volume is unavailable.

---

Issue 4 — RBD image remains mapped after transfer, blocking VM start

After the dd transfer completes, the RBD image stays mapped as a kernel block device (e.g. /dev/rbd0). When QEMU then tries to open /dev/rbd-pve/<uuid>/vm_nvme/vm-106-disk-0, it fails with:

kvm: 'host_device' driver requires '/dev/rbd-pve/...' to be either a character or block device
TASK ERROR: start failed: QEMU exited with code 1

This happens because PVE's QEMU uses the symlink path under /dev/rbd-pve/ which only exists when the image is mapped via the PVE-managed krbd path — not via a raw rbd map call which creates /dev/rbd<N> instead.

Fix suggestion: After the transfer, call rbd unmap (or pvesm deactivate-volume if available) before handing off to qm start.

---

Both issues are specifically triggered by the pvesm activate-volume fallback path on PVE 9.2.2. Once the rbd map fallback lands, Issue 4 should also be addressed by ensuring the unmap happens via the correct PVE-managed path so the symlink under /dev/rbd-pve/ is properly cleaned up before VM start.

[mkellermann97]

Thanks for the version and the retest — good to hear the dd-parser fix landed cleanly (50 GB reported correctly). Issues 3 and 4 fit together neatly: on PVE 9.2.2 there's no pvesm activate-volume, and a raw rbd map hands you /dev/rbdN while QEMU expects PVE's managed /dev/rbd-pve/<uuid>/… krbd symlink — so the disk looks written but isn't reachable at boot. So the fallback has to map through PVE's krbd naming (not raw rbd0) and unmap/deactivate before qm start, exactly as you diagnosed.

I don't have a Ceph RBD target in my dev environment, so rather than ship a blind guess I'll wire that path and ping you to test it on your 9.2.2 box once it's on Testing — you've effectively been my RBD test rig here, which is hugely appreciated. opencollective.com/pegaprox if you'd like to fuel the next round. — Marcus

[oetti77]

Happy to be your RBD test rig!
Unfortunately SSH access to our environment isn't possible, but we'll test immediately once you push to Testing and report back with full logs. Take your time and do it right!

[mkellermann97]

Got the fix direction nailed. Since pvesm activate-volume doesn't exist on 9.2.2, and a raw rbd map hands you /dev/rbdN instead of the /dev/rbd-pve/<fsid>/… path QEMU expects, the clean answer is to stop using a kernel-mapped device for RBD targets entirely: I'm switching the RBD copy to qemu-img's native rbd: writer, which streams straight into the image through librbd — no rbd map, no /dev/rbd-pve dependency, so it kills Issue 3 and Issue 4 together.

To wire the exact rbd: URI for your environment (internal vs external Ceph changes how the monitors/keyring are referenced), could you paste pvesm config <your-rbd-storage> — pool / monhost / username / krbd, no secrets in there, just connection params — and confirm qemu-img is installed on the target nodes? I'll get it onto Testing right after and ping you to verify on your box. opencollective.com/pegaprox if PegaProx is saving you migration grief. — Marcus

[mkellermann97]

Ah — no SSH on your side is completely fine, ignore my last request. PegaProx already runs the migration writes on the node using its own cluster connection, so it can read the RBD storage config itself (pool / monitors / keyring) at runtime — you don't have to fetch anything. The qemu-img rbd: switch happens entirely inside the tool.

It touches a couple of the copy paths, so I'd rather wire it properly than rush a half-fix — I'll push it to Testing and ping you here. At that point the only thing I need from you is to re-run the ESXi→RBD migration from the UI and send the task log, success or failure. opencollective.com/pegaprox if PegaProx is earning its keep. — Marcus