[RFC] End-to-End RDMA Data Plane for vLLM-Omni EPD and Omni Stage Transfer

[GrayMiao123]

Changes proposed

Summary

This RFC proposes connecting the existing Mooncake transfer paths into one end-to-end disaggregated vLLM-Omni pipeline and validating it on a physical multi-node InfiniBand cluster.

The initial target is Qwen3-Omni:

Multimodal Encoder
        │ encoder embeddings
        ▼
Thinker Prefill
        │ KV cache
        ▼
Thinker Decode
        │ hidden states / embeddings
        ▼
Talker
        │ codec tensors
        ▼
Code2Wav

The proposal reuses existing infrastructure:

• Encoder → Prefill: Mooncake-backed vLLM EC connector
• Prefill → Decode: existing vLLM MooncakeConnector
• Thinker → Talker / Talker → Code2Wav: vLLM-Omni MooncakeTransferEngineConnector

The main work is to complete the missing integration, use tensor-native payloads for Qwen3-Omni stage outputs, and provide reproducible validation over real InfiniBand verbs/RDMA rather than only local or simulated deployment.

Motivation

Mooncake and vLLM-Omni already provide most of the required building blocks, but the complete EPD + Omni pipeline has not yet been integrated and validated as one system.

Related work includes:

• the broader Mooncake and vLLM-Omni collaboration roadmap:
  vllm-project/vllm-omni#2904
• the Qwen3-Omni tensor-native stage-transfer design:
  vllm-project/vllm-omni#3635
• the Mooncake-backed vLLM encoder-cache connector:
  vllm-project/vllm#40695
• the existing Qwen3-Omni Prefill–Decode disaggregation path
• Mooncake roadmap: #1883

The remaining gaps are:

1. integrate Encoder → Prefill transfer into the vLLM-Omni pipeline;
2. implement the Qwen3-Omni tensor-native payload contract from
   vllm-project/vllm-omni#3635;
3. keep large embeddings, hidden states, and codec tensors out of generic Python object serialization;
4. validate the complete pipeline on real multi-node RDMA hardware;
5. identify and upstream any generic Mooncake Transfer Engine issues exposed by the deployment.

Our team has access to a physical InfiniBand cluster and can test actual RDMA transport across machines. The team also includes @xiaohajiayou, an active vLLM-Omni contributor, which allows the model-specific integration to be developed directly against the existing vLLM-Omni execution and configuration paths.

Proposed Design

Encoder ── embeddings ──► Prefill ── KV cache ──► Decode
          Mooncake EC              MooncakeConnector

Decode ── hidden states ──► Talker ── codec tensors ──► Code2Wav
       MooncakeTransferEngineConnector

The ownership boundary remains unchanged:

vLLM / vLLM-Omni owns

• model-stage scheduling and request lifecycle;
• Qwen3-Omni tensor selection, validation, packing, and reconstruction;
• fallback behavior and model-level correctness;
• destination buffer lifecycle at the execution layer.

Mooncake owns

• registered CPU/GPU memory transfer;
• TCP and RDMA transport;
• transfer completion and error reporting;
• transport and memory-pool observability.

Large tensors must use tensor-native connector entries. Only compact, versioned metadata should travel through the control path.

A downstream stage must not resume until every required tensor transfer has completed successfully.

Scope

The initial phase contains three integration tasks.

1. Encoder → Thinker Prefill

Use the Mooncake-backed EC connector from vllm-project/vllm#40695 to run the multimodal Encoder and Thinker Prefill on different workers.

The first implementation may use pinned CPU staging. We will validate that the selected transport is actual InfiniBand verbs/RDMA and separately evaluate a GPUDirect path where the GPU–NIC topology supports it.

2. Thinker Prefill → Thinker Decode

Reuse the existing vLLM Mooncake KV-transfer path.

This RFC does not introduce another PD or KV-transfer protocol.

3. Qwen3-Omni inter-stage transfer

Implement the non-async tensor-native payload path proposed in
vllm-project/vllm-omni#3635 for:

• Thinker → Talker
• Talker → Code2Wav

Full-payload transfer is the first milestone. Async codec-chunk streaming remains a follow-up because it requires additional ordering, backpressure, and terminal-event semantics.

Real RDMA Validation

The integration will be tested on at least two physical machines connected through an InfiniBand switch.

The validation must distinguish:

• TCP;
• IP over InfiniBand;
• InfiniBand verbs/RDMA with pinned host memory;
• GPUDirect RDMA, when supported by the actual PCIe topology.

Published test results will include:

• HCA model and link rate;
• selected Mooncake protocol and RDMA device;
• ibv_devinfo / link-state evidence;
• an ib_write_bw or equivalent hardware baseline;
• GPU–NIC topology;
• Mooncake runtime logs confirming the selected transport;
• payload sizes, latency, and effective bandwidth.

The end-to-end benchmark will compare the current communication path with Mooncake RDMA for:

• Encoder → Prefill;
• Thinker → Talker;
• Talker → Code2Wav;
• complete request latency and throughput under concurrency.

This validation is intended to complement existing connector work for which public physical multi-node RDMA results are still limited.

Expected Changes

vLLM / vLLM-Omni

• connect the Mooncake EC path to the vLLM-Omni Encoder/Prefill deployment;
• add Qwen3-Omni tensor-native packing and reconstruction;
• avoid opaque Python-object transfer for large stage tensors;
• add multi-worker configuration and end-to-end tests.

Mooncake

The integration should reuse current Transfer Engine APIs where possible.

Mooncake changes should be limited to generic issues discovered during real deployment, such as:

• multi-buffer completion or fencing;
• registered-memory correctness;
• cleanup after timeout or failure;
• transport selection and observability;
• reproducible integration documentation.

Qwen3-Omni-specific payload semantics must not be added to Mooncake core.

Non-Goals

This RFC does not propose:

• a new scheduler or serving framework;
• replacing the existing Prefill–Decode connector;
• Agent State Cloning;
• Hidden State Prefix Caching;
• approximate KV reuse;
• a new Mooncake metadata service;
• automatic topology-aware placement;
• async codec streaming in the first phase.

Deliverables

1. Encoder → Prefill transfer over physical InfiniBand RDMA.
2. Reuse of the existing Mooncake PD/KV path.
3. Tensor-native Thinker → Talker full-payload transfer.
4. Tensor-native Talker → Code2Wav full-payload transfer.
5. End-to-end Qwen3-Omni multi-node demo.
6. Reproducible correctness and RDMA benchmark report.
7. Separate upstream PRs for generic Mooncake fixes and vLLM-Omni integration.

Open Questions

1. Is the current EC connector contract sufficient for Qwen3-Omni Encoder outputs?
2. Should destination buffers be connector-owned or model-runner-owned?
3. Are existing Mooncake completion APIs sufficient for publishing stage-ready metadata safely?
4. Should the first implementation PR cover only Encoder → Prefill, or Encoder → Prefill plus Thinker → Talker?

Authors

• @GrayMiao123
• @xiaohajiayou

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation

[github-actions]

Thanks for opening this issue, @GrayMiao123!

Field	Value
Issue	#2840
GitHub user ID	175080557
Reporter	@GrayMiao123

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.