Experience and lessons learned from serving multi-stage qwen3-omni in vLLM-Omni#258
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6c50f0766c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
|
||
| ### 8. Validation Results | ||
|
|
||
| We validated the optimization stack with a controlled local benchmark on `Qwen3-Omni-30B-A3B-Instruct` (`128` prompts, concurrency `32`, `5` warmups, three GPUs mapped as `0/1/2`: Thinker on GPU 0, Talker and Code2Wav each with 2 replicas on GPUs 1 and 2). Each configuration restarted the server with an isolated deploy profile. The charts below start from the **Batch** baseline and compare Batch, CUDA Graph, Async chunk, Async output, and Stage replicas. |
There was a problem hiding this comment.
This setup sentence says the whole sweep used two Talker and two Code2Wav replicas, but the table below treats Stage replicas as a distinct final configuration. That changes the resource baseline readers use to interpret the numbers: either the first four rows were single-replica and the setup is wrong, or the replica row is not an incremental optimization. Please specify the replica count per row instead of applying the 2× layout to the entire sweep.
Useful? React with 👍 / 👎.
| --- | ||
| layout: post | ||
| title: "Qwen3-Omni in vLLM-Omni: Staged Serving for Real-Time Multimodal Generation" | ||
| author: "vLLM-Omni Team" |
There was a problem hiding this comment.
add the ant group xx team
|
|
||
| - **A three-stage pipeline:** Thinker for multimodal reasoning, Talker for speech codec generation, and Code2Wav for waveform reconstruction. | ||
| - **OpenAI-compatible serving:** `/v1/chat/completions` is the primary endpoint for Qwen3-Omni text and audio generation. | ||
| - **Async chunking:** Thinker-to-Talker and Talker-to-Code2Wav handoffs emit partial payloads instead of waiting for full-stage completion. |
There was a problem hiding this comment.
async chunking or async chunk?
There was a problem hiding this comment.
we can provide a link to the doc
| <em>Figure 7: Mean audio TTFP in milliseconds (log scale). Async chunking drops c=32 TTFP from ~4975 ms (Batch) to ~497 ms.</em> | ||
| </p> | ||
|
|
||
| | Config (c=32) | Output tok/s | Audio-s/s | Mean audio TTFP | Mean audio RTF | |
There was a problem hiding this comment.
do we have a step by step comparison in terms of througput as we did not seem too much gain in the last 2 optimizations: 1) async output 2) replicas
|
|
||
| This is the main lesson from Qwen3-Omni support in vLLM-Omni: omni-modality serving is a dataflow problem as much as a kernel problem. Efficient inference depends on carrying the right tensors across stage boundaries, preserving request-scoped async-chunk handoff state, and choosing execution policy per component. | ||
|
|
||
| ## References |
There was a problem hiding this comment.
add the ack before the reference similar to #254
amy-why-3459
changed the title
…ration Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
| #### Backend policy across all three stages | ||
|
|
||
| | Platform | Thinker / Talker | Code2Wav | | ||
| |---|---|---| | ||
| | CUDA (default) | Outer graph on (`enforce_eager: false`) | Inner graph on (`enforce_eager: false`) | | ||
| | ROCm | Outer graph on | **Eager** — `conv_transpose1d` via MIOpen is not capture-safe | | ||
| | XPU | Eager, `max_cudagraph_capture_size: 0` | Eager, `max_cudagraph_capture_size: 0` | |
There was a problem hiding this comment.
I think we don't need to mention this.
There was a problem hiding this comment.
yes, it's too detail
| | Platform | Deploy delta (from `qwen3_omni_moe.yaml`) | | ||
| |---|---| | ||
| | CUDA (default) | CUDA Graph on Thinker/Talker (outer) and Code2Wav (inner) | | ||
| | NPU | Ascend-tuned TP and device layout; Talker code predictor uses platform graphs | | ||
| | ROCm | Code2Wav runs eager — `conv_transpose1d` via MIOpen is not capture-safe | | ||
| | XPU | All stages eager with `max_cudagraph_capture_size: 0`; multi-GPU device layout | |
|
|
||
| - **`torch.compile`** fuses the 5-layer predictor forward (`dynamic=False`, `epilogue_fusion=False`) so RMSNorm/RoPE stay numerically aligned with the reference path while still reducing kernel count per step. | ||
| - On CUDA, the code predictor does **not** enable a second manual CUDA Graph layer by default (`use_cuda_graphs=False`), because that would conflict with vLLM's Talker `CUDAGraphWrapper`. The outer Talker graph and compiled inner forward are complementary: one captures the AR stage loop, the other fuses the codec-prediction micro-forward. | ||
| - On NPU, the same wrapper can fall back to platform graphs (`use_cuda_graphs=True`) where Inductor is unavailable. |
There was a problem hiding this comment.
| - On NPU, the same wrapper can fall back to platform graphs (`use_cuda_graphs=True`) where Inductor is unavailable. |
|
|
||
| **Why it works.** CUDA Graph captures a fixed operator sequence once and replays it with minimal CPU work. Each stage has a different capture point, but the principle is the same: decode shapes bucket into stable `(batch, seq, frames)` profiles, so the runtime records the graph at warmup and reuses it on the hot path. In the benchmark sweep, the **CUDA Graph** step sets `enforce_eager: false` on stages 0, 1, and 2 (the **Batch** baseline sets `enforce_eager: true` everywhere). | ||
|
|
||
| #### Stage 0 — Thinker: vLLM outer decode graph |
There was a problem hiding this comment.
Do we have an image for every stage model architectures? It would be clearer.
| #### Backend policy across all three stages | ||
|
|
||
| | Platform | Thinker / Talker | Code2Wav | | ||
| |---|---|---| | ||
| | CUDA (default) | Outer graph on (`enforce_eager: false`) | Inner graph on (`enforce_eager: false`) | | ||
| | ROCm | Outer graph on | **Eager** — `conv_transpose1d` via MIOpen is not capture-safe | | ||
| | XPU | Eager, `max_cudagraph_capture_size: 0` | Eager, `max_cudagraph_capture_size: 0` | |
There was a problem hiding this comment.
yes, it's too detail
|
|
||
| The default Qwen3-Omni-MoE deploy profile is designed for a staged layout. In the checked-in profile, stage 0 runs on one GPU, while stages 1 and 2 run on another: | ||
|
|
||
| ```yaml |
There was a problem hiding this comment.
it this the latest yaml?
| - Thinker has the multimodal prompt and deterministic text-generation profile. | ||
| - Talker has speech-generation sampling parameters. | ||
| - Code2Wav has audio reconstruction parameters and backend-specific graph policy. | ||
| - Shared-memory connectors carry structured payloads between adjacent stages. |
There was a problem hiding this comment.
do we use shared memory by default?
|
|
||
| ## What This Enables | ||
|
|
||
| With this design, vLLM-Omni can serve Qwen3-Omni as more than a monolithic checkpoint: |
There was a problem hiding this comment.
I do not understand this paragraph
|
I think we need to add a short summary section before ack |
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com> Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
4 participants
…ration