From ad7ca54379d57324e2778e8227e8647f80a01516 Mon Sep 17 00:00:00 2001 From: amy-why-3459 Date: Sat, 27 Jun 2026 17:41:41 +0800 Subject: [PATCH 1/2] Qwen3-Omni in vLLM-Omni: Staged Serving for Real-Time Multimodal Generation Signed-off-by: amy-why-3459 --- _posts/2026-06-27-qwen3-omni-optimization.md | 369 ++++++++++++++++++ .../qwen3-omni-async-replica.svg | 56 +++ .../qwen3-omni-bench-reqps.svg | 64 +++ .../qwen3-omni-bench-rtf.svg | 64 +++ .../qwen3-omni-bench-throughput.svg | 64 +++ .../qwen3-omni-bench-ttfp.svg | 58 +++ .../qwen3-omni-optimization-stack.svg | 59 +++ .../qwen3-omni-serving-flow.svg | 78 ++++ 8 files changed, 812 insertions(+) create mode 100644 _posts/2026-06-27-qwen3-omni-optimization.md create mode 100644 assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-async-replica.svg create mode 100644 assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-reqps.svg create mode 100644 assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-rtf.svg create mode 100644 assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-throughput.svg create mode 100644 assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-ttfp.svg create mode 100644 assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-optimization-stack.svg create mode 100644 assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-serving-flow.svg diff --git a/_posts/2026-06-27-qwen3-omni-optimization.md b/_posts/2026-06-27-qwen3-omni-optimization.md new file mode 100644 index 00000000..d951a5e1 --- /dev/null +++ b/_posts/2026-06-27-qwen3-omni-optimization.md @@ -0,0 +1,369 @@ +--- +layout: post +title: "Qwen3-Omni in vLLM-Omni: Staged Serving for Real-Time Multimodal Generation" +author: "vLLM-Omni Team" +summary: "How vLLM-Omni serves and optimizes Qwen3-Omni with staged Thinker-Talker-Code2Wav execution, batching, CUDA Graphs, async chunking, replicas, hot-path cleanup, and perf validation." +image: /assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-serving-flow.svg +social_image: /assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-serving-flow.svg +read_time_minutes: 12 +tags: + - performance + - multimodal + - vllm-omni +--- + +Qwen3-Omni combines multimodal understanding with speech generation. This post explains how [vLLM-Omni](https://github.com/vllm-project/vllm-omni) serves it as a staged pipeline and optimizes each stage for online workloads. + +## TL;DR + +vLLM-Omni's Qwen3-Omni serving stack includes: + +- **A three-stage pipeline:** Thinker for multimodal reasoning, Talker for speech codec generation, and Code2Wav for waveform reconstruction. +- **OpenAI-compatible serving:** `/v1/chat/completions` is the primary endpoint for Qwen3-Omni text and audio generation. +- **Async chunking:** Thinker-to-Talker and Talker-to-Code2Wav handoffs emit partial payloads instead of waiting for full-stage completion. +- **Batching, replicas, and CUDA Graphs:** stage-level batching, Talker/Code2Wav replicas, and per-stage graph capture on Thinker, Talker, and Code2Wav improve high-concurrency throughput. +- **Performance validation:** controlled benchmark sweeps and DFX perf runs show lower TTFP, lower RTF, and higher throughput as each optimization layer is enabled. + +## Quickstart + +The default Qwen3-Omni deploy profile is resolved automatically when serving the model with `--omni`: + +```bash +vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \ + --omni \ + --port 8091 \ +``` + +For explicit configuration, pass the staged deploy profile: + +```bash +vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \ + --omni \ + --port 8091 \ + --stage-configs-path vllm_omni/deploy/qwen3_omni_moe.yaml +``` + +Requests should use `/v1/chat/completions`. Set `modalities` in the request body to declare output types — e.g. `["text"]` for text only, or `["text", "audio"]` for text plus speech. + +## The Qwen3-Omni Serving Model + +Text-only LLM serving is one loop: prefill, decode, detokenize. Qwen3-Omni adds two speech stages after multimodal reasoning, each with a different compute profile: + +```text +Thinker -> multimodal understanding + text generation +Talker -> hidden states and embeddings to RVQ codec codes +Code2Wav -> codec codes to waveform audio +``` + +

+ Qwen3-Omni serving flow in vLLM-Omni, showing multimodal inputs, Thinker, Talker, Code2Wav, and text/audio outputs. +
+ Figure 1: Qwen3-Omni serving in vLLM-Omni is a staged dataflow: Thinker produces text and hidden states, Talker produces codec codes, and Code2Wav reconstructs audio. +

+ +Three details drive the serving design: + +1. **Stage handoffs carry tensors, not just token IDs.** Talker needs Thinker hidden states, prompt embeddings, and speech control tokens; Code2Wav needs codec codes and chunk metadata such as `left_context_size`. +2. **First-audio latency is a first-class metric.** Connector chunk size sets TTFP; decode left context preserves audio continuity. vLLM-Omni keeps chunk cadence (`codec_chunk_frames`, `initial_codec_chunk_frames`) separate from vocoder context (`codec_left_context_frames`). +3. **Output routing follows request modalities.** Each `/v1/chat/completions` call carries a `modalities` list that declares the desired outputs. vLLM-Omni reads it per request: `"text"` alone stops after Thinker, while `"audio"` (typically together with `"text"`) continues through Talker and Code2Wav — so the same deployed pipeline can serve text-only and speech-generation requests without separate server configs. + +| Stage | Component | Mode | Output | +|---|---|---|---| +| 0 | Thinker | `LLM_AR` | Text + latent side outputs | +| 1 | Talker | `LLM_AR` | RVQ codec codes | +| 2 | Code2Wav | `LLM_GENERATION` | Audio waveform | + +Custom processors build each handoff in full-payload or async-chunk mode; shared-memory connectors move payloads between stages. At concurrency 32, a Batch-only baseline still reaches only **35.0** output tok/s and **4975 ms** mean audio TTFP (RTF **0.98**). The sections below optimize this pipeline stage by stage. + +## Optimization Overview + +Different parts of the Qwen3-Omni pipeline hit different bottlenecks. vLLM-Omni does not apply one fixed recipe; each optimization targets a specific stage or handoff. The sections below follow the order we validated in the benchmark sweep. Each one answers three questions: **why** the problem exists, **what you gain**, and **why that gain follows** from the mechanism. + +| Technique | Target stage / path | Problem it addresses | Primary benefit | +|---|---|---|---| +| Stage decomposition | Thinker → Talker → Code2Wav | One monolithic loop cannot tune batching, graphs, or devices per sub-path | Independent runtime policy per stage | +| AR + Code2Wav batching | Talker MTP path, Code2Wav async chunks | Single-request micro-work leaves GPU underutilized at high concurrency | Higher occupancy and output tok/s | +| CUDA Graph | Thinker / Talker / Code2Wav decode paths | Repeated CPU kernel dispatch on every decode step | Lower TPOT/RTF; ~4× throughput jump in sweep | +| Async chunk + SHM connectors | Thinker→Talker, Talker→Code2Wav | Full-payload barriers inflate TTFP | Pipelined handoffs; largest TTFP reduction | +| Async omni output | Client-facing audio emission | Synchronous client I/O stalls stage workers | Throughput recovery without TTFP regression | +| Stage replicas | Talker, Code2Wav | Speech stages queue up while Thinker has headroom | Horizontal scale on bottleneck stages only | +| Hot-path cleanup | Talker code predictor, connector payloads | Per-step Python/allocation/sync tax compounds over long utterances | Lower per-step latency; stacks with all layers above | + +The rest of this section follows Qwen3-Omni through that stack in the order we validated it. + +## Qwen3-Omni: A Full Optimization Path + +Among omni-modality models in vLLM-Omni, Qwen3-Omni has the most complete three-stage speech path: Thinker for multimodal reasoning, Talker for codec generation, and Code2Wav for waveform reconstruction. + +We validated each layer with a controlled benchmark sweep at concurrency 32 (`128` prompts, `5` warmups, three GPUs mapped as `0/1/2`: Thinker on GPU 0, Talker and Code2Wav each with 2 replicas on GPUs 1 and 2). Each configuration restarted the server with an isolated deploy profile. The charts in [Validation Results](#validation-results) start from the **Batch** baseline and add one optimization at a time. + +| Step | Config added | Output tok/s | Mean audio TTFP | Mean audio RTF | +|---|---|---:|---:|---:| +| Baseline | Batch | 35.0 | 4975 ms | 0.98 | +| + CUDA Graph | Graph capture on Thinker, Talker, Code2Wav | 135.2 (+286%) | 1798 ms (−64%) | 0.38 (−61%) | +| + Async chunk | Async chunk stage handoffs | 152.5 (+13%) | 497 ms (−72%) | 0.39 | +| + Async output | Async omni output path | 137.4 | 464 ms (−7%) | 0.31 (−21%) | +| + Stage replicas | 2× Talker + 2× Code2Wav | 185.8 (+35%) | 482 ms | 0.31 | + +

+ Qwen3-Omni optimization stack in vLLM-Omni, showing staged execution, batching and replicas, CUDA Graph, async chunking, and hot-path cleanup. +
+ Figure 2: Qwen3-Omni performance comes from optimizing the staged dataflow, stage runtime, and decode hot path together. +

+ +### 1. Stage Decomposition: Making Internal Boundaries Explicit + +**Why.** Qwen3-Omni is not one homogeneous decode loop. Thinker runs multimodal AR text generation, Talker runs a codec-predictor AR path, and Code2Wav runs parallel vocoder decode. Folding all three into a single serving path forces one batching policy, one graph policy, and one device layout on workloads with very different shapes and bottlenecks. + +**What you gain.** Explicit stages let vLLM-Omni treat each component as an independent runtime: separate `max_num_seqs`, sampling params, connectors, graph/eager policy, and optional replicas. That is the prerequisite for every optimization below — without it, batching and CUDA Graph would still be constrained by whichever sub-path is slowest in a monolithic loop. + +**Why it works.** Stage boundaries turn model-internal handoffs into first-class serving objects. Connectors define what crosses each boundary (hidden states, embeddings, codec codes, chunk metadata), and the scheduler can schedule, batch, and scale each stage on its own critical path. Code2Wav can batch vocoder forwards while Talker still runs single-token decode; audio can surface from stage 2 while text still streams from stage 0; only stages 1 and 2 need replicas under concurrent speech load. + +In practice, stage 0 owns the tokenizer and multimodal prompt; stage 1 consumes Thinker hidden states and embeddings; stage 2 consumes codec codes and emits audio. The default MoE deploy profile places Thinker on one GPU and Talker/Code2Wav on another, matching their different memory and compute profiles. Without explicit boundaries, connector payloads, async-chunk handoff state, and per-stage execution policy would remain buried inside model code. + +### 2. Batching: AR Decode and Code2Wav Micro-Work + +**Why.** After stage decomposition, the speech path still spent most GPU time on single-request micro-work. Each Talker decode step runs a short code-predictor forward; each Code2Wav chunk is a small vocoder forward. At concurrency 32, processing those steps one request at a time leaves SMs idle between launches and prevents the scheduler from amortizing fixed per-step costs. + +**What you gain.** Batching raises GPU occupancy on the speech-generation side: more requests share the same Talker MTP invocation and the same Code2Wav forward. In reported tests, Code2Wav batching alone cut E2E from 9697 ms to 6435 ms and improved RTF by **62.36%** at high concurrency. Batching is the baseline layer in our sweep — without it, later optimizations have little to stack on. + +**Why it works.** Talker and Code2Wav decode steps are largely independent across requests once stage inputs are prepared. The optimized Talker runner collects single-token non-prefill requests into a decode batch, writes `input_ids`, embeddings, last Talker hidden states, and text-step embeddings into preallocated GPU buffers, and invokes the Talker MTP path once. Batched Code2Wav returns lists of audio tensors, reuses `ubatch_slices`, and runs one forward for multiple connector chunks. Fixed kernel launch and Python dispatch costs are paid once per batch instead of once per request. + +Even with batching alone, the benchmark sweep at concurrency 32 still reaches only **35.0** output tok/s, **4975 ms** mean audio TTFP, and **0.98** mean audio RTF — near realtime on the audio path. Batching improves utilization but does not remove per-step launch overhead or full-payload stage waits; the next layers address those. + +### 3. CUDA Graph: Per-Stage Decode Capture + +**Why.** Batching raised occupancy, but each decode step still paid repeated CPU-side kernel dispatch. Qwen3-Omni runs three decode-heavy stages; Talker alone may execute hundreds of short steps per utterance, and each step previously re-launched the same stable operator sequence from Python. At concurrency 32, that launch tax dominated TPOT and kept RTF above realtime even after batching. + +**What you gain.** Turning on CUDA Graph for all three stages in the benchmark sweep raises output throughput from **35.0** to **135.2** tok/s (+286%), cuts mean audio TTFP from **4975 ms** to **1798 ms**, and drops mean audio RTF from **0.98** to **0.38**. Most of the win comes from removing launch overhead across Thinker text generation, Talker codec decode, and Code2Wav vocoder forwards together — not from Code2Wav alone. + +**Why it works.** CUDA Graph captures a fixed operator sequence once and replays it with minimal CPU work. Each stage has a different capture point, but the principle is the same: decode shapes bucket into stable `(batch, seq, frames)` profiles, so the runtime records the graph at warmup and reuses it on the hot path. In the benchmark sweep, the **CUDA Graph** step sets `enforce_eager: false` on stages 0, 1, and 2 (the **Batch** baseline sets `enforce_eager: true` everywhere). + +#### Stage 0 — Thinker: vLLM outer decode graph + +The Thinker is an autoregressive multimodal stage (`LLM_AR`). When `enforce_eager` is false, it uses vLLM's standard CUDA Graph capture on the decode path — the same mechanism as text-only LLM serving. This removes repeated CPU-side kernel dispatch during long Thinker generations. + +The default deploy profile leaves stage 0 on graph-friendly settings on CUDA. Platform overrides can disable outer capture where the backend is unsafe: for example, XPU sets `enforce_eager: true` and `max_cudagraph_capture_size: 0` on stage 0. + +#### Stage 1 — Talker: outer decode graph + compiled code predictor + +The Talker stage also runs through vLLM's outer CUDA Graph path when `enforce_eager: false`. Each Talker decode step additionally invokes the **code predictor** — a short re-prefill transformer that emits RVQ codec codes. That inner path is optimized separately: + +- **`torch.compile`** fuses the 5-layer predictor forward (`dynamic=False`, `epilogue_fusion=False`) so RMSNorm/RoPE stay numerically aligned with the reference path while still reducing kernel count per step. +- On CUDA, the code predictor does **not** enable a second manual CUDA Graph layer by default (`use_cuda_graphs=False`), because that would conflict with vLLM's Talker `CUDAGraphWrapper`. The outer Talker graph and compiled inner forward are complementary: one captures the AR stage loop, the other fuses the codec-prediction micro-forward. +- On NPU, the same wrapper can fall back to platform graphs (`use_cuda_graphs=True`) where Inductor is unavailable. + +Optional prefix-graph buckets (`code_predictor_prefix_graphs` in connector config) can capture additional stable predictor shapes when explicitly enabled. + +#### Stage 2 — Code2Wav: inner vocoder graph + +Code2Wav is a generation stage (`LLM_GENERATION`), not an AR loop. Its graph path is an **inner** `CUDAGraphDecoderWrapper` rather than vLLM's outer wrapper: + +```python +# Enabled during weight load when stage enforce_eager is false +self.code2wav.enable_cudagraph( + codec_chunk_frames=chunk_frames, + codec_left_context_frames=left_frames, +) +``` + +**Shape bucketing from connector config.** Before warmup, the wrapper reads `codec_chunk_frames` and `codec_left_context_frames` from the stage connector config. Capture enumerates the `(batch, num_quantizers, frames)` buckets that async-chunk and full-payload decode will hit at runtime — including the smaller first chunk from `initial_codec_chunk_frames`. + +**Vocoder warmup.** `precompute_snake_caches()` runs before graph capture so SnakeBeta activations do not pay repeated setup inside the captured decode loop. + +**Chunk dispatch.** In async-chunk mode, `chunked_decode_streaming` delegates stable chunks to `_cudagraph_wrapper.chunked_decode_with_cudagraph`; full-payload paths use the wrapper's batched decode entry points when shapes match captured buckets. + +#### Backend policy across all three stages + +| Platform | Thinker / Talker | Code2Wav | +|---|---|---| +| CUDA (default) | Outer graph on (`enforce_eager: false`) | Inner graph on (`enforce_eager: false`) | +| ROCm | Outer graph on | **Eager** — `conv_transpose1d` via MIOpen is not capture-safe | +| XPU | Eager, `max_cudagraph_capture_size: 0` | Eager, `max_cudagraph_capture_size: 0` | + +### 4. Async Chunk: Inter-Stage Handoffs and Overlapped Transfer + +**Why.** CUDA Graph made each stage faster, but the pipeline was still **barrier-synchronized**: Talker could not start until Thinker finished, and Code2Wav could not emit audio until Talker accumulated a full payload. First-audio latency therefore tracked full Thinker generation plus full Talker prefill — even when only a few codec frames were needed to produce the first audible chunk. + +**What you gain.** Async chunking is the largest TTFP win in the sweep: mean audio TTFP drops from **1798 ms** (CUDA Graph) to **497 ms**, and from **4975 ms** (Batch alone) overall. Output throughput rises to **152.5** tok/s because pipelined handoffs keep Talker and Code2Wav busy while Thinker is still decoding. In a reported long-run benchmark, enabling async put/get cut E2E from 390.5 s to 271.2 s and mean audio TTFP from 175.2 s to 4.0 s. + +**Why it works.** Async chunking turns sequential stage completion into **pipelined partial handoffs**. Thinker emits embedding rows incrementally; Talker accumulates codec frames and slices on `initial_codec_chunk_frames` / `codec_chunk_frames` boundaries; Code2Wav decodes each slice with `codec_left_context_frames` for continuity. The async scheduler and shared-memory connector overlap chunk transfer with stage compute, so the next stage starts work while the previous stage is still decoding. + +**Connector chunk policy.** The deploy config controls output chunk cadence independently from Code2Wav decode context: + +```yaml +initial_codec_chunk_frames: 4 +codec_chunk_frames: 25 +codec_left_context_frames: 25 +``` + +The connector can emit a small first codec chunk for lower TTFP while Code2Wav keeps enough left context for audio continuity across chunk boundaries. + +**Incremental handoffs instead of full replay.** On the worker connector path, Thinker→Talker async chunking tracks `put_req_chunk` plus pending chunk-0 prefill state; Talker→Code2Wav async chunking accumulates codec rows in `code_prompt_token_ids` and slices them using `codec_chunk_frames`, `initial_codec_chunk_frames`, and `codec_left_context_frames`. + +**Transfer overlap.** The async scheduler and shared-memory connector overlap chunk IO with stage compute. Large payloads use key-addressed POSIX SHM; small payloads can inline to avoid mmap and file-lock overhead. Pending keys are tracked and unlinked on request cleanup so aborted async chunks do not leak `/dev/shm` segments. + +### 5. Async Output: Non-Blocking Client Emission + +**Why.** Async chunking speeds inter-stage handoffs, but the **client-facing emission path** can still stall stage workers. If Code2Wav must synchronously serialize, buffer, and push each audio chunk through the HTTP serving loop before starting the next decode step, GPU time is lost to I/O and Python scheduling even though the stage-to-stage pipeline is already incremental. + +**What you gain.** On top of async chunking in the benchmark sweep, async output keeps mean audio TTFP near **464 ms** while lowering RTF from **0.39** to **0.31** at concurrency 32 — decoupling client emission removes scheduling stalls without giving back the TTFP win from async chunking. + +**Why it works.** `use_async_omni_output` decouples chunk production from chunk delivery. Code2Wav can finish a vocoder forward and hand audio to a non-blocking output path while Talker and the connector keep advancing. The serving loop surfaces packets to `/v1/chat/completions` asynchronously instead of making decode workers wait on client I/O. + +### 6. Stage Replicas: Scaling Talker and Code2Wav + +**Why.** Under concurrent speech generation, Thinker, Talker, and Code2Wav do not saturate equally. Thinker handles one multimodal prompt per request; Talker and Code2Wav execute many short decode steps and vocoder forwards per utterance. At concurrency 32, a single Talker/Code2Wav instance becomes the tail bottleneck even when Thinker still has headroom — replicating the entire pipeline would waste memory duplicating the multimodal stage. + +**What you gain.** Adding replicas on top of async output reaches **185.8** output tok/s (+35%) at concurrency 32 — the highest throughput in the sweep — while keeping mean audio TTFP near **482 ms** and RTF near **0.31**. The tracked multi-replica perf suite separately targets **2100 ms** mean audio TTFP and **0.39** mean audio RTF for long text-plus-audio workloads at concurrency 32. + +**Why it works.** Stage replication is horizontal scaling applied only where queue depth builds up. One Thinker on GPU 0 fans out to two Talker replicas on GPUs 1 and 2; two Code2Wav replicas share the same pair of GPUs. Load spreads across the speech-generation side without copying the full three-stage pipeline per replica. The benchmark deploy config keeps one Thinker and runs two replicas each for Talker and Code2Wav: + +```json +{ + "stage_overrides": { + "1": {"num_replicas": 2, "devices": "1,2"}, + "2": {"num_replicas": 2, "devices": "1,2"} + }, + "extra_cli_args": ["--async-chunk"] +} +``` + +

+ Qwen3-Omni async chunk and stage replica serving, showing one Thinker feeding two Talker replicas and two Code2Wav replicas. +
+ Figure 3: Async chunking and stage replicas target the speech-generation side of the pipeline, where Talker and Code2Wav can become the bottleneck under concurrent load. +

+ +### 7. Hot-Path Cleanup: Talker Decode and Connector Payloads + +**Why.** Batching, graphs, async chunking, and replicas address structural bottlenecks, but the Talker decode loop still executed hundreds of small per-step costs: Python `generate()` dispatch, repeated `torch.cat` while building connector payloads, and device-to-host reads of decode state that the next step immediately needed back on GPU. + +**What you gain.** Hot-path cleanup removes overhead that scales with utterance length. In a long-context single-request test, E2E dropped from 21.28 s to 7.37 s, audio TTFP from 3197 ms to 1796 ms, and audio RTF from 0.71 to 0.28. These changes stack with the layers above and show up in the DFX perf suite baselines rather than as a separate sweep row. + +**Why it works.** Each fix targets a repeated micro-cost on the critical path: + +**Connector payload construction.** Large async chunks previously paid for repeated `torch.cat` while accumulating Thinker and Talker payloads. Passing decode embeddings per token and trimming redundant payload assembly removes allocation and copy work on every chunk boundary. + +**Talker code predictor rewrite.** The older path used Hugging Face `generate()` for very short codec-predictor sequences. That added Python dispatch, dynamic allocation, and KV-cache overhead on every Talker decode step. The optimized path uses re-prefill with SDPA, native GQA, inline top-k sampling, cached module references, and `torch.compile` on the inner transformer (`epilogue_fusion=False` for fp32-safe RMSNorm/RoPE). On CUDA this compile path sits below vLLM's Talker CUDA Graph rather than adding a conflicting second graph layer; see §3 above. + +**GPU-resident decode state.** Intermediate keys such as `hidden_states.last`, `hidden_states.trailing_text`, `embed.tts_pad_projected`, and `codes.audio` stay in `model_intermediate_buffer` so the next decode step reuses GPU-resident tensors instead of forcing device-to-host synchronization — eliminating round trips that would otherwise serialize every Talker step. + +**Numerical precision on the code predictor.** Where audio quality is sensitive, RMSNorm variance and RoPE stay in fp32, attention uses SDPA with native GQA, and per-call embedding buffers avoid cross-request aliasing. Qwen3-Omni configures the wrapper with parallel embedding, stored sampling, and returned projection buffers so Talker can feed the next decode step without rebuilding state on CPU. The compile and graph layers (§3) then optimize this cleaned-up path instead of fighting legacy `generate()` overhead. + +These hot-path changes stack with the batching, graph, and async-chunk optimizations above. Qwen3-Omni performance tests in the DFX perf suite — text-only serving and async-chunk workloads — give tracked baselines instead of one-off local numbers. + +### 8. Validation Results + +We validated the optimization stack with a controlled local benchmark on `Qwen3-Omni-30B-A3B-Instruct` (`128` prompts, concurrency `32`, `5` warmups, three GPUs mapped as `0/1/2`: Thinker on GPU 0, Talker and Code2Wav each with 2 replicas on GPUs 1 and 2). Each configuration restarted the server with an isolated deploy profile. The charts below start from the **Batch** baseline and compare Batch, CUDA Graph, Async chunk, Async output, and Stage replicas. + +

+ Qwen3-Omni output throughput at concurrency 1 vs 32 across optimization configs from Batch to Replicas. +
+ Figure 4: Output throughput (tok/s) at concurrency 1 (orange) vs concurrency 32 (green). CUDA Graph, async output, and stage replicas raise c=32 output tok/s from 35.0 (Batch) to 185.8 (Replicas). +

+ +

+ Qwen3-Omni request throughput at concurrency 1 vs 32 across optimization configs from Batch to Replicas. +
+ Figure 5: Request throughput (req/s) at concurrency 1 (orange) vs concurrency 32 (green). Stage replicas reach 11.1 req/s at c=32, up from 2.0 req/s (Batch). +

+ +

+ Qwen3-Omni mean audio RTF at concurrency 1 vs 32 across optimization configs from Batch to Replicas. +
+ Figure 6: Mean audio RTF at concurrency 1 (orange) vs concurrency 32 (green). Batch alone stays near realtime at c=32 (RTF 0.98); async chunking and replicas keep RTF at or below ~0.40. +

+ +

+ Qwen3-Omni mean audio TTFP at concurrency 1 vs 32 across optimization configs from Batch to Replicas. +
+ Figure 7: Mean audio TTFP in milliseconds (log scale). Async chunking drops c=32 TTFP from ~4975 ms (Batch) to ~497 ms. +

+ +| Config (c=32) | Output tok/s | Audio-s/s | Mean audio TTFP | Mean audio RTF | +|---|---:|---:|---:|---:| +| Batch | 35.0 | 10.8 | 4975 ms | 0.98 | +| CUDA Graph | 135.2 | 40.9 | 1798 ms | 0.38 | +| Async chunk | 152.5 | 45.9 | 497 ms | 0.39 | +| Async output | 137.4 | 42.4 | 464 ms | 0.31 | +| Stage replicas | 185.8 | 57.1 | 482 ms | 0.31 | + +Stage replicas deliver the best throughput on this sweep: **185.8** output tok/s at concurrency 32, with TTFP and RTF staying in the same **~460–500 ms / ~0.31** band as async output. Async chunking is the largest latency win over Batch alone, cutting TTFP from **4975 ms** to **497 ms** and RTF from **0.98** to **0.39**. + +The following results come from the Qwen3-Omni DFX perf artifacts for `Qwen/Qwen3-Omni-30B-A3B-Instruct` using the `openai-chat-omni` backend. They reflect the accumulated optimization stack above. The percentages compare the measured run against the tracked perf-suite baseline for the same workload. Lower is better for TTFT, E2EL, TTFP, and audio RTF; higher is better for throughput. + +| Workload | Key result | Change vs. baseline | +|---|---:|---:| +| Long text + audio output, 128 prompts, concurrency 32, 2500 input / 900 output tokens | 80.43 audio-s/s, 0.399 mean audio RTF, 1065 ms mean audio TTFP | TTFT -20.3%, E2EL -26.3%, audio RTF -29.1% | +| Audio-input multimodal, 10 prompts at 0.1 req/s | 3.15 audio-s/s, 0.117 mean audio RTF, 396 ms mean audio TTFP | TTFT -12.7%, E2EL -11.6%, audio RTF -10.3% | +| Image + video multimodal, 40 prompts at 0.5 req/s | 13.74 audio-s/s, 0.146 mean audio RTF, 332 ms mean audio TTFP | TTFT -9.2%, E2EL -24.0%, audio RTF -19.8% | +| Image + video + audio multimodal, 100 prompts at 1.0 req/s | 29.07 audio-s/s, 0.186 mean audio RTF, 553 ms mean audio TTFP | TTFT -14.3%, E2EL -34.5%, audio RTF -31.4% | + +The same infrastructure also benefits text-only routing through Qwen3-Omni. In the text-only long-output workload with 128 prompts at concurrency 32, the measured run reached 2,032 output tok/s and 7,698 total tok/s, while mean E2EL dropped from the tracked 18.15 s baseline to 14.15 s (-22.1%). + +The multi-replica perf suite separately validates the deployment shape where stages 1 and 2 each run two replicas. That suite tracks long text-plus-audio serving at concurrency 8, 16, 24, and 32, with baseline targets for TTFT, TPOT, audio TTFP, and audio RTF. The Stage replicas row in the benchmark sweep above shows the same scaling idea in practice: replicating Talker and Code2Wav improves throughput and latency without duplicating the Thinker stage. + +The important result is that the gains show up across different request shapes: long text-plus-audio generation, audio input, image/video input, and mixed image/video/audio input. That is what we want from a pipeline optimization. It improves the common dataflow instead of only helping one narrow prompt pattern. + +## Deployment Profile + +The default Qwen3-Omni-MoE deploy profile is designed for a staged layout. In the checked-in profile, stage 0 runs on one GPU, while stages 1 and 2 run on another: + +```yaml +stages: + - stage_id: 0 + devices: "0" + max_num_seqs: 64 + default_sampling_params: + temperature: 0.0 + max_tokens: 2048 + + - stage_id: 1 + devices: "1" + max_num_seqs: 64 + input_connectors: + from_stage_0: connector_of_shared_memory + default_sampling_params: + temperature: 0.9 + top_k: 50 + max_tokens: 4096 + + - stage_id: 2 + devices: "1" + max_num_seqs: 64 + input_connectors: + from_stage_1: connector_of_shared_memory + default_sampling_params: + temperature: 0.0 + max_tokens: 65536 +``` + +The exact device assignment is not a universal recommendation. The useful part is the separation of concerns: + +- Thinker has the multimodal prompt and deterministic text-generation profile. +- Talker has speech-generation sampling parameters. +- Code2Wav has audio reconstruction parameters and backend-specific graph policy. +- Shared-memory connectors carry structured payloads between adjacent stages. + +## What This Enables + +With this design, vLLM-Omni can serve Qwen3-Omni as more than a monolithic checkpoint: + +- Per-request `modalities` control how far the pipeline runs: text-only requests finish at Thinker; adding `"audio"` activates Talker and Code2Wav. +- Chat serving can receive text while audio serving consumes Code2Wav output metadata such as sample rate. +- Platform-specific deployment differences can live in YAML rather than model branches. + +This is the main lesson from Qwen3-Omni support in vLLM-Omni: omni-modality serving is a dataflow problem as much as a kernel problem. Efficient inference depends on carrying the right tensors across stage boundaries, preserving request-scoped async-chunk handoff state, and choosing execution policy per component. + +## References + +- Qwen3-Omni pipeline topology in vLLM-Omni: [`pipeline.py`](https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/model_executor/models/qwen3_omni/pipeline.py) +- Qwen3-Omni model wrapper in vLLM-Omni: [`qwen3_omni.py`](https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/model_executor/models/qwen3_omni/qwen3_omni.py) +- Qwen3-Omni stage input processors: [`stage_input_processors/qwen3_omni.py`](https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/model_executor/stage_input_processors/qwen3_omni.py) +- Qwen3-Omni deploy profile: [`qwen3_omni_moe.yaml`](https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/deploy/qwen3_omni_moe.yaml) +- Qwen3-Omni async-chunk perf config: [`test_qwen3_omni_async_chunk.json`](https://github.com/vllm-project/vllm-omni/blob/main/tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json) +- Qwen3-Omni multi-replica perf config: [`test_qwen3_omni_multi_replicas.json`](https://github.com/vllm-project/vllm-omni/blob/main/tests/dfx/perf/tests/test_qwen3_omni_multi_replicas.json) +- Qwen3-Omni model repository: [Qwen/Qwen3-Omni-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct) + +If you are interested in Qwen3-Omni serving or omni-modality inference, join the `#vllm-omni` channel in [vLLM Slack](https://slack.vllm.ai), or open an issue in [vLLM-Omni GitHub](https://github.com/vllm-project/vllm-omni). diff --git a/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-async-replica.svg b/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-async-replica.svg new file mode 100644 index 00000000..e1fc4249 --- /dev/null +++ b/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-async-replica.svg @@ -0,0 +1,56 @@ + + Qwen3-Omni async chunk and stage replicas + One Thinker feeding two Talker replicas and two Code2Wav replicas. + + + + + + + + + Async chunk + stage replicas + + + Thinker + text + hidden states + + + Talker R0 + codec generation + + + Talker R1 + codec generation + + + Code2Wav R0 + waveform chunks + + + Code2Wav R1 + waveform chunks + + + Audio + streaming output + + + + + + + + + diff --git a/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-reqps.svg b/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-reqps.svg new file mode 100644 index 00000000..c12736e4 --- /dev/null +++ b/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-reqps.svg @@ -0,0 +1,64 @@ + +Request throughput (req/s) across optimization configs + + +Request throughput (req/s) across optimization configs + +c=1 + +c=32 + + + +0.0 + +3.2 + +6.4 + +9.5 + +12.7 + + +0.4 + +1.8 + +2.0 + +2.3 + +2.4 + + +2.0 + +7.9 + +9.0 + +7.8 + +11.1 +Batch +CUDA +Graph +Async +chunk +Async +output +Replicas + diff --git a/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-rtf.svg b/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-rtf.svg new file mode 100644 index 00000000..e712183a --- /dev/null +++ b/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-rtf.svg @@ -0,0 +1,64 @@ + +Mean audio RTF across optimization configs + + +Mean audio RTF across optimization configs + +c=1 + +c=32 + + + +0.00 + +0.28 + +0.56 + +0.84 + +1.12 + + +0.85 + +0.17 + +0.16 + +0.14 + +0.14 + + +0.98 + +0.38 + +0.39 + +0.31 + +0.31 +Batch +CUDA +Graph +Async +chunk +Async +output +Replicas + diff --git a/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-throughput.svg b/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-throughput.svg new file mode 100644 index 00000000..1d29eab2 --- /dev/null +++ b/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-throughput.svg @@ -0,0 +1,64 @@ + +Output throughput (tok/s) across optimization configs + + +Output throughput (tok/s) across optimization configs + +c=1 + +c=32 + + + +0.0 + +53.4 + +107 + +160 + +214 + + +4.9 + +22.5 + +25.1 + +28.4 + +30.0 + + +35.0 + +135 + +152 + +137 + +186 +Batch +CUDA +Graph +Async +chunk +Async +output +Replicas + diff --git a/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-ttfp.svg b/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-ttfp.svg new file mode 100644 index 00000000..97fea100 --- /dev/null +++ b/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-ttfp.svg @@ -0,0 +1,58 @@ + +Mean audio TTFP (ms) across optimization configs + + +Mean audio TTFP (ms) across optimization configs + +c=1 + +c=32 + + + +100 ms + +1000 ms + + +2556 ms + +552 ms + +129 ms + +124 ms + +128 ms + + +4975 ms + +1798 ms + +497 ms + +464 ms + +482 ms +Batch +CUDA +Graph +Async +chunk +Async +output +Replicas + diff --git a/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-optimization-stack.svg b/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-optimization-stack.svg new file mode 100644 index 00000000..843b6223 --- /dev/null +++ b/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-optimization-stack.svg @@ -0,0 +1,59 @@ + + Qwen3-Omni optimization stack in vLLM-Omni + Optimization areas for Qwen3-Omni staged serving in vLLM-Omni. + + + + + + + + + Qwen3-Omni optimization stack + + + Stage split + Thinker / Talker / C2W + + + Batch + AR + Code2Wav + + + CUDA Graph + stable chunks + + + Async chunk + overlap IO + compute + + + Async output + async omni output + + + Replicas + scale Talker / C2W + + + Hot-path cleanup + GPU-resident decode state + + + + + + + + diff --git a/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-serving-flow.svg b/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-serving-flow.svg new file mode 100644 index 00000000..48614f74 --- /dev/null +++ b/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-serving-flow.svg @@ -0,0 +1,78 @@ + + Qwen3-Omni three-stage serving flow in vLLM-Omni + Text, image, audio, and video inputs feed Thinker, then Talker and Code2Wav produce text and audio outputs. + + + + + + + + + Qwen3-Omni serving flow + + + + Text + + + Image + + + Audio + + + Video + + + + Stage 0 + Thinker + reasoning + text + + + Stage 1 + Talker + RVQ codec codes + + + Stage 2 + Code2Wav + waveform audio + + + + Text + + + Audio + + + + + + + + + + + + + + + From e6e3edf4ad7278775df104f90c4908b54ac47bae Mon Sep 17 00:00:00 2001 From: amy-why-3459 Date: Tue, 30 Jun 2026 08:55:08 +0800 Subject: [PATCH 2/2] Apply suggestions from code review Co-authored-by: Hongsheng Liu Signed-off-by: amy-why-3459 --- _posts/2026-06-27-qwen3-omni-optimization.md | 369 ------------------ _posts/2026-07-01-qwen3-omni-optimization.md | 284 ++++++++++++++ .../qwen3-omni-async-replica.svg | 56 --- .../qwen3-omni-bench-reqps.svg | 64 --- .../qwen3-omni-bench-rtf.svg | 64 --- .../qwen3-omni-bench-throughput.svg | 64 --- .../qwen3-omni-bench-ttfp.svg | 58 --- .../qwen3-omni-optimization-stack.svg | 59 --- .../qwen3-omni-async-chunk-timeline.svg | 141 +++++++ .../qwen3-omni-async-output-step-gap.svg | 121 ++++++ .../qwen3-omni-async-replica.svg | 69 ++++ .../qwen3-omni-bench-reqps.svg | 124 ++++++ .../qwen3-omni-bench-rtf.svg | 121 ++++++ .../qwen3-omni-bench-ttfp.svg | 121 ++++++ .../qwen3-omni-cuda-graph-stages.svg | 83 ++++ .../qwen3-omni-optimization-stack.svg | 59 +++ .../qwen3-omni-serving-flow.svg | 0 17 files changed, 1123 insertions(+), 734 deletions(-) delete mode 100644 _posts/2026-06-27-qwen3-omni-optimization.md create mode 100644 _posts/2026-07-01-qwen3-omni-optimization.md delete mode 100644 assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-async-replica.svg delete mode 100644 assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-reqps.svg delete mode 100644 assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-rtf.svg delete mode 100644 assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-throughput.svg delete mode 100644 assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-ttfp.svg delete mode 100644 assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-optimization-stack.svg create mode 100644 assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-async-chunk-timeline.svg create mode 100644 assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-async-output-step-gap.svg create mode 100644 assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-async-replica.svg create mode 100644 assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-bench-reqps.svg create mode 100644 assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-bench-rtf.svg create mode 100644 assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-bench-ttfp.svg create mode 100644 assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-cuda-graph-stages.svg create mode 100644 assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-optimization-stack.svg rename assets/figures/{2026-06-27-qwen3-omni-optimization => 2026-07-01-qwen3-omni-optimization}/qwen3-omni-serving-flow.svg (100%) diff --git a/_posts/2026-06-27-qwen3-omni-optimization.md b/_posts/2026-06-27-qwen3-omni-optimization.md deleted file mode 100644 index d951a5e1..00000000 --- a/_posts/2026-06-27-qwen3-omni-optimization.md +++ /dev/null @@ -1,369 +0,0 @@ ---- -layout: post -title: "Qwen3-Omni in vLLM-Omni: Staged Serving for Real-Time Multimodal Generation" -author: "vLLM-Omni Team" -summary: "How vLLM-Omni serves and optimizes Qwen3-Omni with staged Thinker-Talker-Code2Wav execution, batching, CUDA Graphs, async chunking, replicas, hot-path cleanup, and perf validation." -image: /assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-serving-flow.svg -social_image: /assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-serving-flow.svg -read_time_minutes: 12 -tags: - - performance - - multimodal - - vllm-omni ---- - -Qwen3-Omni combines multimodal understanding with speech generation. This post explains how [vLLM-Omni](https://github.com/vllm-project/vllm-omni) serves it as a staged pipeline and optimizes each stage for online workloads. - -## TL;DR - -vLLM-Omni's Qwen3-Omni serving stack includes: - -- **A three-stage pipeline:** Thinker for multimodal reasoning, Talker for speech codec generation, and Code2Wav for waveform reconstruction. -- **OpenAI-compatible serving:** `/v1/chat/completions` is the primary endpoint for Qwen3-Omni text and audio generation. -- **Async chunking:** Thinker-to-Talker and Talker-to-Code2Wav handoffs emit partial payloads instead of waiting for full-stage completion. -- **Batching, replicas, and CUDA Graphs:** stage-level batching, Talker/Code2Wav replicas, and per-stage graph capture on Thinker, Talker, and Code2Wav improve high-concurrency throughput. -- **Performance validation:** controlled benchmark sweeps and DFX perf runs show lower TTFP, lower RTF, and higher throughput as each optimization layer is enabled. - -## Quickstart - -The default Qwen3-Omni deploy profile is resolved automatically when serving the model with `--omni`: - -```bash -vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \ - --omni \ - --port 8091 \ -``` - -For explicit configuration, pass the staged deploy profile: - -```bash -vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \ - --omni \ - --port 8091 \ - --stage-configs-path vllm_omni/deploy/qwen3_omni_moe.yaml -``` - -Requests should use `/v1/chat/completions`. Set `modalities` in the request body to declare output types — e.g. `["text"]` for text only, or `["text", "audio"]` for text plus speech. - -## The Qwen3-Omni Serving Model - -Text-only LLM serving is one loop: prefill, decode, detokenize. Qwen3-Omni adds two speech stages after multimodal reasoning, each with a different compute profile: - -```text -Thinker -> multimodal understanding + text generation -Talker -> hidden states and embeddings to RVQ codec codes -Code2Wav -> codec codes to waveform audio -``` - -

- Qwen3-Omni serving flow in vLLM-Omni, showing multimodal inputs, Thinker, Talker, Code2Wav, and text/audio outputs. -
- Figure 1: Qwen3-Omni serving in vLLM-Omni is a staged dataflow: Thinker produces text and hidden states, Talker produces codec codes, and Code2Wav reconstructs audio. -

- -Three details drive the serving design: - -1. **Stage handoffs carry tensors, not just token IDs.** Talker needs Thinker hidden states, prompt embeddings, and speech control tokens; Code2Wav needs codec codes and chunk metadata such as `left_context_size`. -2. **First-audio latency is a first-class metric.** Connector chunk size sets TTFP; decode left context preserves audio continuity. vLLM-Omni keeps chunk cadence (`codec_chunk_frames`, `initial_codec_chunk_frames`) separate from vocoder context (`codec_left_context_frames`). -3. **Output routing follows request modalities.** Each `/v1/chat/completions` call carries a `modalities` list that declares the desired outputs. vLLM-Omni reads it per request: `"text"` alone stops after Thinker, while `"audio"` (typically together with `"text"`) continues through Talker and Code2Wav — so the same deployed pipeline can serve text-only and speech-generation requests without separate server configs. - -| Stage | Component | Mode | Output | -|---|---|---|---| -| 0 | Thinker | `LLM_AR` | Text + latent side outputs | -| 1 | Talker | `LLM_AR` | RVQ codec codes | -| 2 | Code2Wav | `LLM_GENERATION` | Audio waveform | - -Custom processors build each handoff in full-payload or async-chunk mode; shared-memory connectors move payloads between stages. At concurrency 32, a Batch-only baseline still reaches only **35.0** output tok/s and **4975 ms** mean audio TTFP (RTF **0.98**). The sections below optimize this pipeline stage by stage. - -## Optimization Overview - -Different parts of the Qwen3-Omni pipeline hit different bottlenecks. vLLM-Omni does not apply one fixed recipe; each optimization targets a specific stage or handoff. The sections below follow the order we validated in the benchmark sweep. Each one answers three questions: **why** the problem exists, **what you gain**, and **why that gain follows** from the mechanism. - -| Technique | Target stage / path | Problem it addresses | Primary benefit | -|---|---|---|---| -| Stage decomposition | Thinker → Talker → Code2Wav | One monolithic loop cannot tune batching, graphs, or devices per sub-path | Independent runtime policy per stage | -| AR + Code2Wav batching | Talker MTP path, Code2Wav async chunks | Single-request micro-work leaves GPU underutilized at high concurrency | Higher occupancy and output tok/s | -| CUDA Graph | Thinker / Talker / Code2Wav decode paths | Repeated CPU kernel dispatch on every decode step | Lower TPOT/RTF; ~4× throughput jump in sweep | -| Async chunk + SHM connectors | Thinker→Talker, Talker→Code2Wav | Full-payload barriers inflate TTFP | Pipelined handoffs; largest TTFP reduction | -| Async omni output | Client-facing audio emission | Synchronous client I/O stalls stage workers | Throughput recovery without TTFP regression | -| Stage replicas | Talker, Code2Wav | Speech stages queue up while Thinker has headroom | Horizontal scale on bottleneck stages only | -| Hot-path cleanup | Talker code predictor, connector payloads | Per-step Python/allocation/sync tax compounds over long utterances | Lower per-step latency; stacks with all layers above | - -The rest of this section follows Qwen3-Omni through that stack in the order we validated it. - -## Qwen3-Omni: A Full Optimization Path - -Among omni-modality models in vLLM-Omni, Qwen3-Omni has the most complete three-stage speech path: Thinker for multimodal reasoning, Talker for codec generation, and Code2Wav for waveform reconstruction. - -We validated each layer with a controlled benchmark sweep at concurrency 32 (`128` prompts, `5` warmups, three GPUs mapped as `0/1/2`: Thinker on GPU 0, Talker and Code2Wav each with 2 replicas on GPUs 1 and 2). Each configuration restarted the server with an isolated deploy profile. The charts in [Validation Results](#validation-results) start from the **Batch** baseline and add one optimization at a time. - -| Step | Config added | Output tok/s | Mean audio TTFP | Mean audio RTF | -|---|---|---:|---:|---:| -| Baseline | Batch | 35.0 | 4975 ms | 0.98 | -| + CUDA Graph | Graph capture on Thinker, Talker, Code2Wav | 135.2 (+286%) | 1798 ms (−64%) | 0.38 (−61%) | -| + Async chunk | Async chunk stage handoffs | 152.5 (+13%) | 497 ms (−72%) | 0.39 | -| + Async output | Async omni output path | 137.4 | 464 ms (−7%) | 0.31 (−21%) | -| + Stage replicas | 2× Talker + 2× Code2Wav | 185.8 (+35%) | 482 ms | 0.31 | - -

- Qwen3-Omni optimization stack in vLLM-Omni, showing staged execution, batching and replicas, CUDA Graph, async chunking, and hot-path cleanup. -
- Figure 2: Qwen3-Omni performance comes from optimizing the staged dataflow, stage runtime, and decode hot path together. -

- -### 1. Stage Decomposition: Making Internal Boundaries Explicit - -**Why.** Qwen3-Omni is not one homogeneous decode loop. Thinker runs multimodal AR text generation, Talker runs a codec-predictor AR path, and Code2Wav runs parallel vocoder decode. Folding all three into a single serving path forces one batching policy, one graph policy, and one device layout on workloads with very different shapes and bottlenecks. - -**What you gain.** Explicit stages let vLLM-Omni treat each component as an independent runtime: separate `max_num_seqs`, sampling params, connectors, graph/eager policy, and optional replicas. That is the prerequisite for every optimization below — without it, batching and CUDA Graph would still be constrained by whichever sub-path is slowest in a monolithic loop. - -**Why it works.** Stage boundaries turn model-internal handoffs into first-class serving objects. Connectors define what crosses each boundary (hidden states, embeddings, codec codes, chunk metadata), and the scheduler can schedule, batch, and scale each stage on its own critical path. Code2Wav can batch vocoder forwards while Talker still runs single-token decode; audio can surface from stage 2 while text still streams from stage 0; only stages 1 and 2 need replicas under concurrent speech load. - -In practice, stage 0 owns the tokenizer and multimodal prompt; stage 1 consumes Thinker hidden states and embeddings; stage 2 consumes codec codes and emits audio. The default MoE deploy profile places Thinker on one GPU and Talker/Code2Wav on another, matching their different memory and compute profiles. Without explicit boundaries, connector payloads, async-chunk handoff state, and per-stage execution policy would remain buried inside model code. - -### 2. Batching: AR Decode and Code2Wav Micro-Work - -**Why.** After stage decomposition, the speech path still spent most GPU time on single-request micro-work. Each Talker decode step runs a short code-predictor forward; each Code2Wav chunk is a small vocoder forward. At concurrency 32, processing those steps one request at a time leaves SMs idle between launches and prevents the scheduler from amortizing fixed per-step costs. - -**What you gain.** Batching raises GPU occupancy on the speech-generation side: more requests share the same Talker MTP invocation and the same Code2Wav forward. In reported tests, Code2Wav batching alone cut E2E from 9697 ms to 6435 ms and improved RTF by **62.36%** at high concurrency. Batching is the baseline layer in our sweep — without it, later optimizations have little to stack on. - -**Why it works.** Talker and Code2Wav decode steps are largely independent across requests once stage inputs are prepared. The optimized Talker runner collects single-token non-prefill requests into a decode batch, writes `input_ids`, embeddings, last Talker hidden states, and text-step embeddings into preallocated GPU buffers, and invokes the Talker MTP path once. Batched Code2Wav returns lists of audio tensors, reuses `ubatch_slices`, and runs one forward for multiple connector chunks. Fixed kernel launch and Python dispatch costs are paid once per batch instead of once per request. - -Even with batching alone, the benchmark sweep at concurrency 32 still reaches only **35.0** output tok/s, **4975 ms** mean audio TTFP, and **0.98** mean audio RTF — near realtime on the audio path. Batching improves utilization but does not remove per-step launch overhead or full-payload stage waits; the next layers address those. - -### 3. CUDA Graph: Per-Stage Decode Capture - -**Why.** Batching raised occupancy, but each decode step still paid repeated CPU-side kernel dispatch. Qwen3-Omni runs three decode-heavy stages; Talker alone may execute hundreds of short steps per utterance, and each step previously re-launched the same stable operator sequence from Python. At concurrency 32, that launch tax dominated TPOT and kept RTF above realtime even after batching. - -**What you gain.** Turning on CUDA Graph for all three stages in the benchmark sweep raises output throughput from **35.0** to **135.2** tok/s (+286%), cuts mean audio TTFP from **4975 ms** to **1798 ms**, and drops mean audio RTF from **0.98** to **0.38**. Most of the win comes from removing launch overhead across Thinker text generation, Talker codec decode, and Code2Wav vocoder forwards together — not from Code2Wav alone. - -**Why it works.** CUDA Graph captures a fixed operator sequence once and replays it with minimal CPU work. Each stage has a different capture point, but the principle is the same: decode shapes bucket into stable `(batch, seq, frames)` profiles, so the runtime records the graph at warmup and reuses it on the hot path. In the benchmark sweep, the **CUDA Graph** step sets `enforce_eager: false` on stages 0, 1, and 2 (the **Batch** baseline sets `enforce_eager: true` everywhere). - -#### Stage 0 — Thinker: vLLM outer decode graph - -The Thinker is an autoregressive multimodal stage (`LLM_AR`). When `enforce_eager` is false, it uses vLLM's standard CUDA Graph capture on the decode path — the same mechanism as text-only LLM serving. This removes repeated CPU-side kernel dispatch during long Thinker generations. - -The default deploy profile leaves stage 0 on graph-friendly settings on CUDA. Platform overrides can disable outer capture where the backend is unsafe: for example, XPU sets `enforce_eager: true` and `max_cudagraph_capture_size: 0` on stage 0. - -#### Stage 1 — Talker: outer decode graph + compiled code predictor - -The Talker stage also runs through vLLM's outer CUDA Graph path when `enforce_eager: false`. Each Talker decode step additionally invokes the **code predictor** — a short re-prefill transformer that emits RVQ codec codes. That inner path is optimized separately: - -- **`torch.compile`** fuses the 5-layer predictor forward (`dynamic=False`, `epilogue_fusion=False`) so RMSNorm/RoPE stay numerically aligned with the reference path while still reducing kernel count per step. -- On CUDA, the code predictor does **not** enable a second manual CUDA Graph layer by default (`use_cuda_graphs=False`), because that would conflict with vLLM's Talker `CUDAGraphWrapper`. The outer Talker graph and compiled inner forward are complementary: one captures the AR stage loop, the other fuses the codec-prediction micro-forward. -- On NPU, the same wrapper can fall back to platform graphs (`use_cuda_graphs=True`) where Inductor is unavailable. - -Optional prefix-graph buckets (`code_predictor_prefix_graphs` in connector config) can capture additional stable predictor shapes when explicitly enabled. - -#### Stage 2 — Code2Wav: inner vocoder graph - -Code2Wav is a generation stage (`LLM_GENERATION`), not an AR loop. Its graph path is an **inner** `CUDAGraphDecoderWrapper` rather than vLLM's outer wrapper: - -```python -# Enabled during weight load when stage enforce_eager is false -self.code2wav.enable_cudagraph( - codec_chunk_frames=chunk_frames, - codec_left_context_frames=left_frames, -) -``` - -**Shape bucketing from connector config.** Before warmup, the wrapper reads `codec_chunk_frames` and `codec_left_context_frames` from the stage connector config. Capture enumerates the `(batch, num_quantizers, frames)` buckets that async-chunk and full-payload decode will hit at runtime — including the smaller first chunk from `initial_codec_chunk_frames`. - -**Vocoder warmup.** `precompute_snake_caches()` runs before graph capture so SnakeBeta activations do not pay repeated setup inside the captured decode loop. - -**Chunk dispatch.** In async-chunk mode, `chunked_decode_streaming` delegates stable chunks to `_cudagraph_wrapper.chunked_decode_with_cudagraph`; full-payload paths use the wrapper's batched decode entry points when shapes match captured buckets. - -#### Backend policy across all three stages - -| Platform | Thinker / Talker | Code2Wav | -|---|---|---| -| CUDA (default) | Outer graph on (`enforce_eager: false`) | Inner graph on (`enforce_eager: false`) | -| ROCm | Outer graph on | **Eager** — `conv_transpose1d` via MIOpen is not capture-safe | -| XPU | Eager, `max_cudagraph_capture_size: 0` | Eager, `max_cudagraph_capture_size: 0` | - -### 4. Async Chunk: Inter-Stage Handoffs and Overlapped Transfer - -**Why.** CUDA Graph made each stage faster, but the pipeline was still **barrier-synchronized**: Talker could not start until Thinker finished, and Code2Wav could not emit audio until Talker accumulated a full payload. First-audio latency therefore tracked full Thinker generation plus full Talker prefill — even when only a few codec frames were needed to produce the first audible chunk. - -**What you gain.** Async chunking is the largest TTFP win in the sweep: mean audio TTFP drops from **1798 ms** (CUDA Graph) to **497 ms**, and from **4975 ms** (Batch alone) overall. Output throughput rises to **152.5** tok/s because pipelined handoffs keep Talker and Code2Wav busy while Thinker is still decoding. In a reported long-run benchmark, enabling async put/get cut E2E from 390.5 s to 271.2 s and mean audio TTFP from 175.2 s to 4.0 s. - -**Why it works.** Async chunking turns sequential stage completion into **pipelined partial handoffs**. Thinker emits embedding rows incrementally; Talker accumulates codec frames and slices on `initial_codec_chunk_frames` / `codec_chunk_frames` boundaries; Code2Wav decodes each slice with `codec_left_context_frames` for continuity. The async scheduler and shared-memory connector overlap chunk transfer with stage compute, so the next stage starts work while the previous stage is still decoding. - -**Connector chunk policy.** The deploy config controls output chunk cadence independently from Code2Wav decode context: - -```yaml -initial_codec_chunk_frames: 4 -codec_chunk_frames: 25 -codec_left_context_frames: 25 -``` - -The connector can emit a small first codec chunk for lower TTFP while Code2Wav keeps enough left context for audio continuity across chunk boundaries. - -**Incremental handoffs instead of full replay.** On the worker connector path, Thinker→Talker async chunking tracks `put_req_chunk` plus pending chunk-0 prefill state; Talker→Code2Wav async chunking accumulates codec rows in `code_prompt_token_ids` and slices them using `codec_chunk_frames`, `initial_codec_chunk_frames`, and `codec_left_context_frames`. - -**Transfer overlap.** The async scheduler and shared-memory connector overlap chunk IO with stage compute. Large payloads use key-addressed POSIX SHM; small payloads can inline to avoid mmap and file-lock overhead. Pending keys are tracked and unlinked on request cleanup so aborted async chunks do not leak `/dev/shm` segments. - -### 5. Async Output: Non-Blocking Client Emission - -**Why.** Async chunking speeds inter-stage handoffs, but the **client-facing emission path** can still stall stage workers. If Code2Wav must synchronously serialize, buffer, and push each audio chunk through the HTTP serving loop before starting the next decode step, GPU time is lost to I/O and Python scheduling even though the stage-to-stage pipeline is already incremental. - -**What you gain.** On top of async chunking in the benchmark sweep, async output keeps mean audio TTFP near **464 ms** while lowering RTF from **0.39** to **0.31** at concurrency 32 — decoupling client emission removes scheduling stalls without giving back the TTFP win from async chunking. - -**Why it works.** `use_async_omni_output` decouples chunk production from chunk delivery. Code2Wav can finish a vocoder forward and hand audio to a non-blocking output path while Talker and the connector keep advancing. The serving loop surfaces packets to `/v1/chat/completions` asynchronously instead of making decode workers wait on client I/O. - -### 6. Stage Replicas: Scaling Talker and Code2Wav - -**Why.** Under concurrent speech generation, Thinker, Talker, and Code2Wav do not saturate equally. Thinker handles one multimodal prompt per request; Talker and Code2Wav execute many short decode steps and vocoder forwards per utterance. At concurrency 32, a single Talker/Code2Wav instance becomes the tail bottleneck even when Thinker still has headroom — replicating the entire pipeline would waste memory duplicating the multimodal stage. - -**What you gain.** Adding replicas on top of async output reaches **185.8** output tok/s (+35%) at concurrency 32 — the highest throughput in the sweep — while keeping mean audio TTFP near **482 ms** and RTF near **0.31**. The tracked multi-replica perf suite separately targets **2100 ms** mean audio TTFP and **0.39** mean audio RTF for long text-plus-audio workloads at concurrency 32. - -**Why it works.** Stage replication is horizontal scaling applied only where queue depth builds up. One Thinker on GPU 0 fans out to two Talker replicas on GPUs 1 and 2; two Code2Wav replicas share the same pair of GPUs. Load spreads across the speech-generation side without copying the full three-stage pipeline per replica. The benchmark deploy config keeps one Thinker and runs two replicas each for Talker and Code2Wav: - -```json -{ - "stage_overrides": { - "1": {"num_replicas": 2, "devices": "1,2"}, - "2": {"num_replicas": 2, "devices": "1,2"} - }, - "extra_cli_args": ["--async-chunk"] -} -``` - -

- Qwen3-Omni async chunk and stage replica serving, showing one Thinker feeding two Talker replicas and two Code2Wav replicas. -
- Figure 3: Async chunking and stage replicas target the speech-generation side of the pipeline, where Talker and Code2Wav can become the bottleneck under concurrent load. -

- -### 7. Hot-Path Cleanup: Talker Decode and Connector Payloads - -**Why.** Batching, graphs, async chunking, and replicas address structural bottlenecks, but the Talker decode loop still executed hundreds of small per-step costs: Python `generate()` dispatch, repeated `torch.cat` while building connector payloads, and device-to-host reads of decode state that the next step immediately needed back on GPU. - -**What you gain.** Hot-path cleanup removes overhead that scales with utterance length. In a long-context single-request test, E2E dropped from 21.28 s to 7.37 s, audio TTFP from 3197 ms to 1796 ms, and audio RTF from 0.71 to 0.28. These changes stack with the layers above and show up in the DFX perf suite baselines rather than as a separate sweep row. - -**Why it works.** Each fix targets a repeated micro-cost on the critical path: - -**Connector payload construction.** Large async chunks previously paid for repeated `torch.cat` while accumulating Thinker and Talker payloads. Passing decode embeddings per token and trimming redundant payload assembly removes allocation and copy work on every chunk boundary. - -**Talker code predictor rewrite.** The older path used Hugging Face `generate()` for very short codec-predictor sequences. That added Python dispatch, dynamic allocation, and KV-cache overhead on every Talker decode step. The optimized path uses re-prefill with SDPA, native GQA, inline top-k sampling, cached module references, and `torch.compile` on the inner transformer (`epilogue_fusion=False` for fp32-safe RMSNorm/RoPE). On CUDA this compile path sits below vLLM's Talker CUDA Graph rather than adding a conflicting second graph layer; see §3 above. - -**GPU-resident decode state.** Intermediate keys such as `hidden_states.last`, `hidden_states.trailing_text`, `embed.tts_pad_projected`, and `codes.audio` stay in `model_intermediate_buffer` so the next decode step reuses GPU-resident tensors instead of forcing device-to-host synchronization — eliminating round trips that would otherwise serialize every Talker step. - -**Numerical precision on the code predictor.** Where audio quality is sensitive, RMSNorm variance and RoPE stay in fp32, attention uses SDPA with native GQA, and per-call embedding buffers avoid cross-request aliasing. Qwen3-Omni configures the wrapper with parallel embedding, stored sampling, and returned projection buffers so Talker can feed the next decode step without rebuilding state on CPU. The compile and graph layers (§3) then optimize this cleaned-up path instead of fighting legacy `generate()` overhead. - -These hot-path changes stack with the batching, graph, and async-chunk optimizations above. Qwen3-Omni performance tests in the DFX perf suite — text-only serving and async-chunk workloads — give tracked baselines instead of one-off local numbers. - -### 8. Validation Results - -We validated the optimization stack with a controlled local benchmark on `Qwen3-Omni-30B-A3B-Instruct` (`128` prompts, concurrency `32`, `5` warmups, three GPUs mapped as `0/1/2`: Thinker on GPU 0, Talker and Code2Wav each with 2 replicas on GPUs 1 and 2). Each configuration restarted the server with an isolated deploy profile. The charts below start from the **Batch** baseline and compare Batch, CUDA Graph, Async chunk, Async output, and Stage replicas. - -

- Qwen3-Omni output throughput at concurrency 1 vs 32 across optimization configs from Batch to Replicas. -
- Figure 4: Output throughput (tok/s) at concurrency 1 (orange) vs concurrency 32 (green). CUDA Graph, async output, and stage replicas raise c=32 output tok/s from 35.0 (Batch) to 185.8 (Replicas). -

- -

- Qwen3-Omni request throughput at concurrency 1 vs 32 across optimization configs from Batch to Replicas. -
- Figure 5: Request throughput (req/s) at concurrency 1 (orange) vs concurrency 32 (green). Stage replicas reach 11.1 req/s at c=32, up from 2.0 req/s (Batch). -

- -

- Qwen3-Omni mean audio RTF at concurrency 1 vs 32 across optimization configs from Batch to Replicas. -
- Figure 6: Mean audio RTF at concurrency 1 (orange) vs concurrency 32 (green). Batch alone stays near realtime at c=32 (RTF 0.98); async chunking and replicas keep RTF at or below ~0.40. -

- -

- Qwen3-Omni mean audio TTFP at concurrency 1 vs 32 across optimization configs from Batch to Replicas. -
- Figure 7: Mean audio TTFP in milliseconds (log scale). Async chunking drops c=32 TTFP from ~4975 ms (Batch) to ~497 ms. -

- -| Config (c=32) | Output tok/s | Audio-s/s | Mean audio TTFP | Mean audio RTF | -|---|---:|---:|---:|---:| -| Batch | 35.0 | 10.8 | 4975 ms | 0.98 | -| CUDA Graph | 135.2 | 40.9 | 1798 ms | 0.38 | -| Async chunk | 152.5 | 45.9 | 497 ms | 0.39 | -| Async output | 137.4 | 42.4 | 464 ms | 0.31 | -| Stage replicas | 185.8 | 57.1 | 482 ms | 0.31 | - -Stage replicas deliver the best throughput on this sweep: **185.8** output tok/s at concurrency 32, with TTFP and RTF staying in the same **~460–500 ms / ~0.31** band as async output. Async chunking is the largest latency win over Batch alone, cutting TTFP from **4975 ms** to **497 ms** and RTF from **0.98** to **0.39**. - -The following results come from the Qwen3-Omni DFX perf artifacts for `Qwen/Qwen3-Omni-30B-A3B-Instruct` using the `openai-chat-omni` backend. They reflect the accumulated optimization stack above. The percentages compare the measured run against the tracked perf-suite baseline for the same workload. Lower is better for TTFT, E2EL, TTFP, and audio RTF; higher is better for throughput. - -| Workload | Key result | Change vs. baseline | -|---|---:|---:| -| Long text + audio output, 128 prompts, concurrency 32, 2500 input / 900 output tokens | 80.43 audio-s/s, 0.399 mean audio RTF, 1065 ms mean audio TTFP | TTFT -20.3%, E2EL -26.3%, audio RTF -29.1% | -| Audio-input multimodal, 10 prompts at 0.1 req/s | 3.15 audio-s/s, 0.117 mean audio RTF, 396 ms mean audio TTFP | TTFT -12.7%, E2EL -11.6%, audio RTF -10.3% | -| Image + video multimodal, 40 prompts at 0.5 req/s | 13.74 audio-s/s, 0.146 mean audio RTF, 332 ms mean audio TTFP | TTFT -9.2%, E2EL -24.0%, audio RTF -19.8% | -| Image + video + audio multimodal, 100 prompts at 1.0 req/s | 29.07 audio-s/s, 0.186 mean audio RTF, 553 ms mean audio TTFP | TTFT -14.3%, E2EL -34.5%, audio RTF -31.4% | - -The same infrastructure also benefits text-only routing through Qwen3-Omni. In the text-only long-output workload with 128 prompts at concurrency 32, the measured run reached 2,032 output tok/s and 7,698 total tok/s, while mean E2EL dropped from the tracked 18.15 s baseline to 14.15 s (-22.1%). - -The multi-replica perf suite separately validates the deployment shape where stages 1 and 2 each run two replicas. That suite tracks long text-plus-audio serving at concurrency 8, 16, 24, and 32, with baseline targets for TTFT, TPOT, audio TTFP, and audio RTF. The Stage replicas row in the benchmark sweep above shows the same scaling idea in practice: replicating Talker and Code2Wav improves throughput and latency without duplicating the Thinker stage. - -The important result is that the gains show up across different request shapes: long text-plus-audio generation, audio input, image/video input, and mixed image/video/audio input. That is what we want from a pipeline optimization. It improves the common dataflow instead of only helping one narrow prompt pattern. - -## Deployment Profile - -The default Qwen3-Omni-MoE deploy profile is designed for a staged layout. In the checked-in profile, stage 0 runs on one GPU, while stages 1 and 2 run on another: - -```yaml -stages: - - stage_id: 0 - devices: "0" - max_num_seqs: 64 - default_sampling_params: - temperature: 0.0 - max_tokens: 2048 - - - stage_id: 1 - devices: "1" - max_num_seqs: 64 - input_connectors: - from_stage_0: connector_of_shared_memory - default_sampling_params: - temperature: 0.9 - top_k: 50 - max_tokens: 4096 - - - stage_id: 2 - devices: "1" - max_num_seqs: 64 - input_connectors: - from_stage_1: connector_of_shared_memory - default_sampling_params: - temperature: 0.0 - max_tokens: 65536 -``` - -The exact device assignment is not a universal recommendation. The useful part is the separation of concerns: - -- Thinker has the multimodal prompt and deterministic text-generation profile. -- Talker has speech-generation sampling parameters. -- Code2Wav has audio reconstruction parameters and backend-specific graph policy. -- Shared-memory connectors carry structured payloads between adjacent stages. - -## What This Enables - -With this design, vLLM-Omni can serve Qwen3-Omni as more than a monolithic checkpoint: - -- Per-request `modalities` control how far the pipeline runs: text-only requests finish at Thinker; adding `"audio"` activates Talker and Code2Wav. -- Chat serving can receive text while audio serving consumes Code2Wav output metadata such as sample rate. -- Platform-specific deployment differences can live in YAML rather than model branches. - -This is the main lesson from Qwen3-Omni support in vLLM-Omni: omni-modality serving is a dataflow problem as much as a kernel problem. Efficient inference depends on carrying the right tensors across stage boundaries, preserving request-scoped async-chunk handoff state, and choosing execution policy per component. - -## References - -- Qwen3-Omni pipeline topology in vLLM-Omni: [`pipeline.py`](https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/model_executor/models/qwen3_omni/pipeline.py) -- Qwen3-Omni model wrapper in vLLM-Omni: [`qwen3_omni.py`](https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/model_executor/models/qwen3_omni/qwen3_omni.py) -- Qwen3-Omni stage input processors: [`stage_input_processors/qwen3_omni.py`](https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/model_executor/stage_input_processors/qwen3_omni.py) -- Qwen3-Omni deploy profile: [`qwen3_omni_moe.yaml`](https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/deploy/qwen3_omni_moe.yaml) -- Qwen3-Omni async-chunk perf config: [`test_qwen3_omni_async_chunk.json`](https://github.com/vllm-project/vllm-omni/blob/main/tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json) -- Qwen3-Omni multi-replica perf config: [`test_qwen3_omni_multi_replicas.json`](https://github.com/vllm-project/vllm-omni/blob/main/tests/dfx/perf/tests/test_qwen3_omni_multi_replicas.json) -- Qwen3-Omni model repository: [Qwen/Qwen3-Omni-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct) - -If you are interested in Qwen3-Omni serving or omni-modality inference, join the `#vllm-omni` channel in [vLLM Slack](https://slack.vllm.ai), or open an issue in [vLLM-Omni GitHub](https://github.com/vllm-project/vllm-omni). diff --git a/_posts/2026-07-01-qwen3-omni-optimization.md b/_posts/2026-07-01-qwen3-omni-optimization.md new file mode 100644 index 00000000..0dad7cad --- /dev/null +++ b/_posts/2026-07-01-qwen3-omni-optimization.md @@ -0,0 +1,284 @@ +--- +layout: post +title: "Experience and Lessons Learned from Serving Multi-Stage Qwen3-Omni in vLLM-Omni" +author: "vLLM-Omni Team and Ant Group SCT Team" +summary: "How vLLM-Omni serves and optimizes Qwen3-Omni with staged Thinker-Talker-Code2Wav execution, batching, CUDA Graphs, async chunk, async output, replicas, hot-path cleanup, and perf validation." +image: /assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-serving-flow.svg +social_image: /assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-serving-flow.svg +read_time_minutes: 12 +tags: + - performance + - multimodal + - vllm-omni +--- + +Qwen3-Omni combines multimodal understanding with speech generation. This post explains how [vLLM-Omni](https://github.com/vllm-project/vllm-omni) serves it as a staged pipeline and optimizes each stage for online workloads. + +## TL;DR + +vLLM-Omni's Qwen3-Omni serving stack includes: + +- **A three-stage pipeline:** Thinker for multimodal reasoning, Talker for speech codec generation, and Code2Wav for waveform reconstruction. +- **OpenAI-compatible serving:** `/v1/chat/completions` is the primary endpoint for Qwen3-Omni text and audio generation. +- **Batching, CUDA Graphs, async chunk, async output, replicas, and hot-path cleanup:** stage-level batching and per-stage graph capture on Thinker, Talker, and Code2Wav improve high-concurrency throughput; async-chunk handoffs and async output keep the pipeline and decode workers from stalling on full-payload barriers and synchronous payload construction; Talker/Code2Wav replicas scale the speech-generation stages; and hot-path cleanup trims the per-step model-internal overhead that scales with utterance length. +- **Performance validation:** controlled benchmark sweeps and DFX perf runs show lower audio TTFP (time to first audio packet), lower audio RTF (real-time factor), and higher throughput as each optimization layer is enabled. + +## Quickstart + +The default Qwen3-Omni deploy profile is resolved automatically when serving the model with `--omni`: + +```bash +vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \ + --omni \ + --port 8091 +``` + +For explicit configuration, pass the staged deploy profile: + +```bash +vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \ + --omni \ + --port 8091 \ + --deploy-config vllm_omni/deploy/qwen3_omni_moe.yaml +``` + +The bundled profile includes a `platforms:` section. vLLM-Omni detects the runtime backend (CUDA, NPU, ROCm or XPU) and merges the matching deltas automatically — no extra CLI flag is required. The same launch command works across hardware. + +Requests should use `/v1/chat/completions`. Set `modalities` in the request body to declare output types — e.g. `["text"]` for text only, or `["text", "audio"]` for text plus speech. + +For deployment options, async-chunk settings, and multi-replica layouts, see the [Qwen3-Omni online serving guide](https://github.com/vllm-project/vllm-omni/blob/main/examples/online_serving/qwen3_omni/README.md). + +## Qwen3-Omni Serving Model + +Text-only LLM serving is one loop: prefill, decode, detokenize. Qwen3-Omni adds two speech stages after multimodal reasoning, each with a different compute profile: + +```text +Thinker -> multimodal understanding + text generation +Talker -> hidden states and embeddings to RVQ codec codes +Code2Wav -> codec codes to waveform audio +``` + +

+ Qwen3-Omni serving flow in vLLM-Omni, showing multimodal inputs, Thinker, Talker, Code2Wav, and text/audio outputs. +
+ Figure 1: Qwen3-Omni serving in vLLM-Omni is a staged dataflow: Thinker produces text and hidden states, Talker produces codec codes, and Code2Wav reconstructs audio. +

+ +## Optimization Overview + +Different parts of the Qwen3-Omni pipeline hit different bottlenecks. vLLM-Omni does not apply one fixed recipe; each optimization targets a specific stage or handoff. The walkthrough that follows takes each one in turn — in the order we validated it — and answers three questions in order: **Why** the problem exists, **Why it works** (how the mechanism removes that problem), and **What you gain** (the measured payoff). + +| Technique | Target stage / path | Problem it addresses | Primary benefit | +|---|---|---|---| +| Stage decomposition | Thinker → Talker → Code2Wav | Three stages with very different compute profiles share one loop, so a single batching/graph/device policy lets the slowest sub-path gate the rest — capping throughput and blocking per-stage tuning and scaling | Independent runtime policy per stage | +| AR + Code2Wav batching | Talker MTP path, Code2Wav async chunks | Single-request micro-work leaves SMs idle between launches at high concurrency, capping GPU occupancy and req/s | Higher occupancy and req/s | +| CUDA Graph | Thinker / Talker / Code2Wav decode paths | Repeated CPU-side kernel dispatch on every decode step inflates TPOT and keeps audio RTF above real-time | Lower TPOT and audio RTF; ~4× throughput jump in sweep | +| Async chunk | Thinker→Talker, Talker→Code2Wav | Full-payload stage barriers force Code2Wav to wait for a full Talker payload, delaying first audio and inflating audio TTFP | Pipelined handoffs; largest audio TTFP reduction | +| Async omni output | Thinker connector payloads | Synchronous payload construction blocks Thinker decode workers between chunks, wasting GPU time on allocation and lowering throughput | Throughput recovery without audio TTFP regression | +| Stage replicas | Talker, Code2Wav | Talker and Code2Wav saturate and queue while Thinker still has headroom, becoming the tail bottleneck under load | Horizontal scale on bottleneck stages only | +| Hot-path cleanup | Talker code predictor, connector payloads | Per-step Python, allocation, and sync overhead compounds over long utterances, inflating E2EL and audio TTFP | Lower per-step latency; stacks with all layers above | + +We validated each layer with a controlled benchmark sweep on Seed-TTS `en` (`Qwen3-Omni-30B-A3B-Instruct`, `10`/`160`/`320`/`640` prompts at concurrency `1`/`16`/`32`/`64`, `5` warmups, three visible GPUs mapped as `0/1/2`). Each configuration restarted the server with an isolated deploy profile and added one optimization on top of the previous row. **Batch** through **Async output** pin one stage per GPU (Thinker / Talker / Code2Wav on GPUs 0 / 1 / 2, single replica each); the **Stage replicas** row keeps Thinker on GPU 0 and runs 2× Talker + 2× Code2Wav on GPUs 1 and 2. The table below summarizes concurrency 64; [Validation Results](#validation-results) charts all four concurrency levels. + +| Step | Config added | Talker / Code2Wav replicas | Req/s | Mean audio TTFP | Mean audio RTF | +|---|---|---|---:|---:|---:| +| Baseline | Batch | 1 / 1 | 2.2 | 5884 ms | 1.15 | +| + CUDA Graph | Graph capture on Thinker, Talker, Code2Wav | 1 / 1 | 8.6 (+299%) | 2790 ms (−53%) | 0.59 (−49%) | +| + Async chunk | Async-chunk stage handoffs | 1 / 1 | 9.3 (+8%) | 655 ms (−77%) | 0.63 | +| + Async output | Async omni output path | 1 / 1 | 11.3 (+22%) | 631 ms (−4%) | 0.47 (−25%) | +| + Stage replicas | 2× Talker + 2× Code2Wav | 2 / 2 | 11.7 (+4%) | 632 ms | 0.47 | + +

+ Qwen3-Omni optimization stack in vLLM-Omni, showing staged execution, batching and replicas, CUDA Graph, async chunk, and hot-path cleanup. +
+ Figure 2: Qwen3-Omni performance comes from optimizing the staged dataflow, stage runtime, and decode hot path together. +

+ +## Optimization Stack, Stage by Stage + +Each optimization builds on the previous one, so the numbers reported in each step assume every layer above it is already enabled. + +### 1. Stage Decomposition and Batching: The Baseline + +**Why.** Qwen3-Omni is not one homogeneous decode loop: Thinker does multimodal AR text generation, Talker runs a codec-predictor AR path, and Code2Wav runs parallel vocoder decode. Folding these three very different workloads into a single serving path forces the same batching policy, graph policy, and device layout on all of them — and lets the slowest sub-path gate the rest. Separating the stages removes that coupling, but it exposes a second problem: the speech path still spends most of its GPU time on single-request micro-work. Each Talker decode step is a short code-predictor forward and each Code2Wav chunk is a small vocoder forward, so at concurrency 64, running them one request at a time leaves SMs idle between launches and never amortizes the fixed per-step cost. + +**Why it works.** This addresses the two problems in turn. First, stage decomposition breaks the coupling: stage boundaries become first-class serving objects, connectors define what crosses each one (hidden states, embeddings, codec codes, chunk metadata), and the scheduler can schedule, batch, and graph each stage on its own critical path — so no single policy is forced on all three and the slowest sub-path no longer gates the rest. Second, per-stage batching closes the idle-SM gap: collecting concurrent requests into one Talker MTP invocation and one Code2Wav forward fills the SMs that single-request micro-work left idle, and amortizes the fixed per-step cost across the batch. + +**What you gain.** Explicit stages let vLLM-Omni treat each component as an independent runtime — separate `max_num_seqs`, sampling params, connectors, graph/eager policy, and optional replicas — which is the prerequisite for every optimization below. This batched, stage-decomposed configuration is the **Batch** baseline that every later row builds on. + +### 2. CUDA Graph: Per-Stage Decode Capture + +**Why.** Batching raised occupancy, but each decode step still paid repeated CPU-side kernel dispatch. Qwen3-Omni runs three decode-heavy stages; Talker alone may execute hundreds of short steps per utterance, and each step previously re-launched the same stable operator sequence from Python. At concurrency 64, that launch tax dominated TPOT and kept audio RTF above real-time even after batching. + +**Why it works.** CUDA Graph removes the per-step kernel dispatch that dominated TPOT: it captures a fixed operator sequence once and replays it with minimal CPU work. Each stage has a different capture point, but the principle is the same: decode shapes bucket into stable `(batch, seq, frames)` profiles, so the runtime records the graph at warmup and reuses it on the hot path. + +

+ Per-stage CUDA Graph capture in Qwen3-Omni: Thinker and Talker run under vLLM's outer decode graph, Talker's inner code predictor is torch.compiled, and Code2Wav uses an inner CUDAGraphDecoderWrapper. +
+ Figure 3: Each stage captures graphs at a different point. Thinker and Talker decode under vLLM's outer graph; Talker's inner code predictor is torch.compiled instead of given a second graph; Code2Wav uses an inner CUDAGraphDecoderWrapper. +

+ +#### Stage 0 — Thinker: vLLM outer decode graph + +The Thinker is an autoregressive multimodal stage (`LLM_AR`). When `enforce_eager` is false, it uses vLLM's standard CUDA Graph capture on the decode path — the same mechanism as text-only LLM serving. This removes repeated CPU-side kernel dispatch during long Thinker generations. + +#### Stage 1 — Talker: outer decode graph + compiled code predictor + +The Talker stage also runs through vLLM's outer CUDA Graph path when `enforce_eager: false`. Each Talker decode step additionally invokes the **code predictor** — a short re-prefill transformer that emits RVQ codec codes. That inner path is optimized separately: + +- **`torch.compile`** fuses the 5-layer predictor forward (`dynamic=False`, `epilogue_fusion=False`) so RMSNorm/RoPE stay numerically aligned with the reference path while still reducing kernel count per step. +- On CUDA, the code predictor does **not** enable a second manual CUDA Graph layer by default (`use_cuda_graphs=False`), because that would conflict with vLLM's Talker `CUDAGraphWrapper`. The outer Talker graph and compiled inner forward are complementary: one captures the AR stage loop, the other fuses the codec-prediction micro-forward. + +Optional prefix-graph buckets (`code_predictor_prefix_graphs` in connector config) can capture additional stable predictor shapes when explicitly enabled. + +#### Stage 2 — Code2Wav: inner vocoder graph + +Code2Wav is a generation stage (`LLM_GENERATION`), not an AR loop. Its graph path is an **inner** `CUDAGraphDecoderWrapper` rather than vLLM's outer wrapper: + +```python +# Enabled during weight load when stage enforce_eager is false +self.code2wav.enable_cudagraph( + codec_chunk_frames=chunk_frames, + codec_left_context_frames=left_frames, +) +``` + +**Shape bucketing from connector config.** Before warmup, the wrapper reads `codec_chunk_frames` and `codec_left_context_frames` from the stage connector config. Capture enumerates the `(batch, num_quantizers, frames)` buckets that async-chunk and full-payload decode will hit at runtime — including the smaller first chunk from `initial_codec_chunk_frames`. + +**Vocoder warmup.** `precompute_snake_caches()` runs before graph capture so SnakeBeta activations do not pay repeated setup inside the captured decode loop. + +**Chunk dispatch.** In async-chunk mode, `chunked_decode_streaming` delegates stable chunks to `_cudagraph_wrapper.chunked_decode_with_cudagraph`; full-payload paths use the wrapper's batched decode entry points when shapes match captured buckets. + +**What you gain.** Turning on CUDA Graph for all three stages in the benchmark sweep raises req/s from **2.2** to **8.6** (+299%), cuts mean audio TTFP from **5884 ms** to **2790 ms**, and drops mean audio RTF from **1.15** to **0.59**. Most of the win comes from removing launch overhead across Thinker text generation, Talker codec decode, and Code2Wav vocoder forwards together. + +### 3. Async Chunk: Pipelined Inter-Stage Handoffs + +**Why.** CUDA Graph made each stage faster, but the pipeline was still **barrier-synchronized**: Talker could not start until Thinker finished, and Code2Wav could not emit audio until Talker accumulated a full payload. First-audio latency therefore tracked full Thinker generation plus full Talker prefill — even when only a few codec frames were needed to produce the first audible chunk. + +**Why it works.** Async chunk replaces the full-payload barrier with **pipelined partial handoffs**. Thinker emits embedding rows incrementally; Talker accumulates codec frames and slices them on `initial_codec_chunk_frames` / `codec_chunk_frames` boundaries; The async scheduler overlaps chunk transfer with stage compute, so each stage starts work while the previous one is still decoding — first audio is ready after a few codec frames instead of a full Thinker generation plus Talker prefill. + +

+ Async chunk before/after timeline: the barrier-synchronized pipeline waits for full Thinker and Talker payloads before first audio, while async chunk overlaps partial Thinker, Talker, and Code2Wav chunks so first audio arrives much earlier. +
+ Figure 4: Without async chunk, each stage waits for the previous stage's full payload, so first audio tracks full Thinker generation plus Talker prefill. Async chunk overlaps partial handoffs, so Code2Wav starts emitting audio after only a few codec frames. +

+ +**What you gain.** Async chunk is the largest audio TTFP win in the sweep: mean audio TTFP drops from **2790 ms** (CUDA Graph) to **655 ms**. + +### 4. Async Output: Non-Blocking Payload Construction + +**Why.** Async chunk pipelines Thinker→Talker→Code2Wav, but **synchronous payload construction** can still block decode workers. If Thinker must fully assemble each connector payload — copying embeddings and hidden states on every chunk boundary — before the next decode step can start, GPU time is lost to Python scheduling even though stage handoffs are already incremental. + +**Why it works.** `async_omni_output` decouples payload construction from stage handoff: Thinker hands decode state to a non-blocking output path and immediately returns to the next token, while the connector assembles and ships chunks asynchronously. + +

+ Async output before/after profiler trace: before, decode steps are separated by ~2.8 ms idle gaps while the connector payload is built synchronously; after, steps pack back-to-back with ~41 us gaps and the GPU stays busy. +
+ Figure 5: Decode step gap before and after async output. With synchronous payload construction, the GPU sits idle ~2.8 ms between Talker steps; moving payload assembly off the decode path packs steps back-to-back, shrinking the inter-step gap to ~41 µs. +

+ +**What you gain.** On top of async chunk at concurrency 64, async output keeps mean audio TTFP near **631 ms** while lowering mean audio RTF from **0.63** to **0.47**. + +### 5. Stage Replicas: Scaling Talker and Code2Wav + +**Why.** The three stages do not saturate equally under load. For each request, Thinker generates text once, but Talker and Code2Wav then run hundreds of short decode steps and vocoder forwards to render that text as audio — far more sustained small-step work on the speech side. So as concurrency climbs, Talker and Code2Wav saturate first: at concurrency 64 a single replica of either becomes the tail bottleneck while Thinker still has headroom. Scaling the whole pipeline would clear it, but waste memory duplicating the large multimodal Thinker. + +**Why it works.** Replication adds capacity only to the stages that saturate, not the whole pipeline. A single Thinker on GPU 0 feeds 2× Talker and 2× Code2Wav replicas spread across GPUs 1 and 2: the extra replicas absorb the speech-side backlog, while the heavy multimodal stage stays unduplicated. The benchmark deploy config enables this with: + +```json +{ + "stage_overrides": { + "1": {"num_replicas": 2, "devices": "1,2"}, + "2": {"num_replicas": 2, "devices": "1,2"} + } +} +``` + +

+ Qwen3-Omni async chunk and stage replica serving, showing one Thinker feeding two Talker replicas and two Code2Wav replicas. +
+ Figure 6: Async chunk and stage replicas target the speech-generation side of the pipeline, where Talker and Code2Wav can become the bottleneck under concurrent load. +

+ +**What you gain.** Adding replicas on top of async output reaches **11.7** req/s at concurrency 64 — the highest throughput in the sweep — while holding mean audio TTFP near **632 ms** and mean audio RTF near **0.47**. The replica margin over a single Talker/Code2Wav widens as concurrency climbs, since the speech stages are the first to saturate. + +### 6. Hot-Path Cleanup: Talker Decode and Connector Payloads + +**Why.** Batching, graphs, async chunk, and replicas eliminate the structural, framework-level bottlenecks — optimizations that apply to most multimodal serving pipelines. What remains is model-internal: profiling the Talker decode loop still shows a long tail of small costs that repeat on every step and so scale with utterance length — redundant connector traffic, per-step `torch.cat` and CPU serialization while building payloads, Python dispatch in the codec predictor, and device-to-host reads of decode state that the next step needs straight back on the GPU. + +**Why it works.** Each fix below removes one of those repeated costs — or trims fixed per-deployment overhead — without changing the audio output: + +**Decode-only connector handoffs.** Chunk 0 still ships the full Thinker prefill; every later decode step sends only the new `embed.decode` row instead of re-transmitting the full prefill embedding and hidden-state tensors. Connector traffic stays **O(1) per step** rather than growing with prompt length, and each handoff avoids redundant CPU serialization and cross-stage copies that would otherwise repeat on every Talker step in a long utterance. + +**Single-GPU executor default.** Removing the implicit `"mp"` default for `distributed_executor_backend` lets single-GPU deployments use the `uni` executor and avoid multiprocess startup, IPC, and worker-sync overhead on the low-concurrency paths where per-step latency matters most. + +**Connector payload construction.** Accumulating Thinker and Talker payloads previously paid for repeated `torch.cat` on every chunk boundary. Passing decode embeddings per token and trimming redundant assembly removes that allocation and copy work. The same change set also skips building downstream pooler/multimodal CPU payloads when a request's final stage is already local, avoiding hidden-state D2H on paths that do not feed another stage. + +**Talker code predictor rewrite.** The old path drove very short codec-predictor sequences through Hugging Face `generate()`, adding Python dispatch, dynamic allocation, and KV-cache overhead on every Talker step. The rewrite uses re-prefill with SDPA, native GQA, inline top-k sampling, cached module references, and `torch.compile` on the inner transformer. On CUDA this compile path sits below vLLM's Talker CUDA Graph rather than adding a conflicting second graph layer (see §2). + +**GPU-resident decode state and stage-local fast paths.** Intermediate tensors such as `hidden_states.last`, `hidden_states.trailing_text`, `embed.tts_pad_projected`, and `codes.audio` stay in `model_intermediate_buffer`, so the next step reuses them on-device instead of forcing a device-to-host round trip that would serialize every Talker step. Talker and Code2Wav also skip multimodal `get_mrope_input_positions` — they only need cheap linear positions — and `_store_value` avoids redundant `.to("cpu")` work when a tensor is already on CPU. + +**Numerical precision guardrail.** These rewrites must not regress audio quality, so RMSNorm variance and RoPE stay in fp32 (`epilogue_fusion=False`), and per-call embedding buffers avoid cross-request aliasing — the speed-ups above run on top of this constraint, not against it. + +**What you gain.** Hot-path cleanup removes overhead that scales with utterance length. In a long-context single-request test, E2EL dropped from 21.28 s to 7.37 s, audio TTFP from 3197 ms to 1796 ms, and audio RTF from 0.71 to 0.28. These changes stack with the layers above and show up in the DFX perf suite baselines rather than as a separate sweep row. + +## Validation Results + +The charts below plot the same benchmark sweep summarized in [Optimization Overview](#optimization-overview) — Batch, CUDA Graph, Async chunk, Async output, and Stage replicas — across all four concurrency levels (`1`/`16`/`32`/`64`), each starting from the **Batch** baseline. + +

+ Qwen3-Omni request throughput at concurrency 1, 8, 16, and 32 across optimization configs from Batch to Replicas. +
+ Figure 7: Request throughput (req/s) at c=1 (orange), c=16 (purple), c=32 (green), and c=64 (red). Stage replicas reach 11.7 req/s at c=64 and 6.8 req/s at c=32, up from 2.2 req/s (Batch at c=64). +

+ +

+ Qwen3-Omni mean audio RTF at concurrency 1, 8, 16, and 32 across optimization configs from Batch to Replicas. +
+ Figure 8: Mean audio RTF at c=1, 16, 32, and 64. Batch sits at or above real-time under load (RTF up to 1.15 at c=64); async output and replicas keep c=32/c=64 RTF at or below ~0.47. +

+ +

+ Qwen3-Omni mean audio TTFP at concurrency 1, 8, 16, and 32 across optimization configs from Batch to Replicas. +
+ Figure 9: Mean audio TTFP in milliseconds (log scale) at c=1, 16, 32, and 64. Async chunk drops c=64 TTFP from ~5884 ms (Batch) to ~655 ms. +

+ + +### Reading the Sweeps Together + +Across the three sweeps, the story is consistent: each layer targets a different bottleneck, and together they compound. + +- **Throughput (Figure 7).** Request throughput climbs from the Batch baseline of **2.2 req/s** to **11.7 req/s** at concurrency 64 (**~5.4×**), and from **1.1** to **6.8 req/s** at concurrency 32. The largest single jump is CUDA Graph (**~4×**); async output supplies the last big push at high concurrency, and stage replicas take throughput to its peak with headroom that grows as concurrency rises. +- **Real-time factor (Figure 8).** Mean audio RTF drops from an above-real-time **1.15** under Batch to **0.47** at concurrency 64 — decode moves from lagging playback under load to running comfortably ahead of it. +- **First-packet latency (Figure 9).** Mean audio TTFP falls from **~5884 ms** to **~632 ms** at concurrency 64, with async chunk contributing the largest single cut (to **~655 ms**) and the later layers holding that gain. + +The takeaway is that no single layer carries the whole win: CUDA Graph and async chunk dominate the latency reductions, while async output and stage replicas add the throughput headroom at concurrency 32 and 64. Stacking them turns a pipeline that barely keeps up under load into one with real-time headroom to spare. + +## Acknowledgements + +We thank the Qwen3-Omni contributors in [vLLM-Omni](https://github.com/vllm-project/vllm-omni), including Haiyan Wu, Taichang Zhou, Canlin Guo, Ruirui Yang, Ziming Huang, Wengang Zheng, Lianhao Xu, Han Gao, Junhong Liu, Samit Huang, Hao Chen, Alex Brooks, Chenguang Zheng, Peiqi Yin, Wenjing Chen, Nick Cao, Shunyang Li, Yong Yang, Divyansh Singhvi, Yueqian Lin, Dayu Qiu, Roger Wang and Hongsheng Liu, for their contributions and feedback. + +--- + +## References + +**Source and configuration** + +- Qwen3-Omni pipeline topology in vLLM-Omni: [`pipeline.py`](https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/model_executor/models/qwen3_omni/pipeline.py) +- Qwen3-Omni model wrapper in vLLM-Omni: [`qwen3_omni.py`](https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/model_executor/models/qwen3_omni/qwen3_omni.py) +- Qwen3-Omni stage input processors: [`stage_input_processors/qwen3_omni.py`](https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/model_executor/stage_input_processors/qwen3_omni.py) +- Qwen3-Omni deploy profile: [`qwen3_omni_moe.yaml`](https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/deploy/qwen3_omni_moe.yaml) +- Qwen3-Omni async-chunk perf config: [`test_qwen3_omni_async_chunk.json`](https://github.com/vllm-project/vllm-omni/blob/main/tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json) +- Qwen3-Omni multi-replica perf config: [`test_qwen3_omni_multi_replicas.json`](https://github.com/vllm-project/vllm-omni/blob/main/tests/dfx/perf/tests/test_qwen3_omni_multi_replicas.json) +- Qwen3-Omni model repository: [Qwen/Qwen3-Omni-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct) + +**Optimization pull requests** + +- CUDA Graph (§2): Thinker [vllm-omni#523](https://github.com/vllm-project/vllm-omni/pull/523), Talker [vllm-omni#669](https://github.com/vllm-project/vllm-omni/pull/669), Code2Wav [vllm-omni#2376](https://github.com/vllm-project/vllm-omni/pull/2376) +- Async chunk (§3): cross-stage chunked compute/communication [vllm-omni#727](https://github.com/vllm-project/vllm-omni/pull/727), async scheduling to overlap chunk IO and compute [vllm-omni#951](https://github.com/vllm-project/vllm-omni/pull/951), inter-packet latency optimization [vllm-omni#1656](https://github.com/vllm-project/vllm-omni/pull/1656) +- Async output (§4): async omni output materialization [vllm-omni#4476](https://github.com/vllm-project/vllm-omni/pull/4476) +- Stage replicas (§5): support multi-stage deployment[vllm-omni#2396](https://github.com/vllm-project/vllm-omni/pull/2396), omni stage runtime and distributed replica control plane [vllm-omni#3855](https://github.com/vllm-project/vllm-omni/pull/3855) +- Hot-path cleanup (§6): [vllm-omni#3007](https://github.com/vllm-project/vllm-omni/pull/3007), [vllm-omni#3164](https://github.com/vllm-project/vllm-omni/pull/3164), [vllm-omni#3878](https://github.com/vllm-project/vllm-omni/pull/3878) + +If you are interested in Qwen3-Omni serving or omni-modality inference, join the `#vllm-omni` channel in [vLLM Slack](https://slack.vllm.ai), or open an issue in [vLLM-Omni GitHub](https://github.com/vllm-project/vllm-omni). diff --git a/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-async-replica.svg b/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-async-replica.svg deleted file mode 100644 index e1fc4249..00000000 --- a/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-async-replica.svg +++ /dev/null @@ -1,56 +0,0 @@ - - Qwen3-Omni async chunk and stage replicas - One Thinker feeding two Talker replicas and two Code2Wav replicas. - - - - - - - - - Async chunk + stage replicas - - - Thinker - text + hidden states - - - Talker R0 - codec generation - - - Talker R1 - codec generation - - - Code2Wav R0 - waveform chunks - - - Code2Wav R1 - waveform chunks - - - Audio - streaming output - - - - - - - - - diff --git a/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-reqps.svg b/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-reqps.svg deleted file mode 100644 index c12736e4..00000000 --- a/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-reqps.svg +++ /dev/null @@ -1,64 +0,0 @@ - -Request throughput (req/s) across optimization configs - - -Request throughput (req/s) across optimization configs - -c=1 - -c=32 - - - -0.0 - -3.2 - -6.4 - -9.5 - -12.7 - - -0.4 - -1.8 - -2.0 - -2.3 - -2.4 - - -2.0 - -7.9 - -9.0 - -7.8 - -11.1 -Batch -CUDA -Graph -Async -chunk -Async -output -Replicas - diff --git a/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-rtf.svg b/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-rtf.svg deleted file mode 100644 index e712183a..00000000 --- a/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-rtf.svg +++ /dev/null @@ -1,64 +0,0 @@ - -Mean audio RTF across optimization configs - - -Mean audio RTF across optimization configs - -c=1 - -c=32 - - - -0.00 - -0.28 - -0.56 - -0.84 - -1.12 - - -0.85 - -0.17 - -0.16 - -0.14 - -0.14 - - -0.98 - -0.38 - -0.39 - -0.31 - -0.31 -Batch -CUDA -Graph -Async -chunk -Async -output -Replicas - diff --git a/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-throughput.svg b/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-throughput.svg deleted file mode 100644 index 1d29eab2..00000000 --- a/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-throughput.svg +++ /dev/null @@ -1,64 +0,0 @@ - -Output throughput (tok/s) across optimization configs - - -Output throughput (tok/s) across optimization configs - -c=1 - -c=32 - - - -0.0 - -53.4 - -107 - -160 - -214 - - -4.9 - -22.5 - -25.1 - -28.4 - -30.0 - - -35.0 - -135 - -152 - -137 - -186 -Batch -CUDA -Graph -Async -chunk -Async -output -Replicas - diff --git a/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-ttfp.svg b/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-ttfp.svg deleted file mode 100644 index 97fea100..00000000 --- a/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-bench-ttfp.svg +++ /dev/null @@ -1,58 +0,0 @@ - -Mean audio TTFP (ms) across optimization configs - - -Mean audio TTFP (ms) across optimization configs - -c=1 - -c=32 - - - -100 ms - -1000 ms - - -2556 ms - -552 ms - -129 ms - -124 ms - -128 ms - - -4975 ms - -1798 ms - -497 ms - -464 ms - -482 ms -Batch -CUDA -Graph -Async -chunk -Async -output -Replicas - diff --git a/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-optimization-stack.svg b/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-optimization-stack.svg deleted file mode 100644 index 843b6223..00000000 --- a/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-optimization-stack.svg +++ /dev/null @@ -1,59 +0,0 @@ - - Qwen3-Omni optimization stack in vLLM-Omni - Optimization areas for Qwen3-Omni staged serving in vLLM-Omni. - - - - - - - - - Qwen3-Omni optimization stack - - - Stage split - Thinker / Talker / C2W - - - Batch - AR + Code2Wav - - - CUDA Graph - stable chunks - - - Async chunk - overlap IO + compute - - - Async output - async omni output - - - Replicas - scale Talker / C2W - - - Hot-path cleanup - GPU-resident decode state - - - - - - - - diff --git a/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-async-chunk-timeline.svg b/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-async-chunk-timeline.svg new file mode 100644 index 00000000..98f12988 --- /dev/null +++ b/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-async-chunk-timeline.svg @@ -0,0 +1,141 @@ + + Async chunk before/after timeline + Barrier-synchronized execution waits for full Thinker and Talker payloads before first audio; async chunk pipelines partial handoffs so first audio arrives much earlier. + + + + + + + + + + + + + + + + + + + + + + Async chunk: barrier-synchronized vs pipelined + Stage handoff timeline · Thinker → Talker → Code2Wav + + + + Before + full-payload barrier + each stage waits + + Thinker + Talker + Code2Wav + + + + + full generation + + + full prefill + codec + + + decode + + + + + + first audio (late) + + + + After + pipelined partial + handoffs + + Thinker + Talker + Code2Wav + + + + + + + + + + + + + + + + + + + + + + + time + + + first audio (early) + + + + + + early 655 ms + late 2790 ms + + + + Mean audio TTFP (c=64) · −77% in sweep + 2790 ms → 655 ms + + + Stage idle between handoffs + eliminated + stages overlap in pipeline + + + Mechanism + async chunk + incremental stage handoffs + diff --git a/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-async-output-step-gap.svg b/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-async-output-step-gap.svg new file mode 100644 index 00000000..14fa313a --- /dev/null +++ b/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-async-output-step-gap.svg @@ -0,0 +1,121 @@ + + Qwen3-Omni async output decode step gap before and after + Before async output, decode steps are separated by ~2.8 ms idle gaps while the connector payload is built synchronously; after async output, steps pack back-to-back and a zoom inset shows the inter-step gap is only ~41 us. + + + + + + + + + + + + + + + + Async output: decode step gap before and after + Torch profiler trace · consecutive Talker decode steps in a fixed window + + + + Before + sync payload + construction + + GPU kernels + + + + + + + + + + + + + ~2.8 ms idle + + + + After + async output + (off decode path) + + GPU kernels + + + + + + + + + + + time + + + + + + + + + Zoom · one inter-step gap (GPU kernels) + + + + step N + step N+1 + + + ~41 µs + + + + Step-to-step gap + 2.8 ms → 41 µs + ~68× tighter + + + GPU idle between steps + eliminated + decode workers stay busy + + + Mechanism + async output + payload build off decode path + diff --git a/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-async-replica.svg b/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-async-replica.svg new file mode 100644 index 00000000..03aeb01d --- /dev/null +++ b/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-async-replica.svg @@ -0,0 +1,69 @@ + + Qwen3-Omni async chunk and stage replicas + One Thinker feeding two Talker replicas and two Code2Wav replicas. + + + + + + + + + Async chunk + stage replicas + Parallel Talker / Code2Wav replicas under one Thinker + + + + Thinker + text + hidden states + + + + Talker R0 + codec generation + + + Talker R1 + codec generation + + + + Code2Wav R0 + waveform chunks + + + Code2Wav R1 + waveform chunks + + + + Output + text + audio + + + + + + + + + + + + + + + + diff --git a/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-bench-reqps.svg b/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-bench-reqps.svg new file mode 100644 index 00000000..8e20ca8d --- /dev/null +++ b/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-bench-reqps.svg @@ -0,0 +1,124 @@ + +Request throughput (req/s) across optimization configs + + + +Request throughput (req/s) across optimization configs + +c=1 + +c=16 + +c=32 + +c=64 + + + + + + + + + +0 +5 +10 +15 +20 + + + + + + + +0.29 +0.29 + +1.34 +1.34 + +1.67 +1.67 + +1.91 +1.91 + +1.90 +1.90 + + +0.55 +0.55 + +2.74 +2.74 + +3.01 +3.01 + +3.49 +3.49 + +3.60 +3.60 + + +1.09 +1.09 + +5.72 +5.72 + +5.69 +5.69 + +6.55 +6.55 + +6.77 +6.77 + + +2.16 +2.16 + +8.60 +8.60 + +9.30 +9.30 + +11.3 +11.3 + +11.7 +11.7 +Batch +CUDA +Graph +Async +chunk +Async +output +Replicas + diff --git a/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-bench-rtf.svg b/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-bench-rtf.svg new file mode 100644 index 00000000..e8653f8b --- /dev/null +++ b/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-bench-rtf.svg @@ -0,0 +1,121 @@ + +Mean audio RTF across optimization configs + + + +Mean audio RTF across optimization configs + +c=1 + +c=16 + +c=32 + +c=64 + + + + + + + + + +0 rtf +0.33 rtf +0.66 rtf +1 rtf +1.3 rtf + + + + + + + +0.82 +0.82 + +0.19 +0.19 + +0.16 +0.16 + +0.14 +0.14 + +0.13 +0.13 + +0.93 +0.93 + +0.27 +0.27 + +0.29 +0.29 + +0.23 +0.23 + +0.23 +0.23 + +0.99 +0.99 + +0.38 +0.38 + +0.39 +0.39 + +0.31 +0.31 + +0.31 +0.31 + +1.15 +1.15 + +0.59 +0.59 + +0.63 +0.63 + +0.47 +0.47 + +0.47 +0.47 +Batch +CUDA +Graph +Async +chunk +Async +output +Replicas + diff --git a/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-bench-ttfp.svg b/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-bench-ttfp.svg new file mode 100644 index 00000000..897c87a0 --- /dev/null +++ b/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-bench-ttfp.svg @@ -0,0 +1,121 @@ + +Mean audio TTFP (ms) across optimization configs + + + +Mean audio TTFP (ms) across optimization configs + +c=1 + +c=16 + +c=32 + +c=64 + + + + + + + + +100 ms +500 ms +2000 ms +10000 ms + + + + + + + +3456 ms +3456 ms + +745 ms +745 ms + +129 ms +129 ms + +121 ms +121 ms + +129 ms +129 ms + +6377 ms +6377 ms + +1624 ms +1624 ms + +315 ms +315 ms + +279 ms +279 ms + +280 ms +280 ms + + +5824 ms +5824 ms + +1932 ms +1932 ms + +425 ms +425 ms + +433 ms +433 ms + +409 ms +409 ms + + +5884 ms +5884 ms + +2790 ms +2790 ms + +655 ms +655 ms + +631 ms +631 ms + +632 ms +632 ms +Batch +CUDA +Graph +Async +chunk +Async +output +Replicas + diff --git a/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-cuda-graph-stages.svg b/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-cuda-graph-stages.svg new file mode 100644 index 00000000..b0e649f7 --- /dev/null +++ b/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-cuda-graph-stages.svg @@ -0,0 +1,83 @@ + + Per-stage CUDA Graph capture in Qwen3-Omni + Thinker uses vLLM's outer decode graph; Talker adds a torch.compile inner code predictor under its outer graph; Code2Wav uses an inner CUDAGraphDecoderWrapper. + + + + + + + + + Per-stage CUDA Graph capture + + + + Stage 0 + Thinker + multimodal AR text + + + CUDA Graph · outer (vLLM) + + AR decode step + vLLM CUDAGraphWrapper + + + + Stage 1 + Talker + codec-predictor AR + + + CUDA Graph · outer (vLLM) + + AR decode step + outer stage loop + + Code predictor + torch.compile · inner forward + + + + Stage 2 + Code2Wav + parallel vocoder + + + CUDA Graph · inner + CUDAGraphDecoderWrapper + + Vocoder decode + shape-bucketed chunks + + + + + + + + Dashed = CUDA Graph capture region (outer = vLLM, inner = custom wrapper) + + Yellow = torch.compile (no second graph layer) + diff --git a/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-optimization-stack.svg b/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-optimization-stack.svg new file mode 100644 index 00000000..38599ac2 --- /dev/null +++ b/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-optimization-stack.svg @@ -0,0 +1,59 @@ + + Qwen3-Omni optimization stack in vLLM-Omni + Optimization areas for Qwen3-Omni staged serving in vLLM-Omni. + + + + + + + + + Qwen3-Omni optimization stack + + + Stage split + Thinker / Talker / Code2Wav + + + Batch + AR + Code2Wav + + + CUDA Graph + stable chunks + + + Async chunk + overlap IO + compute + + + Async output + async omni output + + + Replicas + scale Talker / Code2Wav + + + Hot-path cleanup + GPU-resident decode state + + + + + + + + diff --git a/assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-serving-flow.svg b/assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-serving-flow.svg similarity index 100% rename from assets/figures/2026-06-27-qwen3-omni-optimization/qwen3-omni-serving-flow.svg rename to assets/figures/2026-07-01-qwen3-omni-optimization/qwen3-omni-serving-flow.svg