[rollout, data] feat: integrate TransferQueue for multi-modal image data transfer by RobotGF · Pull Request #5780 · verl-project/verl

RobotGF · 2026-03-27T12:01:44Z

What does this PR do?

Add TQ-based zero-copy image data transfer between AgentLoop workers and vLLM server, using tq:// URL scheme to avoid serialising large PIL Images over Ray RPC.

Checklist Before Starting

Search for similar PRs. Paste at least one query link here: ...
Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
- {modules} include fsdp, megatron, veomni, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data, cfg, reward, fully_async, one_step_off
- If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
- {type} is in feat, fix, refactor, chore, test
- If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
- Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc.

API and Usage Example

Demonstrate how the API changes if any, and provide usage example(s) if possible.

# Add code snippet or script demonstrating how to use this

Design & Code Changes

Demonstrate the high-level design if this PR is complex, and list the specific changes.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

Read the Contribute Guide.
Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
Add / Update the documentation.
Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: ...
Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)
If your PR is related to the recipe submodule, please also update the reference to the submodule commit via git submodule update --remote or cd recipe && git pull origin main.

…ata transfer Add TQ-based zero-copy image data transfer between AgentLoop workers and vLLM server, using tq:// URL scheme to avoid serialising large PIL Images over Ray RPC. Changes: - New verl/utils/tq_multimodal.py: store/resolve images via TQ with zero-copy path (PIL→numpy→tensor→msgpack memoryview→ZMQ) - vllm_async_server.py: resolve tq:// URLs in generate(), inject TQ env vars into vLLM server Ray actor runtime_env - single_turn_agent_loop.py & tool_agent_loop.py: store images to TQ before generate() when TQ is enabled; no-op fallback otherwise Non-breaking: all changes are guarded by TQ availability checks. When TQ is not enabled, the existing PIL-Image-over-Ray-RPC path is used. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

gemini-code-assist

Code Review

This pull request introduces TransferQueue (TQ) support for multi-modal data to optimize communication by avoiding the serialization of large image blobs over Ray RPC. Key changes include the addition of a utility module for TQ operations, integration of media storage in agent loops, and URL resolution in the vLLM server. Feedback identifies a logic error in environment variable parsing, the use of inconsistent request IDs for tracing, and a risky dependency on a private external API for URL parsing.

…rformance - Fix hash collision: replace hash()-based index with uuid4 as explicit KV key - Batch images: flatten all images into one contiguous tensor per store call, reducing N TQ round-trips to 1 (store) and grouping fetches on resolve - Use async_kv_put with explicit key instead of async_put, ensuring deterministic retrieval by batch_key - Move inline imports to module top-level across all 4 files - Remove dead try/except ImportError in maybe_store_media_to_tq - Optimize _maybe_resolve_tq_media to check only first element Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ingleTurnAgentLoop Generate request_id once and pass it to both maybe_store_media_to_tq and server_manager.generate, ensuring consistent tracing across distributed components. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Store path: - Replace per-image np.copy() + torch.cat with pre-allocated contiguous buffer and direct copy_ into it (N+1 allocations → 1 allocation) Resolve path: - Output numpy.ndarray in channels-first (C,H,W) format directly, which vLLM multi_modal_data accepts natively - Skip PIL Image.fromarray round-trip, eliminating 1 memcpy per image on resolve and another inside vLLM's HF processor (PIL→tensor) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

pickle.loads on env vars crossing process boundaries allows arbitrary code execution. Switch serialize_tq_info/deserialize_tq_info to JSON. The vllm-side _deserialize_env_var already handles both formats (JSON preferred, pickle fallback with warning), so this is backward-compatible. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Cover URL helpers (make/parse/is_tq_url), TQ client singleton, JSON serialization helpers, and full store→resolve round-trip with mock TQ client including: - RGB, RGBA, grayscale image round-trips - Mixed image batch in single TQ entry - Channels-first (C,H,W) output format verification - Batch key uniqueness (uuid4) - maybe_store_media_to_tq enable/disable/fallback paths - TensorDict internal layout validation All tests run on CPU without real TransferQueue dependency. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

wuxibin89 · 2026-03-30T02:25:45Z

Hold until #5401 merged, do you any have performance comparison before and after this PR?

RobotGF · 2026-03-30T12:36:44Z

Hold until #5780 merged, do you any have performance comparison before and after this PR?

geo3k测试基本持平，小图+4nnodes，测试和Baseline基本没有影响，传输开销不是瓶颈

SamitHuang · 2026-03-31T04:09:17Z

+        ``(image_data, video_data)`` — either ``tq://`` URL lists or the
+        original PIL Image lists.
+    """
+    tq_enabled = os.environ.get("TRANSFER_QUEUE_ENABLE", False)


For ease-of-use concern, do we have a better way to trigger transfer queue rather env var?

btw, main_ppo uses os.environ.get("TRANSFER_QUEUE_ENABLE", "0") == "1", which is safer if the env var is set to "0"

For ease-of-use concern, do we have a better way to trigger transfer queue rather env var?

Yes，maybe the config system of transfer_queue field is better way

0oshowero0 · 2026-04-01T08:39:24Z

Do we have to use a URL-like format to pass the essential information required for TransferQueue to retrieve the data? Directly passing a KVBatchMeta obj maybe a simpler solution?
The current implementation lacks proper lifetime management for multimodal data. We should ensure MM data is deleted/released once it’s no longer needed to avoid unbounded memory/storage growth.
From a performance standpoint, the main bottleneck likely sits between AgentLoopManager and AgentLoopWorker. Since AgentLoopManager effectively acts as a secondary single-controller during inference, optimizing transfers across this boundary seems more critical. Is there any plan to introduce a distributed dataloader to let AgentLoopWorker directly process the data? @wuxibin89

RobotGF · 2026-04-02T03:20:31Z

Do we have to use a URL-like format to pass the essential information required for TransferQueue to retrieve the data? Directly passing a KVBatchMeta obj maybe a simpler solution?

The current implementation lacks proper lifetime management for multimodal data. We should ensure MM data is deleted/released once it’s no longer needed to avoid unbounded memory/storage growth.

From a performance standpoint, the main bottleneck likely sits between AgentLoopManager and AgentLoopWorker. Since AgentLoopManager effectively acts as a secondary single-controller during inference, optimizing transfers across this boundary seems more critical. Is there any plan to introduce a distributed dataloader to let AgentLoopWorker directly process the data? @wuxibin89

if use url pass data, can use http post with this PR feat: add TQMediaConnector plugin for TransferQueue image data support vllm-project/vllm#38355
yes, maybe need to add lifetime management is better

0oshowero0 · 2026-04-02T03:44:37Z

+        partition_id: TQ partition that holds the image data.
+
+    Returns:
+        A list of ``tq://<partition_id>/<batch_key>/<index>`` URL strings,


batch_key may lead to some misunderstanding for the upcoming agentic session level management. Can we refer to the key format design in https://github.com/verl-project/verl/pull/5401/changes#diff-4c72a1d4b0dd65c951d50ff4f1db44c5b9572658e7fcaaf380c1de576f9b8f6aR375 to propose a more clear name?

Now it sounds like we are retrieving the mm data for the whole batch

0oshowero0 · 2026-04-02T04:00:17Z

if use url pass data, can use http post with this PR feat: add TQMediaConnector plugin for TransferQueue image data support vllm-project/vllm#38355

yes, maybe need to add lifetime management is better

Thank you for your great work! I’ll study your implementation.

0oshowero0 · 2026-04-12T07:53:51Z

#5401 has been merged~

You can join our chat group for further discussion: https://raw.githubusercontent.com/TransferQueue/community_doc/refs/heads/main/other_assets/WeChatGroup.png

RobotGF requested review from ArronHZG, PeterSH6, chenhaiq and wuxibin89 as code owners March 27, 2026 12:01

gemini-code-assist Bot reviewed Mar 27, 2026

View reviewed changes

Comment thread verl/experimental/agent_loop/single_turn_agent_loop.py

Comment thread verl/utils/tq_multimodal.py Outdated

Comment thread verl/utils/tq_multimodal.py Outdated

RobotGF mentioned this pull request Mar 27, 2026

feat: add TQMediaConnector plugin for TransferQueue image data support vllm-project/vllm#38355

Closed

5 tasks

RobotGF and others added 5 commits March 27, 2026 22:46

SamitHuang reviewed Mar 31, 2026

View reviewed changes

0oshowero0 reviewed Apr 2, 2026

View reviewed changes

Uh oh!

Conversation

RobotGF commented Mar 27, 2026

What does this PR do?

Checklist Before Starting

Test

API and Usage Example

Design & Code Changes

Checklist Before Submitting

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wuxibin89 commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RobotGF commented Mar 30, 2026

Uh oh!

SamitHuang Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

SamitHuang Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

RobotGF Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

0oshowero0 commented Apr 1, 2026

Uh oh!

RobotGF commented Apr 2, 2026

Uh oh!

0oshowero0 Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

0oshowero0 Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

0oshowero0 commented Apr 2, 2026

Uh oh!

0oshowero0 commented Apr 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wuxibin89 commented Mar 30, 2026 •

edited

Loading