fix: NIXL KV transfer patches for Kimi PD disaggregated serving#1
Open
wz1qqx wants to merge 4 commits into
Open
fix: NIXL KV transfer patches for Kimi PD disaggregated serving#1wz1qqx wants to merge 4 commits into
wz1qqx wants to merge 4 commits into
Conversation
added 4 commits
March 20, 2026 22:59
Cherry-picked from vllm-kimi2.5-pd (v0.17.1 based): - fix: keep engine active for pending KV transfers - fix: add post-transfer explicit notification for NIXL KV connector - fix: correct MLA pre-transfer notification request ID - perf: revert GIL-yield sleep to 0.001s
…agnostics 1. Handshake listener: handle non-3-frame ZMQ messages gracefully instead of crashing the thread (fixes ValueError on recv_multipart) 2. Add [DIAG] logs throughout KV transfer pipeline: - build_connector_meta: queue depths (recv/send/save/batch) - request_finished(P): block delay-free tracking - start_load_kv: batch summary + pending state - _nixl_handshake: connection timing - _read_blocks: RDMA READ posting with full context - get_finished: transfer pipeline state per step
Replace verbose [DIAG] diagnostic logs with minimal [KV]-prefixed logs on the NIXL transfer critical path. Each log carries req_id for easy request-level tracing via: grep '\[KV\]' | grep 'req=<id>' Kept (4 critical path events): [KV] request_finished(P): req=..., blocks=... (P: prefill done) [KV] handshake completed: engine=..., Xs (D: first connect) [KV] RDMA READ posted: req=..., blocks=... (D: transfer submitted) [KV] send_done(P) / recv_done(D): req=... (transfer complete) Removed (investigation-only, not critical path): - build_connector_meta queue depths - _nixl_handshake connecting (kept completed) - start_load_kv batch summary - start_load_kv per-request (reverted to debug) - MLA send_notif debug tracking - get_finished pending pipeline state
Cherry-pick upstream fix vllm/vllm#36553 (006aea1). The assert fails when DP dummy batch has padding requests with query_lens=0, triggered more frequently in KVBM PD mode.
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run You ask your reviewers to trigger select CI tests on top of Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. 🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Patches on top of vLLM v0.16.0 for Dynamo PD disaggregated serving with NIXL KV transfer, addressing KV cache leak and stability issues discovered during Kimi-K2.5 deployment.
Changes
MLA notification request ID fix (
nixl_connector.pyline ~2228)send_notifwas using decode-sidereq_idinstead ofmeta.remote.request_idScheduler KV send tracking (
scheduler.py)_pending_kv_send_countto track requests waiting for decode to fetch KV blockshas_finished_requests()now returns True when pending KV sends exist, preventing engine idle blockingEngine GIL yield fix (
core.py)has_unfinished_requests()tohas_requests()in GIL-yield conditionHandshake defensive handling (
nixl_connector.py)KV transfer diagnostic logs (
nixl_connector.py)[KV] request_finished(P)/send_done(P)/recv_done(D)/RDMA READ postedreqs_to_sendsampleAttention assert fix (
utils.py)split_decodes_and_prefills(upstream [BugFix] Remove incorrect assert in split_decodes_and_prefills vllm-project/vllm#36553)Files changed (4)
nixl_connector.pyscheduler.py_pending_kv_send_counttrackingcore.pyutils.pyTest plan
gpu_cache_usage_perccorrectly decreases