[NV][AgentX] Add Kimi-K3 B300 vLLM DSpark sweep / [NV][AgentX] 新增 Kimi-K3 B300 vLLM DSpark 并发扫描 - #2616
[NV][AgentX] Add Kimi-K3 B300 vLLM DSpark sweep / [NV][AgentX] 新增 Kimi-K3 B300 vLLM DSpark 并发扫描#2616xinli-sw wants to merge 13 commits into
Conversation
…ooncake Move the recipe to TP8 x DCP8 with Mooncake as the external KV tier, on an image that carries the Kimi-K3 DCP, DSpark-under-DCP and Mooncake hybrid-attention changes. Sweep concurrency 1-70, drafting with DSpark level 7 at 8 and below. 将该配方迁移到 TP8 x DCP8，并以 Mooncake 作为外部 KV 层，所用镜像已包含 Kimi-K3 DCP、 DCP 下的 DSpark 以及 Mooncake 混合注意力相关改动。并发扫描 1-70，其中并发 8 及以下使用 DSpark level 7。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Claude finished @xinli-sw's task in 5m 53s —— View job PR Review
SummaryNo blocking issues found. One 🟡 warning posted inline: the refresh drops Checks that passed:
One note: the image moved from a digest-pinned nightly to the tag |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| --speculative-config "$SPEC_CONFIG" | ||
| --stream-interval 10 | ||
| --attention-backend TOKENSPEED_MLA | ||
| --attention-config '{"mla_prefill_backend":"TRTLLM_RAGGED","use_prefill_query_quantization":true}' |
There was a problem hiding this comment.
🟡 WARNING: This refresh drops --reasoning-parser kimi_k3, --tool-call-parser kimi_k3, and --enable-auto-tool-choice, which the previous version of this script had and which every other vLLM agentic MTP recipe retains (dsv4_fp4_b300_vllm_mtp.sh:317-319, minimaxm3_fp8_h200_mtp.sh:192-194 — the latter also uses --language-model-only, so that flag doesn't preclude the parsers).
Why it matters: The throughput replay discards live responses, so it's unaffected — but the EVAL_ONLY SWE-bench path consumes live completions via mini-swe-agent. Without a reasoning parser, Kimi-K3's thinking tokens stay inline in content, where they can confuse the agent's action-block parsing and depress or zero the eval score (the same failure mode the synthetic-acceptance split exists to avoid).
Fix: If the agentx-k3 fork image genuinely can't register these parsers, note that in the header comment; otherwise restore them:
| --attention-config '{"mla_prefill_backend":"TRTLLM_RAGGED","use_prefill_query_quantization":true}' | |
| --attention-config '{"mla_prefill_backend":"TRTLLM_RAGGED","use_prefill_query_quantization":true}' | |
| --reasoning-parser kimi_k3 | |
| --tool-call-parser kimi_k3 | |
| --enable-auto-tool-choice |
| MAX_NUM_SEQS=$((2 * CONC)) | ||
|
|
||
| # With spec decoding, cudagraph capture sizes are in TOKENS. vLLM rounds | ||
| # configured sizes up to multiples of (1 + num_speculative_tokens) and dedups | ||
| # them (adjust_cudagraph_sizes_for_spec_decode), so a plain 1..MAX_NUM_SEQS list | ||
| # would collapse to covering only MAX_NUM_SEQS/(1+N) sequences and drop the | ||
| # largest decode batches to eager. Enumerate the multiples explicitly so there | ||
| # is one graph per decode batch of 1..MAX_NUM_SEQS sequences. The graph mode is | ||
| # left at the recipe default that the non-MTP K3 arm already validated. | ||
| CUDA_GRAPH_CAPTURE_SIZES="" | ||
| for ((num_seqs = 1; num_seqs <= MAX_NUM_SEQS; num_seqs++)); do | ||
| if [ -n "$CUDA_GRAPH_CAPTURE_SIZES" ]; then | ||
| CUDA_GRAPH_CAPTURE_SIZES+="," | ||
| # Capture sizes: step * 1..min(max-num-seqs, 128), then the fixed powers of two | ||
| # above that. Sizes are tokens when drafting, sequences when not; the 128 caps | ||
| # the number of dense entries, not their value. | ||
| CAPTURE_STEP=$((1 + NUM_SPEC_TOKENS)) | ||
| DENSE_COUNT=$MAX_NUM_SEQS | ||
| if [ "$DENSE_COUNT" -gt 128 ]; then | ||
| DENSE_COUNT=128 | ||
| fi | ||
| CAPTURE_SIZES="" | ||
| for ((n = 1; n <= DENSE_COUNT; n++)); do | ||
| CAPTURE_SIZES+="${CAPTURE_SIZES:+,}$((n * CAPTURE_STEP))" | ||
| done | ||
| DENSE_MAX=$((DENSE_COUNT * CAPTURE_STEP)) | ||
| for t in 64 128 256 512 1024 2048 4096 8192; do | ||
| if [ "$t" -gt "$DENSE_MAX" ]; then | ||
| CAPTURE_SIZES+=",$t" |
There was a problem hiding this comment.
🟡 For the no-drafting arm (CONC>16), the CUDA-graph capture list is dense only up to 128 entries then jumps to fixed powers of two (256, 512, ...); at the newly-added CONC=70, MAX_NUM_SEQS=140 falls in the uncaptured 129-255 gap, so the largest decode batches at this arm silently pad to the 256-wide graph (~1.8x wasted compute). A simple fix is to append MAX_NUM_SEQS to the capture-size list explicitly.
Extended reasoning...
The bug
kimik3_fp4_b300_vllm_mtp.sh:207-224 builds the CUDA-graph capture-size list as 1..min(MAX_NUM_SEQS, 128) densely, then appends only the fixed powers of two (64, 128, 256, ...) that exceed DENSE_MAX. For the no-drafting arm (CONC > 16, so CAPTURE_STEP = 1), MAX_NUM_SEQS = 2*CONC. The new sweep entry CONC=70 (added in configs/nvidia-master.yaml) gives MAX_NUM_SEQS=140, but the dense list stops at 128 and the next fixed size is 256 — so nothing in [129, 255] is ever captured. CONC=70 is the only value in the new conc-list [1,2,4,8,16,24,32,40,48,56,70] where 2*CONC exceeds 128; every other value (1..56) stays fully dense.
Why this is reachable, not just headroom
The verifier refutation argues this gap is harmless because MAX_NUM_SEQS = 2*CONC is unused "scheduler headroom" that a CONC=70 agentic workload never actually reaches — i.e., the real running-sequence count stays near CONC (~70), safely inside the dense range. That reasoning is contradicted by this repos own established convention for this exact line. The sibling recipes dsv4_fp4_b300_vllm_mtp.sh:258, dsv4_fp4_b200_vllm_mtp.sh:234-236, and dsv4_fp4_mi355x_vllm_mtp.sh:374-376 all set MAX_NUM_SEQS = 2 * CONC with the explicit comment: "AgentX concurrency counts live session trees, not individual requests. Subagent fan-out can push instantaneous request concurrency above CONC, so leave 2x headroom rather than clipping those bursts at the scheduler." That is, 2*CONC is not a decorative bound — it is sized specifically because real bursts of concurrently-running sequences are expected to approach it. If the 129-140 region were truly unreachable, the recipe would not need 2*CONC scheduler capacity there in the first place.
Why the existing code doesnt prevent it
Prior to this PR, this same script enumerated cudagraph capture sizes as a plain 1..MAX_NUM_SEQS loop with no cap (visible in the diffs removed lines, and still how dsv4_fp4_b300_vllm_mtp.sh:279-282 does it today) — every decode batch size got an exact graph. This PR introduces the DENSE_COUNT cap at 128 for the first time, for legitimate reasons (a 93-layer 2.8T model makes capturing the full ladder slow), but the cap only tracks a count, not MAX_NUM_SEQS itself, so it silently drops coverage whenever 2*CONC > 128. Nothing in the loop checks whether MAX_NUM_SEQS itself ended up outside the capture set.
Proof by walkthrough
CONC=70, no-drafting arm ->NUM_SPEC_TOKENS=0->CAPTURE_STEP = 1 + 0 = 1.MAX_NUM_SEQS = 2*70 = 140.DENSE_COUNT = min(140, 128) = 128-> dense sizes are1,2,...,128(step 1) ->DENSE_MAX = 128.- The fixed-size loop appends only
t > 128from{64,128,256,512,...}, i.e. it appends256next. - Resulting capture set:
{1,...,128} ∪ {256,512,1024,2048,4096,8192}— nothing in[129,255]. - At runtime,
--max-num-seqs 140lets the scheduler admit decode batches up to 140 sequences; a batch of, say, 135 sequences has no matching entry and vLLM pads it to the next captured size, 256 (~1.9x the tokens actually needed), or falls back to piecewise/eager for that step. - This only happens for
CONC=70; every other new sweep point (1,2,4,8,16,24,32,40,48,56) has2*CONC <= 112 <= 128, staying fully dense.
Impact and fix
This does not crash the run or corrupt output (256 exists, so it pads rather than erroring), and it only affects the top of the batch-size range at a single concurrency point in the sweep, so nit rather than blocking severity is appropriate. But it does mean the measured throughput at CONC=70 — the top and newest point in this sweep, where accuracy matters most for capacity planning — is measured with some fraction of decode steps running on an oversized, less efficient graph. The fix is small: explicitly append MAX_NUM_SEQS to CAPTURE_SIZES if it is not already in the dense range (or raise DENSE_COUNTs cap to track MAX_NUM_SEQS when it modestly exceeds 128).
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31867498343 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31872521583 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31873444383 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31873444383 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31890409571 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31890585094 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31893747354 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31893747354 |
|
/reuse-sweep-run |
|
/reuse-sweep-run 31893747354 |
…cake # Conflicts: # perf-changelog.yaml
xinli-sw
changed the title
Ankur-singh
changed the title
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:Scope. This PR moves Validation and eval evidence, and the head it came from — read this before relying on the links above. The linked run 31893747354 executed at commit Two facts make the reuse substantive rather than nominal, and both are independently checkable. First, Eval scope. The single Speculative decoding — acceptance lengths match the golden curve. Draft length is selected by concurrency, and the pinned acceptance tracks it in both drafted arms: concurrency <= 8 runs DSpark level 7 with Spec-decode eval guard — satisfied in-script. Recorded exception — six of the eleven published points run without drafting while the master entry declares Recorded exception — the tool-call and reasoning parsers were removed. This PR deletes Recorded exception — the image lost its digest pin. The previous image was digest-pinned ( Single-node recipe publication. This config is single-node vLLM, so the publication requirement applies and is met by the already-published upstream recipe Model and scenario scope, architecture, and patching. MODELS.md lists Kimi-K3 as active for agentic coding with the native/upstream vLLM engine and the Append-only item — not applicable. The new Recorded exception — creator-preflight findings that are pre-existing. Mergeability. The PR is Signed: |
❌❌❌ REJECTED ❌❌❌@Ankur-singh — two blockers: (1) six of the eleven published points (conc 24–70) launch with no ✅ Check 0 (CODEOWNER): PASS — |
4 participants
Description
Move the B300 Kimi-K3 NVFP4 AgentX DSpark recipe to a TP8 × DCP8 layout with
Mooncake as the external KV tier, and extend the concurrency sweep to eleven
points.
--decode-context-parallel-size 8and--dcp-comm-backend a2a; TP8 remains the only single-node layout because the~1.5 TB MXFP4 checkpoint does not fit below 8 GPUs.
SimpleCPUOffloadConnectorwithMooncakeStoreConnector(
mooncake-transfer-engine-cuda13==0.3.11.post1), sized fromdram-utilization: 0.75divided across the 8 TP ranks. The RDMA rail ischosen at runtime from the first ACTIVE device, since
mlx5_0is down on somenodes and a hardcoded rail has no fallback.
TOKENSPEED_MLAfor decode withTRTLLM_RAGGEDMLA prefill andprefill query quantisation, under fp8 KV cache.
tracking the draft length in each arm:
search-space entry. The arms are kept disjoint in concurrency because the
exp-name carries concurrency and spec but not the arm.
vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-5894fdf, which carriesthe Kimi-K3 DCP, DSpark-under-DCP and Mooncake-hybrid changes
(
vllm-project/vllmagentx-k3@5894fdf).Both acceptance lengths come from the committed golden curve
golden_al_distribution/kimik3_dspark_probabilistic_sample_method_block_rejection_sample_method.yaml(
thinking_on: 7 → 3.84, 3 → 3.00), which is the curve measured withdraft_sample_method=probabilisticandrejection_sample_method=block— thedraft configuration this recipe runs. Throughput jobs pin synthetic acceptance;
EVAL_ONLY=trueswitches to real block verification, since synthetic acceptancecommits drafted tokens regardless of the target's logits and would zero the eval
score.
Benchmark evidence. Run 31893747354
executed all eleven
agentic /points plus theagentic eval /job green atcommit
630dc8c73. The branch head then advanced to8f8185deby mergingmain(which brought in the MiniMax-M3 GB200 work from #2609); no
kimik3filediffers between the two commits, and reuse of that run is authorised on this PR.
中文说明
将 B300 Kimi-K3 NVFP4 AgentX DSpark 配方迁移到 TP8 × DCP8 布局，采用 Mooncake 作为
外部 KV 层，并将并发扫描扩展为十一个测试点。
--decode-context-parallel-size 8与--dcp-comm-backend a2a；由于 ~1.5 TB 的 MXFP4 权重无法在少于 8 卡的情况下装下，TP8 仍是唯一的单节点布局。
MooncakeStoreConnector取代SimpleCPUOffloadConnector（
mooncake-transfer-engine-cuda13==0.3.11.post1），容量由dram-utilization: 0.75在 8 个 TP rank 间均分。RDMA 通道在运行时从第一个 ACTIVE 设备中选取，因为部分节点的
mlx5_0处于关闭状态，硬编码通道没有回退路径。TOKENSPEED_MLA，预填充使用TRTLLM_RAGGED并启用查询量化，KV 缓存为 fp8。
条目。由于 exp-name 只携带并发与 spec 而不含分支信息，各分支的并发保持互不重叠。
vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-5894fdf，其中包含 Kimi-K3 的DCP、DCP 下的 DSpark 以及 Mooncake 混合改动（
vllm-project/vllmagentx-k3@5894fdf）。两个接受长度均取自已提交的黄金曲线
golden_al_distribution/kimik3_dspark_probabilistic_sample_method_block_rejection_sample_method.yaml（
thinking_on：7 → 3.84，3 → 3.00），该曲线正是在draft_sample_method=probabilistic与rejection_sample_method=block下测得，与本配方的草稿配置一致。吞吐作业固定合成接受率；
EVAL_ONLY=true时切换为真实的 block校验，因为合成接受会不顾目标模型 logits 直接采纳草稿 token，会使评测分数归零。
基准测试证据：Run 31893747354
在提交
630dc8c73上完整执行了全部十一个agentic /测试点以及agentic eval /作业，且全部通过。随后分支头因合并
main（引入了 #2609 的 MiniMax-M3 GB200 改动）推进到8f8185de；两个提交之间没有任何kimik3文件发生变化，且本 PR 已授权复用该次运行。Related Issue
N/A
Type of Change
Checklist
630dc8c73perf-changelog.yamlentry added forkimik3-fp4-b300-vllm-agentic-dsparkEVAL_ONLYruns use real block verification rather than synthetic acceptance8f8185de— the current-head run reuse-gated, so its benchmark and eval leaves are skipped; the evidence above is from630dc8c73, which is byte-identical for everykimik3path