deepseek_v4_flash-b200: align serve args with recipe + fix cu13 TileLang JIT#27
Open
khluu wants to merge 6 commits into
Open
deepseek_v4_flash-b200: align serve args with recipe + fix cu13 TileLang JIT#27khluu wants to merge 6 commits into
khluu wants to merge 6 commits into
Conversation
Some vLLM release images don't include pytest, but cupy.testing._random imports it unconditionally. When torch.compile traces through cupy modules during model loading, this causes a ModuleNotFoundError that crashes the server. Installing pytest after container startup fixes this. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Switch the DeepSeek-V4-Flash B200 workload to the published vLLM recipe serve args: TP=8 + expert parallel (was TP=2 × DP=4), and drop the --max-model-len 32768 and MTP speculative-decode flags that aren't part of the recipe. deep_gemm_mega_moe MoE backend and fp4 indexer cache are retained. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…chain
Build 235 (and nightly build 233) fail at vLLM engine startup on B200: the
release image ships a mismatched nvidia-cu13 set (nvcc/cccl vs cuda runtime
headers), so the DeepSeek-V4 mHC TileLang kernel can't JIT-compile for sm_100a
("CUDA compiler and CUDA toolkit headers are incompatible"). This is an image
packaging bug (cf. vllm-project/vllm#44305), not a workload issue, and there is
no serve-arg/env workaround — MHCPreOp.forward_cuda hardcodes the tilelang op
with enabled() pinned True, so the torch fallback is unreachable.
Set nightly: false so the nightly stops going red on a known, currently
unfixable-from-our-side image bug. Still triggerable via WORKLOADS; re-enable
when a vLLM image with an aligned cu13 toolchain ships.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
khluu
changed the title
The default VLLM_IMAGE is an x86_64 CUDA 12.9 build, but Blackwell/B200 needs a CUDA 13 image (DeepSeek-V4's mHC TileLang kernels JIT-compile against the image's CUDA toolchain at startup). Add a generic per-profile image override: a GPU profile may declare `image_env: <VAR>`, and when that var is set it takes precedence over VLLM_IMAGE / VLLM_COMMIT for that profile's steps only. B200 sets `image_env: VLLM_IMAGE_CU13`, so a CUDA 13 image can be routed to Blackwell while H200/AMD keep using VLLM_IMAGE. Both image-resolution paths honor it: generate_pipeline.py bakes it into the K8s pod spec, and parse_workload.py uses it in-job (deriving the recorded commit from the cu13 image tag). The var is propagated to the step env so the in-job parser records the image/commit that actually ran. Kept generic per CLAUDE.md (GPU specifics live in gpu_profiles.yaml, not the pipeline code). README updated: new optional env var + the profile `image_env` key. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
khluu
changed the title
This reverts commit c01870f.
…Y_CHECK
The B200 failure isn't a CUDA-major problem (the image is already CUDA 13).
The image's nvidia-cu13 wheels are minor-skewed (nvcc minor != cudart-headers
minor), so the DeepSeek-V4 mHC TileLang kernel's sm_100a JIT compile trips
CCCL's nvcc-vs-headers guard in cuda_toolkit.h ("CUDA compiler and CUDA toolkit
headers are incompatible").
CCCL ships an official opt-out for exactly this ("use a newer CTK than the
compiler ships"): define CCCL_DISABLE_CTK_COMPATIBILITY_CHECK. nvcc honors
NVCC_APPEND_FLAGS and TileLang's compile subprocess inherits os.environ on
Linux, so set NVCC_APPEND_FLAGS=-DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK in the
workload env to skip the (minor-only, same CUDA 13 major) check. Un-park the
workload (nightly: true) now that it has a real fix.
Drop the env override once the image ships version-aligned cu13 wheels.
See vllm-project/vllm#44305.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
khluu
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two changes to the
deepseek_v4_flash-b200workload (net diff vsmainis just this one file):vllm.serve_argswith the published vLLM recipe — TP=8 + EP.vllm.env.1. Serve-args changes
--tensor-parallel-size 2 --data-parallel-size 4→--tensor-parallel-size 8(still--enable-expert-parallel)--max-model-len 32768and the MTP spec-decode flags--moe-backend deep_gemm_mega_moe,--attention_config.use_fp4_indexer_cache=True, and the deepseek_v4 tokenizer/tool-call/reasoning parsers2. cu13 TileLang JIT fix
Builds #233 and #235 failed at engine startup: the DeepSeek-V4 mHC TileLang kernel JIT-compiles for
sm_100aand aborts with "CUDA compiler and CUDA toolkit headers are incompatible."Root cause is not a CUDA-major mismatch — the image is already CUDA 13. The image's
nvidia-cu13wheels are minor-skewed (nvccminor ≠cudartheader minor), and CCCL'scuda_toolkit.hguard rejects any minor mismatch. Same root cause as vllm#44305.CCCL provides an official opt-out for exactly this case ("some users might want to use a newer CTK than the compiler ships"): define
CCCL_DISABLE_CTK_COMPATIBILITY_CHECK.nvcchonorsNVCC_APPEND_FLAGS, and TileLang's compile subprocess inheritsos.environon Linux, so the workload sets:This only disables a minor-version equality check within the same CUDA 13 major (low risk; it's the documented use case). Drop the override once the image ships version-aligned cu13 wheels.
The workload is un-parked (
nightly: true) now that it has a real fix.Notes
VLLM_IMAGE_CU13per-GPU image override; that was based on a wrong premise (the image is already CUDA 13, so a different image isn't the fix) and has been reverted — no pipeline/profile changes remain.Testing
NVCC_APPEND_FLAGS=-DCCCL_DISABLE_CTK_COMPATIBILITY_CHECKis emitted inWORKLOAD_ENV, merged over the B200 profile env;server.sh's native path exports it cleanly.WORKLOAD_BENCH_TPreflects 8.This PR was authored with assistance from Claude Code (AI-assisted).
🤖 Generated with Claude Code