Summary
A tensor-parallel vLLM deployment could stop generating tokens while using HAMi-core GPU sharing. The affected processes remained alive, but progress stopped after process-slot cleanup in the shared memory region.
The issue was first observed while starting and running vLLM with tensor parallelism set to 2. Investigation showed that compaction of dead process slots can leave a stale cached slot valid by PID, even though that slot is no longer
inside the active proc_num range. Later status or memory updates may then target an inactive slot, causing accounting to be lost and status synchronization to stall.
Environment
- HAMi-core base revision:
52f33fc
- GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition (SM 12.0)
- NVIDIA driver: 580.159.03
- HAMi-core build CUDA version: 12.9.2
- Runtime CUDA version: 13.0
- PyTorch: 2.11.0+cu130
- vLLM: 0.25.1
- FlashInfer: 0.6.13
- Workload: Qwen3.6-27B-NVFP4, tensor parallel size 2
- Two virtual GPUs with 60 GiB memory each
Steps to reproduce
- Run a vLLM workload with
--tensor-parallel-size 2 through HAMi-core GPU sharing.
- Ensure that multiple CUDA processes share the same HAMi-core memory region.
- Allow one process to exit while other processes remain active.
- Trigger cleanup of the dead process slot.
- Continue CUDA activity in a process whose slot may have been moved during compaction.
Actual behavior
The workload can stop making progress: vLLM no longer generates tokens, while the involved processes remain alive. Memory accounting and process-status synchronization can target an inactive shared-memory slot.
Expected behavior
After compaction, every active process must resolve to its slot within the active process range. CUDA work, memory accounting, and process-status synchronization should continue normally.
Proposed fix
- Clear the original source slot after moving the last live slot during compaction.
- Validate a cached process slot against the active
proc_num range, not only by PID.
- Re-discover the process slot in the active range when the cached slot is stale.
- Skip process-status waiting when GPU SM limiting is disabled.
Validation performed
- Built the modified library successfully.
- Ran the existing CUDA hook consistency check successfully.
- Verified the updated library in a TP=2 vLLM deployment.
- Observed the workload under load with
CUDA_DEVICE_SM_LIMIT=50, FLASH_ATTN, and CUDA Graphs disabled (cudagraph_mode=NONE). During the observation window, the service remained ready, produced tokens, and had no worker crash or
engine failure.
Summary
A tensor-parallel vLLM deployment could stop generating tokens while using HAMi-core GPU sharing. The affected processes remained alive, but progress stopped after process-slot cleanup in the shared memory region.
The issue was first observed while starting and running vLLM with tensor parallelism set to 2. Investigation showed that compaction of dead process slots can leave a stale cached slot valid by PID, even though that slot is no longer
inside the active
proc_numrange. Later status or memory updates may then target an inactive slot, causing accounting to be lost and status synchronization to stall.Environment
52f33fcSteps to reproduce
--tensor-parallel-size 2through HAMi-core GPU sharing.Actual behavior
The workload can stop making progress: vLLM no longer generates tokens, while the involved processes remain alive. Memory accounting and process-status synchronization can target an inactive shared-memory slot.
Expected behavior
After compaction, every active process must resolve to its slot within the active process range. CUDA work, memory accounting, and process-status synchronization should continue normally.
Proposed fix
proc_numrange, not only by PID.Validation performed
CUDA_DEVICE_SM_LIMIT=50,FLASH_ATTN, and CUDA Graphs disabled (cudagraph_mode=NONE). During the observation window, the service remained ready, produced tokens, and had no worker crash orengine failure.