Error with vllm

[chenhr18thu]

No relation to package version.

[chenhr18thu]

No relation to config.

[ultmaster]

It's a known issue and I think there's a candidate solution proposed by community folks. I'll submit a PR and you can test whether that works.

[chenhr18thu]

@ultmaster This issue has troubled my team quite a lot. I search memory solutions in "pull request". I did not find the solution you mentioned. could you link that solution to this issue?

[chenhr18thu]

@ultmaster Thank you for your kind reply!

[ultmaster]

Hold on a sec. I'm just looking for the solution from the chat and developing the PR.

[ultmaster]

I've took a look and it's not as easy as I originally thought.

First part: The related issue is: vllm-project/vllm#17103

This sleep() is called within verl; and I found that in v0.6.1, reset_prefix_cache() is already called:

So that's not the real problem although I couldn't confirm it for sure that verl is executed in this colocated path.

Second part: The community folks (from Tencent) claims that before that sleep, we need to actually abort the running requests (FINISHED_ABORTED). After that, we can proceed to reset_prefix_cache and sleep to release the memory. The code snippet should be inserted into vLLM. I think they are suggesting some changes like this:

(to copy the code:)

        from typing import cast
        from vllm.v1.core.sched.scheduler import Scheduler
        scheduler = cast(Scheduler, self.scheduler)

        requests_to_abort: list[str] = []
        if self.scheduler.has_unfinished_requests():
            for request in scheduler.waiting:
                requests_to_abort.append(request.request_id)  # is this usage correct???
            for request in scheduler.running:
                requests_to_abort.append(request.request_id)

            self.abort_requests(requests_to_abort)

I'm not very sure about this; and I really need your feedback before we can do anything. The issue is not very reproducible at our side.

[chenhr18thu]

Thank you! The proposed solution works fine.
The bug is from vllm, which is unexpected since we consider it as a mature framework.

[chenhr18thu]

Sadly, after 20 steps training, this bug still exists.

Training Progress: 0%| | 21/10200 [6:15:15<3031:35:56, 1072.18s/it]
�[36m(WorkerDict pid=6244)�[0m model_output = self.model(
�[36m(WorkerDict pid=6244)�[0m hidden_states = self.model(input_ids, positions, intermediate_tensors,
�[36m(WorkerDict pid=6244)�[0m File "/mnt/tenant-home_speed/chr/conda_env/lightning_flash/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 173, in call
�[36m(WorkerDict pid=6244)�[0m return self.forward(*args, **kwargs)
�[36m(WorkerDict pid=6244)�[0m hidden_states, residual = layer(
�[36m(WorkerDict pid=6244)�[0m hidden_states = self.mlp(hidden_states)
�[36m(WorkerDict pid=6244)�[0m gate_up, _ = self.gate_up_proj(x)
�[36m(WorkerDict pid=6244)�[0m output_parallel = self.quant_method.apply(self, input_, bias)
�[36m(WorkerDict pid=6244)�[0m File "/mnt/tenant-home_speed/chr/conda_env/lightning_flash/lib/python3.10/site-packages/vllm/model_executor/layers/linear.py", line 226, in apply
�[36m(WorkerDict pid=6244)�[0m return dispatch_unquantized_gemm()(layer, x, layer.weight, bias)
�[36m(WorkerDict pid=6244)�[0m File "/mnt/tenant-home_speed/chr/conda_env/lightning_flash/lib/python3.10/site-packages/vllm/model_executor/layers/utils.py", line 70, in default_unquantized_gemm
�[36m(WorkerDict pid=6244)�[0m return torch.nn.functional.linear(x, weight, bias)
�[36m(WorkerDict pid=6244)�[0m RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16BF, lda, b, CUDA_R_16BF, ldb, &fbeta, c, CUDA_R_16BF, ldc, compute_type, CUBLAS_GEMM_DEFAULT_TENSOR_OP)
�[36m(TaskRunner pid=5909)�[0m INFO:2025-12-12 20:16:24,658:127.0.0.1 - - [12/Dec/2025 20:16:24] "�[35m�[1mPOST /v1/chat/completions HTTP/1.1�[0m" 503 -
�[36m(TaskRunner pid=5909)�[0m INFO:2025-12-12 20:16:26,902:127.0.0.1 - - [12/Dec/2025 20:16:26] "�[35m�[1mPOST /v1/chat/completions HTTP/1.1�[0m" 503 -
�[36m(TaskRunner pid=5909)�[0m INFO:2025-12-12 20:16:27,296:127.0.0.1 - - [12/Dec/2025 20:16:27] "�[35m�[1mPOST /v1/chat/completions HTTP/1.1�[0m" 503 -
�[36m(TaskRunner pid=5909)�[0m INFO:2025-12-12 20:16:28,239:127.0.0.1 - - [12/Dec/2025 20:16:28] "�[35m�[1mPOST /v1/chat/completions HTTP/1.1�[0m" 503 -
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] File "/mnt/tenant-home_speed/chr/conda_env/lightning_flash/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] return func(*args, **kwargs)
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] File "/mnt/tenant-home_speed/chr/conda_env/lightning_flash/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1430, in execute_model
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] sampler_output = self.sampler(
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] File "/mnt/tenant-home_speed/chr/conda_env/lightning_flash/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] return self._call_impl(*args, **kwargs)
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] File "/mnt/tenant-home_speed/chr/conda_env/lightning_flash/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] return forward_call(*args, **kwargs)
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] File "/mnt/tenant-home_speed/chr/conda_env/lightning_flash/lib/python3.10/site-packages/vllm/v1/sample/sampler.py", line 54, in forward
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] sampled = self.sample(logits, sampling_metadata)
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] File "/mnt/tenant-home_speed/chr/conda_env/lightning_flash/lib/python3.10/site-packages/vllm/v1/sample/sampler.py", line 121, in sample
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] random_sampled = self.topk_topp_sampler(
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] File "/mnt/tenant-home_speed/chr/conda_env/lightning_flash/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] return self._call_impl(*args, **kwargs)
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] File "/mnt/tenant-home_speed/chr/conda_env/lightning_flash/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] return forward_call(*args, **kwargs)
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] File "/mnt/tenant-home_speed/chr/conda_env/lightning_flash/lib/python3.10/site-packages/vllm/v1/sample/ops/topk_topp_sampler.py", line 81, in forward_native
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] logits = apply_top_k_top_p(logits, k, p)
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] File "/mnt/tenant-home_speed/chr/conda_env/lightning_flash/lib/python3.10/site-packages/vllm/v1/sample/ops/topk_topp_sampler.py", line 187, in apply_top_k_top_p
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] logits_sort, logits_idx = logits.sort(dim=-1, descending=False)
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] RuntimeError: CUDA error: an illegal memory access was encountered
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
�[36m(WorkerDict pid=6438)�[0m ERROR 12-12 20:16:23 [worker_base.py:622] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.