Under concurrency levels of 64 / 128, the program triggers a memory access fault and crashes once the cumulative number of requests exceeds
approximately 3000.
docker run -itd \
--name deepseek-v4-flash \
--device amd.com/gpu=all \
--device /dev/kfd \
--device /dev/dri \
--group-add=video \
--cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
--ipc=host \
--network=host \
--shm-size=16g \
--ulimit memlock=-1 \
--ulimit stack=67108864 \
--entrypoint=/bin/bash \
-v /data/:/data/ \
-e AITER_BF16_FP8_MOE_BOUND=0 \
-e ATOM_MOE_GU_ITLV=1 \
m.daocloud.io/docker.io/rocm/atom-dev:atom0.1.5-aiter0.1.16 \
-c "python -m atom.entrypoints.openai_server \
--model /data/DeepSeek-V4-Flash \
--kv_cache_dtype fp8 \
-tp 8 \
--gpu-memory-utilization 0.6 \
--hf-overrides '{\"use_index_cache\": true, \"index_topk_freq\": 4}' \
--method mtp --num-speculative-tokens 3 \
&>> /data/atom-log/deepseek_v4_flash.log 2>&1"
[aiter] shape is M:839, N:1024, K:4096 dtype='torch.bfloat16' otype='torch.bfloat16' bias=False, scaleAB=False, bpreshuffle=False, not found tuned config in /tmp/aiter_configs/bf16_tuned_gemm.csv, will use default config! using torch solution:0
[aiter] shape is M:839, N:1024, K:4096 dtype='torch.bfloat16' otype='torch.bfloat16' bias=False, scaleAB=False, bpreshuffle=False, not found tuned config in /tmp/aiter_configs/bf16_tuned_gemm.csv, will use default config! using torch solution:0
[aiter] shape is M:839, N:1024, K:4096 dtype='torch.bfloat16' otype='torch.bfloat16' bias=False, scaleAB=False, bpreshuffle=False, not found tuned config in /tmp/aiter_configs/bf16_tuned_gemm.csv, will use default config! using torch solution:0
[aiter] shape is M:839, N:1024, K:4096 dtype='torch.bfloat16' otype='torch.bfloat16' bias=False, scaleAB=False, bpreshuffle=False, not found tuned config in /tmp/aiter_configs/bf16_tuned_gemm.csv, will use default config! using torch solution:0
[aiter] shape is M:839, N:2048, K:4096 dtype='torch.bfloat16' otype='torch.bfloat16' bias=False, scaleAB=False, bpreshuffle=False, not found tuned config in /tmp/aiter_configs/bf16_tuned_gemm.csv, will use default config! using torch solution:0
[aiter] shape is M:839, N:512, K:4096 dtype='torch.bfloat16' otype='torch.bfloat16' bias=False, scaleAB=False, bpreshuffle=False, not found tuned config in /tmp/aiter_configs/bf16_tuned_gemm.csv, will use default config! using torch solution:0
[aiter] shape is M:839, N:8192, K:1024, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config!
[aiter] shape is M:839, N:64, K:4096 dtype='torch.bfloat16' otype='torch.bfloat16' bias=False, scaleAB=False, bpreshuffle=False, not found tuned config in /tmp/aiter_configs/bf16_tuned_gemm.csv, will use default config! using torch solution:0
[aiter] shape is M:839, N:1024, K:4096 dtype='torch.bfloat16' otype='torch.bfloat16' bias=False, scaleAB=False, bpreshuffle=False, not found tuned config in /tmp/aiter_configs/bf16_tuned_gemm.csv, will use default config! using torch solution:0
Memory access fault by GPU node-9 (Agent handle: 0x5ac6ac6a3020) on address 0x7b34b98ab000. Reason: Unknown.
Memory access fault by GPU node-4 (Agent handle: 0x5dc503d862e0) on address 0x763211ea9000. Reason: Unknown.
Memory access fault by GPU node-7 (Agent handle: 0x5f6e6fffd0c0) on address 0x7a6e097dd000. Reason: Unknown.
GPU coredump: execvp failed: No such file or directory
GPU coredump: execvp failed: No such file or directory
GPU coredump: execvp failed: No such file or directory
Failed to write program header to pipe: Broken pipe
GPU coredump: handler exited with error (status: 1)
GPU core dump failed
Failed to write program header to pipe: Broken pipe
GPU coredump: handler exited with error (status: 1)
GPU core dump failed
Failed to write program header to pipe: Broken pipe
GPU coredump: handler exited with error (status: 1)
GPU core dump failed
[atom 07:16:14] AsyncIOProcManager(ModelRunner): [ModelRunner7/8] proc died unexpectedly (exitcode=-6), shutting down.
[atom 07:16:14] AsyncIOProcManager(ModelRunner): shutdown all runners...
[rank3]:[W629 07:16:40.220947732 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=28, addr=[localhost]:52040, remote=[localhost]:38869): Connection reset by peer
Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:679 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x75102f1259f8 in /opt/venv/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x6965f51 (0x7510be9c9f51 in /opt/venv/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x1cc (0x7510be9c5e5c in /opt/venv/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c7 (0x7510a58adab7 in /opt/venv/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
frame #4: <unknown function> + 0xdc253 (0x75102d430253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #5: <unknown function> + 0x94ac3 (0x7510cee83ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: <unknown function> + 0x1268d0 (0x7510cef158d0 in /lib/x86_64-linux-gnu/libc.so.6)
[rank3]:[W629 07:16:40.225439301 ProcessGroupNCCL.cpp:1804] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Connection reset by peer
[atom 07:16:41] AsyncIOProcManager(ModelRunner): All runners are shutdown.
Problem Description
On 8*MI355X with
rocm/atom-dev:atom0.1.5-aiter0.1.16and DeepSeek-V4-Flash.Under concurrency levels of 64 / 128, the program triggers a memory access fault and crashes once the cumulative number of requests exceeds
approximately 3000.
Operating System
Ubuntu 24.04.2 LTS (Noble Numbat)
CPU
2 x AMD EPYC 9575F 64-Core Processor
GPU
8 x AMD Instinct MI355X
ROCm Version
ROCm 7.2.4
ROCm Component
No response
Steps to Reproduce
Serve configuration
Stack trace
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
rocminfo --support output
Additional Information
No response