[Bug]: [SSD] DSV4 enable SSD offload，stress testing, repeat offloading the same batch key, lead to OBJECT_ALREADY_EXISTS/persist failed/INVALID_KEY

### Bug Report

# 测试场景：
DeepSeekV4模型，PD分离，4机，1P1D，P和D都是双机。

## P节点的1台机器的vllm启动脚本。
```shell
unset ftp_proxy
unset https_proxy
unset http_proxy
rm -rf ~/ascend/log
rm -rf /opt/XXX/Test/vLLM/mooncake-ssd/logs/*
#!/bin/bash
TIMESTAMP=$(date +"%Y%m%d%H%M")

#export HCCL_INTRA_PCIE_ENABLE=1
#export HCCL_INTRA_ROCE_ENABLE=0

MASTER_IP_ADDRESS="x.xxx.xx.xx"
IP_ADDRESS="x.xxx.xx.xx"
SERVICE_PORT=10000
LOG_FILE="/opt/XXX/Test/vLLM/mooncake-ssd/logs/deepseekv4_start_prefill_${TIMESTAMP}.log"

NETWORK_INTERFACE="eth0"
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
PREFILL_DATA_PARALLEL_RANK=0

PREFILL_DATA_PARALLEL_SIZE=2
PREFILL_TENSOR_PARALLEL_SIZE=8

MODEL_PATH="/opt/Dsv4-flash-bf16"
MODEL_NAME="DeepSeek-V4-Flash"

# # mooncake
export MOONCAKE_CONFIG_PATH="/opt/XXX/Test/vLLM/mooncake-ssd/configs/mooncake.json"
export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib64:${LD_LIBRARY_PATH}
export PYTHONHASHSEED=0
export HCCL_INTRA_ROCE_ENABLE=1
export ASCEND_CONNECT_TIMEOUT=10000
export ASCEND_TRANSFER_TIMEOUT=10000

# # jemalloc
export LD_PRELOAD=/usr/lib64/libjemalloc.so.2:$LD_PRELOAD
# # AIV
export HCCL_OP_EXPANSION_MODE="AIV"
export TASK_QUEUE_ENABLE=1
export VLLM_RPC_TIMEOUT=3600000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=1200
export HCCL_CONNECT_TIMEOUT=1200

export HCCL_IF_IP=$IP_ADDRESS
export GLOO_SOCKET_IFNAME=$NETWORK_INTERFACE
export TP_SOCKET_IFNAME=$NETWORK_INTERFACE
export HCCL_SOCKET_IFNAME=$NETWORK_INTERFACE
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=1024

#export VLLM_VERSION=0.20.2
export VLLM_ASCEND_APPLY_DSV4_PATCH=1
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
#export USE_MULTI_GROUPS_KV_CACHE=1
#export USE_MULTI_BLOCK_POOL=1

vllm serve "$MODEL_PATH" \
--host "$IP_ADDRESS" \
--port "$SERVICE_PORT" \
--data-parallel-size "$PREFILL_DATA_PARALLEL_SIZE" \
--data-parallel-rank "$PREFILL_DATA_PARALLEL_RANK" \
--data-parallel-address "$MASTER_IP_ADDRESS" \
--data-parallel-rpc-port 12321 \
--tensor-parallel-size "$PREFILL_TENSOR_PARALLEL_SIZE" \
--enable-expert-parallel \
--served-model-name "$MODEL_NAME" \
--seed 1024 \
--max-model-len 204800 \
--max-num-batched-tokens 4096 \
--max-num-seqs 16 \
--block-size 128 \
--enforce-eager \
--async-scheduling \
--no-disable-hybrid-kv-cache-manager \
--no-enable-prefix-caching \
--trust-remote-code \
--gpu_memory_utilization 0.95 \
--speculative-config '{"num_speculative_tokens": 1,"method": "mtp","enforce_eager": true}' \
--safetensors-load-strategy 'prefetch' \
--model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 128}' \
--tokenizer-mode deepseek_v4 \
--tool-call-parser deepseek_v4 \
--enable-auto-tool-choice \
--reasoning-parser deepseek_v4 \
--additional-config '{"enable_cpu_binding": true, "enable_shared_expert_dp": true}' \
--kv-transfer-config \
    '{"kv_connector": "MultiConnector",
    "kv_role": "kv_producer",
    "kv_connector_extra_config": {
        "connectors": [{
	    "kv_connector": "MooncakeHybridConnector",
	    "kv_role": "kv_producer",
	    "kv_port": "30000",
	    "kv_connector_extra_config": {
                "prefill": {
                        "dp_size": 2,
                        "tp_size": 8
                },
                "decode": {
                        "dp_size": 2,
                        "tp_size": 8
                }
	    }
        },
	{
	    "kv_connector": "AscendStoreConnector",
	    "kv_role": "kv_producer",
	    "kv_connector_extra_config": {
                "backend": "mooncake",
		"lookup_rpc_port": "0"
            }
	}]
    }
    }' > "$LOG_FILE" 2>&1 &
```
## mooncake_master的启动命令。
```shell
export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib64:${LD_LIBRARY_PATH}

mooncake_master \
  --rpc_port=39486 \
  --eviction_high_watermark_ratio=0.7 \
  --eviction_ratio=0.2 \
  --default_kv_lease_ttl=11000 \
  --enable_offload=true \
  --offload_on_evict=true
```

# P节点的vllm日志
```text
W20260708 12:39:38.314945  7519 storage_backend.cpp:1353] Duplicate key detected in BatchOffload: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@cb74f08f554769a92285d5d1a278df180c9d1763a35ca22a68da22ddf5fbec7f, bucket_id=7304518881347. Returning OBJECT_ALREADY_EXISTS.
...
E20260708 12:48:28.416213  7519 file_storage.cpp:731] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@797000e494ea5584f4407dbdd0f8087369dc029a48ccecbc37b63875e55dc7a2
E20260708 12:48:28.416296  7519 file_storage.cpp:382] BatchQuerySlices failed with error: INVALID_KEY
E20260708 12:48:30.313634  7528 file_storage.cpp:731] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@8d1310d6592a04f35eb7eb83c04b6f407b857e81bd428f18339d54580aef6d9a
E20260708 12:48:30.313707  7528 file_storage.cpp:382] BatchQuerySlices failed with error: INVALID_KEY
E20260708 12:48:30.458350  7511 file_storage.cpp:731] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@6b288f4977b946d91003ddcca6332024e5f82185632c7f1c16f600edde81a2fc
E20260708 12:48:30.458447  7511 file_storage.cpp:382] BatchQuerySlices failed with error: INVALID_KEY
E20260708 12:48:30.642908  7526 file_storage.cpp:731] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@dceb210cdce4c83faf29c754daf4918a4ab549fd0ab270dff3f2305735bbf059
E20260708 12:48:30.643005  7526 file_storage.cpp:382] BatchQuerySlices failed with error: INVALID_KEY
E20260708 12:48:30.651410  7513 file_storage.cpp:731] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:5@cache_role:kv@cache_family:c1@b748b1174ac20e00b2329dbdaf9d84f54fc91145820592d0085169055936f49b
E20260708 12:48:30.651494  7513 file_storage.cpp:382] BatchQuerySlices failed with error: INVALID_KEY
E20260708 12:48:30.946017  7517 file_storage.cpp:731] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:5@cache_role:kv@cache_family:c1@b2a33ffe860be7487d2f3abe1dd8062ad2f01d474a9378cda4507368b1786913
E20260708 12:48:30.946106  7517 file_storage.cpp:382] BatchQuerySlices failed with error: INVALID_KEY
...
E20260708 12:57:39.994860 11520 storage_backend.cpp:1424] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@16e8aff8f59dd4f5396cff3500306159171b8eb1745e182f9f3f7ccd94827d35
E20260708 12:57:39.995028 11521 storage_backend.cpp:1424] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@16e8aff8f59dd4f5396cff3500306159171b8eb1745e182f9f3f7ccd94827d35
E20260708 12:57:39.995077 11521 file_storage.cpp:690] Batch load object failed,err_code = INVALID_KEY
E20260708 12:57:39.995029 11520 file_storage.cpp:690] Batch load object failed,err_code = INVALID_KEY
E20260708 12:57:39.995108 11520 file_storage.cpp:311] Batch load object failed,err_code = INVALID_KEY
E20260708 12:57:39.995096 11521 file_storage.cpp:311] Batch load object failed,err_code = INVALID_KEY
E20260708 12:57:39.995134 11520 real_client.cpp:5434] Batch get offload object failed,err_code = INVALID_KEY
E20260708 12:57:39.995139 11521 real_client.cpp:5434] Batch get offload object failed,err_code = INVALID_KEY
...

```

单条请求测试（50K输入，1K输出）时，mooncake + 池化 + ssd没有问题。
32并发，50K输入，1K输出时，会报错，报错见：deepseekv4_start_prefill_XX.log

# 分析
## 问题定位结论

32 并发下失败的核心链路是 **Mooncake SSD Offload 并发写冲突 → 整批 persist 失败 → Master 与本地 SSD 元数据不一致 → D 侧读 KV 时 `INVALID_KEY`**。单请求不触发，是因为没有并发重复 offload。

---

### 1. 错误时间线（日志）

| 时间 | 现象 | 含义 |
|------|------|------|
| **12:39:38** | 首次 `Duplicate key detected in BatchOffload` → `OBJECT_ALREADY_EXISTS` | 多线程/多请求同时 offload 同一 KV block |
| **12:48 ~ 13:10+** | 大量 `Key not found` → `BatchQuerySlices/BatchLoad INVALID_KEY` | SSD 本地索引找不到 block |
| **12:57:39** | `Batch get offload object failed, err_code = INVALID_KEY`（约 4.7 万次） | D 节点经 RPC 从 P 节点 SSD 拉 KV 失败 |
| **14:19+** | 32 并发 POST 开始，`missing_count=N` 持续出现 | KV 传输时 block 缺失，需补存 |

---

### 2. 根因分析

#### 根因 A：并发 offload 同一 KV key（主因）

日志典型片段：

```
W ... Duplicate key detected in BatchOffload: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@...@{block_hash}
    bucket_id=7304518881347. Returning OBJECT_ALREADY_EXISTS.
I ... Cleaned up orphaned bucket data file: /opt/mooncake_offload/rank_8/7304518881347.bucket
E ... Failed to store objects with error: OBJECT_ALREADY_EXISTS
E ... Failed to persist objects with error: OBJECT_ALREADY_EXISTS
```

触发条件（32 并发 + 池化 + prefix caching）：
- 多请求共享 prefix block → **相同 block hash**
- Mooncake 池化后多请求同时触发 offload
- 本地 `object_bucket_map_` 检测到 key 已存在，返回 `OBJECT_ALREADY_EXISTS`

本地代码路径（`storage_backend.cpp`）：

```1344:1353:Mooncake/mooncake-store/src/storage_backend.cpp
        // Pre-check for duplicates before modifying any state
        for (const auto& key : committed_keys) {
            if (object_bucket_map_.find(key) != object_bucket_map_.end()) {
                LOG(WARNING)
                    << "Duplicate key detected in BatchOffload: " << key
                    << ", bucket_id=" << bucket_id
                    << ". Returning OBJECT_ALREADY_EXISTS.";
                lock.unlock();
                CleanupOrphanedBucket(bucket_id);
                return tl::make_unexpected(ErrorCode::OBJECT_ALREADY_EXISTS);
```

#### 根因 B：`file_storage.cpp` 未将 `OBJECT_ALREADY_EXISTS` 视为幂等成功

`PutStart` 路径会把 `OBJECT_ALREADY_EXISTS` 当成功：

```1553:1555:Mooncake/mooncake-store/src/client_service.cpp
        if (err == ErrorCode::OBJECT_ALREADY_EXISTS) {
            VLOG(1) << "object_already_exists key=" << key;
            return {};
```

但 **Offload persist 路径**遇到该错误会直接失败整批：

```563:577:Mooncake/mooncake-store/src/file_storage.cpp
        if (!offload_res) {
            LOG(ERROR) << "Failed to store objects with error: "
                       << offload_res.error();
            ...
            } else {
                return tl::make_unexpected(offload_res.error());
            }
        }
```

后果：
1. 一个 bucket 里只要有一个 duplicate key，**整批 keys 都 persist 失败**
2. 部分 key 可能已在 Master 注册 LOCAL_DISK replica，但本地 SSD 实际未写入
3. D 侧 `BatchGet` → `Key not found` → **`INVALID_KEY`**

#### 根因 C：配置加剧并发冲突

| 配置项 | 当前值 | 影响 |
|--------|--------|------|
| `enable_prefix_caching` | 日志中为 **True**（`run_pd-p.sh` 写了 `--no-enable-prefix-caching`，但实际运行未生效） | 多请求共享 block，duplicate offload 概率高 |
| `default_kv_lease_ttl` | **11000ms（11 秒）** | 50K prefill 远超 11s，lease 过期可能导致 Master 侧 key 失效 |
| `max_num_seqs` | 16 | 32 并发下仍有足够并行度触发 race |
| `offload_on_evict=true` | 开启 | 高并发下 eviction + 重复 offload 叠加 |

---

### 3. 错误传播链

```mermaid
sequenceDiagram
    participant P as P节点 vLLM
    participant MC as Mooncake Client
    participant SSD as 本地 SSD Backend
    participant M as mooncake_master
    participant D as D节点

    P->>MC: 多请求并发 offload 同一 block
    MC->>SSD: BatchOffload (WriteBucket)
    SSD-->>SSD: 检测到 Duplicate key
    SSD-->>MC: OBJECT_ALREADY_EXISTS
    MC-->>MC: file_storage 整批 persist 失败
    Note over M,SSD: Master 可能有 replica 记录，SSD 无数据
    D->>MC: BatchGet offload object
    MC->>SSD: BatchLoad
    SSD-->>D: INVALID_KEY (Key not found)
```

---

### 4. 非主因（可忽略）

- `libjemalloc.so.2` LD_PRELOAD 警告：环境缺库，不影响功能
- Traceback `ValueError: ... is not a multimodal model`：个别请求带了 image 内容，与 Mooncake KV 无关
- `reasoning token IDs` 警告：DSV4 parser 配置问题，非本次根因

---

### 5. 修复建议（按优先级）

**P0 — Mooncake 代码修复**

1. 在 `file_storage.cpp` 的 `OffloadObjects` 中，对 `OBJECT_ALREADY_EXISTS` **按幂等成功处理**（与 `PutStart` 一致），不要 fail 整批。
2. 或在 `storage_backend.cpp` 的 `BatchOffload` 中，duplicate key 时 **跳过该 key、继续提交其余 key**，而不是 return error。
3. 同一 bucket 内 duplicate 时，只对 duplicate key 跳过，不影响 batch 内其他 key。

**P1 — 配置调整**

```bash
# start_mooncake_master.sh - lease 应覆盖完整 prefill + transfer 时间
--default_kv_lease_ttl=600000   # 建议 10 分钟，按实际 prefill 时长调整

# run_pd-p.sh - 确认 prefix caching 关闭（与脚本意图一致）
--no-enable-prefix-caching

# 若 32 并发仍冲突，可先压测验证
--max-num-seqs 32
```

**P2 — 验证步骤**

1. 修复后重跑 32 并发，确认不再出现 `Failed to persist objects with error: OBJECT_ALREADY_EXISTS`
2. 确认 `Batch get offload object failed, INVALID_KEY` 归零
3. 确认 D 侧 `missing_count` 不再持续增长
4. 检查 `/opt/mooncake_offload/rank_*/` 下 bucket 文件与 Master 元数据一致

---

### 6. 总结

| 项目 | 结论 |
|------|------|
| **直接报错** | `OBJECT_ALREADY_EXISTS` → `INVALID_KEY` |
| **根本原因** | 32 并发下同一 KV block 被重复 offload，`file_storage` 未做幂等处理，整批 persist 失败 |
| **为何单请求正常** | 无并发 race，duplicate 不触发 |
| **修复方向** | Offload 路径对 `OBJECT_ALREADY_EXISTS` 幂等 + 增大 lease TTL + 确认关闭 prefix caching |


### Before submitting...

- [x] Ensure you searched for relevant issues and read the [documentation]

时间	现象	含义
12:39:38	首次 `Duplicate key detected in BatchOffload` → `OBJECT_ALREADY_EXISTS`	多线程/多请求同时 offload 同一 KV block
12:48 ~ 13:10+	大量 `Key not found` → `BatchQuerySlices/BatchLoad INVALID_KEY`	SSD 本地索引找不到 block
12:57:39	`Batch get offload object failed, err_code = INVALID_KEY`（约 4.7 万次）	D 节点经 RPC 从 P 节点 SSD 拉 KV 失败
14:19+	32 并发 POST 开始，`missing_count=N` 持续出现	KV 传输时 block 缺失，需补存

项目	结论
直接报错	`OBJECT_ALREADY_EXISTS` → `INVALID_KEY`
根本原因	32 并发下同一 KV block 被重复 offload，`file_storage` 未做幂等处理，整批 persist 失败
为何单请求正常	无并发 race，duplicate 不触发
修复方向	Offload 路径对 `OBJECT_ALREADY_EXISTS` 幂等 + 增大 lease TTL + 确认关闭 prefix caching

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Bug]: [SSD] DSV4 enable SSD offload，stress testing, repeat offloading the same batch key, lead to OBJECT_ALREADY_EXISTS/persist failed/INVALID_KEY #2827

Bug Report

测试场景：

P节点的1台机器的vllm启动脚本。

mooncake_master的启动命令。

P节点的vllm日志

分析

问题定位结论

1. 错误时间线（日志）

2. 根因分析

根因 A：并发 offload 同一 KV key（主因）

根因 B：`file_storage.cpp` 未将 `OBJECT_ALREADY_EXISTS` 视为幂等成功

根因 C：配置加剧并发冲突

3. 错误传播链

4. 非主因（可忽略）

5. 修复建议（按优先级）

6. 总结

Before submitting...

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

配置项	当前值	影响
`enable_prefix_caching`	日志中为 True（`run_pd-p.sh` 写了 `--no-enable-prefix-caching`，但实际运行未生效）	多请求共享 block，duplicate offload 概率高
`default_kv_lease_ttl`	11000ms（11 秒）	50K prefill 远超 11s，lease 过期可能导致 Master 侧 key 失效
`max_num_seqs`	16	32 并发下仍有足够并行度触发 race
`offload_on_evict=true`	开启	高并发下 eviction + 重复 offload 叠加

Uh oh!

[Bug]: [SSD] DSV4 enable SSD offload，stress testing, repeat offloading the same batch key, lead to OBJECT_ALREADY_EXISTS/persist failed/INVALID_KEY #2827

Description

Bug Report

测试场景：

P节点的1台机器的vllm启动脚本。

mooncake_master的启动命令。

P节点的vllm日志

分析

问题定位结论

1. 错误时间线（日志）

2. 根因分析

根因 A：并发 offload 同一 KV key（主因）

根因 B：file_storage.cpp 未将 OBJECT_ALREADY_EXISTS 视为幂等成功

根因 C：配置加剧并发冲突

3. 错误传播链

4. 非主因（可忽略）

5. 修复建议（按优先级）

6. 总结

Before submitting...

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

根因 B：`file_storage.cpp` 未将 `OBJECT_ALREADY_EXISTS` 视为幂等成功