Add Mooncake Store file-level weight connector for SGLang

[xianzhiT]

Motivation

This connector lets SGLang load model weights directly from a remote MooncakeStore over RDMA, treating the store as a generic model cache: --model-path mooncake://<checkpoint> --load-format remote, without staging weight files to local disk.

The design deliberately keeps storage decoupled from the inference engine. Weights are stored as plain files, and the engine loads files exactly as it would from disk — only the source changes (disk → MooncakeStore). This trades some raw load throughput for generality (any file-based loader can adopt it) and operability (import / list / delete are managed independently of the serving process). It is complementary to the tightly-coupled, higher-performance approach in #21661.

Related design discussion / RFC on the Mooncake side: kvcache-ai/Mooncake#2282 (Unified KVCache and Model Weight Management). The Mooncake-side control plane (import / list / inspect / verify / delete) lives in a separate Mooncake PR: kvcache-ai/Mooncake#2829.

Modifications

• connector/mooncake_store_file.py: MooncakeStoreFileConnector. Reads safetensors headers from Mooncake chunks, identifies tensor boundaries from data_offsets, coalesces each tensor batch into a single get_into_ranges() against a reused registered buffer on the target CUDA device (GPUDirect RDMA), and yields (name, tensor) to DefaultLoader. Only metadata/tokenizer/config files are materialized locally.
• connector/__init__.py: register the mooncake:// scheme.
• model_loader/loader.py: pass model_loader_extra_config through to the remote connector.
• configs/model_config.py: resolve remote model config/tokenizer files and propagate the Mooncake store config (also via MOONCAKE_WEIGHT_STORE_CONFIG env) to the loader.
• server_args.py: derive a filesystem-safe served_model_name for remote (mooncake://) model paths.
• managers/{scheduler,tokenizer_manager,tp_worker}.py: thread remote tokenizer metadata paths through the manager processes.
• test/registered/model_loading/test_mooncake_store_file_connector.py: connector unit tests.

Usage

1. Import weights into MooncakeStore (Mooncake side; control plane lives in the separate Mooncake PR kvcache-ai/Mooncake#2829):

python -m mooncake.weight_store.cli \
    --master-server-addr <master_ip:50051> \
    --metadata-server <metadata_ip:8080> \
    --protocol rdma --rdma-devices <mlx5_x> \
    --local-hostname <host_ip:port> \
    model import \
    --checkpoint-id Qwen3-235B-A22B \
    --model-id Qwen/Qwen3-235B-A22B \
    --revision main \
    --source /path/to/local/weights

# inspect / list to confirm it landed
python -m mooncake.weight_store.cli ... model list
python -m mooncake.weight_store.cli ... model inspect Qwen3-235B-A22B

2. Launch SGLang loading from the store (this PR). Point --model-path at
the mooncake://<checkpoint-id> scheme and use `--load-for

python -m sglang.launch_server \
    --model-path mooncake://Qwen3-235B-A22B \
    --load-format remote \
    --tp 8

The store connection is read from the Mooncake config file (or the MOONCAKE_WEIGHT_STORE_CONFIG env var). No extra `--model-is needed — the connector defaults to the GPUDirect RDMA fast path.

Accuracy Tests

End-to-end, cross-machine (one storage node + one 8×H20 node): loaded Qwen3-235B-A22B from MooncakeStore and verified inference output is correct. The connector only changes the weight source, not compute, so model outputs are
unaffected.

Speed Tests and Profiling

Cross-machine setup (one storage node + one 8×H20 node), weights loaded from MooncakeStore over RDMA. Zero extra config needed — the connector defaults to the GPUDirect RDMA fast path, reading weights straight into device memory.

Model	Weights (dtype)	TP	Weight Loading Time
Qwen3-235B-A22B	~438 GB (BF16)	8	~30s

Only Qwen3-235B-A22B was measured on our setup; more models to follow.

Known limitation: generic file-level storage makes TP-aware loading harder than a format-coupled approach, so raw load throughput currently trails the tensor-level, format-coupled path in #21661 (which reports 16s for the same model). Closing that gap (e.g. optional TP-aware layout hints on top of the generic file layout) is the main follow-up.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ❌ Run #29066099590
Latest PR Test (Extra): ❌ Run #29066099499

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!