feat(store): prefix-aware KV cache reuse and E2E validation

[yanchaomei]

Description

This PR packages the Mooncake Store KVCache optimization work used for the competition branch:

• Adds prefix-aware KVCache reuse in Mooncake Store, including fast negative lookup, radix-tree prefix fallback, and safer deferred lease touch flushing around replica removal paths.
• Extends master service coverage for prefix fallback and lease touch behavior.
• Adds an end-to-end SGLang + HiCache + Mooncake Store benchmark harness, sanitized deployment/validation docs, and configurable wheel build directory support via BUILD_DIR.

Module

• Transfer Engine (mooncake-transfer-engine)
• Mooncake Store (mooncake-store)
• Mooncake EP (mooncake-ep)
• Integration (mooncake-integration)
• P2P Store (mooncake-p2p-store)
• Python Wheel (mooncake-wheel)
• PyTorch Backend (mooncake-pg)
• Mooncake RL (mooncake-rl)
• CI/CD
• Docs
• Other

Type of Change

• Bug fix
• New feature
• Refactor
• Breaking change
• Documentation update
• Other

How Has This Been Tested?

• bash -n benchmarks/e2e_hicache_benchmark.sh
• git diff --check
• Remote Mooncake Store test suite: GLOG_minloglevel=2 ./build/mooncake-store/tests/master_service_test passed 73/73 tests.
• Remote E2E smoke with Qwen2.5-0.5B through SGLang + HiCache + Mooncake Store completed 60/60 requests successfully after cleanup hardening.
• The full 7B E2E path is documented but was blocked in the available remote snapshot because the local model snapshot lacked weight shards; the 0.5B run was used as the integration smoke.

Checklist

• I have performed a self-review of my own code.
• I have formatted my own code using ./scripts/code_format.sh before submitting.
  • clang-format-20 was not available in the local environment, so this checklist item is intentionally left unchecked.
• I have updated the documentation.
• I have added tests to prove my changes are effective.

[gemini-code-assist]

Code Review

This pull request introduces several major optimizations to the Mooncake Store, including a lock-free Counting Bloom Filter, a prefix-aware Radix Tree index, an S3-FIFO eviction strategy, an adaptive cache scheduler, and a PMEM buffer allocator. The code review identifies several critical issues and improvement opportunities: a potential runtime crash in PmemBufferAllocator due to public instantiation with std::enable_shared_from_this, an O(N) performance bottleneck in S3-FIFO's RemoveKey that can be simplified to O(1), memory ordering issues in deferred lease operations, a sign-extension bug in the Bloom Filter's FNV-1a hash function, and a fragile node-lifetime pattern in the Radix Tree's removal logic.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

PmemBufferAllocator inherits from std::enable_shared_from_this<PmemBufferAllocator> and calls shared_from_this() inside allocate(). However, its constructor is public, allowing users to instantiate it on the stack or via std::unique_ptr. This will cause a runtime crash (std::bad_weak_ptr) when shared_from_this() is invoked. To prevent this, the constructor should be made private or protected, and a static factory method (e.g., create()) returning std::shared_ptr<PmemBufferAllocator> should be provided.

[gemini-code-assist]

The RemoveKey implementation performs linear scans (O(N)) over small_queue_ and main_queue_ to eagerly remove the key. This defeats the O(1) complexity of the eviction strategy and introduces a severe performance bottleneck when the cache contains many keys. Since EvictKey already performs lazy deletion by checking if the popped key exists in freq_ (it == freq_.end()), eager queue removal is completely redundant. Simplifying RemoveKey to only erase from freq_ and ghost_set_ achieves true O(1) performance.

    ErrorCode RemoveKey(const std::string& key) override {
        freq_.erase(key);
        ghost_set_.erase(key);

        // Remove from base class tracking
        return EvictionStrategy::RemoveKey(key);
    }

[gemini-code-assist]

TouchDeferred and FlushDeferredLease use std::memory_order_relaxed for both recently_accessed and recent_access_ms. Because relaxed operations do not impose any synchronization or ordering guarantees, the CPU or compiler can reorder the store of recently_accessed before recent_access_ms in TouchDeferred. Consequently, the eviction thread calling FlushDeferredLease might see recently_accessed as true but read a stale or uninitialized value from recent_access_ms. To ensure correctness, use std::memory_order_release when setting recently_accessed to true, and std::memory_order_acquire when exchanging it in FlushDeferredLease.

        void TouchDeferred() const {
            const auto now = std::chrono::system_clock::now();
            const auto now_ms =
                std::chrono::duration_cast<std::chrono::milliseconds>(
                    now.time_since_epoch())
                    .count();
            recent_access_ms.store(now_ms, std::memory_order_relaxed);
            recently_accessed.store(true, std::memory_order_release);
        }

[gemini-code-assist]

Use std::memory_order_acquire when exchanging recently_accessed to ensure that the thread reads the most up-to-date value of recent_access_ms written by TouchDeferred.

        void FlushDeferredLease(const uint64_t ttl,
                                const uint64_t soft_ttl) const {
            if (recently_accessed.exchange(false, std::memory_order_acquire)) {
                const auto access_ms =
                    recent_access_ms.load(std::memory_order_relaxed);
                const auto access_time =
                    access_ms > 0
                        ? std::chrono::system_clock::time_point(
                              std::chrono::milliseconds(access_ms))
                        : std::chrono::system_clock::now();
                SpinLocker locker(&lock);
                lease_timeout =
                    std::max(lease_timeout,
                             access_time + std::chrono::milliseconds(ttl));
                if (soft_pin_timeout) {
                    soft_pin_timeout =
                        std::max(*soft_pin_timeout,
                                 access_time +
                                     std::chrono::milliseconds(soft_ttl));
                }
            }
        }

[gemini-code-assist]

In fnv1a, c is cast directly from char to size_t. On platforms where char is signed by default, any character with a value greater than 127 (such as non-ASCII UTF-8 bytes) will undergo sign extension, resulting in a very large size_t value (e.g., 0xffffffffffffff80 instead of 0x80). This can cause inconsistent hash values across different platforms/compilers and degrade hash distribution. Cast c to unsigned char first to prevent sign extension.

        hash ^= static_cast<size_t>(static_cast<unsigned char>(c));

[gemini-code-assist]

In the merge step of removeImpl, only_child is accessed as a reference to a std::unique_ptr owned by child->children. When child->children = std::move(grandchildren); is executed, the old map is destroyed, which immediately deletes the node only_child refers to. While the code currently accesses only_child's members before this assignment, this pattern is fragile and prone to lifetime/aliasing bugs if modified. Moving only_child out of the map and clearing the map first is much safer and cleaner.

Suggested change

	else if (child->children.size() == 1 && !child->is_terminal) {
	auto& only_child = child->children.begin()->second;
	child->label += only_child->label;
	child->is_terminal = only_child->is_terminal;
	child->full_key = std::move(only_child->full_key);
	auto grandchildren = std::move(only_child->children);
	child->children = std::move(grandchildren);
	else if (child->children.size() == 1 && !child->is_terminal) {
	auto only_child = std::move(child->children.begin()->second);
	child->children.clear();
	child->label += only_child->label;
	child->is_terminal = only_child->is_terminal;
	child->full_key = std::move(only_child->full_key);
	child->children = std::move(only_child->children);
	}

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 71.90022% with 383 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...cake-store/benchmarks/adaptive_scheduler_bench.cpp	0.00%	143 Missing ⚠️
mooncake-store/benchmarks/bloom_filter_bench.cpp	0.00%	103 Missing ⚠️
mooncake-store/src/allocator.cpp	0.00%	97 Missing ⚠️
mooncake-store/src/master_snapshot_manager.cpp	74.41%	11 Missing ⚠️
mooncake-store/src/master_service.cpp	92.37%	9 Missing ⚠️
mooncake-store/include/eviction_strategy.h	88.88%	8 Missing ⚠️
mooncake-store/include/allocator.h	0.00%	5 Missing ⚠️
mooncake-store/include/adaptive_cache_scheduler.h	97.18%	2 Missing ⚠️
mooncake-store/src/master_metric_manager.cpp	66.66%	2 Missing ⚠️
mooncake-store/tests/bloom_filter_test.cpp	99.42%	1 Missing ⚠️
... and 2 more

📢 Thoughts on this report? Let us know!