+
+
+
+
+
+
diff --git a/docs/source/_static/modelconfig.json b/docs/source/_static/modelconfig.json
new file mode 100644
index 00000000000..b74ba949277
--- /dev/null
+++ b/docs/source/_static/modelconfig.json
@@ -0,0 +1,131 @@
+{
+ "meta-llama/Llama-3.1-8B-Instruct": {
+ "hidden_size": 4096,
+ "num_attention_heads": 32,
+ "num_hidden_layers": 32,
+ "num_key_value_heads": 8
+ },
+ "meta-llama/Llama-3.1-70B-Instruct": {
+ "hidden_size": 8192,
+ "num_attention_heads": 64,
+ "num_hidden_layers": 80,
+ "num_key_value_heads": 8
+ },
+ "mistralai/Mistral-7B-Instruct-v0.2": {
+ "hidden_size": 4096,
+ "num_attention_heads": 32,
+ "num_hidden_layers": 32,
+ "num_key_value_heads": 8
+ },
+ "mistralai/Mistral-Large-Instruct-2407": {
+ "hidden_size": 12288,
+ "num_attention_heads": 96,
+ "num_hidden_layers": 88,
+ "num_key_value_heads": 8
+ },
+ "lmsys/longchat-7b-16k": {
+ "hidden_size": 4096,
+ "num_attention_heads": 32,
+ "num_hidden_layers": 32,
+ "num_key_value_heads": 32
+ },
+ "Sao10K/L3-8B-Lunaris-v1": {
+ "hidden_size": 4096,
+ "num_attention_heads": 32,
+ "num_hidden_layers": 32,
+ "num_key_value_heads": 8
+ },
+ "meta-llama/Llama-3.2-3B-Instruct": {
+ "hidden_size": 3072,
+ "num_attention_heads": 24,
+ "num_hidden_layers": 28,
+ "num_key_value_heads": 8
+ },
+ "deepseek-ai/DeepSeek-V3": {
+ "hidden_size": 7168,
+ "num_attention_heads": 128,
+ "num_hidden_layers": 61,
+ "num_key_value_heads": 128,
+ "kv_lora_rank": 512,
+ "qk_rope_head_dim": 64
+ },
+ "deepseek-ai/DeepSeek-R1": {
+ "hidden_size": 7168,
+ "num_attention_heads": 128,
+ "num_hidden_layers": 61,
+ "num_key_value_heads": 128,
+ "kv_lora_rank": 512,
+ "qk_rope_head_dim": 64
+ },
+ "meta-llama/Llama-3.1-405B": {
+ "hidden_size": 16384,
+ "num_attention_heads": 128,
+ "num_hidden_layers": 126,
+ "num_key_value_heads": 8
+ },
+ "meta-llama/Llama-3.2-1B-Instruct": {
+ "hidden_size": 2048,
+ "num_attention_heads": 32,
+ "num_hidden_layers": 16,
+ "num_key_value_heads": 8
+ },
+ "Qwen/Qwen3-32B": {
+ "hidden_size": 5120,
+ "num_attention_heads": 64,
+ "num_hidden_layers": 64,
+ "num_key_value_heads": 8,
+ "head_dim": 128
+ },
+ "Qwen/Qwen3-14B": {
+ "hidden_size": 5120,
+ "num_attention_heads": 40,
+ "num_hidden_layers": 40,
+ "num_key_value_heads": 8,
+ "head_dim": 128
+ },
+ "Qwen/Qwen3-8B": {
+ "hidden_size": 4096,
+ "num_attention_heads": 32,
+ "num_hidden_layers": 36,
+ "num_key_value_heads": 8,
+ "head_dim": 128
+ },
+ "Qwen/Qwen3-4B": {
+ "hidden_size": 2560,
+ "num_attention_heads": 32,
+ "num_hidden_layers": 36,
+ "num_key_value_heads": 8,
+ "head_dim": 128
+ },
+ "Qwen/Qwen3-0.6B": {
+ "hidden_size": 1024,
+ "num_attention_heads": 16,
+ "num_hidden_layers": 28,
+ "num_key_value_heads": 8,
+ "head_dim": 128
+ },
+ "Qwen/Qwen2.5-7B-Instruct": {
+ "hidden_size": 3584,
+ "num_attention_heads": 28,
+ "num_hidden_layers": 28,
+ "num_key_value_heads": 4
+ },
+ "Qwen/Qwen2.5-3B-Instruct": {
+ "hidden_size": 2048,
+ "num_attention_heads": 16,
+ "num_hidden_layers": 36,
+ "num_key_value_heads": 2
+ },
+ "Qwen/Qwen2.5-0.5B": {
+ "hidden_size": 896,
+ "num_attention_heads": 14,
+ "num_hidden_layers": 24,
+ "num_key_value_heads": 2
+ },
+ "Qwen/Qwen-7B": {
+ "hidden_size": 4096,
+ "num_attention_heads": 32,
+ "num_hidden_layers": 32,
+ "num_key_value_heads": 32
+ }
+}
diff --git a/docs/source/api_reference/configurations.rst b/docs/source/api_reference/configurations.rst
index 3b1252521c6..efcf8083687 100644
--- a/docs/source/api_reference/configurations.rst
+++ b/docs/source/api_reference/configurations.rst
@@ -79,6 +79,47 @@ Basic cache settings that control the core functionality of LMCache.
* - extra_config
- LMCACHE_EXTRA_CONFIG={"key": value, ...}
- Additional configuration as JSON dict. For NUMA manual mode, include "gpu_to_numa_mapping": {gpu_id: numa_node, ...}. Default: {}
+
+Lazy Memory Allocator Configurations
+------------------------------------
+
+Settings for the lazy memory allocator that enables gradual memory allocation to reduce startup time and initial memory footprint.
+
+.. note::
+
+ The lazy memory allocator is designed for scenarios with large CPU memory configurations. It starts with a small initial allocation and gradually expands as needed, reducing startup wait time and avoiding unnecessary memory consumption when the full capacity is not required.
+
+ **Key characteristics:**
+
+ - **One-time expansion**: Memory expands until target size is reached, then stops
+ - **No shrinking**: Once allocated, memory is never released back to the system
+ - **Automatic activation**: Only activates when ``max_local_cpu_size`` exceeds ``lazy_memory_safe_size``
+
+.. list-table::
+ :header-rows: 1
+ :widths: 30 30 40
+
+ * - YAML Config Name
+ - Environment Variable
+ - Description
+ * - enable_lazy_memory_allocator
+ - LMCACHE_ENABLE_LAZY_MEMORY_ALLOCATOR
+ - Whether to enable lazy memory allocator. Values: true/false. Default: false
+ * - lazy_memory_initial_ratio
+ - LMCACHE_LAZY_MEMORY_INITIAL_RATIO
+ - Initial memory allocation ratio (0.0-1.0). Determines the fraction of max_local_cpu_size to allocate at startup. Default: 0.2 (20%)
+ * - lazy_memory_expand_trigger_ratio
+ - LMCACHE_LAZY_MEMORY_EXPAND_TRIGGER_RATIO
+ - Memory usage ratio (0.0-1.0) that triggers expansion. When used memory exceeds this ratio of current capacity, expansion begins. Default: 0.5 (50%)
+ * - lazy_memory_step_ratio
+ - LMCACHE_LAZY_MEMORY_STEP_RATIO
+ - Memory expansion step ratio (0.0-1.0). Each expansion adds this fraction of max_local_cpu_size. Default: 0.1 (10%)
+ * - lazy_memory_safe_size
+ - LMCACHE_LAZY_MEMORY_SAFE_SIZE
+ - Threshold in GB above which lazy allocator activates. If max_local_cpu_size ≤ this value, lazy allocator is disabled regardless of enable_lazy_memory_allocator setting. Default: 0.0
+ * - reserve_local_cpu_size
+ - LMCACHE_RESERVE_LOCAL_CPU_SIZE
+ - Reserved system memory in GB that should not be allocated by LMCache. Used to prevent out-of-memory conditions. Default: 0.0
Cache Blending Configurations
-----------------------------
diff --git a/docs/source/community/meetings.rst b/docs/source/community/meetings.rst
index 70dfb0f2fc0..11d40331f24 100644
--- a/docs/source/community/meetings.rst
+++ b/docs/source/community/meetings.rst
@@ -4,6 +4,7 @@ Community meetings
LMCache hosts regular community meetings to discuss updates, address new feature requests,
and feedback from the community. If you are interested in contributing to the LMCache projects
(core LMCache or Production Stack), we encourage you to join the meetings.
+We also host a monthly "Office Hour" on select topics.
Meeting schedule
-----------------
@@ -21,7 +22,7 @@ Please find the meeting invite link below:
- **Meeting link**: `Zoom link KV Cache Size Calculator
+ +Loading model configurations...
+
+
+
+
+
diff --git a/docs/source/getting_started/quickstart.rst b/docs/source/getting_started/quickstart.rst
new file mode 100644
index 00000000000..a580b61b327
--- /dev/null
+++ b/docs/source/getting_started/quickstart.rst
@@ -0,0 +1,92 @@
+.. _quickstart:
+
+Quickstart
+==========
+
+This guide will help you get LMCache up and running quickly within 2 minutes. You'll see LMCache in action with a complete end-to-end example.
+
+(Terminal 1) Install LMCache
+----------------------------
+
+First, install LMCache with these three commands:
+
+.. code-block:: bash
+
+ uv venv --python 3.12
+ source .venv/bin/activate
+ uv pip install lmcache vllm
+
+Start vLLM with LMCache using a single command:
+
+.. code-block:: bash
+
+ # The chunk size here is only for illustration purpose, use default one (256) later
+ LMCACHE_CHUNK_SIZE=8 \
+ vllm serve Qwen/Qwen3-8B \
+ --port 8000 --kv-transfer-config \
+ '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
+
+.. note::
+ If you want to customize configurations further, you can create a configuration file. See the :doc:`../api_reference/configurations` page to learn about all available options.
+
+(Terminal 2) Test LMCache in Action
+-----------------------------------
+
+Now let's see LMCache working! Open a new terminal and send your first request:
+
+.. code-block:: bash
+
+ curl http://localhost:8000/v1/completions \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "Qwen/Qwen3-8B",
+ "prompt": "Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts",
+ "max_tokens": 100,
+ "temperature": 0.7
+ }'
+
+You should see LMCache logs like this:
+
+.. code-block:: text
+
+ (EngineCore_DP0 pid=458469) [2025-09-30 00:08:43,982] LMCache INFO: Stored 27 out of total 27 tokens. size: 0.0037 gb, cost 1.8470 ms, throughput: 2.0075 GB/s; offload_time: 1.7962 ms, put_time: 0.0509 ms
+
+**What this means:** The 27 tokens from your prompt are being stored in CPU RAM because this is the first time the system processes this text. LMCache is caching the KV cache for future reuse.
+
+Now send a second request with a prefix that overlaps with the first:
+
+.. code-block:: bash
+
+ curl http://localhost:8000/v1/completions \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "Qwen/Qwen3-8B",
+ "prompt": "Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts (MoE) models",
+ "max_tokens": 100,
+ "temperature": 0.7
+ }'
+
+You should see logs like this:
+
+.. code-block:: text
+
+ Reqid: cmpl-6709d8795d3c4464b01999c9f3fffede-0, Total tokens 32, LMCache hit tokens: 24, need to load: 8
+ (EngineCore_DP0 pid=494270) [2025-09-30 01:12:36,502] LMCache INFO: Retrieved 8 out of total 8 out of total 24 tokens. size: 0.0011 gb, cost 0.5547 ms, throughput: 1.9808 GB/s;
+ (EngineCore_DP0 pid=494270) [2025-09-30 01:12:36,509] LMCache INFO: Storing KV cache for 8 out of 32 tokens (skip_leading_tokens=24)
+ (EngineCore_DP0 pid=494270) [2025-09-30 01:12:36,510] LMCache INFO: Stored 8 out of total 8 tokens. size: 0.0011 gb, cost 0.4274 ms, throughput: 2.5702 GB/s; offload_time: 0.4013 ms, put_time: 0.0262 ms
+
+**What this means:**
+
+- **Total tokens 32**: The new prompt has 32 tokens after tokenization
+- **LMCache hit tokens: 24**: 24 tokens were found in the cache (24 is a multiple of 8, our chunk size in this example)
+- **Need to load: 8**: vLLM has automatic prefix caching enabled with block size 16. Although there are 24 hit tokens, 16 are already in GPU RAM managed by vLLM, so LMCache only needs to load 24-16=8 tokens
+- **Why 24 hit tokens instead of 27?** LMCache hashes every 8 tokens incrementally (8, 16, 24, 27). When the new request comes in, it checks every 8-token chunk, so it uses the 24-token hash instead of checking the 27-token hash
+- **Stored another 8 tokens**: The new 8 tokens form a complete chunk that gets hashed and stored in CPU RAM for future use
+
+🎉 **Congratulations!** You've just seen LMCache automatically cache and reuse KV caches, reducing computation for overlapping text.
+
+Next Steps
+----------
+
+- **Performance Testing**: Try the :doc:`benchmarking` section to experience LMCache's performance benefits with more comprehensive examples
+- **More Examples**: Explore the :doc:`quickstart/index` section for detailed examples including KV cache sharing across instances and disaggregated prefill
diff --git a/docs/source/getting_started/quickstart/disaggregated_prefill.rst b/docs/source/getting_started/quickstart/disaggregated_prefill.rst
index 1f81e3c0ca5..93297033e53 100644
--- a/docs/source/getting_started/quickstart/disaggregated_prefill.rst
+++ b/docs/source/getting_started/quickstart/disaggregated_prefill.rst
@@ -162,13 +162,13 @@ Send requests to the proxy server (port 9000) using either the completions or ch
"max_tokens": 100
}'
-You can also test the setup with the following command, which runs the `benchmark_serving.py ` @@ -111,6 +114,7 @@ Documentation kv_cache_optimizations/compression/index kv_cache_optimizations/blending + kv_cache_optimizations/layerwise :raw-html:`
` @@ -162,5 +166,5 @@ Documentation community/meetings community/blogs -raw-html:`
` +:raw-html:`
` diff --git a/docs/source/kv_cache/async_loading.rst b/docs/source/kv_cache/async_loading.rst new file mode 100644 index 00000000000..9a55374f11d --- /dev/null +++ b/docs/source/kv_cache/async_loading.rst @@ -0,0 +1,129 @@ +Async Loading in LMCache +========================= + +.. contents:: Table of Contents + :local: + :depth: 2 + +Overview +-------- + +This document explains the principle, benefits, differences from vLLM PR `19330
ZMQ PULL/PUSH"] + B["StorageManager
Async loop (thread)"] + C["AsyncSerializer
WeightedSemaphore"] + D["EventManager
EventType.LOADING"] + end + + subgraph Backends + E["LocalCPUBackend
contains/get"] + F["LocalDiskBackend
async contains/get"] + G["FSConnector
remote FS"] + end + + A --> B + B --> C + B --> D + B -.contains/get.-> E + B -.contains/get.-> F + B -.contains/get.-> G + + style E fill:#dff,stroke:#333,stroke-width:1px + style F fill:#ffd,stroke:#333,stroke-width:1px + style G fill:#dfd,stroke:#333,stroke-width:1px + + +Benefits +-------- + +- Performance overlap + - **I/O–Compute Overlap**: Decoupling lookup/prefetch from loading enables fetching KV chunks while vLLM continues scheduling/computation. +- Robustness and error handling + - **Event-driven Synchronization**: ``EventManager`` ensures safe hand-off of futures and avoids race conditions between threads and the async loop. + - **Backpressure & Deadlock Avoidance**: ``AsyncSerializer`` with a weighted semaphore caps concurrent chunk retrievals based on allocator budget, preventing starvation or allocator lockups. + - **Graceful Miss Path**: Immediate response with ``None`` hit tokens when nothing is retrievable; worker returns quickly without stalling the scheduler. + +Comparison with vLLM Load Failure Recovery feature +--------------------------------------------------- + +The `VLLM_PR_19330
+
+ 
+
+
+
+Architecture Overview
+---------------------
+
+**CacheEngine**
+ The main orchestrator containing two primary generators:
+
+ * **Retrieval Generator** (N + 2 yields): Handles layer-by-layer KV cache loading with on-demand memory allocation
+ * **Storage Generator** (N + 1 yields): Manages layer-by-layer KV cache saving with upfront CPU memory allocation
+
+**LayerwiseGPUConnector**
+ Manages GPU-CPU memory transfers with dedicated CUDA streams:
+
+ * **Load GPU Buffer**: Temporary GPU memory for CPU→GPU transfers (``use_gpu: true``)
+ * **Store GPU Buffer**: Temporary GPU memory for GPU→CPU transfers (``use_gpu: true``)
+ * **Nested Generators**: ``batched_to_gpu()`` and ``batched_from_gpu()`` handle actual memory operations
+
+**StorageManager**
+ Handles persistent storage operations:
+
+ * ``layerwise_batched_get()``: Asynchronous retrieval with ``.result()`` for request-level concurrency
+ * ``batched_put()``: Stores memory objects to persistent backends
+
+Execution Flow
+~~~~~~~~~~~~~~
+
+The layerwise pipeline follows a numbered execution sequence:
+
+**1. start_load_kv()**
+ * Initializes Retrieval Generator via ``lmcache_engine.retrieve_layer()``
+ * Performs setup (1st ``next()``) and loads layer 0 (2nd ``next()``)
+ * Creates ``layerwise_retrievers`` list for ongoing layer processing
+
+**2. wait_for_layer_load()** (repeated for each layer)
+ * Advances Retrieval Generator via ``next()`` to process layer i
+ * Triggers ``StorageManager.layerwise_batched_get()`` for async cache retrieval
+ * Calls GPU Load Generator's ``batched_to_gpu()`` to transfer memory objects to GPU
+ * **Last request in batch**: Synchronizes ``current_stream.wait_stream(load_stream)``
+
+**3. save_kv_layer()** (repeated for each layer)
+ * **First call only**: Creates Storage Generator with upfront CPU memory allocation
+ * Advances Storage Generator via ``next()`` to process layer i
+ * Calls GPU Store Generator's ``batched_from_gpu()`` to transfer GPU data to CPU
+ * **First request in batch**: Synchronizes ``store_stream.wait_stream(current_stream)``
+
+**4. wait_for_save()**
+ * Finalizes Storage Generator with last ``next()`` call
+ * Completes all ``StorageManager.batched_put()`` operations
+ * Performs GPU Store Generator cleanup
+
+Key Optimizations
+~~~~~~~~~~~~~~~~~
+
+**Pipelined Memory Operations**
+ The system overlaps layer N+1 computation with layer N storage.
+
+**Stream Synchronization**
+ Three CUDA streams coordinate operations:
+
+ * ``current_stream``: vLLM's forward pass computation
+ * ``load_stream``: KV cache loading operations
+ * ``store_stream``: KV cache storing operations
+
+**Batch-Level Coordination**
+ Multiple requests are processed together with specialized synchronization:
+
+ * **First request**: Provides store stream synchronization to prevent GPU buffer corruption
+ * **Last request**: Provides load stream synchronization to ensure KV cache availability
+
+**Memory Allocation Strategies**
+ * **Retrieval**: Layer-by-layer allocation
+ * **Storage**: Upfront allocation for all layers
+
+**Cache Key Management**
+ Multi-layer cache engine keys use ``split_layers(N)`` to create per-layer kubernetes_deployment
+
+Configuration
+~~~~~~~~~~~~~
+
+Enable layerwise caching by setting:
+
+.. code-block:: yaml
+
+ use_layerwise: true
+
+The system automatically selects appropriate layerwise GPU connectors based on configuration:
+
+* ``VLLMPagedMemLayerwiseGPUConnector``: For standard layerwise operations
+* ``VLLMBufferLayerwiseGPUConnector``: When blending is enabled
diff --git a/docs/source/production/kubernetes_deployment.rst b/docs/source/production/kubernetes_deployment.rst
index 519a7dad2b6..cdf9c56e371 100644
--- a/docs/source/production/kubernetes_deployment.rst
+++ b/docs/source/production/kubernetes_deployment.rst
@@ -1,4 +1,37 @@
-Kubernetes deployment
+Kubernetes Deployment
=====================
-Coming soon...
+For Kubernetes deployment of vLLM with LMCache integration, we recommend using the `vLLM Production Stack
+
+
+ Click to open full-size
+
+