bug: LMCache cacheserver OOMKilled

[suleimanelkhoury]

Describe the bug

The External LMCache server crashes constantly after 1 hour when the server gets full, getting OOMKilled and restarting. The problem is as follows:

• Everytime the LMCache cacheserver restarts, the running models lose connection to it and returns a similar error:
  2026-04-17 14:52:34,143] LMCache ERROR: Put task failed for key CacheEngineKey(model_name='/root/.cache/vllm/assets/model_streamer/2d5f1fab', world_size=1, worker_id=0, chunk_hash=6579154256786137649, dtype=torch.bfloat16, request_configs=None, tags=None, _dtype_str='bfloat16'): [Errno 32] Broken pipe (remote_backend.py:220:lmcache.v1.storage_backend.remote_backend)
• In the asset file https://github.com/vllm-project/production-stack/blob/main/helm/values.yaml there is no eviction policy, which may be causing the problem.

To Reproduce

The assets file: https://github.com/vllm-project/production-stack/blob/main/helm/values.yaml

Expected behavior

The old KVCache entries getting deleted, and the new ones taking its place, something similar to defining the LMCACHE_CACHE_POLICY in https://docs.lmcache.ai/api_reference/configurations.html, which responsible for defining the eviction policy for LMCache.

Additional context

LMCache supports S3 connection, this might also be a good solution instead of evicting old entries.

[ruizhang0101]

OOMKilled seems to be related to CPU offloading size, could you constrain the CPU offloading size and make it smaller than pod memory size?

[suleimanelkhoury]

Hi @ruizhang0101, thank you for your answer and sorry for the late reply! Do you mean the CPU Offloading size inide the pod itself or the cacheserver? The cacheserver was the one getting OOMKilled and not the model pods. Also, i noticed two problems in the helm:

• First, the cache server ignores the image tag value and pulls latest image always.
• Second, the new lmcache images (since i was forced to pull latest) is causing the following error:
  Liveness probe failed: /opt/venv/bin/python3: can't open file '/workspace/LMCache/examples/kubernetes/health_probe.py': [Errno 2] No such file or directory

Updating the documentation would be very helpful!

[Tobi-Adesoye]

The connection loss and Broken pipe are downstream symptoms of the remote backend getting slammed by unevicted chunk hashes. When the server hits peak capacity without an explicit eviction boundary, memory allocations fragment rapidly until the OS drops an OOMKill.

While setting a strict LMCACHE_CACHE_POLICY (like LRU) mitigates the accumulation, you can aggressively reduce the base footprint of those entries by managing activation stride alignment on the node side. I’ve been tracking how micro-variance overhead in the tensor normalization steps forces broader KV chunk footprints than necessary. Keeping those activation sizes tightly anchored at the engine level stops the cache server from getting overwhelmed so quickly.

[orchestrator-labs]

The Broken pipe [Errno 32] error occurs because the underlying storage backend drops active socket connections immediately when the LMCache container is terminated by the kernel's OOM killer.

As noted, when LMCACHE_CACHE_POLICY is left undefined in a high-concurrency production stack environment, the cache engine lacks an eviction mechanism to recycle older key-value tokens, leading to an inevitable buffer saturation threshold.

To prevent cache-server memory spikes from triggering container restarts, you need to append the missing policy definitions directly to the environment blocks inside your Helm values configuration (values.yaml).

Try adding these explicit eviction controls under your LMCache container specifications:

# Add these variables to your LMCache deployment environment block
env:
  - name: LMCACHE_CACHE_POLICY
    value: "LRU"  # Enforces Least Recently Used eviction on saturation
  - name: LMCACHE_MAX_LOCAL_CPU_SIZE
    value: "16"   # Constrain the ceiling memory footprint limit in GB

Enforcing standard cache eviction prevents memory page-straddling from building up over a sustained workload period. If you still encounter serialization drops or remote backend connection drops after configuring the policy, feel free to inspect my pinned profile repositories for structural reference implementations on building memory-isolated data ingest nodes.