[RFC]: Workload-Aware KV Cache Hints for Mooncake: `session_id` and `priority`

[Lin-z-w]

Changes proposed

Summary

This RFC proposes a first-stage workload-aware KV Cache interface for Mooncake.

The first stage only introduces two optional external hints:

• session_id
• priority

The goal is to let upper layers such as SGLang, vLLM, or llm-d pass lightweight workload information to Mooncake without changing the correctness semantics of existing PUT / GET operations.

If these hints are not provided, Mooncake behavior should remain unchanged.

Motivation

Agentic and multi-turn workloads often reuse long prefixes across requests. However, the KV Cache objects generated by these workloads are not all equally valuable:

• KV objects belonging to the same session are often related in lifecycle and reuse pattern.
• Some requests are more latency-sensitive or more valuable to retain than others.
• Upper layers already have partial knowledge about session identity and request priority.

Mooncake currently provides object-level KV storage, lookup, transfer, replication, and eviction. To support workload-aware KV Cache management, Mooncake should be able to consume a small amount of external context while still keeping policy decisions inside Mooncake.

This RFC intentionally starts with the smallest useful surface: session_id and priority.

Background

Recent SGLang main already exposes both signals:

• session_id is available as a top-level request field in SGLang's internal request path and OpenAI-compatible serving path.
• priority is available as an integer request field and is already used by SGLang scheduling / radix-cache logic.

SGLang also already has a Mooncake HiCache path that can pass ReplicateConfig.group_ids. Those group_ids currently represent page-level physical grouping, such as K/V objects, split-head shards, MLA objects, and sidecar objects derived from the same logical HiCache page.

This RFC does not propose replacing that existing group_ids behavior.

Goals

1. Add optional session_id and priority hints to Mooncake object-level write paths.
2. Preserve existing behavior when the hints are absent.
3. Keep existing group_ids semantics unchanged.
4. Allow Mooncake to store session/priority metadata for KV objects.
5. Allow Mooncake to use priority as a coarse retention hint.
6. Provide a foundation for future session-level statistics, lifecycle handling, and eviction policy improvements.

Non-Goals

This RFC does not propose:

• A new agent hint standard.
• Changes to OpenAI-compatible API standards.
• Changing GET semantics.
• Treating session_id as a cache key namespace.
• Treating session_id as hard pinning.
• Treating priority as a lease.
• Replacing existing group_ids.
• Adding placement hints such as preferred_segment.
• Adding replica hints such as replica_num.
• Adding explicit movement APIs such as prefetch/copy/move/demote.

Those can be discussed in later phases.

Proposed API Semantics

session_id

session_id is an optional string hint.

Semantics:

• It identifies the session or workload that caused the KV object to be written.
• It is metadata associated with the object.
• It should not be part of the object key by default.
• It should not prevent eviction by itself.
• It should not override group_ids.

Expected usage:

• Associate KV objects with sessions.
• Support session-level observability.
• Enable future session-level lifecycle operations, such as bulk release, demotion, or statistics.
• Help upper layers correlate Mooncake cache behavior with request/session routing.

Important distinction:

• group_ids groups related physical objects derived from the same logical page.
• session_id identifies the workload/session that produced or used the object.

These are orthogonal and should both be preserved.

priority

priority is an optional integer hint.

Mooncake should treat it as a cache retention hint, not as a scheduler priority.

Recommended Mooncake-side semantics:

• Higher normalized priority means the object is more valuable to retain.
• Default priority should behave exactly like today's behavior.
• Priority should influence eviction only as a soft preference.
• Priority should not directly change lease duration.
• Priority should not imply hard pinning.

Because different upper layers may define request priority differently, the integration layer should normalize priority before passing it to Mooncake when needed.

For example, SGLang uses integer priority, but its scheduling direction may be configured. Therefore, Mooncake should not need to understand SGLang's scheduling flags. The SGLang-Mooncake adapter should map SGLang request priority into Mooncake's cache-retention priority.

Proposed Data Model

The exact implementation should follow Mooncake's current coding style and binding constraints. Conceptually, the write configuration should support:

struct ReplicateConfig {
  // Existing fields.
  // ...

  // Optional workload/session identity.
  // Empty string means unset.
  std::string session_id;

  // Optional cache-retention priority.
  // 0 means default/normal behavior.
  int32_t priority = 0;
};

If Mooncake already has a better internal optional-field convention, we should follow it.

For batch_put, the minimal first-stage behavior can treat session_id and priority as applying to all objects in the batch.

If a caller needs to write objects from multiple sessions or priority classes in the same batch, it can split the batch in the first stage. If this becomes a performance issue, we can add per-object vectors later, similar to the existing group_ids pattern:

std::vector<std::string> session_ids;
std::vector<int32_t> priorities;

This RFC does not require per-object vectors in the first implementation.

Expected Mooncake Behavior

When hints are absent

Behavior must be unchanged.

Existing clients, tests, and SGLang integrations should continue to work without modification.

When session_id is present

Mooncake should record the session association in object metadata where feasible.

For the first implementation, this can be metadata-only. It does not have to immediately change eviction behavior.

If an object already exists and a new PUT is skipped or deduplicated, there are two possible acceptable first-stage behaviors:

1. Best-effort metadata update: record that this session also touched the object.
2. Insert-only metadata: keep only the metadata from the original write.

Option 1 is more useful for session-level accounting, but option 2 is acceptable for an initial minimal implementation if updating metadata for existing objects is invasive. The chosen behavior should be documented.

When priority is present

Mooncake should store the priority in object metadata.

If the current eviction path can consume object priority with a small change, Mooncake should use it as a soft eviction preference. For example:

• normal priority: existing behavior
• high priority: evict later than normal objects when possible

If the current eviction path does not have a clean priority hook, the first stage may only store and expose priority metadata, then add eviction integration in a follow-up PR.

Priority should be normalized into a small number of internal levels at first. Mooncake should not depend on arbitrary raw integer values from upper layers.

Interaction With Existing group_ids

Existing group_ids behavior must remain unchanged.

For example, if SGLang writes physical objects derived from the same logical HiCache page:

page0_k
page0_v
page0_sidecar

they may share the same group_ids value.

If the request also has session_id = "session-A", then both pieces of information should coexist:

object key: page0_k
group_id:   sglang-hicache:page0
session_id: session-A
priority:   normal/high

session_id must not overwrite group_ids.

Integration With SGLang

The initial SGLang-Mooncake integration can map:

• SGLang session_id -> Mooncake session_id
• SGLang request priority -> normalized Mooncake priority

Other SGLang signals should be left for later phases:

• routed_dp_rank
• disagg_prefill_dp_rank
• routing_key
• extra_key / cache_salt
• cached_tokens_details
• KV events

For placement, the first stage can keep the existing local-first PUT behavior. Mooncake does not need routed_dp_rank or disagg_prefill_dp_rank for correctness.

Compatibility

This proposal must be backward-compatible:

• Existing clients that do not set session_id or priority should behave exactly the same.
• Existing ReplicateConfig.group_ids users should continue to work.
• Existing Python bindings should accept old call patterns.
• Mixed-version behavior should degrade gracefully when the installed Mooncake package does not expose the new fields.

Testing Plan

The implementation should add tests covering:

1. Existing put / batch_put behavior without hints.
2. put with session_id.
3. batch_put with session_id.
4. put with priority.
5. batch_put with priority.
6. group_ids and session_id being present at the same time.
7. group_ids not being overwritten by session_id.
8. Default priority behaving the same as current behavior.
9. Priority metadata being observable or consumed by eviction logic, depending on the implementation stage.

Rollout Plan

Suggested implementation stages:

Stage 1: API and metadata

• Add session_id and priority to the write config path.
• Preserve default behavior when unset.
• Store the metadata internally.
• Add tests and documentation.

Stage 2: Observability

• Expose basic session/priority statistics if practical.
• Examples:
  • bytes written by session
  • objects written by session
  • evictions by priority class

Stage 3: Eviction integration

• Use normalized priority as a soft eviction preference.
• Keep priority coarse-grained initially.
• Avoid lease changes and hard pinning unless explicitly requested by a separate policy.

Open Questions

1. Should batch_put support only scalar session_id / priority in the first stage, or should it immediately support per-object vectors?
2. When a PUT finds that the object already exists, should Mooncake update session metadata for the existing object?
3. What internal priority levels should Mooncake use initially?
4. Should priority influence eviction in the first PR, or should the first PR only add metadata and API support?
5. What observability signals are most useful for llm-d/SGLang routing feedback?

Proposed First PR Scope

For the first PR, I suggest keeping the scope narrow:

• Add optional session_id and priority to the object write config.
• Preserve existing behavior when unset.
• Keep group_ids unchanged.
• Store metadata for inserted objects.
• Add tests for compatibility and metadata propagation.
• If there is a clean eviction hook, map priority into a coarse soft-retention level; otherwise defer eviction changes to a follow-up PR.

This gives Mooncake a concrete first step toward workload-aware KV Cache without requiring new agent-level APIs or large policy changes.

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation

[github-actions]

Thanks for opening this issue, @Lin-z-w!

Field	Value
Issue	#2819
GitHub user ID	99333079
Reporter	@Lin-z-w

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

[Lin-z-w]

One additional clarification on the proposed API shape:

I think we should separate the information carried by ReplicateConfig into two categories:

1. Explicit policy
2. Advisory workload hints

Explicit policy fields are directly executable by Mooncake. Examples include existing or planned fields such as:

• replica_num
• preferred_segments
• with_soft_pin
• with_hard_pin
• group_ids

These fields tell Mooncake what policy the caller wants to apply, subject to capacity and placement constraints.

Workload hints, on the other hand, are not direct policies. They provide context from which Mooncake may derive policy internally. For the first stage, this could include:

• session_id
• retention_priority

The intended precedence is:

explicit policy > Mooncake-derived policy from hints > Mooncake default policy

For example, if replica_num is explicitly provided, Mooncake should use it directly. If it is not provided, Mooncake may use workload hints such as session_id or retention_priority to decide whether a different replication or retention behavior is appropriate.

This split keeps upper layers from needing to understand Mooncake's internal policy logic. Upper layers can either provide explicit policy when they know exactly what they want, or provide workload hints and let Mooncake make the policy decision.

One detail worth calling out: priority should probably not mean raw request scheduling priority from SGLang or any other runtime. If we include this signal, I suggest naming it retention_priority or clearly documenting it as a Mooncake-side cache-retention priority after normalization.

Also, group_ids should remain an explicit object/grouping field rather than a workload hint. It currently describes the relationship between physical KV objects derived from the same logical cache page, and should not be replaced by session_id.

[Lin-z-w]

One clarification on the two proposed workload hint fields:

I suggest that the first-stage hint struct should contain:

• session_id
• retention_priority

The name retention_priority is intentional. It should not be interpreted as the raw request priority used by runtimes such as SGLang or vLLM.

In SGLang, for example, priority is primarily a request scheduling priority. It tells the scheduler which request should be served first. That is not always the same as cache retention importance. A high-priority request may be urgent but short-lived, while a normal-priority request may contain a shared system prompt or long-lived prefix that is much more valuable to keep in KV cache.

Therefore, Mooncake should not directly consume raw SGLang/vLLM request priority as cache importance. Instead, the runtime integration layer should explicitly translate runtime-specific request priority into Mooncake-defined retention_priority.

The intended semantic chain is:

runtime-specific request priority
    -> integration-layer normalization
    -> Mooncake retention_priority
    -> Mooncake internal retention/eviction policy

This keeps Mooncake's API independent from any particular runtime's scheduling semantics.

For example:

SGLang priority = 10
    does not automatically mean
Mooncake retention_priority = high

The adapter must make that decision based on workload policy.

Suggested first-stage definition:

struct WorkloadHints {
  std::string session_id;

  // Cache-retention priority after runtime-specific normalization.
  // This is not raw SGLang/vLLM request scheduling priority.
  int32_t retention_priority = 0;
};

Mooncake may then map retention_priority to internal mechanisms such as eviction_priority, soft-retention classes, or other eviction policy inputs. The external field should express the retention intent, while Mooncake keeps control over the actual eviction implementation.