Maximize KV-cache reuse for agentic Responses workloads

[franciscojavierarceo]

Objective

Maximize vLLM KV-cache reuse for long-running Responses and Conversation agent workloads deployed through llm-d, while preserving exact model-visible token identity, tenant isolation, and safe fallback behavior.

The primary production win is routing each rehydrated continuation to the replica that owns its longest reusable KV prefix. Compact token artifacts remain valuable, but they are a second-stage CPU, request-transfer, and rendering optimization after fleet-level KV affinity works.

Design and measured evidence: #65

Validated baseline

A two-replica GPT-OSS-20B experiment sent 12 sequential previous_response_id continuation pairs through agentic-api and llm-d for each routing profile. Each continuation contained 6,653 input tokens and used a unique unpredictable cache_salt per pair.

Profile	Warm returns	Mean KV hit	Mean continuation latency
Load-only	6/12	49.54%	811.0 ms
Approximate prefix	12/12	99.08%	314.2 ms
Precise Responses prefix	12/12	99.08%	316.5 ms

Precise routing improved cache reuse by 49.54 percentage points and reduced mean continuation latency by 494.5 ms, or 60.97%, relative to load-only routing. Precise and approximate were effectively tied in this idle deterministic workload; exact routing must next be differentiated under renderer-sensitive and production-load conditions.

A separate fixed-replica gate confirmed cache-salt isolation: a different salt reused zero tokens, while repeating the same salt reused 12,928 tokens.

Target architecture

Client
  -> agentic-api rehydrates durable logical state
  -> llm-d renders or validates exact router-visible prefix identity
  -> llm-d selects the best vLLM replica using its KV-event index
  -> vLLM performs its normal APC lookup and generates
  -> vLLM publishes new KV events to llm-d
  -> agentic-api atomically persists the accepted response/checkpoint

The llm-d inference router must run after agentic-api rehydration. Database response IDs, token artifacts, llm-d canonical block keys, and resident vLLM KV blocks are related but distinct objects and must not be treated as interchangeable handles.

Revised delivery plan

Milestone 1: exact KV-affine Responses routing

• Preserve or derive an unpredictable tenant-scoped cache_salt in agentic-api.
• Add Responses-aware vLLM /tokenize support using the same renderer as inference.
• Add llm-d-router Responses token production through /tokenize.
• Propagate same-host endpoint rankIndex so every vLLM replica has a distinct KV-event subscription.
• Configure agentic-api's execution LLM URL to point at llm-d after rehydration.
• Prove prompt-token parity and compare load-only, approximate, and precise routing.
• Extend the current proof to gateway tool rounds, concurrent conversations, event lag, eviction pressure, restarts, and active-active EPP.

Current status: the single-client parity/cache-hit proof is complete in #65. The agentic-api salt fix is split into #96. The vLLM and llm-d-router benchmark patches still need upstream PRs. Production-load acceptance remains open.

Milestone 2: reliable continuation checkpoints

• Persist immutable execution fingerprints, lineage, safe renderer boundaries, and any token-artifact identity with the accepted response.
• Fail closed for missing, corrupt, unknown, or out-of-order certified history.
• Atomically commit logical output and checkpoint metadata.
• Add conversation ordering, idempotency, and concurrent-turn controls.
• Carry ephemeral affinity state through gateway-tool subturns without promoting rejected/intermediate state.

Milestone 3: portable token artifacts

• Replace process-local prompt handles with bounded, tenant-scoped, expiring artifacts that any selected replica can resolve or reseed.
• Bind artifacts to model, tokenizer, renderer/template/parser, tools, instructions, salt, LoRA, multimodal identity, and version.
• Add typed miss/reseed behavior and safe fallback to full logical rendering.
• Add authenticated llm-d router receipts or server-owned artifact-to-canonical-key resolution; agentic-api must not reimplement llm-d hashing.
• Measure request transfer, reconstruction, rendering, tokenization, and TTFT while holding routing/cache residency constant.

Milestone 4: certified incremental suffix execution

• Certify renderer-specific append boundaries, beginning with Harmony/GPT-OSS.
• Support prompt-plus-append and advanced-checkpoint artifacts without changing model-visible prompt IDs.
• Accumulate complete generated token sequences across streaming and tool rounds.
• Certify additional renderer/model families independently.

Cross-project ownership

• agentic-api: rehydration, tenant salt policy, durable checkpoint correctness, conversation concurrency, tool-loop state, and end-to-end observability.
• vLLM: authoritative Responses rendering/tokenization, artifact execution and validation, APC ownership, generated-token capture, and cache usage reporting.
• llm-d-router: exact Responses token production, canonical prefix scoring, endpoint selection, rank/event discovery, router receipts, and routing metrics.
• llm-d-kv-cache: vLLM-event-to-canonical-key mapping, key-format/version ownership, receipt helpers, active-active index behavior, and storage-medium-aware scoring.
• llm-d deployment: compatible images, block sizes, KV-event endpoints, EPP configuration, and operational rollout.

Existing work items

• Upstream exact Responses prefix routing through llm-d #73 - Stage 1 llm-d routing and router-visible exact prefix identity
• Persist reliable Responses continuation checkpoints #71 - Stage 2 durable checkpoint identity and validation in agentic-api
• Add portable vLLM token artifacts for Responses continuations #70 - Stage 3 portable vLLM token-artifact execution and fallback
• Benchmark KV-affine Responses routing and token artifacts #74 - Benchmark coverage across routing, replay, concurrency, tools, eviction, and model families
• Add Responses WebSocket support for Codex continuations #72 - Responses WebSocket transport support (completed prerequisite)

Acceptance criteria

• Under concurrent agent and tool-loop traffic, precise routing materially improves cached prompt tokens and continuation latency relative to load-only routing.
• /tokenize and inference produce identical prompt token IDs for every certified renderer/model profile.
• Different tenant salts cannot share APC blocks; identical permitted salts retain reuse.
• Wrong-pod selection, eviction, event lag, restart, artifact miss, and AllBlocksCleared remain correctness-safe and observable.
• Durable checkpoints cannot be incorrectly promoted by cancellation, retry, concurrent turns, or corrupt history.
• Portable artifacts preserve the Stage 1 cache-hit ratio while reducing long-context transfer/render/tokenization overhead.
• Neutral or negative results are reported rather than hidden.

Non-goals

• Treating a response ID or process-local vLLM handle as proof of fleet-wide KV residency.
• Reimplementing vLLM rendering or llm-d canonical hashing in agentic-api.
• Introducing a second KV-cache namespace outside vLLM APC.
• Claiming precise routing is faster than approximate routing until template-sensitive or loaded experiments demonstrate it.