feat(llm): normalize prompt cache retention into KV hints

[ishandhanani]

This draft normalizes current OpenAI and Anthropic prompt-cache controls into one provider-neutral full-prompt KV retention envelope and carries it through Dynamo to SGLang. It creates the first narrow KV-hint contract without changing sticky sessions, router placement, admission ordering, or scheduler policy.

CLOSES: DYN-3451
CLOSES: DYN-3420
CLOSES: DYN-3419

Depends on frontend-crates #107, SGLang #30796, and Mooncake #2835. Parent design: SGLang RFC #27574.

How This Was Implemented

• Defines KvHintEnvelope { retain_full_prompt: Option<KvRetentionHint> }, where the retention item carries only ttl_seconds, plus a shared Rust/Python JSON fixture.
• Maps current OpenAI prompt_cache_options implicit 30-minute retention and Anthropic top-level automatic cache_control 5-minute/1-hour retention into serialized request metadata before tokenization.
• Materializes the same envelope after Rust or SGLang tokenization and forwards it through aggregated, prefill, and decode worker paths.
• Rejects unsupported explicit/breakpoint-scoped controls rather than widening them to the full prompt; prompt_cache_key and legacy OpenAI retention remain pass-through only.
• Adds normalized KV hints to DYN_REQUEST_TRACE so replay can attribute the intervention.
• Fails hinted requests visibly when the installed SGLang lacks kv_hints; unhinted traffic remains backward-compatible.

Walkthrough

Mental model

Provider-native fields stop at Dynamo's HTTP boundary. Dynamo emits one small intent envelope after tokenization, and SGLang decides whether and how that intent can be applied safely to its cache.

flowchart LR
    A["OpenAI prompt_cache_options"] --> C["Dynamo provider normalization"]
    B["Anthropic top-level cache_control"] --> C
    C --> D["PipelineContext metadata"]
    D --> E["PreprocessedRequest KvHintEnvelope"]
    E --> F["Dynamo routing and admission unchanged"]
    F --> G["SGLang Engine.async_generate kv_hints"]
    G --> H["HiCache / Mooncake L3 retention"]

Loading

State and ordering

The only v1 action is retain_full_prompt. OpenAI implicit mode maps to 1,800 seconds; Anthropic top-level ephemeral maps to its typed 300- or 3,600-second TTL. The envelope accepts unknown future fields so a v1 receiver can preserve its known action instead of rejecting the entire request.

Retention metadata lives in serialized request-plane context rather than a process-local registry. This covers Rust-tokenized, --use-sglang-tokenizer, ModelInput::Text, multimodal, aggregated, prefill, and decode paths without teaching the router about provider semantics.

Request lifecycle

1. The HTTP adapter validates the provider-native cache control and records only the normalized TTL.
2. The preprocessor attaches a full-prompt envelope after tokenization, alongside existing routing metadata.
3. Router and admission components forward the envelope but do not interpret it.
4. Python SGLang handlers reconstruct the envelope from request metadata and pass kv_hints to Engine.async_generate.
5. Request tracing serializes the normalized envelope with replay hashes and cache outcomes.
6. Invalid provider controls return a client error; an old SGLang engine returns a visible compatibility error; cache-layer application remains SGLang's fail-open responsibility.

Boundaries and limitations

• OpenAI support is latest-only: implicit prompt_cache_options with the typed 30-minute TTL. Explicit mode is rejected.
• Anthropic v1 supports top-level automatic cache_control; system/tool/message block breakpoints are rejected until formatter-aware token offsets exist.
• prompt_cache_key is neither retention intent nor a new affinity key. Existing Dynamo sticky sessions remain authoritative for affinity.
• This PR does not add prefetch, demote, share, release, routing weights, or admission behavior.
• The temporary dynamo-protocols git patch is draft-only and must be replaced by a released frontend-crates version.

Validation

• CARGO_INCREMENTAL=0 cargo test -p dynamo-llm --lib — 1,621 passed, 5 ignored.
• .venv/bin/python -m pytest -q components/src/dynamo/sglang/tests/test_sglang_unit.py -k kv_hint — 2 passed.
• cargo fmt --all -- --check and git diff --check origin/main...HEAD — passed.
• Two-worker MiniMax M2.7 TP4 H100 validation through real Codex and Claude Code, SGLang HiCache, Mooncake, compressed DYN_REQUEST_TRACE, and Tachometer — passed functional retention attribution.

Benchmark Results

The final A/B forced seed and probe requests to different workers, drove Mooncake eviction, flushed local caches, and observed zero Dynamo router overlap on every final probe.

Probe	Control L3 tokens	Retained L3 tokens	Normalized hint	Probe latency delta
Codex Responses	4,640 shared-prefix tokens	20,800 tokens	retain_full_prompt.ttl_seconds=1800	+23.1%
Claude Code Messages	0 tokens	13,600 tokens	retain_full_prompt.ttl_seconds=3600	+34.0%

Pressure outcome	Result
Completed pressure requests	70
Total pressure input	1,146,881 tokens
Mooncake eviction	144,904 keys / 34.27 GiB
Trace attribution	Controls had no hint; treatments carried 1,800/3,600 seconds

This validates the contract and retention path, not a performance win. Mooncake retrieval was slower than recompute on the tested single-node topology, so policy enablement remains separate from this API PR.

[datadog-official]

⚠️ Warnings

🚦 12 Pipeline jobs failed

PR | backend-status-check     

PR | dynamo-runtime / image / Build multi-arch cuda13.0     

PR | dynamo-status-check     

View all 12 failed jobs.

ℹ️ Info

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 0.07% (-47.24%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 530a870 | Docs | Give us feedback!}