Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 19 additions & 8 deletions examples/recipes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,20 +8,31 @@ serve them. Pick a model, pick a combo, `kubectl apply`.
| GLM-5.2-MXFP4 | SGLang | [`glm5.2/`](glm5.2/README.md) |
| Kimi-K3 | vLLM | [`kimi-k3/`](kimi-k3/README.md) |


```{admonition} The directory is `aggregated`, the API field still says `mixed`
:class: note
`aggregated` / `disaggregated` is the industry vocabulary and what these directories
are named. The `role:` field inside each manifest is an **API value consumed by the
operator** — `mixed`, `prefill`, `decode` — and is deliberately left alone: renaming
it would break every deployed configuration and anything outside this repo using the
CRD. So `aggregated/deploy.yaml` legitimately contains `role: mixed`. Read `role:`
as the wire format, and the directory name as what it means.
```

## The four combos

Every recipe comes in the same four shapes. They compose two independent choices:
**how requests are split across GPUs**, and **whether KV survives past the GPU**.

| Combo | Serving | KV cache | Use it when |
|---|---|---|---|
| `mixed` | one worker does prefill + decode | GPU only | default; simplest thing that works |
| `mixed-kvd` | one worker does prefill + decode | + kvd L2 host RAM, L3 on a PVC | repeated/long prefixes — shared system prompts, multi-turn chat, RAG |
| `pd` | prefill and decode on separate nodes | GPU only | prefill and decode want different batching; you have a RoCE fabric |
| `pd-kvd` | prefill and decode on separate nodes | + kvd on each role | both of the above |
| `aggregated` | one worker does prefill + decode | GPU only | default; simplest thing that works |
| `aggregated-kvd` | one worker does prefill + decode | + kvd L2 host RAM, L3 on a PVC | repeated/long prefixes — shared system prompts, multi-turn chat, RAG |
| `disaggregated` | prefill and decode on separate nodes | GPU only | prefill and decode want different batching; you have a RoCE fabric |
| `disaggregated-kvd` | prefill and decode on separate nodes | + kvd on each role | both of the above |

`pd` needs the two nodes on a **mutually routable RoCE fabric** — the KV handoff is
RDMA, and there is no TCP fallback. If you are on one box, use `mixed`.
`disaggregated` needs the two nodes on a **mutually routable RoCE fabric** — the KV handoff is
RDMA, and there is no TCP fallback. If you are on one box, use `aggregated`.

## How these manifests are built

Expand Down Expand Up @@ -59,8 +70,8 @@ The families do not carry the same capabilities, and that is by design:

So each manifest names the capabilities it needs via `INFERA_REQUIRE_NATIVE`
(e.g. `mooncake`, `hipfile`, or `mooncake,hipfile`) and `infera-exec` refuses to
start without them. Combos that need nothing native — `mixed` anywhere, and
`mixed-kvd` on SGLang, which reaches its L2 and file L3 through pure-Python
start without them. Combos that need nothing native — `aggregated` anywhere, and
`aggregated-kvd` on SGLang, which reaches its L2 and file L3 through pure-Python
HiCacheStorage — leave it unset.

See [`deploy/overlay/README.md`](../../deploy/overlay/README.md) for how the payload
Expand Down
30 changes: 15 additions & 15 deletions examples/recipes/glm5.2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,26 +10,26 @@ an upstream SGLang bump is a one-line image-tag edit here.

| Combo | Serving | KV cache | Manifest |
|---|---|---|---|
| **mixed** | one worker, prefill + decode | GPU only | [`mixed/deploy.yaml`](mixed/deploy.yaml) |
| **mixed + kvd** | one worker, prefill + decode | + L2 host RAM, L3 on a PVC | [`mixed-kvd/deploy.yaml`](mixed-kvd/deploy.yaml) |
| **pd** | prefill and decode on separate nodes | GPU only | [`pd/deploy.yaml`](pd/deploy.yaml) |
| **pd + kvd** | prefill and decode on separate nodes | + kvd per role | [`pd-kvd/deploy.yaml`](pd-kvd/deploy.yaml) |
| **aggregated** | one worker, prefill + decode | GPU only | [`aggregated/deploy.yaml`](aggregated/deploy.yaml) |
| **aggregated + kvd** | one worker, prefill + decode | + L2 host RAM, L3 on a PVC | [`aggregated-kvd/deploy.yaml`](aggregated-kvd/deploy.yaml) |
| **disaggregated** | prefill and decode on separate nodes | GPU only | [`disaggregated/deploy.yaml`](disaggregated/deploy.yaml) |
| **disaggregated + kvd** | prefill and decode on separate nodes | + kvd per role | [`disaggregated-kvd/deploy.yaml`](disaggregated-kvd/deploy.yaml) |

Start with **mixed**. Add **kvd** when requests share long prefixes (a common system
prompt, multi-turn chat, RAG) — that is what the L2/L3 tiers pay for. Move to **pd**
Start with **aggregated**. Add **kvd** when requests share long prefixes (a common system
prompt, multi-turn chat, RAG) — that is what the L2/L3 tiers pay for. Move to **disaggregated**
only if you have two nodes on a mutually routable RoCE fabric.

`export COMBO=mixed` (or `mixed-kvd`, `pd`, `pd-kvd`) — the commands below use it.
`export COMBO=aggregated` (or `aggregated-kvd`, `disaggregated`, `disaggregated-kvd`) — the commands below use it.

```{admonition} What each combo needs from the overlay's native tree
The overlay harvests one native tree per ABI family — `rocm7-py310` for SGLang
bases, `rocm7-py312` for vLLM — and each records what it carries:

| combo | needs | why |
|---|---|---|
| `mixed` | nothing | |
| `mixed-kvd` | nothing | SGLang reaches kvd's L2 and file L3 through HiCacheStorage, which is pure Python |
| `pd`, `pd-kvd` | `mooncake` | the KV handoff is Mooncake, a compiled extension |
| `aggregated` | nothing | |
| `aggregated-kvd` | nothing | SGLang reaches kvd's L2 and file L3 through HiCacheStorage, which is pure Python |
| `disaggregated`, `disaggregated-kvd` | `mooncake` | the KV handoff is Mooncake, a compiled extension |

`hipfile` is **absent from the SGLang tree on purpose** — kvd's GPU-direct L3 is
a vLLM-only path — so its absence here is not a broken payload.
Expand All @@ -41,8 +41,8 @@ payload missing Mooncake comes up green and serves with no KV transfer at all.

## 2. Prerequisites

**Hardware.** 8× MI355X (or MI300X+) on one node for `mixed`; two such nodes on a
shared RoCE fabric for `pd`. ~700 GB of host RAM if you enable kvd — its L2 arena is
**Hardware.** 8× MI355X (or MI300X+) on one node for `aggregated`; two such nodes on a
shared RoCE fabric for `disaggregated`. ~700 GB of host RAM if you enable kvd — its L2 arena is
pinned host memory charged to the sidecar's limit.

**Cluster.** Kubernetes 1.28+ with:
Expand Down Expand Up @@ -78,7 +78,7 @@ kubectl create namespace infera --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f examples/recipes/glm5.2/${COMBO}/deploy.yaml
```

For the `pd` combos, first substitute the two node names:
For the `disaggregated` combos, first substitute the two node names:

```bash
sed -e "s/<PREFILL_NODE>/node-a/" -e "s/<DECODE_NODE>/node-b/" \
Expand Down Expand Up @@ -146,11 +146,11 @@ rocm-smi --showpids

| What | Status |
|---|---|
| **`mixed/deploy.yaml` exactly as written** | **validated end-to-end on k3s (MI355X, 8×`amd.com/gpu`)** — see below |
| **`aggregated/deploy.yaml` exactly as written** | **validated end-to-end on k3s (MI355X, 8×`amd.com/gpu`)** — see below |
| kvd sidecar + PVC L3 under SGLang | validated (3674 entries / 421 MB), on Qwen3-0.6B — but see the py310 native-tree note above |
| pd / pd-kvd for this model | the PD path itself is validated cross-node (SGLang + Mooncake over RoCE, chi2800→chi2866); not yet run with these GLM-5.2 settings |

The `mixed` run used the **stock `lmsysorg/sglang` image** with the overlay
The `aggregated` run used the **stock `lmsysorg/sglang` image** with the overlay
mounted in — no infera-built engine image anywhere in the Pod:

| | |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# overlay inferaimage/infera-overlay@sha256:6918eff34f201548a738dd592d2a1ece0627354d2e88f24a87cfa8f787a72a44 (infera + router + kvd + native deps)
# engine infera.engine.sglang TP8
#
# Deploy: kubectl apply -f examples/recipes/glm5.2/mixed-kvd/deploy.yaml
# Deploy: kubectl apply -f examples/recipes/glm5.2/aggregated-kvd/deploy.yaml
# Docs: examples/recipes/glm5.2/README.md
#
# Expects the `model-cache` PVC to hold the weights at /models/GLM-5.2-MXFP4
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# overlay inferaimage/infera-overlay@sha256:6918eff34f201548a738dd592d2a1ece0627354d2e88f24a87cfa8f787a72a44 (infera + router + kvd + native deps)
# engine infera.engine.sglang TP8
#
# Deploy: kubectl apply -f examples/recipes/glm5.2/mixed/deploy.yaml
# Deploy: kubectl apply -f examples/recipes/glm5.2/aggregated/deploy.yaml
# Docs: examples/recipes/glm5.2/README.md
#
# Expects the `model-cache` PVC to hold the weights at /models/GLM-5.2-MXFP4
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# overlay inferaimage/infera-overlay@sha256:6918eff34f201548a738dd592d2a1ece0627354d2e88f24a87cfa8f787a72a44 (infera + router + kvd + native deps)
# engine infera.engine.sglang TP8
#
# Deploy: kubectl apply -f examples/recipes/glm5.2/pd-kvd/deploy.yaml
# Deploy: kubectl apply -f examples/recipes/glm5.2/disaggregated-kvd/deploy.yaml
# Docs: examples/recipes/glm5.2/README.md
#
# Expects the `model-cache` PVC to hold the weights at /models/GLM-5.2-MXFP4
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# overlay inferaimage/infera-overlay@sha256:6918eff34f201548a738dd592d2a1ece0627354d2e88f24a87cfa8f787a72a44 (infera + router + kvd + native deps)
# engine infera.engine.sglang TP8
#
# Deploy: kubectl apply -f examples/recipes/glm5.2/pd/deploy.yaml
# Deploy: kubectl apply -f examples/recipes/glm5.2/disaggregated/deploy.yaml
# Docs: examples/recipes/glm5.2/README.md
#
# Expects the `model-cache` PVC to hold the weights at /models/GLM-5.2-MXFP4
Expand Down
Loading
Loading