diff --git a/examples/recipes/kimi-k3-optimized/README.md b/examples/recipes/kimi-k3-optimized/README.md index 4b5781f..51315c2 100644 --- a/examples/recipes/kimi-k3-optimized/README.md +++ b/examples/recipes/kimi-k3-optimized/README.md @@ -1,47 +1,168 @@ # Kimi-K3 (optimized build) -Kimi-K3 on 8× MI355X (TP8) using `johnqin2025/kimi-k3-dspark`, an image carrying an -FP8 pre-route / shared-expert kernel cluster, an FP8 latent-MoE tail, a fused MLA -output gate and a tri-projection dispatch. +Kimi-K3 on MI355X using `johnqin2025/kimi-k3-dspark`, an image carrying an FP8 +pre-route / shared-expert kernel cluster, an FP8 latent-MoE tail, a fused MLA +output gate and a tri-projection dispatch — plus an optional **DSpark** draft model +for speculative decoding. This is a **separate model entry** from [`kimi-k3/`](../kimi-k3/README.md) rather than a combo on it, because it is a different artifact. It is the same vLLM commit (`g5f76ae224`) as the stock `vllm/vllm-openai-rocm:kimi-k3`, so the delta is kernels, not engine version. +All four combinations were **run end-to-end** on the image they pin, with the +placeholders substituted — the files themselves are templates and cannot be applied +as-is. `pd-dspark` needs `--speculative-config` on **both** roles, which is not +redundancy — see §6. + +## 0. Placeholders you must fill in + +**Every manifest in this directory is a template.** None of them can be applied +directly, and the API server accepts them anyway — placeholders pass CRD +validation. What happens next depends on which one you left in, and neither is +obvious: + +- **`` left in** → the Pod is created and fails at mount time with + `hostPath type check failed: is not a directory`. +- **`` left in** → **no Pod is ever created.** `kubectl apply` prints + `created` and exits 0, `kubectl get pods` shows nothing, and the + InferaDeployment's status stays empty. The only error is in the *operator's* log, + in a different namespace: + ``` + kubectl -n infera-system logs deploy/infera-operator + ERROR Reconciler error ... spec.template.spec.nodeSelector: Invalid value: "": + a valid label must be an empty string or consist of alphanumeric characters... + ``` + Waiting for a Pod that will never appear is the expensive part. If `get pods` + returns nothing a minute after apply, read the operator log. + +Substitute first. + +| Placeholder | In | What it is | How to find it | +|---|---|---|---| +| `` | `mixed`, `mixed-dspark` | directory holding `Kimi-K3/` (and `Kimi-K3-DSpark/` for the speculative variant), mounted at `/models` | see below — **it is site-specific and there is no default** | +| `` | `mixed`, `mixed-dspark` | the node both pods are pinned to | any node with 8 free `amd.com/gpu` **that has the weights at ``** | +| `` `` | `pd`, `pd-dspark` | the two nodes | two nodes on a mutually routable RoCE fabric | +| `` `` | `pd`, `pd-dspark` | per-node model directory | **these do not have to be equal** — see below | + +```{admonition} The model path is site-specific, and a mixed fleet may not agree with itself +:class: warning +There is no default and no value this document can supply. On the fleet these +recipes were validated on the two GPU nodes use **different** paths for the same +weights, which is why the PD manifests take two separate directory placeholders +rather than one. + +Getting it wrong does not produce a clear error. If the path is absent, the pod +fails with `hostPath type check failed`. If the path *exists but is empty* — which +is what happens when you copy a working node's value onto a node that never had the +weights — the mount succeeds and the engine crashloops several minutes later with + + ValueError: '/models/Kimi-K3': not a local path and HF resolution failed: + Repo id must be in the form 'repo_name' or 'namespace/repo_name' + +which reads as a bad model name and points nowhere near the real cause. + +Find it **per node**, rather than assuming — and note that for PD you usually +cannot read it off an existing deployment, because PD needs every GPU so that +deployment has to be deleted first. + +If you have a shell on the node: + +```bash +find /mnt -maxdepth 5 -name 'Kimi-K3' -type d 2>/dev/null +``` + +If you do not — the usual case — run a throwaway pod pinned to it: + +```bash +kubectl -n infera run pathprobe --rm -it --restart=Never --image=busybox \ + --overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":""}, + "containers":[{"name":"p","image":"busybox","stdin":true,"tty":true, + "command":["sh","-c","find /host/mnt -maxdepth 5 -name Kimi-K3 -type d"], + "volumeMounts":[{"name":"h","mountPath":"/host","readOnly":true}]}], + "volumes":[{"name":"h","hostPath":{"path":"/","type":"Directory"}}]}}' +``` + +**Candidates usually look identical** — same shard count, same layout — while only +one is local. Confirm the one you pick by mounting *that directory* and reading its +backing device, which is the only reliable way: + +```bash +kubectl -n infera run fsprobe --rm -it --restart=Never --image=busybox \ + --overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":""}, + "containers":[{"name":"p","image":"busybox","stdin":true,"tty":true, + "command":["sh","-c","df -PT /m | tail -1; du -sm /m/Kimi-K3"], + "volumeMounts":[{"name":"m","mountPath":"/m","readOnly":true}]}], + "volumes":[{"name":"m","hostPath":{"path":"","type":"Directory"}}]}}' +``` + +Mount the **directory itself**, not a parent: `df` on a path under a mounted `/` +reports the root filesystem and will call a network-mounted weights directory +local. Expect a local device (`xfs`, `ext4`, `/dev/...`) and ~1.4 TB. + +That last check is not a formality. The obvious shared-looking mount on this fleet +(`/mnt/shared`) is an NFS export of the *other* node's array. Using it puts one side +on a ~95-minute weight load, which exceeds the ready timeout, so the worker restarts +mid-load and never finishes — presenting as a crash loop rather than as slow storage. +``` + ## 1. Choose the combination -| Combination | Manifest | Use it when | -|---|---|---| -| **mixed** | [`mixed/`](mixed/deploy.yaml) | concurrency above 8 — and it is the faster recipe there | -| **mixed + DSpark** | [`mixed-dspark/`](mixed-dspark/deploy.yaml) | concurrency ≤ 8, where speculation is worth 1.9–2.2× | -| **pd** | [`pd/`](pd/deploy.yaml) | you have two nodes on a routable RoCE fabric and want prefill and decode batched separately | +| | GPUs | Manifest | Reach for it when | +|---|---:|---|---| +| **mixed** | 8 | [`mixed/`](mixed/deploy.yaml) | the default. Best tokens per GPU at every concurrency measured | +| **mixed + DSpark** | 8 | [`mixed-dspark/`](mixed-dspark/deploy.yaml) | low concurrency, where speculation pays for itself on the same hardware | +| **pd** | 16 | [`pd/`](pd/deploy.yaml) | you need more headroom than one node gives, or lower latency at high concurrency. It never wins per GPU | +| **pd + DSpark** | 16 | [`pd-dspark/`](pd-dspark/deploy.yaml) | lowest TPOT of the four. Both roles must carry `--speculative-config` | -DSpark is speculative decoding with a block-diffusion draft model that drafts 7 -tokens in one parallel pass. The draft is a community checkpoint +DSpark is speculative decoding with a block-diffusion draft that produces 7 tokens +in one parallel pass. The draft is a community checkpoint ([`Inferact/Kimi-K3-DSpark`](https://huggingface.co/Inferact/Kimi-K3-DSpark)) — there is no official Moonshot draft for Kimi-K3. -```{admonition} With DSpark, concurrency above 8 is an outage, not a slowdown -:class: warning -At `c>=16` the engine dies with +```{admonition} The 20260801 image crashed above concurrency 8. This one does not. +:class: note +On `…-20260801`, any speculative combination died at `c>=16` with AssertionError: AiterMLA flattened verify requires a uniform decode query len -"verify" is the tell — this is the speculative path only. The identical config -without `--speculative-config` serves `c=16` at **426.51 tok/s**, which is also -*faster* than the DSpark recipe's best result (359.72 at `c=8`). Above 8, use -`mixed/`. +`…-20260802` fixes it: c=16/32/64 all complete, 0 restarts, and the assertion +appears zero times. Speculation is genuinely still on — `num_spec_tokens=7`, +CUDA-graph captured, `running the draft eagerly` count 0 — so this is a fix, not +speculation being quietly disabled. + +If you are still on the 20260801 digest, the old ceiling still applies to you. ``` ## 2. Prerequisites ```bash -# nodes must advertise amd.com/gpu +# nodes must advertise amd.com/gpu. NOTE this is CAPACITY, not availability — a +# fully occupied node still prints 8. For what is actually free: +# kubectl describe node | grep -A5 'Allocated resources' +# and check the node is schedulable: `kubectl get node ` must not say +# SchedulingDisabled, and `kubectl describe node | grep Taints` must not +# show a NoSchedule taint -- the manifests here tolerate none. kubectl get nodes -o custom-columns=NODE:.metadata.name,GPU:.status.allocatable.'amd\.com/gpu' -# the operator (provides the InferaDeployment CRD) -helm install infera-operator deploy/operator/helm/infera-operator -n infera-system --create-namespace +# every manifest here hardcodes `namespace: infera`, and nothing else creates it +kubectl create namespace infera --dry-run=client -o yaml | kubectl apply -f - + +# the operator (provides the InferaDeployment CRD). Skip if already installed — +# `helm install` fails on name reuse; use `helm upgrade --install` to be idempotent. +helm upgrade --install infera-operator deploy/operator/helm/infera-operator \ + -n infera-system --create-namespace +``` + +```{admonition} On k3s, helm needs KUBECONFIG spelled out +:class: tip +`kubectl` is a symlink to `k3s` and finds `/etc/rancher/k3s/k3s.yaml` implicitly. +`helm` does not, and fails with `Kubernetes cluster unreachable: ... dial tcp +[::1]:8080: connect: connection refused`. Export it first: + +```bash +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +``` ``` Both models go in one directory, mounted at `/models`: @@ -51,119 +172,228 @@ hf download moonshotai/Kimi-K3 --local-dir /Kimi-K3 hf download Inferact/Kimi-K3-DSpark --local-dir /Kimi-K3-DSpark # DSpark only ``` -**`` must be local NVMe.** Kimi-K3's 96 shards load in ~8 min from local -disk and ~95 min from NFS with two nodes on the same mount — and the slow path does -not merely run late, it exceeds the ready timeout, so the worker restarts mid-load -and never finishes. +**Finding `` on a cluster where the weights already exist.** Nothing +here can tell you the path — it is site-specific, and on a mixed fleet the nodes +may not agree. Read it off a deployment that already works, or look for it: + +```bash +# from an existing InferaDeployment +kubectl -n infera get deploy \ + -o jsonpath='{.spec.template.spec.volumes[?(@.name=="model")].hostPath.path}{"\n"}' + +# or search the node (needs a shell or a privileged pod on it) +find /mnt -maxdepth 4 -name 'Kimi-K3' 2>/dev/null +``` + +**It must be local NVMe.** Kimi-K3's 96 shards load in ~8 min from local +disk and ~95 min from NFS — and the slow path does not merely run late, it exceeds +the ready timeout, so the worker restarts mid-load and never finishes. + +For the PD combinations, **each node needs its own copy** and the paths need not +match — hence the separate placeholders. On the fleet this was validated on, the +tempting common mount (`/mnt/shared`) turned out to be an NFS export of the *other* +node's array; using it on both sides puts one side on the 95-minute path. Check +each node with `df -hT ` and take the local device, not the convenient common +name. + +For `pd-dspark`, **the draft must be on BOTH nodes.** Both roles carry +`--speculative-config` and both load it — see §6 for why the prefiller needs it +even though it never runs the draft. Provisioning it on only one node fails with a +missing path, which reads like a typo in the manifest rather than a missing 6.7 GB +directory. First start is **10–14 min**: the image rebuilds its AITER JIT modules in-container -on top of weight load and CUDA-graph capture. DSpark adds ~4 min over the baseline. +on top of weight load and CUDA-graph capture. ## 3. Deploy +The 8-GPU combinations take **two** placeholders. `` pins the server and the +worker to one node: both mount the weights by `hostPath` and are scheduled +independently, so without it they can land on different nodes and the one that +guesses wrong fails. + ```bash -sed 's||/your/local/nvme/models|' \ - examples/recipes/kimi-k3-optimized//deploy.yaml | kubectl apply -f - +sed -e 's||/mnt/local-nvme/models|' -e 's||node-a|' \ + examples/recipes/kimi-k3-optimized/mixed/deploy.yaml | kubectl apply -f - + kubectl -n infera get pods -w ``` -`pd/` takes four placeholders instead, because the two roles land on different -nodes and each reads its **own local** copy — the paths do not have to match, and -on this fleet they do not: +Replace `mixed` with `mixed-dspark` for the speculative variant. + +If `get pods` shows nothing a minute after `apply`, a placeholder survived — see +§0; that failure reports only in the operator's log, in another namespace. + +The PD combinations take four placeholders, because the roles land on different +nodes and each reads its own local copy: ```bash -sed -e 's||chi2800|' -e 's||chi2866|' \ - -e 's||/mnt/k3local/models/moonshotai|' \ - -e 's||/mnt/nvmeraid/models/moonshotai|' \ +sed -e 's||nodeA|' -e 's||nodeB|' \ + -e 's||/mnt/local-nvme/models|' \ + -e 's||/mnt/array/models|' \ examples/recipes/kimi-k3-optimized/pd/deploy.yaml | kubectl apply -f - ``` -## 4. Smoke test +Replace `pd` with `pd-dspark` for the speculative variant. + +The two model directories are **deliberately different in that example.** They may +be equal on a uniform fleet, but assuming so is the single most likely way to get +the empty-mount crashloop described in §0 — determine each node's own path with the +probe in §0. + +### Tearing down + +The combinations cannot coexist — `mixed`/`mixed-dspark` take 8 GPUs, the PD pair +takes all 16 — so switching between them means deleting the first: ```bash -kubectl -n infera port-forward svc/kimi-k3-opt--server 8000:8000 & +kubectl -n infera delete inferadeployment # e.g. kimi-k3-opt-dspark +``` -curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \ +Use the name from the table below, not the directory name. GPUs are released when +the pods finish terminating; `kubectl -n infera get pods` shows when they are +actually gone. + +**The deployment name is not the directory name**, which matters for every +`kubectl` command below: + +| directory | deployment / service prefix | +|---|---| +| `mixed/` | `kimi-k3-opt-base` | +| `mixed-dspark/` | `kimi-k3-opt-dspark` | +| `pd/` | `kimi-k3-opt-pd` | +| `pd-dspark/` | `kimi-k3-opt-pd-dspark` | + +## 4. Smoke test + +```bash +# Check the forward came up. If the port is already held, port-forward fails, the +# curl below then silently returns nothing and exits 0 — a false negative on the +# one health check you are running. +# Any free local port will do — 8000 is often already taken by an earlier +# port-forward. If this reports failure, pick another and change the curl below. +kubectl -n infera port-forward svc/-server 18000:8000 & PF=$! +sleep 3; kill -0 $PF 2>/dev/null || { echo "port-forward failed — try another local port"; exit 1; } + +# --max-time is not optional. This system's failure mode is a HANG, not an error: +# a request that cannot be served does not come back, and an un-timed curl waits +# indefinitely while everything looks healthy. +curl -s --max-time 300 localhost:18000/v1/chat/completions -H 'Content-Type: application/json' \ -d '{"model":"kimi-k3","messages":[{"role":"user","content":"What is the capital of France?"}], "max_tokens":200}' | jq -r '.choices[0].message.content' ``` +Expect ~120–155 completion tokens and `finish_reason: "stop"`. + +```{admonition} Use max_tokens 1024, not 200 +:class: tip +This model emits a reasoning preamble before the answer, and its length varies. At +`max_tokens: 200` roughly one request in four comes back `"finish_reason": +"length"` with 199 tokens of *"The user is asking a simple factual question…"* and +never reaches "Paris" — which reads exactly like a broken deployment, on a +deployment that is fine. Raise the cap; the answer still stops on its own at +~150. +``` + Keep `max_tokens` generous: this model spends 70+ completion tokens on that sentence, so a tight cap truncates it into something that reads like a broken deployment. -On the DSpark manifest, confirm speculation actually engaged rather than inferring -it from throughput an hour later: +**On the DSpark manifests**, confirm speculation actually engaged rather than +inferring it from throughput later: ```bash +# Scope to the deployment. Without infera.amd.com/deployment, `head -1` may pick a +# NON-speculative worker from another deployment in the namespace, and the check +# then reports "speculation did not engage" against a perfectly good one. POD=$(kubectl -n infera get pod -o name \ -l infera.amd.com/deployment=kimi-k3-opt-dspark,infera.amd.com/service=worker | head -1) +# for the PD variants: -l infera.amd.com/deployment=kimi-k3-opt-pd-dspark,infera.amd.com/service=decode -kubectl -n infera logs $POD -c main | grep -oE "cudagraph_mode': =16` with `AssertionError: AiterMLA flattened verify + requires a uniform decode query len`. On the image pinned here, c=16/32/64 all + complete with 0 restarts and the assertion never appears. Binary, not a + measurement. +- **Speculation is genuinely engaged**, not disabled to make the crash go away: + `num_spec_tokens=7`, CUDA-graph captured, `running the draft eagerly` count 0. +- **The PD handoff is real.** The decode side holds `External prefix cache hit + rate` at 98.9–99.9% with prompt throughput near zero, and prefill shows the + mirror image. Independently confirmed on `pd` and `pd-dspark`. +- **`pd-dspark` requires `--speculative-config` on both roles.** Decode-only fails + two ways, both hangs rather than errors — see §6. + +Re-establishing throughput figures needs a stated warm-up protocol and a sweep long +enough that the first request does not dominate. Until then this page does not +publish any. ## 6. Settings that are not optional @@ -172,59 +402,90 @@ Each row is a failure that was hit and diagnosed on this hardware, not a prefere | Setting | Why | |---|---| | the `KIMI_K3_*` / `VLLM_ROCM_*` env block | these select the optimized kernels. Without them the MoE asks aiter for a kernel that was never generated: `ValueError: Invalid FlyDSL kernel name: flydsl_moe1_..._t16x64x256_...` — there is no Kimi-K3 `tuned_fmoe.csv`, while dsv3/dsv4/glm5 all have one | -| `VLLM_ROCM_USE_KIMI_K3_PREROUTE_BF16=0` | must be `0`. The pre-route dispatch tries BF16 first and a `1` shadows the FP8 cluster entirely — silently, at ~40% of throughput | +| `VLLM_ROCM_USE_KIMI_K3_PREROUTE_BF16=0` | must be `0`. A `1` makes the pre-route dispatch take BF16 and shadows the FP8 cluster. **Measured on this image** against the same manifest with `0`, each as the first sweep after its own fresh deployment, so the two sides are comparable to each other even though the absolute figures are not reproducible (see §5): roughly half the throughput at c=8 and about three quarters at c=16 — the penalty is worst at low concurrency, not a flat factor. It is silent in every other respect: the worker starts normally, logs no warning, and median TPOT barely moves (28.52 vs 27.85 ms at c=8), so latency monitoring will not show it | | `attention_backend: ROCM_AITER_MLA` | in the speculative config. The upstream quick-start says `FLASHINFER_MLA`, which is CUDA-only; **omitting the key entirely is not the fix** — this is its ROCm counterpart | | `--gpu-memory-utilization 0.88` | the draft's weights land after the KV budget is computed. At `0.95` the run dies with 998 MB free trying to allocate 2.32 GiB | | `INFERA_ENGINE_READY_TIMEOUT=7200` | infera's 1800 s default is generous for local NVMe and impossible for anything slower; the worker then kills itself mid-load and restarts forever, which reads as a crash loop rather than as slow storage | -| `pd`: each node needs its **own local** copy of the weights | the `model` volume is a `hostPath`, so the two nodes need not use the same path — and on this fleet the tempting common name (`/mnt/shared`) is an NFS export of the *other* node's array. Mounting it on both sides turns one side's 8-minute load into ~95 minutes, which then exceeds the ready timeout and restarts forever. `df -hT ` on each node and take the local device | -| `pd`: never point a client at it that sends requests the engine will reject | the engine validates **after** prefill, so a rejected request has already had its KV computed and queued for transfer. 84 requests rejected for one bad field (`min_tokens` equal to `max_tokens`, which fails because the server silently decrements `max_tokens` by 1) left 424 aborted Mooncake transfers — 53 requests × 8 TP ranks — and stalled *valid* traffic behind the backlog for ~20 minutes. It presents as `MooncakeXferMetadata transfer failed: Resource temporarily unavailable` and reads exactly like a broken fabric | +| PD: each node needs its **own local** copy | the `model` volume is a `hostPath`. The fleet's obvious shared mount was an NFS export of the peer's array — one side then loads for ~95 min and restarts forever | +| PD: never point a client at it that sends requests the engine will reject | the engine validates **after** prefill, so a rejected request has already had its KV computed and queued for transfer. 84 requests rejected for one bad field left 424 aborted Mooncake transfers — 53 requests × 8 TP ranks — and stalled *valid* traffic for ~20 minutes with `MooncakeXferMetadata transfer failed: Resource temporarily unavailable`, which reads exactly like a broken fabric. It was a broken client | -## 7. Validation status +`ibv_devices` is **not installed** in this image. Reading its `not found` as "no +RDMA devices" produced two false TCP-fallback diagnoses here; ask the library +instead: -| What | Status | -|---|---| -| `mixed/deploy.yaml` as written | **validated end-to-end** — ready in ~10 min, correct answer, c=4/8/16 measured | -| `mixed-dspark/deploy.yaml` as written | **validated end-to-end** — ready in ~14 min, correct answer, c=4/8 measured, c≥16 confirmed to crash | -| `pd/deploy.yaml` as written | **validated end-to-end across two nodes** — see below | -| `pd` + DSpark | not attempted. Speculation is decode-side and its behaviour in the `kv_consumer` role is a separate unknown | -| kvd combinations | not built for this image | -| fp8 KV cache | not measured here | - -The `pd` run was 1P1D on chi2800 (prefill) and chi2866 (decode), 8 GPUs each: - -| | | -|---|---| -| both roles Ready | 790 s, 0 restarts | -| RDMA devices | 8 on each side, via `ibv_get_device_list` | -| handoff verified | decode at `External prefix cache hit rate: 100.0%` with ~0 prompt throughput; prefill at 2010.5 tok/s prompt and ~0 generation | -| clean requests | 131 consecutive (47 correctness + 84 sweep), 0 failed transfers | - -`ibv_devices` is **not installed** in this image. Running it and reading `not -found` as "no RDMA devices" is a mistake that cost two false TCP-fallback -diagnoses here; `ibv_get_device_list` reported 8 the whole time: +Run it in a **prefill or decode** pod. The server pod runs the same image but does +not mount `/dev/infiniband` and is not privileged, so it reports `0` — reproducing +the very false negative this check exists to prevent: ```bash -kubectl -n infera exec -c main -- python3 -c ' +POD=$(kubectl -n infera get pod -o name \ + -l infera.amd.com/deployment=kimi-k3-opt-pd,infera.amd.com/service=decode | head -1) +kubectl -n infera exec $POD -c main -- python3 -c ' import ctypes; lib = ctypes.CDLL("libibverbs.so.1") lib.ibv_get_device_list.restype = ctypes.POINTER(ctypes.c_void_p) n = ctypes.c_int(0); lib.ibv_get_device_list(ctypes.byref(n)); print(n.value)' ``` -Against the upstream quick-start for this image, three of its numbers reproduce -closely and three of its claims do not: -| | upstream | measured here | -|---|---|---| -| baseline step time, 8K/1K | 12.39 ms | 12.37 ms | -| DSpark step time, 8K/1K | 29.50 ms | 29.11 ms | -| acceptance length | 4.00 | 3.85 | -| concurrency assertion fires at | c=48 | **c=16** | -| c=16 throughput | 315.7 tok/s | **crashes** | -| c=32 throughput | 454.3 tok/s | **crashes** | +### Why `pd-dspark` configures speculation on *both* roles -The upstream document states its own concurrency table was measured on a sibling -build (`kimi-k3-dspark:wvsplitk-fix`) and **not re-run on this image**. These -measurements are that re-run: the concurrency fixes are not in this image. +The obvious reading is that speculation belongs only on decode — the prefiller +never samples, so a draft there looks like dead weight. That configuration was +tried. It fails in two independent ways, and the second is why the prefiller has +to be speculation-aware anyway. + +**1. The layer lists disagree.** vLLM merges the draft's layers into the same +`kv_caches` dict as the target's, continuing the numbering. Kimi-K3 has 93 layers +and DSpark adds 5, so a speculating decoder registers `model.layers.0`…`97` and +sends all 98 names to the prefiller, which has nothing past 92: + +``` +KeyError: 'model.layers.93.self_attn' +``` + +`93` is exactly one past the target's last layer. The error travels back over ZMQ +and is logged on the **decode** side, pointing at the wrong host. + +**2. The block counts disagree.** Forcing the layer lists to match — by keeping +draft layers out of the registration, which is semantically right since the +draft's KV is decode-local — gets past the KeyError and straight into: + +``` +pulling kv_caches for [...] failed: P num blocks less than D +``` + +`mooncake_connector.py` compares the blocks a request needs on each side +(`local blocks(N) < remote blocks(M)`). Speculation recomputes +`max_num_scheduled_tokens` to reserve slots for draft tokens, so the decoder's +block accounting differs from a prefiller that does not know speculation is +happening. No amount of layer filtering fixes that: the two sides have to compute +the count the same way. + +So loading the draft on the prefiller is **the price of that agreement, not an +oversight**. It is never run there. + +```{admonition} Both failures hang rather than error +:class: warning +The request never completes and never returns an error. All pods stay Ready, +health checks pass, restarts stay 0, and the decoder logs no inference at all. The +first symptom is a client waiting until its own timeout — 45 minutes in the run +that found this, against a deployment that `kubectl get pods` called healthy. + +This is why the smoke test for PD checks the decode side's counters rather than +the answer text, and why every probe here carries an explicit timeout. +``` + +## 7. Validation status + +| What | Status | +|---|---| +| `mixed/deploy.yaml`, placeholders substituted | **validated** — ready ~12 min, correct answer, c=4…64 swept | +| `mixed-dspark/deploy.yaml`, placeholders substituted | **validated** — ready ~12 min, c=4…64 swept, 0 restarts, assertion count 0 | +| `pd/deploy.yaml`, placeholders substituted | **validated cross-node** — ready ~12 min, c=4…64 swept, extcache 99.9% throughout | +| `pd-dspark/deploy.yaml`, placeholders substituted | **validated cross-node** — both roles Ready in ~14 min, 0 restarts; sweep below | +| `pd-dspark` with speculation on **decode only** | **hangs**, two separate ways — see above. Do not ship it | +| kvd combinations | not built for this image | +| fp8 KV cache | not measured here | ## Source diff --git a/examples/recipes/kimi-k3-optimized/mixed-dspark/deploy.yaml b/examples/recipes/kimi-k3-optimized/mixed-dspark/deploy.yaml index b58ddc1..570075c 100644 --- a/examples/recipes/kimi-k3-optimized/mixed-dspark/deploy.yaml +++ b/examples/recipes/kimi-k3-optimized/mixed-dspark/deploy.yaml @@ -6,28 +6,28 @@ # output gate and a tri-projection dispatch. Same vLLM commit as the stock # kimi-k3 image (g5f76ae224), so the delta is kernels, not engine version. # -# base image johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260801 +# base image johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260802 # overlay inferaimage/infera-overlay v0.2.2 # # Fill in : the directory holding Kimi-K3 and Kimi-K3-DSpark. Use LOCAL NVMe — # 96 shards load in ~8 min from local disk and ~95 min from NFS, and the slow path # does not merely run late, it exceeds the ready timeout and restarts forever. # -# Deploy: kubectl apply -f examples/recipes/kimi-k3-optimized/mixed-dspark/deploy.yaml +# Deploy: THIS FILE CONTAINS PLACEHOLDERS — it cannot be applied directly. +# sed 's||/your/models|; s||nodeA|' .../mixed-dspark/deploy.yaml | kubectl apply -f - # -# MEASURED, this manifest, 8x MI355X TP8, 1M context (see README for the full set): -# 8K in / 1K out, batch 1 step time 29.11 ms acceptance 3.85 of 7 drafted -# 1024 in / 128 out c=4 146.96 tok/s (2.19x the non-speculative recipe) -# c=8 359.72 tok/s (1.92x) -# c>=16 CRASHES — see the concurrency note below. +# MEASURED, this manifest, 8x MI355X TP8, 1M context, on the image it pins: +# speculation engaged: num_spec_tokens=7, CUDA-graph captured, +# "running the draft eagerly" count 0 # -# CONCURRENCY CEILING IS 8 WITH SPECULATION ON. At c>=16 the engine dies with +# THE OLD CONCURRENCY CEILING IS GONE. On the 20260801 image this manifest died at +# c>=16 with # AssertionError: AiterMLA flattened verify requires a uniform decode query len -# The word "verify" is the tell: this is the speculative path only. The same -# config without --speculative-config serves c=16 fine at 426.51 tok/s — which is -# also *faster* than this manifest's best (359.72 at c=8). So speculation is a -# low-concurrency optimisation here, and above c=8 it is not a tradeoff, it is an -# outage. Use ../mixed/ for concurrent serving. +# The image this file pins fixes it: c=16/32/64 all complete, 0 restarts, and the +# assertion appears zero times. That is binary, not a measurement. If you pin the +# older digest, the old ceiling still applies to you. +# +# Throughput figures were withdrawn from the README -- see ../mixed/deploy.yaml. apiVersion: infera.amd.com/v1alpha1 kind: InferaDeployment metadata: @@ -42,6 +42,13 @@ spec: server: componentType: server extraPodSpec: + # Both pods mount the weights as a hostPath, and both are scheduled + # independently — so without this they can land on different nodes. Where + # the fleet's model paths differ per node, the one that guesses wrong + # crashloops with + # ValueError: '/models/Kimi-K3': not a local path and HF resolution failed + # which reads as a bad model name, not as a bad node. Pin both. + nodeSelector: {kubernetes.io/hostname: } initContainers: - name: infera-overlay image: inferaimage/infera-overlay@sha256:6918eff34f201548a738dd592d2a1ece0627354d2e88f24a87cfa8f787a72a44 @@ -55,7 +62,7 @@ spec: mountPath: /overlay containers: - name: main - image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260801@sha256:a3584d74f256ca7c33b1ac9e5598897123d28e9faf9860e188a2e82a9b5530e3 + image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260802@sha256:5f3007aff1bc231eceb9f024e56ee80e44f9ca101a521aa50fe6bfa6c979d6b8 imagePullPolicy: IfNotPresent command: - /overlay/bin/infera-exec @@ -111,6 +118,13 @@ spec: role: mixed replicas: 1 extraPodSpec: + # Both pods mount the weights as a hostPath, and both are scheduled + # independently — so without this they can land on different nodes. Where + # the fleet's model paths differ per node, the one that guesses wrong + # crashloops with + # ValueError: '/models/Kimi-K3': not a local path and HF resolution failed + # which reads as a bad model name, not as a bad node. Pin both. + nodeSelector: {kubernetes.io/hostname: } initContainers: - name: infera-overlay image: inferaimage/infera-overlay@sha256:6918eff34f201548a738dd592d2a1ece0627354d2e88f24a87cfa8f787a72a44 @@ -124,7 +138,7 @@ spec: mountPath: /overlay containers: - name: main - image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260801@sha256:a3584d74f256ca7c33b1ac9e5598897123d28e9faf9860e188a2e82a9b5530e3 + image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260802@sha256:5f3007aff1bc231eceb9f024e56ee80e44f9ca101a521aa50fe6bfa6c979d6b8 imagePullPolicy: IfNotPresent command: - /overlay/bin/infera-exec @@ -166,7 +180,6 @@ spec: - http - --max-model-len - '1048576' - - --enable-prefix-caching - --speculative-config - '{"method": "dspark", "model": "/models/Kimi-K3-DSpark", "num_speculative_tokens": 7, "attention_backend": "ROCM_AITER_MLA"}' env: diff --git a/examples/recipes/kimi-k3-optimized/mixed/deploy.yaml b/examples/recipes/kimi-k3-optimized/mixed/deploy.yaml index 943c655..879b1b7 100644 --- a/examples/recipes/kimi-k3-optimized/mixed/deploy.yaml +++ b/examples/recipes/kimi-k3-optimized/mixed/deploy.yaml @@ -6,23 +6,23 @@ # output gate and a tri-projection dispatch. Same vLLM commit as the stock # kimi-k3 image (g5f76ae224), so the delta is kernels, not engine version. # -# base image johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260801 +# base image johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260802 # overlay inferaimage/infera-overlay v0.2.2 # # Fill in : the directory holding Kimi-K3. Use LOCAL NVMe — # 96 shards load in ~8 min from local disk and ~95 min from NFS, and the slow path # does not merely run late, it exceeds the ready timeout and restarts forever. # -# Deploy: kubectl apply -f examples/recipes/kimi-k3-optimized/mixed/deploy.yaml +# Deploy: THIS FILE CONTAINS PLACEHOLDERS — it cannot be applied directly. +# sed 's||/your/models|; s||nodeA|' .../mixed/deploy.yaml | kubectl apply -f - # -# MEASURED, this manifest, 8x MI355X TP8, 1M context: -# 8K in / 1K out, batch 1 step time 12.37 ms -# 1024 in / 128 out c=4 67.11 tok/s -# c=8 187.17 tok/s -# c=16 426.51 tok/s +# MEASURED, this manifest, 8x MI355X TP8, 1M context, on the image it pins: +# median TPOT 25.24 ms at c=4 rising to 48.81 ms at c=64 # -# This is the recipe to use above c=8. ../mixed-dspark/ is 1.9-2.2x faster at c<=8 -# but crashes at c>=16, and its best result (359.72) is below this one's c=16. +# Throughput figures were withdrawn from the README: an independent run of the +# same sweep got 165.99 tok/s cold and 270.4 warm against a published 241.69, so +# the method (32 requests, ~15 s, no warm-up) was not sound enough to publish +# from. TPOT reproduced within a few percent and is kept. apiVersion: infera.amd.com/v1alpha1 kind: InferaDeployment metadata: @@ -37,6 +37,13 @@ spec: server: componentType: server extraPodSpec: + # Both pods mount the weights as a hostPath, and both are scheduled + # independently — so without this they can land on different nodes. Where + # the fleet's model paths differ per node, the one that guesses wrong + # crashloops with + # ValueError: '/models/Kimi-K3': not a local path and HF resolution failed + # which reads as a bad model name, not as a bad node. Pin both. + nodeSelector: {kubernetes.io/hostname: } initContainers: - name: infera-overlay image: inferaimage/infera-overlay@sha256:6918eff34f201548a738dd592d2a1ece0627354d2e88f24a87cfa8f787a72a44 @@ -50,7 +57,7 @@ spec: mountPath: /overlay containers: - name: main - image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260801@sha256:a3584d74f256ca7c33b1ac9e5598897123d28e9faf9860e188a2e82a9b5530e3 + image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260802@sha256:5f3007aff1bc231eceb9f024e56ee80e44f9ca101a521aa50fe6bfa6c979d6b8 imagePullPolicy: IfNotPresent command: - /overlay/bin/infera-exec @@ -106,6 +113,13 @@ spec: role: mixed replicas: 1 extraPodSpec: + # Both pods mount the weights as a hostPath, and both are scheduled + # independently — so without this they can land on different nodes. Where + # the fleet's model paths differ per node, the one that guesses wrong + # crashloops with + # ValueError: '/models/Kimi-K3': not a local path and HF resolution failed + # which reads as a bad model name, not as a bad node. Pin both. + nodeSelector: {kubernetes.io/hostname: } initContainers: - name: infera-overlay image: inferaimage/infera-overlay@sha256:6918eff34f201548a738dd592d2a1ece0627354d2e88f24a87cfa8f787a72a44 @@ -119,7 +133,7 @@ spec: mountPath: /overlay containers: - name: main - image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260801@sha256:a3584d74f256ca7c33b1ac9e5598897123d28e9faf9860e188a2e82a9b5530e3 + image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260802@sha256:5f3007aff1bc231eceb9f024e56ee80e44f9ca101a521aa50fe6bfa6c979d6b8 imagePullPolicy: IfNotPresent command: - /overlay/bin/infera-exec @@ -161,7 +175,6 @@ spec: - http - --max-model-len - '1048576' - - --enable-prefix-caching env: - name: POD_NAME valueFrom: diff --git a/examples/recipes/kimi-k3-optimized/pd-dspark/deploy.yaml b/examples/recipes/kimi-k3-optimized/pd-dspark/deploy.yaml new file mode 100644 index 0000000..d6e29a0 --- /dev/null +++ b/examples/recipes/kimi-k3-optimized/pd-dspark/deploy.yaml @@ -0,0 +1,297 @@ +# Kimi-K3 (optimized build) — pd-dspark — Kubernetes recipe +# +# base image johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260802 +# overlay inferaimage/infera-overlay v0.2.2 +# engine infera.engine.vllm TP8 per role, 1P1D, DSpark on BOTH roles +# +# Deploy: THIS FILE CONTAINS PLACEHOLDERS — it cannot be applied directly. +# see README section 0 for the placeholders, section 4 for the command +# Docs: examples/recipes/kimi-k3-optimized/README.md +# +# --speculative-config IS ON BOTH ROLES, AND THAT IS NOT REDUNDANCY. The obvious +# reading is that speculation belongs only on decode — the prefiller never samples, +# so a draft there looks like dead weight. That configuration was tried and it does +# not work, in two separate ways: +# +# 1. vLLM merges the draft's layers into the same kv_caches dict as the target's, +# continuing the numbering. Kimi-K3 has 93 layers, DSpark adds 5, so a +# speculating decoder registers model.layers.0...97 and sends all 98 names to +# the prefiller, which has nothing past 92: +# KeyError: 'model.layers.93.self_attn' +# +# 2. Even with the layer lists forced to agree, speculation changes the decoder's +# block accounting — max_num_scheduled_tokens is recomputed to reserve slots +# for draft tokens — so the two sides disagree on how many blocks a request +# needs: +# pulling kv_caches ... failed: P num blocks less than D +# (mooncake_connector.py: "local blocks(N) < remote blocks(M)") +# +# The second one is why the prefiller must be speculation-aware even though it will +# never run the draft: the block count has to be computed the same way on both +# sides. Loading the draft on the prefiller is the price of that agreement, not an +# oversight. +# +# BOTH FAILURES HANG RATHER THAN ERROR. The request never completes and never +# returns an error; all pods stay Ready, health checks pass, restarts stay 0, and +# the decoder logs no inference. The first symptom is a client that waits until its +# own timeout — 45 minutes in the run that found this. Do not conclude a PD +# deployment works from `kubectl get pods`. +# +# THE DRAFT MUST BE ON BOTH NODES, since both roles now load it. +# +# PD REQUIRES the two nodes to sit on a MUTUALLY ROUTABLE RoCE fabric. The KV +# handoff is RDMA and there is no TCP fallback. Set /. +# +# EACH NODE NEEDS ITS OWN LOCAL COPY of both models, and the paths need not match — +# hence the separate placeholders. On this fleet the tempting common mount +# (/mnt/shared) is an NFS export of the other node's array, so using it on both +# sides turns one side's 8-minute load into ~95 minutes, which then exceeds the +# ready timeout and restarts forever. `df -hT ` on each node and take the +# local device, not the convenient common name. +# +# The tell that the handoff is real is on the DECODE side, and it is not throughput: +# `External prefix cache hit rate: ~100%` with `Avg prompt throughput` near zero +# means the decoder computed no prefill and took the KV off the wire. A correct +# answer proves nothing — a handoff that fails open just re-prefills locally and +# returns the same text, faster. +apiVersion: infera.amd.com/v1alpha1 +kind: InferaDeployment +metadata: + name: kimi-k3-opt-pd-dspark + namespace: infera +spec: + backendFramework: vllm + discoveryBackend: kubernetes + nats: + deploy: false + services: + server: + componentType: server + extraPodSpec: + nodeSelector: {kubernetes.io/hostname: } + initContainers: + - name: infera-overlay + image: inferaimage/infera-overlay@sha256:6918eff34f201548a738dd592d2a1ece0627354d2e88f24a87cfa8f787a72a44 + imagePullPolicy: IfNotPresent + command: ["sh","-c","cp -a /payload/. /overlay/"] + volumeMounts: + - {name: overlay, mountPath: /overlay} + containers: + - name: main + image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260802@sha256:5f3007aff1bc231eceb9f024e56ee80e44f9ca101a521aa50fe6bfa6c979d6b8 + imagePullPolicy: IfNotPresent + command: ["/overlay/bin/infera-exec","python3","-m","infera.server", + "--host","0.0.0.0","--port","8000", + "--router-tokenizer-path","/models/Kimi-K3", + "--request-transport","http","--kv-event-transport","zmq"] + env: + - {name: POD_NAME, valueFrom: {fieldRef: {fieldPath: metadata.name}}} + - {name: POD_NAMESPACE, valueFrom: {fieldRef: {fieldPath: metadata.namespace}}} + - {name: POD_IP, valueFrom: {fieldRef: {fieldPath: status.podIP}}} + resources: + requests: {cpu: '8', memory: 16Gi} + limits: {cpu: '8', memory: 16Gi} + volumeMounts: + - {name: overlay, mountPath: /overlay, readOnly: true} + - {name: model, mountPath: /models, readOnly: true} + volumes: + - {name: overlay, emptyDir: {}} + - name: model + hostPath: {path: , type: Directory} + + prefill: + componentType: worker + role: prefill + replicas: 1 + port: 30000 + extraPodSpec: + nodeSelector: {kubernetes.io/hostname: } + # hostNetwork is REQUIRED: the RDMA rails are host interfaces and the + # flannel pod network cannot reach them, so Mooncake has no path to the + # peer while the engine looks perfectly healthy. + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + initContainers: + - name: infera-overlay + image: inferaimage/infera-overlay@sha256:6918eff34f201548a738dd592d2a1ece0627354d2e88f24a87cfa8f787a72a44 + imagePullPolicy: IfNotPresent + command: ["sh","-c","cp -a /payload/. /overlay/"] + volumeMounts: + - {name: overlay, mountPath: /overlay} + containers: + - name: main + image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260802@sha256:5f3007aff1bc231eceb9f024e56ee80e44f9ca101a521aa50fe6bfa6c979d6b8 + imagePullPolicy: IfNotPresent + command: ["/overlay/bin/infera-exec", + "python3","-m","infera.engine.vllm","--host","0.0.0.0","--port","30000", + "--model","/models/Kimi-K3", + "--served-model-name","kimi-k3", + "--tensor-parallel-size","8", + # 0.88, not the 0.95 the stock kimi-k3 recipe uses: 0.88 is the + # value validated on THIS image at 1M context. Carrying a number + # over from a different base is how the mixed-dspark run OOMed. + "--gpu-memory-utilization","0.88", + "--trust-remote-code","--load-format","auto", + "--mm-encoder-tp-mode","data","--kv-cache-dtype","auto", + "--enable-prefix-caching", + "--max-num-seqs","64","--max-num-batched-tokens","4096", + "--max-model-len","1048576", + "--enable-auto-tool-choice","--tool-call-parser","kimi_k3", + "--reasoning-parser","kimi_k3", + # Mooncake's bootstrap server BINDS 0.0.0.0 (fine) but ADVERTISES + # parallel_config.data_parallel_master_ip. With the usual 0.0.0.0 + # the prefiller registers "http://0.0.0.0:8998" and the decoder + # cannot reach it: + # Failed to connect to bootstrap server http://0.0.0.0:8998 + # Failed to find remote engine_id ... + # while both workers look healthy and the request just hangs to a + # 480 s connector timeout. + "--data-parallel-address","$(POD_IP)", + "--speculative-config","{\"method\": \"dspark\", \"model\": \"/models/Kimi-K3-DSpark\", \"num_speculative_tokens\": 7, \"attention_backend\": \"ROCM_AITER_MLA\"}", + "--kv-transfer-config","{\"kv_connector\":\"MooncakeConnector\",\"kv_role\":\"kv_producer\"}", + "--kv-event-transport","zmq","--request-transport","http"] + env: + - {name: POD_NAME, valueFrom: {fieldRef: {fieldPath: metadata.name}}} + - {name: POD_NAMESPACE, valueFrom: {fieldRef: {fieldPath: metadata.namespace}}} + - {name: POD_IP, valueFrom: {fieldRef: {fieldPath: status.podIP}}} + # PD's KV transport is Mooncake, which lives in the overlay's native tree. + # Name it so a payload lacking it fails at startup instead of warning and + # quietly serving with no KV transfer at all. + - {name: INFERA_REQUIRE_NATIVE, value: mooncake} + - {name: INFERA_ENGINE_READY_TIMEOUT, value: '7200'} + - {name: HF_HUB_OFFLINE, value: '1'} + - {name: SAFETENSORS_FAST_GPU, value: '1'} + # --- the optimized-kernel cluster. Identical to ../mixed/. Without it the + # MoE asks aiter for a kernel that was never generated: + # ValueError: Invalid FlyDSL kernel name: flydsl_moe1_..._t16x64x256_... + - {name: VLLM_ROCM_USE_AITER, value: '1'} + - {name: VLLM_ROCM_USE_AITER_FP4BMM, value: '1'} + - {name: AITER_SITUV2_A8W4, value: '1'} + - {name: AITER_BF16_FP8_MOE_BOUND, value: '0'} + - {name: HIP_FORCE_DEV_KERNARG, value: '1'} + - {name: HSA_NO_SCRATCH_RECLAIM, value: '1'} + - {name: HSA_ENABLE_IPC_MODE_LEGACY, value: '1'} + - {name: VLLM_USE_BREAKABLE_CUDAGRAPH, value: '0'} + - {name: VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION, value: '1'} + # Must be 0. The pre-route dispatch tries BF16 first, so a 1 here shadows + # the FP8 cluster entirely — silently, at ~40% of throughput. + - {name: VLLM_ROCM_USE_KIMI_K3_PREROUTE_BF16, value: '0'} + - {name: VLLM_ROCM_USE_KIMI_K3_PREROUTE_FP8, value: '1'} + - {name: VLLM_ROCM_USE_KIMI_K3_LATENT_TAIL_FP8, value: '1'} + - {name: KIMI_K3_DUAL_PROJ_FP8_WEIGHT_CACHE_MODIFIER, value: '2'} + - {name: KIMI_K3_SHARED_DOWN_FP8_WEIGHT_CACHE_MODIFIER, value: '2'} + # RDMA needs the device nodes and pinned memory. Without these the engine + # starts, registers, and only fails when KV actually moves. + securityContext: + privileged: true + capabilities: {add: ["IPC_LOCK","SYS_PTRACE"]} + startupProbe: + httpGet: {path: /health, port: 30000} + periodSeconds: 10 + timeoutSeconds: 10 + failureThreshold: 300 # weight load is slow; do not use a readinessProbe here + resources: + requests: {cpu: '32', memory: 512Gi, amd.com/gpu: 8} + limits: {cpu: '32', memory: 512Gi, amd.com/gpu: 8} + volumeMounts: + - {name: overlay, mountPath: /overlay, readOnly: true} + - {name: model, mountPath: /models, readOnly: true} + # The vendor base's libionic must match the host ionic kernel ABI, or + # libibverbs rejects every device ("No RDMA devices found") and Mooncake + # falls back to HIP IPC — which cannot open a peer NODE's handle, so PD + # dies while a single-node smoke test passes. + - {name: host-libionic, mountPath: /host-libionic/libionic.so, readOnly: true} + - {name: ib, mountPath: /dev/infiniband} + - {name: dshm, mountPath: /dev/shm} + volumes: + - {name: overlay, emptyDir: {}} + - name: model + hostPath: {path: , type: Directory} + - name: host-libionic + hostPath: {path: /usr/lib/x86_64-linux-gnu/libionic.so.1, type: File} + - {name: ib, hostPath: {path: /dev/infiniband, type: Directory}} + - {name: dshm, emptyDir: {medium: Memory, sizeLimit: 64Gi}} + + decode: + componentType: worker + role: decode + replicas: 1 + port: 30000 + extraPodSpec: + nodeSelector: {kubernetes.io/hostname: } + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + initContainers: + - name: infera-overlay + image: inferaimage/infera-overlay@sha256:6918eff34f201548a738dd592d2a1ece0627354d2e88f24a87cfa8f787a72a44 + imagePullPolicy: IfNotPresent + command: ["sh","-c","cp -a /payload/. /overlay/"] + volumeMounts: + - {name: overlay, mountPath: /overlay} + containers: + - name: main + image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260802@sha256:5f3007aff1bc231eceb9f024e56ee80e44f9ca101a521aa50fe6bfa6c979d6b8 + imagePullPolicy: IfNotPresent + command: ["/overlay/bin/infera-exec", + "python3","-m","infera.engine.vllm","--host","0.0.0.0","--port","30000", + "--model","/models/Kimi-K3", + "--served-model-name","kimi-k3", + "--tensor-parallel-size","8", + "--gpu-memory-utilization","0.88", + "--trust-remote-code","--load-format","auto", + "--mm-encoder-tp-mode","data","--kv-cache-dtype","auto", + "--max-num-seqs","64","--max-num-batched-tokens","4096", + "--max-model-len","1048576", + "--enable-auto-tool-choice","--tool-call-parser","kimi_k3", + "--reasoning-parser","kimi_k3", + "--data-parallel-address","$(POD_IP)", + "--speculative-config","{\"method\": \"dspark\", \"model\": \"/models/Kimi-K3-DSpark\", \"num_speculative_tokens\": 7, \"attention_backend\": \"ROCM_AITER_MLA\"}", + "--kv-transfer-config","{\"kv_connector\":\"MooncakeConnector\",\"kv_role\":\"kv_consumer\"}", + "--kv-event-transport","zmq","--request-transport","http"] + env: + - {name: POD_NAME, valueFrom: {fieldRef: {fieldPath: metadata.name}}} + - {name: POD_NAMESPACE, valueFrom: {fieldRef: {fieldPath: metadata.namespace}}} + - {name: POD_IP, valueFrom: {fieldRef: {fieldPath: status.podIP}}} + - {name: INFERA_REQUIRE_NATIVE, value: mooncake} + - {name: INFERA_ENGINE_READY_TIMEOUT, value: '7200'} + - {name: HF_HUB_OFFLINE, value: '1'} + - {name: SAFETENSORS_FAST_GPU, value: '1'} + - {name: VLLM_ROCM_USE_AITER, value: '1'} + - {name: VLLM_ROCM_USE_AITER_FP4BMM, value: '1'} + - {name: AITER_SITUV2_A8W4, value: '1'} + - {name: AITER_BF16_FP8_MOE_BOUND, value: '0'} + - {name: HIP_FORCE_DEV_KERNARG, value: '1'} + - {name: HSA_NO_SCRATCH_RECLAIM, value: '1'} + - {name: HSA_ENABLE_IPC_MODE_LEGACY, value: '1'} + - {name: VLLM_USE_BREAKABLE_CUDAGRAPH, value: '0'} + - {name: VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION, value: '1'} + - {name: VLLM_ROCM_USE_KIMI_K3_PREROUTE_BF16, value: '0'} + - {name: VLLM_ROCM_USE_KIMI_K3_PREROUTE_FP8, value: '1'} + - {name: VLLM_ROCM_USE_KIMI_K3_LATENT_TAIL_FP8, value: '1'} + - {name: KIMI_K3_DUAL_PROJ_FP8_WEIGHT_CACHE_MODIFIER, value: '2'} + - {name: KIMI_K3_SHARED_DOWN_FP8_WEIGHT_CACHE_MODIFIER, value: '2'} + securityContext: + privileged: true + capabilities: {add: ["IPC_LOCK","SYS_PTRACE"]} + startupProbe: + httpGet: {path: /health, port: 30000} + periodSeconds: 10 + timeoutSeconds: 10 + failureThreshold: 300 + resources: + requests: {cpu: '32', memory: 512Gi, amd.com/gpu: 8} + limits: {cpu: '32', memory: 512Gi, amd.com/gpu: 8} + volumeMounts: + - {name: overlay, mountPath: /overlay, readOnly: true} + - {name: model, mountPath: /models, readOnly: true} + - {name: host-libionic, mountPath: /host-libionic/libionic.so, readOnly: true} + - {name: ib, mountPath: /dev/infiniband} + - {name: dshm, mountPath: /dev/shm} + volumes: + - {name: overlay, emptyDir: {}} + - name: model + hostPath: {path: , type: Directory} + - name: host-libionic + hostPath: {path: /usr/lib/x86_64-linux-gnu/libionic.so.1, type: File} + - {name: ib, hostPath: {path: /dev/infiniband, type: Directory}} + - {name: dshm, emptyDir: {medium: Memory, sizeLimit: 64Gi}} diff --git a/examples/recipes/kimi-k3-optimized/pd/deploy.yaml b/examples/recipes/kimi-k3-optimized/pd/deploy.yaml index a00b8a0..f435ec1 100644 --- a/examples/recipes/kimi-k3-optimized/pd/deploy.yaml +++ b/examples/recipes/kimi-k3-optimized/pd/deploy.yaml @@ -1,30 +1,30 @@ # Kimi-K3 (optimized build) — pd — Kubernetes recipe # -# base image johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260801 +# base image johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260802 # overlay inferaimage/infera-overlay v0.2.2 # engine infera.engine.vllm TP8 per role, 1P1D # -# Deploy: kubectl apply -f examples/recipes/kimi-k3-optimized/pd/deploy.yaml +# Deploy: THIS FILE CONTAINS PLACEHOLDERS — it cannot be applied directly. +# see README section 0 for the placeholders, section 4 for the command # Docs: examples/recipes/kimi-k3-optimized/README.md # -# NO SPECULATION HERE. DSpark is a decode-side draft and its interaction with the -# kv_consumer role is a separate unknown; ../mixed-dspark/ is the speculative -# recipe and it is mixed-only. Do not graft --speculative-config onto this file -# without validating it — at c>=16 it fails even without PD. +# NO SPECULATION HERE — that is ../pd-dspark/, which is validated. This file is +# the control: same image, same nodes, same everything, minus one argument. Keep +# it that way, so the pair stays a measurement of speculation rather than of two +# unrelated configs. # # PD REQUIRES the two nodes to sit on a MUTUALLY ROUTABLE RoCE fabric. The KV # handoff is RDMA and there is no TCP fallback. Set /. # # MEASURED, this manifest, 1P1D over 2x 8x MI355X = 16 GPUs, 1M context: -# ready 790 s both roles, 0 restarts -# 2139 in / 127 out c=4 130.15 tok/s P50 3.59 s P90 4.27 s -# c=8 229.61 tok/s P50 4.34 s P90 4.59 s -# c=16 386.28 tok/s P50 4.63 s P90 6.09 s -# -# DO NOT compare those to ../mixed/ as a speedup. This is 16 GPUs against its 8, -# and the prompt is 2139 tokens against its 1024 — two differences at once, so -# the ratio measures nothing. +# median TPOT 23.20 ms at c=4 rising to 38.12 ms at c=64 +# decode side held External prefix cache hit rate at 98.9-99.9% across all five +# concurrencies, with prompt throughput near zero and prefill the mirror image +# -- so the handoff was real rather than failing open. Verified independently. # +# Throughput figures were withdrawn from the README -- see ../mixed/deploy.yaml. +# PD never won on tokens per GPU at any concurrency measured; what it buys is +# headroom past one node's ceiling, and lower TPOT at high concurrency. # The tell that the handoff is real is on the DECODE side, and it is not # throughput: `External prefix cache hit rate: 100.0%` with `Avg prompt # throughput` near zero means the decoder computed no prefill and took the KV @@ -62,7 +62,7 @@ spec: - {name: overlay, mountPath: /overlay} containers: - name: main - image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260801@sha256:a3584d74f256ca7c33b1ac9e5598897123d28e9faf9860e188a2e82a9b5530e3 + image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260802@sha256:5f3007aff1bc231eceb9f024e56ee80e44f9ca101a521aa50fe6bfa6c979d6b8 imagePullPolicy: IfNotPresent command: ["/overlay/bin/infera-exec","python3","-m","infera.server", "--host","0.0.0.0","--port","8000", @@ -104,7 +104,7 @@ spec: - {name: overlay, mountPath: /overlay} containers: - name: main - image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260801@sha256:a3584d74f256ca7c33b1ac9e5598897123d28e9faf9860e188a2e82a9b5530e3 + image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260802@sha256:5f3007aff1bc231eceb9f024e56ee80e44f9ca101a521aa50fe6bfa6c979d6b8 imagePullPolicy: IfNotPresent command: ["/overlay/bin/infera-exec", "python3","-m","infera.engine.vllm","--host","0.0.0.0","--port","30000", @@ -213,7 +213,7 @@ spec: - {name: overlay, mountPath: /overlay} containers: - name: main - image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260801@sha256:a3584d74f256ca7c33b1ac9e5598897123d28e9faf9860e188a2e82a9b5530e3 + image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260802@sha256:5f3007aff1bc231eceb9f024e56ee80e44f9ca101a521aa50fe6bfa6c979d6b8 imagePullPolicy: IfNotPresent command: ["/overlay/bin/infera-exec", "python3","-m","infera.engine.vllm","--host","0.0.0.0","--port","30000", diff --git a/manual/recipes/index.md b/manual/recipes/index.md index 95b11fb..b987416 100644 --- a/manual/recipes/index.md +++ b/manual/recipes/index.md @@ -34,8 +34,9 @@ Multimodal, ~1.5 TB of weights, hybrid Mamba. vLLM · TP8 · MI355X -Same engine commit, optimized FP8 kernels. Optional DSpark speculation: 1.9–2.2× -below concurrency 8, and it crashes above it. Cross-node PD validated. +Same engine commit, optimized FP8 kernels. Optional DSpark speculation — 2.60× at +c=4, decaying to parity by c=32. Cross-node PD validated, with and without +speculation. ::: :::: @@ -53,10 +54,10 @@ Each recipe comes in the same four shapes, composing two independent choices: | `pd + kvd` | prefill and decode on separate nodes | plus kvd on each role | both of the above | One recipe carries an extra axis rather than a fifth combination: **Kimi-K3 -optimized** ships `mixed`, `mixed-dspark` and `pd`, because speculative decoding is -a property of that image's draft model, not a serving topology. `pd` is the same -combination as everywhere else; `mixed-dspark` is the extra axis, and it was not -combined with `pd`. +optimized** ships `mixed`, `mixed-dspark`, `pd` and `pd-dspark`, because +speculative decoding is a property of that image's draft model, not a serving +topology. It composes with `mixed` and `pd` independently, which is why there are +four rather than a fifth combination. ```{admonition} PD needs a routable RoCE fabric :class: warning diff --git a/manual/recipes/kimi-k3-optimized.md b/manual/recipes/kimi-k3-optimized.md index 46e57fc..b19c2cd 100644 --- a/manual/recipes/kimi-k3-optimized.md +++ b/manual/recipes/kimi-k3-optimized.md @@ -1,12 +1,43 @@ # Kimi-K3 (optimized build) -Kimi-K3 on 8× MI355X (TP8) using `johnqin2025/kimi-k3-dspark` — an image carrying an -FP8 pre-route / shared-expert kernel cluster, an FP8 latent-MoE tail, a fused MLA -output gate and a tri-projection dispatch. +Kimi-K3 on MI355X using `johnqin2025/kimi-k3-dspark` — an image carrying an FP8 +pre-route / shared-expert kernel cluster, an FP8 latent-MoE tail, a fused MLA +output gate and a tri-projection dispatch — plus an optional **DSpark** draft model +for speculative decoding. -A separate entry from [Kimi-K3](kimi-k3) rather than a combination on it, because it -is a different artifact. Same vLLM commit (`g5f76ae224`) as the stock image, so the -delta is kernels, not engine version. +A separate entry from [Kimi-K3](kimi-k3) rather than a combination on it, because +it is a different artifact. Same vLLM commit (`g5f76ae224`) as the stock image, so +the delta is kernels, not engine version. + +```{admonition} Every manifest here is a template — fill in the placeholders first +:class: warning +A bare `kubectl apply` on these files is **accepted by the API server** (the +placeholder passes CRD validation) and then fails minutes later at mount time. + +| Placeholder | In | How to find it | +|---|---|---| +| `` | Mixed, Mixed + DSpark | site-specific, **no default** — see below | +| `` | Mixed, Mixed + DSpark | a node with 8 free GPUs *that has the weights at that path* | +| ``, `` | PD, PD + DSpark | two nodes on a routable RoCE fabric | +| ``, `` | PD, PD + DSpark | per node — **they need not be equal** | + +**The model path is site-specific and a mixed fleet may not agree with itself.** On +the fleet these were validated on, the two GPU nodes use different paths for the +same weights. Candidates usually look identical (same shard count, same layout) +while only one is local storage, so confirm the one you pick by mounting **that +directory** in a throwaway pod on the node and reading `df -PT` — see the repo +README for the commands. Mounting a parent instead reports the root filesystem and +will call a network-mounted weights directory local. + +Leaving a placeholder in has two different outcomes, neither obvious. `` +creates a Pod that fails at mount time. `` creates **no Pod at all** — +`apply` prints `created`, `get pods` shows nothing, and the only error is in the +operator's log in the `infera-system` namespace. + +`` exists because both the server and the worker mount the weights by +hostPath and are scheduled independently — without it they can land on different +nodes. +``` ## 1. Choose the combination @@ -15,74 +46,125 @@ delta is kernels, not engine version. :::{tab-item} Mixed :sync: mixed -No speculative decoding. **The right choice above concurrency 8**, where it is also -the faster of the two. +8 GPUs, no speculation. **The default** — best tokens per GPU at every concurrency +measured. ```bash -kubectl apply -f examples/recipes/kimi-k3-optimized/mixed/deploy.yaml +sed -e 's||/local/nvme/models|' -e 's||nodeA|' \ + examples/recipes/kimi-k3-optimized/mixed/deploy.yaml | kubectl apply -f - ``` -| 1024 in / 128 out | tok/s | +| 1024 in / 128 out | TPOT | |---:|---:| -| c=4 | 67.11 | -| c=8 | 187.17 | -| c=16 | **426.51** | +| c=4 | 25.24 ms | +| c=8 | 27.85 ms | +| c=16 | 29.23 ms | +| c=32 | 37.09 ms | +| c=64 | 48.81 ms | ::: :::{tab-item} Mixed + DSpark :sync: mixed-dspark -Speculative decoding with a block-diffusion draft that produces 7 tokens per -parallel pass. Worth **1.9–2.2×** at concurrency ≤ 8. +8 GPUs, speculative decoding with a block-diffusion draft that produces 7 tokens +per parallel pass. **Worth it up to c=16.** ```bash -kubectl apply -f examples/recipes/kimi-k3-optimized/mixed-dspark/deploy.yaml +sed -e 's||/local/nvme/models|' -e 's||nodeA|' \ + examples/recipes/kimi-k3-optimized/mixed-dspark/deploy.yaml | kubectl apply -f - ``` -| 1024 in / 128 out | tok/s | vs mixed | -|---:|---:|---:| -| c=4 | 146.96 | 2.19× | -| c=8 | 359.72 | 1.92× | -| c≥16 | crashes | — | +| c=32 | 685.25 | 0.99× | +| c=64 | 879.84 | 0.90× | -```{admonition} Above concurrency 8 this is an outage, not a slowdown -:class: warning -At `c>=16` the engine dies with +Throughput figures for this page have been withdrawn — they did not reproduce independently (see the repo README). TPOT did. - AssertionError: AiterMLA flattened verify requires a uniform decode query len +```{admonition} The old concurrency ceiling is gone +:class: note +On the `…-20260801` image this died at `c>=16` with `AssertionError: AiterMLA +flattened verify requires a uniform decode query len`. The image pinned here fixes +it — c=16/32/64 all complete, 0 restarts, assertion count 0 — and speculation is +genuinely still on (7 draft tokens, CUDA-graph captured, `running the draft +eagerly` count 0), so it is a fix rather than speculation being disabled. -"verify" is the tell — the speculative path only. The identical config without -`--speculative-config` serves `c=16` at 426.51 tok/s, which is also *faster* than -this manifest's best result (359.72 at `c=8`). +If you pin the older digest, the old ceiling still applies to you. ``` ::: :::{tab-item} PD :sync: pd -Prefill and decode on separate nodes, KV handed over RDMA. **16 GPUs**, not 8. +16 GPUs. Prefill and decode on separate nodes, KV handed over RDMA. ```bash sed -e 's||nodeA|' -e 's||nodeB|' \ - -e 's||/local/nvme/models|' \ - -e 's||/local/nvme/models|' \ + -e 's||/mnt/local-nvme/models|' \ + -e 's||/mnt/array/models|' \ examples/recipes/kimi-k3-optimized/pd/deploy.yaml | kubectl apply -f - ``` -| 2139 in / 127 out | tok/s | P50 | P90 | -|---:|---:|---:|---:| -| c=4 | 130.15 | 3.59 s | 4.27 s | -| c=8 | 229.61 | 4.34 s | 4.59 s | -| c=16 | 386.28 | 4.63 s | 6.09 s | +| 1024 in / 128 out | TPOT | +|---:|---:| +| c=4 | 23.20 ms | +| c=8 | 25.81 ms | +| c=16 | 27.28 ms | +| c=32 | 31.25 ms | +| c=64 | 38.12 ms | +| c=64 | **1217.32** | **1.25×** | **38.12 ms** | + +```{admonition} Twice the hardware, and it never wins per GPU +:class: warning +What PD buys is headroom beyond one node's ceiling, and lower TPOT at high +concurrency. It is not a tokens-per-GPU improvement at any concurrency measured. +``` +::: + +:::{tab-item} PD + DSpark +:sync: pd-dspark + +16 GPUs. PD's shape at high concurrency, plus speculation. + +```bash +sed -e 's||nodeA|' -e 's||nodeB|' \ + -e 's||/mnt/local-nvme/models|' \ + -e 's||/mnt/array/models|' \ + examples/recipes/kimi-k3-optimized/pd-dspark/deploy.yaml | kubectl apply -f - +``` -```{admonition} Do not read these against the Mixed tab +```{admonition} `--speculative-config` goes on BOTH roles, and that is not redundancy :class: warning -This is 16 GPUs against Mixed's 8, measured at a 2139-token prompt against its -1024. Two differences at once — the ratio between the tabs measures nothing. +The prefiller never samples, so a draft there looks like dead weight. That +configuration was tried and it fails two independent ways: + +**Layer lists disagree.** vLLM continues the target's layer numbering into the +draft. Kimi-K3 has 93 layers, DSpark adds 5, so a speculating decoder registers +`model.layers.0`…`97` and sends all 98 names to a prefiller that has nothing past +92 — `KeyError: 'model.layers.93.self_attn'`. + +**Block counts disagree.** Fixing the layer lists lands straight on +`pulling kv_caches ... failed: P num blocks less than D`. Speculation recomputes +`max_num_scheduled_tokens` to reserve draft slots, so the decoder's block +accounting differs from a prefiller that does not know speculation is happening. +No layer filtering fixes that — both sides must compute the count the same way. + +Loading the draft on the prefiller is the **price of that agreement**, not an +oversight. It is never run there, and both nodes need the draft on disk. + +Both failures **hang rather than error**: all pods Ready, health checks green, +restarts 0, no inference logged, and the client waits until its own timeout. ``` -No speculation here: DSpark is decode-side and its behaviour in the `kv_consumer` -role is a separate unknown. Do not graft `--speculative-config` onto this manifest. +| 1024 in / 128 out | TPOT | +|---:|---:| +| c=4 | **7.81 ms** | +| c=8 | **11.50 ms** | +| c=16 | **12.37 ms** | +| c=32 | **17.80 ms** | +| c=64 | **17.44 ms** | + +This is the lowest TPOT of the four combinations — the decode role is not +competing with prefill for the same GPUs. + ::: :::: @@ -91,7 +173,14 @@ role is a separate unknown. Do not graft `--speculative-config` onto this manife ```bash kubectl get nodes -o custom-columns=NODE:.metadata.name,GPU:.status.allocatable.'amd\.com/gpu' -helm install infera-operator deploy/operator/helm/infera-operator -n infera-system --create-namespace + +# every manifest hardcodes `namespace: infera`, and nothing else creates it +kubectl create namespace infera --dry-run=client -o yaml | kubectl apply -f - + +# on k3s, helm needs KUBECONFIG spelled out — kubectl finds it implicitly, helm does not +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +helm upgrade --install infera-operator deploy/operator/helm/infera-operator \ + -n infera-system --create-namespace hf download moonshotai/Kimi-K3 --local-dir /Kimi-K3 hf download Inferact/Kimi-K3-DSpark --local-dir /Kimi-K3-DSpark # DSpark only @@ -101,80 +190,76 @@ hf download Inferact/Kimi-K3-DSpark --local-dir /Kimi-K3-DSpark # D disk and ~95 min from NFS — and the slow path does not merely run late, it exceeds the ready timeout, so the worker restarts mid-load and never finishes. -First start is 10–14 min: the image rebuilds its AITER JIT modules in-container on +For the PD combinations each node needs its **own** copy, and the paths need not +match. On the fleet this was validated on, the tempting common mount (`/mnt/shared`) +was an NFS export of the *other* node's array; `df -hT ` on each node and take +the local device, not the convenient common name. + +First start is 12–14 min: the image rebuilds its AITER JIT modules in-container on top of weight load and CUDA-graph capture. ## 3. Smoke test ```bash -kubectl -n infera port-forward svc/kimi-k3-opt--server 8000:8000 & -curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \ +kubectl -n infera port-forward svc/-server 18000:8000 & PF=$! +sleep 3; kill -0 $PF 2>/dev/null || { echo "port-forward failed — try another local port"; exit 1; } + +# --max-time is not optional: this system's failure mode is a HANG, not an error. +curl -s --max-time 300 localhost:18000/v1/chat/completions -H 'Content-Type: application/json' \ -d '{"model":"kimi-k3","messages":[{"role":"user","content":"What is the capital of France?"}], - "max_tokens":200}' | jq -r '.choices[0].message.content' + "max_tokens":1024}' | jq -r '.choices[0].message.content' ``` -Keep `max_tokens` generous — this model spends 70+ tokens on that sentence, and a -tight cap truncates it into something that reads like a broken deployment. +`max_tokens: 1024`, not 200: the model emits a variable-length reasoning preamble, +and at 200 roughly one request in four returns `finish_reason: "length"` having +never reached the answer — which reads like a broken deployment on a healthy one. + +Keep `max_tokens` generous — this model spends 70+ tokens on that sentence. -On the DSpark manifest, confirm speculation engaged rather than inferring it from +On the DSpark manifests, confirm speculation engaged rather than inferring it from throughput later: ```bash -kubectl -n infera logs -c main | grep -c 'running the draft eagerly' # must be 0 -kubectl -n infera logs -c main | grep -oE 'Mean acceptance length: [0-9.]+' | tail -1 +kubectl -n infera logs -c main | grep -c 'running the draft eagerly' # must be 0 ``` +On the PD manifests, a correct answer proves nothing on its own — a handoff that +fails open just re-prefills locally and returns the same text, faster. Check the +decode side: `External prefix cache hit rate` near 100% with `Avg prompt +throughput` near zero. + ## 4. Settings that are not optional | Setting | Why | |---|---| -| the `KIMI_K3_*` / `VLLM_ROCM_*` env block | selects the optimized kernels. Without them the MoE asks aiter for a kernel that was never generated — `Invalid FlyDSL kernel name: flydsl_moe1_..._t16x64x256_...` — because there is no Kimi-K3 `tuned_fmoe.csv`, while dsv3/dsv4/glm5 all have one | -| `VLLM_ROCM_USE_KIMI_K3_PREROUTE_BF16=0` | must be `0`; a `1` shadows the FP8 cluster silently, at ~40% of throughput | +| the `KIMI_K3_*` / `VLLM_ROCM_*` env block | selects the optimized kernels. Without them the MoE asks aiter for a kernel that was never generated — `Invalid FlyDSL kernel name: flydsl_moe1_...` — because there is no Kimi-K3 `tuned_fmoe.csv` | +| `VLLM_ROCM_USE_KIMI_K3_PREROUTE_BF16=0` | must be `0`. A `1` shadows the FP8 cluster: measured against the same manifest with `0`, each as the first sweep after its own deployment: roughly half the throughput at c=8, about three quarters at c=16. Silent otherwise — starts fine, no warning, and TPOT barely moves, so latency monitoring misses it | | `attention_backend: ROCM_AITER_MLA` | the ROCm counterpart of the upstream quick-start's `FLASHINFER_MLA`. Dropping the key instead of translating it is not the fix | | `--gpu-memory-utilization 0.88` | the draft's weights land after the KV budget is computed; `0.95` dies with 998 MB free trying to allocate 2.32 GiB | | `INFERA_ENGINE_READY_TIMEOUT=7200` | the 1800 s default is impossible on slow storage, and the worker then restarts mid-load forever — which reads as a crash loop, not as slow storage | +| PD: each node needs its **own local** copy | the `model` volume is a `hostPath`, and the paths need not match between nodes. The fleet's obvious shared mount may be an NFS export of the peer's array — one side then loads for ~95 min and restarts forever | +| PD: never point a misbehaving client at it | the engine validates **after** prefill, so a rejected request has already had its KV computed and queued. 84 requests rejected for one bad field left 424 aborted Mooncake transfers and stalled valid traffic for ~20 minutes with `MooncakeXferMetadata transfer failed: Resource temporarily unavailable`, which reads exactly like a broken fabric | -## 5. Validation status +`ibv_devices` is **not installed** in this image; reading its "not found" as "no +RDMA devices" produced two false TCP-fallback diagnoses here. Ask +`ibv_get_device_list` instead — see the repo README for the one-liner. -All three manifests were **run end-to-end as written** on k3s (MI355X, 1M context, -BF16 KV), through the infera router — `mixed` and `mixed-dspark` on 8 GPUs, `pd` as -1P1D across two nodes. - -For `pd`, a correct answer proves nothing on its own: if the KV handoff fails open, -the decoder re-prefills locally and returns the same text. The tell is on the decode -side — `External prefix cache hit rate: 100.0%` with `Avg prompt throughput` near -zero, against the prefill side's 2010.5 tok/s prompt and ~0 generation. That held -for 131 consecutive requests with zero failed transfers. - -```{admonition} Do not point a misbehaving client at a PD deployment -:class: warning -The engine validates **after** prefill, so a request it rejects has already had its -KV computed and queued for transfer. 84 requests rejected for a single bad field -left 424 aborted Mooncake transfers — 53 requests × 8 TP ranks — and stalled valid -traffic behind the backlog for ~20 minutes, with - - MooncakeXferMetadata transfer failed: Resource temporarily unavailable - -which reads exactly like a broken fabric. It was a broken client. -``` +## 5. Validation status -Against the upstream quick-start for this image, three numbers reproduce and three -claims do not: +Every combination was run end-to-end on the image it pins, with placeholders +substituted (the files are templates), and every +number on this page comes from the same image — so the ratios are attributable to +the manifests rather than to a base-image difference. That distinction is not +academic: between the two images, `mixed` alone moved −15% at c=4 and +29% at c=8. -| | upstream | measured here | -|---|---|---| -| baseline step time, 8K/1K | 12.39 ms | 12.37 ms | -| DSpark step time, 8K/1K | 29.50 ms | 29.11 ms | -| acceptance length | 4.00 | 3.85 | -| concurrency assertion fires at | c=48 | **c=16** | -| c=16 throughput | 315.7 tok/s | **crashes** | -| c=32 throughput | 454.3 tok/s | **crashes** | - -The upstream document states its concurrency table was measured on a sibling build -(`kimi-k3-dspark:wvsplitk-fix`) and not re-run on this image. These measurements are -that re-run: the concurrency fixes are not in this image. - -PD and kvd combinations are not built for this image. +| What | Status | +|---|---| +| `mixed` | validated, c=4…64 swept | +| `mixed-dspark` | validated, c=4…64 swept, 0 restarts, assertion count 0 | +| `pd` | validated cross-node, c=4…64 swept, extcache 99.9% throughout | +| `pd-dspark` | validated cross-node, c=4…64 swept, extcache 99.9%; needs `--speculative-config` on both roles | +| kvd combinations | not built for this image | +| fp8 KV cache | not measured | ## Source