Skip to content

docs(recipes): re-measure kimi-k3-optimized on the 20260802 image, add pd-dspark #74

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jiejingzhangamd merged 10 commits into from
Aug 3, 2026
Merged

docs(recipes): re-measure kimi-k3-optimized on the 20260802 image, add pd-dspark #74

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
e2a784a
docs(recipes): re-measure kimi-k3-optimized on the 20260802 image
jiejingzhangamd Aug 2, 2026
950c663
feat(recipes): pd-dspark for kimi-k3-optimized
jiejingzhangamd Aug 2, 2026
d8f799f
docs(recipes): fix the base-image line in three manifest headers
jiejingzhangamd Aug 2, 2026
9121b78
fix(recipes): make kimi-k3-optimized followable — 16 findings from an…
jiejingzhangamd Aug 2, 2026
e446b78
docs(recipes): put the placeholders, and the path warning, up front
jiejingzhangamd Aug 2, 2026
52cd356
feat(recipes): pre-flight check for kimi-k3-optimized
jiejingzhangamd Aug 2, 2026
4bca96b
fix(recipes): second independent run — both combinations now serve, 9…
jiejingzhangamd Aug 2, 2026
5ccb6ba
docs(recipes): measure the PREROUTE_BF16 penalty instead of asserting it
jiejingzhangamd Aug 2, 2026
17c5e42
fix(recipes): third independent run — preflight gave two false PASSes
jiejingzhangamd Aug 2, 2026
ea0fa69
fix(recipes): withdraw preflight.sh and the throughput tables
jiejingzhangamd Aug 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
493 changes: 377 additions & 116 deletions examples/recipes/kimi-k3-optimized/README.md
View file
Open in desktop

Large diffs are not rendered by default.

45 changes: 29 additions & 16 deletions examples/recipes/kimi-k3-optimized/mixed-dspark/deploy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,28 +6,28 @@
# output gate and a tri-projection dispatch. Same vLLM commit as the stock
# kimi-k3 image (g5f76ae224), so the delta is kernels, not engine version.
#
# base image johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260801
# base image johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260802
# overlay inferaimage/infera-overlay v0.2.2
#
# Fill in <MODEL_DIR>: the directory holding Kimi-K3 and Kimi-K3-DSpark. Use LOCAL NVMe —
# 96 shards load in ~8 min from local disk and ~95 min from NFS, and the slow path
# does not merely run late, it exceeds the ready timeout and restarts forever.
#
# Deploy: kubectl apply -f examples/recipes/kimi-k3-optimized/mixed-dspark/deploy.yaml
# Deploy: THIS FILE CONTAINS PLACEHOLDERS — it cannot be applied directly.
# sed 's|<MODEL_DIR>|/your/models|; s|<NODE>|nodeA|' .../mixed-dspark/deploy.yaml | kubectl apply -f -
#
# MEASURED, this manifest, 8x MI355X TP8, 1M context (see README for the full set):
# 8K in / 1K out, batch 1 step time 29.11 ms acceptance 3.85 of 7 drafted
# 1024 in / 128 out c=4 146.96 tok/s (2.19x the non-speculative recipe)
# c=8 359.72 tok/s (1.92x)
# c>=16 CRASHES — see the concurrency note below.
# MEASURED, this manifest, 8x MI355X TP8, 1M context, on the image it pins:
# speculation engaged: num_spec_tokens=7, CUDA-graph captured,
# "running the draft eagerly" count 0
#
# CONCURRENCY CEILING IS 8 WITH SPECULATION ON. At c>=16 the engine dies with
# THE OLD CONCURRENCY CEILING IS GONE. On the 20260801 image this manifest died at
# c>=16 with
# AssertionError: AiterMLA flattened verify requires a uniform decode query len
# The word "verify" is the tell: this is the speculative path only. The same
# config without --speculative-config serves c=16 fine at 426.51 tok/s — which is
# also *faster* than this manifest's best (359.72 at c=8). So speculation is a
# low-concurrency optimisation here, and above c=8 it is not a tradeoff, it is an
# outage. Use ../mixed/ for concurrent serving.
# The image this file pins fixes it: c=16/32/64 all complete, 0 restarts, and the
# assertion appears zero times. That is binary, not a measurement. If you pin the
# older digest, the old ceiling still applies to you.
#
# Throughput figures were withdrawn from the README -- see ../mixed/deploy.yaml.
apiVersion: infera.amd.com/v1alpha1
kind: InferaDeployment
metadata:
Expand All @@ -42,6 +42,13 @@ spec:
server:
componentType: server
extraPodSpec:
# Both pods mount the weights as a hostPath, and both are scheduled
# independently — so without this they can land on different nodes. Where
# the fleet's model paths differ per node, the one that guesses wrong
# crashloops with
# ValueError: '/models/Kimi-K3': not a local path and HF resolution failed
# which reads as a bad model name, not as a bad node. Pin both.
nodeSelector: {kubernetes.io/hostname: <NODE>}
initContainers:
- name: infera-overlay
image: inferaimage/infera-overlay@sha256:6918eff34f201548a738dd592d2a1ece0627354d2e88f24a87cfa8f787a72a44
Expand All @@ -55,7 +62,7 @@ spec:
mountPath: /overlay
containers:
- name: main
image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260801@sha256:a3584d74f256ca7c33b1ac9e5598897123d28e9faf9860e188a2e82a9b5530e3
image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260802@sha256:5f3007aff1bc231eceb9f024e56ee80e44f9ca101a521aa50fe6bfa6c979d6b8
imagePullPolicy: IfNotPresent
command:
- /overlay/bin/infera-exec
Expand Down Expand Up @@ -111,6 +118,13 @@ spec:
role: mixed
replicas: 1
extraPodSpec:
# Both pods mount the weights as a hostPath, and both are scheduled
# independently — so without this they can land on different nodes. Where
# the fleet's model paths differ per node, the one that guesses wrong
# crashloops with
# ValueError: '/models/Kimi-K3': not a local path and HF resolution failed
# which reads as a bad model name, not as a bad node. Pin both.
nodeSelector: {kubernetes.io/hostname: <NODE>}
initContainers:
- name: infera-overlay
image: inferaimage/infera-overlay@sha256:6918eff34f201548a738dd592d2a1ece0627354d2e88f24a87cfa8f787a72a44
Expand All @@ -124,7 +138,7 @@ spec:
mountPath: /overlay
containers:
- name: main
image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260801@sha256:a3584d74f256ca7c33b1ac9e5598897123d28e9faf9860e188a2e82a9b5530e3
image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260802@sha256:5f3007aff1bc231eceb9f024e56ee80e44f9ca101a521aa50fe6bfa6c979d6b8
imagePullPolicy: IfNotPresent
command:
- /overlay/bin/infera-exec
Expand Down Expand Up @@ -166,7 +180,6 @@ spec:
- http
- --max-model-len
- '1048576'
- --enable-prefix-caching
- --speculative-config
- '{"method": "dspark", "model": "/models/Kimi-K3-DSpark", "num_speculative_tokens": 7, "attention_backend": "ROCM_AITER_MLA"}'
env:
Expand Down
37 changes: 25 additions & 12 deletions examples/recipes/kimi-k3-optimized/mixed/deploy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,23 @@
# output gate and a tri-projection dispatch. Same vLLM commit as the stock
# kimi-k3 image (g5f76ae224), so the delta is kernels, not engine version.
#
# base image johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260801
# base image johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260802
# overlay inferaimage/infera-overlay v0.2.2
#
# Fill in <MODEL_DIR>: the directory holding Kimi-K3. Use LOCAL NVMe —
# 96 shards load in ~8 min from local disk and ~95 min from NFS, and the slow path
# does not merely run late, it exceeds the ready timeout and restarts forever.
#
# Deploy: kubectl apply -f examples/recipes/kimi-k3-optimized/mixed/deploy.yaml
# Deploy: THIS FILE CONTAINS PLACEHOLDERS — it cannot be applied directly.
# sed 's|<MODEL_DIR>|/your/models|; s|<NODE>|nodeA|' .../mixed/deploy.yaml | kubectl apply -f -
#
# MEASURED, this manifest, 8x MI355X TP8, 1M context:
# 8K in / 1K out, batch 1 step time 12.37 ms
# 1024 in / 128 out c=4 67.11 tok/s
# c=8 187.17 tok/s
# c=16 426.51 tok/s
# MEASURED, this manifest, 8x MI355X TP8, 1M context, on the image it pins:
# median TPOT 25.24 ms at c=4 rising to 48.81 ms at c=64
#
# This is the recipe to use above c=8. ../mixed-dspark/ is 1.9-2.2x faster at c<=8
# but crashes at c>=16, and its best result (359.72) is below this one's c=16.
# Throughput figures were withdrawn from the README: an independent run of the
# same sweep got 165.99 tok/s cold and 270.4 warm against a published 241.69, so
# the method (32 requests, ~15 s, no warm-up) was not sound enough to publish
# from. TPOT reproduced within a few percent and is kept.
apiVersion: infera.amd.com/v1alpha1
kind: InferaDeployment
metadata:
Expand All @@ -37,6 +37,13 @@ spec:
server:
componentType: server
extraPodSpec:
# Both pods mount the weights as a hostPath, and both are scheduled
# independently — so without this they can land on different nodes. Where
# the fleet's model paths differ per node, the one that guesses wrong
# crashloops with
# ValueError: '/models/Kimi-K3': not a local path and HF resolution failed
# which reads as a bad model name, not as a bad node. Pin both.
nodeSelector: {kubernetes.io/hostname: <NODE>}
initContainers:
- name: infera-overlay
image: inferaimage/infera-overlay@sha256:6918eff34f201548a738dd592d2a1ece0627354d2e88f24a87cfa8f787a72a44
Expand All @@ -50,7 +57,7 @@ spec:
mountPath: /overlay
containers:
- name: main
image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260801@sha256:a3584d74f256ca7c33b1ac9e5598897123d28e9faf9860e188a2e82a9b5530e3
image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260802@sha256:5f3007aff1bc231eceb9f024e56ee80e44f9ca101a521aa50fe6bfa6c979d6b8
imagePullPolicy: IfNotPresent
command:
- /overlay/bin/infera-exec
Expand Down Expand Up @@ -106,6 +113,13 @@ spec:
role: mixed
replicas: 1
extraPodSpec:
# Both pods mount the weights as a hostPath, and both are scheduled
# independently — so without this they can land on different nodes. Where
# the fleet's model paths differ per node, the one that guesses wrong
# crashloops with
# ValueError: '/models/Kimi-K3': not a local path and HF resolution failed
# which reads as a bad model name, not as a bad node. Pin both.
nodeSelector: {kubernetes.io/hostname: <NODE>}
initContainers:
- name: infera-overlay
image: inferaimage/infera-overlay@sha256:6918eff34f201548a738dd592d2a1ece0627354d2e88f24a87cfa8f787a72a44
Expand All @@ -119,7 +133,7 @@ spec:
mountPath: /overlay
containers:
- name: main
image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260801@sha256:a3584d74f256ca7c33b1ac9e5598897123d28e9faf9860e188a2e82a9b5530e3
image: johnqin2025/kimi-k3-dspark:1.1.0-mi355x-rocm7.2.3-20260802@sha256:5f3007aff1bc231eceb9f024e56ee80e44f9ca101a521aa50fe6bfa6c979d6b8
imagePullPolicy: IfNotPresent
command:
- /overlay/bin/infera-exec
Expand Down Expand Up @@ -161,7 +175,6 @@ spec:
- http
- --max-model-len
- '1048576'
- --enable-prefix-caching
env:
- name: POD_NAME
valueFrom:
Expand Down
Loading
Loading