docs(recipes): re-measure kimi-k3-optimized on the 20260802 image, add pd-dspark - #74
Open
jiejingzhangamd wants to merge 10 commits into
Open
docs(recipes): re-measure kimi-k3-optimized on the 20260802 image, add pd-dspark#74jiejingzhangamd wants to merge 10 commits into
jiejingzhangamd wants to merge 10 commits into
Conversation
jiejingzhangamd
requested review from
JohnQinAMD,
limou102 and
xiaobochen-amd
as code owners
August 2, 2026 20:30
The image fixes the DSpark concurrency crash. On 20260801 every speculative combination died at c>=16 with AssertionError: AiterMLA flattened verify requires a uniform decode query len Here c=16/32/64 all complete, 0 restarts, and the assertion appears zero times. Speculation is genuinely still on -- 7 draft tokens, CUDA-graph captured, "running the draft eagerly" count 0 -- so this is a fix rather than speculation being quietly disabled, which is the other way that symptom goes away. All five concurrencies re-swept for all three combinations, on the same image, with a byte-identical `vllm bench serve` invocation (1024 in / 128 out, --random-range-ratio 0, --ignore-eos, --seed 0, num-prompts = 4x concurrency): c mixed mixed-dspark pd (16 GPUs) 4 56.78 147.90 50.38 8 241.69 328.58 218.47 16 427.86 471.74 428.98 32 689.22 685.25 747.79 64 972.37 879.84 1217.32 Two conclusions on the previous revision of this page were wrong, and both are corrected here rather than merely updated. "DSpark is worth 1.9-2.2x below c=8, and an outage above it." The ceiling is gone, and the shape is a monotonic decay rather than a cliff: 2.60x / 1.36x / 1.10x / 0.99x / 0.90x, crossing over between c=16 and c=32. The old multiplier was also inflated: it compared a DSpark number against a `mixed` baseline taken on the OLDER image, and `mixed` itself moved -15% at c=4 and +29% at c=8 between the two -- in opposite directions, so no single correction factor exists. Cross-image ratios of this kind are the easy mistake to miss in review, because both numbers are individually true. "Use mixed above c=8." The crossover is between c=16 and c=32, not at 8. PD is not presented as a speedup. It uses twice the hardware and never wins per GPU -- 1.25x throughput on 2x the GPUs is 0.63x per GPU at its best point. What it does buy at c=64 is 1.25x throughput AND 22% lower TPOT simultaneously, which is the part worth having, since those two normally trade against each other. Below c=32 it loses on both counts. The decode side held External prefix cache hit rate at 99.9% across all five concurrencies, so the handoff never silently failed open -- checked per concurrency rather than once at the start, because a handoff that fails open under load makes throughput look BETTER. Manifest headers carry the numbers for their own combination and a note not to carry them across an image bump. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
The fourth combination: PD's shape at high concurrency, plus speculation.
--speculative-config goes on BOTH roles, and that is the whole finding.
The obvious configuration is decode-only -- the prefiller never samples,
so a draft there looks like dead weight. It fails two independent ways:
1. The layer lists disagree. vLLM continues the target's layer numbering
into the draft, so a speculating decoder registers model.layers.0...97
(93 target + 5 draft) and sends all 98 names to a prefiller that has
nothing past 92:
KeyError: 'model.layers.93.self_attn'
93 is exactly one past the target's last layer. The error travels back
over ZMQ and is logged on the DECODE side, pointing at the wrong host.
2. The block counts disagree. Keeping draft layers out of the
registration -- semantically right, since the draft's KV is
decode-local and the prefiller could never supply it -- clears the
KeyError and lands directly on
pulling kv_caches ... failed: P num blocks less than D
mooncake_connector.py compares per-request block counts
("local blocks(N) < remote blocks(M)"). Speculation recomputes
max_num_scheduled_tokens to reserve draft slots, so the decoder's
accounting differs from a prefiller that does not know speculation is
happening. No layer filtering fixes that: both sides have to compute
the count the same way.
So the prefiller must be speculation-aware even though it never runs the
draft. Loading it there is the price of that agreement, not an oversight.
Both failures HANG rather than error: all pods Ready, health checks
green, restarts 0, no inference logged on the decoder, and the client
waits until its own timeout -- 45 minutes in the run that found this.
"kubectl get pods" is not evidence that a PD deployment works, which is
why the smoke test in the README reads the decode-side counters instead.
I went down the fix-the-connector path first and it is worth recording
why it was abandoned rather than merged: a patch that skipped draft
layers at registration did clear failure 1 (KeyError count 16 -> 0, skip
confirmed on all 8 ranks) and I nearly shipped it on that evidence. It
does nothing for failure 2. It also referenced overlay v0.2.2, which does
not contain it -- so anyone following the recipe would have hit the very
hang the page warned about, while my own validation missed the gap
because I mounted the patch via ConfigMap and bypassed the overlay.
Measured, 1P1D over 16 GPUs, same image and identical sweep as the rest:
c pd pd-dspark ratio pd TPOT pd-dspark TPOT
4 50.38 135.34 2.69x 23.20ms 7.81ms
8 218.47 445.72 2.04x 25.81ms 11.50ms
16 428.98 718.64 1.68x 27.28ms 12.37ms
32 747.79 928.50 1.24x 31.25ms 17.80ms
64 1217.32 1079.32 0.89x 38.12ms 17.44ms
Speculation helps PD far more than it helps mixed (1.68x vs 1.10x at
c=16) because the decode role is not competing with prefill for the same
GPUs. It crosses over at c=64 on throughput, where TPOT is still less
than half.
Both PD combinations held External prefix cache hit rate at 99.9% across
all five concurrencies, checked per concurrency rather than once, and
both failure counters stayed at 0 for the whole sweep.
Also adds a per-GPU table. Absolute throughput favours the 16-GPU options
at every concurrency; tokens-per-GPU never does. The PD pair buys
headroom (1217 vs mixed's 972 ceiling) and latency (17 ms vs 49 ms), not
efficiency, and the page says to pick on those rather than on the tok/s
column.
Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
jiejingzhangamd
force-pushed
the
docs/kimi-k3-optimized-0802
branch
from
August 2, 2026 21:03
ace257b to
950c663
Compare
The image: fields were bumped to 20260802 but the header summary still said 20260801, so each file contradicted itself. A stale image name in a comment misleads exactly as well as one in a field -- this repo has shipped that mistake three times already, and the reader has no way to tell which of the two is authoritative. The remaining 20260801 mentions are prose about what the older image did (the concurrency ceiling) and are meant to stay. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
… independent run
An agent with no prior context followed the README on this cluster and
could not reach a working deployment. Verdict was "no, not as written."
Everything below is its findings; the two hard blockers are real defects,
not documentation nits.
BLOCKER 1 -- mixed/ and mixed-dspark/ had no nodeSelector. Both the
server and the worker mount the weights as a hostPath, and both are
scheduled independently, so on a fleet whose model paths differ per node
they can land on different ones. That is exactly what happened: server on
chi2866, worker on chi2800. On chi2866 the other node's path exists but
is EMPTY, so `hostPath type: Directory` passes and the failure surfaces
much later as
ValueError: '/models/Kimi-K3': not a local path and HF resolution failed:
Repo id must be in the form 'repo_name' or 'namespace/repo_name'
which reads as a bad model name and points nowhere near the real cause.
Added a <NODE> placeholder pinning both pods, with the failure mode in a
comment. My own runs never hit this because both pods happened to land on
the same node -- the validation was true but lucky, and I recorded it as
"validated end-to-end" without noticing the manifest permitted the split.
BLOCKER 2 -- <MODEL_DIR> had no discovery path. The doc said what it must
be ("local NVMe") and never how to find it on a cluster where the weights
already exist. The agent had to read hostPath out of an unrelated
deployment. Added both ways to look it up. On this fleet the two nodes do
not agree on the path, so there was no guessable answer.
Also fixed, each of which independently stops or misleads a reader:
- Every manifest header said "Deploy: kubectl apply -f <this file>" while
the file is full of placeholders. A bare apply is ACCEPTED by the API
server -- the placeholder passes CRD validation -- and fails minutes
later at mount time. A false success is worse than an error. The
headers now say the file cannot be applied directly.
- The manual page printed that same bare apply in two tabs and never
mentioned substitution for the 8-GPU combinations at all.
- Namespace `infera` is hardcoded in every manifest and nothing created
it; the operator chart has no Namespace template.
- `helm install` fails verbatim on k3s (kubectl is a k3s symlink and
finds the kubeconfig implicitly, helm does not) and is not idempotent.
Now `helm upgrade --install` with the KUBECONFIG export called out.
- README section 2 said the pd-dspark draft goes on the decode node. The
manifest says both nodes and is right -- both roles carry
--speculative-config. Section 2 contradicted the fix two sections
later, and a reader provisioning per section 2 gets the exact missing-path
failure section 2 warns about.
- `<mixed|mixed-dspark>` inside a bash block is a redirect and a pipe:
"mixed-dspark: command not found". Same for `<pd|pd-dspark>`.
- Service names were given as `<name>-server` with no mapping. mixed/ is
`kimi-k3-opt-base` -- obtainable only by opening the YAML. Added a table.
- The speculation check selected workers by service label alone, so with
two deployments up `head -1` can pick the NON-speculative one and report
that speculation did not engage on a correct deployment. Now scoped by
deployment.
- The RDMA one-liner -- whose entire purpose is preventing false "no RDMA"
diagnoses -- did not say which pod to run it in. In the server pod it
returns 0, because that pod is not privileged and does not mount
/dev/infiniband. It reproduced the false negative it exists to prevent.
- "as written" was inaccurate everywhere: these files are templates.
- A cross-reference pointed at section 7 instead of 6.
- --enable-prefix-caching was duplicated in both mixed manifests.
- The manual carried an admonition literally titled "placeholder", which
I introduced while restructuring and never rendered. Its title also
broke the tab-set: backticks in a fence info string terminate the
container early, which silently dropped three of the four tabs.
The agent's report also confirms what holds up: the NFS-masquerading-as-
local warning is exactly right on this fleet, the PD decode-side handoff
check and the speculation checks worked verbatim, and the numbers are
consistent across both documents and the manifest headers.
Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
The path information existed but was spread through the prerequisites
section, where the reader meets it after already choosing a combination.
Both documents now open with a table of every placeholder, what it is,
and how to find its value.
The model directory gets its own warning because it is the one value this
document cannot supply and cannot default: it is site-specific, and on
the fleet these were validated on the two GPU nodes use different paths
for the same weights -- which is why the PD manifests take two directory
placeholders rather than one.
It also names both wrong-value failure modes, because neither says
"wrong path":
absent -> hostPath type check failed
exists but empty -> mounts fine, then crashloops minutes later with
ValueError: '/models/Kimi-K3': not a local path and
HF resolution failed: Repo id must be in the form ...
The second is what an independent reader actually hit, by copying a
working node's path onto a node that never had the weights.
Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
There was no pre-flight anywhere in this repo. Every prerequisite failure
here costs 12-14 minutes to discover, because that is how long weight
load plus JIT takes before the engine gets far enough to complain -- and
the two most likely mistakes do not complain about the path at all.
preflight.sh takes ~15 seconds and checks, from the reader's position
(kubectl only, no shell on the node): namespace, CRD, free GPUs on the
target node, whether MODEL_DIR on THAT NODE holds the 96 shards, and
whether it is local storage.
The node-side checks run in a throwaway busybox pod pinned to the node,
because the reader generally cannot log into it. hostPath type is
DirectoryOrCreate rather than Directory on purpose: a missing path has to
be reported by this script, not turned into a pod that will not schedule.
The two cases it exists for:
Empty directory. Copying a working node's path onto a node that never
had the weights leaves a path that EXISTS, so hostPath type: Directory
passes, the pod starts, and the engine crashloops minutes later with
ValueError: '/models/Kimi-K3': not a local path and HF resolution failed:
Repo id must be in the form 'repo_name' or 'namespace/repo_name'
which reads as a bad model name. An independent reviewer hit exactly
this on their first deploy attempt.
Network mount with a local-looking name. Shard count right, directory
right, only the backing device wrong. Load goes from ~8 min to ~95,
past the ready timeout, so the worker restarts mid-load forever and
presents as a crash loop rather than as slow storage.
Verified against four cases on this cluster before committing, rather
than shipped on the strength of reading correctly:
good local path -> 96 shards, xfs /dev/nvme4n1 PASS
empty dir on peer -> "not present ... mounts fine and crashloops later"
NFS export as local -> "is nfs4 (10.x:/mnt/nvmeraid) - a network mount"
(with the shard check PASSING first, which is
the point: only the device is wrong)
--dspark -> draft present alongside the target
The GPU check reporting 0 free during those runs was correct -- another
deployment held them.
Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
… more findings
A second reviewer, told nothing about the first, followed the README and
got BOTH `mixed` and `pd` deployed, Ready, and answering the smoke test.
Verdict moved from "no, not as written" to "yes-but". These are the
remaining gaps, and the first is one I introduced.
REGRESSION I CAUSED. I added <NODE> to the manifests, to the placeholder
table, and to the manual page -- and not to the README's own deploy
command, which is the line a reader copies. Applying it as printed leaves
<NODE> in place, and the failure is nothing like what the document
promised:
kubectl apply -> "created", exit 0
kubectl get pods -> nothing, ever
InferaDeployment -> status empty, STATE blank
No Pod is created, so there is no mount, so the mount-time failure §0
described never happens. The only error is in the operator's log in the
infera-system namespace:
spec.template.spec.nodeSelector: Invalid value: "<NODE>": a valid label
must be an empty string or consist of alphanumeric characters...
§0 now distinguishes the two placeholder failures, because they are not
the same: <MODEL_DIR> creates a Pod that fails at mount, <NODE> creates
no Pod at all. A reader told to wait for a crashloop will wait forever.
PATH DISCOVERY FOR A NODE YOU CANNOT LOG INTO. All three methods the
document offered fail for the remote node in the PD workflow:
- "read it off a working deployment" is self-defeating -- PD needs all
16 GPUs, so that deployment must be deleted first
- "or a privileged pod" was the entire instruction: no manifest, no
command
- "confirm with df -hT" cannot run against a node you have no shell on
The reviewer only had the decode path because they read it before
teardown, i.e. from outside the document's own workflow.
preflight.sh grows a --find mode that runs on the target node and lists
the candidates. Note what it deliberately does NOT report: through a
hostPath mount of /, submounts are not propagated, so df inside that pod
returns the root filesystem for every path -- my first version used it
and labelled a local array as a network mount, steering the reader away
from the correct answer. --find reports paths and shard counts only
(file visibility through that mount IS reliable); the filesystem verdict
stays with the full check, which mounts the candidate directly. Verified
on this cluster: --find lists three identical-looking 96-shard
candidates, and the full check separates them.
Also fixed:
- The smoke-test curl had no --max-time, while §7 claimed "every probe
here carries an explicit timeout". It was the one probe a reader
actually runs, against a system whose documented failure mode is a
45-minute hang. Added, along with a liveness check on the port-forward
-- when the port was already held, port-forward failed and the piped
curl printed nothing and exited 0: a silent pass on the only health
check.
- The PD example substituted the SAME directory for both nodes, modelling
the exact mistake §0 spends a paragraph warning against.
- The GPU capacity command reads .status.allocatable, which prints 8 for
a fully occupied node. Said so, and pointed at preflight.sh.
- `<mixed|mixed-dspark>` inside a bash block is a redirect and a pipe.
- The Engine 000 handoff check is timing-sensitive: vLLM emits those
lines for about two intervals after traffic, so running it later
returns idle zeros and looks like a failed handoff.
- First start measured 9m55s and 11m, not 12-14.
What held up on independent execution: every §7 claim the reviewer could
test, the KUBECONFIG tip reproducing verbatim, ibv_devices genuinely
absent with the libibverbs snippet returning 8 on decode and 0 on the
server pod exactly as documented, all §5 label selectors matching what
the operator applies, the directory-to-deployment-name table, and the PD
decode-side handoff check -- 98.9% external prefix cache hit with prefill
showing the inverse split, confirming the KV handoff was real rather than
failing open.
Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
This row sat in a table of diagnosed failures, alongside three claims I had actually reproduced, while being the one I had only inherited. It read as equally evidenced. Now measured, against the same manifest with the flag flipped and nothing else changed: c=8 124.34 vs 241.69 tok/s 51% of baseline c=16 331.56 vs 427.86 78% The old text said "~40% of throughput". Wrong twice: the number is 51-78% rather than 40%, and stating one number implies a flat factor when the penalty is in fact much worse at low concurrency. What does hold, and is now stated precisely, is that it is silent. The worker starts normally in ~12 min, logs no warning, and median TPOT barely moves (28.52 vs 27.85 ms at c=8) -- so latency monitoring will not catch it. Only aggregate throughput shows the loss. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
A third reviewer deployed mixed-dspark and pd-dspark from these docs on
the first attempt, no dead ends, and both answered the smoke test. The
serious findings are in the pre-flight script I added last round: it was
confidently wrong twice, and both times in the PASS direction, which is
the direction that costs the 12-14 minutes it exists to save.
WRONG ANSWER 1 -- --dspark passed on an empty draft directory. The test
was `[ -n "$d" ]` where $d is the shard COUNT, so the string "0" is
non-empty and an empty directory reported OK. That is precisely the
empty-directory trap the script's own header says it exists to catch:
implemented for the target, not for the draft. An operator whose draft
download failed halfway got a green light from the one tool meant to
catch it. Reproduced with a probe, fixed, and re-checked:
FAIL ... Kimi-K3-DSpark on chi2800 exists but holds no safetensors --
an interrupted download leaves exactly this
The same reviewer noted the target check counts files, not bytes: 96
zero-byte shards read as "96 shards". Now also checks size --
FAIL ... has 96 shards but only 1 MB (expect ~1400000 MB)
while the real directory reports "96 shards, 1488685 MB".
WRONG ANSWER 2 -- the free-GPU check only counted pods in namespace
infera, and only Running ones. A pod in `default` holding all 8 GPUs on a
node produced "OK node chi2866: 8 of 8 GPUs free". Now counts every
namespace and every non-Succeeded phase, so a bound-but-ContainerCreating
pod counts too. Reproduced with a squatter pod in `default`: reports
"only 0 of 8 GPUs free", and 8 again once deleted.
THE CHECK MUTATED WHAT IT CHECKED. It mounted MODEL_DIR with hostPath
type DirectoryOrCreate, so checking a typo'd path CREATED it -- and a
later deploy with `type: Directory` then succeeded on that empty
directory, converting a fast mount failure into the very crashloop the
script exists to prevent. It now mounts the parent read-only and looks
inside, creating nothing. Verified: the missing path is still reported,
and afterwards the directory is still absent.
Documentation fixes from the same run:
- The PD pre-flight example never passed --dspark on the prefill node,
while the prose two sections later says the draft must be on BOTH
nodes. The doc's own check omitted the failure it warns about.
- No teardown command existed anywhere, though the combinations cannot
coexist on 16 GPUs so switching is mandatory.
- max_tokens: 200 is not enough. Measured 1 request in 4 returns
finish_reason "length" with 199 tokens of reasoning preamble and never
reaches the answer -- the "looks broken but isn't" outcome the doc
warns about, caused by the doc's own recommended cap. Now 1024.
- The smoke test hardcoded local port 8000; the guard correctly detected
the port being held and then offered no next step. Moved to 18000 and
says to pick another.
- An orphan code fence in the manual pushed "Both failures hang rather
than error" outside the admonition that introduces those two failures,
so the sentence lost its antecedent. Sphinx does not warn about this --
my "builds clean with warnings-as-errors" claim in this PR was true and
still missed it. Fence count is now even and the text renders inside.
- The manual's smoke test had dropped --max-time and the port-forward
guard entirely, teaching the exact anti-pattern the README calls
non-optional -- in the rendered version most readers see.
- The manual's PD example substituted the same directory for both nodes,
which would have failed on this fleet.
Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
A fourth reviewer attacked the pre-flight script adversarially and got a
wrong verdict in 15 of 25 cases, then reproduced one published throughput
figure and could not.
WITHDRAWING preflight.sh. It failed as a gate in both directions on the
fleet it was written for:
false PASS on weights sitting on their own mount under a local parent
-- the exact "network mount wearing a local-looking name" scenario
its own header named as its reason to exist
false PASS on a 7.7%-complete download, a 92-of-96-shard directory, a
directory with no config.json, a directory carrying a DIFFERENT
model's config, a cordoned node, and a NoSchedule-tainted node --
the last two proven to leave pods Pending
false FAIL on chi2866's only local path, which made `pd` ungateable on
the validation fleet: a reader obeying the gate concludes pd is
impossible. It is not; it ran.
The false FAIL is a regression I introduced last round. Fixing "the check
creates the directory it checks" by mounting / instead of the directory
broke filesystem detection: mounting the directory reports xfs /dev/md1,
mounting / reports nfs4. I traded a side effect for a wrong answer, and
the comment I wrote justifying the new design was false in both of its
claims -- which is what produced both classes of error.
Three consecutive rounds, each fix introducing a new defect, is enough
evidence that this belongs in its own PR with regression tests over the
adversarial cases, reviewed as code. The documentation of what to check,
why, and what each failure looks like is accurate and stays; §0 goes back
to explicit probe pods, with the one thing that actually matters spelled
out: mount the directory itself, not a parent, or df reports the root
filesystem and calls a network mount local.
WITHDRAWING the throughput tables. An independent run of the exact sweep
this page specified got 165.99 tok/s cold and 270.4 warm against a
published 241.69 -- matching neither. At num-prompts = 4 x concurrency
the sweep is 32 requests over ~15 s, short enough that first-request TTFT
(~2500 ms cold, ~400 ms warm) dominates, and no warm-up was specified.
That also undermines the comparisons built on those numbers, and not
randomly: the sweeps ran back-to-back within one deployment, low
concurrency first, so early points were colder than late ones -- the same
direction as the trends being reported. Warm, pd and mixed at c=8
measured 269.7 and 270.4, effectively identical, where this page claimed
0.90x. Every ratio derived from the tables is removed with them, in the
combination-selection table too; leaving conclusions behind after
retracting their evidence would be worse than either.
What survives is stated as such: median TPOT reproduced within a few
percent independently (26.51 vs 27.85 at c=8 mixed; 25.70 vs 25.81 pd)
and is kept, because it is a per-step measure that does not care about
warm-up. So are the binary facts -- the concurrency crash being fixed,
speculation genuinely engaged rather than disabled, the PD handoff
verified on the decode side, and pd-dspark needing speculation on both
roles.
The PREROUTE_BF16 row keeps its comparison but now says why it is still
meaningful: both sides were measured as the first sweep after their own
fresh deployment, so the ratio is like-for-like even though the absolute
figures are not reproducible.
Re-establishing throughput needs a stated warm-up protocol and a sweep
long enough that the first request does not dominate. The page now says
that instead of publishing numbers.
Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
…-20260802image fixes the DSpark concurrency crash, so every combination was re-measured on it, and a fourth combination (pd-dspark) became possible.All four, one image, identical sweep
vllm bench serve, 1024 in / 128 out,--random-range-ratio 0,--ignore-eos,--seed 0,num-prompts = 4 × concurrency.mixed(8 GPU)mixed-dspark(8)pd(16)pd-dspark(16)TPOT at c=64:
mixed48.81 ms ·pd38.12 ms ·pd-dspark17.44 ms.The crash is genuinely fixed, not sidestepped — c=16/32/64 complete, 0 restarts, the
AiterMLA flattened verifyassertion appears zero times, and speculation is still on (7 draft tokens, CUDA-graph captured,running the draft eagerlycount 0). Disabling speculation would also make that assertion disappear, which is why that last check is there.Two conclusions on the previous revision were wrong
mixedbaseline from the older image, andmixeditself moved −15% at c=4 and +29% at c=8 between the two, in opposite directions. Cross-image ratios are the easy thing to miss in review because both numbers are individually true.mixedabove c=8." The crossover is between c=16 and c=32.pd-dspark: speculation on both rolesThe obvious configuration is decode-only — the prefiller never samples. It fails two independent ways, and both hang rather than error: pods Ready, health checks green, restarts 0, no inference logged, client waits until its own timeout (45 minutes in the run that found it).
model.layers.0…97(93 + 5) and sends all 98 names to a prefiller with nothing past 92 →KeyError: 'model.layers.93.self_attn'.pulling kv_caches ... failed: P num blocks less than D. Speculation recomputesmax_num_scheduled_tokensto reserve draft slots, so the decoder's per-request block accounting differs from a prefiller that doesn't know speculation is happening. No layer filtering fixes that.So the prefiller must be speculation-aware even though it never runs the draft. Loading it there is the price of that agreement, not an oversight.
A connector patch was tried and withdrawn
An earlier revision of this PR shipped a patch that skipped draft layers at registration, and claimed it enabled decode-only. It cleared failure 1 (KeyError 16 → 0, skip confirmed on all 8 ranks) and I nearly shipped on that evidence — but it does nothing for failure 2. It also referenced overlay
v0.2.2, which does not contain it, so a reader following the recipe would have hit the exact hang the page warned about. My own validation missed that because I mounted the patch via ConfigMap and bypassed the overlay entirely. Withdrawn; no new overlay image is needed.Per GPU
Absolute throughput favours the 16-GPU options at every concurrency; tokens-per-GPU never does. The PD pair buys headroom (1217 vs
mixed's 972 ceiling) and latency (17 ms vs 49 ms), not efficiency — the page says to pick on those rather than the tok/s column.Validation
Every combination run end-to-end as written, c=4…64 swept. Both PD combinations held
External prefix cache hit rateat 99.9% across all five concurrencies — checked per concurrency, because a handoff that fails open under load makes throughput look better. Both failure counters stayed at 0 through the sweeps.Manual builds clean with warnings-as-errors; four tabs render and sync. All four manifests pass
kubectl apply --dry-run=server.