refactor(recipes): aggregated / disaggregated naming, and the RDMA check on the published page - #76
Merged
Conversation
`mixed` never said what was mixed, and `pd` needed you to already know it
abbreviates prefill/decode. `aggregated` / `disaggregated` is the
vocabulary the field uses -- it is what NVIDIA Dynamo's recipe pages use,
and what someone searching for this feature will type.
Renamed across all three models so the combination set stays comparable
between them, which is the whole reason it is a fixed set:
mixed -> aggregated pd -> disaggregated
mixed-kvd -> aggregated-kvd pd-kvd -> disaggregated-kvd
mixed-dspark -> aggregated-dspark pd-dspark -> disaggregated-dspark
pd-sglang -> disaggregated-sglang
13 directories, plus every reference in prose, link text, link targets,
tab labels and tab sync ids. Renaming only one model would have broken
cross-model comparability; renaming only the directories and leaving the
prose would have left text and links disagreeing.
THE API VALUES ARE DELIBERATELY UNCHANGED. `role:` inside each manifest
is consumed by the operator and is still `mixed` / `prefill` / `decode`,
as is DisaggMode.MIXED in the Python. Renaming those would break every
deployed configuration and anything outside this repo using the CRD, and
it is a separate decision for the team to make on its own terms.
That leaves `aggregated/deploy.yaml` legitimately containing `role:
mixed`, which reads like a mistake if you meet it cold -- so both the
repo README and the manual index now say so explicitly: the directory
name is what the combination means, `role:` is the wire format. A reader
who spots the discrepancy finds it already accounted for rather than
concluding the docs are stale.
Verified: all 13 manifests still pass `kubectl apply --dry-run=server`;
every deploy.yaml path referenced from any document exists; the manual
builds clean with warnings-as-errors and all four tabs on each of the
three model pages render with the new labels and sync ids.
Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
The published page told the reader that `ibv_devices` is absent and that `ibv_get_device_list` is the right question, then sent them to the repo README for the actual command. Someone reading the docs site to find out whether their NICs are visible could not get an answer from the docs site. The Disaggregated tab now carries both forms: a standalone probe pod for before you deploy, and the in-pod version for after. Both state the thing that makes this check worth running at all -- when the container cannot see the RDMA devices, nothing says so. Mooncake falls back to a transport that cannot cross nodes, and the symptom is a request that hangs while every pod stays Ready. It also says which pod to run it in. The router runs the same image but is not privileged and does not mount /dev/infiniband, so it reports 0 -- reproducing the exact false negative this check exists to prevent. That was observed twice during validation. There is no pre-flight script to point at: the one added earlier was withdrawn for giving wrong verdicts in both directions, so the fabric check is documented as a command rather than a tool. Also repaired a sentence in the validation section that an earlier edit of mine had left spliced together mid-clause. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
jiejingzhangamd
requested review from
JohnQinAMD,
limou102 and
xiaobochen-amd
as code owners
August 3, 2026 01:36
I was wrong, and the correction matters more than the edit.
When I added a pre-flight script to this recipe two revisions ago, I
justified it with "There was no pre-flight anywhere in this repo" -- in a
commit message that is now merged. I had grepped two directories,
examples/recipes/ and manual/recipes/, and reported the result as if I
had searched the repository.
The repository has two layers of preflight, both better than what I
wrote:
infera/tools/preflight/ a cluster diagnostic covering RDMA
device and link state, cross-node RoCE
bandwidth, Mooncake KV-transfer
bandwidth measured separately over
rdma and tcp, GPU compute/HBM/P2P,
firmware and cross-node consistency,
and storage -- including "KV not on
local NVMe -> FAIL", which is the
local-versus-network check my script
got wrong in both directions before it
was withdrawn. Renders one HTML report,
with a SLURM path for multi-node.
infera/common/disagg_preflight.py config validation at launch, before
the engine subprocess starts, for the
two silent-failure modes of cross-node
PD: advertising a non-routable host to
etcd, and configurations prone to
silent TCP fallback.
Both are already referenced from other recipes in this repo
(examples/deepseek_v4, examples/sglang_1p2d_kimi2.6, bench/pd_mori_1p1d).
This recipe was the one that did not know about them, and I made that
worse by building a replacement instead of looking.
Both documents now send the reader to the real tooling, calling out the
Mooncake rdma-versus-tcp rows as the ones that matter here -- a fabric
that will silently serve at TCP speed shows up as a number rather than as
a deployment that is inexplicably slow. The manual keeps the
ibv_get_device_list one-liner for the narrower "can the container see the
devices at all" question, with the warning about which pod to run it in.
Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
Asking where the docs explain how to use preflight turned up the real
answer: they do not. The tool is referenced from six places across the
examples and troubleshooting pages, each of which links out to a file on
GitHub -- so a reader on the docs site has to leave it to learn how to
run the thing those pages tell them to run first. It was not in the toc.
The invocation was inconsistent too, depending on which page you landed
on: `infera-preflight --firmware` in troubleshooting, `python -m
infera.tools.preflight --dump-path <dir> --netperf --mooncake` in the
DSv4 example. Both work -- `infera-preflight` is a console script in
pyproject.toml -- but nothing said so.
The new page covers what it checks, how to run it single-node and under
SLURM, what the report contains, and the three things that are easy to
get wrong:
* run it INSIDE the engine container for the full set. torch,
ais-check, Mooncake and Mori exist only in the image, and without the
host's libionic.so mounted, in-container ib_write_bw / Mooncake /
Mori cannot see the ionic RDMA devices -- which looks identical to
the node not having any.
* use a fresh --dump-path. Rank 0 decides everyone has reported by
counting *.json in the directory, so a stale file makes it render
early.
* read the Mooncake rows for PD. They report KV-move bandwidth over
RDMA and over TCP separately, which is what turns "silently serving
at TCP speed" from a mystery into a number.
It also leads with why the tool exists at all: nearly everything it
checks fails silently. RDMA the container cannot see does not raise --
Mooncake falls back to a transport that cannot cross nodes and the
request simply never returns. A KV path on network storage does not
raise either; the weight load just takes an order of magnitude longer,
exceeds the ready timeout, and presents as a crash loop.
The separate launch-time config validation in
infera/common/disagg_preflight.py is documented alongside it, with the
distinction stated: it is config-only and cannot tell you the hardware is
healthy.
References in the kimi-k3-optimized recipe and the DSv4 example now point
at this page instead of at GitHub.
Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two changes. Both were pushed after #75 merged, so neither made it in.
1.
mixed/pd→aggregated/disaggregatedmixednever said what was mixed, andpdneeded you to already know it abbreviates prefill/decode.aggregated/disaggregatedis the vocabulary the field uses — what NVIDIA Dynamo's recipe pages use, and what someone searching for this feature will type.Renamed across all three models, so the combination set stays comparable between them — which is the whole reason it is a fixed set:
13 directories, plus every reference in prose, link text, link targets, tab labels and tab sync ids. Renaming only one model would have broken cross-model comparability; renaming the directories and leaving the prose would have left text and links disagreeing.
The API values are deliberately unchanged
role:inside each manifest is consumed by the operator and is stillmixed/prefill/decode, as isDisaggMode.MIXEDin the Python. Renaming those would break every deployed configuration and anything outside this repo using the CRD — a separate decision, on its own terms.That leaves
aggregated/deploy.yamllegitimately containingrole: mixed, which reads like a mistake if you meet it cold. Both the repo README and the manual index now say so explicitly: the directory name is what the combination means,role:is the wire format. A reader who spots the discrepancy finds it already accounted for rather than concluding the docs are stale.Verified: all 13 manifests still pass
kubectl apply --dry-run=server; everydeploy.yamlpath referenced from any document exists; the manual builds clean with warnings-as-errors and all four tabs on each of the three model pages render with the new labels and sync ids.2. The RDMA device check is now on the published page
The page told readers that
ibv_devicesis absent and thatibv_get_device_listis the right question — then sent them to the repo README for the command. Someone reading the docs site to find out whether their NICs are visible could not get an answer from the docs site.The Disaggregated tab now carries both forms: a standalone probe pod for before you deploy, and the in-pod version for after. Both state why it is worth running: when the container cannot see the RDMA devices, nothing says so. Mooncake falls back to a transport that cannot cross nodes, and the symptom is a request that hangs while every pod stays Ready.
It also says which pod to run it in. The router runs the same image but is not privileged and does not mount
/dev/infiniband, so it reports0— reproducing the exact false negative this check exists to prevent. That was observed twice during validation.There is no pre-flight script to point at: the one added earlier was withdrawn in #74 for giving wrong verdicts in both directions, so the fabric check is documented as a command rather than a tool.
Also repaired a sentence in the validation section that an earlier edit had left spliced together mid-clause.