[wip]fix(kvaware+kvd): two bugs blocking KV-aware routing + kvd together on SGLang, with image, docs and repro

[dorado269]

Two bug fixes found while bringing up KV-aware routing + kvd (tiered KV offload) together on the SGLang engine, plus the deployment image and the operator documentation for that configuration, plus a reproduction kit.

Verified end-to-end on GLM-5.2-MXFP4, two-node PD over Mooncake RDMA, DP-attention 8/8 on both legs, both features on.

---

The two fixes

fix(net) — free_tcp_port_block returned the same base to every caller

It scanned downward from a fixed start (ip_local_port_range.low - count) and released the probe sockets before returning, so two engines launched on one host picked the same block deterministically. The second leg died with:

zmq.error.ZMQError: Address already in use (addr='tcp://*:32765')

Reached only from the kv-aware path (sglang/worker.py:77, when enable_kv_events is set and SGLang binds one KV-event publisher per DP rank at base + rank).

The reservation cannot be made exclusive — the probe binds 127.0.0.1:P while the real listener binds 0.0.0.0:P (zmq tcp://*). Two alternatives were tried and rejected against live MVPs:

• Hold the reservation until the child binds → the child then fails with errno 98; we would be locking out our own subprocess.
• 0.0.0.0 + SO_REUSEADDR → a second probe can still take the same port, so it buys nothing.

So the fix removes the determinism instead: 64 random bases in the same sub-ephemeral window, then the original exhaustive scan as fallback.

Blast radius: exactly one caller in the repo. vLLM and ATOM use the other function, free_tcp_port(), which is untouched.

fix(kvd) — bind-mounted L3 silently lost O_DIRECT

For a bind mount findmnt reports the source with the bind subpath in brackets:

/dev/md0[/mnt/nvme-raid/kvd-long]

That whole string went to lsblk, which answers "not a block device". The failure is silent: classification falls back to buffered I/O even on hardware that qualifies for O_DIRECT — i.e. exactly the normal deployment, an L3 directory bind-mounted into the engine container.

Blast radius: storage_classify is private to the kvd daemon, but the daemon is shared by all three engines, so vLLM's kvd L3 path benefits identically. Second-order effect checked deliberately: mount_source is also compared against /proc/mounts (src == source) for the NFS nconnect/rsize/wsize probes. /proc/mounts never carries the bracket form, so stripping moves that comparison from always fails to can succeed — a fix there too, not a regression.

Both fixes have tests that fail on the pre-fix code

git checkout <base> -- infera/common/net.py infera/kvd/storage_classify.py
pytest tests/unit/common/test_net_port_block.py tests/unit/kvd/test_storage_classify.py -q
# 3 failed, 48 passed        <- fixes reverted
# 51 passed                  <- fixes present

A trap worth flagging for reviewers: the first version of the storage_classify test passed on the broken code, because the fake _run returned success for lsblk regardless of its argument. It had to be taught the real contract (real lsblk exits non-zero on a bracketed target). If you extend these tests, revert the fix and confirm they go red.

Full unit suite, differential, same image: main = 83 failed / 965 passed / 58 errors; this branch = 83 failed / 971 passed / 58 errors. Identical failures, +6 passes (the new tests). The 83/58 are pre-existing container-environment limitations present on main, not introduced here.

---

build(sglang) — the deployment image

deploy/docker/Dockerfile.sglang.kvaware-kvd layers on Dockerfile.sglang rather than duplicating a 78.6 GB build — the existing image already contains engine + wrapper + kvd daemon + router + statctl, and one image runs all four roles.

What it adds is the contract:

• a build-time self-check that imports InferaKvdBackend, wire_infera_kvd_backend, attach_to_radix_cache, statctl, and asserts free_tcp_port_block no longer returns a fixed base. It discriminates — 10 distinct bases post-fix, 1 pre-fix. A build that loses the kvd adapter (bad merge, or a base bump that moves HiCacheStorage) now fails at build time instead of starting fine and serving with no L3 at all.
• PYTHONHASHSEED=0 baked in — for kvd this is a correctness requirement, not tuning: unset it is random per process, the two PD legs hash the same prompt differently, and every restart orphans the whole L3 cache.
• a digest-pinnable base arg, because the unified-Mooncake rebuild is exactly the layer you don't want silently swapped.

---

Docs

docs(serving) — new manual/serving/kvaware_kvd_operations.md: which settings are experiment-only and what production should use instead, the recommended all-switches-on config, how to verify each switch actually took effect (with a negative case for each — all three features fail silently by degrading to a slower-but-correct path), and which numbers are tunable with recommended values and which direction to move them.

One trap documented there: --kv-overlap-weight defaults to 1.0 and the per-role weights fall back to it, so a PD deployment that doesn't set them explicitly behaves nearly round-robin and looks like kv-aware "does nothing".

docs(kvd) — ⚠️ this reverses #26 / 29a69ca ("kvd offload is vLLM-only"), so please sanity-check me here @jiejingzhangamd. The reasoning:

• InferaKvdBackend (a SGLang HiCacheStorage backend) has been in the tree since Infera v0.1.0 — it predates the doc saying it doesn't exist.
• The likely reason for that doc is that CI's kvd e2e covers only vLLM (tests/e2e/pd_mixed/vllm/test_mixed_kvd.py; there is no SGLang equivalent), so the path was untested rather than absent.
• This PR tested it on hardware (below).

AIC GPU-Direct remains vLLM-only; SGLang reads via the daemon's POSIX path. Happy to drop this commit if you'd rather gate the doc change on a SGLang kvd e2e case landing first.

---

Hardware verification

Image built from this branch, sha256:c9292019…, same digest on both nodes. 8× MI355X per node, ionic RoCE, all 8 rails active.

Test	Result
Correctness (temp=0 factual probe)	4/4, coherent
conc=32, 128 req, ISL/OSL 1024	128/128 needle correct, 0 HTTP errors, 128/128 finish=stop, 3.1 req/s
conc=128, 512 req	502/512 needle correct, 0 HTTP errors, 6.8 req/s

conc=128's 10 non-clean responses are all finish=length at exactly the 1024-token cap — the run-on mode of a harness that sends no chat template and suppresses EOS for throughput. The split is clean: 502 good at finish=stop median 149 tokens, 10 bad at exactly 1024. No KV-corruption signature at either concurrency.

kvd is shown to actually serve, not merely to be wired

This is the part that is easy to fake, so it is worth spelling out. During the stress tests kvd wrote 32 GB and read nothing (gets_total=0) — SGLang's in-GPU radix cache serves repeated prefixes without ever touching L3. A passing test proves nothing here, and neither does a latency win (in earlier work a reuse phase ran 2.7× faster with kvd's counters flat at zero).

The discriminating check is restart-and-replay — restarting the engine empties the GPU cache while the daemon and its L3 survive:

	gets	hits	sets	misses
before restart	0	0	18272	0
after restart	102	102	18272 (unchanged)	0

Reads with no new writes on an empty GPU cache. That reuse could only have come from L3.

---

Reproduction kit

kvaware_kvd_pr.packup_20260731/ — spec with criteria-vs-result, ordered reproduce steps, environment (hardware / drivers / fabric / image digests), per-patch what-why-how-context, the Dockerfiles, every script verbatim, raw JSON results, and logs.

Verified self-contained: a fresh clone at 8692fb4 plus these patches produces a tree hashing identical to this branch's HEAD.

notes.md records the wrong turns, including two general traps: a docker save | gzip that was silently truncated yet passed gzip -t (verify the tar, not the gzip), and a stress classifier that reported CORRUPT_REASONING for a correct answer because GLM-5.2 legitimately reasoned in Chinese.

---

Known gaps (deliberate, stated so nothing reads as more than it is)

• No kvd latency benefit observed. kvd demonstrably served 102 blocks; it did not demonstrably make anything faster at this scale.
• The overlap weights (20.0 / 2.0) were used but not validated — one prefill and one decode worker means the scorer never had a choice to make. They are documented as a starting point, not benchmarked optima.
• No SGLang kvd e2e case added. That is what would permanently settle the doc question above.
• 854ebf7 (long-context Mooncake wait_event fix) is deliberately not included — it lives on another branch and is tracked separately.

🤖 Generated with Claude Code

[jiejingzhangamd]

better remove

minimize code change

[jiejingzhangamd]

@yihou heads up — I force-pushed this branch (52d71195 → 5a4dfba2). If you have unpushed local work on it, rebase it onto the new head before doing anything else; --force-with-lease protects what was on the remote, not what is only on your machine.

What changed, and nothing else:

Rebased onto main. The branch was 39 commits behind. All 7 commits replayed with no conflicts, authorship untouched.

Fixed the lint job (one new commit). Four hooks were failing — trailing-whitespace, end-of-file-fixer, ruff-format, ruff-check. Almost all whitespace, but two of them touch files that are evidence rather than source, so they were checked rather than waved through:

• the .patch files lose the single space on blank context lines. Verified this does not break them: git apply reconstructs blank context lines and a stripped patch still applies with the correct result.
• results/raw/*.json gain a trailing newline. Content untouched — the diff is \ No newline at end of file only.

The one real fix is E402 in scripts/stress_capture.py: import os sat mid-file next to the IDX-replay block, hoisted into the top import block.

Added the missing DCO sign-off — please read this part. The dco job was failing on all 7 commits. .github/workflows/dco.yml requires the author's email to appear in the sign-off, so signing them under my name would have kept the check red with a different message. Each commit is therefore signed off as its own author — yours carry Signed-off-by: yihou <yihou@amd.com>.

I did this at @kzjeef's instruction, on the statement that you had authorised it. Flagging it explicitly because a DCO sign-off is a certification made in your name, and it should not appear in the history without you knowing it was added by someone else's rebase. If you did not authorise it, say so and I will strip the trailers so you can sign them yourself — for the record that is just:

git fetch && git checkout yihou.dev.glm5.2.mxfp4.kvaware.kvd.pr
git rebase --signoff origin/main && git push --force-with-lease

The trees are byte-identical to before the sign-off; only trailers were added.

Still outstanding: REVIEW_REQUIRED.

[jiejingzhangamd]

Reviewed the ~40 lines of behaviour change (infera/common/net.py, infera/kvd/storage_classify.py); the remaining ~10k lines are the packup archive. Both diagnoses are convincing and the tests are good — 4 new for the port block, 50 for storage classification including two bind-mount cases. One blocking issue.

Blocking — free_tcp_port_block crashes on a tuned ip_local_port_range

highest = low - count
randomised = [random.randint(1024, highest) for _ in range(_PORT_BLOCK_TRIES)]

random.randint raises when highest < 1024, which happens whenever the host's ephemeral range starts at or near 1024. net.ipv4.ip_local_port_range = 1024 65535 is a real high-concurrency tuning, not a hypothetical.

Verified against both versions with the range faked to 1024 65535:

main (original) -> RuntimeError: could not find 8 contiguous free TCP ports
PR  (randomised)-> ValueError: empty range in randrange(1024, 1017)

So it is a regression in two ways. The old code's range(low - count, 1024, -1) was simply empty and fell through to the deliberate RuntimeError, which names the problem. The new code raises ValueError out of random before probing a single port — on a host where the old code would also have failed, but legibly.

Suggest guarding the window and keeping the existing failure path:

highest = low - count
if highest < 1024:
    raise RuntimeError(
        f"no room for a {count}-port block below the ephemeral range "
        f"(ip_local_port_range starts at {low})"
    )

Worth a test — the existing four all run against the real /proc value, so this path cannot be hit in CI.

Non-blocking — find("[") misses one bind-mount shape

bracket = source.find("[")
if bracket > 0:

> 0 rather than >= 0 is right, and I assume deliberate: it stops an IPv6 source like [2001:db8::1]:/export from being truncated to nothing. Worth a comment saying so, since it reads like an off-by-one.

It does leave one case: an IPv6 NFS source that is bind-mounted, [2001:db8::1]:/export[/sub], keeps its subpath and hits the same "not a block device" path this PR fixes. rfind handles that and is identical everywhere else:

/dev/md0[/mnt/nvme-raid/kvd-long]   find -> /dev/md0                      rfind -> /dev/md0
[2001:db8::1]:/export               find -> unchanged (correct)           rfind -> unchanged
[2001:db8::1]:/export[/sub]         find -> unchanged (subpath kept)      rfind -> [2001:db8::1]:/export

Marginal, since NFS classifies as buffered anyway — but it is a one-character change.

Non-blocking — the 64 random probes can repeat

[random.randint(...) for _ in range(64)] samples with replacement, so some of the 64 tries are wasted on bases already probed. random.sample(range(1024, highest + 1), k=min(64, highest - 1023)) would not, and would also make the empty window above impossible to express. Only matters under contention, which is exactly when the tries are worth the most.

Checked and fine

• Fork safety. The failure this fixes is two engines starting concurrently on one host, so it matters whether forked children draw the same sequence — CPython reseeds random on fork, and two children produced different picks in a direct test. Not an issue.
• The exhaustive fallback still starts deterministically at highest. If all 64 random tries fail, two callers re-converge — but at that point the host is nearly out of blocks and the RuntimeError is imminent regardless.
• Reservation is loopback-only and released before returning. The docstring explains why this cannot be exclusive; the reasoning holds.

Note on the branch

I rebased this onto main (was 39 commits behind), fixed the lint job, and added the DCO sign-off — see the comment above for exactly what changed and how to undo the sign-off if it was applied in error.

[jiejingzhangamd]

Pushed 2cd56f43 fixing the blocking issue I raised, rather than leaving it for a round-trip.

free_tcp_port_block no longer crashes on a tuned ip_local_port_range. Guarded the empty window and kept the deliberate RuntimeError, so the message names the cause:

before this branch   RuntimeError: could not find 8 contiguous free TCP ports
on this branch       ValueError: empty range in randrange(1024, 1017)
after 2cd56f43       RuntimeError: no room for a 8-port block below the
                     ephemeral range (ip_local_port_range starts at 1024)

Switched the 64 probes to random.sample. Independent randint draws repeat bases, wasting tries under contention — which is when each try is worth the most. It also makes the empty window unrepresentable rather than merely guarded, so the guard and the sampling cannot drift apart.

Added the regression test. The existing four read the real /proc value, so this path cannot be reached in CI. Verified it fails with the original ValueError when the fix is reverted — a test that cannot fail is not worth having.

Behaviour on a normal host is unchanged: still a random base below the ephemeral range.

The two non-blocking points from the review are untouched and yours to judge — find("[") vs rfind("[") for an IPv6 NFS bind mount, which is marginal since NFS classifies as buffered anyway.