ccl: hierarchical cross-node AllGather (intra-node SDMA + inter-node RDMA) - #441

Merged

inkcherry merged 230 commits into

mainfrom

inkcherry/sdma-hier-allgather

Jul 30, 2026

Commits on Jun 30, 2026

ccl: hierarchical cross-node AllGather (intra-node SDMA + inter-node RDMA)
inkcherry
committed

Commits on Jul 3, 2026

examples/fsdp_sdma: transparent FSDP2 all-gather over MORI (intra SDMA + inter RDMA)
inkcherry
committed
ccl: param-contiguous zero-copy for HierAllGather (kill FSDP copy-out to beat RCCL xnode)
inkcherry
committed
ccl: fuse HierAllGather param-contiguous scatter into ONE launch (kill N*P launch overhead to beat RCCL xnode)
inkcherry
committed
ccl: overlap HierAllGather param-contiguous local-block scatter with the inter-node RDMA ring (recover ring||gather overlap zero-copy lost to serial path; lever to beat RCCL xnode)
inkcherry
committed
ccl: gate HierAllGather param-contiguous overlap OFF by default (HSA fault under FSDP; standalone bit-exact)
inkcherry
committed
ccl: standalone bit-exact test for intra param-contiguous (HSDP winning path); catches non-bit-exact scatter (num_blocks=1/W=G) behind the +17.6% HSDP result
inkcherry
committed
ccl: system-scope fences in fused subgroup param-contiguous scatter + register-once regression test (isolates num_blocks=1 scatter bug; num_blocks=N cross-node path re-confirmed bit-exact)
inkcherry
committed
ccl: drop ineffective threadfence_system from fused subgroup param-contiguous kernel (align to proven flat kernel); add receiver-slot diagnostic to intra test
inkcherry
committed
ccl: gated measurement to skip inter-ring entry barrier — proves the per-op cross-node barrier is the FSDP gap (SDMA 102.9->112.1, beats RCCL 105.6 when removed); justifies generation-counter barri…
inkcherry
committed
ccl: kill per-AG-call D2H syncs in param-contiguous zero-copy path (.item()/.tolist() drained the pipeline every call, destroying AG<->backward overlap under FSDP); cache u32 split tensors
inkcherry
committed
ccl: MORI_HIER_NO_ALL_BARRIER measurement switch (strip entry+finish+intra barriers) — decisive A/B falsifies barrier-skew: all-barrier removal gives 0% recovery (111.98->111.79), gap is not per-op…
inkcherry
committed
ccl: FIX the 17-turn num_blocks=1 param-contiguous scatter 'bug' — it was an undersized-output IPC-registration artifact, NOT a concurrent-put race
inkcherry
committed
ccl: fix overlap zero-copy spin-deadlock — serialize side/main param-contiguous scatters (shared per-groupPos flag slot)
inkcherry
committed
ccl: fix overlap zero-copy loss-drift race — record_stream input/output on the side stream so the caching allocator does not recycle FSDP buffers mid-scatter
inkcherry
committed
ccl: MORI_HIER_REG_STATS diagnostic — count output-ptr registration churn per AG call (each change = a cross-node ShmemSymmetric register/deregister collective that cannot overlap; candidate for th…
inkcherry
committed
ccl: MORI_HIER_DEBUG_SYNC isolation switch proves the FSDP copy-out loss-nondeterminism is an async completion-capture bug — forced stream.synchronize() at op return gives bit-exact loss==native (1…
inkcherry
committed
ccl: fix FSDP copy-out loss-nondeterminism — SDMA gather receiver spins on the peer-completion flag with a SYSTEM-scope acquire (AtomicLoadSeqCstSystem) + __threadfence_system instead of an AGENT-s…
inkcherry
committed
ccl: CU-domain Phase-B copy-OUT (transit out_ accessor + single torch elementwise copy-out) to test the copy-engine<->CU coherence root cause for FSDP loss-drift; both CU-write-output and CU-read-t…
inkcherry
committed

Commits on Jul 4, 2026

Commits on Jul 8, 2026

ccl(hier): crown fill + chunk-pipelined DEEP_PIPE + host-proxy inter leg; MI300X/mlx5 port (bit-exact, >native at steady state)
inkcherry
committed

Commits on Jul 9, 2026

ccl(hier): DEEP_PIPE=16 -> all-size UT parity (>=1.0x) + reassembly SDMA-engine auto-fan
inkcherry
committed

Commits on Jul 13, 2026

Commits on Jul 14, 2026

Commits on Jul 15, 2026

Commits on Jul 17, 2026

Commits on Jul 20, 2026

Commits on Jul 21, 2026

Commits on Jul 22, 2026

Commits on Jul 23, 2026

Commits on Jul 27, 2026

Commits on Jul 30, 2026