[TENT] Gate queued dispatch with receiver credits

[catyans]

Summary

Stacked on #2861. Please review that prerequisite first.

This draft integrates the bounded receiver-credit model with the TENT runtime queue:

• snapshot CreditKey, epoch, and CreditCharge when a queue owner is admitted;
• atomically reserve DataBytes + RequestSlots immediately before dispatch;
• commit only after transport/staging submission succeeds;
• roll back the exact reservation token when submission fails;
• defer an owner without transport submission when credit is exhausted;
• reject stale queued snapshots after receiver session/epoch changes.

The integration is explicitly opt-in and defaults off:

enable_runtime_queue=true
receiver_credit/enabled=true
receiver_credit/default_qos_class=0

The control-plane owner installs shared context/ledger state through
installReceiverCreditDispatch(). Enabling the feature without installing that
state fails closed before queue tasks are published.

Change shape

• Base prerequisite: [TENT] Add bounded receiver-credit control protocol model #2861 (d2a3333)
• This change: one squashed commit (8160d44)
• Diff relative to [TENT] Add bounded receiver-credit control protocol model #2861: 11 files, +754/-7
• Default behavior: unchanged (receiver_credit/enabled=false)

Cluster test environment

Tests ran remotely on the Lingjun H20 cluster, not on the local workstation.

Item	Value
Main node	lingjun-100, container codex_mooncake_pr_eval
Kernel	Linux 5.10.134-16.3.al8.x86_64
CPU / memory	Intel Xeon Platinum 8575C, 192 visible vCPUs, 2.0 TiB RAM
GPU	8 x NVIDIA H20-3e, 143771 MiB each
Driver	580.105.08
Compiler / CMake	GCC 13.3.0 / CMake 3.28.3
Builds	Release, ASan+LSan Debug, TSan, CUDA (USE_CUDA=ON)

No public IPs, credentials, or SSH material are included here. The remote tree
was updated with Git; no local-to-remote scp was used.

Results

Final tree

Mode	Suites	Cases	Elapsed	Result	Findings
Release	3	55	0.58 s	55/55	none
ASan + LSan	3	55	1.75 s	55/55	0 memory errors, 0 leaks
TSan	3	55	1.06 s	55/55	0 data races
CUDA build	3	55	1.40 s	55/55	none

Repeated stress

Campaign	Cases/round	Rounds	Executions	Passed	Failed
Admission queue state machine	31	1000	31,000	31,000	0
Atomic dispatch gate	4	1000	4,000	4,000	0
Formal API E2E suite	17	1000	17,000	17,000	0
Opt-in/config API boundary	4	1000	4,000	4,000	0
Review concurrency regressions	2	1000	2,000	2,000	0
Total			58,000	58,000	0

xychart-beta
    title "Repeated test executions (all passed)"
    x-axis [Admission, Gate, E2E, OptInAPI]
    y-axis "case executions" 0 --> 32000
    bar [31000, 4000, 17000, 4000]

Loading

Credit and submit invariants

Scenario	Initial bytes/slots	Final bytes/slots	Transport submits	Outcome
Successful submit	4096/1	0/0	1	COMPLETED
Injected submit failure	4096/1	4096/1	1 failed call	FAILED; exact rollback
Exhausted credit	0/0	0/0	0	deferred, retryable
Stale snapshot after restart	old session 0/0	old session 0/0	0	rejected
Two owners, one-owner grant	4096/1	0/0	1 before new grant	no overdraw
Cumulative grant becomes 8192/2	+4096/+1 available	0/0	2 total	both COMPLETED

Failures found during testing

• Release caught a CHECK_STATUS(status) macro name collision that could return
  an uninitialized status; renamed the outer variable and repeated the suite.
• Stress iteration 507 exposed a timing-invalid assertion in the progress-worker
  test. The worker may dispatch owner 2 before the submitter observes owner 1;
  the test now checks timing-independent final invariants.
• The multi-owner test initially treated grants as deltas. [TENT] Add bounded receiver-credit control protocol model #2861 deliberately
  defines monotonic cumulative grants, so the second total is correctly 8192/2.
• The prototype held context/ledger references. The formal API now retains shared
  ownership, validated under ASan/LSan.

Conclusion and limits

The data supports the intended claim: receiver capacity becomes a hard,
atomic dispatch constraint, submission failure does not leak credit, stale
receiver identity cannot be charged, and multiple queued owners cannot overdraw
the ledger.

This PR does not claim TCP/RDMA bandwidth or TTFT improvement. Deterministic
submission and failure assertions use a fake RDMA transport; the CUDA build
checks H20/CUDA compatibility. Real cross-host performance testing belongs after
the control RPC/telemetry producer is connected in a follow-up.

Checklist

• Default-off compatibility
• Fail-closed opt-in configuration
• Exact multi-resource rollback
• Exhaustion/defer/retry
• Receiver restart/stale snapshot
• Multi-owner no-overdraw
• Release stress
• ASan/LSan
• TSan
• CUDA build
• Maintainer agreement on the [RFC]: TENT Receiver-Advertised Credits and Upstream Backpressure #2849 control-path direction

[gemini-code-assist]

Code Review

This pull request introduces a receiver-side credit-based flow control mechanism to the transfer engine, adding components for credit ledger management, capability negotiation, control-plane message codecs, and a dispatch gate that integrates with the runtime admission queue. Feedback on the changes highlights a performance bottleneck and potential state leak in TransferEngineImpl::refillDispatchWindow where the dispatch budget is restricted to one owner when the credit gate is enabled; a suggestion is provided to allow full concurrency while safely deferring remaining owners if a dispatch fails.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!