Skip to content

[TENT] Add receiver-credit ledger model and protocol invariants#2860

Open
catyans wants to merge 4 commits into
kvcache-ai:mainfrom
catyans:feat/receiver-credit-model
Open

[TENT] Add receiver-credit ledger model and protocol invariants#2860
catyans wants to merge 4 commits into
kvcache-ai:mainfrom
catyans:feat/receiver-credit-model

Conversation

@catyans

@catyans catyans commented Jul 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

Implements PR 1 of #2849: the private receiver-credit state model and deterministic protocol invariants.

This PR adds:

  • private CreditKey, CreditResource, CreditCharge, and versioned ReceiverCreditUpdateV1 semantic types;
  • a bounded sender-side cumulative-grant ledger;
  • atomic multi-resource reservation and pre-dispatch rollback;
  • duplicate/reordered update handling and sequence-gap visibility;
  • session/epoch fencing, including activation-replay protection;
  • fail-closed validation for stale epochs, decreasing grants, unknown/duplicate resources, and rollback underflow;
  • deterministic concurrency and protocol tests.

Scope boundary

Intentionally excluded from this PR:

  • network/control messages;
  • capability negotiation;
  • Admission or dispatch integration;
  • background threads;
  • public Request API changes;
  • any default runtime behavior change.

The model is transport-neutral so the control path can be selected separately after maintainer feedback on #2849.

Invariants covered

  • Duplicate or reordered updates cannot mint usable credit.
  • Same-epoch activation replay is idempotent and cannot reset consumption.
  • Older activation/update epochs fail closed.
  • Only a newer epoch fences the previous ledger state.
  • A multi-resource charge is reserved completely or not at all.
  • Concurrent reservations cannot exceed the advertised grant.
  • Ledger state is bounded by a configured maximum entry count.

Cluster validation

Validated from clean remote git clone/fetch on the 4-node H20 lab; no local-to-remote file copy was used.

  • lingjun-100, GCC 13, Release: focused CTest passed.
  • lingjun-100: ctest --repeat until-fail:1000 passed.
  • lingjun-100: TSan passed with no reported data race.
  • lingjun-100: ASan + LeakSanitizer passed.
  • lingjun-100: CUDA-enabled tent_runtime build passed.
  • lingjun-102, independent clean clone/container: 1000 repeated runs passed.

The four-node RDMA throughput/P99 matrix belongs to the benchmark-only #2859 and later dispatch-gating PR because this PR deliberately has no data-path integration.

Test plan

cmake --build build --target receiver_credit_test
ctest --test-dir build -R receiver_credit_test --output-on-failure

Related: #2849

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the SenderCreditLedger class and associated data structures to manage receiver credits on the sender side, along with comprehensive unit tests. The feedback highlights opportunities to optimize the activate method by avoiding a redundant hash lookup when updating an existing entry, and notes the lack of a mechanism to clean up or deactivate stale sessions, which could lead to ledger capacity exhaustion over time.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread mooncake-transfer-engine/tent/src/runtime/receiver_credit.cpp
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@catyans

catyans commented Jul 11, 2026

Copy link
Copy Markdown
Collaborator Author

CI follow-up: the failed-job rerun completed successfully.

  • x86 builds: Python 3.10 and 3.12 passed, including full tests/coverage and wheel build.
  • ARM64 builds: Python 3.10 and 3.12 passed.
  • TENT CUDA on/off, MUSA, EFA/CUDA wheels, Ubuntu wheel E2E, and Docker passed.
  • The downstream Ascend Hixl Mooncake Store test also ran and passed.

The initial failures were runner infrastructure issues (disk exhaustion and sccache DNS/fail-fast), and neither reproduced. Workflow: https://github.com/kvcache-ai/Mooncake/actions/runs/29156606254

Keeping the PR in Draft while waiting for maintainer feedback on the protocol/scope boundary in #2849.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants