[TENT] Add bounded receiver-credit control protocol model#2861
[TENT] Add bounded receiver-credit control protocol model#2861catyans wants to merge 24 commits into
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a receiver credit flow control mechanism for the mooncake transfer engine, including the SenderCreditLedger for tracking resource grants and reservations, and control-plane components for capability negotiation, session activation, and message serialization. The feedback highlights a critical security issue in CreditPeerContextTable::activate where a session could be hijacked by a different sender peer if the epoch increases without validating the sender's identity. Additionally, it is recommended to use a 64-bit hash mixing constant in LookupKeyHash for consistency and improved hash distribution on 64-bit platforms.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Final consolidated results: data, charts, and conclusion
Executive conclusionThe receiver-credit control model passed the tested correctness, boundedness, restart, mixed-version, concurrency, and memory-safety gates. Under deliberately severe receiver slowdown, both loopback and cross-host runs eventually accepted 100% of 16,000 offered updates, produced zero invalid accepts, and ended with a credit-amplification ratio of exactly 1.0000. Queue saturation was observable and retryable rather than silent. This is sufficient evidence for reviewing the transport-neutral control model in this Draft. It is not evidence that production dispatch gating or TTFT is complete; production receiver-push versus sender-pull remains a design decision in #2849. 1. Validation scale
xychart-beta
title "Validation event volume"
x-axis ["Release runs (K)", "Restarts (K)", "Queue events (M)", "Malformed inputs (M)"]
y-axis "Count in stated unit" 0 --> 100
bar [2, 10, 32, 100]
2. Saturation resultsWorkload: 16 concurrent clients, 1,000 logical cumulative updates per client, bounded inbox capacity 64, drain batch 31, and 1 ms delay after each receiver drain.
xychart-beta
title "Retryable backpressure versus accepted updates"
x-axis ["Loopback", "Cross-host"]
y-axis "Events" 0 --> 70000
bar [66178, 53754]
line [16000, 16000]
Bars are retryable queue-full responses; the line is accepted logical updates. Even with 3.36-4.14 queue-full responses per accepted update, no logical update was lost. Raw receiver outputs: 3. Conservation invariants
4. Mixed-version behaviorflowchart LR
A[Capability offer] --> B{Peer supports v1?}
B -- yes: 2,1 --> C[Required -> Active v1]
B -- no: 2 only --> D{Rollout mode}
D -- Optional --> E[Legacy fallback]
D -- Required --> F[Failed / NotImplemented]
5. Fault and restart coverageDeterministic coverage includes loss, duplicate, reorder, sequence gaps, disconnect/reconnect, same-epoch activation replay, new-session restart, stale cleanup, capacity recovery, zero/full inbox, multi-producer drain, every truncation length, bad magic/schema/reserved fields, oversized messages, duplicate/unknown resources, wrong session/QoS/epoch, decreasing grants, and malformed fuzz. The peer-context stress ran 10,000 receiver restarts against 16 concurrent readers. Every snapshot preserved a matching 6. Final assessmentflowchart TD
P[PASS: PR2 control-model evidence] --> P1[Bounded wire and state]
P --> P2[Explicit retryable backpressure]
P --> P3[No logical loss]
P --> P4[No credit amplification]
P --> P5[Restart and mixed-version fencing]
R[Remaining production work] --> R1[Choose receiver-push or sender-pull]
R --> R2[Formal ControlService RPC]
R --> R3[Dispatch reserve and rollback]
R --> R4[Data-plane throughput, P99, TTFT]
Decision: keep #2861 as a stacked Draft for protocol review, dependent on #2860. Do not mark it Ready until #2860 merges, the diff shrinks, the upstream CI matrix is green, and the ControlService direction in #2849 is aligned. Data-plane performance claims remain in #2859/#2849, not this control-model PR. |
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
7227fd7 to
d2a3333
Compare
# Conflicts: # mooncake-transfer-engine/tent/tests/CMakeLists.txt
# Conflicts: # mooncake-transfer-engine/tent/tests/CMakeLists.txt
# Conflicts: # mooncake-transfer-engine/tent/tests/CMakeLists.txt
…t-control-model # Conflicts: # mooncake-transfer-engine/tent/src/runtime/receiver_credit.cpp # mooncake-transfer-engine/tent/tests/CMakeLists.txt # mooncake-transfer-engine/tent/tests/receiver_credit_test.cpp
|
Merged current upstream/main into this branch and resolved the receiver-credit file/test-list conflicts after #2860 landed. Resolution summary:
Cluster validation on
|
2 participants
Summary
Adds the transport-neutral control protocol model for receiver-owned credits proposed in #2849. This is a stacked Draft and depends on #2860; the overlapping ledger commits disappear from this diff after #2860 merges.
This PR adds:
(target_id, qos_class)peer-context table with restart fencing and exact cleanup;Scope boundary
This Draft intentionally does not allocate a production
ControlServiceRPC ID or choose receiver-push versus sender-pull. The currentCoroRpcAgenthandler does not expose caller identity, so receiver-push requires an explicit sender binding in the wire contract; sender-pull has different polling semantics. That choice is requested in #2849.Also excluded:
RequestAPI changes or priority-to-QoS mapping;Key invariants
Cluster validation
No local-to-remote file copy was used; cluster nodes fetched the branch directly.
tent_runtimebuild: PASS;accepted=16000 queue_full=53754 invalid=0 available=16000, all processes exit 0;{2,1}+ Required -> Active v1;{2}+ Optional -> Legacy;{2}+ Required -> Failed;all RPC servers report
DONE=0 calls=1 invalid=0.Test plan
Depends on #2860. Related RFC: #2849.