[RFC]: TENT Receiver-Advertised Credits and Upstream Backpressure

[catyans]

Changes proposed

Summary

This RFC proposes a receiver-advertised credit layer for TENT. It allows the node that will actually receive payload data to reserve and advertise bounded capacity before a sender dispatches work into a transport or remote staging path.

The proposal is a Phase 3 extension of #2132. It builds on the existing local runtime queue and dispatch window; it does not replace transport execution, RDMA congestion control, or the priority/byte-deficit work in #2655.

The key distinction is:

• DispatchByteCredit: sender-local scheduler accounting, as discussed in [TENT] add priority and byte credits to runtime queue dispatch #2655.
• ReceiverAdvertisedCredit: receiver-owned resource reservation shared across nodes, proposed here.

The first implementation PR should add only the private credit state model, versioned message schema, and deterministic fault tests. It should not modify Admission scheduling or enable distributed credits by default.

This is the detailed follow-up for the receiver-credit part of #2132 Phase 3. It does not replace #2132's local queue design, and it deliberately does not duplicate #2655's sender-local dispatch byte credits.

Motivation

The current runtime queue can bound local outstanding owners/bytes and the amount of work handed to transports:

sender submit
  -> local queue admission
  -> local scheduling / dispatch window
  -> Transport::submitTransferTasks() or ProxyManager::submit()

This is necessary but insufficient for many-to-one traffic. With 16 Prefill senders targeting one Decode receiver, every sender can independently observe local capacity while the receiver has exhausted one of these resources:

• bytes that may safely be in flight;
• logical request/completion slots;
• remote staging chunks;
• consumer readiness or Decode-side commit capacity;
• receiver CPU/QP/CQ progress capacity.

PFC/ECN/DCQCN protect the fabric but do not express application buffer or consumer readiness. One-sided RDMA WRITE also does not synchronize the initiator with the remote consumer. TENT therefore needs an explicit receiver-side resource contract before remote dispatch.

Goals

1. Bound aggregate many-to-one work by receiver capacity.
2. Apply backpressure before work is buried in transport or ProxyManager queues.
3. Support Direct and Staged plans with different resource charges.
4. Make duplicate, reordered, stale, and post-restart credit updates safe.
5. Preserve old-peer and default-disabled behavior.
6. Expose structured backpressure to SGLang/vLLM without blocking scheduler threads.

Expected Benefits and Claims Boundary

If the implementation satisfies the invariants and acceptance criteria in this RFC, it provides the following enforceable properties. These properties do not depend on an assumed throughput or latency improvement:

1. Receiver-capacity safety for modeled resources. The sum of outstanding reservations across senders cannot exceed the receiver's configured capacity for DATA_BYTES, REQUEST_SLOTS, STAGING_SLOTS, and CONSUMER_SLOTS. A resource that is not modeled or charged is not covered by this guarantee.
2. Early, bounded backpressure. Work that lacks receiver capacity remains in the sender's bounded runtime queue instead of being buried in transport, ProxyManager, or remote-staging queues. This converts receiver overload into explicit credit stall rather than unbounded downstream accumulation.
3. Atomic multi-resource eligibility. A transfer is dispatched only after its complete charge vector is reserved. Partial reservations, such as bytes without a staging or consumer slot, are never exposed.
4. Staged-path progress isolation. Separately bounded control and staging-internal capacity prevents user payload from consuming the resources required to return credit or complete the inner work that releases an outer reservation.
5. Replay and restart safety. Duplicate, reordered, decreasing, overflowing, stale-epoch, and pre-restart updates cannot mint additional usable credit. Old sessions are fenced before their resources are reused.
6. Control-plane recovery at zero data credit. Credit, heartbeat, reset, cancel, and health traffic can make progress through a bounded control budget even when payload dispatch is stopped.
7. Actionable pressure attribution. The runtime can distinguish receiver-credit, local-queue, staging, and transport pressure. A later connector can use that typed signal to pause, retry, select another Decode node, fallback, or recompute instead of treating every condition as generic TooManyRequests.
8. Incremental deployment. Default-disabled behavior remains unchanged, capability negotiation supports mixed-version peers, and required mode fails or falls back explicitly rather than treating missing credit as unlimited capacity.

This RFC does not claim a numerical P99/TTFT reduction, higher total throughput, better SLO attainment, or tenant fairness before the real-cluster matrix below is completed. Those are experimental outcomes, not protocol invariants. Tenant guarantees, caps, weights, borrowing, and authorization also require a separate QoS-contract/policing design. Healthy-path utilization must be measured against a correctly tuned fixed window, while overload tests must report receiver occupancy, stalls, failures, and data integrity in addition to latency and throughput.

Non-goals

• A second WDRR/priority scheduler.
• Per-RTT congestion control or replacement of DCQCN/HPCC/ECN/PFC.
• A cluster-wide per-transfer executor.
• Revoking already-dispatched work.
• Solving tenant authentication or QoS contract policy in this RFC.
• Reusing Status::TooManyRequests without identifying the pressure source.

Terminology and Direction

receiver means the node whose memory/resource is receiving payload, not necessarily the API target:

Plan	Data receiver	Remote receiver credit?
RDMA WRITE to peer	remote target	yes
remote staging/delegate	remote staging/consumer	yes
RDMA READ from peer	local initiator	no; apply local resource limits
local SHM/NVLink	local process/host	local accounting only

A plan that does not consume remote receiver resources must not be stalled by an unrelated remote credit ledger.

Design Principles

1. A grant is a reservation, not an estimate

The receiver may use telemetry to decide how much capacity to reserve, but advertised credit itself means the receiver has committed that resource to the peer/class. Unknown telemetry must not create credit.

2. Absolute cumulative grants within an epoch

Delta grants can be double-applied after retransmission or reordering. Instead, for every credit key the receiver advertises monotonically increasing cumulative totals:

available(resource) = grant_total(resource) - consumed_total(resource)

The sender accepts an update only when:

• session_id and epoch match the active peer session;
• sequence is newer than the last accepted update;
• every cumulative grant is non-decreasing;
• subtraction and addition pass overflow/underflow checks.

Duplicate and old updates become no-ops.

3. Credits are non-revocable inside a live session

The receiver can stop granting more credit but cannot safely retract credit that the sender might already have consumed. Unused credit is returned explicitly or reclaimed after the peer session is fenced and old QPs/inflight operations are drained or failed.

The first version should keep initial grants small rather than implement unsafe timer-only reclamation.

4. Control traffic never consumes data credit

Credit update, usage, return, heartbeat, reset, cancel, and health messages use a bounded control budget. A zero data window must not prevent the message that reopens the window.

Credit Key and Resource Model

struct CreditKey {
    PeerSessionId receiver_session;
    PeerId sender_peer;
    uint32_t qos_class;
};

enum class CreditResource : uint16_t {
    DATA_BYTES = 1,
    REQUEST_SLOTS = 2,
    STAGING_SLOTS = 3,
    CONSUMER_SLOTS = 4,
};

struct CreditAmount {
    CreditResource resource;
    uint64_t grant_total;
};

The receiver owns a global capacity allocator. Per-peer/per-class grants are carved from that allocator so that the sum of outstanding reservations cannot exceed configured physical capacity.

An Execution Plan produces a charge vector:

struct CreditCharge {
    std::vector<std::pair<CreditResource, uint64_t>> resources;
};

Examples:

• Direct WRITE: DATA_BYTES + REQUEST_SLOTS + CONSUMER_SLOTS.
• Remote staged WRITE: the above plus STAGING_SLOTS.
• Local RDMA READ receive: local resource accounting, no remote ledger.

Proposed Wire Schema

The exact serialization can change, but the semantic fields must remain explicit and versioned:

struct ReceiverCreditUpdateV1 {
    uint16_t schema_version{1};
    uint16_t flags{0};
    uint32_t qos_class{0};

    UInt128 receiver_session_id;
    uint64_t epoch{0};
    uint64_t sequence{0};

    // Freshness hint, evaluated from the sender's local receive time.
    // It is not permission for the receiver to reclaim resources.
    uint32_t freshness_ttl_ms{0};

    std::vector<CreditAmount> grants;
};

struct SenderCreditUsageV1 {
    uint16_t schema_version{1};
    UInt128 receiver_session_id;
    uint64_t epoch{0};
    uint64_t sequence{0};
    uint32_t qos_class{0};
    std::vector<CreditAmount> consumed_totals;
    std::vector<CreditAmount> pending_demand;
};

Capability negotiation must happen during peer/bootstrap setup. The negotiated state is one of:

• Unsupported: peer has no receiver-credit capability.
• OptionalLegacy: configured rollout allows legacy bounded local dispatch.
• Required: unsupported or stale credit blocks/fallbacks instead of assuming infinity.
• Active(version): typed updates are accepted.

State Machine

Disabled ------------------------------> Legacy
   |
   v
Negotiating -- unsupported -----------> Legacy or Failed(required)
   |
   +-- compatible --------------------> Active
                                          |
                     stale/disconnect ----+----> Stale
                                          |       |
                                          |       +-- newer update -> Active
                                          |
                     new session/epoch ---+----> Fencing -> Active(new epoch)

Rules:

1. Active: new dispatch may reserve credit.
2. Stale: stop new credit-gated dispatch; existing transport-owned work continues to terminal state.
3. New epoch: invalidate unused old ledger state, but do not reuse old target resources until old QPs/inflight work are fenced.
4. Credit violation: fail/quarantine the peer and emit a protocol error metric; do not silently overdraw.
5. Counter overflow, unknown required resource, malformed schema, or decreasing cumulative total: reject the update.

Dispatch Integration

Receiver credit belongs at dispatch eligibility, not queue admission:

TryAdmit(local owners/bytes/lifetime)
  -> PickByLocalPolicy(priority/deadline/deficit)
  -> CreditLedger::TryReserve(plan.credit_charge)
       success: hand off to transport/proxy
       blocked: keep queued; record stall; request/update demand

The current pickForDispatch() transitions selected owners to Dispatching. A production integration therefore needs one agreed boundary:

1. an eligibility callback/filter before state transition; or
2. a safe requeueForDispatch() preserving original enqueue/aging time.

This should be coordinated with #2655 rather than implemented in the first credit PR.

Local dispatch capacity and remote credit are both required:

effective dispatch = local window
                   AND local scheduler eligibility
                   AND receiver credit for every required resource

Staging and Deadlock Avoidance

Credits should shape the physical inner transfers generated by ProxyManager, not only the outer user-visible staged request, matching #2132.

Staging-internal/control work needs reserved capacity that user payload cannot consume. Otherwise:

outer staged work reserves receiver capacity
  -> inner staging transfer waits for the same exhausted class
  -> outer work cannot complete and release capacity

The resource allocator must distinguish user payload, staging-internal progress, and control messages. Bypass is allowed only when separately bounded and accounted.

Control Transport

TENT's existing notification QP is a useful prototype path because it is separate from data QPs and already has configurable SL/TC. It is not yet the protocol contract:

• current notifications are string name/msg payloads;
• peer/session identity must be bound to the endpoint, not trusted from payload text;
• publisher enqueue must be nonblocking;
• updates need periodic refresh and failure metrics;
• control WR/CQ slots must remain available under payload saturation.

The first wire PR may use a fake transport and ControlService for tests. Using the notification QP in production should be a separate reviewed step.

Control transport alternatives

Alternative	Advantages	Risks / limitations	Recommendation
Extend BootstrapDesc	Reuses peer setup and capability negotiation	Bootstrap is not a recurring update channel; reconnect semantics become overloaded	Capability/version negotiation only
Typed ControlService RPC	Explicit schema, identity and error reporting; easy to fault-inject	Requires a bounded async client/server path and independent progress guarantees	Preferred first production transport
Typed notification message	Reuses the existing notification QP and configured control SL/TC	Current notification payload is string-based and its retry/identity contract is insufficient	Follow-up after typed framing and reserved WR/CQ budget
Piggyback on data completion	Low message count	Credit cannot recover when payload is stopped; creates a circular dependency	Do not use as the only update path

The protocol model must remain independent of the selected transport. PR 1 and PR 2 should test through a fake bounded transport so that transport selection cannot weaken epoch, ordering or capacity invariants.

Backpressure API

TooManyRequests currently conflates queue admission, batch capacity, transport pressure, and staging exhaustion. Receiver-credit stall should expose structured information:

struct BackpressureInfo {
    enum class Source { RECEIVER_CREDIT, LOCAL_QUEUE, STAGING, TRANSPORT };
    Source source;
    PeerId peer;
    uint32_t qos_class;
    std::vector<CreditResource> exhausted;
    uint64_t retry_after_us;
    bool fallback_allowed;
};

Initially this can remain internal and observable through metrics. A later connector API can surface BACKPRESSURE/RETRY_AFTER so SGLang/vLLM can pause submission, choose another Decode node, fallback, or recompute locally.

Metrics

Per bounded peer/class labels:

• granted/consumed/available by resource;
• credit-stalled owners/bytes and stall duration;
• update age, sequence gaps, duplicate/stale updates;
• epoch resets and fenced inflight work;
• explicit returns and idle reserved credit;
• protocol violations and legacy fallbacks;
• receiver global committed/free capacity.

Request IDs and unbounded tenant IDs belong in traces/sampling, not metric labels.

Compatibility and Rollout

• Default: disabled; current behavior unchanged.
• Optional mode: negotiate capability; legacy peers retain current bounded local dispatch with a visible metric.
• Required mode: unsupported/stale peers cannot enter the gated path; use configured fallback or fail explicitly.
• No existing public Request field changes in the first PR.
• Disablement is a config rollback; it does not require wire downgrade during an active session.

Proposed Code Placement and Concurrency Model

The exact file names are subject to maintainer feedback, but the ownership boundary should remain explicit:

Component	Tentative placement	Responsibility
CreditKey, CreditCharge, wire-neutral update types	tent/include/tent/runtime/receiver_credit.h	Private types and checked arithmetic
SenderCreditLedger	tent/src/runtime/receiver_credit.cpp	Apply cumulative grants and atomically reserve a complete charge vector
ReceiverCreditAllocator	tent/src/runtime/receiver_credit.cpp	Carve bounded grants from global receiver resources
Versioned wire codec	tent/include/tent/control/receiver_credit_protocol.h	Validate schema/version/length before constructing runtime types
Dispatch eligibility adapter	adjacent to Admission/runtime queue implementation	Gate transport handoff without changing queue admission semantics
Metrics	existing bounded TENT metrics registry	Stall, stale update, protocol violation and capacity gauges

Concurrency rules:

1. Control callbacks validate framing and enqueue updates into a bounded queue; they do not mutate Admission state or call a transport while holding a credit lock.
2. The runtime/dispatch owner drains updates and applies each credit-key mutation serially, or under a dedicated ledger shard lock if the current thread model requires it.
3. Multi-resource reserve is all-or-nothing under one ledger critical section. A partial reservation is never externally visible.
4. Receiver capacity allocation and resource release use the same ownership domain. Grant publication happens after reservation is committed.
5. No credit mutex may be held across transport submission, user callbacks, RPC, notification send, or blocking wait.
6. Completion only reports receiver resource release; it must not mint sender-side credit locally.

Memory is bounded by configured limits, not request count:

ledger entries <= max_peers * max_qos_classes * negotiated_resources
queued control updates <= control_queue_capacity
wire resources per update <= max_resources_per_update

Unknown peers/classes/resources, oversized updates and capacity multiplication overflow are rejected before allocation. Request IDs and tenant strings are not retained as unbounded ledger keys.

Security and Abuse Boundaries

• A sender cannot self-assign a QoS class or increase a grant; peer identity and authorized class come from the negotiated endpoint/policy context.
• A receiver never grants more than committed capacity, even if demand reports are forged or duplicated.
• Per-peer demand, update frequency and control-queue occupancy are bounded to prevent control-plane memory or CPU exhaustion.
• Malformed, decreasing, overflowing or wrong-epoch updates fail closed and increment a bounded protocol-error metric.
• A disconnected sender cannot retain reservations indefinitely: explicit return is preferred, while session fencing provides the safe reclamation boundary.
• Optional legacy mode is a rollout policy, not an implicit unlimited grant. Required mode must reject or take an explicit configured fallback.
• Debug traces may contain peer/request identifiers, but metrics must not use unbounded identifiers as labels.

Existing PG Mechanism: Cluster Evidence and Gaps

This RFC does not yet have a TENT implementation to benchmark. We therefore used Mooncake PG #1971 only as mechanism evidence, comparing upstream commit 4188e3a with the parent of #1971 (c3bafbed) on an 8×H20 host with four ~392 Gbit/s RDMA rails.

• The current receiver-driven PG passed 8/8 CPU/CUDA P2P functional tests; the pre-[PG] Refactor P2PProxy with shared chunk pools and receiver-driven credit-based flow control #1971 baseline passed 4/8.
• A single-rail isolation matrix passed 15/15 runs with wrong=0.
• The same high-risk cases on four rails had 5/15 abnormal runs: two silent data-corruption results and three P2P operation failures.
• A valid 32/64/128 MiB pool/chunk matrix passed 9/10 runs; one 64 MiB pool + 8 MiB chunk run lost progress and hit a 150-second watchdog.
• CUDA failed-rank and replacement-recovery tests both passed.
• A two-node run published the correct RDMA IPs but could not establish PG connectivity because the peer store value was shorter than sizeof(SegmentInfo); both sides timed out.

These results support receiver-owned credits as a correctness direction, but they do not validate TENT throughput, SLO, or isolation. Before dispatch gating is enabled, the acceptance gate must include strict data validation across one and multiple rails, bounded pool liveness, and a versioned cross-node metadata handshake. Full details are recorded in cluster-validation.md beside this draft.

Implementation Plan

PR 1 — Credit model and protocol invariants

• private CreditKey, CreditLedger, CreditUpdateV1;
• deterministic tests for duplicate, reorder, sequence gaps, stale epoch, reset, overflow, underflow, multi-resource atomic reserve and return;
• no background thread, network messages, Admission changes, or behavior change.

PR 2 — Capability and control transport

• version negotiation and optional/required rollout mode;
• fake transport fault injection: loss, delay, duplication, reordering, disconnect;
• endpoint-bound peer identity;
• bounded nonblocking publisher and freshness metrics.

PR 3 — Runtime dispatch gating

• coordinate eligibility/requeue contract with [TENT] add priority and byte credits to runtime queue dispatch #2655;
• reserve all resource dimensions atomically before transport handoff;
• preserve original enqueue time while credit-stalled;
• direct WRITE first; RDMA READ remains local-only.

PR 4 — Staging and connector backpressure

• remote staging slots and consumer readiness;
• reserved staging-internal/control class;
• SGLang/vLLM typed backpressure and retry/fallback behavior.

Test and Experiment Matrix

Deterministic protocol tests

• duplicate/reordered/lost update;
• decreasing totals and integer overflow;
• sender/receiver restart and epoch mismatch;
• old completion after new epoch;
• control disconnect while payload is inflight;
• atomic multi-resource reservation failure;
• explicit unused-credit return;
• unsupported old peer in optional/required modes.

Real cluster experiments

Dimension	Values
Senders	1 / 4 / 16
Path	Direct / Staged / mixed
Mode	disabled / fixed local window / receiver credit
Receiver condition	normal / consumer slowdown / stage exhaustion / restart
Workload	foreground KV + checkpoint/background

Report:

• foreground P50/P99 and SLO attainment;
• background throughput and total utilization;
• receiver staging/CQ/queue peak occupancy;
• credit stall time and update overhead;
• failure/fallback/degraded-action counts;
• no data corruption and no unbounded receiver growth.

Acceptance Criteria

1. Aggregate reserved receiver resources never exceed configured capacity.
2. A sender never dispatches a required remote plan without all credit dimensions.
3. Duplicate, reordered, and stale updates never increase available credit.
4. Restarted peers cannot reuse old credit or old allocation generation.
5. Zero data credit cannot block control updates.
6. Under 1/4/16-sender incast, receiver queue/staging growth remains bounded.
7. Healthy-path utilization does not regress materially versus a correctly tuned fixed window.
8. Default-disabled and old-peer behavior remain compatible.

Related Work

• [RFC]: TENT Local Transfer Admission Queue #2132 — local runtime admission queue and Phase 3 pressure/credits boundary.
• [TENT] add priority and byte credits to runtime queue dispatch #2655 — sender-local priority lanes and byte deficit; deliberately not duplicated here.
• [PG] Refactor P2PProxy with shared chunk pools and receiver-driven credit-based flow control #1971 — Mooncake PG receiver-driven per-chunk credit protocol and epoch recovery.
• [TransferEngine] Apply configured SL/TC to the TENT notification QP #2526 — configured SL/TC on TENT notification QP.
• RFC 9000 / RFC 9113 / RFC 9893 — receiver-owned flow-control window semantics.
• Homa / SIRD — receiver-driven scheduling and the need to account for shared bottlenecks.

Open Questions for Maintainers

1. Should the first wire transport extend BootstrapDesc, add a ControlService RPC, or define a typed notification protocol?
2. Should dispatch gating use an eligibility callback or a requeue transition in the runtime queue?
3. Which first path is most useful for production validation: direct RDMA WRITE or remote staging WRITE?
4. Should optional legacy mode remain available long term, or only during rollout?
5. Which component owns consumer readiness: TENT, Mooncake Store, or the SGLang/vLLM connector?

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation

[github-actions]

Thanks for opening this issue, @catyans!

Field	Value
Issue	#2849
GitHub user ID	18214026
Reporter	@catyans

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

[catyans]

Final consolidated receiver-credit experiment report

This comment supersedes all earlier experiment comments on this RFC. The benchmark-only branch is receiver-credit-experiment, at commit 8e01c8d5. It does not modify production Admission or dispatch.

What was corrected

The receiver JSON p99_us starts at a multi-slot lease demand and ends as individual slots complete. It is a lease-lifecycle / receiver-backlog metric, not a per-transfer end-to-end latency metric, so it must not be compared directly with fixed per-transfer demand latency.

Commit 8e01c8d5 moves the sender timer before credit acquisition. The sender P99 below therefore covers credit wait + request construction + RDMA transfer. Earlier interpretations of receiver p99_us as a per-transfer SLO metric are superseded.

Direct capacity-safety evidence

On the backup RDMA hosts, sender-local fixed behavior exceeded receiver capacity, while receiver-owned credits enforced it:

Condition	Configured capacity	Fixed peak / violations	Credit peak / violations
4-sender incast	8 slots	11 / 657	8 / 0
16-sender incast	8 slots	22 / 14,657	8 / 0
16 senders, 5 ms consumer slowdown	64 slots	81 / 14,103	64 / 0

The restored main cluster also reproduced the risk after reducing benchmark control polling from 50 us to the retained 10 us setting: one 16-sender fixed run reached 16/8 slots with 9 violations. Every corresponding credit run remained within capacity.

Final main-cluster 16-sender A/B

Five interleaved repetitions used 16 senders, an 8-slot receiver cap, 4 MiB transfers, lease batch 2, and the corrected sender end-to-end timer:

Metric (median of 5)	Fixed	Receiver credit	Delta
Receiver throughput	7.363 GB/s	7.393 GB/s	+0.41%
Mean of 16 sender P99s	119.200 us	119.675 us	+0.40%
Worst-sender P99 per run	128.0 us	122.0 us	-4.69%
Sender Jain fairness	0.999896	0.999615	-0.000281

All credit runs completed all offered operations with zero data errors, zero capacity violations, zero invalid releases, and zero final occupancy.

Main-cluster throughput coverage

The broader three-repetition matrix produced these receiver-throughput medians:

Condition	Fixed	Receiver credit	Delta
1 sender, healthy, 1 rail	35.219 GB/s	35.756 GB/s	+1.53%
4-sender incast, 8-slot cap	16.269 GB/s	16.186 GB/s	-0.51%
16 senders, 5 ms consumer slowdown	6.720 GB/s	7.365 GB/s	+9.59%
4 senders / 4 RDMA rails	28.601 GB/s	28.778 GB/s	+0.62%

Three integrity repetitions completed 12,288 4 MiB WRITE→READ pairs with zero data errors.

Parameter selection

• batch 1 caused a 7.75% 16-sender throughput deficit in the initial main-cluster matrix;
• batch 2 removed that deficit while retaining near-perfect fairness;
• batch 4 increased variance and produced about 6.7k receiver stalls per run;
• batch 8 reduced median throughput and produced about 14k stalls per run;
• 1 us polling improved the lease-lifecycle metric by only about 3 us over 10 us while increasing busy-poll cost, so it was reverted.

The retained benchmark operating point is batch 2 with 10 us polling. Batch 2 is evidence for an operating point, not a proposed production constant; production should adapt grant size to capacity, active senders, and SLO pressure.

Verification and claims boundary

The focused C++ tests, Python result-verifier tests, JSONL mechanical checks, offered/completed equality, capacity guards, and integrity checks pass.

These results support the RFC's core claim: receiver-owned credits can enforce an aggregate resource bound without a material throughput, end-to-end P99, or fairness regression in the tested workload. They do not yet validate production restart fencing, replay/reorder handling, typed ControlService messages, STAGING_SLOTS / CONSUMER_SLOTS, connector backpressure, or SGLang/vLLM TTFT improvements.

[catyans]

Draft benchmark PR opened: #2859

It contains the cleaned, benchmark-only tebench harness used to validate this RFC: receiver byte/slot capacity tracking, fixed-vs-credit demand/grant/release experiments, fixed-count integrity runs, corrected end-to-end sender timing, versioned JSONL, and mechanical verification.

The PR deliberately excludes production Admission/dispatch integration and links back to this RFC for design discussion and experimental evidence.

[catyans]

Draft PR1 is now open: #2860. It is intentionally limited to the private cumulative-grant ledger, versioned semantic types, epoch/replay fencing, bounded state, atomic multi-resource reserve/rollback, and deterministic protocol/concurrency tests. It does not add a control transport, capability negotiation, Admission/dispatch integration, background threads, or default behavior changes. Cluster validation on two independent H20 nodes includes 2000 repeated Release runs total, TSan, ASan/LeakSanitizer, and a CUDA-enabled TENT runtime build. I will keep PR2 transport-neutral behind a fake bounded control channel until the control-path choice in this RFC is aligned.

[catyans]

PR1 #2860 is now Ready for Review with the full upstream matrix green and all modified coverable lines covered. The transport-neutral PR2 prototype has also passed two-node Release (1000 repeats per node), ASan/LSan, TSan, CUDA runtime build, bounded-inbox saturation, receiver restart fencing, and real cross-host mixed-version negotiation.

One production control-path decision is now blocking a clean PR2 interface:

1. Receiver push: the receiver pushes cumulative grants to the sender. This requires the negotiation request/update envelope to bind sender identity (the current CoroRpcAgent handler does not expose caller identity).
2. Sender pull: the sender polls by target/QoS/last-sequence and receives activation/update in the response. This naturally preserves target context but introduces polling semantics.

Both can reuse the validated capability, activation, epoch, bounded ingress, and peer-context models. Is push or pull preferred for TENT ControlService? I will keep the production RPC/default-off integration out of PR2 until this identity/control-flow choice is aligned, rather than baking a test-only constant into the interface.

[catyans]

Implementation progress update (2026-07-12)

This comment records the current implementation boundary for the RFC. The core
credit invariants and a sender-side dispatch prototype now exist, but the RFC is
not yet implemented end to end. In particular, production control transport,
receiver allocation/telemetry, staged resources, typed backpressure, and
connectors remain follow-up work.

PR stack

Stage	PR	State	Implemented boundary
Benchmark mechanism/evidence	#2859	Draft	Benchmark-only fixed-vs-credit capacity, integrity, and performance harness; no production dispatch changes
PR1: ledger/invariants	#2860	Ready for review; 26 successful checks	Cumulative grants, epoch/session fencing, bounded sender ledger, atomic multi-resource reserve/rollback, malformed/replay/overflow/concurrency tests
PR2: control model	#2861	Draft; retry CI has no failing checks, 2 jobs still running	Capability state/codec, activation, endpoint/QoS peer context, bounded nonblocking inbox, wire validation, fake control-channel fault tests, mixed-version harness
PR3: dispatch gate	#2862	Draft; CI running	Queued-owner identity/charge snapshot, pre-dispatch atomic reserve, exhaustion defer/retry, submit-success commit, exact submit-failure rollback, restart rejection, default-off opt-in API

PR3 is a single squashed delta (8160d44) over the PR2 head (d2a3333). The
upstream GitHub view temporarily includes prerequisite commits because a PR
cannot use a fork-only PR2 branch as its base; after the prerequisites merge,
the visible diff will collapse to the single PR3 commit.

What is implemented now

RFC property	Status	Evidence / qualification
Monotonic absolute cumulative grants	Implemented	Duplicate/old/decreasing/overflow updates cannot mint credit
Session/epoch replay and restart safety	Implemented in model and dispatch snapshot	Old queued snapshots fail before transport submission
Atomic charge-vector reserve	Implemented	All resources reserve or none reserve
Exact rollback on submission failure	Implemented	Reservation token restores exact bytes/slots; committed tokens cannot be rolled back
Backpressure before transport handoff	Implemented in PR3 prototype	Insufficient credit produces zero transport submissions and requeues the owner
Bounded multi-owner dispatch	Implemented for modeled sender ledger	Two owners with one-owner grant submit exactly one before the cumulative grant increases
Default-disabled behavior	Implemented	receiver_credit/enabled=false by default
Fail-closed opt-in setup	Implemented	Receiver credit requires runtime queue; enabled-but-uninstalled control state rejects before task publication
Context/ledger lifetime	Implemented	Formal installation API retains shared ownership
Configured default QoS	Implemented	Optional provider; otherwise receiver_credit/default_qos_class

Current opt-in surface:

enable_runtime_queue=true
receiver_credit/enabled=true
receiver_credit/default_qos_class=0

installReceiverCreditDispatch(contexts, ledger, qos_provider);

PR3 cluster validation

All validation was run remotely on the Lingjun H20 cluster, primarily on an
8 x H20-3e host (192 visible vCPUs, 2 TiB RAM, GCC 13.3, CMake 3.28.3,
NVIDIA driver 580.105.08). No workstation results are used as cluster evidence.

Mode / campaign	Executions	Passed	Failed	Notes
Final Release suite	55	55	0	3 suites, 0.58 s
Final ASan + LSan suite	55	55	0	0 memory errors, 0 leaks; 1.75 s
Final TSan suite	55	55	0	0 data races; 1.06 s
Final CUDA-enabled suite	55	55	0	USE_CUDA=ON; 1.40 s
Admission queue repeat	31,000	31,000	0	31 cases x 1000
Atomic dispatch gate repeat	4,000	4,000	0	4 cases x 1000
Formal API E2E repeat	17,000	17,000	0	17 cases x 1000
Opt-in/config API repeat	4,000	4,000	0	4 cases x 1000
Review concurrency regressions	2,000	2,000	0	2 cases x 1000
Repeated-test total	58,000	58,000	0	Excludes the four final one-shot build matrices

xychart-beta
    title "PR3 repeated test executions (all passed)"
    x-axis [Admission, Gate, E2E, OptInAPI]
    y-axis "case executions" 0 --> 32000
    bar [31000, 4000, 17000, 4000]

Loading

Key deterministic observations:

Scenario	Initial bytes/slots	Final bytes/slots	Transport submits	Result
Successful submit	4096/1	0/0	1	committed
Injected submit failure	4096/1	4096/1	1 failed call	exact rollback
Exhausted credit	0/0	0/0	0	deferred/retryable
Stale snapshot after restart	old session 0/0	old session 0/0	0	rejected
Two owners, cumulative grant 4096/1	4096/1	0/0	1	no overdraw
Cumulative grant increases to 8192/2	+4096/+1 usable	0/0	2 total	both complete

Important remaining work

RFC area	Status / next requirement
Production control transport	Not implemented. Need maintainer direction on receiver push vs sender pull, then a typed bounded ControlService path with endpoint-bound identity
Bootstrap negotiation wiring	Model/codec exists, but it is not connected to real peer bootstrap/session lifecycle
Receiver global capacity allocator	Not implemented. Production grants are not yet carved from a global receiver-owned capacity pool across senders/classes
Receiver telemetry -> grant production	Not implemented. Current updates are supplied by tests/harnesses
Plan-aware charging	Incomplete. PR3 currently demonstrates DATA_BYTES + REQUEST_SLOTS; production gating must classify remote Direct WRITE vs local/READ and avoid charging unrelated paths
CONSUMER_SLOTS	Type exists; no production execution-plan charge/release integration yet
STAGING_SLOTS / ProxyManager inner work	Not implemented. Belongs to the staged-path follow-up
Independent control budget	Model principle only; zero-data-credit recovery has not been proven through a production control channel
Structured BackpressureInfo and metrics	Not implemented. No typed connector-facing source/retry/fallback signal or production stall/capacity metric set yet
SGLang/vLLM connector behavior	Not implemented. Retry, alternate Decode selection, fallback, and recompute remain PR4 work
Production 1/4/16-sender acceptance matrix	Benchmark mechanism evidence exists, but the matrix has not been rerun through the production TENT control+dispatch path because that path is not connected yet

Revised next steps

1. Review/merge PR1 and align the control-flow question before productionizing PR2.
2. Implement the selected typed ControlService push/pull flow, endpoint-bound
   identity, bootstrap negotiation, and receiver global allocator.
3. Connect Direct remote WRITE plan charging first; explicitly leave RDMA READ
   and local SHM/NVLink under local accounting.
4. Add bounded metrics and internal structured pressure attribution.
5. Run the RFC's real 1/4/16-sender normal/slowdown/restart matrix through the
   connected production path.
6. Add staging/consumer resources and connector-facing backpressure in PR4.

Explicitly not implemented

The following items are not present in the production path today. Passing
PR1/PR2/PR3 tests must not be interpreted as completing them:

• A real receiver-to-sender credit update channel in ControlService.
• Maintainer-selected receiver-push or sender-pull control semantics.
• Endpoint-authenticated sender identity on production credit messages.
• Real bootstrap capability/version negotiation between TENT peers.
• Automatic activation/deactivation tied to connection/session lifecycle.
• A receiver-global allocator that proves the sum of grants across all
  senders and QoS classes is at most physical receiver capacity.
• Receiver telemetry sampling and policy that converts free capacity into
  committed cumulative grants.
• Sender usage/demand reports on a production wire path.
• Explicit unused-credit return on a production wire path.
• Safe receiver-side reclamation after disconnect, including QP/inflight
  fencing before old capacity is reused.
• Periodic refresh, heartbeat, update-age handling, and reconnect recovery
  through the real control transport.
• A separately reserved control WR/CQ/message budget proven to progress at
  zero data credit.
• Execution-plan classification that gates only remote receiver-consuming
  operations; the prototype has not finalized Direct WRITE vs READ/local
  accounting.
• Production CONSUMER_SLOTS charging and release.
• Production STAGING_SLOTS charging and release.
• ProxyManager inner-transfer credit ownership and deadlock avoidance.
• Separate bounded staging-internal/control credit classes.
• Typed BackpressureInfo with exhausted resources, peer, QoS, retry hint,
  and fallback permission.
• Receiver-credit stall duration, grant/consume/available, update-age,
  violation, fencing, and global-capacity production metrics.
• Protocol-violation quarantine/failure policy in the real peer path.
• Optional-legacy and required-mode production fallback behavior.
• SGLang/vLLM pause, retry, alternate Decode selection, fallback, or local
  recompute integration.
• Production-path 1/4/16-sender incast validation.
• Production-path consumer slowdown, staging exhaustion, disconnect, and
  restart validation.
• Healthy-path comparison against a tuned fixed window after real control
  and dispatch wiring.
• Production-path P50/P99, SLO, occupancy, stall overhead, failure counts,
  and data-integrity report.
• A demonstrated TENT TTFT, bandwidth, throughput, or fairness improvement.

RFC acceptance-criteria status

RFC acceptance criterion	Current status	Reason
1. Aggregate reserved receiver resources never exceed configured capacity	Not yet satisfied end to end	Benchmark and sender-ledger evidence exists, but production receiver-global allocation is absent
2. Required remote plan never dispatches without every credit dimension	Partially satisfied	Atomic gate works for modeled bytes/slots; plan classification, consumer and staging dimensions remain
3. Duplicate/reordered/stale updates never increase available credit	Satisfied in protocol model	PR1/PR2 deterministic tests; production transport wiring remains
4. Restarted peers cannot reuse old credit/allocation generation	Partially satisfied	Sender snapshot/session fencing is tested; receiver QP/inflight reclamation is not implemented
5. Zero data credit cannot block control updates	Not yet satisfied in production	Bounded inbox model exists; no independent production control transport/budget
6. Receiver growth remains bounded under 1/4/16-sender incast	Not yet validated on production path	Benchmark harness evidence is not the connected TENT implementation
7. Healthy utilization does not regress materially vs tuned fixed window	Not yet validated on production path	Requires real control transport and receiver allocator
8. Default-disabled and old-peer behavior remain compatible	Default-disabled implemented; old-peer rollout partial	Default-off PR3 tests pass; real bootstrap optional/required negotiation is not wired

Current claims boundary

The implemented stack now validates the credit arithmetic, replay/restart
fencing, bounded sender-side dispatch decision, and exact rollback behavior. It
does not yet establish end-to-end receiver-capacity safety in production,
because real receiver capacity allocation and the production update transport
are not connected. PR3 also does not claim a new TCP/RDMA throughput or TTFT
gain; those measurements become meaningful only after that end-to-end path is
complete.