[RFC]: TENT QoS Contract Schema and Effective Policy Model

[catyans]

Summary

This RFC proposes a TENT QoS Contract schema that defines network-transfer QoS intent at the tenant -> intent -> request level.

The goal is to provide a stable control-plane contract for priority authorization, bandwidth caps, burst budgets, inflight limits, degraded actions, and future receiver-side credit allocation.

This RFC does not propose a full scheduler rewrite. The first implementation milestone only adds schema parsing, validation, effective-policy resolution, and explain/debug output. Runtime enforcement is intentionally split into follow-up milestones.

Motivation

TENT already has several QoS-related building blocks:

• request priority
• deadline-aware admission
• intent type
• transport policy selection
• QP pool / SL / TC mapping
• RDMA bandwidth telemetry
• benchmark-side QoS metrics

However, these mechanisms currently lack a stable contract that answers:

• which tenant or workload is allowed to request HIGH priority?
• what is the maximum impact of background traffic?
• what minimum share should foreground traffic receive under contention?
• how much burst is allowed before throttling?
• which degraded actions are allowed for each intent?
• how should receiver-side credits be allocated once receiver capacity becomes visible?

Without this contract, adding more scheduling algorithms can improve local behavior, but does not provide production-grade multi-tenant isolation.

Non-goals

This RFC does not attempt to:

• replace Mooncake Store tenant storage quota
• define object/cache capacity quota
• implement full weighted fair queuing in the first PR
• implement receiver-advertised credits in the first PR
• guarantee bandwidth in the first PR
• expose hardware-specific SL/TC details directly to applications
• require all upper-layer integrations to pass tenant identity immediately

Store tenant quota controls storage capacity. TENT QoS Contract controls network transfer behavior.

Receiver Credits expose current receiver capacity. QoS Contract defines how that capacity should be shared.

Terminology

• Tenant: an isolation domain such as user, workload, service, model group, or deployment namespace.
• Intent: business purpose of a transfer, for example foreground_get, background_prefetch, migration, checkpoint, weight_loading, staging_internal.
• Contract: the configured QoS rule for a tenant and intent.
• Effective Policy: the resolved policy attached to a request after applying defaults, tenant overrides, intent overrides, and validation.
• Enforcement: runtime behavior that consumes the effective policy, such as policing, admission control, credit allocation, or scheduling.
• Degraded Action: allowed fallback behavior when the preferred path cannot satisfy policy or deadline.

Proposed Schema

Example:

qos:
  version: 1

  defaults:
    priority: medium
    weight: 1
    max_inflight_bytes: 1GiB
    max_inflight_requests: 1024
    allowed_degraded_actions:
      - fallback_transport
      - reject

  tenants:
    - name: default
      intents:
        foreground_get:
          priority: high
          min_bandwidth_gbps: 40
          max_bandwidth_gbps: 100
          weight: 8
          burst_bytes: 2GiB
          max_inflight_bytes: 4GiB
          max_inflight_requests: 4096
          deadline_profile: interactive
          allowed_degraded_actions:
            - fallback_transport
            - local_recompute
            - reject

        background_prefetch:
          priority: low
          max_bandwidth_gbps: 20
          weight: 1
          burst_bytes: 512MiB
          max_inflight_bytes: 1GiB
          max_inflight_requests: 512
          deadline_profile: background
          allowed_degraded_actions:
            - delay
            - reject

        checkpoint:
          priority: low
          max_bandwidth_gbps: 10
          weight: 1
          burst_bytes: 256MiB
          allowed_degraded_actions:
            - delay
            - reject

Field Semantics

priority

The authorized scheduling priority for this tenant and intent.

Applications may request a priority, but the effective priority should be derived from the contract. A request should not be able to obtain HIGH priority merely by setting Request.priority = HIGH.

If a requested priority exceeds the authorized priority, the implementation should either:

• downgrade to the authorized priority, or
• fail closed if strict mode is enabled.

min_bandwidth_gbps

Minimum target bandwidth share under contention.

This is a service objective, not a hard guarantee in the first implementation milestone. Enforcement requires scheduler support and telemetry.

max_bandwidth_gbps

Maximum bandwidth budget for this tenant and intent.

This prevents background or noisy traffic from consuming unlimited bandwidth when the system is shared.

weight

Relative share when multiple contracts compete for the same constrained resource.

This is intended for future WDRR / weighted admission / receiver credit allocation.

burst_bytes

Amount of temporary budget a contract may consume above its steady-state cap.

This supports work-conserving behavior without permanently starving other tenants.

max_inflight_bytes

Maximum admitted but unfinished bytes for this tenant and intent.

This is useful before full bandwidth policing exists, because it bounds queue growth and memory pressure.

max_inflight_requests

Maximum admitted but unfinished request count.

This prevents many small requests from bypassing byte-based limits.

deadline_profile

Named deadline behavior profile, for example:

• interactive
• background
• best_effort
• bulk

The exact mapping to deadline-aware admission may be implemented in a follow-up PR.

allowed_degraded_actions

Actions the runtime may take when the request cannot satisfy its preferred path or deadline.

Potential values:

• delay
• fallback_transport
• local_recompute
• compress
• reject

The runtime must not apply an undeclared degraded action for a contract.

Effective Policy Resolution

Resolution order:

request explicit policy_name
    -> exact named contract, if authorized
tenant + intent
    -> tenant-specific intent contract
tenant default
    -> tenant-level default contract
global intent default
    -> default for this intent
global default
    -> compatibility default

If a request contains unknown tenant or unknown intent:

• compatibility mode: fall back to global default
• strict mode: reject or downgrade to best-effort

The resolved effective policy should be visible in debug/explain output.

Example explain output:

{
  "tenant": "tenant-a",
  "intent": "foreground_get",
  "matched_contract": "tenant-a.foreground_get",
  "requested_priority": "high",
  "effective_priority": "high",
  "max_bandwidth_gbps": 100,
  "weight": 8,
  "burst_bytes": 2147483648,
  "max_inflight_bytes": 4294967296,
  "allowed_degraded_actions": ["fallback_transport", "local_recompute", "reject"],
  "enforcement": {
    "priority_authorization": "planned",
    "max_inflight_bytes": "planned",
    "bandwidth_cap": "planned",
    "receiver_credits": "planned"
  }
}

Compatibility

Default behavior must remain unchanged if no QoS contract is configured.

Existing fields remain valid:

• Request.priority
• Request.deadline_ns
• Request.policy_name
• Request.intent_type
• existing transport selection policies
• existing QP pool / SL / TC configuration

The first implementation should not change transport selection unless explicitly configured.

Relationship to Existing Work

Intent API

The contract consumes intent_type as the semantic key. This builds on the existing Transfer Intent API work.

Transport Selection Policy

SelectionPolicy maps requests to transport, QP pool, SL, and TC. QoS Contract is one level above it: it defines what the request is allowed to ask for and what resource envelope it belongs to.

Admission Queue

Admission Queue is a natural enforcement point for max_inflight_bytes, max_inflight_requests, and future bandwidth budgets.

Receiver Credits

Receiver Credits expose receiver-side capacity. QoS Contract defines how credits should be divided across tenants and intents.

Store Tenant Quota

Store tenant quota controls object/cache capacity. TENT QoS Contract controls network transfer resources. These two should share tenant identity where possible but should not share quota semantics.

Implementation Plan

Milestone 1: Schema and Effective Policy Model

• Add QoS contract config schema.
• Parse YAML/JSON config.
• Validate fields and units.
• Resolve effective policy for request.
• Add debug/explain output.
• Add unit tests for matching, fallback, invalid config, strict/compat mode.
• No runtime enforcement yet except safe priority normalization if accepted by maintainers.

Milestone 2: Priority Authorization and Inflight Policing

• Derive effective priority from contract.
• Add tenant/intent aggregate counters.
• Enforce max_inflight_bytes.
• Enforce max_inflight_requests.
• Add tests for small-request bypass prevention.

Milestone 3: Bandwidth Cap and Borrowing

• Add token-bucket style budget.
• Support work-conserving borrowing.
• Reclaim borrowed budget when foreground traffic resumes.
• Report budget debt and throttling metrics.

Milestone 4: Receiver Credit Allocation

• Use the effective contract to allocate receiver-advertised credits by tenant and intent.
• Prevent receiver-side incast and staging-buffer overload.
• Integrate with Receiver Credits RFC.

Milestone 5: Weighted Scheduling

• Add WDRR or equivalent weighted scheduler.
• Validate fairness, goodput, SLO attainment, and isolation leakage using tebench QoS metrics.

Validation Plan

Milestone 1 validation:

• parser unit tests
• invalid schema tests
• fallback behavior tests
• effective policy resolution tests
• explain output golden tests
• backward compatibility test with no QoS contract

Later validation:

• two-tenant bandwidth share test
• noisy-background isolation test
• foreground SLO attainment test
• burst and debt recovery test
• many-small-request policing test
• receiver incast test
• tebench weighted goodput / Jain fairness / isolation leakage metrics

Open Questions

1. Should tenant_id be added directly to TENT Request, or should it remain an advisory metadata field initially?
2. Should unauthorized priority be downgraded by default or rejected in strict mode?
3. Should min_bandwidth_gbps be accepted before enforcement exists, or introduced only when bandwidth scheduling lands?
4. Should the schema live under existing TENT config or a separate QoS policy file?
5. Should contract matching support wildcard tenant / wildcard intent?
6. Which degraded actions should be standardized first?
7. How should the contract interact with explicit policy_name override?

Expected Outcome

After this RFC and the first implementation milestone, TENT will have a stable, validated QoS contract model that future scheduling, policing, receiver-credit, and wire-QoS features can consume.

This creates a clear path from application intent to enforceable network behavior without overloading low-level transport policy fields.

[github-actions]

Thanks for opening this issue, @catyans!

Field	Value
Issue	#2856
GitHub user ID	18214026
Reporter	@catyans

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.