DEP: Session-aware admission strategies in the local router

[ishandhanani]

Summary

Add a stateful admission-strategy boundary to the local router. Session-aware routing is the first use case, but the interface should remain generic. The router continues to own request payloads, queues, cancellation, worker selection, and dispatch. A strategy owns only the state needed to decide whether a request is ready, deferred, or outside its scope. This DEP covers one frontend router/WorkerSet. Cross-frontend and cross-cluster coordination are out of scope for Phase 1.

Motivation

Session-aware admission needs both immutable request facts, such as session ID and initial context, and live facts, such as generated-token progress. Sending every token update through the serialized scheduler actor adds allocations, contention, and best-effort channel delivery.

At the same time, putting session aware routing directly in PolicyQueue would couple generic queue ownership to one strategy.

Proposed architecture

• PolicyQueue owns ready/deferred request storage, ordering, limits, and counters.
• PolicyClassAdmissionStrategy owns algorithm state and returns Bypass, Ready, or Defer.
• The existing selector remains responsible for final worker validation, scoring, and booking.
• Strategies receive narrow typed inputs:
  • immutable facts in AdmissionRequest;
  • live host-owned readers such as RequestProgress;
  • serialized lifecycle events such as Dispatched, Completed, and Aborted.

sequenceDiagram
    participant F as Frontend / RequestGuard
    participant Q as SchedulerQueueActor
    participant A as Admission strategy
    participant W as Worker selector

    F->>Q: schedule(request, progress reader)
    Q->>A: admit(request facts + typed readers)
    alt Bypass or Ready
        A-->>Q: Ready(optional worker constraint)
        Q->>W: validate, select, and book
        W-->>F: response lifecycle owner
        loop response stream
            F->>F: atomic progress update
        end
        F->>Q: Completed(final context) or Aborted
        Q->>A: terminal lifecycle event
    else Defer
        A-->>Q: Defer
        Q->>Q: retain in PolicyQueue
        A-->>Q: MakeReady
        Q->>W: validate, select, and book
    end

Loading

RequestProgress uses a shared monotonic atomic value. The strategy reads the latest value without actor messages. Completed { context_tokens } is authoritative; Aborted does not commit new logical context.

One router-owned record tracks the lifecycle:

pending -> selected -> dispatched -> completed | aborted

Cancellation ownership transfers from the scheduling future to the response lifecycle owner after dispatch. Duplicate or stale lifecycle events must be harmless.

Session-aware strategy contract

A concrete strategy must define:

• same-session concurrency and deferral;
• worker affinity and invalid-worker behavior;
• completion versus abort accounting;
• bounded state, expiry, and cleanup;
• behavior for requests without a session, normally Bypass.

Session-close semantics can be added later as generic typed metadata or a lifecycle event rather than an algorithm-specific boolean.

Multiple frontends

Phase 1 state is frontend-local: strategy instances, admission IDs, deferred requests, lifecycle records, and progress handles are not shared.

A shared third-party program table alone does not provide distributed correctness. Strict same-session serialization across frontend replicas requires session-affine ingress for Phase 1. A future design would also need fenced ownership, idempotent terminal updates, deferred-request ownership, global accounting, and frontend-failure recovery.

Performance contract

• No external RPC or async callback on the scheduler actor path.
• No actor command or request-ID allocation per progress update.
• No progress handle or lifecycle overhead when no strategy is configured.
• Bypass creates no retained strategy lifecycle.
• Normal admission must not scan all retained sessions.

Local benchmarks measured approximately 2.525 ns per atomic progress update and 0.486 ns per read. The previous actor-command approach measured 220-560 ns per update and dropped 48-82% of burst updates after channel saturation.

Non-goals

• Rejection thresholds, policy-class DRR, and frontend/worker rejection gates (DEP: Request Admission and Rejection Controls #9755).
• Cross-cluster session placement (DEP: Session-Aware Placement Policy for the Multi-DC Global Router #11403).
• Distributed strategy state or global admission authority.
• Engine preemption, KV eviction, or cache-retention APIs.
• A second strategy-owned request queue.

Delivery

• feat(router): define queue admission strategy API #11434 defines the minimal strategy API and configuration surface.
• Stacked host PR adds queue-owned lifecycle management and typed live progress.
• feat(router): add ThunderAgent admission strategy #11447 should contain ThunderAgent-specific state and behavior on top of that host.
• feat(router): unify worker availability admission #11482 is exploratory; rejection and availability work should remain under DEP: Request Admission and Rejection Controls #9755.

References

• DEP: Request Admission and Rejection Controls #9755 — request admission and rejection controls
• feat(router): add exact-worker policy queue lanes #11363 — exact-worker queue lanes
• DEP: Session-Aware Placement Policy for the Multi-DC Global Router #11403 — session-aware multi-cluster placement
• DEP: Worker-owned slot tracker lifecycle events #10511 — worker-owned lifecycle tracking
• feat(router): define queue admission strategy API #11434 — generic admission API
• feat(router): add ThunderAgent admission strategy #11447 — ThunderAgent strategy
• feat(router): unify worker availability admission #11482 — worker-availability exploration

[ishandhanani]

#11434 (review)

Moving forward with this. See #11434 as a very small basic API