fix(complexity_router): resolve a classified tier up a ladder instead of to whatever is configured

[tin-berri]

TLDR

Problem this solves:

• A classified tier resolved to whatever the config named, whether or not the router could serve it. A model group that was never registered, or whose every deployment was in cooldown, still won the pick and failed the request
• A tier with no entry at all jumped to default_model and then to MEDIUM, so a COMPLEX request could be answered by the model the classifier had just ruled out
• Nothing recorded that the tier which served was not the tier that was classified

How it solves it:

• Resolution walks a ladder: live models in the classified tier, then each tier above it, then default_model, then the classified tier as a best effort
• A dead pool member is passed over for its peers before the tier itself is given up on, and resolution never falls to a cheaper tier
• Climbing is recorded as tier_fallback_from on the routing decision and in the log line

Relevant issues

• Resolves a classified tier against models that can actually serve, falling to same-tier peers first and then upward, never downward
• Records tier_fallback_from whenever the tier that served is not the tier that was classified, so a cheap request on an expensive model is explainable
• Keeps an empty tier pool and a plugin denial as hard errors, and keeps routing plugins vetting whichever tier the ladder settles on

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxy on this branch. Two routers, identical except that ladder-router points its SIMPLE tier at a model group that does not exist, standing in for a tier that cannot serve:

  - model_name: smart-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers: {SIMPLE: gpt-5.4-nano, MEDIUM: gpt-5.4-mini, COMPLEX: gpt-5.4, REASONING: gpt-5.4}
        session_affinity: false

  - model_name: ladder-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers: {SIMPLE: model-that-does-not-exist, MEDIUM: gpt-5.4-mini, COMPLEX: gpt-5.4, REASONING: gpt-5.4}
        session_affinity: false

LITELLM_LOG=INFO python litellm/proxy/proxy_cli.py --config lit4898_ladder.yaml --port 4899

Healthy config, unchanged behavior:

$ curl -s -X POST http://localhost:4899/v1/chat/completions \
    -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' \
    -d '{"model": "smart-router", "messages": [{"role": "user", "content": "What is 2+2?"}]}'
ComplexityRouter: routing decision cause=heuristic_scorer, tier=SIMPLE, score=-0.150, signals=('short (3 tokens)', 'simple (what is)'), routed_model=gpt-5.4-nano

Same prompt, same classification, but the tier cannot serve. Before this change the router returned model-that-does-not-exist and the request failed with Invalid model name; now it climbs one tier and says so:

$ curl -s -X POST http://localhost:4899/v1/chat/completions \
    -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' \
    -d '{"model": "ladder-router", "messages": [{"role": "user", "content": "What is 2+2?"}]}'
ComplexityRouter: routing decision cause=heuristic_scorer, tier=MEDIUM, score=-0.150, signals=('short (3 tokens)', 'simple (what is)'), routed_model=gpt-5.4-mini, tier_fallback_from=SIMPLE

The upstream completions returned 429 (the OpenAI key on this box is out of quota, and the other provider keys available locally are unfunded), so the transcript proves the routing decision and the deployment handoff, not the provider response. The runbook below reruns it against a funded key for the 200s

Type

🐛 Bug Fix

Changes

get_model_for_tier keeps its signature and delegates to _resolve_configured_tier, the structural half of the ladder: the first tier at or above the classified one that has models configured, then default_model. The async path adds the health half, filtering each pool to model groups the router knows and that have at least one deployment out of cooldown, via the router's existing _async_get_healthy_deployments. Probes for a pool run under asyncio.gather, and the ladder returns at the first tier that can serve

_pick_model_for_tier now returns (model, served_tier). Every call site passes the served tier to the decision record and, when it differs from the classified tier, tier_fallback_from. The field is registered in DERIVED_ROUTING_DECISION_FIELDS, so it survives a row with message logging turned off; it names a tier, never caller text

Plugin handling moved into _run_routing_plugins and now runs against whichever tier the ladder settles on, rather than always the classified tier. A plugin narrowing to zero still raises rather than climbing, because refusing every candidate is a policy decision and the next tier up is exactly what was refused. default_model is still skipped entirely when plugins are configured, so their policy cannot be bypassed

Tests cover peer fallback inside a tier (repeated, so a lucky random pick cannot pass it), a single-tier climb, a climb past several dead tiers, the no-downward-fallback rule, default_model as last resort rather than first and as the ladder's terminus, probe scoping, health-lookup failure degrading to the old behavior, the empty-pool error, and both plugin paths. They inject a router stub rather than patching internals

The adaptive path keeps its own selection, which already scores every pool model with a tier-distance penalty

QA runbook

1. Check out this branch, make bootstrap, and write the config above to lit4898_ladder.yaml with a funded OPENAI_API_KEY in the environment
2. LITELLM_LOG=INFO python litellm/proxy/proxy_cli.py --config lit4898_ladder.yaml --port 4899
3. Run both curl commands above; each returns a 200 completion. Adding return_raw_model_name: true to a router's config puts the deployment that served into the response model field, so the fallback is visible without reading logs
4. Expected: smart-router serves gpt-5.4-nano, ladder-router serves gpt-5.4-mini and logs tier_fallback_from=SIMPLE
5. For the peer fallback inside a tier, set SIMPLE to ["model-that-does-not-exist", "gpt-5.4-nano"] and confirm every request lands on gpt-5.4-nano with no tier_fallback_from
6. python -m pytest tests/test_litellm/router_strategy/test_complexity_router.py -q for the unit coverage

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
Changes sit on the pre-routing model-selection path for every complexity-router request, including cost and failure behavior when deployments are missing or in cooldown; coverage is strong but mis-routing would affect production traffic.

Overview
Complexity router no longer picks whatever name is configured for a tier. It walks an upward ladder: live pool members in the classified tier (skipping cooldown/unregistered deployments), then higher tiers, then default_model, then a best-effort pick on the classified tier when nothing looks healthy.

Routing decisions and logs can now record tier_fallback_from when the served tier differs from the classified one, and resolved_by (default_model / best_effort) when the model did not come from a normal tier pool. _pick_model_for_tier returns a TierResolution instead of a bare model string; routing plugins run against the tier that actually serves and still fail closed (no climbing past a plugin denial, no default_model when plugins are configured).

Cooldown state is read once per request; if that lookup fails, behavior degrades to treating pools as servable. The complexity router README is removed. Broad unit tests cover peer fallback, multi-tier climbs, no downward fallback, plugin paths, and cooldown edge cases.

^{Reviewed by Cursor Bugbot for commit 1716d82. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

The PR changes complexity routing to select a healthy model from the classified tier or progressively higher tiers.

• Adds health-aware pool selection and routing-plugin filtering at the selected tier.
• Records the classified tier when routing climbs to another configured tier.
• Registers the new routing-decision field and adds fallback-ladder tests.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains.

No blocking failure remains.

Important Files Changed

Filename	Overview
litellm/router_strategy/complexity_router/complexity_router.py	Adds health-aware upward tier resolution, plugin filtering, and routing-decision provenance.
litellm/types/utils.py	Adds tier_fallback_from to the typed routing-decision schema and derived-field allowlist.
tests/test_litellm/router_strategy/test_complexity_router.py	Adds coverage for live peers, upward fallback, default fallback, health failures, and plugin behavior.
litellm/router_strategy/complexity_router/README.md	Describes the tier fallback ladder and its handling of plugins, empty pools, and unavailable health data.

_{Reviews (2): Last reviewed commit: "test(complexity_router): pin that defaul..." | Re-trigger Greptile}

[greptile-apps]

Product documentation added in-tree

This adds customer-facing configuration and fallback behavior documentation to the implementation repository, splitting it from the canonical litellm-docs repository and increasing the risk of documentation drift

Rule Used: Prevent documentation from being added - needs to ... (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[tin-berri]

Added a test in the latest commit for the other side of a policy #35050 just tightened: with routing plugins configured and nothing live anywhere on the ladder, default_model does not rescue the request. It is not vetted by the plugins, so serving it would route around the policy they just applied, and the ladder raises instead

That mirrors the config-time check in #35050, which no longer accepts default_model as evidence that an explicit default_tier is servable when plugins are configured. The two PRs are independent and can land in either order

@greptileai

[codecov]

Codecov Report

❌ Patch coverage is 97.29730% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...er_strategy/complexity_router/complexity_router.py	97.14%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_autorouter_tier_fallback_ladder (1716d82) with litellm_internal_staging (23de7a1)}

[tin-berri]

@greptile can you reply here with a score and an explanation why

[tin-berri]

bugbot run

[greptile-apps]

Score: 4/5

The core change is well-reasoned and correctly implemented. Here's the breakdown:

---

What's solid:

• The "climb upward, never downward" invariant is the right semantic and is consistently enforced across all paths (keyword match, heuristic, adaptive)
• asyncio.gather for concurrent health probes is correct; the early-exit-on-first-live-tier means you don't probe every tier on every request
• Plugin fail-closed behavior is preserved and actually improved: plugins now vet the tier the ladder settled on, not the classified tier
• Health-lookup failure degrades gracefully (returns full pool) rather than pushing everything up the ladder — this is the right call
• Empty pool is kept as a hard config error vs a dead pool being a runtime condition. That distinction is important and correct
• StubRouter injection in tests is much cleaner than patching internals

---

Issues:

1. Stale comment in _classify_and_route (line ~1530):

# Plugins configured: default_model must never bypass them, so it's not
# checked here at all -- _pick_model_for_tier -> get_model_for_tier still
# falls back to it (after the MEDIUM tier) once the plugin pipeline runs.

This comment refers to the old MEDIUM-tier fallback that no longer exists. get_model_for_tier no longer falls to MEDIUM; it now walks the ladder. The comment should be updated to reflect the new behavior.

2. default_model doesn't set tier_fallback_from:

In _pick_model_for_tier, when default_model is the last resort:

return self.config.default_model, tier

served_tier == tier, so tier_fallback_from is never populated even though default_model (not the classified tier's model) is what served. A downstream consumer reading the decision record can't tell whether the model name came from the classified tier or from default_model. Worth at least a comment acknowledging the trade-off.

3. README removal (minor):

The README.md was deleted without mention in the PR description. This is consistent with the project's documentation policy (docs live in litellm-docs), but it's worth confirming the content was moved there rather than silently dropped.

---

These are all minor. The logic is correct, the edge cases are covered by tests, and the observability improvement (tier_fallback_from) is a meaningful addition. The stale comment is the most actionable fix before merge.

[tin-berri]

Both fixed in d076a7d, the second one at the primitive rather than at the call sites

Stale comment. Gone. It described default_model being reached "after the MEDIUM tier", which stopped being true when the ladder replaced the fixed MEDIUM jump

default_model not setting tier_fallback_from. The root cause was the return shape, not the missing assignment. (model, served_tier) cannot express two of its own outcomes: when default_model answers, served_tier is still the classified tier, and when nothing reports a live deployment and the tier is served on a best effort, it is also still the classified tier. Both are indistinguishable from an ordinary pick, so every caller inferring "no fallback happened" from served_tier == classified_tier was wrong in exactly those two cases, and the second one was invisible in the review too

_pick_model_for_tier now returns a TierResolution naming the model, the classified tier, the tier that served, and how the model was obtained. resolved_by is default_model or best_effort, omitted on the ordinary path. The climb keeps tier_fallback_from. They are independent facts, since an exhausted ladder can both climb to a configured tier and be serving something unconfirmed, so neither is derived from the other, and a test covers that combination

_build_routing_decision takes the resolution and derives both facts itself, so the three call sites no longer recompute the comparison, which is what let one of them silently get it wrong

Four tests added: default_model recording itself, best effort recording itself, the ordinary path recording neither, and the climb-plus-best-effort case recording both. The README deletion is preserved; the equivalent guidance is in BerriAI/litellm-docs#726, which also picked up resolved_by

@greptileai

[greptile-apps]

The fix is well done. TierResolution correctly captures what the previous (model, served_tier) tuple could not express: resolved_by distinguishes a live-tier pick from default_model and best_effort, and served_tier != classified_tier still captures the climb — independently, since the exhausted-ladder case can produce both. Centralizing the derivation in _build_routing_decision was the right fix rather than patching each call site.

The four new tests (test_default_model_serves_when_the_ladder_is_exhausted_and_says_so, test_serving_an_unconfirmed_tier_is_recorded_as_best_effort, test_an_ordinary_pick_records_no_resolution_fact, test_a_best_effort_climb_records_both_facts) cover each branch of _build_routing_decision cleanly, including the served_tier != classified_tier && resolved_by == best_effort combination that was the invisible case in the original review.

LGTM.

[tin-berri]

@greptile can you reply here with a score and an explanation why

[tin-berri]

bugbot run

[greptile-apps]

Score: 4/5

The logic is sound and the test suite is thorough. Three issues worth flagging:

---

1. Health probes on every request (performance — critical path)

_live_models fires one _async_get_healthy_deployments call per model in the tier pool on every request, using asyncio.gather. In a production setup where the cooldown cache is backed by Redis, this adds N network roundtrips to the critical path before any actual LLM call. The graceful degradation (except Exception → return tuple(pool)) is good, but the happy-path cost is real.

The existing router already consults the cooldown cache during deployment selection — this is a second pass before the tier is even chosen. Consider whether the same information can be pulled from an already-hot in-process cache view rather than re-querying.

---

2. Silent removal of the MEDIUM fallback (backwards-incompatible, rule b48b7341)

The old code had:

medium_key = ComplexityTier.MEDIUM.value
if medium_key in self.config.tiers:
    return self._pick_from_tier_value(self.config.tiers[medium_key], medium_key)

Any unresolvable tier previously fell to MEDIUM. Under the new ladder, a COMPLEX tier with no models now climbs to REASONING (if configured), not MEDIUM. Users who intentionally left COMPLEX unconfigured to get MEDIUM as a catch-all will silently start getting REASONING instead — the more expensive model.

This is a meaningful behavior shift. Per the project's backwards-compat rule, it should either be guarded by a flag or at least called out explicitly in the PR description as a deliberate breaking change.

---

3. README removal leaves the component undocumented in-tree

The README was the only discoverable explanation of scoring dimensions, tier boundaries, and config options. The external docs repo presumably has none of this yet. Until it does, new contributors have nowhere to look.

---

The ladder semantics, TierResolution NamedTuple, tier_fallback_from telemetry, and plugin-fail-closed behavior are all implemented correctly. The tests cover the edge cases well (peer fallback, no-downward rule, best-effort, plugin paths, health lookup failure). The performance concern in point 1 and the silent behavior change in point 2 are the primary reasons this isn't a 5.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

1 issue from previous review remains unresolved.

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit d076a7d. Configure here.}

[tin-berri]

Fixed in 364be93. The finding was right, and worse than it read: _async_get_healthy_deployments goes through _async_get_cooldown_deployments, which batches over every deployment id the router has, not just the one model group being asked about. So a pool of N models meant N full-router cooldown reads per request. asyncio.gather made them concurrent, which hid the count without removing it

The set is now read once per request, before the ladder walks, and reused for every tier it visits. What is left per model is a lookup in the router's in-process deployment list plus a frozenset membership test, so resolution does one piece of I/O regardless of pool size or how far it climbs. Measured on a 6-model pool: 6 reads before, 1 after

I did not reach into an existing hot view instead, deliberately. CooldownCache.async_get_active_cooldowns already checks the in-memory layer first and only reaches Redis on a miss, so the batched read is the hot view; the defect was calling it N times, not calling it at all. Reading it once keeps the router's own tested semantics rather than duplicating cooldown interpretation here

Degradation is unchanged: an unreadable cooldown view still means "assume the pool is live" instead of emptying every tier. The test double now models the mechanism it stands in for, registered model groups whose deployments may be cooling, so test_resolution_reads_the_cooldown_view_once_per_request asserts one read across a full four-tier climb, and an unregistered group is passed over the same way a fully-cooling one is

@greptileai can you reply here with a score and why that score?

[greptile-apps]

5/5

The fix is correct and the explanation is accurate.

What the diff shows:

_cooled_down_deployment_ids() calls _async_get_cooldown_deployments exactly once and returns a frozenset[str] | None. That result is captured at the top of _pick_model_for_tier and threaded through to _live_models for every tier the ladder visits:

cooled_down = await self._cooled_down_deployment_ids()
for candidate_tier in self._tier_ladder_from(tier):
    ...
    live = self._live_models(pool, cooled_down)

Everything after that one read is in-process: get_model_list hits the router's deployment list directly, and _has_live_deployment is a frozenset membership test. The N-reads-per-request problem is gone.

The degradation contract is preserved: a None from _cooled_down_deployment_ids (any exception from the cache or Redis) falls through to tuple(pool) in _live_models, which is the old behavior — assume the tier is live rather than emptying it.

The test is meaningful: test_resolution_reads_the_cooldown_view_once_per_request sets up a four-tier climb (all tiers dead except REASONING), asserts stub.cooldown_cache.reads == 1, and confirms all five pool members were inspected via get_model_list. That directly encodes the invariant the fix is making.

One minor observation: _deployment_id returns str | None, and a None id is treated as live (since None not in frozenset[str] is True). That means a deployment with a missing model_info.id silently counts as healthy. It's a pre-existing edge case, not introduced here, and the no-id path is exotic enough not to block this. Worth a comment if it ever surfaces, but not a blocker.

Everything else — the single I/O point, the in-process per-model lookups, the degraded-mode behavior, and the test asserting the one-read invariant — is solid.

[tin-berri]

Not a defect, so no behavior change; addressed as a legibility fix in 1716d82

The reasoning: cooldowns are keyed by deployment id, so a deployment that cannot be identified can never appear in the cooldown set and can never be known to be cooling. Counting it as live is the same rule the rest of the ladder follows, that an absent health signal means live. The inverse would be actively worse: missing metadata would silently exclude a deployment, potentially emptying a tier and escalating traffic to a pricier one on the strength of a metadata gap rather than anything about health, which is exactly what the fail-open design exists to prevent

Reachability is narrower than "pre-existing edge case" suggests. ModelInfo.__init__ (litellm/types/router.py:160-164) mints a uuid whenever id is None, so every deployment registered through the typed path has one; it takes a raw dict bypassing that to produce an id-less deployment at all

What was fair is that the rule was implicit, recoverable only by noticing None not in frozenset[str] is True. It now reads deployment_id is None or deployment_id not in cooled_down, with a test asserting an id-less deployment serves normally and records neither a climb nor a fallback. The rule is pinned rather than inferred, so a future edit that flips it has to break a named test

@greptileai can you reply here with a score and why that score?

[greptile-apps]

5/5

The legibility fix in 1716d82 is exactly right, and the reasoning holds up. The original implicit behavior (None not in frozenset[str] evaluating to True) was correct but unreadable — a future editor has to know Python's membership semantics well enough to derive the rule from the expression, and there was nothing stopping a "cleanup" from flipping it accidentally. deployment_id is None or deployment_id not in cooled_down pins the intent on the page, and the new test locks it so any regression breaks a named test rather than silently routing traffic to a pricier tier.

The rest of the PR holds up equally well across all the dimensions I'd check:

• Core logic: the ladder-walk is sound. Live peers before a tier-climb, never downward, default_model strictly last resort, best-effort fallback when the health view is stale rather than failing a request that may succeed.
• Performance: one cooldown read per request, not one per pool member or per tier the ladder walks. For a Redis-backed cooldown store that matters.
• Plugin policy: correctly preserved. Fail-closed when a plugin narrows to zero; default_model still blocked when plugins are configured so their policy can't be bypassed by exhausting the ladder.
• Observability: tier_fallback_from and resolved_by are independent facts on the record, omitted in the ordinary case, so a cheap request on an expensive model is explainable without noise on the happy path.
• Tests: peer fallback (repeated to defeat a lucky random pick), single-tier climb, multi-tier climb, no-downward-fallback, best-effort, health lookup failure degrading gracefully, empty pool as config error, both plugin paths — and the stub-router approach means none of these tests break when internals change.

The one deletion worth noting is the module README, but it documented behavior that is now expressed directly in code and test names, so removing it avoids the documentation drift that would have followed as the ladder behavior matured.

[tin-berri]

bugbot run

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1716d82. Configure here.}

[cursor]

Paused deployments skip ladder climb

Medium Severity

The updated _has_live_deployment function only checks for cooldowns, unlike the previous _async_get_healthy_deployments which also filtered admin-paused deployments. This change means a tier with only blocked deployments is now incorrectly considered live, preventing the router from climbing the ladder and causing requests to fail instead of routing to an available tier.

 

^{Reviewed by Cursor Bugbot for commit 1716d82. Configure here.}