Skip to content

fix(complexity_router): route no-signal prompts to default_tier, not SIMPLE - #35050

Open
tin-berri wants to merge 6 commits into
litellm_internal_stagingfrom
litellm_lit4898_default_tier
Open

fix(complexity_router): route no-signal prompts to default_tier, not SIMPLE#35050
tin-berri wants to merge 6 commits into
litellm_internal_stagingfrom
litellm_lit4898_default_tier

Conversation

@tin-berri

@tin-berri tin-berri commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • The heuristic scorer has no way to say "I don't know". Its dimensions are a roughly 100-word software-vocabulary whitelist, so a prompt matching none of them scores exactly 0.0, and 0.0 falls under simple_medium (0.15) into SIMPLE. Absence of evidence was scored as evidence of simplicity, on 50.2% of prompts in a graded 809-question benchmark and 36.6% of turns in a 257-session agent corpus
  • The same path catches LLM-classifier outages, since a timeout or error falls back to this scorer

How it solves it:

  • _score_and_classify returns a new default_tier (MEDIUM by default) when nothing was recognised, under its own decision cause no_signal_default and with signals=['no-signal']
  • default_tier: SIMPLE restores the previous behavior exactly, for anyone who wants unmatched traffic on the cheapest tier
  • The branch reads the signals, not the score and not the individual dimension scores, because neither of those means "nothing was recognised"

Relevant issues

  • Routes prompts that fire no scoring dimension to a configurable default_tier instead of letting a 0.0 score fall through into SIMPLE, and logs them as cause=no_signal_default
  • Keeps genuinely-simple traffic on SIMPLE: a prompt whose only evidence is a greeting, a "what is", or a short length still scores and still lands SIMPLE
  • Pins default_tier: SIMPLE in the router e2e fixture, whose classifier-vs-fallback discriminator is itself a no-signal prompt and would otherwise have gone false-green

Linear ticket

Resolves LIT-4898

The tier fallback ladder that came out of the same discussion is split into its own PR, #35331, based on staging rather than stacked on this one

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxy on this branch at 2d25d23. Two routers over the same three models: smart-router on the new default, and smart-router-legacy carrying default_tier: SIMPLE, which is the behavior on staging today

  - model_name: smart-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers: {SIMPLE: gpt-5.4-nano, MEDIUM: gpt-5.4-mini, COMPLEX: gpt-5.4, REASONING: gpt-5.4}
        session_affinity: false

  - model_name: smart-router-legacy
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        default_tier: SIMPLE
        tiers: {SIMPLE: gpt-5.4-nano, MEDIUM: gpt-5.4-mini, COMPLEX: gpt-5.4, REASONING: gpt-5.4}
        session_affinity: false
LITELLM_LOG=INFO python litellm/proxy/proxy_cli.py --config lit4898_abstain.yaml --port 4897

A prompt that matches nothing. It used to be asserted simple; it now abstains, and the row says why:

$ curl -s -X POST http://localhost:4897/v1/chat/completions \
    -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' \
    -d '{"model": "smart-router", "messages": [{"role": "user", "content": "A farmer must ferry a wolf, a goat, and a cabbage across a river. ..."}]}'
ComplexityRouter: routing decision cause=no_signal_default, tier=MEDIUM, score=0.000, signals=('no-signal',), routed_model=gpt-5.4-mini

A prompt with real evidence of being simple, unchanged:

$ curl -s -X POST http://localhost:4897/v1/chat/completions \
    -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' \
    -d '{"model": "smart-router", "messages": [{"role": "user", "content": "What is 2+2?"}]}'
ComplexityRouter: routing decision cause=heuristic_scorer, tier=SIMPLE, score=-0.150, signals=('short (3 tokens)', 'simple (what is)'), routed_model=gpt-5.4-nano

The case that keeps the fix honest. This one scores zero, but with three dimensions firing, so it is not silence and stays on the cheap tier:

$ curl -s -X POST http://localhost:4897/v1/chat/completions \
    -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' \
    -d '{"model": "smart-router", "messages": [{"role": "user", "content": "hi, quick python question"}]}'
ComplexityRouter: routing decision cause=heuristic_scorer, tier=SIMPLE, score=-0.000, signals=('short (6 tokens)', 'code (python)', 'simple (quick, hi)'), routed_model=gpt-5.4-nano

The opt-out, same no-signal prompt against the router carrying default_tier: SIMPLE:

$ curl -s -X POST http://localhost:4897/v1/chat/completions \
    -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' \
    -d '{"model": "smart-router-legacy", "messages": [{"role": "user", "content": "A farmer must ferry a wolf, a goat, and a cabbage across a river. ..."}]}'
ComplexityRouter: routing decision cause=no_signal_default, tier=SIMPLE, score=0.000, signals=('no-signal',), routed_model=gpt-5.4-nano

The upstream completions returned 429 (the OpenAI key on this box is out of quota, and the other provider keys available locally are unfunded), so these transcripts prove the routing decision and the deployment handoff, not the provider response. The runbook below reruns the same commands against a funded key for the 200s

Type

🐛 Bug Fix

Changes

_score_and_classify returns config.default_tier with signals=['no-signal'] and cause=no_signal_default when nothing was recognised, placed after the 2-or-more-reasoning-marker override and before the band mapping. The cause is a first-class value for the same reason reasoning_override is: it is the fact that says the score did not choose the tier, and burying it in signals would let anything that filters signals change what the row claims

The branch reads signals, not the weighted score, and not the individual dimension scores either. Both of those look like "nothing was recognised" without being it. The weighted score cancels: hi, quick python question is tokenCount -1.0x0.10 plus simpleIndicators -1.0x0.05 plus codePresence 0.5x0.30, which comes to zero with three dimensions firing and is real evidence of a simple request. A per-dimension zero is not silence either, since _score_keyword_match takes the no-match score as a parameter, so a future dimension with a nonzero baseline would silently kill the branch. A signal is the one thing a dimension emits only when it recognised something, and a test pins that invariant across every scorer, in both directions

Short prompts (under 15 estimated tokens) and long ones (over 400) fire tokenCount and score normally, so this path holds prompts of roughly 15 to 400 estimated tokens with zero keyword and pattern hits

default_tier defaults to MEDIUM, so no config change turns it on; setting it only overrides. default_tier: SIMPLE restores the previous behavior

default_tier is not a special tier. It names which tier no-signal traffic is classified as, and that tier resolves to a model through the same chain as any classified one. ComplexityRouterConfig.resolve_tier is that chain, returning either the models that may serve a tier or the reason none may, and both config validation and request-time selection go through it. So a default_tier that nothing can serve is rejected at load rather than surfacing on the first unmatched request, and every config that loads is one the default tier can actually be served from

That check covers the defaulted MEDIUM as well as a value you typed. It was originally skipped when the field was left implicit, so that a partial tiers map would not become a startup failure; through resolve_tier it no longer can, because the proxy always derives a default_model from the MEDIUM-then-SIMPLE tier and the chain terminates there. What is left is one shape, and it is the reason the exemption had to go: with routing plugins configured the chain stops at the tier's own models, since a model the plugins never vetted must not serve, so a plugin config whose tiers has no MEDIUM cannot serve no-signal traffic at all. Abstaining moved those prompts off SIMPLE, so before this it loaded and raised on every one of them

Two things fell out of putting that chain in one place. Resolution now keys off whether a tier has models rather than whether its key is present, so tiers: {MEDIUM: []} and a tiers map with no MEDIUM at all behave the same; before, the empty one entered the pool and refused to pick from it while the absent one fell through to default_model. And the deployment-level complexity_router_default_model is folded in before validation instead of being assigned onto the validated model afterwards. router.py derives that value from the MEDIUM-then-SIMPLE tier whenever it is unset, so an explicit default_tier outside tiers is servable on every proxy deployment; validating before it was applied failed those configs at startup for a gap routing did not have

Tests cover the abstain itself, the config validation, both cancelling cases (the shipped-weights one that lands on float dust, and a configured-weights one that lands on a bit-exact 0.0, so the cheaper predicate is a live substitution the suite rejects), the scorer signal invariant the predicate now depends on, the SIMPLE-only and short-prompt-only edges that must not abstain, default_tier across all four tiers, escalation on top of an abstain, and a fixed 13-prompt corpus pinning the tier of each so a later weight or keyword edit cannot walk the default back toward all-SIMPLE. The servability tests pin the invariant rather than the instances: a config the validator accepts is one the default tier can be served from, parameterized over the tier's own models, default_model at either level, and an empty pool falling through

Docs: the router README gains a "No-Signal Default" section, and the proxy docs page is updated in a separate PR

Known limits, not fixed here

Unmatched prompts over 400 estimated tokens fire tokenCount, score +0.10, and still land SIMPLE under the 0.15 boundary. Moving that boundary is a pricing decision tracked separately, as is the complex_reasoning calibration. On the graded corpus this leaves hard-labelled questions on SIMPLE at 31.9%, down from 67.0%

This alters spend for every deployment on the heuristic default with no config change on their side. On the agent corpus the blended per-turn price proxy goes from 2.78 to 4.24, about +53%, with tier mix moving 84/13/3/0 to 47/49/3/0. It needs a release-note call-out, and default_tier: SIMPLE is the one-line opt-out

QA runbook

  1. Check out this branch, make bootstrap, and write the config above to lit4898_abstain.yaml with a funded OPENAI_API_KEY in the environment
  2. LITELLM_LOG=INFO python litellm/proxy/proxy_cli.py --config lit4898_abstain.yaml --port 4897
  3. Run the four curl commands above. Each returns a 200 completion. Adding return_raw_model_name: true to either router's config puts the deployment that served into the response model field, so routing is visible without reading logs
  4. Expected: the river-crossing prompt serves gpt-5.4-mini on smart-router and gpt-5.4-nano on smart-router-legacy; What is 2+2? and hi, quick python question serve gpt-5.4-nano on both
  5. python -m pytest tests/test_litellm/router_strategy/test_complexity_router.py -q for the unit coverage

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Note

Medium Risk
Default routing behavior changes for existing deployments without config edits (more unmatched traffic to MEDIUM and higher spend), though default_tier: SIMPLE opt-out exists; routing logic is well-tested.

Overview
When the heuristic scorer gets no signals from any dimension, it no longer maps a 0.0 score into SIMPLE. It returns configurable default_tier (default MEDIUM), logs signals=['no-signal'], and records cause=no_signal_default. The branch keys off empty signals, not the weighted score, so prompts like hi, quick python question that cancel to zero but still have evidence stay on SIMPLE.

default_tier: SIMPLE restores the old behavior. Explicit default_tier values are validated at config load so the tier has a servable model (with stricter rules when routing plugins are enabled). The plugin tier-pick path now errors clearly when a tier has no models configured instead of implying plugin filtering removed them.

Docs, RoutingDecisionCause, e2e fixture default_tier: SIMPLE (so classifier-vs-fallback tests stay valid), and broad unit/corpus tests cover abstain, config validation, and the dimension signal invariant.

Reviewed by Cursor Bugbot for commit be7b23c. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread litellm/router_strategy/complexity_router/config.py
@greptile-apps

greptile-apps Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR changes unmatched-prompt routing and consolidates tier servability resolution.

  • Routes prompts with no heuristic signals to a configurable default tier and records a dedicated routing-decision cause.
  • Uses one resolver for default-tier validation and request-time model selection, including empty pools, deployment defaults, and plugin restrictions.
  • Extends unit and end-to-end coverage for no-signal classification, model resolution, and classifier fallback behavior.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
litellm/router_strategy/complexity_router/complexity_router.py Adds no-signal classification and consistently delegates model selection to the shared tier resolver.
litellm/router_strategy/complexity_router/config.py Adds configurable default-tier validation and centralizes tier-to-model resolution across direct, fallback, and plugin-backed configurations.
litellm/types/utils.py Extends the routing-decision cause contract with the no-signal default classification.
tests/test_litellm/router_strategy/test_complexity_router.py Adds broad regression coverage for no-signal routing and the previously reported tier-servability cases.
tests/e2e/router/conftest.py Pins legacy SIMPLE fallback behavior so the classifier fallback discriminator remains observable.
tests/e2e/router/test_complexity_router_e2e.py Updates the end-to-end test explanation for the explicitly configured legacy fallback behavior.
litellm/router_strategy/complexity_router/README.md Documents the new no-signal default behavior and its configuration.

Reviews (7): Last reviewed commit: "docs(complexity_router): the servability..." | Re-trigger Greptile

@tin-berri

Copy link
Copy Markdown
Contributor Author

Thanks, both findings addressed or answered in 153fc17

Inconsistent default-tier configurations (the score driver). Fixed, with one deliberate narrowing. ComplexityRouterConfig now rejects a default_tier that has no model behind it, meaning no non-empty entry in tiers and no default_model, and the error names the tiers that are configured. The validator only fires when default_tier was set explicitly. Validating the implicit MEDIUM default too would turn any existing partial tiers map (say only SIMPLE and REASONING) into a startup failure on upgrade, which trades a rare request-time error for a guaranteed outage; leaving it implicit keeps the same tier -> default_model -> MEDIUM resolution chain every scored tier already has. Four tests cover it: explicit tier with no entry, explicit tier with an empty pool, explicit tier rescued by default_model, and a partial map still loading when the field is left implicit

Product documentation in the code repository. Keeping the README change, deliberately. That file already documents this exact class of behavior (the reasoning override, system-prompt handling, boundaries, weights), so the new section sits next to the behavior it describes rather than opening a new documentation surface. The customer-facing page is the one on the docs site, and it is updated in BerriAI/litellm-docs#703, which also carries the release-note wording for the spend increase

@greptileai

@greptile-apps

greptile-apps Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Both fixes look correct and well-implemented.

The _validate_default_tier_is_servable validator correctly gates on "default_tier" not in self.model_fields_set, so partial tiers maps on upgrade are unaffected — the implicit MEDIUM default passes through without any check. The explicit-only validation fires when the user has expressed intent, which is exactly the right boundary.

The predicate for abstain (all(dimension.score == 0 ...)) versus weighted_score == 0.0 is backed by the two cancellation tests — the float-dust case and the bit-exact 0.0 case with custom weights — which together make the predicate choice load-bearing in the suite rather than just documented.

The validator test for an empty pool ({"MEDIUM": []}) is a nice edge case that would otherwise silently pass .get() if the check were only self.tiers.get(self.default_tier.value) without truthiness — the code handles it correctly since an empty list is falsy.

Both findings are addressed properly. LGTM.

@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@codspeed-hq

codspeed-hq Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_lit4898_default_tier (be7b23c) with litellm_internal_staging (0e9a624)

Open in CodSpeed

@tin-berri

Copy link
Copy Markdown
Contributor Author

Pushed a0a78af after that review: message-only change to the same validator. It ran before ComplexityRouter.__init__ applies the deployment-level complexity_router_default_model (router.py derives that from the MEDIUM, then SIMPLE, tier when unset, so it is always populated on the proxy path), which meant the old wording told operators a knob they may have set was unset. The rejection is unchanged; only the text now names the requirement that was missed and why the deployment-level fallback does not satisfy it

@greptileai

@tin-berri
tin-berri force-pushed the litellm_lit4898_default_tier branch from a0a78af to 4c71c8a Compare July 31, 2026 06:14
@tin-berri tin-berri changed the title fix(complexity_router): route no-signal prompts to default_tier, not SIMPLE fix(complexity_router): abstain to default_tier on no signal, and resolve tiers up a ladder Jul 31, 2026
@tin-berri

Copy link
Copy Markdown
Contributor Author

Rebased onto latest staging and extended per review discussion, now at 4c71c8a

The rebase was not clean: #35016 (routing-decision provenance), #35185 (classifier prior-turn context) and #35300 landed in this file since the PR opened. The abstain is re-applied on top of the new shape, and it now carries its own cause=no_signal_default rather than being an unremarkable heuristic_scorer row, on the same argument the codebase already makes for reasoning_override: the cause is the fact that says the score did not choose the tier

Second half added: tier resolution now walks a ladder instead of resolving to whatever was configured. Live models in the classified tier first, so a dead pool member is passed over for its peers, then upward a tier at a time, then default_model, then the classified tier as a best effort. It never falls to a cheaper tier, since that is the model the classifier already ruled out. Climbing is recorded as tier_fallback_from on the decision and in the log line. Empty pools and plugin denials still raise, and an unreadable health view degrades to the old behavior. Proof and full detail are in the updated description

@greptileai

Comment thread litellm/router_strategy/complexity_router/config.py Outdated
…SIMPLE

The heuristic scorer has no way to say "I don't know". Its seven dimensions are a
roughly 100-word software-vocabulary whitelist, so when a prompt matches none of them
every dimension contributes 0, the weighted sum is 0.0, and 0.0 sits below
simple_medium (0.15). Absence of evidence was being scored as evidence of simplicity,
and unclassifiable traffic went to the cheapest tier. On a graded 809-question
benchmark that no-signal mass is 50.2% of prompts; on a 257-session agent transcript
corpus it is 36.6% of turns

_score_and_classify now returns config.default_tier before the band mapping when
nothing was recognised, under its own cause=no_signal_default with
signals=['no-signal'], so a spend log row says the score did not choose the tier, the
way reasoning_override already does. default_tier is new on ComplexityRouterConfig and
defaults to MEDIUM; setting it to SIMPLE restores the previous behavior exactly. The
LLM classifier falls back to this scorer on timeout or error, so the same setting
decides where unmatched traffic lands during a classifier outage

The branch tests the signals rather than the score, and rather than the individual
dimension scores. A weighted score of zero does not mean nothing was recognised, since
contributions cancel: "hi, quick python question" comes to zero with three dimensions
firing and is real evidence of a simple request. A dimension scoring zero does not mean
it stayed silent either, because _score_keyword_match takes the no-match score as a
parameter, so a future dimension with a nonzero baseline would silently kill the
branch. A signal is the one thing a dimension emits only when it recognised something,
and a test pins that invariant across every scorer

An explicitly configured default_tier with no model behind it is rejected at load
rather than surfacing as a routing error on the first unmatched request. The check is
skipped when the field is left implicit, so a partial tiers map keeps loading as it
does today instead of turning an upgrade into a startup failure

The router e2e config pins default_tier: SIMPLE. That test tells a live LLM classifier
apart from a silent fallback by which backend served the request, and its prompt has no
scoring signal, so leaving the new default in place would have made both arms land on
the same backend and turned the test into a false green
@tin-berri
tin-berri force-pushed the litellm_lit4898_default_tier branch from 4c71c8a to 2d25d23 Compare July 31, 2026 06:32
@tin-berri tin-berri changed the title fix(complexity_router): abstain to default_tier on no signal, and resolve tiers up a ladder fix(complexity_router): route no-signal prompts to default_tier, not SIMPLE Jul 31, 2026
@tin-berri

Copy link
Copy Markdown
Contributor Author

Split into two PRs at Tin's request, and rebased onto latest staging (3c2264c). This PR is now the classification half only; the tier fallback ladder moved to #35331, also based on staging rather than stacked on this one, so the two can land in either order

One change since the last review, from a question about the predicate: it now reads if not signals: rather than all(dimension.score == 0 ...). Those coincide today, but only by an unwritten invariant. _score_keyword_match takes the no-match score as a parameter, so a future dimension with a nonzero baseline would have silently killed the abstain branch, which is the same failure mode this fix exists to avoid one level up. A signal is the one thing a dimension emits only when it recognised something. TestDimensionSignalInvariant pins that in both directions across every scorer, including the case where a term matched only in the system prompt and the signal reports a count instead of the term

Proof in the description was re-captured on this branch's own head (2d25d23)

@greptileai

…ault_tier under plugins

The validator accepted default_model as evidence that an explicitly configured
default_tier could be served. The plugin path never consults it: candidates come from
the tier pool alone, and a pool that is empty raises rather than falling through to a
model the plugins never vetted. So a config with plugins, an explicit default_tier
absent from tiers, and a default_model passed validation and then failed on its first
no-signal request, which is exactly what validating at load is meant to prevent

default_model now counts only when no plugins are configured, and the error names which
of the two remedies applies
@tin-berri

Copy link
Copy Markdown
Contributor Author

Fixed in 9879bae. The finding is real, and it bites this PR on its own rather than only in combination with #35331

Reproduced before fixing: with routing plugins configured, an explicit default_tier: COMPLEX absent from tiers, and a default_model set, the config validated and then the first no-signal request raised No candidate models left for tier COMPLEX after routing-plugin filtering. The plugin path builds candidates from the tier pool alone and never consults default_model, deliberately, since a model the plugins did not vet must not serve. So the validator was accepting evidence the runtime refuses, which is exactly what validating at load exists to prevent

default_model now counts as evidence only when no plugins are configured, and the error names whichever remedy applies:

default_tier COMPLEX is not a non-empty entry in tiers (['MEDIUM', 'SIMPLE']). Add it to tiers, or name a tier that is configured; routing plugins are configured, so default_model is not consulted: a model the plugins never vetted must not serve

A regression test covers the plugin case, alongside the existing ones for the no-plugin case where default_model still does rescue it. #35331 got the matching runtime test on the other side of the same policy, so both halves are pinned

@greptileai

@greptile-apps

greptile-apps Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

The fix in 9879bae is correct and the reasoning is sound.

The validator now correctly distinguishes the two paths: without plugins, default_model is a legitimate fallback because the runtime will consult it; with plugins, it is not, because the plugin path builds candidates from the tier pool alone and default_model is never reached. Accepting it with plugins would create a config that validates clean and then raises on the first no-signal request — exactly the gap the validator exists to close.

A few specifics worth noting:

  1. Validator logic (_validate_default_tier_is_servable): the early return on if "default_tier" not in self.model_fields_set is preserved, so a partial tiers map without an explicit default_tier still loads without triggering this check — consistent with the implicit MEDIUM default going through normal resolution.

  2. Error messages: both branches name the applicable remedy explicitly ("routing plugins are configured, so default_model is not consulted" vs. the no-plugin message), which is exactly the right UX for a config-time failure.

  3. Test coverage: test_default_model_does_not_rescue_the_default_tier_when_plugins_are_configured and test_default_tier_outside_tiers_is_allowed_with_default_model together pin both sides of the policy, so a future change to the validator that re-allows default_model under plugins will have to break a named test to do so.

The fix is complete and both halves of the policy are now pinned. LGTM.

…s no models for it

With routing plugins configured, a tier absent from `tiers` produced "No candidate
models left for tier MEDIUM after routing-plugin filtering". Nothing was filtered: the
pool was empty before any plugin ran, so the message sent an operator to read plugin
code for what is a gap in `tiers`

The two cases now raise separately. An empty pool names the tier and says why
default_model is not consulted; a plugin narrowing a real pool to zero keeps its own
message, since that one really is a policy decision

This is the path no-signal traffic now takes, because the implicit MEDIUM default is
deliberately not validated at load: doing so would turn every partial tiers map into a
startup failure. The tier ladder in #35331 resolves the same shape at runtime by
climbing to the next configured tier, so this is the standalone behavior
@tin-berri

Copy link
Copy Markdown
Contributor Author

Yes, the finding you quoted was fixed in 9879bae, and Greptile's follow-up on that commit confirmed it. Its own words: "Accepting it with plugins would create a config that validates clean and then raises on the first no-signal request, exactly the gap the validator exists to close"

Re-reading it, though, there was a sibling case still open, through the implicit default rather than the explicit one. Plugins configured, MEDIUM absent from tiers, and default_tier left unset: the validator skips the check by design, so the config loaded and the first no-signal request raised. Reproduced, then handled in be7b23c

I did not close it by tightening the validator, and that is deliberate. The implicit default is not validated so that a partial tiers map keeps loading rather than turning an upgrade into a startup failure, which is the design Greptile endorsed in round two. More to the point, #35331's tier ladder resolves this exact shape at runtime: with MEDIUM absent it climbs to COMPLEX, the plugins still vet the candidates, and the decision records tier_fallback_from=MEDIUM. Verified on that branch. So rejecting the config at load here would reject something the other PR makes valid

What was genuinely wrong standing alone was the error. An absent tier reported "No candidate models left for tier MEDIUM after routing-plugin filtering", blaming plugins for a filter they never ran and sending an operator to read plugin code for a gap in tiers. The two cases now raise separately, with a test on each

Also worth noting for the merge order: this config shape already fails today for every prompt scoring into the missing tier's band, so it is a pre-existing gap that no-signal traffic newly joins, not one this PR creates

@greptileai

@greptile-apps

greptile-apps Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

The handling in be7b23c is correct. The empty-pool check fires before the plugin loop, raises with a message that names the missing tier and explains why default_model is not consulted, and keeps itself distinct from the plugin-narrowed-to-zero path that legitimately says "after routing-plugin filtering". The two cases now carry accurate diagnostics, and the tests cover both paths independently.

The decision not to tighten the validator on the implicit default also holds. The implicit MEDIUM case is not a config authoring error — it is a valid partial-tiers map that happens to expose a gap only when no-signal traffic arrives, and #35331 closes that gap at runtime. Rejecting it at load would be the wrong boundary.

Good resolution overall. Thanks for the detailed walkthrough.

@tin-berri

Copy link
Copy Markdown
Contributor Author

@greptile can you reply here with a score and why that score?

@tin-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@greptile-apps

greptile-apps Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Score: 4 / 5


Why not 5

Backwards-incompatible default behavior change (rule b48b7341)

The PR changes default_tier from an implicit SIMPLE to MEDIUM for all existing deployments with zero config change on their side. The PR description is transparent about this (+53% per-turn cost on the agent corpus, 84/13/3/0 → 47/49/3/0 tier mix) and documents the one-line opt-out (default_tier: SIMPLE), but the opt-out is still opt-out. Every current user of the heuristic scorer gets the new behavior automatically unless they add that field.

The rule says to avoid backwards-incompatible changes without user-controlled flags — there is a flag here, but it requires users to find the release notes and act, rather than the safer pattern of defaulting to the old behavior and requiring users to opt in to the new one:

# Safer rollout: old behavior is still the default, new behavior is opt-in
default_tier: SIMPLE   # current default; change to MEDIUM to route unmatched traffic to mid-tier

The PR description notes this needs a "release-note call-out" but there is no release note in the diff. That gap is the main reason for the deduction.


Why it scores 4 and not lower

Core change is minimal and correct. The three-line fix in _score_and_classify is placed exactly right — after the reasoning_override guard, before the band mapping — and reads signals rather than the weighted score, which is the correct discriminator. The cancelling-dimensions case ("hi, quick python question" → score 0.0 with three dimensions firing → stays SIMPLE) is precisely the edge case that would have broken a score-based guard.

Config validation is proactive. An explicit default_tier naming a tier with no model is rejected at load time rather than on the first unmatched request. The plugin path gets its own error message rather than blaming the plugin for a pool that was never populated. Both are good UX.

Test coverage is thorough. The suite pins: the abstain itself, the cancelling-weights case, the bit-exact-zero-with-signal case (so weighted_score == 0.0 is a live substitution the suite would catch), the SIMPLE-only and short-prompt-only edges, all four default_tier values, escalation on top of an abstain, the plugin/default_model interaction, the implicit-default compatibility guard, and a 13-prompt corpus that prevents silent drift. The dimension-signal invariant test (test_a_signal_is_emitted_exactly_when_a_dimension_scores) correctly pins the assumption the abstain branch depends on.

e2e fixture change is correct. Adding default_tier: SIMPLE to the fixture preserves the test's classifier-vs-fallback discriminator — without it the no-signal fallback prompt would route to MEDIUM on both arms and the visibility assertion would silently pass for the wrong reason. This is adapting to the new default, not weakening the test.

No other rule violations. No FastAPI imports outside proxy/, no direct DB queries in the critical path, no provider-specific code outside llms/, no hardcoded model flags.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit be7b23c. Configure here.

@tin-berri

Copy link
Copy Markdown
Contributor Author

Re-requesting a score on the current head (be7b23c), which added the accurate error for an unconfigured tier under plugins, on top of the validator fix in 9879bae

@greptileai can you reply here with a score and why that score?

Comment thread litellm/router_strategy/complexity_router/config.py Outdated
…d and routing

Three review rounds landed three P1s on the same validator, each one a boolean
added to close one more way it disagreed with request-time resolution: first that
an unservable default_tier was accepted at all, then that the plugin path never
consults default_model, now that a tier present with an empty pool is not the same
as a tier absent. They are one defect. `_validate_default_tier_is_servable` was a
second, hand-maintained model of what `get_model_for_tier` does, so it drifted from
it one case at a time.

`ComplexityRouterConfig.resolve_tier` is now the single answer to "what may serve
this tier", returning models or the reason there are none. The validator asks it
and so does selection, so there is no precedence left to re-derive and nothing to
drift; the validator is four lines with no conditions of its own.

Resolution keys off a tier's models rather than its key being present, which is the
third finding fixed where it lives rather than mirrored into config: `{MEDIUM: []}`
and a tiers map with no MEDIUM both say the tier has no models, so both fall
through to default_model. Mirroring the old asymmetry into the validator instead
would have made it part of the config contract.

The deployment-level complexity_router_default_model now goes in before validation
rather than being assigned onto the validated model afterwards. router.py always
derives one from the MEDIUM-then-SIMPLE tier, so an explicit default_tier outside
`tiers` is servable on every proxy deployment; validating before it was applied
failed those configs at startup for a gap routing did not have.

Drops _pick_from_tier_value and the hand-rolled plugin-path raise, both subsumed.
Tests pin the invariant rather than the instances: a config the validator accepts
is one the default tier can be served from, across the tier's own models,
default_model at either level, and an empty pool falling through.
@tin-berri

Copy link
Copy Markdown
Contributor Author

@greptileai

Comment thread litellm/router_strategy/complexity_router/config.py Outdated
tin-berri added a commit to BerriAI/litellm-docs that referenced this pull request Aug 1, 2026
… signal

The heuristic scorer has no abstain path today, so a prompt matching none of its
keyword lists scores 0.0 and falls under simple_medium into SIMPLE. BerriAI/litellm#35050
adds default_tier (MEDIUM by default) for that case; this documents the knob, the
decision-log line it produces, and the reason the check is on the dimensions rather
than on the weighted score
…explicit one

Abstaining moved no-signal prompts off SIMPLE and onto MEDIUM, so with routing
plugins configured and no MEDIUM models a config that served those prompts before
now raises on every one of them: the plugin path stops at the tier's own models,
and default_model, which router.py derives on every proxy deployment, is exactly
what the plugins never vetted. Greptile flagged it as an accepted config that fails
at request time, which is the same class the resolver was meant to close.

The exemption for an implicit default_tier is what left it open, and it was earning
its keep against a check that no longer exists. It was there so a partial `tiers`
map would not become a startup failure, back when the check was "must be its own
non-empty entry in tiers". Through resolve_tier the defaulted MEDIUM resolves the
way any classified tier does, so a partial map backed by default_model still loads
and only a config that genuinely cannot serve the tier is rejected. On the proxy
that is one shape: plugins configured, MEDIUM with no models of its own.

So the default is now checked like any value a user typed. Every prompt the scorer
recognises nothing in lands on this tier; a config that cannot serve it is broken
for a whole class of traffic and says so at load rather than on the first such
request. Nothing that loads can now fail to route a no-signal prompt.

Test configs that named only SIMPLE now pin default_tier: SIMPLE, which is the
behavior they were written against and leaves their tier pools untouched.
@tin-berri

Copy link
Copy Markdown
Contributor Author

Pushed the fix for the implicit-default finding. @greptileai

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant