fix(complexity_router): route no-signal prompts to default_tier, not SIMPLE - #35050
fix(complexity_router): route no-signal prompts to default_tier, not SIMPLE#35050tin-berri wants to merge 6 commits into
Conversation
Greptile SummaryThe PR changes unmatched-prompt routing and consolidates tier servability resolution.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| litellm/router_strategy/complexity_router/complexity_router.py | Adds no-signal classification and consistently delegates model selection to the shared tier resolver. |
| litellm/router_strategy/complexity_router/config.py | Adds configurable default-tier validation and centralizes tier-to-model resolution across direct, fallback, and plugin-backed configurations. |
| litellm/types/utils.py | Extends the routing-decision cause contract with the no-signal default classification. |
| tests/test_litellm/router_strategy/test_complexity_router.py | Adds broad regression coverage for no-signal routing and the previously reported tier-servability cases. |
| tests/e2e/router/conftest.py | Pins legacy SIMPLE fallback behavior so the classifier fallback discriminator remains observable. |
| tests/e2e/router/test_complexity_router_e2e.py | Updates the end-to-end test explanation for the explicitly configured legacy fallback behavior. |
| litellm/router_strategy/complexity_router/README.md | Documents the new no-signal default behavior and its configuration. |
Reviews (7): Last reviewed commit: "docs(complexity_router): the servability..." | Re-trigger Greptile
|
Thanks, both findings addressed or answered in 153fc17 Inconsistent default-tier configurations (the score driver). Fixed, with one deliberate narrowing. Product documentation in the code repository. Keeping the README change, deliberately. That file already documents this exact class of behavior (the reasoning override, system-prompt handling, boundaries, weights), so the new section sits next to the behavior it describes rather than opening a new documentation surface. The customer-facing page is the one on the docs site, and it is updated in BerriAI/litellm-docs#703, which also carries the release-note wording for the spend increase |
|
Both fixes look correct and well-implemented. The The predicate for abstain ( The validator test for an empty pool ( Both findings are addressed properly. LGTM. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
Pushed a0a78af after that review: message-only change to the same validator. It ran before ComplexityRouter.__init__ applies the deployment-level |
a0a78af to
4c71c8a
Compare
|
Rebased onto latest staging and extended per review discussion, now at 4c71c8a The rebase was not clean: #35016 (routing-decision provenance), #35185 (classifier prior-turn context) and #35300 landed in this file since the PR opened. The abstain is re-applied on top of the new shape, and it now carries its own Second half added: tier resolution now walks a ladder instead of resolving to whatever was configured. Live models in the classified tier first, so a dead pool member is passed over for its peers, then upward a tier at a time, then |
…SIMPLE The heuristic scorer has no way to say "I don't know". Its seven dimensions are a roughly 100-word software-vocabulary whitelist, so when a prompt matches none of them every dimension contributes 0, the weighted sum is 0.0, and 0.0 sits below simple_medium (0.15). Absence of evidence was being scored as evidence of simplicity, and unclassifiable traffic went to the cheapest tier. On a graded 809-question benchmark that no-signal mass is 50.2% of prompts; on a 257-session agent transcript corpus it is 36.6% of turns _score_and_classify now returns config.default_tier before the band mapping when nothing was recognised, under its own cause=no_signal_default with signals=['no-signal'], so a spend log row says the score did not choose the tier, the way reasoning_override already does. default_tier is new on ComplexityRouterConfig and defaults to MEDIUM; setting it to SIMPLE restores the previous behavior exactly. The LLM classifier falls back to this scorer on timeout or error, so the same setting decides where unmatched traffic lands during a classifier outage The branch tests the signals rather than the score, and rather than the individual dimension scores. A weighted score of zero does not mean nothing was recognised, since contributions cancel: "hi, quick python question" comes to zero with three dimensions firing and is real evidence of a simple request. A dimension scoring zero does not mean it stayed silent either, because _score_keyword_match takes the no-match score as a parameter, so a future dimension with a nonzero baseline would silently kill the branch. A signal is the one thing a dimension emits only when it recognised something, and a test pins that invariant across every scorer An explicitly configured default_tier with no model behind it is rejected at load rather than surfacing as a routing error on the first unmatched request. The check is skipped when the field is left implicit, so a partial tiers map keeps loading as it does today instead of turning an upgrade into a startup failure The router e2e config pins default_tier: SIMPLE. That test tells a live LLM classifier apart from a silent fallback by which backend served the request, and its prompt has no scoring signal, so leaving the new default in place would have made both arms land on the same backend and turned the test into a false green
4c71c8a to
2d25d23
Compare
|
Split into two PRs at Tin's request, and rebased onto latest staging (3c2264c). This PR is now the classification half only; the tier fallback ladder moved to #35331, also based on staging rather than stacked on this one, so the two can land in either order One change since the last review, from a question about the predicate: it now reads Proof in the description was re-captured on this branch's own head (2d25d23) |
…ault_tier under plugins The validator accepted default_model as evidence that an explicitly configured default_tier could be served. The plugin path never consults it: candidates come from the tier pool alone, and a pool that is empty raises rather than falling through to a model the plugins never vetted. So a config with plugins, an explicit default_tier absent from tiers, and a default_model passed validation and then failed on its first no-signal request, which is exactly what validating at load is meant to prevent default_model now counts only when no plugins are configured, and the error names which of the two remedies applies
|
Fixed in 9879bae. The finding is real, and it bites this PR on its own rather than only in combination with #35331 Reproduced before fixing: with routing plugins configured, an explicit
A regression test covers the plugin case, alongside the existing ones for the no-plugin case where |
|
The fix in 9879bae is correct and the reasoning is sound. The validator now correctly distinguishes the two paths: without plugins, A few specifics worth noting:
The fix is complete and both halves of the policy are now pinned. LGTM. |
…s no models for it With routing plugins configured, a tier absent from `tiers` produced "No candidate models left for tier MEDIUM after routing-plugin filtering". Nothing was filtered: the pool was empty before any plugin ran, so the message sent an operator to read plugin code for what is a gap in `tiers` The two cases now raise separately. An empty pool names the tier and says why default_model is not consulted; a plugin narrowing a real pool to zero keeps its own message, since that one really is a policy decision This is the path no-signal traffic now takes, because the implicit MEDIUM default is deliberately not validated at load: doing so would turn every partial tiers map into a startup failure. The tier ladder in #35331 resolves the same shape at runtime by climbing to the next configured tier, so this is the standalone behavior
|
Yes, the finding you quoted was fixed in 9879bae, and Greptile's follow-up on that commit confirmed it. Its own words: "Accepting it with plugins would create a config that validates clean and then raises on the first no-signal request, exactly the gap the validator exists to close" Re-reading it, though, there was a sibling case still open, through the implicit default rather than the explicit one. Plugins configured, MEDIUM absent from I did not close it by tightening the validator, and that is deliberate. The implicit default is not validated so that a partial What was genuinely wrong standing alone was the error. An absent tier reported "No candidate models left for tier MEDIUM after routing-plugin filtering", blaming plugins for a filter they never ran and sending an operator to read plugin code for a gap in Also worth noting for the merge order: this config shape already fails today for every prompt scoring into the missing tier's band, so it is a pre-existing gap that no-signal traffic newly joins, not one this PR creates |
|
The handling in be7b23c is correct. The empty-pool check fires before the plugin loop, raises with a message that names the missing tier and explains why The decision not to tighten the validator on the implicit default also holds. The implicit MEDIUM case is not a config authoring error — it is a valid partial-tiers map that happens to expose a gap only when no-signal traffic arrives, and #35331 closes that gap at runtime. Rejecting it at load would be the wrong boundary. Good resolution overall. Thanks for the detailed walkthrough. |
|
@greptile can you reply here with a score and why that score? |
|
bugbot run |
|
Score: 4 / 5 Why not 5Backwards-incompatible default behavior change (rule b48b7341) The PR changes The rule says to avoid backwards-incompatible changes without user-controlled flags — there is a flag here, but it requires users to find the release notes and act, rather than the safer pattern of defaulting to the old behavior and requiring users to opt in to the new one: # Safer rollout: old behavior is still the default, new behavior is opt-in
default_tier: SIMPLE # current default; change to MEDIUM to route unmatched traffic to mid-tierThe PR description notes this needs a "release-note call-out" but there is no release note in the diff. That gap is the main reason for the deduction. Why it scores 4 and not lowerCore change is minimal and correct. The three-line fix in Config validation is proactive. An explicit Test coverage is thorough. The suite pins: the abstain itself, the cancelling-weights case, the bit-exact-zero-with-signal case (so e2e fixture change is correct. Adding No other rule violations. No FastAPI imports outside |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit be7b23c. Configure here.
|
Re-requesting a score on the current head (be7b23c), which added the accurate error for an unconfigured tier under plugins, on top of the validator fix in 9879bae @greptileai can you reply here with a score and why that score? |
…d and routing
Three review rounds landed three P1s on the same validator, each one a boolean
added to close one more way it disagreed with request-time resolution: first that
an unservable default_tier was accepted at all, then that the plugin path never
consults default_model, now that a tier present with an empty pool is not the same
as a tier absent. They are one defect. `_validate_default_tier_is_servable` was a
second, hand-maintained model of what `get_model_for_tier` does, so it drifted from
it one case at a time.
`ComplexityRouterConfig.resolve_tier` is now the single answer to "what may serve
this tier", returning models or the reason there are none. The validator asks it
and so does selection, so there is no precedence left to re-derive and nothing to
drift; the validator is four lines with no conditions of its own.
Resolution keys off a tier's models rather than its key being present, which is the
third finding fixed where it lives rather than mirrored into config: `{MEDIUM: []}`
and a tiers map with no MEDIUM both say the tier has no models, so both fall
through to default_model. Mirroring the old asymmetry into the validator instead
would have made it part of the config contract.
The deployment-level complexity_router_default_model now goes in before validation
rather than being assigned onto the validated model afterwards. router.py always
derives one from the MEDIUM-then-SIMPLE tier, so an explicit default_tier outside
`tiers` is servable on every proxy deployment; validating before it was applied
failed those configs at startup for a gap routing did not have.
Drops _pick_from_tier_value and the hand-rolled plugin-path raise, both subsumed.
Tests pin the invariant rather than the instances: a config the validator accepts
is one the default tier can be served from, across the tier's own models,
default_model at either level, and an empty pool falling through.
… signal The heuristic scorer has no abstain path today, so a prompt matching none of its keyword lists scores 0.0 and falls under simple_medium into SIMPLE. BerriAI/litellm#35050 adds default_tier (MEDIUM by default) for that case; this documents the knob, the decision-log line it produces, and the reason the check is on the dimensions rather than on the weighted score
…explicit one Abstaining moved no-signal prompts off SIMPLE and onto MEDIUM, so with routing plugins configured and no MEDIUM models a config that served those prompts before now raises on every one of them: the plugin path stops at the tier's own models, and default_model, which router.py derives on every proxy deployment, is exactly what the plugins never vetted. Greptile flagged it as an accepted config that fails at request time, which is the same class the resolver was meant to close. The exemption for an implicit default_tier is what left it open, and it was earning its keep against a check that no longer exists. It was there so a partial `tiers` map would not become a startup failure, back when the check was "must be its own non-empty entry in tiers". Through resolve_tier the defaulted MEDIUM resolves the way any classified tier does, so a partial map backed by default_model still loads and only a config that genuinely cannot serve the tier is rejected. On the proxy that is one shape: plugins configured, MEDIUM with no models of its own. So the default is now checked like any value a user typed. Every prompt the scorer recognises nothing in lands on this tier; a config that cannot serve it is broken for a whole class of traffic and says so at load rather than on the first such request. Nothing that loads can now fail to route a no-signal prompt. Test configs that named only SIMPLE now pin default_tier: SIMPLE, which is the behavior they were written against and leaves their tier pools untouched.
|
Pushed the fix for the implicit-default finding. @greptileai |
TLDR
Problem this solves:
simple_medium(0.15) into SIMPLE. Absence of evidence was scored as evidence of simplicity, on 50.2% of prompts in a graded 809-question benchmark and 36.6% of turns in a 257-session agent corpusHow it solves it:
_score_and_classifyreturns a newdefault_tier(MEDIUM by default) when nothing was recognised, under its own decision causeno_signal_defaultand withsignals=['no-signal']default_tier: SIMPLErestores the previous behavior exactly, for anyone who wants unmatched traffic on the cheapest tierRelevant issues
default_tierinstead of letting a 0.0 score fall through into SIMPLE, and logs them ascause=no_signal_defaultdefault_tier: SIMPLEin the router e2e fixture, whose classifier-vs-fallback discriminator is itself a no-signal prompt and would otherwise have gone false-greenLinear ticket
Resolves LIT-4898
The tier fallback ladder that came out of the same discussion is split into its own PR, #35331, based on staging rather than stacked on this one
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Live proxy on this branch at 2d25d23. Two routers over the same three models:
smart-routeron the new default, andsmart-router-legacycarryingdefault_tier: SIMPLE, which is the behavior on staging todayA prompt that matches nothing. It used to be asserted simple; it now abstains, and the row says why:
A prompt with real evidence of being simple, unchanged:
The case that keeps the fix honest. This one scores zero, but with three dimensions firing, so it is not silence and stays on the cheap tier:
The opt-out, same no-signal prompt against the router carrying
default_tier: SIMPLE:The upstream completions returned 429 (the OpenAI key on this box is out of quota, and the other provider keys available locally are unfunded), so these transcripts prove the routing decision and the deployment handoff, not the provider response. The runbook below reruns the same commands against a funded key for the 200s
Type
🐛 Bug Fix
Changes
_score_and_classifyreturnsconfig.default_tierwithsignals=['no-signal']andcause=no_signal_defaultwhen nothing was recognised, placed after the 2-or-more-reasoning-marker override and before the band mapping. The cause is a first-class value for the same reasonreasoning_overrideis: it is the fact that says the score did not choose the tier, and burying it insignalswould let anything that filters signals change what the row claimsThe branch reads
signals, not the weighted score, and not the individual dimension scores either. Both of those look like "nothing was recognised" without being it. The weighted score cancels:hi, quick python questionistokenCount -1.0x0.10plussimpleIndicators -1.0x0.05pluscodePresence 0.5x0.30, which comes to zero with three dimensions firing and is real evidence of a simple request. A per-dimension zero is not silence either, since_score_keyword_matchtakes the no-match score as a parameter, so a future dimension with a nonzero baseline would silently kill the branch. A signal is the one thing a dimension emits only when it recognised something, and a test pins that invariant across every scorer, in both directionsShort prompts (under 15 estimated tokens) and long ones (over 400) fire
tokenCountand score normally, so this path holds prompts of roughly 15 to 400 estimated tokens with zero keyword and pattern hitsdefault_tierdefaults to MEDIUM, so no config change turns it on; setting it only overrides.default_tier: SIMPLErestores the previous behaviordefault_tieris not a special tier. It names which tier no-signal traffic is classified as, and that tier resolves to a model through the same chain as any classified one.ComplexityRouterConfig.resolve_tieris that chain, returning either the models that may serve a tier or the reason none may, and both config validation and request-time selection go through it. So adefault_tierthat nothing can serve is rejected at load rather than surfacing on the first unmatched request, and every config that loads is one the default tier can actually be served fromThat check covers the defaulted MEDIUM as well as a value you typed. It was originally skipped when the field was left implicit, so that a partial
tiersmap would not become a startup failure; throughresolve_tierit no longer can, because the proxy always derives adefault_modelfrom the MEDIUM-then-SIMPLE tier and the chain terminates there. What is left is one shape, and it is the reason the exemption had to go: with routing plugins configured the chain stops at the tier's own models, since a model the plugins never vetted must not serve, so a plugin config whosetiershas no MEDIUM cannot serve no-signal traffic at all. Abstaining moved those prompts off SIMPLE, so before this it loaded and raised on every one of themTwo things fell out of putting that chain in one place. Resolution now keys off whether a tier has models rather than whether its key is present, so
tiers: {MEDIUM: []}and a tiers map with no MEDIUM at all behave the same; before, the empty one entered the pool and refused to pick from it while the absent one fell through todefault_model. And the deployment-levelcomplexity_router_default_modelis folded in before validation instead of being assigned onto the validated model afterwards.router.pyderives that value from the MEDIUM-then-SIMPLE tier whenever it is unset, so an explicitdefault_tieroutsidetiersis servable on every proxy deployment; validating before it was applied failed those configs at startup for a gap routing did not haveTests cover the abstain itself, the config validation, both cancelling cases (the shipped-weights one that lands on float dust, and a configured-weights one that lands on a bit-exact 0.0, so the cheaper predicate is a live substitution the suite rejects), the scorer signal invariant the predicate now depends on, the SIMPLE-only and short-prompt-only edges that must not abstain,
default_tieracross all four tiers, escalation on top of an abstain, and a fixed 13-prompt corpus pinning the tier of each so a later weight or keyword edit cannot walk the default back toward all-SIMPLE. The servability tests pin the invariant rather than the instances: a config the validator accepts is one the default tier can be served from, parameterized over the tier's own models,default_modelat either level, and an empty pool falling throughDocs: the router README gains a "No-Signal Default" section, and the proxy docs page is updated in a separate PR
Known limits, not fixed here
Unmatched prompts over 400 estimated tokens fire
tokenCount, score +0.10, and still land SIMPLE under the 0.15 boundary. Moving that boundary is a pricing decision tracked separately, as is thecomplex_reasoningcalibration. On the graded corpus this leaves hard-labelled questions on SIMPLE at 31.9%, down from 67.0%This alters spend for every deployment on the heuristic default with no config change on their side. On the agent corpus the blended per-turn price proxy goes from 2.78 to 4.24, about +53%, with tier mix moving 84/13/3/0 to 47/49/3/0. It needs a release-note call-out, and
default_tier: SIMPLEis the one-line opt-outQA runbook
make bootstrap, and write the config above tolit4898_abstain.yamlwith a fundedOPENAI_API_KEYin the environmentLITELLM_LOG=INFO python litellm/proxy/proxy_cli.py --config lit4898_abstain.yaml --port 4897return_raw_model_name: trueto either router's config puts the deployment that served into the responsemodelfield, so routing is visible without reading logssmart-routerand gpt-5.4-nano onsmart-router-legacy;What is 2+2?andhi, quick python questionserve gpt-5.4-nano on bothpython -m pytest tests/test_litellm/router_strategy/test_complexity_router.py -qfor the unit coverageFinal Attestation
Note
Medium Risk
Default routing behavior changes for existing deployments without config edits (more unmatched traffic to MEDIUM and higher spend), though default_tier: SIMPLE opt-out exists; routing logic is well-tested.
Overview
When the heuristic scorer gets no signals from any dimension, it no longer maps a 0.0 score into SIMPLE. It returns configurable
default_tier(default MEDIUM), logssignals=['no-signal'], and recordscause=no_signal_default. The branch keys off empty signals, not the weighted score, so prompts likehi, quick python questionthat cancel to zero but still have evidence stay on SIMPLE.default_tier: SIMPLErestores the old behavior. Explicitdefault_tiervalues are validated at config load so the tier has a servable model (with stricter rules when routing plugins are enabled). The plugin tier-pick path now errors clearly when a tier has no models configured instead of implying plugin filtering removed them.Docs,
RoutingDecisionCause, e2e fixturedefault_tier: SIMPLE(so classifier-vs-fallback tests stay valid), and broad unit/corpus tests cover abstain, config validation, and the dimension signal invariant.Reviewed by Cursor Bugbot for commit be7b23c. Bugbot is set up for automated code reviews on this repo. Configure here.