Skip to content
24 changes: 23 additions & 1 deletion docs/proxy/auto_routing.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,9 @@ Every knob v2 exposes. All fields on `complexity_router_config` are optional exc
session_affinity: true
session_affinity_ttl_seconds: 3600

# Tier for prompts that match nothing at all
default_tier: MEDIUM

# Tune heuristic scorer boundaries and weights (all optional)
tier_boundaries:
simple_medium: 0.15
Expand Down Expand Up @@ -138,6 +141,24 @@ Three ways to pick a tier. Pick one; the router falls back to the heuristic scor

Two or more reasoning markers auto-routes to `REASONING` regardless of the weighted score.

**No signal.** The scorer only recognizes what is on its keyword lists, so a prompt can be genuinely hard and match none of them; a logic puzzle, a business tradeoff, a piece of prose to edit. When no dimension fires the router returns `default_tier` (MEDIUM by default) instead of consulting the boundaries, because absence of evidence is not evidence of simplicity. The decision log shows `signals=['no-signal']` for these. Set `default_tier: SIMPLE` to send unmatched traffic to the cheapest tier instead.

`default_tier` is not a special tier; it names which tier no-signal traffic is classified as, and that tier resolves to a model exactly like a classified one, through its own entry in `tiers`, then `default_model`, then the MEDIUM tier. It has to be servable by one of those whether you set it or take the default, and the config is rejected at load rather than failing on the first unmatched request. A partial `tiers` map is unaffected, since the proxy derives `complexity_router_default_model` from the MEDIUM-then-SIMPLE tier whenever you leave it unset and the chain terminates there.

With routing plugins configured that chain stops at the tier's own models, since a model the plugins never vetted must not serve. A plugin config therefore has to give `default_tier` models in `tiers`, or point it at a tier that has them; `default_model` will not stand in, and a plugin config whose `tiers` omits MEDIUM is rejected at load rather than raising on the first unmatched request.

:::warning Upgrading

This changes where unmatched traffic lands on existing deployments, with no config change on your side. On a 257-session agent corpus the tier mix moved from 84/13/3/0 to 47/49/3/0 (SIMPLE/MEDIUM/COMPLEX/REASONING) and the blended per-turn price proxy rose about 53%, because half the turns that were being served as SIMPLE had matched nothing rather than matched as simple. That is the point of the change, but it is a spend increase you should expect rather than discover. Pin `default_tier: SIMPLE` to keep the old behavior.

:::

```yaml
default_tier: MEDIUM # SIMPLE | MEDIUM | COMPLEX | REASONING
```

The check is on the individual dimensions rather than the weighted score, since contributions cancel: "hi, quick python question" comes to zero with three dimensions firing (short prompt, simple indicator, code keyword), and that is real evidence of a simple request, so it stays SIMPLE. Prompts under the `simple` token threshold or over the `complex` one fire `tokenCount` and score normally too. The LLM classifier falls back to this scorer when it times out or errors, so `default_tier` also decides where unmatched traffic lands during a classifier outage.

**LLM classifier.** Uses a small fast model (Haiku, gpt-4o-mini, whatever you point it at) with structured output. Goes through the same `Router` instance, so credentials, budgets, and fallbacks apply. Timeout, empty content, or schema mismatch falls back to the heuristic scorer.

```yaml
Expand Down Expand Up @@ -167,7 +188,7 @@ match_threshold: 0.5
A tier value can be a single model name or a list.

- **Single string:** pins the tier to one model.
- **List:** router random-picks per request (uniform), same idea as simple-shuffle. Empty pools raise at config load rather than falling through to `default_model`.
- **List:** router random-picks per request (uniform), same idea as simple-shuffle. An empty pool means the tier names no models, so it resolves like a tier you left out of the map entirely, falling through to `default_model`; with `adaptive: true` it is rejected at config load instead.
- **List + `adaptive: true`:** Thompson-sample across the pool. Cold requests sample only inside the classified tier so cost weights do not collapse initial traffic on the cheapest model. Models configured in multiple tiers use their minimum distance from the classified tier. Feedback from a later turn attributes back to the model that actually served the previous response.

## Session affinity
Expand Down Expand Up @@ -197,6 +218,7 @@ Every routing decision emits one greppable line naming its cause. `cause=` is gr

```
ComplexityRouter: routing decision cause=complexity_scorer, tier=SIMPLE, score=-0.150, signals=['short (7 tokens)', 'simple (what is)'], routed_model=gpt-4o-mini
ComplexityRouter: routing decision cause=no_signal_default, tier=MEDIUM, score=0.000, signals=['no-signal'], routed_model=gpt-4o
ComplexityRouter: routing decision cause=literal_keyword_match, tier=REASONING, routed_model=gpt-5.5
ComplexityRouter: routing decision cause=semantic_keyword_match, tier=REASONING, routed_model=gpt-5.5
ComplexityRouter: routing decision cause=llm_classifier, tier=COMPLEX, score=1.000, signals=['llm-classifier:COMPLEX'], routed_model=claude-sonnet-5
Expand Down
6 changes: 6 additions & 0 deletions release_notes/v1.95.0rc1/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,12 @@ pip install litellm==1.95.0rc1
</TabItem>
</Tabs>

:::danger Breaking Changes

**Auto-routing sends prompts that match no scoring dimension to `default_tier` (MEDIUM) instead of SIMPLE.** The heuristic scorer had no way to say "I don't know"; a prompt matching none of its keyword lists scored 0.0, and 0.0 fell under the `simple_medium` boundary into SIMPLE, so absence of evidence was read as evidence of simplicity. This applies to every deployment on the heuristic scorer with no config change on your side, and it also covers LLM-classifier timeouts and errors, which fall back to that scorer. On a 257-session agent corpus the tier mix moved from 84/13/3/0 to 47/49/3/0 (SIMPLE/MEDIUM/COMPLEX/REASONING) and the blended per-turn price proxy rose about 53%. Set `default_tier: SIMPLE` on `complexity_router_config` to keep the previous behavior. One config shape is now rejected at startup rather than at request time: auto-routers with `plugins` configured whose `tiers` gives MEDIUM no models, since routing plugins do not consult `default_model` and the tier has nothing else to fall back to. Give MEDIUM models in `tiers`, or point `default_tier` at a tier that has them. See [PR #35050](https://github.com/BerriAI/litellm/pull/35050).

:::

## Key Highlights

- **Claude Opus 5 on day zero, everywhere** - the new 1M-context Opus lands simultaneously on Anthropic, Amazon Bedrock (including the `us`, `eu`, `au`, `jp`, and `global` inference profiles), Google Vertex AI, and Azure AI Foundry, with adaptive thinking, xhigh reasoning effort, computer use, PDF input, and prompt caching all recorded in the cost map.
Expand Down