diff --git a/docs/proxy/request_headers.md b/docs/proxy/request_headers.md index 2e9c26fde..6e24637f1 100644 --- a/docs/proxy/request_headers.md +++ b/docs/proxy/request_headers.md @@ -16,7 +16,7 @@ By default, LiteLLM does not forward client headers to LLM provider APIs. Howeve `x-litellm-tags`: Optional[str]: A comma separated list (e.g. `tag1,tag2,tag3`) of tags to use for [tag-based routing](./tag_routing) **OR** [spend-tracking](./enterprise.md#tracking-spend-for-custom-tags). -`x-litellm-num-retries`: Optional[int]: The number of retries for the request. +`x-litellm-num-retries`: Optional[int]: The number of retries for the request. This outranks a `num_retries` in the request body, in a deployment's `litellm_params`, and in `litellm_settings`. [Learn More](../routing#where-num_retries-can-be-set-and-which-one-wins) `x-litellm-spend-logs-metadata`: Optional[str]: JSON string containing custom metadata to include in spend logs. Example: `{"user_id": "12345", "project_id": "proj_abc", "request_type": "chat_completion"}`. [Learn More](./cost_tracking) diff --git a/docs/routing.md b/docs/routing.md index b11e440d9..f63104896 100644 --- a/docs/routing.md +++ b/docs/routing.md @@ -1428,6 +1428,26 @@ response = router.completion(model="gpt-3.5-turbo", messages=messages) print(f"response: {response}") ``` +#### Where `num_retries` can be set, and which one wins + +`num_retries` can come from four places. They are ranked, highest first: + +1. the `x-litellm-num-retries` request header (proxy only) +2. `num_retries` in the request body +3. `num_retries` in a deployment's `litellm_params` in `model_list` +4. `num_retries` in `litellm_settings` (the router-wide default) + +So a caller can always raise or lower the retry count for one request, including setting it to `0` to +disable retries, no matter what the deployment or the global setting says. A deployment value applies +whenever the request carries none, and it overrides the global default. + +`num_retries` is not the same knob as `max_retries`. `num_retries` is LiteLLM's own retry loop, while +`max_retries` is the provider SDK's internal retry count. For a call that goes through the router, +LiteLLM owns retries and pins the provider client to `max_retries: 0`, so a `max_retries` in the +request body or in `litellm_params` has no effect on a proxy request. That is deliberate: it is what +stops a deployment `num_retries: N` from being applied twice and turning one request into +`(1 + N) ** 2` upstream calls. Use `num_retries` to control how many attempts a request gets. + ### [Advanced]: Custom Retries, Cooldowns based on Error Type - Use `RetryPolicy` if you want to set a `num_retries` based on the Exception received