Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/proxy/request_headers.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ By default, LiteLLM does not forward client headers to LLM provider APIs. Howeve

`x-litellm-tags`: Optional[str]: A comma separated list (e.g. `tag1,tag2,tag3`) of tags to use for [tag-based routing](./tag_routing) **OR** [spend-tracking](./enterprise.md#tracking-spend-for-custom-tags).

`x-litellm-num-retries`: Optional[int]: The number of retries for the request.
`x-litellm-num-retries`: Optional[int]: The number of retries for the request. This outranks a `num_retries` in the request body, in a deployment's `litellm_params`, and in `litellm_settings`. [Learn More](../routing#where-num_retries-can-be-set-and-which-one-wins)

`x-litellm-spend-logs-metadata`: Optional[str]: JSON string containing custom metadata to include in spend logs. Example: `{"user_id": "12345", "project_id": "proj_abc", "request_type": "chat_completion"}`. [Learn More](./cost_tracking)

Expand Down
20 changes: 20 additions & 0 deletions docs/routing.md
Original file line number Diff line number Diff line change
Expand Up @@ -1428,6 +1428,26 @@ response = router.completion(model="gpt-3.5-turbo", messages=messages)
print(f"response: {response}")
```

#### Where `num_retries` can be set, and which one wins

`num_retries` can come from four places. They are ranked, highest first:

1. the `x-litellm-num-retries` request header (proxy only)
2. `num_retries` in the request body
3. `num_retries` in a deployment's `litellm_params` in `model_list`
4. `num_retries` in `litellm_settings` (the router-wide default)

So a caller can always raise or lower the retry count for one request, including setting it to `0` to
disable retries, no matter what the deployment or the global setting says. A deployment value applies
whenever the request carries none, and it overrides the global default.

`num_retries` is not the same knob as `max_retries`. `num_retries` is LiteLLM's own retry loop, while
`max_retries` is the provider SDK's internal retry count. For a call that goes through the router,
LiteLLM owns retries and pins the provider client to `max_retries: 0`, so a `max_retries` in the
request body or in `litellm_params` has no effect on a proxy request. That is deliberate: it is what
stops a deployment `num_retries: N` from being applied twice and turning one request into
`(1 + N) ** 2` upstream calls. Use `num_retries` to control how many attempts a request gets.

### [Advanced]: Custom Retries, Cooldowns based on Error Type

- Use `RetryPolicy` if you want to set a `num_retries` based on the Exception received
Expand Down