[Feat] Add timeouts to VLLM Router configuration#983
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces support for configuring timeouts on route rules in the Helm chart, along with corresponding test cases. The reviewer pointed out a potential YAML parsing issue in values.yaml if a user uncomments the example timeout configuration directly, and suggested a safer way to structure the commented-out example.
| timeouts: {} | ||
| # requests: 300s |
There was a problem hiding this comment.
If a user uncomments the # requests: 300s line directly, it will result in invalid YAML because timeouts is already defined as an inline empty map (timeouts: {}) on the previous line.
To prevent YAML parsing errors when users try to uncomment the example, it is better to comment out the entire block as an example and keep the default timeouts: {} separate.
# timeouts:
# requests: 300s
timeouts: {}Signed-off-by: Peter De Sousa <pd@s-ai.com>
01dda24 to
ed19ef6
Compare
Signed-off-by: Peter De Sousa <pd@s-ai.com>
fdc551b to
681b620
Compare
FILL IN THE PR DESCRIPTION HERE
FIX #706
Currently the router configuration for the backend will be created with defaults, for some models and scenarios this means the Gateway is timing out before vLLM responds. This is a quick fix to allow users to create a timeout. Long term a more elegant solution would be appropriate.
-swhen doinggit commit[Bugfix],[Feat], and[CI].