[FEAT] Add benchmark.sample_requests for GuideLLM output size control#27
Merged
Conversation
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
3 tasks
VincentG1234
added a commit
that referenced
this pull request
Jun 11, 2026
## Summary Add optional `benchmark.rampup` to study configs, forwarding GuideLLM's `--rampup` flag so concurrent benchmarks can ramp load linearly to target concurrency instead of starting at full rate. ## Why Sudden full-concurrency load can skew benchmark metrics (cold caches, queue buildup, OOM risk). GuideLLM supports a ramp-up period; this fork already exposes `warmup` and `cooldown` but not `rampup`, so users could not control how load increases at the start of a run. ## What changed - `auto_tune_vllm/benchmarks/config.py` — add optional `rampup` field; validate `> 0` in `__post_init__` - `auto_tune_vllm/benchmarks/providers.py` — pass `--rampup` to GuideLLM CLI when set - `tests/benchmarks/test_guidellm_command.py` — CLI construction and validation tests for rampup - `docs/configuration.md` — document `rampup` (seconds, included in metrics, unlike warmup) - `examples/study_config.yaml`, `examples/study_config_minimal.yaml` — commented example - `README.md` — fork changelog entry ## How tested - [x] `ruff check .` - [x] `pytest -v tests/benchmarks/test_guidellm_command.py` (11 passed) - [ ] Manual E2E (maintainer): `auto-tune-vllm optimize` with `benchmark.rampup: 10` and verify GuideLLM receives `--rampup 10` ## Risks / limitations - Requires a GuideLLM version that supports `--rampup` (same pattern as existing warmup/cooldown flags). - Ramp-up requests are included in reported metrics; only `warmup`/`cooldown` exclude phases from measurement. - No interaction with fractional warmup/cooldown sum validation (rampup is always absolute seconds). ## Links - Follows [#24](#24) (warmup/cooldown) and [#27](#27) (sample_requests) benchmark config extensions. Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Expose GuideLLM
--sample-requestsviabenchmark.sample_requestsin study YAML (default0) and bump the GuideLLM dependency to>= 0.5.4.Why
GuideLLM benchmark JSON files can grow very large when every request is stored. During Optuna studies this adds unnecessary disk I/O and storage overhead, while aggregate metrics used for optimization are unaffected. The
--sample-requestsflag regressed around GuideLLM v0.4 and was fixed in v0.5.4.What changed
auto_tune_vllm/benchmarks/config.py— addsample_requests: int = 0with validation (>= 0)auto_tune_vllm/benchmarks/providers.py— pass--sample-requeststo the GuideLLM CLI on every benchmark rundocs/configuration.md— documentsample_requestsand minimum GuideLLM versionexamples/study_config.yaml,examples/study_config_minimal.yaml— commented examplepyproject.toml,requirements.txt— pinguidellm>=0.5.4(fix for--sample-requests)How tested
ruff check .pytest -v tests/sample_requests: 20is setLinks