Fix ValidationError when re-exporting benchmarks with console output#926
Fix ValidationError when re-exporting benchmarks with console output#926martin-pfister wants to merge 1 commit into
Conversation
|
Hi @martin-pfister, the DCO check has failed. Please click on DCO in the Checks section for instructions on how to resolve this. |
reimport_benchmarks_report() unconditionally attached a file path to every requested output format, but ConsoleBenchmarkOutputArgs forbids extra fields since console output has no file to write to. This broke `guidellm benchmark from-file` whenever console was among the requested formats (the default). Only attach a path when the resolved output args class actually declares a `path` field. Fixes vllm-project#925 Signed-off-by: Martin Pfister <30149150+martin-pfister@users.noreply.github.com>
d7155e2 to
cbd16c5
Compare
dbutenhof
left a comment
There was a problem hiding this comment.
While this approach may work in the short term, it's a goal to upgrade the benchmark from-file and preprocess dataset subcommands to match the modern registry-friendly format of the 0.7.x run command. That is, --output kind=console --output kind=json,path=bmark.json. (Unfortunately we had a hard deadline to complete 0.7.0 and we missed this.)
We'd prefer to complete the CLI revision rather than take an intermediate short-term fix like this, although we appreciate the contribution and may still decide it's worth accepting.
|
See #927 |
|
Closing because the issue has been resolved by #927. But thanks for donating the time & effort to try to help! |
Summary
reimport_benchmarks_report()(used byguidellm benchmark from-file) unconditionally attached a file path to every requested output format.ConsoleBenchmarkOutputArgsforbids extra fields (nopath), so requesting console output — the default — raisedpydantic_core.ValidationError: console.path Extra inputs are not permitted.pathfield, so console output (and any future path-less format) works while file-based formats (json/yaml/html/csv) keep their paths.Fixes #925
Test plan
test_reimport_with_console_format_does_not_raise) that reproduces the exact crash from the issue and asserts it no longer raises.ruff checkandmypypass on the changed files.pytest tests/unit/entrypoints tests/unit/benchmarkpasses (75 passed, pre-existing unrelated skips only).mainand confirmed the fix resolves it viaguidellm benchmark from-file benchmarks.json --output-formats console.git log
commit cbd16c5
Author: Martin Pfister 30149150+martin-pfister@users.noreply.github.com
Date: Fri Jul 10 15:11:43 2026 +0000
Signed-off-by: Martin Pfister 30149150+martin-pfister@users.noreply.github.com