Skip to content

Fix ValidationError when re-exporting benchmarks with console output#926

Closed
martin-pfister wants to merge 1 commit into
vllm-project:mainfrom
martin-pfister:fix/reimport-console-output-path
Closed

Fix ValidationError when re-exporting benchmarks with console output#926
martin-pfister wants to merge 1 commit into
vllm-project:mainfrom
martin-pfister:fix/reimport-console-output-path

Conversation

@martin-pfister

@martin-pfister martin-pfister commented Jul 10, 2026

Copy link
Copy Markdown

Summary

  • reimport_benchmarks_report() (used by guidellm benchmark from-file) unconditionally attached a file path to every requested output format.
  • ConsoleBenchmarkOutputArgs forbids extra fields (no path), so requesting console output — the default — raised pydantic_core.ValidationError: console.path Extra inputs are not permitted.
  • Now a path is only attached when the resolved output args class for that format actually declares a path field, so console output (and any future path-less format) works while file-based formats (json/yaml/html/csv) keep their paths.

Fixes #925

Test plan

  • Added a regression test (test_reimport_with_console_format_does_not_raise) that reproduces the exact crash from the issue and asserts it no longer raises.
  • ruff check and mypy pass on the changed files.
  • pytest tests/unit/entrypoints tests/unit/benchmark passes (75 passed, pre-existing unrelated skips only).
  • Manually reproduced the crash on main and confirmed the fix resolves it via guidellm benchmark from-file benchmarks.json --output-formats console.

git log

commit cbd16c5
Author: Martin Pfister 30149150+martin-pfister@users.noreply.github.com
Date: Fri Jul 10 15:11:43 2026 +0000

Fix ValidationError when re-exporting benchmarks with console output

reimport_benchmarks_report() unconditionally attached a file path to
every requested output format, but ConsoleBenchmarkOutputArgs forbids
extra fields since console output has no file to write to. This broke
`guidellm benchmark from-file` whenever console was among the requested
formats (the default).

Only attach a path when the resolved output args class actually
declares a `path` field.

Fixes #925

Signed-off-by: Martin Pfister <30149150+martin-pfister@users.noreply.github.com>

Signed-off-by: Martin Pfister 30149150+martin-pfister@users.noreply.github.com

@mergify

mergify Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Hi @martin-pfister, the DCO check has failed. Please click on DCO in the Checks section for instructions on how to resolve this.

reimport_benchmarks_report() unconditionally attached a file path to
every requested output format, but ConsoleBenchmarkOutputArgs forbids
extra fields since console output has no file to write to. This broke
`guidellm benchmark from-file` whenever console was among the requested
formats (the default).

Only attach a path when the resolved output args class actually
declares a `path` field.

Fixes vllm-project#925

Signed-off-by: Martin Pfister <30149150+martin-pfister@users.noreply.github.com>
@martin-pfister martin-pfister force-pushed the fix/reimport-console-output-path branch from d7155e2 to cbd16c5 Compare July 10, 2026 15:28

@dbutenhof dbutenhof left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While this approach may work in the short term, it's a goal to upgrade the benchmark from-file and preprocess dataset subcommands to match the modern registry-friendly format of the 0.7.x run command. That is, --output kind=console --output kind=json,path=bmark.json. (Unfortunately we had a hard deadline to complete 0.7.0 and we missed this.)

We'd prefer to complete the CLI revision rather than take an intermediate short-term fix like this, although we appreciate the contribution and may still decide it's worth accepting.

@dbutenhof

Copy link
Copy Markdown
Collaborator

See #927

@dbutenhof

Copy link
Copy Markdown
Collaborator

Closing because the issue has been resolved by #927.

But thanks for donating the time & effort to try to help!

@dbutenhof dbutenhof closed this Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

guidellm benchmark from-file crashes with ValidationError when output includes console format

2 participants