Modernize benchmark from-file

[dbutenhof]

Summary

Modernize guidellm benchmark from-file by adopting the full registry typed --output style.

Details

The guidellm benchmark from-file command in 0.7.0

1. Fails with --output-formats console due to an attempt to inject a path
2. Does not allow specifying file paths for each format
3. Lacks CLI style consistency with guidellm run

Replace the old --output-formats with the modern --output kind=<kind>,... style.

Test Plan

Manual command testing + tox

Related Issues

• Resolves guidellm benchmark from-file crashes with ValidationError when output includes console format #925

---

• "I certify that all code in this PR is my own, except as noted below."

Use of AI

• Includes code generated or substantially modified by an AI agent
• Includes tests generated or substantially modified by an AI agent

NOTE: the Generated-by or Assisted-by trailers should be used in git commit messages when code or tests were generated or substantially modified by an AI agent, as described in the project's DEVELOPING.md file.

---

git log

commit 32906a2
Author: David Butenhof dbutenho@redhat.com
Date: Mon Jul 13 10:03:20 2026 -0400

Modernize benchmark from-file

Signed-off-by: David Butenhof <dbutenho@redhat.com>

commit 346ed21
Author: David Butenhof dbutenho@redhat.com
Date: Mon Jul 13 12:33:42 2026 -0400

Migration update

Signed-off-by: David Butenhof <dbutenho@redhat.com>

commit c31c71d
Author: David Butenhof dbutenho@redhat.com
Date: Tue Jul 14 15:32:14 2026 -0400

Fix benchmark from-file tests

Replace the disabled re-export tests with working code.

Signed-off-by: David Butenhof <dbutenho@redhat.com>

---

Signed-off-by: David Butenhof dbutenho@redhat.com

[jaredoconnell]

It works well. One thing to consider is that it overwrites the old benchmark, which is fine, but if there is ever a bug in it then it could corrupt the results file.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[dbutenhof]

It works well. One thing to consider is that it overwrites the old benchmark, which is fine, but if there is ever a bug in it then it could corrupt the results file.

Certainly true -- but not new. I had a couple of ideas along the way that I decided not to pursue, but I'm not sure any of it makes much sense. The user controls the output file, and if they don't want to overwrite the input on kind=json they can easily supply a distinct pathname for that output.

[SkiHatDuckie]

Works for me as well. I wonder if it would be a good idea to fix the tests in test_benchmark_from_file_entrypoint.py at some point, so that manual testing isn't as necessary.

[dbutenhof]

Works for me as well. I wonder if it would be a good idea to fix the tests in test_benchmark_from_file_entrypoint.py at some point, so that manual testing isn't as necessary.

I hadn't actually noticed that there are no tests, because they're all disabled. I'm both tempted to fix that and also tempted to just merge the fix and worry about that later...

[jaredoconnell]

What is the benefit of the new dependency?

[dbutenhof]

ruff was unhappy about using Path in the async methods, which I found ... interesting. But faced with disabling the warning or using its recommended alternative (only for the tests), I figured I'd go with the alternative.

ASYNC240 Async functions should not use pathlib.Path methods, use trio.Path or anyio.path
   --> tests/unit/entrypoints/test_benchmark_from_file_entrypoint.py:226:9
    |
224 |     assert exported_yaml.exists()
225 |     assert json.loads(exported_json.read_text()) == json.loads(
226 |         saved_json_report.read_text()
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
227 |     )
228 |     parsed_yaml = yaml.safe_load(exported_yaml.read_text())

[jaredoconnell]

I don't love the use of a new test dependency, but it's a test dependency. It seems okay.

[dbutenhof]

I don't love the use of a new test dependency, but it's a test dependency. It seems okay.

What I find interesting is that we're doing similar things in product code. Possibly not directly in async methods, but I found several in blocking methods called by async methods, and since we don't seem to be disabling ASYNC240 its not clear why we haven't seen complaints about those usages. Maybe ruff just isn't smart enough to find them?