Skip to content

Switch vllm_bench to standalone vllm-bench Rust binary#32

Open
khluu wants to merge 1 commit into
mainfrom
claude/adoring-proskuriakova-8e925e
Open

Switch vllm_bench to standalone vllm-bench Rust binary#32
khluu wants to merge 1 commit into
mainfrom
claude/adoring-proskuriakova-8e925e

Conversation

@khluu

@khluu khluu commented Jul 1, 2026

Copy link
Copy Markdown
Member

Summary

  • Swap the vllm bench serve Python CLI (run via docker exec inside the vllm/vllm-openai container) for the standalone vllm-bench Rust binary, which runs on the host and talks HTTP to the served port. JSON schema is identical, so ingest_perf.py needed no logic changes.
  • The prebuilt Linux musl binary is auto-downloaded to ~/.cache/perf-eval/ on first use, pinned to v0.1.0. Override with VLLM_BENCH_VERSION or point at a local build with VLLM_BENCH_BIN.
  • Drop the SPEED-Bench dataset path from lib/run_vllm_bench.sh — it needed HuggingFace download + docker cp + in-container pandas install, and it was already effectively dead code (every workload uses dataset: random). Also remove the now-unused speed_bench_dataset_subset / speed_bench_category schema fields from parse_workload.py, the run.sh TSV reader, and the four AMD workloads (kimi_k2_5_mi300x, kimi_k2_5_mi355x, gpt_oss_120b_mi355x, deepseek_v4_pro_mi355x) that still carried them.
  • README updated to point at vllm-bench and drop the now-irrelevant dataset: speed_bench warning.

Net: lib/run_vllm_bench.sh shrinks from 168 → 100 lines and no longer needs docker exec, docker cp, or any in-container pip installs.

This PR was authored with assistance from Claude Code.

Test plan

  • bash -n on every helper in lib/
  • Parser stub against all 16 workloads — all parse clean, WORKLOAD_VLLM_BENCH_TSV reduced to 7 columns
  • Buildkite build on qwen3_5_h200 with BENCH_ONLY=1 to confirm the binary download + host-side invocation work end-to-end on a real GPU agent

Replace the `vllm bench serve` Python CLI (run via `docker exec` inside the
vllm/vllm-openai container) with the standalone vllm-bench Rust binary from
github.com/vllm-project/vllm-bench, which runs on the host and talks HTTP to
the served port. JSON schema is identical, so ingest_perf.py is unchanged.

The binary is downloaded on first use to ~/.cache/perf-eval/ and pinned to
v0.1.0 (override with VLLM_BENCH_VERSION / VLLM_BENCH_BIN).

Drops the SPEED-Bench dataset code path — it needed HuggingFace download,
`docker cp` of JSONL into the container, and in-container pandas install,
and it was already effectively dead (all workloads use `dataset: random`).
Removes the unused `speed_bench_dataset_subset` / `speed_bench_category`
fields from the schema and from the four AMD workloads that still carried
them.

This PR was authored with assistance from Claude Code.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant