From 488e54404d53917f6d768c825c8053302d8f9658 Mon Sep 17 00:00:00 2001 From: khluu Date: Tue, 30 Jun 2026 17:40:04 -0700 Subject: [PATCH] Switch vllm_bench to standalone vllm-bench Rust binary MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the `vllm bench serve` Python CLI (run via `docker exec` inside the vllm/vllm-openai container) with the standalone vllm-bench Rust binary from github.com/vllm-project/vllm-bench, which runs on the host and talks HTTP to the served port. JSON schema is identical, so ingest_perf.py is unchanged. The binary is downloaded on first use to ~/.cache/perf-eval/ and pinned to v0.1.0 (override with VLLM_BENCH_VERSION / VLLM_BENCH_BIN). Drops the SPEED-Bench dataset code path — it needed HuggingFace download, `docker cp` of JSONL into the container, and in-container pandas install, and it was already effectively dead (all workloads use `dataset: random`). Removes the unused `speed_bench_dataset_subset` / `speed_bench_category` fields from the schema and from the four AMD workloads that still carried them. This PR was authored with assistance from Claude Code. Co-Authored-By: Claude --- README.md | 5 +- lib/ingest_perf.py | 6 +- lib/parse_workload.py | 5 +- lib/run.sh | 7 +- lib/run_vllm_bench.sh | 159 ++++++++------------------ workloads/deepseek_v4_pro_mi355x.yaml | 2 - workloads/gpt_oss_120b_mi355x.yaml | 2 - workloads/kimi_k2_5_mi300x.yaml | 2 - workloads/kimi_k2_5_mi355x.yaml | 2 - 9 files changed, 59 insertions(+), 131 deletions(-) diff --git a/README.md b/README.md index f01a6e6..0821164 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ A recipe has top-level metadata plus up to three eval blocks: - **`vllm:`** — *how the server runs.* Defines what model to serve and how (`model`, `serve_args`, optional image/env overrides). Required. - **`lm_eval:`** — *what accuracy to measure.* Lists lm-evaluation-harness tasks to run against the live server (e.g. `gsm8k`, `aime25`). Each task's score is saved under `results///`. Optional. -- **`vllm_bench:`** — *what perf to measure.* Lists `vllm bench serve` configs (input/output lengths, concurrency, dataset). Raw JSON is saved and ingested into the perf dashboard. Optional. +- **`vllm_bench:`** — *what perf to measure.* Lists [vllm-bench](https://github.com/vllm-project/vllm-bench) configs (input/output lengths, concurrency, dataset). Raw JSON is saved and ingested into the perf dashboard. Optional. - **`bfcl:`** — *function-calling eval.* Runs [BFCL](https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard) test categories against the live server. Some models need `--enable-auto-tool-choice` and `--tool-call-parser` in `serve_args`. Results are transformed to lm_eval format and ingested as `bfcl_` tasks. Optional. Include one or more of `lm_eval:` / `vllm_bench:` / `bfcl:` depending on what you want out of this recipe. @@ -89,7 +89,8 @@ A few things worth knowing: - **`nightly`** controls only the nightly schedule. Recipes with `nightly: false` (or omitted) are still triggerable explicitly via the `WORKLOADS` env var. - **`lm_eval.tasks` is a list** because each entry runs as a separate `lm_eval` invocation — `--num_fewshot` is a single global flag, so different shot counts need separate runs. Each task's results land in `results///`. - **`vllm_bench` runs first** if both blocks are present — that way perf-pipeline bugs surface quickly instead of waiting on a full lm-eval pass. -- **`vllm_bench` uses the `random` dataset with `--ignore-eos`** so every request prefills exactly `input_len` and decodes exactly `output_len` tokens — that's what makes the per-GPU decode throughput meaningful. Pair it with `backend: openai` (the `/v1/completions` endpoint) for exact token control. Avoid `dataset: speed_bench` for throughput numbers: it requires `--skip-tokenizer-init`, which makes `vllm bench serve` cap every request at a single output token, so output throughput reads as ~0. +- **`vllm_bench` uses the standalone [vllm-bench](https://github.com/vllm-project/vllm-bench) Rust client** (drop-in for `vllm bench serve` with the same flags and identical result-JSON schema). It runs on the host and talks HTTP to the served port, so no `docker exec` is needed. The prebuilt Linux binary is downloaded automatically on first use; pin a different release with `VLLM_BENCH_VERSION` or point at a local build with `VLLM_BENCH_BIN`. Only `dataset: random` is wired up here. +- **`vllm_bench` uses `random` with `--ignore-eos`** so every request prefills exactly `input_len` and decodes exactly `output_len` tokens — that's what makes the per-GPU decode throughput meaningful. Pair it with `backend: openai` (the `/v1/completions` endpoint) for exact token control. - **`bfcl` may need tool-call serve args.** Some models require `--enable-auto-tool-choice` and `--tool-call-parser` for function-calling; the parser warns if `--tool-call-parser` is absent. Each category runs as a separate generate + evaluate pass; scores appear on the eval dashboard as `bfcl_` tasks. - **`bfcl.maximum_step_limit`** caps how many inference steps BFCL allows per multi-turn turn (default 10 in perf-eval; BFCL upstream defaults to 20). Set it in the workload YAML, or override per-run with the `BFCL_MAXIMUM_STEP_LIMIT` env var (env wins over YAML). Useful for agentic / long multi-turn categories. - **`bfcl.max_test_cases`** subsamples a category instead of running the full set — e.g. `multi_turn` (~800 cases) down to 300. For aggregate groups with multiple subcategories, the cap is split evenly across subcategories (by BFCL id order within each). Set a single integer to cap every category, or a map per category (`multi_turn: 240`). Override per-run with `BFCL_MAX_TEST_CASES`. Scores are partial-eval only and are not comparable to full BFCL leaderboard numbers. diff --git a/lib/ingest_perf.py b/lib/ingest_perf.py index 5291d43..74478ca 100644 --- a/lib/ingest_perf.py +++ b/lib/ingest_perf.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Transform a `vllm bench serve` raw JSON result and POST it to the perf +"""Transform a vllm-bench raw JSON result and POST it to the perf dashboard's ingestion endpoint. The dashboard at perf.vllm.ai reads from the `vllm_perf_data_ingest` @@ -42,7 +42,7 @@ def post(endpoint: str, payload: dict) -> None: def transform(raw: dict, args: argparse.Namespace) -> dict: - """Map the raw `vllm bench serve` JSON to the dashboard's row shape.""" + """Map the raw vllm-bench JSON to the dashboard's row shape.""" tp = max(args.tp, 1) total_token_throughput = float(raw.get("total_token_throughput", 0) or 0) output_throughput = float(raw.get("output_throughput", 0) or 0) @@ -91,7 +91,7 @@ def transform(raw: dict, args: argparse.Namespace) -> dict: def main() -> int: p = argparse.ArgumentParser(description=__doc__) - p.add_argument("--raw-result", required=True, help="Raw JSON from `vllm bench serve --save-result`") + p.add_argument("--raw-result", required=True, help="Raw JSON from `vllm-bench --save-result`") p.add_argument("--device", required=True, help="Device tag (e.g. h200)") p.add_argument("--tp", type=int, required=True, help="Effective parallel-degree (TP * DP)") p.add_argument("--precision", required=True, help="Precision tag (e.g. fp8, bf16)") diff --git a/lib/parse_workload.py b/lib/parse_workload.py index b6b108e..8589c80 100644 --- a/lib/parse_workload.py +++ b/lib/parse_workload.py @@ -23,7 +23,6 @@ BENCH_FIELDS = { "name", "backend", "dataset", "input_len", "output_len", "num_prompts", "max_concurrency", - "speed_bench_dataset_subset", "speed_bench_category", } BENCH_REQUIRED = ("name", "input_len", "output_len", "num_prompts", "max_concurrency") BFCL_FIELDS = { @@ -115,7 +114,7 @@ def resolve_image(vllm: dict, profile: dict) -> tuple[str, str]: def parse_tp(serve_args: str) -> int: """Effective parallel degree (TP * DP) from serve_args; defaults to 1. - `vllm bench serve` reports aggregate throughput; we divide by this to get + vllm-bench reports aggregate throughput; we divide by this to get per-GPU metrics for the dashboard. """ toks = serve_args.split() @@ -205,8 +204,6 @@ def opt(key): str(c["output_len"]), str(c["num_prompts"]), str(c["max_concurrency"]), - opt("speed_bench_dataset_subset"), - opt("speed_bench_category"), ] ) ) diff --git a/lib/run.sh b/lib/run.sh index 78f88e5..7fe9423 100755 --- a/lib/run.sh +++ b/lib/run.sh @@ -36,16 +36,15 @@ start_server "$CONTAINER" "$PORT" "$WORKLOAD_IMAGE" "$WORKLOAD_MODEL" \ "$WORKLOAD_SERVE_ARGS" "$WORKLOAD_ENV" "$WORKLOAD_SERVER_RUNTIME" wait_healthy "$PORT" -# vllm bench serve runs first so we can validate perf flow without waiting +# vllm-bench runs first so we can validate perf flow without waiting # on a full lm_eval pass. Each config's raw json lands in # $RESULTS_DIR/bench-.json and is then transformed and POSTed to the # perf dashboard ingest endpoint. -while IFS=$'\t' read -r bname backend dataset isl osl nprompts conc speed_subset speed_category; do +while IFS=$'\t' read -r bname backend dataset isl osl nprompts conc; do [[ -z "$bname" ]] && continue run_vllm_bench "$CONTAINER" "$PORT" "$WORKLOAD_MODEL" \ "$bname" "$backend" "$dataset" "$isl" "$osl" "$nprompts" \ - "$conc" "$speed_subset" "$speed_category" \ - "$BENCH_TRUST_REMOTE_CODE" "$RESULTS_DIR" + "$conc" "$BENCH_TRUST_REMOTE_CODE" "$RESULTS_DIR" python3 "$DIR/ingest_perf.py" \ --raw-result "${RESULTS_DIR}/bench-${bname}.json" \ diff --git a/lib/run_vllm_bench.sh b/lib/run_vllm_bench.sh index edda073..0b90638 100644 --- a/lib/run_vllm_bench.sh +++ b/lib/run_vllm_bench.sh @@ -1,94 +1,63 @@ -# Run a single `vllm bench serve` config against the running vLLM container. +# Run a single vllm-bench config against the running vLLM server. # Source this from run.sh. # # Usage: # run_vllm_bench \ # \ -# \ # # -# Docker runtime invokes `vllm bench serve` inside the vllm/vllm-openai -# container via `docker exec`; native runtime invokes it directly. The raw -# JSON lands in "/bench-.json" so ingest_perf.py can pick -# it up. - -# vLLM's SpeedBench class expects a local .jsonl file built by -# NeMo's prepare.py — the bench CLI does not download the dataset itself. -SPEED_BENCH_PREPARE_URL="https://raw.githubusercontent.com/NVIDIA-NeMo/Skills/refs/heads/main/nemo_skills/dataset/speed-bench/prepare.py" - -pip_install_quiet() { - if python3 -c 'import sys; sys.exit(0 if sys.prefix != sys.base_prefix else 1)'; then - python3 -m pip install --quiet "$@" - else - PIP_BREAK_SYSTEM_PACKAGES=1 python3 -m pip install --user --quiet "$@" +# Uses the standalone vllm-bench Rust CLI from github.com/vllm-project/vllm-bench +# (drop-in for `vllm bench serve` with the same flags and identical result-JSON +# schema). The binary runs on the host and talks HTTP to the served port, so +# the arg is unused — kept for the existing call-site signature. +# The raw JSON lands in "/bench-.json" so ingest_perf.py can +# pick it up. + +# Pinned upstream tag for the prebuilt binary; bump as new releases land. +VLLM_BENCH_VERSION="${VLLM_BENCH_VERSION:-v0.1.0}" +VLLM_BENCH_BIN="${VLLM_BENCH_BIN:-}" + +ensure_vllm_bench() { + if [[ -n "$VLLM_BENCH_BIN" && -x "$VLLM_BENCH_BIN" ]]; then + return fi -} - -prepare_speed_bench_dataset() { - local container=$1 runtime=$2 subset=$3 category=$4 data_dir=$5 - - if [[ ! -s "${data_dir}/${subset}.jsonl" ]]; then - if ! python3 -c 'import importlib.util as u, sys; sys.exit(0 if all(u.find_spec(m) for m in ("datasets","numpy","pandas","tiktoken")) else 1)' 2>/dev/null; then - echo "--- :python: installing SPEED-Bench prep dependencies" - pip_install_quiet datasets numpy pandas tiktoken - fi - mkdir -p "$data_dir" - echo "--- :arrow_down: preparing SPEED-Bench ${subset} dataset in ${data_dir}" - python3 - "$SPEED_BENCH_PREPARE_URL" "$subset" "$category" "$data_dir" <<'PY' -import sys, urllib.request -from pathlib import Path - -prepare_url, subset, category, data_dir = sys.argv[1:] -source = urllib.request.urlopen(prepare_url, timeout=60).read() -ns = {"__name__": "speed_bench_prepare", "__file__": "prepare.py"} -exec(compile(source, "prepare.py", "exec"), ns) - -dataset = ns["load_dataset"]("nvidia/SPEED-Bench", subset, split="test") -if category: - dataset = dataset.filter(lambda ex: ex["category"] == category) -dataset = ns["_resolve_external_data"](dataset, subset) -dataset = dataset.map( - lambda ex: {"messages": [{"role": "user", "content": t} for t in ex["turns"]]}, - remove_columns=["turns"], -) -Path(data_dir).mkdir(parents=True, exist_ok=True) -dataset.to_json(Path(data_dir) / f"{subset}.jsonl") -PY + if command -v vllm-bench >/dev/null 2>&1; then + VLLM_BENCH_BIN="$(command -v vllm-bench)" + return fi - test -s "${data_dir}/${subset}.jsonl" - - # Docker runtime: ship the data into the container and make sure pandas is - # available there (vLLM's SpeedBench loads the JSONL via pandas). - if [[ "$runtime" != "native" ]]; then - docker exec "$container" mkdir -p "$data_dir" - docker cp "${data_dir}/." "${container}:${data_dir}/" - if ! docker exec "$container" python3 -c 'import pandas' 2>/dev/null; then - echo "--- :docker: installing pandas in vLLM container" - docker exec "$container" bash -lc \ - 'PIP_BREAK_SYSTEM_PACKAGES=1 python3 -m pip install --quiet pandas \ - || PIP_BREAK_SYSTEM_PACKAGES=1 python3 -m pip install --user --quiet pandas' - fi + local cache="${HOME}/.cache/perf-eval" + local bin="${cache}/vllm-bench-${VLLM_BENCH_VERSION}" + if [[ ! -x "$bin" ]]; then + mkdir -p "$cache" + local arch; arch="$(uname -m)" + local url="https://github.com/vllm-project/vllm-bench/releases/download/${VLLM_BENCH_VERSION}/vllm-bench-${arch}-linux-musl" + echo "--- :arrow_down: downloading vllm-bench ${VLLM_BENCH_VERSION} (${arch})" + curl -fsSL "$url" -o "${bin}.tmp" + chmod +x "${bin}.tmp" + mv "${bin}.tmp" "$bin" fi + VLLM_BENCH_BIN="$bin" } run_vllm_bench() { - local container=$1 port=$2 model=$3 name=$4 backend=$5 dataset=$6 + local _container=$1 port=$2 model=$3 name=$4 backend=$5 dataset=$6 local input_len=$7 output_len=$8 num_prompts=$9 max_concurrency=${10} - local speed_bench_dataset_subset=${11} speed_bench_category=${12} - local trust_remote_code=${13} outdir=${14} - local runtime="${WORKLOAD_SERVER_RUNTIME:-docker}" - local in_container_json="/tmp/bench-${name}.json" + local trust_remote_code=${11} outdir=${12} local host_json="${outdir}/bench-${name}.json" [[ "$backend" == "-" ]] && backend="" - [[ "$speed_bench_dataset_subset" == "-" ]] && speed_bench_dataset_subset="" - [[ "$speed_bench_category" == "-" ]] && speed_bench_category="" - echo "--- :stopwatch: vllm bench serve ${name} (dataset=${dataset} isl=${input_len} osl=${output_len} conc=${max_concurrency} n=${num_prompts})" + if [[ "$dataset" != "random" ]]; then + echo "unsupported vllm_bench dataset: $dataset" >&2 + return 2 + fi + + ensure_vllm_bench + + echo "--- :stopwatch: vllm-bench ${name} (isl=${input_len} osl=${output_len} conc=${max_concurrency} n=${num_prompts})" mkdir -p "$outdir" - local cmd=(vllm bench serve) - [[ "$runtime" != "native" ]] && cmd=(docker exec "$container" "${cmd[@]}") + local cmd=("$VLLM_BENCH_BIN") if [[ -n "$backend" ]]; then cmd+=(--backend "$backend" --base-url "http://127.0.0.1:${port}") @@ -99,51 +68,21 @@ run_vllm_bench() { cmd+=( --model "$model" - --dataset-name "$dataset" + --dataset-name random --num-prompts "$num_prompts" --max-concurrency "$max_concurrency" + # --ignore-eos forces every request to emit the full output_len; without it + # the model can stop early on the random prompt and decode throughput collapses. + --random-input-len "$input_len" + --random-output-len "$output_len" + --ignore-eos + --save-result + --result-filename "$host_json" ) [[ "$trust_remote_code" == "true" ]] && cmd+=(--trust-remote-code) - case "$dataset" in - random) - # --ignore-eos forces every request to emit the full output_len; without it - # the model can stop early on the random prompt and decode throughput collapses. - cmd+=(--random-input-len "$input_len" --random-output-len "$output_len" --ignore-eos) - ;; - speed_bench) - [[ -z "$speed_bench_dataset_subset" ]] && speed_bench_dataset_subset="qualitative" - local data_dir="${VLLM_SPEED_BENCH_DIR:-/tmp/vllm-speed-bench}/${speed_bench_dataset_subset}" - [[ -n "$speed_bench_category" ]] && data_dir="${data_dir}-${speed_bench_category}" - prepare_speed_bench_dataset "$container" "$runtime" \ - "$speed_bench_dataset_subset" "$speed_bench_category" "$data_dir" - # SPEED-Bench applies the client-side chat template at tokenizer init, - # which breaks for chat-template-less models — rely on server-side - # usage accounting instead. - cmd+=( - --dataset-path "$data_dir" - --speed-bench-output-len "$output_len" - --speed-bench-dataset-subset "$speed_bench_dataset_subset" - --skip-tokenizer-init - ) - [[ -n "$speed_bench_category" ]] && cmd+=(--speed-bench-category "$speed_bench_category") - ;; - *) - echo "unsupported vllm_bench dataset: $dataset" >&2 - return 2 - ;; - esac - - if [[ "$runtime" == "native" ]]; then - cmd+=(--save-result --result-filename "$host_json") - else - cmd+=(--save-result --result-filename "$in_container_json") - fi - "${cmd[@]}" - [[ "$runtime" != "native" ]] && docker cp "${container}:${in_container_json}" "$host_json" - python3 - "$host_json" "$num_prompts" <<'PY' import json, sys path, expected = sys.argv[1], int(sys.argv[2]) @@ -160,7 +99,7 @@ def read_int(*keys, default=None): completed = read_int("completed", "successful", "successful_requests") failed = read_int("failed", "errored", "failed_requests", "num_failed_requests", default=0) if failed or completed != expected: - print(f"vllm bench serve incomplete: completed={completed} failed={failed} expected={expected}", file=sys.stderr) + print(f"vllm-bench incomplete: completed={completed} failed={failed} expected={expected}", file=sys.stderr) sys.exit(1) PY echo " saved $host_json" diff --git a/workloads/deepseek_v4_pro_mi355x.yaml b/workloads/deepseek_v4_pro_mi355x.yaml index ddb77be..e48a4e3 100644 --- a/workloads/deepseek_v4_pro_mi355x.yaml +++ b/workloads/deepseek_v4_pro_mi355x.yaml @@ -37,5 +37,3 @@ vllm_bench: output_len: 1024 num_prompts: 512 max_concurrency: 128 - speed_bench_dataset_subset: throughput_8k - speed_bench_category: low_entropy diff --git a/workloads/gpt_oss_120b_mi355x.yaml b/workloads/gpt_oss_120b_mi355x.yaml index ec8c8a9..8174c4f 100644 --- a/workloads/gpt_oss_120b_mi355x.yaml +++ b/workloads/gpt_oss_120b_mi355x.yaml @@ -35,5 +35,3 @@ vllm_bench: output_len: 1024 num_prompts: 512 max_concurrency: 128 - speed_bench_dataset_subset: throughput_8k - speed_bench_category: low_entropy diff --git a/workloads/kimi_k2_5_mi300x.yaml b/workloads/kimi_k2_5_mi300x.yaml index e482c7b..9719f43 100644 --- a/workloads/kimi_k2_5_mi300x.yaml +++ b/workloads/kimi_k2_5_mi300x.yaml @@ -34,5 +34,3 @@ vllm_bench: output_len: 1024 num_prompts: 512 max_concurrency: 128 - speed_bench_dataset_subset: throughput_8k - speed_bench_category: low_entropy diff --git a/workloads/kimi_k2_5_mi355x.yaml b/workloads/kimi_k2_5_mi355x.yaml index 64965ef..8c4d82d 100644 --- a/workloads/kimi_k2_5_mi355x.yaml +++ b/workloads/kimi_k2_5_mi355x.yaml @@ -34,5 +34,3 @@ vllm_bench: output_len: 1024 num_prompts: 512 max_concurrency: 128 - speed_bench_dataset_subset: throughput_8k - speed_bench_category: low_entropy