Switch vllm_bench to standalone vllm-bench Rust binary

README.md

@@ -28,7 +28,7 @@ A recipe has top-level metadata plus up to three eval blocks:
 
 - **`vllm:`** — *how the server runs.* Defines what model to serve and how (`model`, `serve_args`, optional image/env overrides). Required.
 - **`lm_eval:`** — *what accuracy to measure.* Lists lm-evaluation-harness tasks to run against the live server (e.g. `gsm8k`, `aime25`). Each task's score is saved under `results/<name>/<task-name>/`. Optional.
-- **`vllm_bench:`** — *what perf to measure.* Lists `vllm bench serve` configs (input/output lengths, concurrency, dataset). Raw JSON is saved and ingested into the perf dashboard. Optional.
+- **`vllm_bench:`** — *what perf to measure.* Lists [vllm-bench](https://github.com/vllm-project/vllm-bench) configs (input/output lengths, concurrency, dataset). Raw JSON is saved and ingested into the perf dashboard. Optional.
 - **`bfcl:`** — *function-calling eval.* Runs [BFCL](https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard) test categories against the live server. Some models need `--enable-auto-tool-choice` and `--tool-call-parser` in `serve_args`. Results are transformed to lm_eval format and ingested as `bfcl_<category>` tasks. Optional.
 
 Include one or more of `lm_eval:` / `vllm_bench:` / `bfcl:` depending on what you want out of this recipe.
@@ -89,7 +89,8 @@ A few things worth knowing:
 - **`nightly`** controls only the nightly schedule. Recipes with `nightly: false` (or omitted) are still triggerable explicitly via the `WORKLOADS` env var.
 - **`lm_eval.tasks` is a list** because each entry runs as a separate `lm_eval` invocation — `--num_fewshot` is a single global flag, so different shot counts need separate runs. Each task's results land in `results/<name>/<task-name>/`.
 - **`vllm_bench` runs first** if both blocks are present — that way perf-pipeline bugs surface quickly instead of waiting on a full lm-eval pass.
-- **`vllm_bench` uses the `random` dataset with `--ignore-eos`** so every request prefills exactly `input_len` and decodes exactly `output_len` tokens — that's what makes the per-GPU decode throughput meaningful. Pair it with `backend: openai` (the `/v1/completions` endpoint) for exact token control. Avoid `dataset: speed_bench` for throughput numbers: it requires `--skip-tokenizer-init`, which makes `vllm bench serve` cap every request at a single output token, so output throughput reads as ~0.
+- **`vllm_bench` uses the standalone [vllm-bench](https://github.com/vllm-project/vllm-bench) Rust client** (drop-in for `vllm bench serve` with the same flags and identical result-JSON schema). It runs on the host and talks HTTP to the served port, so no `docker exec` is needed. The prebuilt Linux binary is downloaded automatically on first use; pin a different release with `VLLM_BENCH_VERSION` or point at a local build with `VLLM_BENCH_BIN`. Only `dataset: random` is wired up here.
+- **`vllm_bench` uses `random` with `--ignore-eos`** so every request prefills exactly `input_len` and decodes exactly `output_len` tokens — that's what makes the per-GPU decode throughput meaningful. Pair it with `backend: openai` (the `/v1/completions` endpoint) for exact token control.
 - **`bfcl` may need tool-call serve args.** Some models require `--enable-auto-tool-choice` and `--tool-call-parser` for function-calling; the parser warns if `--tool-call-parser` is absent. Each category runs as a separate generate + evaluate pass; scores appear on the eval dashboard as `bfcl_<category>` tasks.
 - **`bfcl.maximum_step_limit`** caps how many inference steps BFCL allows per multi-turn turn (default 10 in perf-eval; BFCL upstream defaults to 20). Set it in the workload YAML, or override per-run with the `BFCL_MAXIMUM_STEP_LIMIT` env var (env wins over YAML). Useful for agentic / long multi-turn categories.
 - **`bfcl.max_test_cases`** subsamples a category instead of running the full set — e.g. `multi_turn` (~800 cases) down to 300. For aggregate groups with multiple subcategories, the cap is split evenly across subcategories (by BFCL id order within each). Set a single integer to cap every category, or a map per category (`multi_turn: 240`). Override per-run with `BFCL_MAX_TEST_CASES`. Scores are partial-eval only and are not comparable to full BFCL leaderboard numbers.

lib/ingest_perf.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-"""Transform a `vllm bench serve` raw JSON result and POST it to the perf
+"""Transform a vllm-bench raw JSON result and POST it to the perf
 dashboard's ingestion endpoint.
 
 The dashboard at perf.vllm.ai reads from the `vllm_perf_data_ingest`
@@ -42,7 +42,7 @@ def post(endpoint: str, payload: dict) -> None:
 
 
 def transform(raw: dict, args: argparse.Namespace) -> dict:
-    """Map the raw `vllm bench serve` JSON to the dashboard's row shape."""
+    """Map the raw vllm-bench JSON to the dashboard's row shape."""
     tp = max(args.tp, 1)
     total_token_throughput = float(raw.get("total_token_throughput", 0) or 0)
     output_throughput = float(raw.get("output_throughput", 0) or 0)
@@ -91,7 +91,7 @@ def transform(raw: dict, args: argparse.Namespace) -> dict:
 
 def main() -> int:
     p = argparse.ArgumentParser(description=__doc__)
-    p.add_argument("--raw-result", required=True, help="Raw JSON from `vllm bench serve --save-result`")
+    p.add_argument("--raw-result", required=True, help="Raw JSON from `vllm-bench --save-result`")
     p.add_argument("--device", required=True, help="Device tag (e.g. h200)")
     p.add_argument("--tp", type=int, required=True, help="Effective parallel-degree (TP * DP)")
     p.add_argument("--precision", required=True, help="Precision tag (e.g. fp8, bf16)")

lib/parse_workload.py

@@ -23,7 +23,6 @@
 BENCH_FIELDS = {
     "name", "backend", "dataset", "input_len", "output_len",
     "num_prompts", "max_concurrency",
-    "speed_bench_dataset_subset", "speed_bench_category",
 }
 BENCH_REQUIRED = ("name", "input_len", "output_len", "num_prompts", "max_concurrency")
 BFCL_FIELDS = {
@@ -115,7 +114,7 @@ def resolve_image(vllm: dict, profile: dict) -> tuple[str, str]:
 def parse_tp(serve_args: str) -> int:
     """Effective parallel degree (TP * DP) from serve_args; defaults to 1.
 
-    `vllm bench serve` reports aggregate throughput; we divide by this to get
+    vllm-bench reports aggregate throughput; we divide by this to get
     per-GPU metrics for the dashboard.
     """
     toks = serve_args.split()
@@ -205,8 +204,6 @@ def opt(key):
                     str(c["output_len"]),
                     str(c["num_prompts"]),
                     str(c["max_concurrency"]),
-                    opt("speed_bench_dataset_subset"),
-                    opt("speed_bench_category"),
                 ]
             )
         )

lib/run.sh

@@ -36,16 +36,15 @@ start_server "$CONTAINER" "$PORT" "$WORKLOAD_IMAGE" "$WORKLOAD_MODEL" \
              "$WORKLOAD_SERVE_ARGS" "$WORKLOAD_ENV" "$WORKLOAD_SERVER_RUNTIME"
 wait_healthy "$PORT"
 
-# vllm bench serve runs first so we can validate perf flow without waiting
+# vllm-bench runs first so we can validate perf flow without waiting
 # on a full lm_eval pass. Each config's raw json lands in
 # $RESULTS_DIR/bench-<name>.json and is then transformed and POSTed to the
 # perf dashboard ingest endpoint.
-while IFS=$'\t' read -r bname backend dataset isl osl nprompts conc speed_subset speed_category; do
+while IFS=$'\t' read -r bname backend dataset isl osl nprompts conc; do
   [[ -z "$bname" ]] && continue
   run_vllm_bench "$CONTAINER" "$PORT" "$WORKLOAD_MODEL" \
                  "$bname" "$backend" "$dataset" "$isl" "$osl" "$nprompts" \
-                 "$conc" "$speed_subset" "$speed_category" \
-                 "$BENCH_TRUST_REMOTE_CODE" "$RESULTS_DIR"
+                 "$conc" "$BENCH_TRUST_REMOTE_CODE" "$RESULTS_DIR"
 
   python3 "$DIR/ingest_perf.py" \
     --raw-result "${RESULTS_DIR}/bench-${bname}.json" \

lib/run_vllm_bench.sh

@@ -1,94 +1,63 @@
-# Run a single `vllm bench serve` config against the running vLLM container.
+# Run a single vllm-bench config against the running vLLM server.
 # Source this from run.sh.
 #
 # Usage:
 #   run_vllm_bench <container> <port> <model> <name> <backend> <dataset> \
 #                  <input_len> <output_len> <num_prompts> <max_concurrency> \
-#                  <speed_bench_dataset_subset> <speed_bench_category> \
 #                  <trust_remote_code> <output_dir>
 #
-# Docker runtime invokes `vllm bench serve` inside the vllm/vllm-openai
-# container via `docker exec`; native runtime invokes it directly. The raw
-# JSON lands in "<output_dir>/bench-<name>.json" so ingest_perf.py can pick
-# it up.
-
-# vLLM's SpeedBench class expects a local <subset>.jsonl file built by
-# NeMo's prepare.py — the bench CLI does not download the dataset itself.
-SPEED_BENCH_PREPARE_URL="https://raw.githubusercontent.com/NVIDIA-NeMo/Skills/refs/heads/main/nemo_skills/dataset/speed-bench/prepare.py"
-
-pip_install_quiet() {
-  if python3 -c 'import sys; sys.exit(0 if sys.prefix != sys.base_prefix else 1)'; then
-    python3 -m pip install --quiet "$@"
-  else
-    PIP_BREAK_SYSTEM_PACKAGES=1 python3 -m pip install --user --quiet "$@"
+# Uses the standalone vllm-bench Rust CLI from github.com/vllm-project/vllm-bench
+# (drop-in for `vllm bench serve` with the same flags and identical result-JSON
+# schema). The binary runs on the host and talks HTTP to the served port, so
+# the <container> arg is unused — kept for the existing call-site signature.
+# The raw JSON lands in "<output_dir>/bench-<name>.json" so ingest_perf.py can
+# pick it up.
+
+# Pinned upstream tag for the prebuilt binary; bump as new releases land.
+VLLM_BENCH_VERSION="${VLLM_BENCH_VERSION:-v0.1.0}"
+VLLM_BENCH_BIN="${VLLM_BENCH_BIN:-}"
+
+ensure_vllm_bench() {
+  if [[ -n "$VLLM_BENCH_BIN" && -x "$VLLM_BENCH_BIN" ]]; then
+    return
   fi
-}
-
-prepare_speed_bench_dataset() {
-  local container=$1 runtime=$2 subset=$3 category=$4 data_dir=$5
-
-  if [[ ! -s "${data_dir}/${subset}.jsonl" ]]; then
-    if ! python3 -c 'import importlib.util as u, sys; sys.exit(0 if all(u.find_spec(m) for m in ("datasets","numpy","pandas","tiktoken")) else 1)' 2>/dev/null; then
-      echo "--- :python: installing SPEED-Bench prep dependencies"
-      pip_install_quiet datasets numpy pandas tiktoken
-    fi
-    mkdir -p "$data_dir"
-    echo "--- :arrow_down: preparing SPEED-Bench ${subset} dataset in ${data_dir}"
-    python3 - "$SPEED_BENCH_PREPARE_URL" "$subset" "$category" "$data_dir" <<'PY'
-import sys, urllib.request
-from pathlib import Path
-
-prepare_url, subset, category, data_dir = sys.argv[1:]
-source = urllib.request.urlopen(prepare_url, timeout=60).read()
-ns = {"__name__": "speed_bench_prepare", "__file__": "prepare.py"}
-exec(compile(source, "prepare.py", "exec"), ns)
-
-dataset = ns["load_dataset"]("nvidia/SPEED-Bench", subset, split="test")
-if category:
-    dataset = dataset.filter(lambda ex: ex["category"] == category)
-dataset = ns["_resolve_external_data"](dataset, subset)
-dataset = dataset.map(
-    lambda ex: {"messages": [{"role": "user", "content": t} for t in ex["turns"]]},
-    remove_columns=["turns"],
-)
-Path(data_dir).mkdir(parents=True, exist_ok=True)
-dataset.to_json(Path(data_dir) / f"{subset}.jsonl")
-PY
+  if command -v vllm-bench >/dev/null 2>&1; then
+    VLLM_BENCH_BIN="$(command -v vllm-bench)"
+    return
   fi
-  test -s "${data_dir}/${subset}.jsonl"
-
-  # Docker runtime: ship the data into the container and make sure pandas is
-  # available there (vLLM's SpeedBench loads the JSONL via pandas).
-  if [[ "$runtime" != "native" ]]; then
-    docker exec "$container" mkdir -p "$data_dir"
-    docker cp "${data_dir}/." "${container}:${data_dir}/"
-    if ! docker exec "$container" python3 -c 'import pandas' 2>/dev/null; then
-      echo "--- :docker: installing pandas in vLLM container"
-      docker exec "$container" bash -lc \
-        'PIP_BREAK_SYSTEM_PACKAGES=1 python3 -m pip install --quiet pandas \
-          || PIP_BREAK_SYSTEM_PACKAGES=1 python3 -m pip install --user --quiet pandas'
-    fi
+  local cache="${HOME}/.cache/perf-eval"
+  local bin="${cache}/vllm-bench-${VLLM_BENCH_VERSION}"
+  if [[ ! -x "$bin" ]]; then
+    mkdir -p "$cache"
+    local arch; arch="$(uname -m)"
+    local url="https://github.com/vllm-project/vllm-bench/releases/download/${VLLM_BENCH_VERSION}/vllm-bench-${arch}-linux-musl"
+    echo "--- :arrow_down: downloading vllm-bench ${VLLM_BENCH_VERSION} (${arch})"
+    curl -fsSL "$url" -o "${bin}.tmp"
+    chmod +x "${bin}.tmp"
+    mv "${bin}.tmp" "$bin"
   fi
+  VLLM_BENCH_BIN="$bin"
 }
 
 run_vllm_bench() {
-  local container=$1 port=$2 model=$3 name=$4 backend=$5 dataset=$6
+  local _container=$1 port=$2 model=$3 name=$4 backend=$5 dataset=$6
   local input_len=$7 output_len=$8 num_prompts=$9 max_concurrency=${10}
-  local speed_bench_dataset_subset=${11} speed_bench_category=${12}
-  local trust_remote_code=${13} outdir=${14}
-  local runtime="${WORKLOAD_SERVER_RUNTIME:-docker}"
-  local in_container_json="/tmp/bench-${name}.json"
+  local trust_remote_code=${11} outdir=${12}
   local host_json="${outdir}/bench-${name}.json"
 
   [[ "$backend" == "-" ]] && backend=""
-  [[ "$speed_bench_dataset_subset" == "-" ]] && speed_bench_dataset_subset=""
-  [[ "$speed_bench_category" == "-" ]] && speed_bench_category=""
 
-  echo "--- :stopwatch: vllm bench serve ${name} (dataset=${dataset} isl=${input_len} osl=${output_len} conc=${max_concurrency} n=${num_prompts})"
+  if [[ "$dataset" != "random" ]]; then
+    echo "unsupported vllm_bench dataset: $dataset" >&2
+    return 2
+  fi
+
+  ensure_vllm_bench
+
+  echo "--- :stopwatch: vllm-bench ${name} (isl=${input_len} osl=${output_len} conc=${max_concurrency} n=${num_prompts})"
   mkdir -p "$outdir"
 
-  local cmd=(vllm bench serve)
-  [[ "$runtime" != "native" ]] && cmd=(docker exec "$container" "${cmd[@]}")
+  local cmd=("$VLLM_BENCH_BIN")
 
   if [[ -n "$backend" ]]; then
     cmd+=(--backend "$backend" --base-url "http://127.0.0.1:${port}")
@@ -99,51 +68,21 @@ run_vllm_bench() {
 
   cmd+=(
     --model "$model"
-    --dataset-name "$dataset"
+    --dataset-name random
     --num-prompts "$num_prompts"
     --max-concurrency "$max_concurrency"
+    # --ignore-eos forces every request to emit the full output_len; without it
+    # the model can stop early on the random prompt and decode throughput collapses.
+    --random-input-len "$input_len"
+    --random-output-len "$output_len"
+    --ignore-eos
+    --save-result
+    --result-filename "$host_json"
   )
   [[ "$trust_remote_code" == "true" ]] && cmd+=(--trust-remote-code)
 
-  case "$dataset" in
-    random)
-      # --ignore-eos forces every request to emit the full output_len; without it
-      # the model can stop early on the random prompt and decode throughput collapses.
-      cmd+=(--random-input-len "$input_len" --random-output-len "$output_len" --ignore-eos)
-      ;;
-    speed_bench)
-      [[ -z "$speed_bench_dataset_subset" ]] && speed_bench_dataset_subset="qualitative"
-      local data_dir="${VLLM_SPEED_BENCH_DIR:-/tmp/vllm-speed-bench}/${speed_bench_dataset_subset}"
-      [[ -n "$speed_bench_category" ]] && data_dir="${data_dir}-${speed_bench_category}"
-      prepare_speed_bench_dataset "$container" "$runtime" \
-        "$speed_bench_dataset_subset" "$speed_bench_category" "$data_dir"
-      # SPEED-Bench applies the client-side chat template at tokenizer init,
-      # which breaks for chat-template-less models — rely on server-side
-      # usage accounting instead.
-      cmd+=(
-        --dataset-path "$data_dir"
-        --speed-bench-output-len "$output_len"
-        --speed-bench-dataset-subset "$speed_bench_dataset_subset"
-        --skip-tokenizer-init
-      )
-      [[ -n "$speed_bench_category" ]] && cmd+=(--speed-bench-category "$speed_bench_category")
-      ;;
-    *)
-      echo "unsupported vllm_bench dataset: $dataset" >&2
-      return 2
-      ;;
-  esac
-
-  if [[ "$runtime" == "native" ]]; then
-    cmd+=(--save-result --result-filename "$host_json")
-  else
-    cmd+=(--save-result --result-filename "$in_container_json")
-  fi
-
   "${cmd[@]}"
 
-  [[ "$runtime" != "native" ]] && docker cp "${container}:${in_container_json}" "$host_json"
-
   python3 - "$host_json" "$num_prompts" <<'PY'
 import json, sys
 path, expected = sys.argv[1], int(sys.argv[2])
@@ -160,7 +99,7 @@ def read_int(*keys, default=None):
 completed = read_int("completed", "successful", "successful_requests")
 failed = read_int("failed", "errored", "failed_requests", "num_failed_requests", default=0)
 if failed or completed != expected:
-    print(f"vllm bench serve incomplete: completed={completed} failed={failed} expected={expected}", file=sys.stderr)
+    print(f"vllm-bench incomplete: completed={completed} failed={failed} expected={expected}", file=sys.stderr)
     sys.exit(1)
 PY
   echo "  saved $host_json"

workloads/deepseek_v4_pro_mi355x.yaml

@@ -37,5 +37,3 @@ vllm_bench:
       output_len: 1024
       num_prompts: 512
       max_concurrency: 128
-      speed_bench_dataset_subset: throughput_8k
-      speed_bench_category: low_entropy

workloads/gpt_oss_120b_mi355x.yaml

@@ -35,5 +35,3 @@ vllm_bench:
       output_len: 1024
       num_prompts: 512
       max_concurrency: 128
-      speed_bench_dataset_subset: throughput_8k
-      speed_bench_category: low_entropy

workloads/kimi_k2_5_mi300x.yaml

@@ -34,5 +34,3 @@ vllm_bench:
       output_len: 1024
       num_prompts: 512
       max_concurrency: 128
-      speed_bench_dataset_subset: throughput_8k
-      speed_bench_category: low_entropy

workloads/kimi_k2_5_mi355x.yaml

@@ -34,5 +34,3 @@ vllm_bench:
       output_len: 1024
       num_prompts: 512
       max_concurrency: 128
-      speed_bench_dataset_subset: throughput_8k
-      speed_bench_category: low_entropy