Add GB300 Slurm Pyxis smoke path

.buildkite/generate_pipeline.py

@@ -57,6 +57,7 @@ def setup_command(packages):
 GPU_EMOJI = {
     "H200": ":h200:",
     "B200": ":b200:",
+    "GB300_SLURM": ":gb300:",
     "A100": ":a100:",
     "MI355X": ":amd:",
     "MI300X": ":amd:",

README.md

@@ -37,7 +37,10 @@ Include one or more of `lm_eval:` / `vllm_bench:` / `bfcl:` depending on what yo
 name: qwen3_5-h200       # used in container name and results/<name>/
 gpu: H200                # picks queue/image/HF cache from lib/gpu_profiles.yaml
 num_gpus: 8
+num_nodes: 1             # optional; set >1 for multi-node Slurm workloads
+gpus_per_node: 8         # optional; used for Slurm allocation + metadata
 nightly: true            # include in the nightly schedule (default: false)
+bench_only: false        # optional; true skips lm_eval/BFCL for this workload
 
 vllm:                    # how the server is brought up
   model: Qwen/Qwen3.5-397B-A17B-FP8
@@ -73,6 +76,12 @@ bfcl:                    # function-calling eval (optional)
     multi_turn: 100      # or set a single int to cap every category
 
 vllm_bench:              # perf runs (optional) — fed to the perf dashboard
+  metadata:              # optional dashboard labels / throughput denominator
+    device: h200
+    precision: fp8
+    is_multinode: false
+    gpu_count: 8
+    tp: 8
   configs:
     - name: 1k-in-1k-out-conc-256
       backend: openai                 # /v1/completions — exact ISL/OSL, no chat template
@@ -86,13 +95,17 @@ vllm_bench:              # perf runs (optional) — fed to the perf dashboard
 A few things worth knowing:
 
 - **`gpu`** must match a key in `lib/gpu_profiles.yaml`. The profile sets the Buildkite queue, default image, HF cache path, and baseline env vars.
+- **`num_nodes`** and **`gpus_per_node`** describe the allocation shape for multi-node workloads. Existing single-node recipes can omit them.
+- **`bench_only: true`** skips lm-eval and BFCL for one workload, even when the build-wide `BENCH_ONLY` env var is not set. This is useful for new perf-only smoke tests.
 - **`nightly`** controls only the nightly schedule. Recipes with `nightly: false` (or omitted) are still triggerable explicitly via the `WORKLOADS` env var.
 - **`lm_eval.tasks` is a list** because each entry runs as a separate `lm_eval` invocation — `--num_fewshot` is a single global flag, so different shot counts need separate runs. Each task's results land in `results/<name>/<task-name>/`.
 - **`vllm_bench` runs first** if both blocks are present — that way perf-pipeline bugs surface quickly instead of waiting on a full lm-eval pass.
 - **`vllm_bench` uses the `random` dataset with `--ignore-eos`** so every request prefills exactly `input_len` and decodes exactly `output_len` tokens — that's what makes the per-GPU decode throughput meaningful. Pair it with `backend: openai` (the `/v1/completions` endpoint) for exact token control. Avoid `dataset: speed_bench` for throughput numbers: it requires `--skip-tokenizer-init`, which makes `vllm bench serve` cap every request at a single output token, so output throughput reads as ~0.
 - **`bfcl` may need tool-call serve args.** Some models require `--enable-auto-tool-choice` and `--tool-call-parser` for function-calling; the parser warns if `--tool-call-parser` is absent. Each category runs as a separate generate + evaluate pass; scores appear on the eval dashboard as `bfcl_<category>` tasks.
 - **`bfcl.maximum_step_limit`** caps how many inference steps BFCL allows per multi-turn turn (default 10 in perf-eval; BFCL upstream defaults to 20). Set it in the workload YAML, or override per-run with the `BFCL_MAXIMUM_STEP_LIMIT` env var (env wins over YAML). Useful for agentic / long multi-turn categories.
 - **`bfcl.max_test_cases`** subsamples a category instead of running the full set — e.g. `multi_turn` (~800 cases) down to 300. For aggregate groups with multiple subcategories, the cap is split evenly across subcategories (by BFCL id order within each). Set a single integer to cap every category, or a map per category (`multi_turn: 240`). Override per-run with `BFCL_MAX_TEST_CASES`. Scores are partial-eval only and are not comparable to full BFCL leaderboard numbers.
+- **`vllm_bench.metadata.gpu_count`** is the denominator for per-GPU throughput. Set it explicitly for multi-node runs; otherwise the parser falls back to `num_gpus` or the effective TP/DP parsed from `serve_args`.
+- **`vllm_bench.metadata`** can also set dashboard fields such as `is_multinode`, `num_nodes`, `gpus_per_node`, `tp`, `dp`, `ep`, `dp_attention`, `disagg`, and `spec_decoding`.
 
 For everything else (the full set of supported fields, defaults, validation rules), the existing files in `workloads/` are the working reference and `lib/parse_workload.py` is the source of truth.
 
@@ -126,6 +139,37 @@ The pipeline is [**`vllm/perf-eval`**](https://buildkite.com/vllm/perf-eval). Wi
 
 This runs the `qwen3_5_h200` workload against the specified vLLM nightly image. Omit `WORKLOADS` to run all `nightly: true` workloads.
 
+### Run a GB300 Slurm smoke
+
+`workloads/deepseek_v4_flash_gb300_slurm.yaml` is an opt-in multi-node smoke. It does not run in nightly by default. Trigger it explicitly after the `GB300_SLURM` profile's Buildkite queue and Slurm launcher environment are configured:
+
+```bash
+WORKLOADS=deepseek_v4_flash_gb300_slurm
+BENCH_ONLY=1
+VLLM_COMMIT=<vllm-sha>
+VLLM_IMAGE=/home/inf-simon/containers/vllm-openai-v0.20.0-cu130.sqsh
+```
+
+The `GB300_SLURM` profile uses `server_runtime: slurm`, which delegates server startup to `lib/slurm_launch_vllm.sh` unless `PERF_EVAL_SLURM_LAUNCHER` points at a custom launcher. The launcher contract is:
+
+1. Submit a long-running server job.
+2. Write the endpoint env file named by `PERF_EVAL_ENDPOINT_FILE`.
+3. Print the Slurm job id on stdout so `lib/server.sh` can `scancel` it on exit.
+
+Useful Slurm env vars:
+
+- `PERF_EVAL_SLURM_ACCOUNT`, `PERF_EVAL_SLURM_PARTITION`, `PERF_EVAL_SLURM_QOS`, `PERF_EVAL_SLURM_RESERVATION`, `PERF_EVAL_SLURM_TIME`
+- `PERF_EVAL_SLURM_CONTAINER_MOUNTS`, `PERF_EVAL_SLURM_CONTAINER_WORKDIR`, `PERF_EVAL_SLURM_EXTRA_SBATCH_ARGS`, `PERF_EVAL_SLURM_EXTRA_SRUN_ARGS`
+- `PERF_EVAL_SLURM_CONTAINER_RUNTIME=auto|pyxis|none` to choose Pyxis/Enroot container launch or native `srun`; `auto` uses Pyxis only when `srun --help` advertises `--container-image`
+- `PERF_EVAL_SLURM_REQUEST_GPUS=0` for clusters that allocate whole GPU nodes without Slurm GPU GRES, `PERF_EVAL_SLURM_GPUS_PER_NODE=<n>` to override the workload's metadata value for `--gpus-per-node`, or `PERF_EVAL_SLURM_GRES=<gres>` to request GPUs with `--gres` instead
+- `PERF_EVAL_SLURM_VLLM_DISTRIBUTED_BACKEND=mp|none` to control the default native vLLM multi-node launcher. The default is `mp` when `num_nodes > 1`.
+- `PERF_EVAL_SLURM_SERVER_COMMAND` to replace the default non-Ray `vllm serve` command with a cluster-specific launch wrapper
+- `PERF_EVAL_BENCH_CLIENT_RUNTIME=slurm` to run `vllm bench serve` through Slurm instead of requiring a local `vllm` CLI on the Buildkite submission host; `PERF_EVAL_BENCH_CLIENT_CONTAINER_RUNTIME=auto|pyxis|none` overrides the server container runtime for that client step
+
+The GB300 Slurm profile defaults to Pyxis/Enroot containers for both the server and benchmark client, partition `batch`, `PERF_EVAL_SLURM_GRES=gpu:b300:4`, a 3-hour time limit, `--exclusive`, and `HF_HOME=/local_scratch/hf-models`. The profile starts with the Axis mounts `/home/inf-simon:/nfs_home,/raid/users/inf-simon:/local_scratch`; add a CI model-cache mount only after the durable cache path exists on the rack. If `PERF_EVAL_SLURM_GRES` is unset, the launcher uses the workload's `gpus_per_node` value with `--gpus-per-node`.
+
+For `num_nodes > 1`, the default Slurm launcher uses native vLLM MP serving rather than Ray: it starts one `vllm serve` task per node with `--distributed-executor-backend mp`, sets `--nnodes`, `--master-addr`, and `--node-rank`, and adds `--headless` on follower nodes. Pyxis server and benchmark-client steps set `--container-workdir` automatically when the current host checkout is under one of the configured `host:container` mounts; set `PERF_EVAL_SLURM_CONTAINER_WORKDIR` explicitly if the checkout path is not inferable. SRT-Slurm remains a possible future wrapper, but it needs an SRT/Dynamo-capable image and checkout and should not block this substrate path.
+
 **From an agent:** see `CLAUDE.md` for the Buildkite MCP workflow (don't shell out to `curl` or `bk`).
 
 ### Run a recipe end-to-end

lib/gpu_profiles.yaml

@@ -15,6 +15,22 @@ B200:
     VLLM_DEEP_GEMM_WARMUP: skip
     NCCL_CUMEM_HOST_ENABLE: 0
 
+GB300_SLURM:
+  queue: gb300_slurm
+  hf_home: /local_scratch/hf-models
+  server_runtime: slurm
+  env:
+    VLLM_DEEP_GEMM_WARMUP: skip
+    PERF_EVAL_SLURM_CONTAINER_RUNTIME: pyxis
+    PERF_EVAL_BENCH_CLIENT_RUNTIME: slurm
+    PERF_EVAL_BENCH_CLIENT_CONTAINER_RUNTIME: pyxis
+    PERF_EVAL_SLURM_REQUEST_GPUS: "1"
+    PERF_EVAL_SLURM_PARTITION: batch
+    PERF_EVAL_SLURM_GRES: gpu:b300:4
+    PERF_EVAL_SLURM_TIME: "03:00:00"
+    PERF_EVAL_SLURM_CONTAINER_MOUNTS: /home/inf-simon:/nfs_home,/raid/users/inf-simon:/local_scratch
+    PERF_EVAL_SLURM_EXTRA_SBATCH_ARGS: --exclusive
+
 MI300X:
   queue: mi300_perf_eval
   image_repo: vllm/vllm-openai-rocm

lib/ingest_perf.py

@@ -8,8 +8,8 @@
 
 Latencies in the raw result are in milliseconds (e.g., `mean_ttft_ms`).
 The dashboard expects seconds, so the transform drops the `_ms` suffix and
-divides by 1000. Aggregate throughput is divided by `tp` to match the
-dashboard's per-GPU columns (`tput_per_gpu`, `output_tput_per_gpu`,
+divides by 1000. Aggregate throughput is divided by `gpu_count` to populate
+the dashboard's per-GPU columns (`tput_per_gpu`, `output_tput_per_gpu`,
 `input_tput_per_gpu`).
 
 Failures are logged but never fatal: ingestion is best-effort and must not
@@ -41,9 +41,14 @@ def post(endpoint: str, payload: dict) -> None:
             raise RuntimeError(f"HTTP {resp.status}")
 
 
+def bool_text(value: object) -> str:
+    return "true" if str(value).lower() in {"1", "true", "yes"} else "false"
+
+
 def transform(raw: dict, args: argparse.Namespace) -> dict:
     """Map the raw `vllm bench serve` JSON to the dashboard's row shape."""
     tp = max(args.tp, 1)
+    gpu_count = max(args.gpu_count or tp, 1)
     total_token_throughput = float(raw.get("total_token_throughput", 0) or 0)
     output_throughput = float(raw.get("output_throughput", 0) or 0)
     input_throughput = total_token_throughput - output_throughput
@@ -56,18 +61,25 @@ def transform(raw: dict, args: argparse.Namespace) -> dict:
         "model": raw.get("model_id") or args.model,
         "framework": "vllm",
         "precision": args.precision,
-        "spec_decoding": "false",
-        "disagg": "false",
         "isl": int(args.isl),
         "osl": int(args.osl),
-        "is_multinode": "false",
+        "is_multinode": bool_text(args.is_multinode),
         "tp": tp,
-        "ep": 1,
-        "dp_attention": "false",
-        "tput_per_gpu": total_token_throughput / tp,
-        "output_tput_per_gpu": output_throughput / tp,
-        "input_tput_per_gpu": input_throughput / tp,
+        "ep": args.ep or 1,
+        "dp_attention": bool_text(args.dp_attention),
+        "disagg": bool_text(args.disagg),
+        "spec_decoding": bool_text(args.spec_decoding),
+        "tput_per_gpu": total_token_throughput / gpu_count,
+        "output_tput_per_gpu": output_throughput / gpu_count,
+        "input_tput_per_gpu": input_throughput / gpu_count,
     }
+    if args.dp is not None:
+        data["dp"] = args.dp
+    if args.num_nodes is not None:
+        data["num_nodes"] = args.num_nodes
+    if args.gpus_per_node is not None:
+        data["gpus_per_node"] = args.gpus_per_node
+    data["gpu_count"] = gpu_count
 
     if os.environ.get("NIGHTLY") == "1":
         data["nightly"] = True
@@ -89,11 +101,26 @@ def transform(raw: dict, args: argparse.Namespace) -> dict:
     return data
 
 
+def optional_int(value: str) -> int | None:
+    if value in ("", "None", "none", "null"):
+        return None
+    return int(value)
+
+
 def main() -> int:
     p = argparse.ArgumentParser(description=__doc__)
     p.add_argument("--raw-result", required=True, help="Raw JSON from `vllm bench serve --save-result`")
     p.add_argument("--device", required=True, help="Device tag (e.g. h200)")
     p.add_argument("--tp", type=int, required=True, help="Effective parallel-degree (TP * DP)")
+    p.add_argument("--dp", type=optional_int, default=None, help="Data parallel degree, when known")
+    p.add_argument("--ep", type=optional_int, default=None, help="Expert parallel degree, when known")
+    p.add_argument("--gpu-count", type=optional_int, default=None, help="Denominator for per-GPU throughput")
+    p.add_argument("--is-multinode", default="false", help="Whether this row came from a multi-node run")
+    p.add_argument("--num-nodes", type=optional_int, default=None, help="Node count for multi-node runs")
+    p.add_argument("--gpus-per-node", type=optional_int, default=None, help="GPUs per node for multi-node runs")
+    p.add_argument("--dp-attention", default="false", help="Dashboard flag for DP attention")
+    p.add_argument("--disagg", default="false", help="Dashboard flag for disaggregated serving")
+    p.add_argument("--spec-decoding", default="false", help="Dashboard flag for speculative decoding")
     p.add_argument("--precision", required=True, help="Precision tag (e.g. fp8, bf16)")
     p.add_argument("--model", required=True, help="HF model identifier (fallback if raw json lacks model_id)")
     p.add_argument("--image", required=True, help="Docker image used for the run")

lib/parse_workload.py

@@ -10,6 +10,10 @@
 Image precedence: VLLM_IMAGE > VLLM_COMMIT > workload `vllm.image` >
 `vllm/vllm-openai:latest`. When BENCH_ONLY is truthy, lm_eval task names
 are not validated against the registry (because they will not run).
+
+Multi-node workloads can set top-level `num_nodes` and `gpus_per_node`
+fields, plus `vllm_bench.metadata` values that control dashboard labels
+and the throughput denominator used for per-GPU metrics.
 """
 
 import os
@@ -141,6 +145,28 @@ def find(*names):
     return tp * dp
 
 
+def parse_int_arg(serve_args: str, *names: str) -> int | None:
+    toks = serve_args.split()
+    for i, t in enumerate(toks):
+        if "=" in t:
+            key, _, val = t.partition("=")
+            if key in names:
+                try:
+                    return int(val)
+                except ValueError:
+                    return None
+        elif t in names and i + 1 < len(toks):
+            try:
+                return int(toks[i + 1])
+            except ValueError:
+                return None
+    return None
+
+
+def bool_text(value: object) -> str:
+    return "true" if str(value).lower() in {"1", "true", "yes"} else "false"
+
+
 def precision_from_model(model: str) -> str:
     name = model.lower()
     for marker in ("fp4", "fp8", "int4", "int8", "bf16", "fp16"):
@@ -149,10 +175,10 @@ def precision_from_model(model: str) -> str:
     return "bf16"
 
 
-def validate_tasks(tasks: list, path: str) -> None:
+def validate_tasks(tasks: list, path: str, bench_only: bool = False) -> None:
     if not tasks:
         sys.exit(f"{path}: missing or empty `lm_eval.tasks`")
-    skip_registry = env_truthy("BENCH_ONLY")
+    skip_registry = bench_only or env_truthy("BENCH_ONLY")
     known = set() if skip_registry else known_task_names()
     for t in tasks:
         extra = set(t) - TASK_FIELDS
@@ -303,14 +329,15 @@ def main(path: str) -> None:
     tasks = lm_eval.get("tasks") or []
     bfcl = data.get("bfcl") or {}
     bench_configs = bench.get("configs") or []
+    bench_only = bool_text(data.get("bench_only", False))
 
     if not tasks and not bench_configs and not bfcl:
         sys.exit(
             f"{path}: workload must define at least one of lm_eval, vllm_bench, or bfcl"
         )
 
     if tasks:
-        validate_tasks(tasks, path)
+        validate_tasks(tasks, path, bench_only == "true")
 
     serve_args = vllm.get("serve_args") or ""
     if bfcl:
@@ -326,22 +353,59 @@ def main(path: str) -> None:
     if tp is None:
         tp = parse_tp(serve_args)
 
+    num_nodes = data.get("num_nodes") or metadata.get("num_nodes") or 1
+    gpus_per_node = data.get("gpus_per_node") or metadata.get("gpus_per_node")
+    num_gpus = data.get("num_gpus") or metadata.get("num_gpus")
+    if gpus_per_node is None and num_gpus and num_nodes:
+        try:
+            if int(num_gpus) % int(num_nodes) == 0:
+                gpus_per_node = int(num_gpus) // int(num_nodes)
+        except (TypeError, ValueError, ZeroDivisionError):
+            pass
+    if num_gpus is None and gpus_per_node:
+        try:
+            num_gpus = int(num_nodes) * int(gpus_per_node)
+        except (TypeError, ValueError):
+            pass
+
+    is_multinode = metadata.get("is_multinode")
+    if is_multinode is None:
+        try:
+            is_multinode = int(num_nodes) > 1
+        except (TypeError, ValueError):
+            is_multinode = False
+
+    gpu_count = (
+        metadata.get("gpu_count")
+        or metadata.get("per_gpu_denominator")
+        or num_gpus
+        or tp
+    )
+
     emit("NAME", data.get("name", ""))
     emit("IMAGE", image)
     emit("VLLM_COMMIT", vllm_commit)
     emit("MODEL", vllm.get("model", ""))
     emit("SERVE_ARGS", serve_args)
     emit("SERVER_RUNTIME", profile.get("server_runtime", "docker"))
+    emit("NUM_NODES", num_nodes)
+    emit("GPUS_PER_NODE", gpus_per_node or "")
+    emit("NUM_GPUS", num_gpus or "")
     emit("ENV", "\n".join(f"{k}={fmt(v)}" for k, v in env.items()))
     emit("LM_EVAL_TASKS_TSV", task_tsv(tasks, lm_eval.get("model_args") or {}))
     emit("VLLM_BENCH_TSV", bench_tsv(bench_configs, path))
     emit("BFCL_TSV", bfcl_tsv(bfcl) if bfcl else "")
+    emit("BENCH_ONLY", bench_only)
     emit("BENCH_DEVICE", metadata.get("device") or gpu.lower())
     emit("BENCH_TP", tp)
-    emit(
-        "BENCH_PRECISION",
-        metadata.get("precision") or precision_from_model(vllm.get("model") or ""),
-    )
+    emit("BENCH_DP", metadata.get("dp") or parse_int_arg(serve_args, "--data-parallel-size", "-dp", "--dp") or "")
+    emit("BENCH_EP", metadata.get("ep") or ("1" if "--enable-expert-parallel" in serve_args else ""))
+    emit("BENCH_GPU_COUNT", gpu_count)
+    emit("BENCH_IS_MULTINODE", bool_text(is_multinode))
+    emit("BENCH_DP_ATTENTION", bool_text(metadata.get("dp_attention", False)))
+    emit("BENCH_DISAGG", bool_text(metadata.get("disagg", False)))
+    emit("BENCH_SPEC_DECODING", bool_text(metadata.get("spec_decoding", False)))
+    emit("BENCH_PRECISION", metadata.get("precision") or precision_from_model(vllm.get("model") or ""))
 
 
 if __name__ == "__main__":