Add GB300 Slurm Pyxis smoke path

[khluu]

Summary

• Add an opt-in deepseek_v4_flash_gb300_slurm workload and GB300_SLURM profile for the Axis Slurm/Pyxis path.
• Add a Slurm server launcher and Slurm benchmark-client path. For num_nodes > 1, the default launcher now uses non-Ray native vLLM MP serving: one vllm serve task per node, --distributed-executor-backend mp, --nnodes, --master-addr, --node-rank, and --headless on follower nodes.
• Add Pyxis --container-workdir support for server and benchmark-client steps so results can be written from inside the container back to a shared mounted checkout.
• Fix server log streaming teardown so successful Slurm runs cannot hang waiting on a surviving tail -F process.
• Add multi-node workload metadata parsing, dashboard ingest fields, and README docs for the manual GB300 smoke.

AI assistance was used.

Smoke Status

A bounded manual Axis GB300 pass against this PR's Slurm/Pyxis substrate passed outside Buildkite:

• 1-node sanity smoke with a tiny OPT model completed 4/4 benchmark requests with 0 failures.
• 2-node native MP smoke with a tiny OPT model completed 4/4 benchmark requests with 0 failures and produced the expected per-GPU ingest denominator for 8 GPUs.
• Result-file visibility works when Pyxis steps use --container-workdir pointed at the shared checkout mount.

The staged Axis image has vllm, uv, and ninja, but not Dynamo, so SRT-Slurm is not a drop-in path with that image. SRT can be added later through a site wrapper or a dedicated SRT/Dynamo-capable image and checkout; it should not block this Slurm/Pyxis substrate.

Remaining Gates

• Keep this PR draft until a real deepseek_v4_flash_gb300_slurm smoke or equivalent larger workload shape passes.
• Pin VLLM_IMAGE to a local Axis .sqsh, for example /home/inf-simon/containers/vllm-openai-v0.20.0-cu130.sqsh; relying on the default Docker image path risks a Docker Hub import on the rack.
• Confirm the Buildkite queue name and model/cache staging behavior before enabling any nightly run.

Validation

• python3 -m py_compile .buildkite/generate_pipeline.py lib/parse_workload.py lib/ingest_perf.py
• bash -n lib/run.sh && bash -n lib/server.sh && bash -n lib/run_lm_eval.sh && bash -n lib/run_vllm_bench.sh && bash -n lib/slurm_launch_vllm.sh
• Parser smoke for workloads/deepseek_v4_flash_gb300_slurm.yaml, verifying slurm, 2 nodes, 4 GPUs per node, 8 total GPUs, batch, gpu:b300:4, Pyxis, Axis mounts, HF_HOME=/local_scratch/hf-models, and bench-only metadata.
• WORKLOADS=deepseek_v4_flash_gb300_slurm python3 .buildkite/generate_pipeline.py
• Mocked Slurm launcher smoke verifying sbatch and generated per-node srun commands include --gres gpu:b300:4, --partition batch, --exclusive, Pyxis image, --container-workdir, native MP rank flags, and --headless on follower nodes.
• Mocked Slurm benchmark-client smoke verifying Pyxis image, mounts, and --container-workdir are passed to the client srun.
• Local log-stream teardown smoke verifying stop_log_stream kills the stream process group and leaves no tail -F member behind.
• git diff --check