Reap stale perf-eval containers before starting vLLM#26
Open
khluu wants to merge 2 commits into
Open
Conversation
Some vLLM release images don't include pytest, but cupy.testing._random imports it unconditionally. When torch.compile traces through cupy modules during model loading, this causes a ModuleNotFoundError that crashes the server. Installing pytest after container startup fixes this. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
A job killed with SIGKILL (e.g. the Buildkite 120-min step timeout or an agent restart) never runs run.sh's EXIT trap, so its vLLM container survives and squats on the GPUs and port 8000. Because container names are PID-based and unique, a later run's own cleanup never reaps the orphan, and `docker run -p 8000:8000` then fails with "Bind for 0.0.0.0:8000 failed: port is already allocated" -- wedging every subsequent run on that host until someone removes the container by hand. This is what broke build 222 on h200-ci-1. Add reap_stale_servers(), called from start_server before `docker run`, to remove any leftover perf-eval container (by name prefix) or anything still holding the target port. The server port is fixed, so two perf-eval runs can't share a host -- any pre-existing perf-eval container is stale and safe to remove, making the host self-heal on the next run. This change was authored with assistance from Claude Code (AI-assisted). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Build 222 failed on
h200-ci-1instantly with:This was not an eval failure. A leftover container
perf-eval-kimi_k2_5-h200-3640085from an earlier job was still running (up 9h, idle ~7.5h), pinning all 8 GPUs and port 8000. I removed it by hand to unblock the host; this PR prevents recurrence.Root cause
run.shcleans up viatrap 'stop_server "$CONTAINER"' EXIT. When a job is SIGKILLed — the Buildkite 120-min step timeout, or an agent restart — the EXIT trap never fires, so the container survives.perf-eval-${WORKLOAD_NAME}-$$(PID-based, unique per run), so a later run'sdocker rm -f "$CONTAINER"only ever targets its own container — it never reaps a previous run's orphan.docker run -p 8000:8000collides on every subsequent run. One orphan wedges the host indefinitely until someone clears it manually.Fix
Add
reap_stale_servers()inlib/server.sh, called fromstart_serverright beforedocker run. It removes any leftover container matching theperf-eval-name prefix, plus anything still publishing the target port. The server port is fixed at 8000, so two perf-eval runs can't coexist on one host — any pre-existing perf-eval container is by definition stale and safe to remove. The host now self-heals on the next run instead of dead-locking.Testing
bash -n lib/run.sh && bash -n lib/server.sh && bash -n lib/run_lm_eval.sh— passes.docker ps --filter name=perf-eval-and--filter publish=8000filters return cleanly onh200-ci-1(exit 0).This PR was authored with assistance from Claude Code (AI-assisted).
🤖 Generated with Claude Code