Skip to content

fix(agentic): remove warmup drain timeout #2186

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cquil11 merged 1 commit into from
Jul 14, 2026
+3 −0
Merged

fix(agentic): remove warmup drain timeout #2186

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions benchmarks/benchmark_lib.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1479,6 +1479,9 @@ build_replay_cmd() {
# with one-token outputs and no idle delays for 10 minutes. Profiling begins
# only after those requests drain and resumes from the resulting live state.
REPLAY_CMD+=" --agentic-cache-warmup-duration 600"
# Long-context warmup requests can take more than AIPerf's default 300s
# grace period to drain. Wait for every warmup request before profiling.
REPLAY_CMD+=" --warmup-grace-period inf"

@claude claude Bot Jul 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 WARNING: Nothing in this PR (or CI, since the PR is unlabeled) actually exercises --warmup-grace-period inf against the pinned aiperf, and inf removes the only bound on warmup drain short of the job-level timeout.

Why it matters:

  1. The utils/aiperf submodule pin (0d2aa05) was not bumped in this PR, and the stated validation (bash -n + a generated-command assertion) only proves the flag is appended to the command string — not that the pinned aiperf CLI accepts --warmup-grace-period or parses inf for it. If it doesn't, every agentic replay fails at argument parsing (all ~25 single-node recipes plus multi_node/agentic_srt.sh route through build_replay_cmd).
  2. $REPLAY_CMD runs with no timeout wrapper (run_agentic_replay_and_write_outputs, line 1595), so with an infinite grace period a single wedged warmup request (hung server, stuck stream) now blocks the job until the outer GHA/Slurm timeout instead of proceeding after 300s — that's hours of idle GPU time per stuck config in a sweep.

Fix: Run at least one agentic config against this branch (add the full-sweep-fail-fast label) to confirm the pinned aiperf accepts --warmup-grace-period inf before merging. If the goal is just "longer than 300s", a large finite grace (e.g. 1800, matching the bumped configure timeouts at lines 1455–1458) preserves the hang backstop:

Suggested change
REPLAY_CMD+=" --warmup-grace-period inf"
REPLAY_CMD+=" --warmup-grace-period 1800"

# Use server-reported usage fields (prompt_tokens / completion_tokens) for
# ISL/OSL instead of client-side tokenizer.encode(). Auto-enables
# stream_options.include_usage on the OpenAI chat endpoint. Skips the
Expand Down