fix(agentic): remove warmup drain timeout

[cquil11]

Passes --warmup-grace-period inf for agentic replays, removing AIPerf's default 300-second drain cutoff.

Validated with bash -n benchmarks/benchmark_lib.sh and a generated-command assertion.

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[claude]

Claude finished @cquil11's task in 5m 31s —— View job

---

Review complete

• Read the PR diff (benchmarks/benchmark_lib.sh, +3 lines)
• Verify --warmup-grace-period inf against AIPerf — could not be verified (see below)
• Check interaction with surrounding warmup/timeout configuration
• Post review feedback

Summary

The change itself is clean and lands in the right place — build_replay_cmd is the single shared builder, so every agentic recipe (single-node and multi_node/agentic_srt.sh) picks it up. One 🟡 warning posted inline: this PR doesn't bump the utils/aiperf submodule pin, and neither bash -n nor a command-string assertion proves the pinned aiperf CLI accepts --warmup-grace-period inf — if it doesn't, every agentic replay fails at argument parsing. Additionally, inf combined with the fact that $REPLAY_CMD runs with no timeout wrapper means a single wedged warmup request now stalls the job until the outer GHA/Slurm timeout instead of 300s. Recommend validating with a full-sweep-fail-fast labeled run before merge, or using a large finite grace period (e.g. 1800) if unbounded drain isn't strictly required.

Note: I attempted to inspect the pinned aiperf fork (SemiAnalysisAI/aiperf@0d2aa05) to confirm the flag's existence and inf parsing, but network/code-search access is restricted in this review environment.

[claude]

🟡 WARNING: Nothing in this PR (or CI, since the PR is unlabeled) actually exercises --warmup-grace-period inf against the pinned aiperf, and inf removes the only bound on warmup drain short of the job-level timeout.

Why it matters:

1. The utils/aiperf submodule pin (0d2aa05) was not bumped in this PR, and the stated validation (bash -n + a generated-command assertion) only proves the flag is appended to the command string — not that the pinned aiperf CLI accepts --warmup-grace-period or parses inf for it. If it doesn't, every agentic replay fails at argument parsing (all ~25 single-node recipes plus multi_node/agentic_srt.sh route through build_replay_cmd).
2. $REPLAY_CMD runs with no timeout wrapper (run_agentic_replay_and_write_outputs, line 1595), so with an infinite grace period a single wedged warmup request (hung server, stuck stream) now blocks the job until the outer GHA/Slurm timeout instead of proceeding after 300s — that's hours of idle GPU time per stuck config in a sweep.

Fix: Run at least one agentic config against this branch (add the full-sweep-fail-fast label) to confirm the pinned aiperf accepts --warmup-grace-period inf before merging. If the goal is just "longer than 300s", a large finite grace (e.g. 1800, matching the bumped configure timeouts at lines 1455–1458) preserves the hang backstop:

Suggested change

	REPLAY_CMD+=" --warmup-grace-period inf"
	REPLAY_CMD+=" --warmup-grace-period 1800"

[claude]

LGTM — a single-line addition of --warmup-grace-period inf to the agentic replay aiperf command.

Extended reasoning...

Overview

The PR adds one line to build_replay_cmd() in benchmarks/benchmark_lib.sh, appending --warmup-grace-period inf to the aiperf replay command used for agentic benchmarks. No other files change.

Verification

Traced the flag through utils/aiperf: under TimingMode.AGENTIC_REPLAY, when warmup_grace_period is unset, _agentic_warmup_grace_period() (utils/aiperf/src/aiperf/timing/config.py) computes min(agentic_cache_warmup_duration, 300.0). Since this recipe sets --agentic-cache-warmup-duration 600, the effective default grace period is currently capped at 300s, matching the PR's stated motivation of removing a "default 300-second drain cutoff." --warmup-grace-period inf is a valid override — the CLI field is float | None with ge=0 (accepts inf), and user_config.py's validate_warmup_grace_period explicitly exempts AGENTIC_REPLAY mode from requiring --warmup-duration, so the flag is accepted standalone in this scenario.

Security risks

None — this only affects a benchmark harness shell script, no production code paths, auth, or data handling.

Level of scrutiny

Low. This is a one-line, well-commented addition to an internal benchmarking script that only changes CLI flags passed to a load-generation tool, with no logic changes elsewhere.

Other factors

The PR description's rationale checks out against the actual aiperf source and its validator/test coverage (test_grace_period_validation.py covers the agentic-mode-alone case). No CODEOWNER-sensitive or security-relevant code is touched.