[AMD] Qwen3.5-FP8 MI355X SGLang disagg MTP: EAGLE speculative decoding / Qwen3.5-FP8 MI355X SGLang disagg MTP：新增EAGLE投机解码配置

[ChangLiu0709]

Changes

MTP (qwen3.5-fp8-mi355x-sglang-disagg-mtp) — new config

• EAGLE speculative decoding on TP4P+TP8D topology: spec-decoding: mtp, DECODE_MTP_SIZE=3 (EAGLE, num-steps 3, draft-tokens 4, topk 1).
• Concurrency sweep: [8, 16, 32, 64] on TP4P+TP8D (12 GPU), [128] on TP4P+TP4D (8 GPU).
• ISL=8K / OSL=1K.
• models.yaml: add mtp_flags: "--speculative-algorithm EAGLE --speculative-eagle-topk 1" to Qwen3.5-397B-A17B-FP8.

Why two concurrency arms

At conc=128, TP8 decode bottlenecks MTP draft verification (saturates 8 GPUs, causing -38% tput/GPU and TTFT explosion). Switching to TP4D resolves the collapse: +83% tput/GPU, 7.5x better TTFT vs TP8D.

Local benchmark results (v0.5.16, ISL=8K/OSL=1K)

MTP (TP4P+TP8D, 12 GPU) — conc 8-64:

Conc	Interactivity	Tput/GPU	TPOT (ms)	TTFT (ms)
8	121.7	648	8.22	654
16	109.9	1,150	9.10	887
32	96.7	1,923	10.34	1,452
64	87.1	2,955	11.48	3,832

MTP (TP4P+TP4D, 8 GPU) — conc 128:

Conc	Interactivity	Tput/GPU	TPOT (ms)	TTFT (ms)
128	39.5	5,147	25.32	1,715

Split from #2557 — MTP/speculative decoding only; independent of baseline PR #2606.

Authors

@ChangLiu0709
@xiaohuguo2023
@chunfangamd

中文说明

Qwen3.5-FP8 MI355X SGLang 分离式推理新增 MTP（EAGLE 投机解码）配置：

• 新增 qwen3.5-fp8-mi355x-sglang-disagg-mtp 配置，使用 EAGLE 投机解码（depth=3, topk=1）
• 两组并发测试方案：conc 8-64 使用 TP4P+TP8D (12 GPU)，conc 128 使用 TP4P+TP4D (8 GPU)
• 在 conc=128 时，TP8 解码会导致 MTP 性能崩溃；切换至 TP4D 后吞吐量/GPU 提升 83%，TTFT 改善 7.5 倍
• models.yaml：为 Qwen3.5-397B-A17B-FP8 添加 mtp_flags 参数

从 #2557 拆分 — 仅包含 MTP 投机解码配置，与基线 PR #2606 相互独立。

Made with Cursor

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[claude]

🔴 The appended changelog entry's pr-link points to pull/2557, but this change is PR #2607. This will cause utils/merge_with_reuse.sh to fail at the prepare_perf_changelog_merge.py canonicalize --pr-number 2607 step (compare_entries → validate_added_pr_link raises ChangelogValidationError since the link is neither pull/2607 nor an XXX placeholder), blocking the automated merge path. Fix: change the link to https://github.com/SemiAnalysisAI/InferenceX/pull/2607 (or use an XXX placeholder, which the tooling auto-canonicalizes).

Extended reasoning...

The bug: The new changelog entry appended at the end of perf-changelog.yaml (config-keys: qwen3.5-fp8-mi355x-sglang-disagg-mtp) sets pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2557. Per the PR metadata this change is actually PR #2607 — 2557 appears to be a leftover from the PR description's note "Split from #2557", copy-pasted into the wrong field.

Where this actually breaks: It's worth correcting the originating report's attribution here — this is not caught by the check-changelog CI gate in .github/workflows/run-sweep.yml. That job invokes validate_perf_changelog.py without --pr-number, and its main() path only calls validate_matrix_compatible_change → validate_generated_config → process_changelog.py, none of which check pr-link against a PR number. So CI will pass with the wrong link in place.

The real enforcement point is merge time. utils/merge_with_reuse.sh runs utils/prepare_perf_changelog_merge.py canonicalize --pr-number 2607 (and again on resolve-conflict). canonicalize_appended_links() calls compare_entries(..., pr_number), which calls validate_added_pr_link(link, pr_number) in utils/validate_perf_changelog.py. That function requires an appended entry's link to be either an XXX placeholder (in PR_LINK_PLACEHOLDERS) or exactly match https://github.com/SemiAnalysisAI/InferenceX/pull/{pr_number}. I confirmed this via grep on both files: validate_added_pr_link builds expected from pr_number and raises when link not in PR_LINK_PLACEHOLDERS and link != expected.

Step-by-step proof:

1. PR [AMD] Qwen3.5-FP8 MI355X SGLang disagg MTP: EAGLE speculative decoding / Qwen3.5-FP8 MI355X SGLang disagg MTP：新增EAGLE投机解码配置 #2607 is merged/prepared via merge_with_reuse.sh, which calls prepare_perf_changelog_merge.py canonicalize --pr-number 2607.
2. canonicalize_appended_links() iterates newly appended entries and calls compare_entries(base_entries, head_entries, pr_number=2607).
3. For the entry with config-keys: qwen3.5-fp8-mi355x-sglang-disagg-mtp, compare_entries calls validate_added_pr_link("https://github.com/SemiAnalysisAI/InferenceX/pull/2557", 2607).
4. Inside, expected = "https://github.com/SemiAnalysisAI/InferenceX/pull/2607". Since the actual link is neither a placeholder nor equal to expected, the function raises ChangelogValidationError.
5. canonicalize only auto-fixes placeholder links (XXX) — it has no logic to correct a wrong but well-formed PR number — so this concrete mismatch surfaces as a hard failure that aborts the merge script, requiring manual intervention.

Why nothing upstream catches it today: The entry is syntactically valid (matches CANONICAL_PR_LINK) and simply refers to a real, different PR (#2557), so schema/format checks pass; only the PR-number-aware merge-time check catches the mismatch.

Fix: Change pr-link on the appended entry (perf-changelog.yaml, near line 5963) to https://github.com/SemiAnalysisAI/InferenceX/pull/2607, or use one of the accepted XXX placeholders so canonicalize can fill it in automatically at merge time.