Improve Hyperloom Roofline & Kernel-Opt based on Real Workload HW measurements - #1075
Open
rpoornac wants to merge 5 commits into
Open
Improve Hyperloom Roofline & Kernel-Opt based on Real Workload HW measurements#1075rpoornac wants to merge 5 commits into
rpoornac wants to merge 5 commits into
Conversation
The decode ceiling was built from vendor boost clock and vendor peak HBM bandwidth, neither of which an MI355X delivers under real power, thermal and CAC limits. That made the ceiling unreachable by construction, so within% read low and the saturation gate kept the loop chasing headroom that did not exist. Resolve the roof from measurement first, falling back to the tables only when nothing measured is available: - An on-node probe measures matrix-core issue rate per precision and non-temporal streaming bandwidth, cached per architecture and ROCm version. Cached results are validated against the requested GPU type so a session replayed on a foreign part cannot borrow another part's numbers. - Table peaks are derated by the engine clock the benchmark actually sustained, harvested from GPU telemetry. Idle-state samples are excluded; averaging them in would understate the clock and over-derate the ceiling. - Bandwidth falls back to a calibrated fraction of vendor peak (0.89 on MI355X, measured) rather than the raw peak. Every layer is fail-open: an unmeasured part keeps its previous behaviour. Provenance for which layer answered rides on the snapshot, since a probe-derived roof and a table-derived one are not the same quantity. Validated against GPT-OSS-120B MXFP4 on MI355X across the InferenceX gptoss_fp4_mi355x sweep (TP=1 conc 4-128, TP=8 conc 4-16). Measured throughput stays under the ceiling at all nine points, and the ceiling tightens ~11%. Co-authored-by: Cursor <cursoragent@cursor.com>
The untried-kernel queue sorted purely on gpu_pct, so the top_n cut kept whatever owned the most trace time -- including kernels already running at their roofline with nothing left to give. A kernel at 12% of GPU time and 90% efficiency would take the attempt ahead of one at 11% and 30% efficiency, even though the latter is where the recoverable time actually is. Rank on GPU-time share weighted by headroom (1 - efficiency) instead. The bypass report already publishes this as optimization_priority, so prefer that field and keep a single definition of the ROI; recompute it only for the TraceLens path, which carries efficiency_percent but no ROI. Rows whose efficiency was never measured fall back to the raw share, which reproduces the previous ordering rather than inventing headroom for them. The min_gpu_pct floor deliberately stays on the raw share: it answers whether a kernel is big enough to bother with, and headroom must not be able to talk a negligible kernel past it. Co-authored-by: Cursor <cursoragent@cursor.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Comment on lines
+26
to
+36
| from hyperloom.orchestrator.kernel.hw_probe import ( | ||
| DISABLE_ENV, | ||
| DeviceInfo, | ||
| MfmaRate, | ||
| ProbeResult, | ||
| load_cached, | ||
| normalize_arch, | ||
| probe_compute_peak_tflops, | ||
| probe_hbm_bandwidth_gb_per_sec, | ||
| rocm_version, | ||
| ) |
Comment on lines
+26
to
+36
| from hyperloom.orchestrator.kernel.hw_probe import ( | ||
| DISABLE_ENV, | ||
| DeviceInfo, | ||
| MfmaRate, | ||
| ProbeResult, | ||
| load_cached, | ||
| normalize_arch, | ||
| probe_compute_peak_tflops, | ||
| probe_hbm_bandwidth_gb_per_sec, | ||
| rocm_version, | ||
| ) |
The ROI weighting took headroom from efficiency_percent, which is compute-side by construction: compute_roofline sets it from FLOPs/peak_flops and leaves it near zero for anything memory-bound, publishing the binding-side number separately as roofline_attainment_pct. So every memory-bound kernel scored as pure headroom no matter how saturated it was. On a real GPT-OSS-120B trace, aiter::add_rmsnorm sits at 100% of its bandwidth roof and reported efficiency_percent 0.279, which handed it 99.7% of its GPU-time share as recoverable -- ranking the one kernel with nothing left to give at the top. That is the exact failure the weighting was added to prevent; it happened to work only for compute-bound kernels. Weight by roofline_attainment_pct, which already picks compute-vs-bandwidth utilization from bound_type. The TraceLens route publishes no attainment, so its compute-side number is trusted only when the kernel is compute-bound (where the two coincide) and a memory-bound row without attainment degrades to the raw share rather than being scored on the wrong axis. On the same trace the corrected weighting drops the saturated norm from 1.7416 to 0.0, moves an 87%-attained norm from 1.7638 to 0.2281, and leaves rotary embedding at 0.6702 of its 0.868 share because 22.8% attainment is real headroom. Compute-bound rows are unchanged, as attainment equals efficiency there. Selection on this workload is unchanged: the MoE kernels that dominate carry no analytical roofline and still rank on raw share. Co-authored-by: Cursor <cursoragent@cursor.com>
CI E2E report — ✅ Succeeded
|
A clamped roofline says the closed form does not fit the kernel, not that the kernel is saturated, so reading headroom off it silently retires work that was never measured. On a GPT-OSS-120B eager trace the elementwise form bills aiter::moe_cktile2stages_gemm2_ck for all 128 experts when topk=4 run, implying ~68 TB/s against a 7.13 TB/s roof. The estimate was truncated to 100% attainment, which drove the ROI of a kernel worth 14.6% of GPU time to zero. roofline_estimate_capped was already recorded but unread; both ROI paths now treat it as unknown attainment and fall back to raw share. Co-authored-by: Cursor <cursoragent@cursor.com>
Collaborator
|
@rpoornac can you comment on the motivation behind this feature? The default path today runs via the TraceLens Agent for prioritization and using MAF values. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Keep roofline assertions off real workload hardware measurements instead of default SoC Peak/static config.
Rank kernel-opt candidates by roofline headroom not just GPU-time share