Syncing up with the main branch - #594
Open
devalshahamd wants to merge 791 commits into
Open
Conversation
feat/std_dev_flag
Added reasoning section to standalone analysis
Kernel Fusion Perf. Model
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Previously, TraceDiff could only parse through one root node. This is
problematic for some traces (training traces) that contain many root
nodes. Now TraceDiff properly parses through every root node.
Additionally, TraceDiff performs an initial tree compression via
truncating the top level nodes that only have one children. Ex:
```
node A
|____ node B
|____ node C
|____ node D
|____ node E
```
becomes
```
node A
|____ node D
|____ node E
```
This is problematic if every node in the subtree has only children. In
these cases the entire tree compresses to nothing. Now, the tree
compression will stop when a CPU operation or kernel is encountered.
TraceDiff tests in private repo pass.
Closes #569
Closes #570
Standalone Agentic Analysis Update (4/3)
new patch files with flags
…-internal into update_patches
Update patches
Corrected the formatting of 'SGLang' section header and updated example script links for inference analysis.
This pull request introduces new scripts and patches to enable advanced profiling for SGLang and vLLM inference workflows, with a focus on CUDA graph capture and detailed trace annotation. The main changes include new Docker build scripts for SGLang and vLLM, and a set of patches that add options for shape discovery, roofline metrics, and improved trace collection in both SGLang and vLLM codebases. ** Work under progress to upstream these changes with vLLM and SGLang main repositories. ** <!-- Copyright (c) 2024 - 2025 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. --> --------- Co-authored-by: Deval Shah <devashah@amd.com>
## Summary
Adds a pull-based Prometheus metrics server to the TraceLens performance
harness. Previously the harness could only push metrics to Prometheus.
This PR adds a `--serve-only` mode that starts a long-running HTTP
endpoint so Prometheus (or any compatible scraper) can pull metrics on
its own schedule.
## Motivation
The existing OTLP push model requires Grafana Cloud credentials. A
pull-based endpoint decouples metric collection from the profiling job:
the batch job writes the metrics and exits, and Prometheus scrapes the
server independently on its own interval.
## Changes
### `tests/perf_tracker/tracelens_perf_harness.py`
**New function: `serve_prometheus_metrics`**
Starts a `prometheus_client` HTTP server that exposes metrics. The
server re-reads whatever file is passed to `--serve-only` on every
scrape.
**New CLI flags:**
| Flag | Default | Description |
|------|---------|-------------|
| `--serve-only <timing.json>` | — | Skip profiling; serve metrics from
an existing timing JSON. Blocks until Ctrl+C. |
| `--metrics-port <port>` | `9100` | Port for the Prometheus HTTP
endpoint. |
### `tests/perf_tracker/README.md`
Updated README to reflect changes.
## Usage
```bash
# Start the server
python tests/perf_tracker/tracelens_perf_harness.py \
--serve-only ./perf_results/timing.json
# Custom port
python tests/perf_tracker/tracelens_perf_harness.py \
--serve-only ./perf_results/timing.json --metrics-port 9200
```
Metrics are available at `http://<ip>:<port>/metrics`. Updating the file
passed to `--serve-only` will update the metrics on endpoint upon
refresh of the page.
---------
Co-authored-by: devalshahamd <deval.shah@amd.com>
Co-authored-by: Adeem Jassani <adeem.jassani@amd.com>
RN and CM --------- Co-authored-by: Matt Williams <Matt.Williams+amdeng@amd.com>
## Summary Refactors the top-level `README.md` to be shorter, more scannable, and better organized around how people actually use TraceLens. Adds a dedicated trace-collection step, splits usage into an SDK path and an agent path, trims the wall of per-format CLI snippets down to a format table plus documentation links **NOTE**: Working PR, feel free to directly edit ## Why | Pain point | Old | New | |---|---|---| | No guidance on obtaining a trace | Quick Start jumped straight to report generation; trace collection was a one-line afterthought | Dedicated `2. Collect a trace` step covering training/eager and vLLM/SGLang, with the profiling guide and Magpie-based Profiling skill | | README dense with CLI snippets | Five per-format bash blocks (PyTorch, compare, multi-rank, rocprofv3, pftrace) duplicated content already in `docs_original/` | Kept the format summary table; details deferred to the linked per-format docs | | No clear entry point per audience | Examples and the agent were listed without framing | New `## Usage` section split into an SDK subsection (examples/notebooks) and a `TraceLens Agent` subsection | --------- Co-authored-by: Adeem Jassani <ajassani@amd.com>
<!-- Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. --> Manager feedback --------- Co-authored-by: Matt Williams <Matt.Williams+amdeng@amd.com>
## Summary Follow-up to #825. Clarifies the hardware section of the compatibility matrix and restores the section ordering. - **TraceLens runs on a CPU host.** It parses trace files and never executes kernels or talks to a GPU, so no accelerator is required to install or run it — you can analyze traces on a laptop or CI runner with no GPU present. The hardware section now says this explicitly. - **Reframes the accelerator table** as the *target accelerator whose traces you analyze* (the tracing environment) used for roofline analysis — not a requirement for the machine running TraceLens. This avoids the earlier layout reading as "you need an MI300X to use TraceLens." - **Restores `ROCm and CUDA compatibility` as the leading section**, with `Hardware requirements` following it (the accelerator detail now sits under a `Target accelerators for roofline analysis` subsection). ## Test plan - [x] Copyright header intact on `compatibility.md` - [x] No broken internal links introduced by this change - [ ] Read the Docs build renders the reordered/reworded section Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary
Graph replay events sometimes appear in different order from capture
events. The reasons for this are not clear and is something that needs
to be investigated.
For one trace the events were ordered as follows:
```
Capture cycle: NORM G128 ROPE KVC ATTN_QKV ATTN_RED OTHER NORM G512 ACT G256
Graph replay cycle: ACT G256 NORM G128 ROPE KVC ATTN_QKV ATTN_RED OTHER NORM G512
```
The activation in some attention->FFN cycles appeared at the start in
some of the graph replay cycles which is incorrect. However, the count
of each kernel invocation is the same between the two traces.
To account for this, there is now a phase 3 fallback. After normal fully
positional graph capture merge and greedy graph capture merge fails,
this new attempt at merging is used.
## Approach
Group both event lists by kernel name, then match positionally within
each group. Reorder **graph replay** events to match **capture** order
(not the reverse), so the merged tree preserves the logical graph-node
execution structure.
### Algorithm
1. **Group by name**: Build `{kernel_name: [event, ...]}` for both
capture and graph events, preserving within-group order.
2. **Verify counts**: For every kernel name, check that the count in
capture equals the count in graph. If any name has a mismatch, return
`None` (alignment failed).
3. **Reassemble in capture order**: Walk the capture event list. For
each capture event, pull the next unused graph event with the same
kernel name from its group. The result is the graph events reordered to
match capture (logical) order.
```
capture (CPU dispatch order): [A0, B0, C0, A1, B1, A2]
graph (GPU execution order): [A0, C0, B0, A1, B1, A2]
Step 1 - group by name:
capture_groups: { A: [A0, A1, A2], B: [B0, B1], C: [C0] }
graph_groups: { A: [A0, A1, A2], B: [B0, B1], C: [C0] }
Step 2 - counts match: A=3, B=2, C=1 OK
Step 3 - walk capture order, pull from graph groups:
capture[0] = A -> graph_groups[A][0] = A0
capture[1] = B -> graph_groups[B][0] = B0
capture[2] = C -> graph_groups[C][0] = C0
capture[3] = A -> graph_groups[A][1] = A1
capture[4] = B -> graph_groups[B][1] = B1
capture[5] = A -> graph_groups[A][2] = A2
Result: graph events reordered to [A0, B0, C0, A1, B1, A2]
-> matches capture order
-> capture[i] <-> graph[i] pairing is correct
```
<!-- Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. -->
## Summary - The compatibility matrix listed the Operating system requirement as `Linux (OS-independent core)`, which is self-contradictory: it reads as "Linux is required" while also claiming OS independence. - Replaced the cell with `N/A` and moved the nuance into the Notes column: TraceLens analysis and report generation are pure Python and run on any OS; capturing ROCm-based traces requires a Linux ROCm environment, and TraceLens only reads those traces afterward. - Matches the `N/A` convention used elsewhere in the compatibility docs. ## Test plan - [ ] Docs build renders the compatibility table with the updated Operating system row. Co-authored-by: Cursor <cursoragent@cursor.com>
<!-- Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. -->
## Summary
This PR adds name disambiguation to TraceDiff's Wagner-Fischer alignment
along with other minor changes.
---
## Problem
Wagner-Fischer aligns child node lists by normalized name. When multiple
children share the same name, WF treats them as interchangeable (all
have edit cost 0) and picks matches based on positional alignment. This
can pair structurally unrelated nodes.
Consider a transformer attention block where both traces have multiple
`aten::bmm` children, but one trace has an extra wrapping layer that
shifts positions:
```
trace1 children: [aten::bmm_A, aten::softmax, aten::bmm_B, aten::linear]
trace2 children: [aten::dropout, aten::bmm_A, aten::softmax, aten::bmm_B]
```
Here `bmm_A` computes Q×K^T (children: `aten::transpose`,
`aten::expand`) and `bmm_B` computes Attn×V (children: `aten::expand`,
`aten::reshape`). They have completely different subtrees but identical
names.
WF aligns by position and produces:
```
match: trace1[0] aten::bmm_A ↔ trace2[1] aten::bmm_A ✓ correct
match: trace1[1] aten::softmax ↔ trace2[2] aten::softmax ✓ correct
match: trace1[2] aten::bmm_B ↔ trace2[3] aten::bmm_B ✓ correct
insert: trace2[0] aten::dropout ✓ correct
delete: trace1[3] aten::linear ✓ correct
```
But if the position shift is different — say trace2 has the `aten::bmm`
nodes in a different order relative to other ops — WF may instead
produce:
```
match: trace1[0] aten::bmm_A ↔ trace2[3] aten::bmm_B ✗ wrong!
```
WF sees both as `aten::bmm` with cost 0 and has no way to distinguish
them. The resulting diff falsely reports that bmm_A's children changed
(transpose/expand → expand/reshape) when in reality the operations are
identical across traces — they were just matched with the wrong
counterpart.
---
## Solution: Post-WF Disambiguation
A new post-processing pass (`_disambiguate_same_name_candidates`) runs
after every WF alignment. It identifies ambiguous matches and resolves
them using subtree structure.
### How it works
**Step 1 — Index unmatched nodes by name.**
Partition WF ops into match/delete/insert. Group deleted indices (trace1
unmatched) and inserted indices (trace2 unmatched) by name.
**Step 2 — Detect ambiguity.**
For each WF match `(i, j)` with name N, check if there are also
unmatched nodes named N on either side. If not, the match is unambiguous
— skip it.
**Step 3 — Disambiguate using BFS child-name levels.**
When ambiguity exists:
- **Anchor**: the matched node on the fixed side (e.g., `children1[i]`
when disambiguating trace2 candidates).
- **Candidates**: the current WF match plus all same-named unmatched
nodes on the other side.
`_bfs_child_name_levels` computes a fingerprint for each node: a list of
sorted tuples, one per BFS depth level, containing the normalized names
of all GPU-path children at that depth. Example:
```
aten::linear
├── aten::mm ← level 1
├── aten::add ← level 1
│ ├── aten::copy_ ← level 2
│ └── aten::mul ← level 2
└── aten::reshape ← level 1
fingerprint = [
("aten::add", "aten::mm", "aten::reshape"),
("aten::copy_", "aten::mul"),
]
```
`_find_bfs_survivors` compares each candidate's fingerprint against the
anchor's. A candidate is evicted if any level differs. If exactly one
candidate survives and it differs from WF's original choice, a
reassignment is recorded.
**Step 4 — Apply reassignments.**
Substitution maps (`swap_i`, `swap_j`) rewrite the ops list: demoted
match sides become delete/insert, promoted nodes are consumed from the
delete/insert pool, and the new match is emitted in place.
### Integration
The disambiguation pass is called at the end of `wagner_fischer`, so all
callers (Phase 2, Phase 3, Phase 4 in `traverse_and_merge`) get
disambiguated results automatically.
This new post-WF disambiguation does not change the number of
matches/inserts/deletes. It only changes existing matches.
---
## Other Changes
### `cuda_driver` category support
`cuda_driver` events (e.g., `cuLaunchKernel`) are now treated
equivalently to `cuda_runtime` events throughout the TraceDiff pipeline:
### Kernel launch name equivalence
A new mapping (`_KERNEL_LAUNCH_EQUIVALENTS`) normalizes vendor-specific
kernel launch API names to a common name during comparison:
```python
{
"hipModuleLaunchKernel": "__kernel_launch__",
"cuLaunchKernel": "__kernel_launch__",
}
```
This prevents spurious diffs when comparing HIP and CUDA traces that use
different launch APIs for the same operation.
### `collapse_single_gpu_child` stops at kernel-dispatching cpu_ops
Previously, `collapse_single_gpu_child` only stopped at kernel dispatch
leaf nodes or nodes with != 1 GPU-path child. Now it also stops at
`cpu_op` nodes that have a kernel dispatch child. This prevents
collapsing through the last meaningful cpu_op before a kernel launch,
preserving it as a comparison point in the diff.
### Phase 3 reconciliation preserves Phase 2 matches
When `reconcile_unmatched` expands wrapper nodes, the resulting children
lists are re-indexed. Previously, all children were re-aligned with a
fresh WF pass, which could undo correct Phase 2 matches (after the new
post-WF disambiguation). Now, Phase 2 match ops are pinned by UID pair,
and WF only runs on the unpinned (newly expanded) children.
---------
Co-authored-by: Spandan More <spandan.more@amd.com>
## Background In phase 4 of matching, TraceDiff takes the children that have already been passed through wagner fischer and filters them to only the unmatched children. It then collapses the children down to the first event with more than one child and performs wagner fischer again on just these unmatched children. ## Summary This PR adds a final wagner fischer at the end of phase 5 to more aggressively match events. If two events are python functions but slightly differ in name, `_normalize_name_for_comparison` now strips the function name as well as the path leading to the file. ## Details If the event name is ``` vllm/model_executor/layers/fused_moe/gpt_oss_triton_kernels_moe.py(148): legacy_routing_from_sparsematrix ``` _normalize_name_for_comparison(event_name, strip_details=True) now changes event name to be: ``` gpt_oss_triton_kernels_moe.py ``` The regex pattern only applies to python functions. This more aggressive version of wagner fischer is only applied at the end of phase 4, to avoid matching unrelated events in the first wagner fischer passes when there may be better matches. For instance: ``` trace1 (collapsed children): trace2 (collapsed children): [0] kernels.py: compute_attention [0] kernels.py: compute_ffn [1] kernels.py: compute_ffn [1] kernels.py: compute_attention ``` If we do aggressive WF, children become: ``` trace1: [kernels.py, kernels.py] trace2: [kernels.py, kernels.py] ``` and kernels are matched positionally: ``` [0] kernels.py ↔ [0] kernels.py → compute_attention ↔ compute_ffn ✗ WRONG [1] kernels.py ↔ [1] kernels.py → compute_ffn ↔ compute_attention ✗ WRONG ``` Therefore an initial normal WF pass occurs first, followed by a the more aggressive WF pass on the remaining unmatched events. --------- Co-authored-by: Claude Sonnet 4 <noreply@anthropic.com>
…s guide (#824) ## Summary Completes the documentation migration by porting the remaining legacy topics from `docs_original/` into the rocm-docs-core site under `docs/`, then reorganizing the core-feature docs so the SDK story lives in one place. Follows the conventions used across the existing pages (MyST, `{meta}` blocks, "Related topics", sentence-case headings, copyright headers). **Migrated pages + 3 tutorial notebooks:** - **Reference** — `perf-report-columns` (full column glossary) - **Concepts** (new section) — `trace2tree` (concept-only), `torch-profiling-analysis`, `shape-metadata`, `gemm-analysis` (merged `aimodels_gemms` + `gemm_dim_eff`), `inference-analysis` (paged-attention roofline model), `triton-perf-model-walkthrough` - **How to** — `sdk-analysis`, `perf-model-without-trace`, `generate-perf-report-pytorch-inference` (end-to-end workflow), `compare-perf-reports`, `nccl-analysis`, `origami-integration`, `agentic-mode` - **Tutorials** (new section) — `torch-profiling`, `distributed-profiling`, `anomaly-detection` notebooks, rendered via `myst-nb` **Core-feature docs reorg:** - `how-to/sdk-analysis` — new consolidated SDK guide: tree loading/traversal (`TreePerfAnalyzer`), GPU-timeline breakdown (`GPUEventAnalyser`), per-op GPU time, roofline metrics, adding a new op, and `nn.Module` attribution. Absorbs the former `tree-perf-analysis` and `gpu-event-analysis` how-tos. - `conceptual/trace2tree` — slimmed to the call-tree concept; SDK traversal how-to content moved into `sdk-analysis`. - `how-to/perf-model-without-trace` — new guide for analytical FLOPs/bytes from shapes alone (no trace). - `how-to/generate-perf-report-pytorch-inference` — restructured into collect → split → generate → merge, with the vLLM/SGLang build tables, profiler flags, and the steady-state `split_inference_trace_annotation` workflow; conceptual roofline math split out to `conceptual/inference-analysis`. **Infra:** - `_toc.yml.in` and `index.rst` landing cards updated with Concepts, Tutorials, and the new Reference/How-to entries - `rocm-docs-core` bumped `1.22.0` → `1.32.0` (its built-in `myst-nb` renders the notebooks; execution disabled at build time via `nb_execution_mode = "off"`) - All inbound links repointed from `docs_original` to the new pages (README, CODEOWNERS, API reference, agent READMEs) - `CODEOWNERS` — added inference-doc ownership (`how-to/generate-perf-report-pytorch-inference`, `conceptual/inference-analysis`) - `docs_original/` fully retired (directory removed) - Merged latest `main`. ## Content-fidelity check Every example in `sdk-analysis` is traceable to a real source (verified against git history / source / the torch-conference poster): GPU-timeline / per-op / roofline tables from the legacy `tree-perf-analysis`; subtree + parent-chain examples from the legacy `trace2tree`; timeline metric definitions from the legacy `gpu-event-analysis`; the `nn.Module` GPU-time tree from the poster; and all SDK method names verified present in `TraceLens/`. ## Decisions worth a look - **SDK consolidation** — `TreePerf` + `GPUEventAnalyser` how-tos merged into one `sdk-analysis` page; `trace2tree` kept concept-only. - **Analytical perf model** given its own `perf-model-without-trace` how-to (no trace required). - **Inference** split into a how-to (workflow) and a concept (roofline math). - **GEMM** docs merged into one `conceptual/gemm-analysis` page. - **triton-perf-model-walkthrough** placed under Concepts (explains how metrics are computed) rather than How-to. - Notebook content ported as-is; outputs preserved, not re-executed. ## Validation - Read the Docs build: green on the latest push. - Lint: green. - Internal/anchor/repo-asset link check is clean (0 broken links). ## Test plan - [ ] Read the Docs build succeeds with the ROCm theme - [ ] Tutorials render (myst-nb) with their captured outputs - [ ] Concepts / Tutorials / Reference nav sections appear correctly - [ ] Spot-check migrated technical content against source for accuracy --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Tharun Adithya Srikrishnan <tsrikris@amd.com> Co-authored-by: Deval Shah <devashah@amd.com> Co-authored-by: Kyle Hoffmeyer <97640268+kyle-hoffmeyer@users.noreply.github.com>
## Summary #824 retired the `docs_original/` directory as part of the docs migration. #823 (branched before #824 and merged just after it) reintroduced a link to `docs_original/Inference_analysis.md`, which no longer exists on `main` — leaving a dead link in `TraceLens/Agent/Profiling/README.md`. This repoints that reference to the migrated how-to page and restores the descriptive link text. - `docs_original/Inference_analysis.md` (deleted) -> `docs/how-to/generate-perf-report-pytorch-inference.md` ## Test plan - [ ] Link resolves to the migrated inference how-to page Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary - Replace `pip install --no-deps /tmp/TraceLens` in inference Docker build workflows - Install TraceLens with runtime dependencies such as `orjson` and `matplotlib` - Add build-time validation for core TraceLens inference imports and CLI commands ## Details The vLLM and SGLang inference Docker workflow scripts previously installed TraceLens with `--no-deps`, which allowed images to contain TraceLens entry points while missing required runtime dependencies. This could make TraceLens post-processing fail later with errors such as: ```text ModuleNotFoundError: No module named 'orjson' ``` Updated scripts: - examples/custom_workflows/inference_analysis/build_docker_vllm.sh - examples/custom_workflows/inference_analysis/build_docker_sglang.sh Both now install TraceLens with dependencies: `pip install --upgrade /tmp/TraceLens`
<!-- Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. --> --------- Co-authored-by: Matt Williams <Matt.Williams+amdeng@amd.com> Co-authored-by: Adeem Jassani <ajassani@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
<!-- Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. --> # Pull Request Template > **Note to AMDers:** > This is a public repository. Please do **not** upload any confidential or customer data. Make sure all such data has been anonymized or removed before making this PR. If you need to attach any private files or links, please insert a Internal OneDrive Link or a Jira Ticket Link instead. --------- Co-authored-by: Matt Williams <Matt.Williams+amdeng@amd.com>
Reorders the documentation header so TraceLens support and project links appear first. Hyperloom remains available as a related ecosystem link at the end of the menu without implying that TraceLens is a Hyperloom component. This is a navigation-only change. Made with [Cursor](https://cursor.com) Co-authored-by: Cursor <cursoragent@cursor.com>
<!-- Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. --> # Pull Request Template > **Note to AMDers:** > This is a public repository. Please do **not** upload any confidential or customer data. Make sure all such data has been anonymized or removed before making this PR. If you need to attach any private files or links, please insert a Internal OneDrive Link or a Jira Ticket Link instead.
- Modify the repeatability test script to be able to run against a local Pi server - Create a script to run a local server (usually vllm but sglang should work), install and configure Pi, and run local evals using it --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Don't rely on curl to see if the local server is ready
## Summary Raises the supported-Python floor from **3.6** (EOL since Dec 2021) to **3.11** and removes the compatibility scaffolding that the old floor required. Because `enum.StrEnum` is in the standard library as of Python 3.11, the two backport dependencies (`backports.strenum`, `StrEnum`) and their try/except import fallbacks are now dead code and have been deleted. CI, Read the Docs, and the README badge are aligned to 3.11 to match. Net: 7 files changed, 8 insertions(+), 23 deletions(-). ## Why The `python_requires=">=3.6"` floor forced the project to carry `StrEnum` shims for pre-3.11 interpreters — two extra install-time dependencies plus duplicated fallback import blocks in `util.py` and `jax_analyses.py`. Raising the floor to 3.11 lets the code import `StrEnum` straight from the stdlib and drops the shims entirely, with CI and docs already running 3.10 (a one-step bump to 3.11). ## What changed ### Packaging (1 file) | File | Change | |---|---| | `setup.py` | `python_requires` `>=3.6` → `>=3.11`; removed the `backports.strenum` and `StrEnum` conditional (`python_version<"3.11"`) deps; added `Programming Language :: Python :: 3.11` classifier | ### Python (2 files) | File | Change | |---|---| | `TraceLens/util.py` | Collapsed the `StrEnum` try/except backport fallback to a plain `from enum import StrEnum` | | `TraceLens/TreePerf/jax_analyses.py` | Same `StrEnum` import simplification | ### CI / Docs config (4 files) | File | Change | |---|---| | `.github/workflows/lint.yml` | `python-version` `3.10` → `3.11` | | `.github/workflows/regression-tests.yml` | `python-version` `3.10` → `3.11` | | `.readthedocs.yaml` | `tools.python` `3.10` → `3.11` | | `README.md` | Python badge `3.6+` → `3.11+` | ## Test plan - [x] `black --check setup.py TraceLens/util.py TraceLens/TreePerf/jax_analyses.py` — clean - [x] Import smoke test — `TraceLens.util` and `TraceLens.TreePerf.jax_analyses` import; `StrEnum` resolves from stdlib `enum` - [x] Regression suites pass: `test_subtract_intervals`, `test_graph_mode`, `test_kernel_launchers`, `test_call_stack_perf_report`, `test_perf_report_regression`, `test_compare_perf_reports`, `test_inference_perf_report`, `test_collective_analysis`, `test_all2allv_summary`, `test_detect_recompute` - [x] Grep check — no remaining `backports.strenum` / `python_version<"3.11"` references, no stale `3.6` / `3.10` version pins in changed configs ## Net diff ``` 7 files changed, 8 insertions(+), 23 deletions(-) ``` Co-authored-by: Tharun Adithya Srikrishnan <tsrikrsi@amd.com>
This pull request makes a targeted change to the `get_compute_precision` method in `moe_perf_model_extensions.py` to ensure that when the input data type is either `"fp8"` or `"bf16"`, the method returns `"fp8"` instead of mapping the type using `torch_dtype_map`. This likely addresses a specific requirement for handling these data types in the performance model. - Precision handling update: * Modified the `get_compute_precision` method in `moe_perf_model_extensions.py` to return `"fp8"` when the input data type is `"fp8"` or `"bf16"`, otherwise it falls back to `torch_dtype_map(dtype)` <!-- Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. --> --------- Co-authored-by: Deval Shah <devashah@amd.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
`hipMemsetAsync` is the name of the dispatcher. It can call kernels without `memset` in the name such as buffer copying kernels like `__amd_rocclr_fillBufferAligned`. `hipMemsetAsync` should be matched with these buffer copying kernels.
`test_inference_perf_report` should test that call stack information is present and correct since a critical goal of capture merging is to add call stack info to operations.
## Summary This PR rewrites `_disambiguate_same_name_candidates` in to fix three correctness issues with how same-named events are matched between traces. Fixes issues in #800. ## Details #### Issue 1: Matched events excluded from candidate pool **Problem:** The old implementation only considered unmatched events (inserts/deletes) as candidates for reassignment. Already-matched events with the same name were never considered. This meant that if WF produced two incorrect matches, the function could never correct them. **Fix:** Candidate pools now include ALL same-named events on each side -- both matched and unmatched. #### Issue 2: Same candidate assigned to multiple matches **Problem:** The old implementation iterated over each match op independently, evaluating candidates for that single match in isolation. If two different match ops both identified the same unmatched candidate as the correct match, that candidate could be reassigned to both -- producing duplicate trace2 (or trace1) indices in the final match list. **Fix:** The new implementation operates per-name-group, scoring all `(i, j)` pairs at once and greedily assigning them from a single sorted list. Each index can only be assigned once. #### Issue 3: Wrong side left unmatched **Problem:** Consider 5 events in trace1 and 3 events in trace2, all with the same name. WF aligns positionally: trace1[0, 1, 2] get matched, trace1[3, 4] become deletes. The old code iterated per-match-op and tried to swap in unmatched candidates, but there was no guarantee that the *correct* trace1 events were left unmatched. Since trace2 has fewer events (3 < 5), all 3 trace2 events should be matched with *some* trace1 event, but the old logic couldn't enforce this. **Fix:** The function now operates per-name-group rather than per-match-op. For each ambiguous name, it collects all indices on both sides and produces exactly `min(n1, n2)` matches using a greedy algorithm scored by BFS subtree match depth. This guarantees the minority side (the side with fewer events of that name) is fully matched. Only majority-side leftovers become deletes/inserts. ## Algorithm For each normalized name that appears on both sides with ambiguity (>1 on at least one side) and participates in at least one existing match: 1. Collect all trace1 and trace2 indices with that name (matched + unmatched) 2. Precompute BFS-level subtree fingerprints for each node 3. Score every `(i, j)` pair as `(full_levels, partial_overlap)` where `full_levels` is the number of BFS levels with identical child-name tuples before the first divergence, and `partial_overlap` is the multiset intersection size at that first diverging level 4. Sort pairs by score descending, breaking ties by: prefer existing WF match, then positional order 5. Greedily assign `min(n1, n2)` non-conflicting pairs from the sorted list 6. Remaining majority-side nodes become deletes/inserts ## Tests - `test_no_ambiguity_returns_ops_unchanged` -- No same-named duplicates; verifies ops are returned as-is - `test_unique_survivor_trace2_triggers_reassignment` -- WF matched trace1[0] to trace2[0], but trace2[1] has an identical subtree; verifies reassignment to the better candidate - `test_multiple_survivors_keeps_original_wf_match` -- Two trace2 candidates with identical (empty) subtrees; verifies WF's original choice is preserved when the tie cannot be broken - `test_no_duplicate_trace2_index_in_matches` -- 3 trace1 vs 4 trace2 events, all same name; verifies no trace2 index appears in more than one match - `test_no_duplicate_trace1_index_in_matches` -- 4 trace1 vs 3 trace2 events, all same name; verifies no trace1 index appears in more than one match - `test_insert_and_delete_preserved_when_no_reassignment` -- Unambiguous inserts and deletes pass through unchanged - `test_multiple_same_name_different_subtrees_correct_matches` -- 2 trace1 vs 3 trace2, each with a distinct kernel child; verifies a wrong positional match is corrected to the right subtree pairing - `test_second_match_falls_back_when_best_candidate_already_claimed` -- Two trace1 events both prefer the same trace2 candidate; verifies the second gets its next-best available match - `test_swapped_matches_corrected_via_matched_candidates` -- WF produced two all-matched events with names swapped; verifies matched events are now candidates (Issue 1) - `test_minority_side_fully_matched` -- 5 trace1 vs 3 trace2 events, all same name; verifies all minority-side events are matched and the correct trace1 events are left unmatched (Issue 3) - `test_non_positional_match_across_sides` -- 5 vs 3 events where two trace1 events share BFS level-1 with a trace2 event but only one matches at level-2; verifies the deeper match wins - `test_partial_overlap_breaks_tie_at_diverging_level` -- Two trace1 events diverge from the same trace2 event at the same BFS depth but with different overlap counts; verifies partial overlap correctly breaks the tie - `test_all_ops_indices_valid` -- All match/delete/insert indices in the result are within bounds --------- Co-authored-by: Spandan More <spandan.more@amd.com>
<!-- Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. --> # Pull Request Template > **Note to AMDers:** > This is a public repository. Please do **not** upload any confidential or customer data. Make sure all such data has been anonymized or removed before making this PR. If you need to attach any private files or links, please insert a Internal OneDrive Link or a Jira Ticket Link instead.
## Summary
Cleanup of TraceDiff:
- Removed code duplication and dead code
- Eliminated unnecessary UID→event and vice versa conversions
- Moved utility functions to utils.py
- Trimmed code by adding helper functions
---
## Changes
### 1. Fix duplicate imports
`re` was imported 3 times, `json` and `os` each twice. Consolidated to
single imports with standard ordering. Removed unused imports
(`typing.Any`, `Callable`, `cast`, `Dict`, `Optional`,
`TraceLens.util`).
### 2. New file: `TraceLens/TraceDiff/util.py`
Extracted constants and pure utility functions into a shared module (79
lines):
| Item | Notes |
|---|---|
| `_TRACELENS_DEBUG`, `_GRAPH_LAUNCH_NAMES`,
`_KERNEL_LAUNCH_EQUIVALENTS`, `_KERNEL_DISPATCH_CATEGORIES` | Constants
previously scattered/duplicated in `trace_diff.py` |
| `_TraceKeys`, `_UID`, `_NAME`, `_CATEGORY`, `_TS`, `_DUR` | Short
aliases for `TraceLens.util.TraceEventUtils.TraceKeys.*`, replacing 7
verbose inline references |
| `_sort_by_ts(nodes)` | Takes nodes, returns UIDs sorted by timestamp.
Replaces two identical `sort_by_ts(uids, uid2node)` closures that
resolved UIDs to nodes internally |
| `_get_name_node(node, strip_details)` | Gets normalized comparison
name from a node dict. Replaces `get_name_uid(uid, tree_num)` which
resolved UID→node internally |
| `_normalize_name_for_comparison(name, strip_details)` | Unchanged
logic, moved to util |
| `_list_to_tuple(obj)` | Unchanged logic, moved from closure to util |
| `_get_node_arg(node, key)` | Replaces 4 identical
`get_input_shape/strides/type/concrete_inputs` closures |
| `_is_gpu_path(node)` | Was `TraceDiff.is_gpu_path(self, node)` — no
dependency on `self` |
| `_is_kernel(node)` | Was `TraceDiff.is_kernel(self, node)` — no
dependency on `self` |
### 3. Eliminate UID→node round-tripping
Functions throughout `merge_trees` passed UIDs, resolved them to nodes
via `uid2node.get()`, extracted UIDs, then the next function resolved
the same UIDs again. Fixed by using `TraceToTree.get_children_events()`
and working with nodes internally:
- **`get_children_with_missing`** — rewritten to use
`get_children_events()` and `_get_name_node()`. Nodes throughout, UIDs
extracted only at return via `_sort_by_ts`. Eliminated 3 redundant
resolution round-trips.
- **`collapse_single_gpu_child`** — uses `get_children_events()` instead
of manual `node["children"]` → `uid2node` resolution.
- **`reconcile_unmatched`** — uses `get_children_events()` and
`_get_name_node()` for immediate children. Substitution maps store
nodes; non-substituted UIDs resolved once in the expansion loop.
- **`subtree_contains_cuda_runtime`** — takes node instead of `(uid,
uid2node)`.
### 4. Collapse 3 copy-pasted row-building blocks in
`generate_diff_stats`
The `traverse` function had three branches (combined, trace1-only,
trace2-only) that each built the same ~45-line row dict. Introduced
three shared helpers:
- **`_collect_gpu_rows(...)`** — iterates GPU event UIDs, walks up to
parent CPU op, builds row dict. Replaces identical code in all three
branches.
- **`_resolve_lca_from_parent(combined_parent_node, uid_key,
tree_num)`** — resolves LCA name/ID/children. Replaces identical 10-line
blocks in trace1/trace2 branches.
- **`_compute_lca_busy_time(gpu_event_uids, uid2node)`** — shared by all
three branches.
### 5. Deduplicate tree printing
`print_merged_subtree` and `print_merged_tree` contained near-identical
~35-line recursive formatters. Extracted `_format_merged_subtree`
generator that yields formatted lines. Both methods now delegate to it.
### 6. Simplify `_gpu_path_child_names_at_bfs_levels`
Removed `normalize_fn` and `get_name_fn` callback parameters — the
function now calls `_get_name_node` directly. Signature went from 6
parameters to 3.
### 7. Consistent use of TraceKeys aliases
All trace event key accesses now use `_UID`, `_NAME`, `_CATEGORY`,
`_TS`, `_DUR` aliases instead of raw strings. Removed unnecessary
`"Name"` (uppercase) and `"category"` fallbacks — neither key exists in
actual trace data.
### 8. Consistent use of `_KERNEL_DISPATCH_CATEGORIES`
Added `_KERNEL_DISPATCH_CATEGORIES = ("cuda_runtime", "cuda_driver")` to
`util.py`. Replaced 5 inline `("cuda_runtime", "cuda_driver")` tuples
throughout `trace_diff.py`.
### 9. Fix `_invalidate_merged_cache` — stale cache bug
`_uid1_to_merged_id` and `_uid2_to_merged_id` were not cleared when the
merged tree was rebuilt. If `merge_trees()` were called twice, the
UID→merged_id lookup maps would return stale results from the previous
merge. Fixed by initializing both to `None` in `__init__` and clearing
them in `_invalidate_merged_cache`.
### 10. Remove dead code
| Function | Lines | How it became dead |
|---|---|---|
| `get_kernel_info_subtree` | 8 | Introduced in PR #241, call sites
removed in PR #479, definition left behind |
| `find_last_cpu_op_on_gpu_path` | 64 | Only called by itself
(recursion). Never called externally. |
| `find_all_last_cpu_ops_on_gpu_path` | 52 | Same — only self-recursive,
never called externally. |
| `add_to_pod` | 29 | Call sites removed in PR #498, definition left
behind |
| `get_corresponding_uid` | 5 | Introduced in PR #241, never called
anywhere in any branch |
| `_get_duration` | 9 | Created during early cleanup, never referenced |
| `safe_children` | 7 | Replaced by `get_children_events()` |
| `get_name_uid` | 3 | Replaced by `_get_name_node()` |
| `normalized_name` | 3 | Inlined at call sites |
Also removed unnecessary `isinstance(node, dict)` check in
`subtree_contains_cuda_runtime` and simplified `_get_op_name` from a
3-key lookup with try/except to a single `node.get(_NAME)`.
### 11. Consistent print prefixes
All print statements now use `[TraceDiff]` prefix for consistent log
identification.
### 12. Regression test
Added `tests/test_tracediff_regression.py` — generates TraceDiff output
for TimeSformer mi300-vs-h100 traces and compares all 6 output files
against reference output generated from upstream-main:
- `diff_stats.csv` — all columns compared with tolerance
- `diff_stats_unique_args_summary.csv` — all columns compared with
tolerance
- `cpu_op_map.json`, `cpu_op_map_trace1.json`, `cpu_op_map_trace2.json`
— unified diff
- `merged_tree_output.txt` — unified diff
Supports `--update-references` flag to regenerate reference output.
## File summary
| File | Before | After | Delta |
|---|---|---|---|
| `TraceLens/TraceDiff/trace_diff.py` | 2,228 | 1,659 | −569 (−26%) |
| `TraceLens/TraceDiff/util.py` | — | 79 | +79 |
| `tests/test_tracediff_regression.py` | — | 94 | +94 |
| `tests/traces/tracediff_test/` | — | 6 files | reference output |
## Summary Merges `staging_agent` into `main`: **52 files changed, +630 / −272**. The release is anchored by a repo-wide migration of the agent specs from `.cursor/` to a portable `skills/` layout, adds graph-capture trace comparison support to the Analysis Agent, consolidates roofline classification onto TraceLens Core, and upgrades the eval pipeline to run both analysis scopes end-to-end in one invocation. --- ## Key Features ### 1. Migrate agent specs to the portable `skills/` layout (PR #750) - Moves every orchestrator, sub-agent, template, and reference file out of `.cursor/` into a standard `skills/<skill-name>/` structure (`analysis-orchestrator`, `magpie-benchmark-profiling`, `eval-post-processing`, `quality-llm-eval`, `workflow-llm-eval`), following the **agent-skills spec**. - All 13 category analyzer specs (`gemm-analyzer.md`, `moe-analyzer.md`, `sdpa-analyzer.md`, etc.) relocate to `skills/analysis-orchestrator/agents/`; the orchestrator prose moves to `skills/analysis-orchestrator/reference.md`, and `analysis_template.md` / `sub_agent_spec.md` move under the skill's `templates/`. - Adds a top-level `SKILL.md` entry point per skill so each is self-describing and discoverable. - Largest single change in the release (38 files, +297 / −133), predominantly renames. ### 2. Graph-capture trace comparison in the Analysis Agent (PR #797) - Enables **comparative analysis of paired graph-capture traces** end-to-end through the agent, wiring in upstream Core support for passing two capture-trace directories and two replay traces. - Hardens `tracediff` handling of **call-stack divergences** that arise when comparing captures across framework versions. - Extends the eval schema in `combined_traces_comparative.csv` to carry the capture-trace directory through the prompt, and refreshes the comparative e2e tarball with a representative paired-capture inference fixture. ### 3. Defer roofline computation to TraceLens Core (PR #837) - `calculate_efficiency()` now reads the authoritative **`Roofline Bound`** CSV column emitted by the core perf pipeline instead of re-deriving compute/memory-bound classification from `FLOPS/Byte` and a recomputed balance point — **one owner** for the ridge-point decision. - Removes a duplicated peak-resolution path that could silently drift from Core, and fixes an edge case where the agent classified ops against a fallback peak that Core intentionally left unclassified. - Function signature and the `efficiency.bound_type` field name/values/JSON location are unchanged, so no callers or downstream consumers are affected. ### 4. Full repeatability pipeline across both scopes (PR #856) - With no scope argument, `run_repeatability_parallel.sh` now runs the **complete pipeline sequentially**: generate golden references (standalone + comparative), run repeatability evals for both, then emit a single combined `pr_report.md` + `fix_ticket_report.md`. - Passing `standalone` / `comparative` (or `COMPARISON_SCOPE`) still restricts to a single scope; golden refs are always regenerated from scratch via the `agent` CLI, with `--pi`/`USE_PI=1` affecting only the repeatability and post-processing stages. - Refreshes the standalone/comparative unit and e2e test tarballs to match. --- ## Lines of Code Changed | Section | Files | Insertions | Deletions | Net | |---------|-------|------------|-----------|-----| | 1. Migrate agent specs to `skills/` | 38 | +297 | −133 | +164 | | 2. Graph-capture comparison | 6 | +58 | −30 | +28 | | 3. Defer roofline to Core | 1 | +6 | −3 | +3 | | 4. Full repeatability pipeline | 5 | +272 | −92 | +180 | | **Grand Total** | **52** | **+630** | **−272** | **+358** | --------- Co-authored-by: Kyle Hoffmeyer <97640268+kyle-hoffmeyer@users.noreply.github.com> Co-authored-by: Claude Sonnet 4 <noreply@anthropic.com> Co-authored-by: root <root@ctr-s93-mi300x-21-vm6.ctr2.amd.com> Co-authored-by: Tharun Adithya Srikrishnan <tsrikrsi@amd.com> Co-authored-by: Gabriel Weisz <162640284+gabeweisz@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Ahmedhasssan-aig <Ahmed.Hasssan@amd.com> Co-authored-by: Hasssan <ahasssan@ctr2-alola-ctrl-01.amd.com> Co-authored-by: Gabe Weisz <gabe.weisz@amd.com>
This pull request refactors how inference trace annotation parsing is handled across the codebase by centralizing annotation parsing logic into the new `TraceLens.TraceUtils.annotation_utils` module. The changes remove scattered regular expressions and custom parsing code, replacing them with reusable utilities and classes. This improves maintainability, consistency, and robustness of annotation handling, especially for different annotation formats and edge cases. The most important changes are: **Centralization and Reuse of Annotation Parsing Logic:** * Introduced and adopted the `IterationAnnotation` class and related utilities from `TraceLens.TraceUtils.annotation_utils` throughout the codebase, replacing custom parsing and regular expression logic in multiple files (`attention_perf_model_extensions.py`, `match_inference_trace_blocks.py`, `generate_perf_report_pytorch_inference.py`, `trace_capture_merge_experimental.py`). [[1]](diffhunk://#diff-11bbfa67e2e4b8f4659996548738b078a104b114c0644e4d3935ffd46b61a345R12-L13) [[2]](diffhunk://#diff-11bbfa67e2e4b8f4659996548738b078a104b114c0644e4d3935ffd46b61a345L96-R97) [[3]](diffhunk://#diff-11bbfa67e2e4b8f4659996548738b078a104b114c0644e4d3935ffd46b61a345L834-R790) [[4]](diffhunk://#diff-65fef6c8c5821e4b6daa3b8c82338b8e7585fd7b1f77695330694539a0d9325eR34-R38) [[5]](diffhunk://#diff-a361820c09001fd17b2d09215e178075fa4e355e907c434103b7f0bd0a40041cL24-R31) [[6]](diffhunk://#diff-8bef1344f5b3f92d3dcbfc8be2ca155630ab9ba9a7052893474e413e7decb3e5L46-L72) [[7]](diffhunk://#diff-95ba412ca7f917219f63ddbadfdf884887586df9c59c050bdc263fd0d845891dR183-R194) **Refactoring and Simplification:** * Removed legacy annotation parsing code, regular expressions, and helper functions, replacing them with calls to the new utilities. This affects how chunk stats, parameter details, and iteration details are extracted and used. [[1]](diffhunk://#diff-8bef1344f5b3f92d3dcbfc8be2ca155630ab9ba9a7052893474e413e7decb3e5L83-L182) [[2]](diffhunk://#diff-8bef1344f5b3f92d3dcbfc8be2ca155630ab9ba9a7052893474e413e7decb3e5L276-R170) [[3]](diffhunk://#diff-8bef1344f5b3f92d3dcbfc8be2ca155630ab9ba9a7052893474e413e7decb3e5L597-R495) [[4]](diffhunk://#diff-65fef6c8c5821e4b6daa3b8c82338b8e7585fd7b1f77695330694539a0d9325eL221-L236) [[5]](diffhunk://#diff-65fef6c8c5821e4b6daa3b8c82338b8e7585fd7b1f77695330694539a0d9325eL200) [[6]](diffhunk://#diff-a361820c09001fd17b2d09215e178075fa4e355e907c434103b7f0bd0a40041cL531-R531) [[7]](diffhunk://#diff-a361820c09001fd17b2d09215e178075fa4e355e907c434103b7f0bd0a40041cL644-R648) **Improved Robustness and Edge Case Handling:** * Enhanced error handling and fallback logic when batch size or annotation details cannot be determined, ensuring the pipeline skips problematic cases gracefully and emits warnings instead of crashing. [[1]](diffhunk://#diff-a361820c09001fd17b2d09215e178075fa4e355e907c434103b7f0bd0a40041cL644-R648) [[2]](diffhunk://#diff-a361820c09001fd17b2d09215e178075fa4e355e907c434103b7f0bd0a40041cR697-R703) **Expanded Phase Handling:** * Added support for the `prefill_only` phase in block discovery and validation, in addition to existing phases, improving flexibility for different inference workloads. [[1]](diffhunk://#diff-8bef1344f5b3f92d3dcbfc8be2ca155630ab9ba9a7052893474e413e7decb3e5L290-R186) [[2]](diffhunk://#diff-8bef1344f5b3f92d3dcbfc8be2ca155630ab9ba9a7052893474e413e7decb3e5L340-R236) [[3]](diffhunk://#diff-8bef1344f5b3f92d3dcbfc8be2ca155630ab9ba9a7052893474e413e7decb3e5L653-R543) **Testing Enhancements:** * Added new regression test steps for annotation parsing and trace splitting to the GitHub Actions workflow (`.github/workflows/regression-tests.yml`), ensuring these utilities are covered by automated tests. <!-- Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. --> --------- Co-authored-by: Deval Shah <devashah@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Syncing up with the main branch