Fix empty insight check in agent evals - #847
Open
kyle-hoffmeyer wants to merge 3 commits into
Open
Conversation
## Summary: This PR introduces support for comparing pairs of graph capture traces via the TraceLens Agent. ## TraceLens changes Updates to `generate_perf_report_pytorch_inference.py` to support passing two graph capture traces: #707 Updates to `trace_capture_merge_experimental.py` to support passing two graph capture traces as well as general improvements: #707, #798 Enhancements to `tracediff.py` to handle call stack divergences that occur when comparing two graph capture traces as well as traces that differ across framework versions: #768, #820, #800 ## Agent changes - Allows two graph replay traces and two capture trace directories to specified in prompt - Appropriately calls `generate_perf_report_pytorch_inference.py` ## Eval changes - Modified schema in `combined_traces_comparative.csv` to support capture trace directory passed into prompt. - Added pair of GPTOSS graph replay traces along with capture traces (vllm v0.19 vs vllm v0.21) into evals. Updated `e2e_tests_comparative.tar.gz`. - Updated `generate_ref.sh` and `run_repeatability_parallel.sh` to be able to pass capture trace directory into prompt --------- Co-authored-by: Claude Sonnet 4 <noreply@anthropic.com>
## Summary The Analysis Agent re-derived each operation's compute/memory bound classification from `FLOPS/Byte` and a recomputed `balance_point`, duplicating the roofline decision the core perf pipeline already makes and writes to the `Roofline Bound` CSV column. This change makes `calculate_efficiency()` read that authoritative column directly, giving the classification a single owner. Net diff: 1 file, +6/−3. ## Why The agent's `balance_point = peak_maf / peak_hbm_bw` test is algebraically identical to the core's `compute_time >= memory_time` comparison in `TreePerf/tree_perf.py`, so it was a second copy of the same ridge-point math — with its own peak-resolution path that could silently drift from the core. The core already emits the result as `Roofline Bound` (`COMPUTE_BOUND`/`MEMORY_BOUND`) into every per-category CSV the agent consumes, so the duplication was pure redundancy. Reading the column also fixes an edge case where the agent classified ops against a *fallback* peak that the core deliberately left unclassified (no resolved compute spec). ## What changed ### Python (1 file) | File | Change | |---|---| | `TraceLens/Agent/Analysis/category_analyses/analysis_utils.py` | In `calculate_efficiency()`, replace the `FLOPS/Byte` vs `balance_point` derivation of `bound_type` with a direct read of the `Roofline Bound` column (`COMPUTE_BOUND`→`compute`, `MEMORY_BOUND`→`memory`, empty→`None`). | The `resolved_peak_maf` / `resolved_peak_hbm_bw` fields and `_resolve_peak_maf()` are retained — they still feed the "X% of Y TFLOPS/TB/s" rendering in `utils/templates/sub_agent_spec.md`. The function signature is unchanged, so no callers are affected. The `efficiency.bound_type` field keeps its name, `compute`/`memory` values, and JSON location, so downstream consumers are unaffected. ## Test plan - [x] `pytest tests/test_analysis_agent.py` — 35 passed (ROCm dev container; black 26.1.0 clean on the changed file). - [x] Equivalence check: agent `bound_type` vs core `Roofline Bound` on a default-mode training trace — 233/233 modelled ops matched, zero compute↔memory disagreements. - [x] End-to-end analysis runs on two traces (one default-mode, one FP4 inference-mode): freshly generated `bound_type` equals the CSV column positionally — 449/449 and 39/39, zero mismatches. Report headers, P-item sets, template markers, and headline metrics all match the prior baseline (only display-precision INFO deltas). ### Rollback Revert the single hunk in `calculate_efficiency()` to restore the `balance_point` derivation; no data-contract or signature changes to unwind. ## Net diff ``` 1 file changed, 6 insertions(+), 3 deletions(-) ``` Co-authored-by: root <root@ctr-s93-mi300x-21-vm6.ctr2.amd.com>
kyle-hoffmeyer
requested review from
Ahmedhasssan-aig and
tsrikris
as code owners
July 23, 2026 20:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
marker_eval_2andmarker_eval_3should pass if report clearly states that there are no compute optimization opportunities. If it doesn't state this and the compute optimization section is blank, it should fail.