Fix empty insight check in agent evals

TraceLens/Agent/Analysis/.cursor/skills/analysis-orchestrator.md

@@ -93,9 +93,9 @@ Use vendor-agnostic terminology throughout such as GPU kernels, collective commu
    - If **Inference (vLLM/SGLang)** is selected, ask **Execution Mode** → `<inference_exec_mode>`:
      1. **Eager mode** (`<inference_exec_mode>` = `eager`) — only the trace file is needed
      2. **Graph replay + capture** (`<inference_exec_mode>` = `graph_capture`) — also requires a capture folder path
-   - If **Graph replay + capture**, ask for **Capture Folder Path** → `<capture_folder_path>`:
+   - If **Graph replay + capture**, ask for **Capture Folder Path** → `<capture_folder_path_1>`:
      - Ask: "Please provide the full path to the graph capture traces folder"
-   - **Unsupported combination:** If `<inference_exec_mode>` = `graph_capture` **and** `<comparison_scope>` = `comparative`, stop immediately. Inform the user: "Graph replay + capture mode is not yet supported for comparative analysis. Please provide eager mode traces instead." Do not misinterpret as two standalone analyses. Do **not** proceed to Step 1 or beyond.
+   - If **Graph replay + capture** and **comparative**, ask for **Trace2 Capture Folder Path** → `<capture_folder_path_2>`
 
 5. **Environment Setup**
    - Ask: "Are you running locally or on a cluster?"
@@ -178,14 +178,23 @@ All commands below append `<suffix_1>` and `<suffix_2>`, resolved by `<compariso
 | `comparative` trace1 | `--output_xlsx_path <output_dir>/perf_report_trace1.xlsx --output_csvs_dir <output_dir>/perf_report_trace1_csvs` |
 | `comparative` trace2 | `--profile_json_path <trace2_path> --output_xlsx_path <output_dir>/perf_report_trace2.xlsx --output_csvs_dir <output_dir>/perf_report_trace2_csvs` |
 
-**`<suffix_2>`** — extension flags:
+**`<suffix_2>`** — comparison flags:
 
 | scope | value |
 |-------|-------|
 | `standalone` | none |
 | `comparative` trace1 | `--comparison_json_path <trace2_path>` |
+| `comparative` trace1 if `<capture_folder_path_2>` provided | `--comparison_json_path <trace2_path> --comparison_capture_folder <capture_folder_path_2>` |
 | `comparative` trace2 | none |
 
+**`<suffix_3>`** — graph capture flags:
+
+| scope | value |
+|-------|-------|
+| `standalone` if `<capture_folder_path_1>` provided | `--capture_folder <capture_folder_path_1>` |
+| `comparative` trace1 if `<capture_folder_path_1>` provided | `--capture_folder <capture_folder_path_1>` |
+| `comparative` trace2 if `<capture_folder_path_2>` provided | `--capture_folder <capture_folder_path_2>` |
+
 **`<suffix_ext>`** — user extension file:
 
 | condition | value |
@@ -229,14 +238,14 @@ All commands below append `<suffix_1>` and `<suffix_2>`, resolved by `<compariso
 ```bash
 <prefix> TraceLens_generate_perf_report_pytorch_inference \
   --profile_json_path <trace_path> \
-  --capture_folder <capture_folder_path> \
   --gpu_arch_json_path <platform_file> \
   --group_by_parent_module \
   --enable_pseudo_ops \
   --group_by_num_kernels \
   --include_call_stack \
   <suffix_1> \
   <suffix_2> \
+  <suffix_3> \
   <suffix_ext>
 ```
 

TraceLens/Agent/Analysis/README.md

@@ -19,7 +19,7 @@ The TraceLens Agent is an agentic performance analysis tool that generates actio
 | | Standalone | Comparative |
 |---|---|---|
 | **Eager** | ✅ | ✅ |
-| **Graph + Capture** | ✅ | ❌ |
+| **Graph + Capture** | ✅ | ✅ |
 | **Graph** | ❌ | ❌ |
 
 ---
@@ -76,14 +76,22 @@ Roofline analysis compares each measured kernel against your GPU's max-achievabl
 ### To run via Cursor chat:
 
 1. **In a Cursor chat with Claude Opus 4.7 High, invoke one of:**
-   - Standalone (single trace):
+   - Standalone (single eager trace):
     ```
     "Follow the analysis orchestrator installed with TraceLens and run the full agentic analysis workflow on <path_to_trace.json>"
     ```
-   - Comparative (two traces):
+   - Standalone (single graph replay trace with capture trace directory):
+    ```
+    "Follow the analysis orchestrator installed with TraceLens and run the full agentic analysis workflow on <path_to_trace.json> with capture folder <path_to_capture.json>"
+    ```
+   - Comparative (two eager traces):
     ```
     "Follow the analysis orchestrator installed with TraceLens and run the full agentic analysis workflow on <path_to_trace1.json> and <path_to_trace2.json>"
     ```
+   - Comparative (two graph replay traces with capture trace directories):
+    ```
+    "Follow the analysis orchestrator installed with TraceLens and run the full agentic analysis workflow on <path_to_trace1.json> with capture folder <path_to_capture1.json> and <path_to_trace2.json> with capture folder <path_to_capture2.json>"
+    ```
     **NOTE**: Always pass **baseline** trace as trace1
 
 
@@ -320,7 +328,7 @@ It queries user inputs, runs TraceLens to pre-compute trace data, and invokes sy
 
 The orchestrator and all 13 sub-agents currently run on **`claude-opus-4-7-high`**, declared in each agent file's front matter under `.cursor/agents/`. The full set: `cpu-idle-analyzer`, `multi-kernel-analyzer`, `kernel-fusion-analyzer`, `model-identification-agent`, `gemm-analyzer`, `sdpa-analyzer`, `elementwise-analyzer`, `reduce-analyzer`, `triton-analyzer`, `moe-analyzer`, `norm-analyzer`, `convolution-analyzer`, `generic-op-analyzer`.
 
-### Supported Standalone Analysis Modes
+### Supported Analysis Modes
 
 The orchestrator supports two analysis modes, selected during Step 0:
 

TraceLens/Agent/Analysis/category_analyses/analysis_utils.py

@@ -408,9 +408,12 @@ def calculate_efficiency(
     result["resolved_peak_maf"] = round(peak_maf, 2) if peak_maf else None
     result["resolved_peak_hbm_bw"] = round(peak_hbm_bw, 2) if peak_hbm_bw else None
 
-    if flops_byte:
-        balance_point = peak_maf / peak_hbm_bw
-        result["bound_type"] = "compute" if flops_byte > balance_point else "memory"
+    roofline_bound = row.get("Roofline Bound")
+    if isinstance(roofline_bound, str):
+        if roofline_bound == "COMPUTE_BOUND":
+            result["bound_type"] = "compute"
+        elif roofline_bound == "MEMORY_BOUND":
+            result["bound_type"] = "memory"
 
     if comparison_scope == "comparative":
         comparative_efficiency(result, row)

agent_evals/Analysis/analysis_tests/combined_traces_comparative.csv

@@ -1,10 +1,11 @@
-id,sub_category,trace1_path,trace2_path,reference_dir,platform,platform2
-test01_moe_fused_vs_unfused,moe,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test01_moe_fused_vs_unfused/trace1.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test01_moe_fused_vs_unfused/trace2.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test01_moe_fused_vs_unfused/analysis_output_ref,MI300X,H100
-test02_attention_both_unfused,attention,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test02_attention_both_unfused/trace1.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test02_attention_both_unfused/trace2.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test02_attention_both_unfused/analysis_output_ref,MI300X,H100
-test03_attention_fused_vs_unfused,attention,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test03_attention_fused_vs_unfused/trace1.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test03_attention_fused_vs_unfused/trace2.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test03_attention_fused_vs_unfused/analysis_output_ref,MI300X,H100
-test04_gemm_mm_vs_linear,gemm,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test04_gemm_mm_vs_linear/trace1.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test04_gemm_mm_vs_linear/trace2.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test04_gemm_mm_vs_linear/analysis_output_ref,MI300X,H100
-test05_gemm_identical_runtime_roofline,gemm,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test05_gemm_identical_runtime_roofline/trace1.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test05_gemm_identical_runtime_roofline/trace2.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test05_gemm_identical_runtime_roofline/analysis_output_ref,MI300X,H100
-test06_conv_gap_exceeds_roofline,conv,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test06_conv_gap_exceeds_roofline/trace1.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test06_conv_gap_exceeds_roofline/trace2.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test06_conv_gap_exceeds_roofline/analysis_output_ref,MI300X,H100
-bert_small,full_model,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/bert_small/gaunernst_bert-small-uncased__1016001_mi300.json.gz,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/bert_small/gaunernst_bert-small-uncased__1016001_h100.json.gz,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/bert_small/analysis_output_ref,MI300X,H100
-llama3_2,full_model,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/llama3_2/llama3_2_mi300.pt.trace_mi300.json.gz,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/llama3_2/llama3_2_h100.pt.trace_h100.json.gz,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/llama3_2/analysis_output_ref,MI300X,H100
-timesformer,full_model,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/timesformer/facebook_timesformer-base-finetuned-k400__1016002_mi300.json.gz,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/timesformer/facebook_timesformer-base-finetuned-k400__1016002_h100.json.gz,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/timesformer/analysis_output_ref,MI300X,H100
+id,sub_category,trace1_path,trace2_path,reference_dir,platform,platform2,capture_folder1,capture_folder2
+test01_moe_fused_vs_unfused,moe,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test01_moe_fused_vs_unfused/trace1.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test01_moe_fused_vs_unfused/trace2.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test01_moe_fused_vs_unfused/analysis_output_ref,MI300X,H100,,
+test02_attention_both_unfused,attention,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test02_attention_both_unfused/trace1.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test02_attention_both_unfused/trace2.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test02_attention_both_unfused/analysis_output_ref,MI300X,H100,,
+test03_attention_fused_vs_unfused,attention,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test03_attention_fused_vs_unfused/trace1.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test03_attention_fused_vs_unfused/trace2.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test03_attention_fused_vs_unfused/analysis_output_ref,MI300X,H100,,
+test04_gemm_mm_vs_linear,gemm,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test04_gemm_mm_vs_linear/trace1.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test04_gemm_mm_vs_linear/trace2.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test04_gemm_mm_vs_linear/analysis_output_ref,MI300X,H100,,
+test05_gemm_identical_runtime_roofline,gemm,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test05_gemm_identical_runtime_roofline/trace1.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test05_gemm_identical_runtime_roofline/trace2.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test05_gemm_identical_runtime_roofline/analysis_output_ref,MI300X,H100,,
+test06_conv_gap_exceeds_roofline,conv,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test06_conv_gap_exceeds_roofline/trace1.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test06_conv_gap_exceeds_roofline/trace2.json,agent_evals/Analysis/analysis_tests/unit_tests_comparative/test06_conv_gap_exceeds_roofline/analysis_output_ref,MI300X,H100,,
+bert_small,full_model,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/bert_small/gaunernst_bert-small-uncased__1016001_mi300.json.gz,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/bert_small/gaunernst_bert-small-uncased__1016001_h100.json.gz,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/bert_small/analysis_output_ref,MI300X,H100,,
+llama3_2,full_model,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/llama3_2/llama3_2_mi300.pt.trace_mi300.json.gz,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/llama3_2/llama3_2_h100.pt.trace_h100.json.gz,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/llama3_2/analysis_output_ref,MI300X,H100,,
+timesformer,full_model,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/timesformer/facebook_timesformer-base-finetuned-k400__1016002_mi300.json.gz,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/timesformer/facebook_timesformer-base-finetuned-k400__1016002_h100.json.gz,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/timesformer/analysis_output_ref,MI300X,H100,,
+gptoss_graph_capture,full_model,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/gptoss_graph_capture/vllm_v21_mi300.json.gz,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/gptoss_graph_capture/vllm_v19_mi300.json.gz,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/gptoss_graph_capture/analysis_output_ref,MI300X,MI300X,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/gptoss_graph_capture/capture_folder_v21,agent_evals/Analysis/analysis_tests/e2e_tests_comparative/gptoss_graph_capture/capture_folder_v19

agent_evals/Analysis/eval_scripts/generate_ref.sh

@@ -69,7 +69,7 @@ log_status() {
 # ---------------------------------------------------------------------------
 
 generate_single_ref() {
-    local id="$1" trace1_path="$2" trace2_path="$3" reference_dir="$4" platform="$5" platform2="$6"
+    local id="$1" trace1_path="$2" trace2_path="$3" reference_dir="$4" platform="$5" platform2="$6" capture_folder1="${7:-}" capture_folder2="${8:-}"
     local tag="[$id]"
 
     local REF_DIR="$REPO_ROOT/$reference_dir"
@@ -100,8 +100,13 @@ generate_single_ref() {
         (
             cd "$ANALYSIS_DIR" || exit
             if [[ "$COMPARISON_SCOPE" == "comparative" ]]; then
+                local capture_suffix=""
+                [[ -n "$capture_folder1" ]] && capture_suffix+=" capture folder for trace1 $REPO_ROOT/$capture_folder1"
+                [[ -n "$capture_folder2" ]] && capture_suffix+=" capture folder for trace2 $REPO_ROOT/$capture_folder2"
+                local analysis_mode="default"
+                [[ -n "$capture_folder1" || -n "$capture_folder2" ]] && analysis_mode="inference"
                 agent --model claude-opus-4-8-thinking-medium --print --force --trust --output-format stream-json \
-                    "Follow the analysis orchestrator installed with the TraceLens pip package (look under TraceLens/Agent/Analysis/.cursor/skills/ in the package installation directory) and run the full agentic analysis workflow on $trace1_path and $trace2_path with platform $platform (trace1) and $platform2 (trace2), analysis mode default, $NODE_LABEL, $RUNTIME_LABEL, output to $OUTPUT_DIR"
+                    "Follow the analysis orchestrator installed with the TraceLens pip package (look under TraceLens/Agent/Analysis/.cursor/skills/ in the package installation directory) and run the full agentic analysis workflow on $trace1_path and $trace2_path${capture_suffix} with platform $platform (trace1) and $platform2 (trace2), analysis mode $analysis_mode, $NODE_LABEL, $RUNTIME_LABEL, output to $OUTPUT_DIR"
             else
                 agent --model claude-opus-4-8-thinking-medium --print --force --trust --output-format stream-json \
                     "Follow the analysis orchestrator installed with the TraceLens pip package (look under TraceLens/Agent/Analysis/.cursor/skills/ in the package installation directory) and run the full agentic analysis workflow on $trace1_path with platform $platform, analysis mode default, $NODE_LABEL, $RUNTIME_LABEL, output to $OUTPUT_DIR"
@@ -202,14 +207,14 @@ should_run_id() {
 setup_semaphore
 
 if [[ "$COMPARISON_SCOPE" == "comparative" ]]; then
-    # comparative CSV: id,sub_category,trace1_path,trace2_path,reference_dir,platform,platform2
-    while IFS=, read -r id sub_category trace1_path trace2_path reference_dir platform platform2 <&3; do
+    # comparative CSV: id,sub_category,trace1_path,trace2_path,reference_dir,platform,platform2,capture_folder1,capture_folder2
+    while IFS=, read -r id sub_category trace1_path trace2_path reference_dir platform platform2 capture_folder1 capture_folder2 <&3; do
         [[ -z "$id" ]] && continue
         should_run_id "$id" || continue
 
         read -u4  # acquire semaphore slot
         (
-            generate_single_ref "$id" "$trace1_path" "$trace2_path" "$reference_dir" "$platform" "$platform2" || true
+            generate_single_ref "$id" "$trace1_path" "$trace2_path" "$reference_dir" "$platform" "$platform2" "${capture_folder1:-}" "${capture_folder2:-}" || true
             sleep "$SLEEP_BETWEEN"
             echo >&4  # release semaphore slot
         ) &