[Perf Regression] 40 config(s) regressed @ 945b3127

[github-actions]

Performance Regression Detected

Commit: 945b3127
Run: https://github.com/ROCm/ATOM/actions/runs/29159908631
Date: 2026-07-12T04:06:19.016554+00:00

Regressed Configurations

Model	ISL/OSL	Conc	Tput (cur)	Tput (base)	Δ%	TPOT (cur)	TPOT (base)	Δ%
DeepSeek-V4-Pro	1024/1024	16	751.2	792.3	-5.2%	19.39	19.45	-0.3%
DeepSeek-V4-Pro DPA	1024/1024	512	8274.7	8502.3	-2.7%	57.54	56.69	1.5%
DeepSeek-V4-Pro DPA	8192/1024	1024	5004.8	4499.9	11.2%	133.79	210.42	-36.4%
DeepSeek-V4-Pro DPA TBO	8192/1024	512	3812.4	3752.2	1.6%	125.38	128.67	-2.6%
DeepSeek-V4-Pro MTP3	1024/1024	4	429.0	404.4	6.1%	8.54	8.98	-4.9%
DeepSeek-V4-Pro MTP3	1024/1024	8	705.1	749.7	-5.9%	10.59	10.18	4.0%
GLM-5.2-FP8	1024/1024	32	1312.2	1311.5	0.1%	23.37	23.50	-0.6%
Kimi-K2.7-Code-MXFP4	1024/1024	16	1222.1	1189.0	2.8%	12.70	13.08	-2.9%
Kimi-K2.7-Code-MXFP4	1024/1024	64	2581.1	2544.0	1.5%	23.95	24.36	-1.7%
Llama-3.3-70B-Instruct-MXFP4	1024/1024	32	1790.7	1786.2	0.3%	17.11	17.18	-0.4%
MiniMax-M3-MXFP4	1024/1024	8	1027.7	1035.1	-0.7%	7.45	7.44	0.1%
MiniMax-M3-MXFP4	1024/1024	16	1675.7	1708.0	-1.9%	9.21	9.07	1.5%
MiniMax-M3-MXFP4	1024/1024	32	2623.1	2561.7	2.4%	11.63	12.03	-3.3%
MiniMax-M3-MXFP4	8192/1024	4	555.2	558.4	-0.6%	6.77	6.75	0.2%
MiniMax-M3-MXFP4 EAGLE3	1024/1024	8	1712.1	1740.2	-1.6%	4.32	4.35	-0.6%
MiniMax-M3-MXFP4 EAGLE3	1024/1024	32	3853.1	3829.2	0.6%	7.76	7.90	-1.7%
MiniMax-M3-MXFP4 EAGLE3	1024/1024	64	5337.0	5417.1	-1.5%	11.23	11.13	0.9%
MiniMax-M3-MXFP8	1024/1024	4	525.4	525.3	0.0%	7.27	7.29	-0.4%
MiniMax-M3-MXFP8	1024/1024	128	4489.7	4515.1	-0.6%	27.52	27.40	0.4%
MiniMax-M3-MXFP8	1024/1024	256	6635.4	6556.4	1.2%	37.27	37.80	-1.4%
MiniMax-M3-MXFP8	8192/1024	16	1115.6	1117.0	-0.1%	13.57	13.60	-0.3%
MiniMax-M3-MXFP8 EAGLE3	1024/1024	8	1328.0	1273.3	4.3%	5.76	5.96	-3.4%
MiniMax-M3-MXFP8 EAGLE3	1024/1024	16	1989.9	2065.2	-3.6%	7.66	7.39	3.7%
MiniMax-M3-MXFP8 EAGLE3	8192/1024	4	694.1	676.2	2.7%	5.20	5.29	-1.8%
Qwen3.5-397B-A17B-FP8	1024/1024	8	736.0	816.4	-9.8%	10.52	9.45	11.4%
Qwen3.5-397B-A17B-FP8	1024/1024	64	2699.7	2815.2	-4.1%	22.89	21.97	4.2%
Qwen3.5-397B-A17B-FP8	1024/1024	256	5827.8	5719.9	1.9%	42.43	43.32	-2.0%
Qwen3.5-397B-A17B-FP8	8192/1024	32	1603.6	1608.6	-0.3%	18.80	18.80	-0.0%
Qwen3.5-397B-A17B-FP8	8192/1024	128	2619.4	2603.4	0.6%	46.32	46.79	-1.0%
Qwen3.5-397B-A17B-FP8 MTP3	8192/1024	4	763.7	832.1	-8.2%	4.84	4.38	10.3%
Qwen3.5-397B-A17B-FP8 MTP3	8192/1024	16	1736.7	1770.9	-1.9%	8.28	8.18	1.2%
Qwen3.5-397B-A17B-FP8 MTP3	8192/1024	32	2130.3	2197.2	-3.0%	13.82	13.53	2.1%
Qwen3.5-397B-A17B-MXFP4	1024/1024	4	412.6	403.1	2.4%	9.21	9.49	-3.0%
Qwen3.5-397B-A17B-MXFP4	1024/1024	64	2947.1	2959.2	-0.4%	20.88	20.83	0.2%
Qwen3.5-397B-A17B-MXFP4	1024/1024	256	5723.2	6372.5	-10.2%	43.28	38.72	11.8%
Qwen3.5-397B-A17B-MXFP4	8192/1024	128	2890.6	2875.4	0.5%	41.79	42.21	-1.0%
gpt-oss-120b	1024/1024	4	916.5	884.1	3.7%	4.16	4.34	-4.2%
gpt-oss-120b	1024/1024	256	12435.4	12690.9	-2.0%	19.42	19.26	0.8%
gpt-oss-120b	8192/1024	16	2389.3	2407.5	-0.8%	6.23	6.23	0.1%
gpt-oss-120b	8192/1024	64	4755.1	4769.8	-0.3%	12.75	12.77	-0.2%

Performance Summary

# Trace Performance Summary

**File:** `Qwen3.5-397B-A17B-FP8_ts_20260712_041608_932.pt.trace.json.gz`

## Prefill

| # | Label | Duration |
|---|-------|----------|
| 0 | `prefill[bs=2 tok=1927 ctx=[936, 991]]` | 96.93 ms |
| 1 | `prefill[bs=6 tok=5527 ctx=[1011, 886, 866]...+3]` | 91.90 ms |
| 2 | `prefill[bs=1 tok=840 ctx=840]` | 132.14 ms |
| 3 | `prefill[bs=1 tok=855 ctx=855]` | 92.58 ms |
| 4 | `prefill[bs=1 tok=906 ctx=906]` | 93.61 ms |
| 5 | `prefill[bs=1 tok=889 ctx=889]` | 93.32 ms |
| 6 | `prefill[bs=1 tok=907 ctx=907]` | 93.10 ms |
| 7 | `prefill[bs=1 tok=959 ctx=959]` | 93.31 ms |
| 8 | `prefill[bs=1 tok=877 ctx=877]` | 94.28 ms |
| 9 | `prefill[bs=1 tok=1012 ctx=1012]` | 95.00 ms |

**Total prefill:** 976.17 ms

## Decode

- **Iterations:** 2006
- **Mean:** 936.3 us
- **Min:** 394.1 us
- **Max:** 2.10 ms
- **Total:** 1878.22 ms

Profiler Traces

Download from workflow artifacts.
Open in Perfetto UI or Chrome chrome://tracing for analysis.

Next Steps

1. Download profiler-analysis-29159908631 artifact
2. Open trace files in Perfetto UI
3. Compare kernel durations against previous traces
4. Identify bottleneck changes