Remove alibi, score_max/sum_exp, and dropout randomization from test_… - #435
Remove alibi, score_max/sum_exp, and dropout randomization from test_…#435vedaanta wants to merge 3 commits into
Conversation
…mhas_v2 Drop is_alibi, with_score_max, with_score_sum_exp, and is_dropout from the randomized SDPA test configurations. The plumbing in the sdpa test package is kept; only the test file stops exercising these options. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughSDPA randomized tests no longer vary selected score, ALiBi, and dropout options. A new deterministic test module covers ALiBi masking, score outputs, and dropout across forward and backward execution. ChangesSDPA test configuration
Estimated code review effort: 3 (Moderate) | ~20 minutes Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
@cudnn-ci-bot run |
|
🚀 Running mirror pipeline Branch: cudnn-gh/pr-435-891f9f7 |
Anerudhan
left a comment
Anerudhan
left a comment
There was a problem hiding this comment.
I am hesitant to remove this. New engines can return not supported. As these are tested into upstream integrations.
|
@Anerudhan Agreed that we have users for these, but we do already run TE and PyT tests directly. Given that we want to focus on important fusions, current testing surface only clobber stats for all the new kernels we are bringing up. I will add a separate individual test for all these fusions. Separate from test_mhas. |
Cover ALiBi, score_max/score_sum_exp, and dropout with a few deterministic fwd/bwd configs in test_sdpa_unpopular_features.py, now that test_mhas_v2.py no longer randomizes over these options. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Added |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/python/test_sdpa_unpopular_features.py`:
- Around line 27-40: Update the ExecConfig construction in
test_sdpa_unpopular_features.py so cfg.is_determin is set to True, ensuring the
backward/dropout cases exercise the deterministic algorithm and bitwise rerun
check.
- Around line 61-86: Add the established cuDNN backend-version and GPU
compute-capability guards to test_sdpa_alibi, test_sdpa_score_max_sum_exp, and
test_sdpa_dropout before exec_sdpa. Reuse the same checks and skip behavior used
elsewhere in the test module, including dtype-specific support conditions, so
unsupported BF16/FP16 combinations are skipped while supported cases retain
their current coverage.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 0225b992-32ad-4534-8fe3-61a9973f4cb5
📒 Files selected for processing (1)
test/python/test_sdpa_unpopular_features.py
| cfg = ExecConfig( | ||
| data_type=data_type, | ||
| rng_data_seed=1234, | ||
| rng_geom_seed=5678, | ||
| is_alibi=is_alibi, | ||
| is_infer=is_infer, | ||
| is_paged=False, | ||
| is_bias=False, | ||
| is_block_mask=False, | ||
| is_padding=False, | ||
| is_cu_seq_len=False, | ||
| is_ragged=False, | ||
| is_dropout=is_dropout, | ||
| is_determin=False, |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== File outline ==\n'
ast-grep outline test/python/test_sdpa_unpopular_features.py --view expanded || true
printf '\n== Relevant symbols ==\n'
rg -n "class ExecConfig|def exec_sdpa|is_determin|determin" -S test python . || trueRepository: NVIDIA/cudnn-frontend
Length of output: 50377
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
p = Path("test/python/test_sdpa_unpopular_features.py")
print("lines", sum(1 for _ in p.open()))
for i, line in enumerate(p.open(), 1):
if 1 <= i <= 220:
if "ExecConfig" in line or "is_determin" in line or "exec_sdpa" in line or "L0" in line:
print(f"{i}: {line.rstrip()}")
PYRepository: NVIDIA/cudnn-frontend
Length of output: 505
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Read the relevant source slices around the referenced lines and the ExecConfig/exec_sdpa definitions.
for f in test/python/test_sdpa_unpopular_features.py $(git ls-files | rg '(^|/)(exec_sdpa|ExecConfig).*\.py$' || true); do
[ -f "$f" ] || continue
echo "===== $f ====="
nl -ba "$f" | sed -n '1,240p'
doneRepository: NVIDIA/cudnn-frontend
Length of output: 255
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
paths = [Path("test/python/test_sdpa_unpopular_features.py")]
# discover the definition files referenced by rg
import subprocess, shlex
out = subprocess.check_output(rg := ["bash", "-lc", "rg -n --glob '*.py' 'class ExecConfig|def exec_sdpa|is_determin' test ."], text=True)
print(out)
PYRepository: NVIDIA/cudnn-frontend
Length of output: 6341
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n "class ExecConfig|def exec_sdpa|is_determin" test . || trueRepository: NVIDIA/cudnn-frontend
Length of output: 12028
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Searching...\n'
rg -n "is_determin|ExecConfig|exec_sdpa" test/python test || trueRepository: NVIDIA/cudnn-frontend
Length of output: 12264
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
def show(path, start, end):
print(f"===== {path} {start}-{end} =====")
lines = Path(path).read_text().splitlines()
for i in range(start, min(end, len(lines)) + 1):
print(f"{i:4d}: {lines[i-1]}")
show("test/python/test_sdpa_unpopular_features.py", 1, 120)
show("test/python/sdpa/fp16.py", 450, 620)
show("test/python/sdpa/random_config.py", 67, 110)
PYRepository: NVIDIA/cudnn-frontend
Length of output: 15272
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
for path, start, end in [
("test/python/test_sdpa_edge_cases.py", 1, 80),
("test/python/test_sdpa_unpopular_features.py", 1, 95),
]:
print(f"===== {path} {start}-{end} =====")
lines = Path(path).read_text().splitlines()
for i in range(start, min(end, len(lines)) + 1):
print(f"{i:4d}: {lines[i-1]}")
PYRepository: NVIDIA/cudnn-frontend
Length of output: 5717
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
path = Path("test/python/sdpa/fp16.py")
lines = path.read_text().splitlines()
for start, end in [(470, 590), (840, 885)]:
print(f"===== {path} {start}-{end} =====")
for i in range(start, min(end, len(lines)) + 1):
print(f"{i:4d}: {lines[i-1]}")
PYRepository: NVIDIA/cudnn-frontend
Length of output: 9171
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
p = Path("test/python/test_sdpa_unpopular_features.py")
for i, line in enumerate(p.read_text().splitlines(), 1):
if i <= 20 or 24 <= i <= 45:
print(f"{i:4d}: {line}")
PYRepository: NVIDIA/cudnn-frontend
Length of output: 1490
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
for path in ["test/python/test_sdpa_edge_cases.py"]:
print(f"===== {path} =====")
for i, line in enumerate(Path(path).read_text().splitlines(), 1):
if i <= 55:
print(f"{i:4d}: {line}")
PYRepository: NVIDIA/cudnn-frontend
Length of output: 1824
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
p = Path("test/python/sdpa/fp16.py")
for i, line in enumerate(p.read_text().splitlines(), 1):
if 475 <= i <= 590:
print(f"{i:4d}: {line}")
PYRepository: NVIDIA/cudnn-frontend
Length of output: 7089
Make these cases deterministic cfg.is_determin drives both use_deterministic_algorithm and the rerun bitwise check in test/python/sdpa/fp16.py, so leaving it False means these backward/dropout cases never exercise the deterministic path described here. Set it to True.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@test/python/test_sdpa_unpopular_features.py` around lines 27 - 40, Update the
ExecConfig construction in test_sdpa_unpopular_features.py so cfg.is_determin is
set to True, ensuring the backward/dropout cases exercise the deterministic
algorithm and bitwise rerun check.
| @pytest.mark.L0 | ||
| @pytest.mark.parametrize("is_infer", [True, False], ids=["fwd", "bwd"]) | ||
| def test_sdpa_alibi(is_infer, request, cudnn_handle): | ||
| # ALiBi requires a causal mask (right_bound=0, top-left alignment). | ||
| cfg = make_config(is_infer=is_infer, is_alibi=True, right_bound=0) | ||
| exec_sdpa(cfg, request, cudnn_handle) | ||
|
|
||
|
|
||
| @pytest.mark.L0 | ||
| @pytest.mark.parametrize("is_infer", [True, False], ids=["fwd", "bwd"]) | ||
| def test_sdpa_score_max_sum_exp(is_infer, request, cudnn_handle): | ||
| cfg = make_config(is_infer=is_infer, with_score_max=True, with_score_sum_exp=True) | ||
| exec_sdpa(cfg, request, cudnn_handle) | ||
|
|
||
|
|
||
| @pytest.mark.L0 | ||
| @pytest.mark.parametrize("is_infer", [True, False], ids=["fwd", "bwd"]) | ||
| def test_sdpa_dropout(is_infer, request, cudnn_handle): | ||
| cfg = make_config( | ||
| data_type=torch.float16, | ||
| is_infer=is_infer, | ||
| is_dropout=True, | ||
| dropout_prob=0.1, | ||
| right_bound=0, | ||
| ) | ||
| exec_sdpa(cfg, request, cudnn_handle) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== test file ==\n'
cat -n test/python/test_sdpa_unpopular_features.py | sed -n '1,220p'
printf '\n== search for support checks ==\n'
rg -n "support check|is_supported|backend_version|device_capability|skipif|pytest\.mark\.skip|cudnn\.backend_version|torch\.cuda\.get_device_capability|validate_config|exec_sdpa" test/python -S
printf '\n== nearby AGENTS ==\n'
find test -name AGENTS.md -print
if [ -f test/AGENTS.md ]; then
printf '\n== test/AGENTS.md ==\n'
cat -n test/AGENTS.md
fiRepository: NVIDIA/cudnn-frontend
Length of output: 29379
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== test/python/test_sdpa_unpopular_features.py =="
sed -n '1,220p' test/python/test_sdpa_unpopular_features.py | nl -ba
echo
echo "== matching helpers in test/python =="
rg -n "def (make_config|exec_sdpa|validate_config|is_supported|supports_|skip|backend_version|get_device_capability)" test/python -S
echo
echo "== AGENTS files under test =="
find test -name AGENTS.md -printRepository: NVIDIA/cudnn-frontend
Length of output: 249
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '== sdpa/fp16.py validate_config and exec_sdpa ==\n'
sed -n '1,180p' test/python/sdpa/fp16.py | cat -n
printf '\n== maybe related support helpers ==\n'
sed -n '180,260p' test/python/sdpa/fp16.py | cat -n
sed -n '840,940p' test/python/sdpa/fp16.py | cat -n
printf '\n== similar SDPA tests with explicit gating ==\n'
sed -n '150,220p' test/python/test_mhas_v2.py | cat -n
sed -n '680,860p' test/python/test_mhas_v2.py | cat -n
sed -n '1000,1045p' test/python/test_mhas_v2.py | cat -nRepository: NVIDIA/cudnn-frontend
Length of output: 33998
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
p = Path('test/python/sdpa/fp16.py')
text = p.read_text().splitlines()
for start, end, label in [
(1, 180, 'validate_config and top-of-file'),
(180, 260, 'mid helpers'),
(840, 940, 'exec_sdpa'),
]:
print(f'== {label} ==')
for i in range(start-1, min(end, len(text))):
print(f'{i+1:4d}: {text[i]}')
print()
PY
python3 - <<'PY'
from pathlib import Path
p = Path('test/python/test_mhas_v2.py')
text = p.read_text().splitlines()
for start, end, label in [
(150, 220, 'gated rope-related tests'),
(680, 860, 'fp8 tests with capability checks'),
(1000, 1045, 'repro driver'),
]:
print(f'== {label} ==')
for i in range(start-1, min(end, len(text))):
print(f'{i+1:4d}: {text[i]}')
print()
PYRepository: NVIDIA/cudnn-frontend
Length of output: 33440
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
def show(path, start, end, label):
text = Path(path).read_text().splitlines()
print(f'== {label} ==')
for i in range(start - 1, min(end, len(text))):
print(f'{i+1:4d}: {text[i]}')
print()
show('test/python/sdpa/fp16.py', 1, 180, 'validate_config and top-of-file')
show('test/python/sdpa/fp16.py', 840, 940, 'exec_sdpa')
show('test/python/test_mhas_v2.py', 150, 220, 'gated rope-related tests')
show('test/python/test_mhas_v2.py', 680, 860, 'fp8 tests with capability checks')
PYRepository: NVIDIA/cudnn-frontend
Length of output: 25788
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
for path, ranges in {
'test/python/sdpa/fp16.py': [(60, 140, 'validate_config'), (840, 900, 'exec_sdpa')],
'test/python/test_mhas_v2.py': [(150, 210, 'existing skip patterns')],
}.items():
text = Path(path).read_text().splitlines()
for start, end, label in ranges:
print(f'== {path} :: {label} ==')
for i in range(start - 1, min(end, len(text))):
print(f'{i+1:4d}: {text[i]}')
print()
PYRepository: NVIDIA/cudnn-frontend
Length of output: 10959
Gate these SDPA cases on backend and GPU capability. These tests always exercise BF16/FP16 paths, but exec_sdpa only applies a few cuDNN-version skips and doesn’t gate architecture or dtype support here. Add the same cudnn.backend_version() and torch.cuda.get_device_capability() checks used elsewhere before calling exec_sdpa, so unsupported combos skip instead of failing CI.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@test/python/test_sdpa_unpopular_features.py` around lines 61 - 86, Add the
established cuDNN backend-version and GPU compute-capability guards to
test_sdpa_alibi, test_sdpa_score_max_sum_exp, and test_sdpa_dropout before
exec_sdpa. Reuse the same checks and skip behavior used elsewhere in the test
module, including dtype-specific support conditions, so unsupported BF16/FP16
combinations are skipped while supported cases retain their current coverage.
Source: Coding guidelines
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
3 participants
…mhas_v2
Drop is_alibi, with_score_max, with_score_sum_exp, and is_dropout from the randomized SDPA test configurations. The plumbing in the sdpa test package is kept; only the test file stops exercising these options.
Before submitting
pre-commit runand committed any formatting changes.Affected area
Summary
Why
Related issues
API and compatibility impact
Testing
Summary by CodeRabbit
Bug Fixes
Tests