NPU port: streaming_session_extra + multi_instance_release_memory_occupation (cases 9, 10) - #1035
Conversation
…ccupation
Port two remaining test cases to Ascend NPU backend:
1. test_npu_streaming_session_extra.py (ported from test_streaming_session_extra.py)
- 5 variants: retract+mixed-chunk, retract+large-page, EAGLE3 v1/v2,
EAGLE3+retract+large-page
- Uses LLAMA_3_1_8B_INSTRUCT + EAGLE3_LLAMA3_1_INSTRUCT_8B weights
- Adds --attention-backend ascend, --disable-cuda-graph,
--disable-piecewise-cuda-graph; page_size adapted to 128/4
2. test_npu_multi_instance_release_memory_occupation.py (ported from
test_multi_instance_release_memory_occupation.py)
- 4-NPU (dp=2 * tp=2) multi-process spawn test
- torch.cuda -> torch.npu (mem_get_info, set_device, empty_cache)
- HCCL backend for dist.init_process_group
- Engine kwargs: attention_backend=ascend, disable_cuda_graph=True
- Uses LLAMA_3_2_1B_INSTRUCT + LLAMA_3_2_1B weights
CI: single-test-npu.yml updates
- Image: swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/
lmsysorg/sglang:main-cann9.0.0-a3
- Runner: linux-aarch64-a3-4 (multi_instance needs 4 NPUs)
- test_cases: the two new files
CLA Signature Passhhhh1252023, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
Three errors occurred in PR sgl-project#1035's CI run: 1. TestNPUStreamingSessionEagle / EagleRetractLargePage: `RuntimeError: npu_rms_norm ... error code is 161002` `Io input dtype or format is not supported, get io input info is x(DT_BFLOAT16, ND) gamma(DT_FLOAT16, ND) ...` Server process killed (exit code -9). Root cause: EAGLE3-LLaMA3.1-Instruct-8B draft's RMSNorm weight is float16 while Llama-3.1-8B main model hidden state is bfloat16. NPU aclnnRmsNorm does not support this dtype combination. Fix: switch all EAGLE3 variants to Qwen3-8B + Qwen3-8B_eagle3 (the same pair already validated in test_npu_streaming_session.py). 2. TestNPUStreamingSessionEagleV2: `AttributeError: 'Envs' object has no attribute 'SGLANG_ENABLE_SPEC_V2'` The CI image's sglang version does not yet expose this attr. Fix: remove the `SGLANG_ENABLE_SPEC_V2` env override. The variable defaults to True in current sglang, so no explicit setting is needed. 3. RetractMixedChunk / RetractLargePage: switch from Llama-3.1-8B-Instruct to Qwen3-8B for consistency with the EAGLE3 variants (single model pair across the whole file reduces cache pressure and CI runtime).
CLA Signature Passhhhh1252023, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
The upstream StreamingSessionKitMixin hard-codes `max_new_tokens=100000` in test_nth_mid_abort_recovery and test_first_mid_abort_recovery. With Qwen3-8B (context_length=40960) the server returns HTTP 400 and the test fails with `KeyError: 'meta_info'` (see .claude/2.log). Add NPUStreamingSessionKitMixin that overrides both methods to use max_new_tokens=40000 (still > context, but fits because the abort fires within 0.5s before the limit is reached; the original 100000 was also never meant to be fully generated). All 5 test classes now inherit from NPUStreamingSessionKitMixin instead of StreamingSessionKitMixin.
CLA Signature Passhhhh1252023, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
CLA Signature Guide@hhhh1252023 , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
CLA Signature Guide@hhhh1252023 , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
CLA Signature Guide@hhhh1252023 , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
CLA Signature Guide@hhhh1252023 , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
CLA Signature Guide@hhhh1252023 , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
CLA Signature Guide@hhhh1252023 , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
CLA Signature Guide@hhhh1252023 , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
CLA Signature Guide@hhhh1252023 , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
CLA Signature Guide@hhhh1252023 , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
CLA Signature Guide@hhhh1252023 , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
CLA Signature Guide@hhhh1252023 , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
CLA Signature Guide@hhhh1252023 , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
CLA Signature Guide@hhhh1252023 , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
CLA Signature Guide@hhhh1252023 , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
CLA Signature Guide@hhhh1252023 , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
CLA Signature Guide@hhhh1252023 , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
CLA Signature Guide@hhhh1252023 , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
| cls.model, | ||
| cls.base_url, | ||
| timeout=DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH, | ||
| other_args=["--enable-streaming-session"] + list(cls.extra_args), |
There was a problem hiding this comment.
--enable-streaming-session is already in _NPU_COMMON_ARGS
CLA Signature Guide@hhhh1252023 , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
CLA Signature Guide@hhhh1252023 , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
CLA Signature Guide@hhhh1252023 , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
| pip install zss pre-commit wandb>=0.16.0 tenacity==8.3.0 loguru openpyxl latex2sympy2 zstandard transformers-stream-generator tqdm-multiprocess pycocoevalcap | ||
| pip install yt-dlp sentencepiece==0.1.99 nltk av ftfy sqlitedict==2.1.0 sacrebleu>=1.5.0 pytablewriter black==24.1.0 isort==5.13.2 peft>=0.2.0 accelerate>=0.29.1 | ||
| pip install jsonlines httpx==0.25.0 evaluate>=0.4.0 datasets==2.16.1 numexpr xgrammar==0.1.25 numpy==1.26.4 dotenv | ||
| pip install /root/.cache/torch_memory_saver-0.0.8-cp311-cp311-linux_aarch64.whl |
There was a problem hiding this comment.
Use /root/.cache/.cache/npu-pkgs/torch_memory_saver-0.0.8-cp311-cp311-linux_aarch64.whl instead of root path of nfs.
4 participants
Summary
Port two remaining test cases to Ascend NPU backend (用例 9、10 移植)。
Changes
1.
test_npu_streaming_session_extra.py(new)Ported from
test/registered/sessions/test_streaming_session_extra.py. 5 variants:TestNPUStreamingSessionRetractMixedChunk— retract +--enable-mixed-chunkTestNPUStreamingSessionRetractLargePage— retract + page=128TestNPUStreamingSessionEagle— EAGLE3 spec v1 (overlap disabled, offset=-1)TestNPUStreamingSessionEagleV2— EAGLE3 spec v2 (overlap on)TestNPUStreamingSessionEagleRetractLargePage— EAGLE3 + retract + page=128NPU adaptations:
LLAMA_3_1_8B_INSTRUCT_WEIGHTS_PATH+EAGLE3_LLAMA3_1_INSTRUCT_8B_WEIGHTS_PATH--attention-backend ascend,--disable-cuda-graph,--disable-piecewise-cuda-graphchunked-prefill-sizedivisible by 128PYTORCH_NPU_ALLOC_CONF=expandable_segments:True,HCCL_EXEC_TIMEOUT=2002.
test_npu_multi_instance_release_memory_occupation.py(new)Ported from
test/registered/rl/test_multi_instance_release_memory_occupation.py. 4-NPU multi-process spawn test.NPU adaptations:
torch.cuda→torch.npu(mem_get_info, set_device, empty_cache)dist.init_process_groupbackend=hccl(instead of nccl default)cuda:{rank}→npu:{rank}attention_backend="ascend",disable_cuda_graph=True,disable_piecewise_cuda_graph=TrueLLAMA_3_2_1B_INSTRUCT_WEIGHTS_PATH+LLAMA_3_2_1B_WEIGHTS_PATHmp.set_start_method("spawn", force=True)at module import3.
.github/workflows/single-test-npu.ymlswr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.0.0-a3linux-aarch64-a3-4(multi_instance needs 4 NPUs for dp=2*tp=2)test_cases: the two new filesCI Registration
Both test files use
register_npu_ci(...):test_npu_streaming_session_extra.py:suite="full-1-npu-a3",est_time=900,nightly=Truetest_npu_multi_instance_release_memory_occupation.py:suite="full-4-npu-a3",est_time=600,nightly=TrueTest Plan
test_npu_streaming_session_extra.py— 5 test classes pass on 1-NPUtest_npu_multi_instance_release_memory_occupation.py— 1 test class passes on 4-NPUCI States
Latest PR Test (Base): ❌ Missing
run-cilabel -- add it to run CI tests.Latest PR Test (Extra): ❌ Blocked --
run-ciis required first.