fix: mask all stop_token_ids in sampling_ids + multi-GPU device support

[lonrencn]

Summary

Two bug fixes for CosyVoice3 inference:

1. sampling_ids() only masks speech_token_size (sos) during ignore_eos=True, not eos or other stop tokens — on certain transformers versions, the LLM can predict EOS with high confidence as the first token, and the unmasked EOS leaks through the ignore_eos window, producing garbage output.

2. Hardcoded torch.device('cuda') defaults to cuda:0 — on multi-GPU systems, torch.cuda.set_device() has no effect and models load on the wrong GPU.

Changes

cosyvoice/llm/llm.py — sampling_ids fix

CosyVoice3LM.stop_token_ids = [speech_token_size + i for i in range(200)] covers sos(6561), eos(6562), task_id(6563), fill_token(6564), etc. The original code only masked index speech_token_size (sos=6561), leaving eos(6562) unmasked during the minimum-length window.

This is usually harmless because a well-functioning LLM doesn't predict EOS early. But with slight numerical differences across transformers versions (e.g., SDPA mask handling changed between 4.x and 5.x), the logits can shift enough (~0.012) to flip EOS from #2 to #1, causing the model to output garbage.

The fix uses hasattr to maintain backward compatibility with CosyVoice1/2.

cosyvoice/cli/model.py + cosyvoice/cli/frontend.py — multi-GPU fix

torch.device('cuda') always defaults to cuda:0, ignoring torch.cuda.set_device(). Changed to torch.device(f'cuda:{torch.cuda.current_device()}') in all model classes.

Testing

Verified with Fun-CosyVoice3-0.5B-2512:

• Chinese: "你好世界" → ASR: "你好，世界。" ✅
• English: "Hello, this is a test." → ASR correct ✅
• Environment: Python 3.11, torch 2.11+cu128, transformers 5.3.0

[LauraGPT]

FunAudioLLM-side triage/validation on current head a12ae5e2c9eadce54c97a1b97f9fde91c0e2075c.

What I checked locally:

git merge-base HEAD origin/main
# 074ca6dc9e80a2f424f1f74b48bdd7d3fea531cc, same as current main

git diff --check origin/main...HEAD
# no whitespace errors

python3 -m py_compile cosyvoice/cli/frontend.py cosyvoice/cli/model.py cosyvoice/llm/llm.py
# passed

The sampling_ids() change is consistent with the class definitions I checked: Qwen2LM.stop_token_ids covers speech_token_size + range(3), and the CosyVoice3 LM class covers speech_token_size + range(200), so masking self.stop_token_ids during the ignore_eos window is aligned with the model-specific stop-token range while preserving the old fallback for older classes without that attribute.

I also sanity-checked the CPU branch of the new device expression in this environment; torch.device(torch.device("cpu")) is accepted by the installed PyTorch, so the expression is not an immediate CPU runtime blocker. I did not run full model inference because this environment does not have the required CosyVoice3 weights and usable CUDA driver, so this is a scoped syntax/static review rather than an end-to-end approval.

[shuvro-sarker]

Independent confirmation of the sampling_ids half of this PR, from benchmarking Fun-CosyVoice3-0.5B-2512 on an L40S. We found the same defect by instrumentation before finding this PR, and reached the same fix.

Confirming the mechanism as you describe it: CosyVoice3LM renumbers its special tokens (sos = speech_token_size + 0, eos_token = speech_token_size + 1) but inherits TransformerLM.sampling_ids, which hard-codes weighted_scores[self.speech_token_size] = -inf. That masks sos, which the model never emits, while the real EOS stays sampleable from step zero — so the minimum-length guard is inoperative rather than merely weak. We verified this by logging which id is masked and the stop_token_offset at the step the loop breaks.

One additional symptom worth recording: an extremely early stop doesn't only produce garbage or near-silence. It can also leave HiFi-GAN's f0 predictor with too few mel frames, surfacing as Kernel size can't be greater than actual input size — i.e. this can present as a crash rather than as bad audio, which makes it easy to misattribute to the vocoder.

Being straight about our evidence, since it argues for merging but not for urgency: under correct conditioning we measured 0/64 failures both with and without the fix on an English verified-transcript reference voice. We did see failures (15/64) under a mis-conditioned setup, but that turned out to be a bug in our own harness — a wrong-language reference voice paired with a mismatched transcript — not in CosyVoice, so we don't offer those numbers as evidence about this defect.

So our data supports your analysis that this is normally latent and only fires when numerical differences flip EOS to the top — it does not independently demonstrate visible breakage at the default operating point. It's still worth merging: the guard is provably a no-op as written, the fix is one line, and the hasattr guard keeps CosyVoice1/2 behaviour intact. A latent correctness bug that depends on transformers-version numerics is exactly the kind that resurfaces later as an unreproducible report.

(We have no data on the multi-GPU half of this PR — we ran single-GPU throughout.)