Feat: Add support for VieNeu-TTS v3 Turbo (48 kHz) Model Family

[phuocnguyen90]

Project Alignment & Vision

This contribution adds native C++ model support for VieNeu-TTS v3 Turbo (48 kHz), enabling highly optimized, torch-free CPU execution with GGUF quantization.

Community Value & Motivation

• Popular Open Source TTS: VieNeu-TTS is a leading open-source Vietnamese Text-to-Speech library with instant voice-cloning support (currently growing rapidly with 2.2k+ stars).
• Overcoming Python Latency: While highly accurate, the original implementation relies on PyTorch or ONNX Runtime on CPU, introducing significant startup and warmup latencies (combined ~11s).
• C++ Speedup: Integrating it into audio.cpp provides a highly optimized compiled C++ pathway that executes 4x faster than real-time (RTF 0.24) with a lightweight disk and memory footprint.

In alignment with audio.cpp's contribution guidelines:

• Decoupled Architecture (Zero Shared Regression): Since VieNeu-TTS v3 Turbo shares base concepts with Qwen3-TTS but uses custom conditioning and token mappings, we isolated the implementation inside a separate vietneu_tts directory and namespace. We did not modify or branch inside the existing qwen3_tts codebase, ensuring zero regression risk to existing models.
• Unified Surface Compositions: Registered the model family specification under model_specs/vietneu_tts.json to compose cleanly with the packaging and metadata surfaces.

---

📊 Validation & Parity Report

1. Build and Run Commands

Compile Target:

chmod +x scripts/build_linux.sh
./scripts/build_linux.sh --backend cpu --target audiocpp_cli

Execution Run:

./build/linux-cpu-release/bin/audiocpp_cli \
  --task tts \
  --family vietneu_tts \
  --model path/to/model.gguf \
  --backend cpu \
  --voice-ref path/to/sample.wav \
  --reference-text "Some call me nature. Others call me Mother Nature. I've been here for over 4.5 billion years. 22,500 times longer than you." \
  --text "sˈin tʃˈaː2w tˈeɜ zˈəːɜj. ɗˈəɪ lˌaː2 bˈaː4n tˈy4 ŋˈiɛ6m." \
  --out output.wav

2. Parity Test Results (Python vs C++)

In deterministic argmax mode (temperature $10^{-8}$), the generated token code sequences and predicted decoder logits match the Python reference implementation exactly.

• Predicted Logits (Step 1, first codebook):

  • Python: 741:8.3750 947:6.9375 286:6.6250 586:6.4062 546:6.2500
  • C++: 741:8.3750 947:6.9375 286:6.6250 586:6.4062 546:6.2500

• Generated Tokens Parity (First 10 steps):

  • Python: [741, 546, 894, 753, 50, 935, 607, 686, 141, 55]
  • C++: [741, 546, 894, 753, 50, 935, 607, 686, 141, 55]

• Output Audio Length Parity:
  Text logits projected from slot 0 are checked against EOS token 6, resulting in a clean generation finish at 3.68s (exactly matching Python speech length, with no silent padding).

3. Backend & Performance Notes

• Backend Tested: CPU (using OpenMP optimization on WSL2 Ubuntu Linux).
• Quantized GGUF Model Size: 163 MB (compared to the original multi-gigabyte safetensors sizes).
• Startup Latency: Instant (<0.01s) model load via mmap lazy file loading (compared to ONNX's 6.4s / PyTorch's 9.3s).
• Inference Time: 0.89 seconds on CPU (average generation speed) yielding a Real-Time Factor (RTF) of 0.24 (4.1x faster than real-time).

4. C++ Test Execution Results

To ensure zero framework regression, we compiled and executed the core unit test suite on both CPU and CUDA:

• sentencepiece_real_model_test: Passed (ok)
• audio_dsp_test: Passed (ok)
• scaled_dot_product_attention_test: Passed (ok on CUDA)
  Executed on an NVIDIA GeForce RTX 3060; compared standard explicit attention with flash attention and verified exact mathematical parity (cosine=1).
• General framework compilation: All core binaries (audiocpp_cli, audiocpp_server, audiocpp_gguf, model_perf) compiled successfully, demonstrating that the if (config_.rope_type >= 0) addition does not break framework builds.

The C++ core execution is verified to be regression-free and stable on both CPU and CUDA backends.

---

🔗 Replication & Testing References

• GGUF Converted Model Weights: phuocnguyen90/VieNeu-TTS-v3-Turbo-GGUF
• Original Upstream ONNX Weights: pnnbao-ump/VieNeu-TTS-v3-Turbo
• Original SDK Codebase: pnnbao97/VieNeu-TTS

---

🛠️ Implementation Details

1. Acoustic Decoder Input Conditioning: Correctly routes target text conditioning to use the Speech Generation Start (SGS) token 117 for VieNeu-TTS.
2. Shared Acoustic Embeddings Mapping (Off-by-One): Offset finer codebooks lookup tables mapping when loading code predictor shared weights.
3. Text/Semantic Logits Early Stopping: Added text logits projection steps and checked the argmax prediction against EOS token 6 to terminate autoregression early.
4. Decoupled Files Added:
   • src/models/vietneu_tts/talker.cpp
   • src/models/vietneu_tts/session.cpp
   • src/models/vietneu_tts/loader.cpp
   • src/models/vietneu_tts/assets.cpp
   • model_specs/vietneu_tts.json

---

🛡️ Shared Framework Safety Note

To integrate the new model family while maintaining strict backward compatibility and zero regression risk for existing models:

• src/framework/modules/attention/qwen_decoder.cpp:
  We added a safe, general check if (config_.rope_type >= 0) surrounding the query/key RoPE builds. This allows downstream layers that do not utilize rotary position embeddings (like the stack blocks in VieNeu-TTS's CodePredictor) to bypass the operator cleanly by setting rope_type = -1 (GGML's standard GGML_ROPE_TYPE_NONE).
  All existing Qwen3-TTS models are completely unaffected, as they specify positive rope_type configurations and will continue executing RoPE as normal.
• app/cli/request.cpp:
  Added CLI parsing arguments for --subtalker-temperature, --subtalker-do-sample, and automatic loading of speaker embeddings sidecar files (<ref_audio>.emb.txt) when --voice-ref is specified.
  This is fully backward-compatible for existing model run tasks, and has been verified to build and run correctly under the local validation suite.

[phuocnguyen90]

Conflict resolved with commit 91dc86d from upstream

[0xShug0]

@phuocnguyen90 Thanks for expanding audio.cpp’s model support!

I recently refactored the project layout to make long-term maintenance easier. Could you move this model under community_models/ instead of models/? Also please update the relevant docs, and update tools/model_manager.py if you want to provide safetensors model download support.

A few specific comments:

• Please drop the app/cli/request.cpp changes.
  The CLI should stay model-agnostic. Model-specific controls can already be passed through --request-option, for example --request-option subtalker_temperature=0.9 or --request-option subtalker_do_sample=true. Please document those options in the model docs instead of adding new global CLI flags.

• Please keep speaker embedding explicit and model-owned.
  The current sidecar behavior that auto-loads --voice-ref + ".emb.txt" is too implicit for the shared CLI. If VietNeU needs an external speaker embedding, please expose it as a VietNeU request option and parse it inside the model path. Also in the current code speaker embedding parsing silently swallows errors.

A better shape would be using request-option:

audiocpp_cli --task clon --family vietneu_tts \
  --model /path/to/VietNeU \
  --backend cuda \
  --voice-ref reference.wav \
  --reference-text "Reference transcript." \
  --request-option speaker_embedding_file=/path/to/reference.emb.txt \
  --text "Hello from VietNeU." \
  --out out.wav

• User-facing error are misleading. "Qwen3 TTS ..."

• Are VoiceDesign and CustomVoice acutally supported? The code seems to load only the Base model.

• It would be better if ModelCliInterface cli() exposes the VietNeU request options.

Also please provide the results based on #54 (comment)

[0xShug0]

@phuocnguyen90 The compile failures are my issue. I will fix it

[0xShug0]

@phuocnguyen90 Pushed b4bdec0. Please use a community-model-scoped constant cache for now. I’ll refactor it into a framework-level feature later.

[phuocnguyen90]

Hi @0xShug0,

Thank you for the detailed feedback and for setting up the community model structure! We have updated the PR to address all of your comments and synced with upstream main (including adopting src/community_models/common/constant_tensor_cache.h from commit b4bdec0).

Here is a summary of the updates applied:

1. Community Model Relocation & Docs

• Directory Layout: Relocated all source and header files under src/community_models/vietneu_tts/ and include/engine/community_models/vietneu_tts/.
• Documentation: Registered vietneu_tts in docs/community_models/models.md and created docs/community_models/vietneu_tts.md.

2. Model-Agnostic CLI & Request Options

• Reverted app/cli/request.cpp: Dropped all custom CLI flag additions to keep request.cpp completely model-agnostic.
• Explicit Speaker Embedding: Removed the implicit .emb.txt auto-loader. Speaker embeddings are now passed explicitly via --request-option speaker_embedding_file=/path/to/reference.emb.txt or --request-option speaker_embedding=<csv_floats>. Parsing errors in session.cpp now throw explicit std::runtime_error exceptions with detailed context.
• CLI Options Exposure: Updated loader.cpp so ModelCliInterface cli() exposes all supported request options (reference_text, speaker_embedding_file, speaker_embedding, subtalker_temperature, subtalker_do_sample).

3. Error Messages & Unused Task Cleanup

• Error Prefixes: Corrected all exception text prefixes from "Qwen3 " to "VieNeu-TTS " across the model runtime files.
• Removed Dead Code: Removed unused VoiceDesign and CustomVoice prompt builder files and structs, keeping only the supported Base model variant with native voice cloning (--voice-ref).

4. Model Manager Integration

• Registered vietneu_tts_v3_turbo in tools/model_manager.py pointing to phuocnguyen90/VieNeu-TTS-v3-Turbo-GGUF to download the 163 MB GGUF model and JSON sidecar configs. Verified that python tools/model_manager.py install vietneu_tts_v3_turbo installs and executes out-of-the-box.

---

5. Community Model Validation Results (With respect to issue #54)

• RTF < 1.0: Verified on OpenMP CPU backend:
  • Short sentence generation: RTF = 0.24 (3.68s speech generated in 0.89s).
  • Multi-sentence long-form text: RTF = 0.41 (22.08s speech generated in 9.10s).
• VRAM / Memory Stability: Tested consecutive requests and verified vietneu_tts.mem_saver=true, which correctly releases the talker step graph memory after each request.
• Long-form Generation: Integrates the framework's native runtime::chunk_text_request helper for seamless long-form text chunking.
• Framework Abstractions: Leverages standard framework modules (LlamaBpeTokenizer, QwenDecoderPositionEncoding::None, VoicePromptContextCache, etc.) to benefit automatically from future framework optimizations.

All changes have been tested locally and pushed to the PR branch. Thanks again for your guidance!

[0xShug0]

@phuocnguyen90 Thank you! PR merged.

[0xShug0]

Updated main README to include your model. Please help spread the word at https://github.com/pnnbao97/VieNeu-TTS or reddit.