TTS OutieTTS 1.0 + cloning audio

[mirek190]

Summary

• add OuteTTS 1.0 1B normal TTS and reference-audio voice cloning
• support the safetensors package and a standalone packed GGUF
• embed IBM DAC, Qwen3 Forced Aligner, tokenizers, configs, sidecars, and the package spec in the one-file GGUF
• add model-manager installation, DAC conversion, CLI/server integration, long-form chunking, reference-profile caching, graph reuse, tracing, and an opt-in memory-saver path
• match the official Python CUDA sampling stream through the framework's PyTorch-compatible exponential sampler
• match official clone prompt construction, including forced-aligner word labels, punctuation, and number formatting

Normal TTS does not initialize the embedded aligner. It is loaded only when cloning needs reference word timing. Safetensors packages and older OuteTTS GGUFs without the embedded aligner can use outetts.aligner_model_path.

Model setup and tested paths

python tools/model_manager.py install outetts_1_0_1b --models-dir models

Package id outetts_1_0_1b installs:

• models/Llama-OuteTTS-1.0-1B
• models/DAC.speech.v1.0
• models/Qwen3-ForcedAligner-0.6B

The standalone path test used only this file outside the PR worktree:

..\models\Llama-OuteTTS-1.0-1B_Q8\Llama-OuteTTS-1.0-1B_Q8.gguf

Exact build commands

Windows CPU:

.\scripts\build_windows.ps1 -Preset windows-cpu-release -ConfigureOnly
cmake -S . -B build\windows-cpu-release -DENGINE_BUILD_WARMBENCH=ON
cmake --build build\windows-cpu-release --parallel 4 --target audiocpp_cli audiocpp_server outetts_warm_bench

Windows CUDA 12.4:

$env:CUDA_PATH = "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4"
.\scripts\build_windows.ps1 -Preset windows-cuda-release -ConfigureOnly
cmake -S . -B build\windows-cuda-release -DENGINE_BUILD_WARMBENCH=ON
cmake --build build\windows-cuda-release --parallel 4 --target audiocpp_cli audiocpp_server outetts_warm_bench

Both command sequences passed. torch_random_test also passed on Windows CPU.

Exact standalone run commands

Normal TTS:

build\windows-cuda-release\bin\audiocpp_cli.exe `
  --task tts --family outetts `
  --model ..\models\Llama-OuteTTS-1.0-1B_Q8\Llama-OuteTTS-1.0-1B_Q8.gguf `
  --backend cuda --text "This is the standalone GGUF path test." `
  --max-tokens 256 --request-option seed=1234 `
  --out ..\outputs\outetts_path_test_tts.wav

One-file voice cloning:

build\windows-cuda-release\bin\audiocpp_cli.exe `
  --task clon --family outetts `
  --model ..\models\Llama-OuteTTS-1.0-1B_Q8\Llama-OuteTTS-1.0-1B_Q8.gguf `
  --backend cuda --voice-ref assets\resources\b.wav `
  --reference-text "Some call me nature. Others call me Mother Nature. I've been here for over 4.5 billion years. 22,500 times longer than you." `
  --request-option reference_language=en `
  --text "This is the standalone GGUF cloning path test." `
  --max-tokens 256 --request-option seed=42 `
  --out ..\outputs\outetts_path_test_clone.wav

Both loaded with no adjacent sidecar or external model_specs directory and passed ffmpeg decoding.

Controlled Python parity

The official HF reference uses OuteTTS commit f5eac6e70d792844c6a6959d900a47af2c061a5b, FP32 CUDA weights, and the official IBM DAC checkpoint. Temperature zero/greedy was deliberately not used: the model's official generation_config.json, README example, and Python API specify sampled generation at temperature 0.4. Both sides use temperature 0.4, repetition penalty 1.1 over 64 tokens, top-k 40, top-p 0.9, min-p 0.05, and the request-file seed.

OuteTTS has no diffusion or latent-noise input; language-model sampling is its only request-time random boundary. C++ CUDA now uses the framework's PyTorch-compatible exponential-race stream. The committed Python driver resets both the CPU and CUDA generators immediately before every model.generate, after prompt/profile construction:

build\reference\venv\Scripts\python.exe `
  tools\audiocpp_cli\outetts_reference.py `
  --model ..\models\Llama-OuteTTS-1.0-1B `
  --dac ..\models\DAC.speech.v1.0\weights_24khz_1.5kbps_v1.0.pth `
  --alignment-json build\reference\b_words.json `
  --request-file tests\outetts\warm_bench_requests.json `
  --device cuda --dtype fp32 `
  --out-dir build\reference\official_script_fp32

Matching C++ FP32 run:

build\windows-cuda-release\bin\outetts_warm_bench.exe `
  --model ..\models\Llama-OuteTTS-1.0-1B `
  --backend cuda --threads 8 `
  --request-file tests\outetts\warm_bench_requests.json `
  --session-option outetts.weight_type=f32 `
  --session-option outetts.aligner_model_path=..\models\Qwen3-ForcedAligner-0.6B `
  --audio-out-dir build\reference\cpp_f32_final_explicit `
  --log-file build\reference\cpp_f32_final_explicit.log

Temporary trace logs compared Python/C++ prompt strings, token IDs, generated IDs, codec codes, and frame counts at component boundaries, then were removed to avoid runtime log spam. The committed Python driver writes boundaries.json. All six ordinary-TTS chunks had zero sampled prompt/generated boundary mismatches. Clone prompts are 2367 tokens on both sides. All FP32 WAV frame counts match exactly. The remaining clone drift is isolated to the reference DAC encoder: among 1046 profile frames, C1 differs on 48 and C2 on 118.

Primary FP32 parity:

Request	WAV cosine	Log-mel cosine	C++ / Python frames
tts_cold	0.999812247	0.999997481	31672 / 31672
tts_repeat	0.999812238	0.999997483	31672 / 31672
tts_longform	0.997324530	0.999294328	135008 / 135008
clone_cold	-0.007753064	0.889403845	36472 / 36472
clone_repeat	-0.007753064	0.889403845	36472 / 36472

Only after FP32 boundary/frame parity was clean was Q8 compared with the same FP32 Python reference. Quantized autoregressive sampling drift is expected:

Request	Q8 WAV cosine	Q8 log-mel cosine	C++ Q8 / Python frames
tts_cold	-0.016326171	0.696795792	31672 / 31672
tts_repeat	-0.016326591	0.696792751	31672 / 31672
tts_longform	0.577927575	0.920741352	133408 / 135008
clone_cold	-0.033990775	0.882715855	36472 / 36472
clone_repeat	-0.033990775	0.882715855	36472 / 36472

Performance per request

Measured on an RTX 3090 with CUDA 12.4 in one loaded session. Python/C++ FP32 are single runs. Q8 values are mean +/- sample standard deviation from three fresh processes; model loading is excluded.

Request	Python HF FP32 wall / RTF	C++ FP32 wall / RTF	C++ Q8 wall / RTF
tts_cold	4599.12 ms / 3.485	5366.24 ms / 4.066	3211.84 +/- 17.30 ms / 2.434
tts_repeat	3777.69 ms / 2.863	3864.98 ms / 2.929	3030.79 +/- 18.60 ms / 2.297
tts_longform	15129.91 ms / 2.690	15547.80 ms / 2.764	13015.83 +/- 171.29 ms / 2.342
clone_cold	5762.51 ms / 3.792	5988.01 ms / 3.940	6432.68 +/- 65.11 ms / 4.233
clone_repeat	5522.67 ms / 3.634	4231.19 ms / 2.784	4220.23 +/- 44.07 ms / 2.777

Memory-saver A/B per request

Three fresh processes per mode were alternated to reduce order bias:

Request	Default CUDA Q8	Memory saver CUDA Q8	Wall delta	Default RTF	Memory-saver RTF
tts_cold	3211.84 +/- 17.30 ms	3223.24 +/- 12.89 ms	+0.35%	2.434	2.443
tts_repeat	3030.79 +/- 18.60 ms	3044.58 +/- 18.76 ms	+0.45%	2.297	2.307
tts_longform	13015.83 +/- 171.29 ms	13070.17 +/- 105.32 ms	+0.42%	2.342	2.351
clone_cold	6432.68 +/- 65.11 ms	6467.00 +/- 46.38 ms	+0.53%	4.233	4.256
clone_repeat	4220.23 +/- 44.07 ms	4238.27 +/- 17.07 ms	+0.43%	2.777	2.789

Memory saver is not faster: the +0.35% to +0.53% per-request differences are close to normal run variance, and the mean sequence cost is +0.44%. It remains useful because measured peak VRAM fell from 17653 MiB to 5780 MiB (67.3%), with the same 294 MiB post-session resident VRAM. It is a memory/latency tradeoff, not a speed optimization.

Generated artifacts

• ..\outputs\outetts_path_test_tts.wav
• ..\outputs\outetts_path_test_clone.wav
• build\reference\official_script_fp32\outputs\*.wav
• build\reference\official_script_fp32\boundaries.json
• build\reference\cpp_f32_final_explicit\*.wav
• build\reference\cpp_q8_torch_sampler\*.wav
• build\logs\warmbench\outetts-ab-torch-sampler\{default,mem_saver}-{1,2,3}-audio\*.wav

These are reproducible local validation outputs and are not committed.

Backend coverage

Backend	Coverage
Windows CPU	CLI/server/warm-benchmark build and complete runtime sequence, including per-request timing and RSS
Windows CUDA 12.4	CLI/server/warm-benchmark build, standalone path, controlled Python FP32 parity, Q8 characterization, per-request A/B, timing/RTF, and VRAM
Linux CPU	GitHub Actions compile check passed; no model runtime claimed
Linux Vulkan	GitHub Actions compile check passed; no model runtime claimed
macOS CPU	GitHub Actions compile check passed; no model runtime claimed
Metal	Not enabled or runtime-tested

Known limitations

• offline only
• cloning requires an accurate transcript and rejects references longer than 20 seconds
• quantized CUDA cloning expands language-model weights to F32 in memory for coherent generation
• long-form output concatenates independently generated chunks; max_tokens applies per chunk
• CUDA uses the PyTorch-compatible sampler; CPU sampling remains deterministic but is not claimed to reproduce PyTorch token-for-token
• quantization can change sampled tokens and output length, even with matching sampler settings and seed

Documentation

• docs/tts.md
• docs/outetts_validation.md
• docs/memory_saver.md

Converted GGUF: https://huggingface.co/mirek190/audio.cpp/tree/main

[0xShug0]

@mirek190 For new models/frameork module change, please check https://github.com/0xShug0/audio.cpp?tab=contributing-ov-file#new-model-prs and report parity and perf mertics as in #19. Your framework changes look fine since they're additive.

For the Python side, you only need to provide the setup instructions. I’ll handle the setup and validation myself.

Beyond #19:

• Use the framework’s text chunker to handle longform, and verify the chunking behavior with logs.
• Use CacheSlot to cache stable state, such as speaker embeddings, and verify cache hits and reuse with logs.
• The same graph should not be cached in multiple instances unless the VRAM overhead is minimal.
• Use logs to verify graph rebuild/reuse behavior in long-lived sessions.
• Use timing logs to identify suspicious hot paths and improve performance.
• Measure peak VRAM usage under multiple requests. If it is too high relative to the model size, the cause may be bugs or simply the use of ggml_backend_alloc_ctx_tensors. How to handle ggml_backend_alloc_ctx_tensors should depend on A/B performance tests. If ggml_backend_alloc_ctx_tensors provides a large performance boost, gate it behind mem_saver mode instead of removing it outright. High peak VRAM may also be caused by graph lifecycle issues, such as not releasing graphs. Likewise, if releasing every graph after use significantly hurts performance, gate that behavior behind mem_saver. Note that the performance impact may only become clear in long-form tests.

(Originally, I only cared about performance and VRAM usage stability, but more and more users asked for VRAM reduction, so I started adding VRAM optimization to new models.)

If you compare the final shape of Moss TTS Local with the original PR, you can see that there were many changes. Because of that, I’d like to make the tests stricter. I’ll create a checklist later.

This is not meant to discourage the PR. The current direction looks good; I just want the validation to be strict enough that we can merge it with confidence and avoid another large cleanup pass later.

[mirek190]

@mirek190 For new models/frameork module change, please check https://github.com/0xShug0/audio.cpp?tab=contributing-ov-file#new-model-prs and report parity and perf mertics as in #19. Your framework changes look fine since they're additive.

For the Python side, you only need to provide the setup instructions. I’ll handle the setup and validation myself.

Beyond #19:

• Use the framework’s text chunker to handle longform, and verify the chunking behavior with logs.
• Use CacheSlot to cache stable state, such as speaker embeddings, and verify cache hits and reuse with logs.
• The same graph should not be cached in multiple instances unless the VRAM overhead is minimal.
• Use logs to verify graph rebuild/reuse behavior in long-lived sessions.
• Use timing logs to identify suspicious hot paths and improve performance.
• Measure peak VRAM usage under multiple requests. If it is too high relative to the model size, the cause may be bugs or simply the use of ggml_backend_alloc_ctx_tensors. How to handle ggml_backend_alloc_ctx_tensors should depend on A/B performance tests. If ggml_backend_alloc_ctx_tensors provides a large performance boost, gate it behind mem_saver mode instead of removing it outright. High peak VRAM may also be caused by graph lifecycle issues, such as not releasing graphs. Likewise, if releasing every graph after use significantly hurts performance, gate that behavior behind mem_saver. Note that the performance impact may only become clear in long-form tests.

(Originally, I only cared about performance and VRAM usage stability, but more and more users asked for VRAM reduction, so I started adding VRAM optimization to new models.)

If you compare the final shape of Moss TTS Local with the original PR, you can see that there were many changes. Because of that, I’d like to make the tests stricter. I’ll create a checklist later.

This is not meant to discourage the PR. The current direction looks good; I just want the validation to be strict enough that we can merge it with confidence and avoid another large cleanup pass later.

Sure

Updated to your request

• Framework text chunking for long-form synthesis.
• CacheSlots caching for prepared cloning reference profiles.
• Reusable generation graph across compatible requests and chunks.
• Only one active OuteTTS Llama runtime, avoiding duplicate graph/model caches.
• Detailed cache, graph lifecycle, chunking, and performance logs.
• Optional outetts.mem_saver=true.
• Reproducible five-request warm benchmark.
• Python reference setup and complete validation documentation.
• Updated PR description with performance, VRAM, parity, and limitations.

RTX 3090 with gguf Q8 benchmark:

• Default peak VRAM: 17,653 MiB
• Memory-saver peak VRAM: 5,780 MiB
• Repeated TTS and cloning outputs were byte-identical.
• All generated WAV files passed FFmpeg validation.

making more tests ...

[0xShug0]

@mirek190 Could you report parity and performance vs. python on a per-request basis instead of only reporting aggregated results? For parity with python, the metric is wav cosine or log-mel similarity.

CPU performance is good to have but not required. CUDA optimization is the priority.

So mem saver on is always faster than off? If this case there is no need to add the mem saver option.

[mirek190]

@mirek190 Could you report parity and performance vs. python on a per-request basis instead of only reporting aggregated results? For parity with python, the metric is wav cosine or log-mel similarity.

CPU performance is good to have but not required. CUDA optimization is the priority.

So mem saver on is always faster than off? If this case there is no need to add the mem saver option.

First post updated

[0xShug0]

@mirek190 Thanks for the update! Generally I expect the log mel > 85%. For most of the models the parity can be 99% when randomness is fully controlled.

several suggestions:

(1) test temperature=0 / greedy, at least it isolates drifts in the sampling part.
(2) use the framework trace log to print the values at the compoent boundaries from upstream to downstream, e.g., compare python vs cpp prompt string and token IDs for TTS and clone (make sure remove the parity debug logs to avoid spamming the logs).
(3) For randomness such as noise, you can add a debug option to pass the noise to fix the noise on both sides.
(4) sampled run only after deterministic path is clean.
(5) use native weights or force fp32. Quant parity drift is expected.
(6) If wav frame count under the deterministic mode differs from ptyhon, that’s a strong sign of a logic mismatch.

Make sure the randomness control is wired correctly in python. E.g., acestep requires two seeds, so setting only the outer one won’t work. The framework has a TorchRandn which matches torch.randn closely enough for model parity tests.

Let me know if you have any questions about my suggestions.

[mirek190]

(1) test temperature=0 / greedy, at least it isolates drifts in the sampling part.

This specific model has ONLY one specific stable temperature and is set already. When you change it the model just collapses.

the rest in progress ....

[mirek190]

@mirek190 Thanks for the update! Generally I expect the log mel > 85%. For most of the models the parity can be 99% when randomness is fully controlled.

several suggestions:

(1) test temperature=0 / greedy, at least it isolates drifts in the sampling part. (2) use the framework trace log to print the values at the compoent boundaries from upstream to downstream, e.g., compare python vs cpp prompt string and token IDs for TTS and clone (make sure remove the parity debug logs to avoid spamming the logs). (3) For randomness such as noise, you can add a debug option to pass the noise to fix the noise on both sides. (4) sampled run only after deterministic path is clean. (5) use native weights or force fp32. Quant parity drift is expected. (6) If wav frame count under the deterministic mode differs from ptyhon, that’s a strong sign of a logic mismatch.

Make sure the randomness control is wired correctly in python. E.g., acestep requires two seeds, so setting only the outer one won’t work. The framework has a TorchRandn which matches torch.randn closely enough for model parity tests.

Let me know if you have any questions about my suggestions.

First post updated with your suggestions

[0xShug0]

@mirek190 Thank you! The results look promising. Would you be willing to refactor the asset and loader code to match the current style (same high-level interface, remove the usage of "as_xx, etc)? This will make the later abstraction easier. I did a large refactor sweep last night, so you can use the updated code as a reference for the new style.

[0xShug0]

@mirek190 Also, could you create a small separate PR for the framework extension? I’d like to merge that first because I’m currently extending the AR template and want to include your changes.

[mirek190]

@mirek190 Also, could you create a small separate PR for the framework extension? I’d like to merge that first because I’m currently extending the AR template and want to include your changes.

done

#68

[mirek190]

@mirek190 Thank you! The results look promising. Would you be willing to refactor the asset and loader code to match the current style (same high-level interface, remove the usage of "as_xx, etc)? This will make the later abstraction easier. I did a large refactor sweep last night, so you can use the updated code as a reference for the new style.

done

[0xShug0]

Thanks! I'll do my tests after I wrap up the Voxtral real-time model.

[mirek190]

Thanks! I'll do my tests after I wrap up the Voxtral real-time model.

Sure
Do not rush.

[mirek190]

Hey 0xShug0 how fast is you higgs v3 implementation as I do not see how it looks your work. I remember you mentioned about you were working on higgs v3. If you want I can push my implementation

I was working on mine for 3 weeks and could only compare to qwen 3 tts which is implemented under audiocpp.

Because mine my Higgs v3 TTS 4B is substantially faster than even much smaller audio.cpp’s Qwen3-TTS 0.6B on the RTX 3090.

Implementation Audio length Inference RTF Speed
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Higgs v3, 4B Q8_0, Flash Attention 4.84 s 344 ms 0.071 14.1× realtime
─────────────────────────────────────── ──────────────
Qwen3-TTS 0.6B, native, first request 4.64 s 1,451 ms 0.313 3.20× realtime
─────────────────────────────────────── ──────────────
Qwen3-TTS 0.6B, warm average 4.64 s 815 ms 0.176 5.69× realtime

I was testing with this sample
EN_2.wav

[0xShug0]

@mirek190 Current warm Higgs TTS RTF is ~7.7x realtime Q8 while Qwen3-TTS 1.7B native is 5.7x realtime on RTX5090. So yes, Higgs TTS is faster than Qwen3. It currently still stays at the "parity safe" stage without further perf optimization (e.g., still using naive attention). I got distracted by the World Cup and by upgrading GGML to v0.16 or v0.17 (just released). Couldn’t decide whether to upgrade yet because the performance impact has been mixed.

Let me know if you're interested in integrating your implementation into audio.cpp. I can push my current half-baked implementation to the dev branch so you can check. (It uses the refactored framework AR template.) No pressure at all. I completely understand if you'd rather publish it as a dedicated project.

[mirek190]

@mirek190 Current warm Higgs TTS RTF is ~7.7x realtime Q8 while Qwen3-TTS 1.7B native is 5.7x realtime on RTX5090. So yes, Higgs TTS is faster than Qwen3. It currently still stays at the "parity safe" stage without further perf optimization (e.g., still using naive attention). I got distracted by the World Cup and by upgrading GGML to v0.16 or v0.17 (just released). Couldn’t decide whether to upgrade yet because the performance impact has been mixed.

Let me know if you're interested in integrating your implementation into audio.cpp. I can push my current half-baked implementation to the dev branch so you can check. (It uses the refactored framework AR template.) No pressure at all. I completely understand if you'd rather publish it as a dedicated project.

Sure pusch to dev and I compare to my implementation

[0xShug0]

@mirek190 Pushed! Feel free to submit a PR to optimize the implementation.