Optimize Higgs Audio v3 inference by mirek190 · Pull Request #79 · 0xShug0/audio.cpp

mirek190 · 2026-07-19T11:36:21Z

Summary

Continuation work of 0xShug0
from commit 2e5c5e5

This PR optimizes the existing Higgs Audio v3 TTS implementation while keeping
the public model/session interface unchanged.

packs Q/K/V and gate/up projections so the shared Qwen decoder issues fewer
matrix multiplications and uses fused SwiGLU;
uses grouped FlashAttention without materializing repeated KV heads;
stores transformer KV in F16, grows it in buckets, and updates it through
direct SET_ROWS graphs that expose the CUDA ROPE -> VIEW -> SET_ROWS
fusion pattern;
retains a cloned-reference KV prefix and prefills only the request-specific
text suffix on subsequent requests with the same reference;
keeps the corrected Higgs sampling/EOC handling used by CLI and server
generation;
adds focused projection, mask, F16 KV, and graph-structure tests;
adds a reproducible five-request CUDA runner and request-level waveform,
log-mel, timing, RTF, and frame-count comparison tool.

voice reference
      |
      +-- first request --> encode + reference prefill --> retained KV prefix
      |                                                   |
      +-- later request ----------------------------------+
                                                          v
text --> suffix-only prefill --> packed QKV / grouped FA --> decode
                                    |
                                    +--> packed gate+up / fused SwiGLU

Base and comparison point

PR base: dev at 698ea146ca5a83ad074c9278522ab810729ee3ce
dev already contains current main at
a0f3b4cc0669d854ef6a239c7cbcf6a90dac1b1d.
Performance baseline: the original Higgs implementation at
2e5c5e568b252241d39d7d521d10c462ac8f442d.
No intervening commit between that baseline and current dev touches the
Higgs model, shared Qwen decoder, or KV-cache paths changed here.

The historical commit did not register higgs_tts_warm_bench as a CMake
target and accepted only inline request JSON. For the detached baseline
worktree I made two harness-only changes: registered that existing source as a
warmbench target and added --request-sequence-file. No baseline runtime or
model source was changed.

Model setup and tested paths

Native model-manager package:

python tools/model_manager.py install higgs_audio_v3_tts_4b --models-root ../models

The benchmark used the standalone Q8 GGUF already produced by the GGUF tool:

E:\LLAMA\llama.cpp\new4\TEST_7_audiocpp\models\higgs-audio-v3-tts-4b_Q8\higgs-audio-v3-tts-4b_Q8.gguf

Voice reference:

E:\LLAMA\llama.cpp\new4\TEST_7_audiocpp\SAMPLES\EN_2.wav
reference text: If you actually care about security.

Exact candidate build commands

Windows PowerShell 5.1, Visual Studio 2022 17.13.5, Ninja, CUDA 12.4, RTX 3090
(sm_86):

& 'C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\Launch-VsDevShell.ps1' `
  -Arch amd64 -HostArch amd64 -SkipAutomaticLocation

$cuda = 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4'
$env:CUDA_PATH = $cuda

cmake -S . -B build/windows-cuda-release -G Ninja `
  -DCMAKE_BUILD_TYPE=Release `
  -DENGINE_ENABLE_CUDA=ON `
  -DENGINE_BUILD_TESTS=ON `
  -DENGINE_BUILD_WARMBENCH=ON `
  -DCMAKE_CUDA_ARCHITECTURES=86 `
  "-DCMAKE_CUDA_COMPILER=$cuda\bin\nvcc.exe" `
  "-DCMAKE_CUDA_FLAGS=--allow-unsupported-compiler" `
  "-DOpenMP_CXX_FLAGS=/openmp:experimental"

cmake --build build/windows-cuda-release --config Release `
  --target qwen_decoder_packed_projection_test higgs_tts_warm_bench audiocpp_cli audiocpp_server -j 8

The detached 2e5c5e5 baseline used the same configure flags and a separate
build/windows-cuda-benchmark directory.

Exact standalone run

audio.cpp\build\windows-cuda-release\bin\audiocpp_cli.exe `
  --task tts `
  --family higgs_tts `
  --model models\higgs-audio-v3-tts-4b_Q8\higgs-audio-v3-tts-4b_Q8.gguf `
  --backend cuda `
  --voice-ref SAMPLES\EN_2.wav `
  --reference-text "If you actually care about security." `
  --text "The optimized Higgs runtime is ready for validation." `
  --temperature 1.0 `
  --top-p 0.95 `
  --top-k 50 `
  --seed 42 `
  --max-tokens 256 `
  --out outputs\higgs_v3_optimized_clone.wav `
  --log

Result: success, 24 kHz mono WAV generated.

Exact performance/parity run

Candidate command (run three times with labels latest_dev_run1 through
latest_dev_run3):

tests/higgs_tts/run_cuda_performance.ps1 `
  -Model ../models/higgs-audio-v3-tts-4b_Q8/higgs-audio-v3-tts-4b_Q8.gguf `
  -Label latest_dev_run1 `
  -Warmup 1 `
  -Iterations 1

Baseline command used the detached historical executable, the same model and
case file, and labels exact_2e5_run1 through exact_2e5_run3:

_bench_higgs_2e5c5e5\build\windows-cuda-benchmark\bin\higgs_tts_warm_bench.exe `
  --model models\higgs-audio-v3-tts-4b_Q8\higgs-audio-v3-tts-4b_Q8.gguf `
  --backend cuda `
  --device 0 `
  --threads 8 `
  --warmup 1 `
  --iterations 1 `
  --request-sequence-file _dev_higgs_tts\tests\higgs_tts\higgs_tts_cuda_perf_cases.json `
  --output-dir _dev_higgs_tts\tests\higgs_tts\results\exact_2e5_run1\audio `
  --timing-file _dev_higgs_tts\tests\higgs_tts\results\exact_2e5_run1\timing.log

Runs were alternated baseline/candidate. Model loading is excluded; each fresh
process performs one warmup request followed by five measured requests. Values
below are per-request mean +/- sample standard deviation across three fresh
processes. Speedup is baseline RTF / candidate RTF, so small output-length
differences do not inflate the result.

Request	Frames baseline / candidate	Baseline ms	Candidate ms	Baseline RTF	Candidate RTF	RTF speedup	WAV cosine	80-band log-mel cosine
0	139200 / 140160	1184.7 +/- 4.4	975.2 +/- 3.7	0.2043 +/- 0.0008	0.1670 +/- 0.0006	1.223x +/- 0.008	0.1294	0.7787
1	134400 / 132480	1184.4 +/- 2.8	925.7 +/- 8.1	0.2115 +/- 0.0005	0.1677 +/- 0.0015	1.261x +/- 0.008	0.2974	0.8822
2	177600 / 176640	1668.1 +/- 4.8	1512.3 +/- 20.4	0.2254 +/- 0.0006	0.2055 +/- 0.0028	1.097x +/- 0.017	0.0435	0.8133
3	65280 / 66240	622.5 +/- 2.8	583.5 +/- 11.9	0.2289 +/- 0.0010	0.2114 +/- 0.0043	1.083x +/- 0.027	0.3628	0.9054
4	132480 / 134400	1136.6 +/- 8.2	934.1 +/- 2.8	0.2059 +/- 0.0015	0.1668 +/- 0.0005	1.234x +/- 0.011	0.1167	0.8341

All five requests improved. Candidate RTF is 0.1668 to 0.2114 on this
workload.

Parity interpretation

These are fixed-seed sampled runs at temperature 1.0, top-p 0.95, top-k
50, matching the non-zero-temperature reference behavior. Greedy mode was
not substituted because it is not the upstream Higgs generation setting.

The optimized default stores KV in F16, so small logit changes can change a
sampled token. Every request differs by only one or two 960-sample codec frames,
but the waveforms are not expected to be bit-identical after the sampled path
diverges. The table therefore reports the drift and similarity rather than
hiding it behind aggregate numbers. -RequireSameFrames is available for
comparisons that are expected to be numerically identical.

This PR compares the optimized C++ path directly with the exact original C++
implementation. A new Python-vs-C++ numerical parity claim is not made for this
Q8/F16 optimization run: the available Python reference uses native weights,
so quantization and sampled-token drift would confound that comparison.

Memory notes

One isolated five-request process per revision, sampled every 100 ms:

Revision	Peak process RSS	GPU memory before	Peak total GPU memory
exact `2e5c5e5`	5583.9 MiB	0 MiB	10258 MiB
candidate	7502.5 MiB	0 MiB	10116 MiB

Windows WDDM did not expose reliable per-process VRAM, so the GPU values are
isolated total-device readings. The candidate reduced observed peak GPU memory
by about 142 MiB. Host RSS is higher; retained reference state and differences
between the historical and current framework are included in that figure.

Generated artifacts

Local, intentionally ignored by Git:

tests/higgs_tts/results/exact_2e5_run{1,2,3}/
tests/higgs_tts/results/latest_dev_run{1,2,3}/
tests/higgs_tts/results/runner_smoke/
tests/higgs_tts/results/cli_smoke/higgs_clone.wav

Each benchmark directory contains:

audio/audio_0.wav ... audio/audio_4.wav
console.log
timing.log
comparison.json              # candidate directories

Tests

Passed:

ctest --test-dir build/windows-cuda-release -C Release `
  -R qwen_decoder_packed_projection_test --output-on-failure

1/1 qwen_decoder_packed_projection_test: passed (0.07 s)

The test covers packed-vs-separate projection values, suffix causal masks,
single and batched F16 KV writes, one grouped FlashAttention op, two direct KV
updates, fused GLU, no repeated KV-head materialization, and the
ROPE -> VIEW -> SET_ROWS graph sequence.

Also passed:

current CLI clone smoke run and WAV generation;
three fresh-process x five-request candidate runs;
three matching historical baseline runs;
comparison helper syntax and end-to-end output;
git diff --check.

The all-target build was attempted. It currently stops in three Moss parity
test targets (codec_encode_parity, codec_dequant_parity, and
codec_decode_parity) because those tests still call pre-refactor path-based
constructors while current origin/dev requires TensorSource. Those files are
unchanged by this PR; the requested Higgs, CLI, and server targets build.

Backend coverage

Tested: CUDA 12.4, RTX 3090, compute capability 8.6, Windows 11.
Built: audiocpp_cli, audiocpp_server, focused unit test, warmbench.
Not measured in this pass: CPU, Vulkan, Metal.

Known limitations

The retained reference-prefix fast path helps repeated cloning requests with
the same reference. A changed reference correctly rebuilds the prefix.
F16 KV is the optimized default and can cause stochastic sampled-token drift,
as reported above.
The packed weights and retained reference state can increase host resident
memory even though observed GPU memory decreased.
Performance numbers are single-request latency on one RTX 3090; they are not
multi-client throughput measurements.

- pack QKV and gate/up projections and use fused SwiGLU in the shared Qwen decoder path - use grouped FlashAttention, F16 bucketed KV caches, direct set-rows updates, and CUDA-friendly RoPE/view/set-rows graphs - retain and reuse cloned-reference KV prefixes so repeated server requests prefill only their text suffix - add graph and F16 KV correctness coverage plus fixed-seed request-level CUDA benchmark/parity tooling - document the optimized Higgs runtime controls and reproducible validation flow

0xShug0 · 2026-07-19T14:04:48Z

@mirek190 Awesome optimization! I'm gonna merge it into dev directly. Sometimes I wonder whether I’ve been too strict about parity and missed out on significant performance gains. Maybe the best approach is to offer two modes: parity-first and performance-first. Users who are particularly sensitive to quality, like in issue #67, could use parity mode, while others could prioritize performance. Honestly, with TTS, both 90% and 99% parity can sound perfectly valid to me.

mirek190 · 2026-07-19T14:14:28Z

I'm glad you like it

0xShug0 · 2026-07-19T14:57:01Z

@mirek190 Is that possible to achve the 2x improvement as you mentioned in #63 (comment)? Does it require bypassing the framework modules/templates, modifying GGML, or using any other special customizations?

mirek190 · 2026-07-19T16:08:06Z

@mirek190 Is that possible to achve the 2x improvement as you mentioned in #63 (comment)? Does it require bypassing the framework modules/templates, modifying GGML, or using any other special customizations?

Actually that was a mistake ;) sorry
The llama.cpp 394 ms result was inaccurate: its phase timers measure asynchronous CUDA submissions without waiting for completion...

0xShug0 · 2026-07-19T16:21:22Z

Ah I see. Your optimization already provides a significant performance boost. I will test and try to merge OutieTTS after the world cup final, and then foucs on Higgs TTS.

Linux + RTX 5090

req	tm before	tm after	RTF before	RTF after	speedup	AR ms/step before	AR ms/step after
0	767.200 ms	546.657 ms	0.1360	0.1020	1.334x	4.6988	3.5714
1	811.752 ms	571.255 ms	0.1450	0.0999	1.451x	4.7222	3.5767
2	1069.092 ms	879.540 ms	0.1350	0.1170	1.154x	4.7435	3.5936
3	392.172 ms	345.377 ms	0.1486	0.1233	1.204x	4.7351	3.6088
4	724.793 ms	577.782 ms	0.1362	0.0996	1.368x	4.7415	3.5714

mirek190 · 2026-07-19T16:46:25Z

With your RTX 5090 is even better .. wow

mirek190 marked this pull request as ready for review July 19, 2026 11:47

0xShug0 merged commit 8007760 into 0xShug0:dev Jul 19, 2026
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimize Higgs Audio v3 inference#79

Optimize Higgs Audio v3 inference#79
0xShug0 merged 1 commit into
0xShug0:devfrom
mirek190:agent/higgs-v3-cuda-performance

mirek190 commented Jul 19, 2026 •

edited

Loading

Uh oh!

0xShug0 commented Jul 19, 2026 •

edited

Loading

Uh oh!

Uh oh!

mirek190 commented Jul 19, 2026

Uh oh!

0xShug0 commented Jul 19, 2026

Uh oh!

mirek190 commented Jul 19, 2026

Uh oh!

0xShug0 commented Jul 19, 2026 •

edited

Loading

Uh oh!

mirek190 commented Jul 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mirek190 commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Base and comparison point

Model setup and tested paths

Exact candidate build commands

Exact standalone run

Exact performance/parity run

Parity interpretation

Memory notes

Generated artifacts

Tests

Backend coverage

Known limitations

Uh oh!

0xShug0 commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

mirek190 commented Jul 19, 2026

Uh oh!

0xShug0 commented Jul 19, 2026

Uh oh!

mirek190 commented Jul 19, 2026

Uh oh!

0xShug0 commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mirek190 commented Jul 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mirek190 commented Jul 19, 2026 •

edited

Loading

0xShug0 commented Jul 19, 2026 •

edited

Loading

0xShug0 commented Jul 19, 2026 •

edited

Loading