Harden PocketTTS FlowLM step allocation against OOM (follow-up to #55 decoder fix)#59
Draft
5uck1ess wants to merge 1 commit into
Draft
Harden PocketTTS FlowLM step allocation against OOM (follow-up to #55 decoder fix)#595uck1ess wants to merge 1 commit into
5uck1ess wants to merge 1 commit into
Conversation
Extends the 0xShug0#55 hardening from the Mimi decoder to FlowLMStepRuntime, the sibling pocket_tts runtime reached via PocketTTSSession::prepare. Under VRAM pressure its unguarded ggml_backend_alloc_ctx_tensors returns null and the following write_tensor calls trip GGML_ASSERT (ggml-backend.cpp:327), SIGABRT-ing the whole server instead of failing the one request. Guards all three allocation sites (params buffer, prompt graph, lazy step graph) with the same reserve/alloc-or-throw idiom as the Mimi decoder fix, and factors the destructor cleanup into an idempotent release_runtime() so a constructor throw does not leak the ggml context or partial buffers. Verified on an RTX 3090: at ~227 MiB free a synthesis that previously SIGABRT'd now returns a clean 500 and the server stays responsive.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft — follow-up to the Mimi decoder OOM hardening on
fix/pocket-tts-mimi-decoder-oom(this PR is based on that branch; retarget tomainonce it merges).While verifying that branch on a 3090 under real VRAM pressure (#55), I found the SIGABRT still reproduces at tighter memory — just in a sibling
pocket_ttsruntime the branch doesn't touch:FlowLMStepRuntime, reached viaPocketTTSSession::prepare. Itsggml_backend_alloc_ctx_tensors(and twoggml_gallocr_alloc_graphcalls) are unguarded, so a failed CUDA buffer alloc returns null and the followingwrite_tensor/graph use tripsGGML_ASSERT(buf != NULL)at ggml-backend.cpp:327.This applies the same treatment you used in the decoder to all three FlowLM allocation sites:
alloc_ctx_tensors) — null-check →runtime_errorggml_gallocr_reserve+alloc_graph-or-throwensure_step_graph_allocated) — same, freeing the partial gallocr so a retry doesn't leakBecause a throw from the constructor skips the destructor, the destructor cleanup is factored into an idempotent
release_runtime()that the failure paths call, so the ggml context and any partial buffers are freed rather than leaked.Verified on an RTX 3090 (A/B, squeezing free VRAM with a filler then synthesizing a cloned voice):
mainSIGABRTs on the ~3 GB Mimi reserve at ~1.1 GB free.mainat 1143 MiB free succeeds at 997 MiB free) — but still SIGABRTs in FlowLM at ~250 MiB free.500(FlowLM step parameter buffer allocation failed (out of memory)) and the server stays responsive — no abort.Draft because you may prefer to fold it into the decoder branch or adjust the messages/idiom. Happy to adapt.