Fit constant tensor descriptor contexts to host memory#73
Merged
Conversation
ConstantTensorCache contexts are created with no_alloc and only ever hold tensor descriptors (~400 B each), but ggml_init() mallocs mem_size regardless of no_alloc -- it calls ggml_aligned_malloc whenever mem_buffer is null. What that reservation costs is entirely OS-dependent, which is why the current defaults look free on one platform and break another: - Linux never faults in the untouched pages. Measured on qwen3-tts-1.7B, the 4G/1.5G/1.5G talker defaults and a 256M/128M/128M build both peak at ~2.05 GB RSS; they differ only in address space (VSZ ~69 GB). - Windows charges the whole reservation against the commit limit up front, so the same defaults can fail outright on a machine with little RAM and a small pagefile. Shrinking the defaults would cost capable hosts their headroom for no gain, so fit the request instead: a context may take up to a quarter of available host memory, with a 64 MiB floor that still holds ~160k descriptors -- far past what any model here builds. Hosts that can afford the full reservation are unaffected, and the fitting is logged under --log rather than silently applied. This sits in ConstantTensorCache, so all model families using it are covered, not just qwen3_tts. Host memory comes from MemAvailable on Linux and GlobalMemoryStatusEx on Windows, and is reported unknown elsewhere (macOS has no portable equivalent), in which case the request is honoured as before. MemFree / sysconf(_SC_AVPHYS_PAGES) is deliberately not used: it excludes reclaimable page cache and so reads near zero on any machine with a warm cache -- on the test box it reported 0.99 GB where MemAvailable reported 4.24 GB, which would have fitted contexts down under no memory pressure at all. This replaces an earlier fix that shrank the talker defaults directly on Windows, where the up-front commit charge was the original failure.
Owner
|
Thank you @patrickjchen! Merged |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ConstantTensorCache contexts are created with no_alloc and only ever hold tensor descriptors (~400 B each), but ggml_init() mallocs mem_size regardless of no_alloc -- it calls ggml_aligned_malloc whenever mem_buffer is null. What that reservation costs is entirely OS-dependent, which is why the current defaults look free on one platform and break another:
Shrinking the defaults would cost capable hosts their headroom for no gain, so fit the request instead: a context may take up to a quarter of available host memory, with a 64 MiB floor that still holds ~160k descriptors -- far past what any model here builds. Hosts that can afford the full reservation are unaffected, and the fitting is logged under --log rather than silently applied.
This sits in ConstantTensorCache, so all model families using it are covered, not just qwen3_tts.
Host memory comes from MemAvailable on Linux and GlobalMemoryStatusEx on Windows, and is reported unknown elsewhere (macOS has no portable equivalent), in which case the request is honoured as before. MemFree / sysconf(_SC_AVPHYS_PAGES) is deliberately not used: it excludes reclaimable page cache and so reads near zero on any machine with a warm cache -- on the test box it reported 0.99 GB where MemAvailable reported 4.24 GB, which would have fitted contexts down under no memory pressure at all.
This replaces an earlier fix that shrank the talker defaults directly on Windows, where the up-front commit charge was the original failure.