Skip to content

Fit constant tensor descriptor contexts to host memory#73

Merged
0xShug0 merged 1 commit into
0xShug0:mainfrom
patrickjchen:fix/constant-context-guard
Jul 18, 2026
Merged

Fit constant tensor descriptor contexts to host memory#73
0xShug0 merged 1 commit into
0xShug0:mainfrom
patrickjchen:fix/constant-context-guard

Conversation

@patrickjchen

Copy link
Copy Markdown
Contributor

ConstantTensorCache contexts are created with no_alloc and only ever hold tensor descriptors (~400 B each), but ggml_init() mallocs mem_size regardless of no_alloc -- it calls ggml_aligned_malloc whenever mem_buffer is null. What that reservation costs is entirely OS-dependent, which is why the current defaults look free on one platform and break another:

  • Linux never faults in the untouched pages. Measured on qwen3-tts-1.7B, the 4G/1.5G/1.5G talker defaults and a 256M/128M/128M build both peak at ~2.05 GB RSS; they differ only in address space (VSZ ~69 GB).
  • Windows charges the whole reservation against the commit limit up front, so the same defaults can fail outright on a machine with little RAM and a small pagefile.

Shrinking the defaults would cost capable hosts their headroom for no gain, so fit the request instead: a context may take up to a quarter of available host memory, with a 64 MiB floor that still holds ~160k descriptors -- far past what any model here builds. Hosts that can afford the full reservation are unaffected, and the fitting is logged under --log rather than silently applied.

This sits in ConstantTensorCache, so all model families using it are covered, not just qwen3_tts.

Host memory comes from MemAvailable on Linux and GlobalMemoryStatusEx on Windows, and is reported unknown elsewhere (macOS has no portable equivalent), in which case the request is honoured as before. MemFree / sysconf(_SC_AVPHYS_PAGES) is deliberately not used: it excludes reclaimable page cache and so reads near zero on any machine with a warm cache -- on the test box it reported 0.99 GB where MemAvailable reported 4.24 GB, which would have fitted contexts down under no memory pressure at all.

This replaces an earlier fix that shrank the talker defaults directly on Windows, where the up-front commit charge was the original failure.

ConstantTensorCache contexts are created with no_alloc and only ever hold tensor
descriptors (~400 B each), but ggml_init() mallocs mem_size regardless of no_alloc
-- it calls ggml_aligned_malloc whenever mem_buffer is null. What that reservation
costs is entirely OS-dependent, which is why the current defaults look free on one
platform and break another:

- Linux never faults in the untouched pages. Measured on qwen3-tts-1.7B, the
  4G/1.5G/1.5G talker defaults and a 256M/128M/128M build both peak at ~2.05 GB
  RSS; they differ only in address space (VSZ ~69 GB).
- Windows charges the whole reservation against the commit limit up front, so the
  same defaults can fail outright on a machine with little RAM and a small
  pagefile.

Shrinking the defaults would cost capable hosts their headroom for no gain, so fit
the request instead: a context may take up to a quarter of available host memory,
with a 64 MiB floor that still holds ~160k descriptors -- far past what any model
here builds. Hosts that can afford the full reservation are unaffected, and the
fitting is logged under --log rather than silently applied.

This sits in ConstantTensorCache, so all model families using it are covered, not
just qwen3_tts.

Host memory comes from MemAvailable on Linux and GlobalMemoryStatusEx on Windows,
and is reported unknown elsewhere (macOS has no portable equivalent), in which
case the request is honoured as before. MemFree / sysconf(_SC_AVPHYS_PAGES) is
deliberately not used: it excludes reclaimable page cache and so reads near zero
on any machine with a warm cache -- on the test box it reported 0.99 GB where
MemAvailable reported 4.24 GB, which would have fitted contexts down under no
memory pressure at all.

This replaces an earlier fix that shrank the talker defaults directly on Windows,
where the up-front commit charge was the original failure.
@0xShug0
0xShug0 merged commit 7554ee6 into 0xShug0:main Jul 18, 2026
4 checks passed
@0xShug0

0xShug0 commented Jul 18, 2026

Copy link
Copy Markdown
Owner

Thank you @patrickjchen! Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants