perf: keep embedding table in FP16 to save ~1GB on 1.7b model - #107
Merged
Conversation
The embed_tokens.bin (594MB FP16) was eagerly converted to FP32 (1.2GB) at load time, but only one token's embedding (8KB) is looked up per decoder step. Store the raw FP16 bytes and convert on-the-fly during lookup. Reduces RSS from ~6GB to ~4.9GB for the 1.7b model.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
embed_tokens.bin(594MB FP16) was eagerly expanded to a 1.2GB FP32Vec<f32>at load time, but only a single token's embedding (~8KB) is looked up per decoder step.This keeps the raw FP16 bytes in memory and converts only the requested token on-the-fly in
lookup_embedding_fp16. The per-call cost is negligible (2048 FP16→FP32 conversions ≈ microseconds).Result: RSS drops from ~6GB to ~4.9GB for the 1.7b Qwen3-ASR model.