fix(quant): avoid F8E4M3 compute dtype in FP8 dequantization

[glaziermag]

Fix: FP8 Unquant Linear Mismatch Crash & Dequantization Truncation

This PR fixes two fatal FP8 runtime bugs:

1. UnquantLinear fallback crash: if an FP8 weight tensor reaches the unquantized path without a corresponding FP8 scale tensor, the old path attempted an unsupported cast and failed with unexpected dtype, expected: BF16, got: F8E4M3. The patched path now rejects this malformed state with an explicit error instead of continuing into the dtype mismatch.

2. Dequantization truncation: BlockwiseFP8Linear, VectorFP8Linear, and distributed MoE expert paths used vb.dtype() as the dequant target. For FP8 models, vb.dtype() can be F8E4M3 (the storage type), so expanded parameters could be coerced back to 8-bit floats. The patch centralizes FP8 dequant dtype selection and keeps dequantized tensors in a compute dtype.

Changes

Centralized helper (lib.rs)

• fp8_dequant_dtype(vb_dtype, bias): determines the output dtype for FP8 dequantization. The invariant is: never dequantize FP8 back into F8E4M3.
• Added durable unit coverage for the invariant on branch head 698933761.

Scale name resolution (distributed/layers.rs)

• fp8_scale_name(vb, base): resolves weight_scale_inv vs weight_scale dynamically and errors if neither exists.
• Applied to distributed MoE paths: PackedExperts and FusedExperts, stacked and per-expert forms, covering gate_proj, up_proj, down_proj, and gate_up_proj scale lookups.

Dequant dtype fixes

• blockwise_fp8/mod.rs: uses fp8_dequant_dtype instead of raw vb.dtype().
• vector_fp8/mod.rs: uses fp8_dequant_dtype instead of raw vb.dtype().
• distributed/layers.rs: blockwise_fp8_moe() calls use a compute dtype rather than FP8 storage dtype.

Missing-scale fallback (unquantized/mod.rs)

• UnquantLinear now explicitly errors on FP8 weights without scale tensors. This is intentionally a clean rejection, not an attempted dequantization path, because the scale tensor needed to recover the values is absent.

Empirical Execution Baseline (GCP g2-standard-32 L4 GPU)

Validated on a representative dynamic FP8 model (CalamitousFelicitousness/Qwen2.5-1.5B-Instruct-fp8-dynamic).

Before (Master Branch)

echo "What is 2+2? Only answer with the number 4." | ./target/release/mistralrs run text -m CalamitousFelicitousness/Qwen2.5-1.5B-Instruct-fp8-dynamic -a qwen2 --max-seq-len 4096

2026-04-27T21:37:48.499309Z ERROR mistralrs_core::engine: step - Model failed with error: unexpected dtype, expected: BF16, got: F8E4M3
2026-04-27T21:37:48.509262Z ERROR mistralrs_core::engine: step - Model failed with error: unexpected dtype, expected: BF16, got: F8E4M3
2026-04-27T21:37:48.522146Z ERROR mistralrs_core::engine: step - Model failed with error: unexpected dtype, expected: BF16, got: F8E4M3

After (Patched Branch)

echo "What is 2+2? Only answer with the number 4." | ./target/release/mistralrs run text -m CalamitousFelicitousness/Qwen2.5-1.5B-Instruct-fp8-dynamic -a qwen2 --max-seq-len 1024 --paged-attn off

4

Stats:
Time to first token: 0.88s
Prompt: 44 tokens, 50.11 T/s
Decode: 2 tokens, 95.24 T/s
Prefix cache: 0 hits / 1 turns
Sampling: temp=0.7, top_k=20, top_p=0.8, min_p=off, rep_pen=1.1

Additional logic-vet validation (698933761)

cargo test -p mistralrs-quant fp8_dequant_dtype_never_uses_fp8_storage_dtype --lib
cargo fmt --all -- --check
git diff --check
cargo clippy --workspace --tests --examples -- -D warnings

All passed locally.

Runtime validation used Qwen2.5-1.5B-Instruct-fp8-dynamic as a representative dynamic FP8/Qwen path. The exact original 27B repro (Qwen/Qwen3.5-27B-FP8) still has not been run and likely requires an A100/H100-class GPU with sufficient VRAM.

[github-actions]

Code Metrics Report

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                  5          305          210           52           43
 CSS                       2         1181         1036           34          111
 CUDA                     59        17706        13869         1637         2200
 Dockerfile                1           39           22            8            9
 HTML                      2          235          197           14           24
 JavaScript               16         3580         2702          486          392
 Jinja2                    7          694          656            5           33
 JSON                     21          409          406            0            3
 Makefile                  1            6            5            0            1
 Metal Shading Lan|       31        11647         9007         1064         1576
 PowerShell                1          300          227           30           43
 Python                  125         8316         6808          412         1096
 Shell                     2          485          329           95           61
 Plain Text                3         3723            0         2413         1310
 TOML                     27         1290         1124           35          131
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         3          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                105        11197            0         8067         3130
 |- BASH                  72          934          691          149           94
 |- Dockerfile             1            1            1            0            0
 |- JSON                  20          719          719            0            0
 |- PowerShell             3            3            3            0            0
 |- Python                23         1038          862           60          116
 |- Rust                  51         2048         1718           54          276
 |- TOML                   6          207          164            0           43
 |- YAML                   2            9            8            1            0
 (Total)                            16156         4166         8331         3659
─────────────────────────────────────────────────────────────────────────────────
 Rust                    547       236072       207590         6565        21917
 |- Markdown             361         8962          452         7385         1125
 (Total)                           245034       208042        13950        23042
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                   961       311435       249055        28614        33766
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[glaziermag]

Housekeeping note: This branch currently bundles the CI fix from #2115 (.typos.toml, openapi_doc.rs, distributed/layers.rs). Once #2115 is merged, this branch will need a rebase onto updated master to drop the duplicate CI fix commit and resolve the resulting conflicts.

[glaziermag]

Closing this rather than continuing to carry an A100/H100-only exact validation caveat. The branch has representative L4 FP8 validation and unit coverage, but the exact original 27B A100/H100 repro was not run.