[Bug]: Pre-quantized MXFP8 checkpoint (config.json quant_method: "mxfp8") fails to load — routed to OnlineQuantizationConfig.from_config() which raises NotImplementedError

[janbernloehr]

Your current environment

The output of python collect_env.py

collect_env.py was not run inside the CI artifact sandbox. Closest captured environment:
- vLLM version: 0.22.1rc1.dev491+g1033ffac2 (public image vllm/vllm-openai:nightly)
- Platform/OS: linux/amd64, Python 3.12
- GPU: reproduced on NVIDIA H100 (config-time failure; GPU/model-arch independent)
- The failure occurs at engine-config creation, before any device/model is touched,
  so it reproduces identically regardless of GPU type or count.

🐛 Describe the bug

Serving a pre-quantized MXFP8 checkpoint whose config.json declares quantization_config.quant_method: "mxfp8" crashes at engine-config creation with NotImplementedError. The new online-quantization frontend registers "mxfp8" as an online-quant shorthand, so vLLM's auto-detect maps the checkpoint's quant_method to OnlineQuantizationConfig, whose from_config() is hard-coded to raise. There is no offline checkpoint path for this quant_method: "mxfp8" format.

Example public model: MiniMaxAI/MiniMax-M3-MXFP8 (config.json quantization_config: quant_method: "mxfp8", activation_scheme: "dynamic", weight_block_size: [1, 32]).

Minimal reproducer (config-time, no GPU/weights needed):

from vllm.model_executor.layers.quantization import (
    get_quantization_config,
    QUANTIZATION_METHODS,
)

# Verbatim from a pre-quantized MXFP8 checkpoint's config.json -> quantization_config
hf_quant_config = dict(
    quant_method="mxfp8",
    activation_scheme="dynamic",
    weight_block_size=[1, 32],
    ignored_layers=["lm_head", "model.embed_tokens"],
)

print("mxfp8 in QUANTIZATION_METHODS:", "mxfp8" in QUANTIZATION_METHODS)  # True
quant_cls = get_quantization_config(hf_quant_config["quant_method"])
print("resolved to:", quant_cls.__name__)                                # OnlineQuantizationConfig
quant_cls.from_config(hf_quant_config)                                    # raises NotImplementedError

Equivalent full command (also fails the same way at startup):

vllm serve MiniMaxAI/MiniMax-M3-MXFP8 --tensor-parallel-size 8 \
  --enable-expert-parallel --trust-remote-code

Full traceback:

File ".../vllm/config/vllm.py", line 627, in _get_quantization_config
    quant_config = get_quant_config(model_config, load_config)
File ".../vllm/model_executor/model_loader/weight_utils.py", line 292, in get_quant_config
    return quant_cls.from_config(hf_quant_config)
File ".../vllm/model_executor/layers/quantization/online/base.py", line 112, in from_config
    raise NotImplementedError(
NotImplementedError: OnlineQuantizationConfig does not support loading from a checkpoint config.
Use quantization_config or quantization='fp8_per_tensor'/'fp8_per_block' instead.

Expected behavior: A checkpoint that declares quant_method: "mxfp8" in config.json should load via an offline MXFP8 checkpoint path (pre-quantized weights + scales), not be routed to the online-quant frontend. The online shorthand "mxfp8" (online quantization of bf16/fp16 weights) and the checkpoint-loading "mxfp8" are colliding on the same method string.

Actual behavior: get_quantization_config("mxfp8") returns OnlineQuantizationConfig because "mxfp8" is registered as an online shorthand; OnlineQuantizationConfig.from_config() unconditionally raises NotImplementedError, so the server never starts. The suggested fp8_per_tensor/fp8_per_block alternatives are not valid substitutes for an MXFP8 checkpoint.

Suggested fix directions (for maintainers):

1. Disambiguate checkpoint auto-detect from the online shorthand: when quant_method comes from a checkpoint config.json, route MXFP8 to a checkpoint-capable config rather than OnlineQuantizationConfig; or
2. Provide an offline MXFP8 checkpoint loader for the quant_method: "mxfp8" + weight_block_size format (analogous to ModelOptMxFp8Config but for this checkpoint schema); or
3. At minimum, have OnlineQuantizationConfig.from_config() raise a clearer, actionable error pointing to the correct offline path for pre-quantized MXFP8 checkpoints.

Before submitting a new issue...

• Searched existing/past issues (only the closed design RFC [RFC]: online quantization user facing API #32412 is related; no bug tracks this checkpoint-loading collision).

This issue was drafted with assistance from the opus AI model.

[cx2009]

Is this still blocking your workflow or already mitigated?

[janbernloehr]

The image from June, 13 still failed - waiting for a more recent run.

[janbernloehr]

fixed by #45381