- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes a dedicated trellis-based weight quantization method tailored for MXFP4 checkpoints. A specific form of grid (MagStrat with exponent reuse) is chosen that can be mapped exactly to the MXFP4 format. The optimal weight encoding is found via the Hessian-aware Viterbi algorithm. The efficacy of the proposed method is validated on the gpt-oss-20b model on several tasks from the lm-eval harness.
Critical strengths
- The paper acknowledges the problem of quantization of already quantized models and the need to develop a dedicated method and weight representation that conforms with the base model precision (FP4 or some other low-precision format). The proposed HEPH grid addresses this problem by proposing a trellis encoding tailored for MXFP4 model quantization that, on the one hand, preserves accuracy as much as possible and, on the other hand, is efficient to decode and allows for large-batch serving via dequantizing the weights back to the original precision.
- HEPH provides efficient inference both in low-batch and high-batch serving scenarios.
Critical weaknesses
- The evaluation is limited—it involves only a single model (gpt-oss-20b) and a limited set of benchmarks (that can be considered outdated as of 2026). To fully appreciate the efficacy of the method, it would be great to consider more models published in MXFP4 precision (DeepSeek-V4, Kimi-K2 / 3). In case it is challenging due to compute constraints, one can consider public MXFP4 quants for Qwen-3.x models released on Hugging Face and measure performance relative to them. In addition, it would be insightful to measure the quality on reasoning benchmarks (GPQA-Diamond, AIME, MATH-500, LiveCodeBench) or agentic benchmarks, as they are more representative measures of the model capabilities.
- The claim that Hadamard transforms incur significant overhead during inference is not accurate. One can fuse small rotations [1] into the dequantization + GEMM kernel without noticeable inference slowdown.
[1] Egiazarian, V., et al. (2025). Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization. arXiv:2509.23202. Google Scholar.
Questions for authors
- In the comparison with QTIP [2], does one project the QTIP-quantized weights onto the MXFP4 grid or perform BF16 quantization? In case it is mapped to the MXFP4 format, is the reason for HEPH providing higher accuracy due to the grid being better suited for the MXFP4 base checkpoint or due to the additional error from QTIP conversion to the MXFP4 format?
- Does the proposed method work for NVFP4-quantized models?
- The 3-bit QTIP models for Llama-3 (even 8B) are almost lossless compared to the BF16 checkpoint, whereas the results in this work show a significant accuracy decline for gpt-oss-20b. Do you have an intuition about why this could be the case?
[2] Tseng, A., Sun, Q., Hou, D., & De Sa, C. (2024). QTIP: Quantization with Trellises and Incoherence Processing. NeurIPS 2024. Google Scholar.
2 comments
Thank you for the response. My concerns and questions were mostly resolved. After reading the answers addressed to me and other reviewers I decide to raise the score.
Response to Reviewer hr3x: gpt-oss-120b and reasoning benchmarks, format vs. rotation cost, QTIP scoring, NVFP4
Thank you for the expert review, for recognizing the need for a dedicated representation that conforms to the base model's precision, and for the pointer to MR-GPTQ. Changes are marked in blue in the revised PDF.
W1 One model, older benchmarks. The revision adds
gpt-oss-120b(36 layers × 128 experts) and reasoning benchmarks you suggested (GPQA-Diamond, AIME 2024+2025), plus GSM8K. Ongpt-oss-120b, HEPH is within 0.8 points of the source on the five zero-shot tasks (78.37 vs 79.16; calibration text doubled because each of its 128 experts receives 4× fewer routed tokens), 96.20 vs 96.25 on GSM8K and 71.40 vs 70.52 on GPQA-Diamond. Ongpt-oss-20bit leads QTIP on GSM8K (+1.10 [0.25, 2.00]) and AIME 2024+2025 (+8.75 [3.75, 14.17]). DeepSeek-V4 releases its routed experts in exactly the same E2M1 × UE8M0/32 format. We measured the released DeepSeek-V4-Flash experts (atom entropy 3.77 bits, exponent-delta entropy 0.047 bpw, both matchinggpt-oss), so the codec and its 3.07-bpw rate carry over unchanged.W2 Hadamard overhead. We agree that small rotations can be fused into the quantization and GEMM kernels at little cost, as MR-GPTQ shows, and we removed the claims that attribute serving speed to avoiding the Hadamard. However, the obstacle HEPH removes is the format, not the arithmetic. MR-GPTQ quantizes a high-precision model into a rotated MXFP4 basis. For a model released only in MXFP4, a rotated reconstruction is no longer in the checkpoint's own E2M1 × UE8M0 set (Proposition 1), so it needs either its own kernels or a second rounding back onto the grid. We measured that second rounding for QTIP: it costs a further 1.5 points (77.72 → 76.20). HEPH reconstructs inside the checkpoint's grid. This is what lets it run on the unmodified vendor kernel in two modes (§4.3): expanded at load, at native speed, or decoded on demand from the compressed stream, which uses 20% less GPU memory and gives 2.7× the KV-cache capacity with 16 GiB of GPU memory, the minimum OpenAI states for
gpt-oss-20b. The related-work and introduction text now make this argument.Q1 How QTIP is scored. QTIP is evaluated with its own BF16 reconstruction (rotation undone), not projected onto MXFP4, which favors QTIP. So the 2.3-point gap is not conversion error. Projecting QTIP onto MXFP4, which it needs in order to run on the FP4 kernel, widens the gap to 3.8 points (Table 1).
Q2 NVFP4. Yes. The construction needs only E2M1 elements and a stored block scale: NVFP4 weights divided by their E4M3 block scale are exact atoms, and the scale bytes are reused and entropy-coded. The scale stream is larger (one scale per 16 weights), so the rate overhead is higher than for MXFP4.
Q3 Why 3-bit QTIP is near-lossless on Llama-3 but not here. The source is different. A BF16 Llama-3 checkpoint carries far more precision than its accuracy needs, so a 3-bit code mostly removes noise.
gpt-oss's experts were trained (quantization-aware) to live on the MXFP4 grid, and their E2M1 atoms carry 3.74 bits of empirical entropy per weight. Even lossless coding needs ≈3.78 bpw with the exponents, so any 3-bit code must discard about 0.7 bits of information per weight that training placed there. Rotating this discrete source into a continuous one discards its structure; HEPH keeps it, which we believe is why it degrades less than QTIP, most visibly on AIME (§4.2).