- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work provides a systematic evaluation of key factors in MXFP4 training, including scaling factor computation, stochastic rounding (SR), and Hadamard transformation (HT). It also adopts Macro Block Scaling (MBS), which is proposed in previous work. MBS maintains higher-precision scale at a coarser granularity beyond the MXFP4 micro-block. The best among the evaluated MXFP4 training recipes reaches performance comparable to NVFP4 training.
Critical strengths
-
The paper provides systematic training results at a large scale. The experiment is done on training LLMs of large size (up to 7B) with at most 640B tokens.
-
The presentation is clear, and the paper is easy to follow.
Critical weaknesses
-
My main concern is that the MXFP4-OCP baseline is insufficiently justified, and the backward quantization implementation in the evaluated methods is unclear, which undermines the claimed contributions. In particular, an important issue about the backward pass is unclear in this paper: for the quantization of activation and weight in the backward pass, is the quantizer input aligned with the quantized-dequantized tensor from the forward pass, ? Or just quantize the high-precision tensors in the backward pass?
This alignment of forward and backward tensors () has been established as an important issue in MXFP4 training, and Quartet [1] and TetraJet [2] are MXFP4 training recipes that ensure this alignment. The comparison against a forward-backward-aligned baseline is valuable for identifying the benefits of MBS, without requiring the full set of their additional proposed techniques. -
The methodological novelty is limited. OAS and MBS were already introduced in [3], while SR and HT are established FP4 training techniques. The authors should clarify what is new beyond composing these components, and clarify the additional insights beyond the prior work.
[1] Quartet: Native FP4 Training Can Be Optimal for Large Language Models, NeurIPS 2025
[2] Oscillation-Reduced MXFP4 Training for Vision Transformers, ICML 2025
[3] Unveiling the Potential of Quantization with MXFP4: Strategies for Quantization Error Reduction, ICML 2026
Questions for authors
Please clarify Weaknesses 1 and 2. These are my main concerns affecting my rating. I have no additional questions currently.
5 comments
Thanks for the additional discussion and results. My concern is only partly resolved, so I can only raise my rating from 1 to 2.
I appreciate the technical contributions to the community, but my concerns about the novelty remain. Because the core methods are from previous work, this work is a good technical report with sufficient ablation studies, but not a conference paper that brings sufficient new insights and ideas.
Thank you for raising your score and for acknowledging the technical contributions to the community. We respect your assessment and would like to clarify two insights that we believe go beyond prior work.
Prior work (Chhugani et al., 2026) introduced OAS and MBS to reduce quantization error in inference, where lower error directly translates into better quality. In pretraining, we find that this no longer holds. As shown in Figure 3(a), MBS128 lowers quantization error but increases the final loss gap relative to BF16 under round-to-nearest gradient quantization (from 6.88% to 8.54%), because it produces a larger bias in the quantized gradients that accumulates over training. Only when combined with gradient stochastic rounding does MBS128 reduce the gap (from 6.47% to 5.12%), and Appendix E analyzes how stochastic rounding removes this bias. This interaction between scale precision and gradient bias does not arise in inference, so it was not observed in prior work.
Our continued-training experiment also decomposes MXFP4's gap to NVFP4 under the same setup. Smaller blocks and OAS recover about half of the 42.42% additional tokens needed to match NVFP4 (down to 19.62%), and the remaining gap closes only when a higher-precision macro-block scale is added (down to 1.33%). This identifies scale precision as the missing piece for MXFP4 in pretraining, and shows that it can be recovered at a coarse granularity while keeping hardware-friendly power-of-two block scales.
Beyond the technical contributions you have mentioned, our work offers insights that, to the best of our knowledge, have not been reported in prior work: the counterintuitive effect of MBS under round-to-nearest gradient quantization, and the decomposition of MXFP4's pretraining gap to NVFP4.
Thank you for your comments and for recognizing our large-scale training results. Below, we present additional experiments on forward-backward alignment and clarify the contributions and practical value of our work.
W1: Forward-backward alignment
A1: Thank you for raising this point. In the paper, backward quantization in all evaluated recipes takes the high-precision X and W as inputs, and we have clarified this in Section 4.2 of the revised paper. Following your suggestion, we implemented forward-backward alignment as in Quartet and TetraJet, where backward quantization takes the forward-quantized X̂ and Ŵ as inputs. The table reports the final loss of 300M models trained for 30B tokens, with and without alignment.
We find that alignment benefits FP4 training, reducing the loss of all three configurations by 0.43%-0.72%. This benefit does not change the conclusions of the paper: with alignment applied to both, MXFP4-MARS still reduces the loss by 1.99% relative to MXFP4-OCP, compared with 2.05% without alignment. Alignment also benefits MXFP4 (0.65%-0.72%) more than NVFP4 (0.43%), narrowing the loss gap between MXFP4-MARS and NVFP4 from 0.93% to 0.70%. Overall, our recipe remains effective with alignment: MXFP4-MARS closes about 75% of the loss gap between MXFP4-OCP and NVFP4 (from 2.75% to 0.70%), and the two are complementary.
W2: Contributions beyond existing techniques
A2: We agree that OAS, MBS, stochastic rounding, and Hadamard rotation build on prior work. Beyond composing them, our work provides the following insights for pretraining:
Scale precision as the missing piece. To our knowledge, this is the first work to apply OAS/MBS to both the forward and backward passes of LLM pretraining. We show that smaller blocks and better power-of-two scale selection narrow the gap to NVFP4 but cannot close it, while adding a higher-precision macro-block scale with MBS closes nearly all of the remaining gap.
Effectiveness of the combination in training. Our ablations show that the components are effective only under specific conditions: MBS helps only with gradient stochastic rounding, Hadamard rotation helps only for weight gradients, and 2D scaling degrades both formats. Building an effective recipe therefore requires understanding these interactions, and our ablations also rule out ineffective configurations that the community would otherwise need costly pretraining runs to test.
MXFP4-NVFP4 parity at scale. To our knowledge, this is also the first work to show MXFP4 training quality comparable to NVFP4 at scale, with 7B models trained on 640B tokens.