- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper analyzes the compounding effect of quantization-induced gradient norm contraction during backpropagation in quantized training. It describes grid geometry, measures quantization errors and analyzes interactions with the commonly used RHT-based normalization techniques. From all of it, it arrives at the conclusion that uniform-grid (INT4/E1M2) 4-bit formats are preferable for quantized training over the commonly-used non-uniform (FP4/E2M1) formats. As the result, the authors propose a fully-E1M2 computation scheme for fully-quantized (both forward and backward passes) training of LLMs. Experimental validation includes dense and MoE models of reasonable sizes for an academic paper.
Critical strengths
The paper is well written.
The core experimental validation is solid and supports the narrative.
The results agree with existing work.
Critical weaknesses
The paper misses important related works and generally lacks novelty.
Missing shrinkage bias related works
The paper crucially omits certain related work on shrinkage bias in backpropagation and any in-depth discussion of related work that explicitly solves it.
Quartet (NeurIPS 2025, Castro et al.) explicitly defined "projection magnitude misalignment" as norm contraction due to quantization and explicitly demonstrated how it compounded and grew during LLM backward pass. Quartet II (ICML 2026, Panferov et al.), a follow-up work, effectively addressed the same problem by explicitly correcting for the shrinkage via the EDEN scaling routing, showing how it fully removes the bias.
In the manuscript, the former related work is not mentioned at all while the latter is only briefly cited on the last page of the appendix.
RHT usage misrepresentation
The paper states that "E2M1 recipes typically restrict RHT to the weight-gradient path to avoid compounding geometric errors", and presents applying RHT to all three GEMM operations as something novel. This is incorrect as there exist numerous work that apply RHT to all backward pass GEMMs: (1) "Training LLMs with MXFP4" by Tseng et al. at AISTATS 2025, Quartet by Castro et al. at NeurIPS 2025, TetraJet-v2 by Chen et al. at ICML 2026, Quartet II by Panferov et al. at ICML 2026.
The "avoid compounding geometric errors" part is solved by using an unbiased quantization technique, such as stochastic rounding (SR) for most or MS-EDEN for Quartet II. Debiasing guarantees that no shrinkage happens. The same debiasing via SR is applied in authors own UFP4 on dY tensors, eliminating the shrinkage there and undermining the usefulness of using the E1M2 format for it. The interaction of SR with various grids is under-explored.
No baseline comparisons
The experimental comparisons included in the paper only include performance breakdowns and self-comparisons. Any comparisons with existing computation schemes, such as Quartet, FP4 All the Way, TetraJet-v2, Four Over Six or Quartet II are missing. As the result, it's hard to gauge how useful the improvements are in relation to alternative approaches and whether the setup is compatible with any of the orthogonal directions such as scale search (Four Over Six) or alternative ubiased quantization primitives (Quartet II).
Section 5.6 asks a question with a commonly-known answer
Fusing RHT with FP4 quantization has been an explicit point of numerous works (QuEST, "Training LLMs with MXFP4", Quartet, MR-GPTQ, Quartet II) on FP4 training and inference, some of which consider much deeper implications for micro-scaling formats such as max reduction pre-fusing and post-op re-rounding. The section does not even touch upon these subjects and has no substance over well-established methods.
Summary
While the paper presents a coherent narrative and reasonable experimental validation, the discussion does not fully reflect the current state of the field and tends to slightly overstate the work's novelty. Overall, the technical contributions appear incremental and, in my opinion, below the acceptance threshold for a top-tier ML conference.
Questions for authors
Questions
How do motivate the choice of using stochastic rounding for dY but nowhere else?
2 comments
Question 4
The paper misses important related works and generally lacks novelty.
As the reviewer observes, the individual components used in our work are established techniques. RHT and SR have been widely used in prior work, and neither E2M1 nor INT4/E1M2 is a newly introduced format.
Our motivation is that the range–resolution trade-off changes as quantization becomes more fine-grained. With per-tensor scaling, the data-element format must accommodate substantial magnitude variation within a tensor. With fine-grained block scaling, part of this responsibility shifts to the block scales. Combined with RHT-based outlier mitigation, this makes local resolution increasingly important and motivates revisiting the allocation of the limited four-bit budget.
An important observation motivating our work is that mainstream accelerators, including NVIDIA Blackwell/Rubin and AMD MI350X, have moved away from INT4 acceleration paths and concentrated their four-bit support on E2M1. We question whether this is the right direction. As low-precision training and inference continue to develop, new techniques will need to address issues associated with the E2M1 codebook, including norm attenuation and directional bias. The additional complexity of corrective operations can, in turn, limit the acceleration benefits of FP4.
We therefore encourage greater attention to fine-grained quantization with uniform formats such as E1M2/INT4. Combined with efficient RHT, we believe this direction offers the potential to raise the performance ceiling of future FP4 systems.
Hi, thank you very much for your valuable feedback. We would like to clarify the following points.
Question 1
At FP8 precision, reducing the quantization block size can lower quantization error and delay the deterioration of the loss gap relative to BF16. At FP4 precision, however, having only 15 distinct representable values makes individual rounding decisions particularly consequential for long-run training. Biased rounding can cause the loss gap relative to BF16 to grow continuously.
For values sampled from a uniform distribution, RTNE (round-to-nearest-even) can be unbiased, while bias in the quantized output can arise from the data distribution itself. SR addresses bias in rounding decisions, but introduces higher variance. Although this strategy preserves expectations, the additional noise makes SR less suitable for the forward pass.
Question 2
Our statement that “E2M1 recipes typically restrict RHT to the weight-gradient path to avoid compounding geometric errors” was not intended to rule out alternative configurations. For example, TetraJet-v2, Quartet II, and NVIDIA’s NVFP4 recipe all apply RHT to WGRAD, while their choices regarding DGRAD differ.
Most FP4 E2M1 training recipes do not introduce RHT in FPROP. Our paper provides an explanation for this observation: RHT can exacerbate Shrinkage Bias under E2M1, leading to degraded training quality.
MS-EDEN compensates for quantization-induced signal attenuation and, under its stated assumptions, produces unbiased vector estimates in expectation. We acknowledge that this guarantee concerns the vector expectation, rather than merely correcting its norm. Nevertheless, unbiasedness does not imply lossless reconstruction: individual quantized realizations can still exhibit magnitude and angular errors, and quantization error and variance remain relevant. The underlying grid can therefore still affect numerical quality, leaving room to investigate whether uniform formats offer a more favorable trade-off.
Through UFP4, we hope to draw attention to the additional correction mechanisms that may accompany continued development around E2M1, and to encourage the community and hardware vendors to explore uniform formats such as INT4 or E1M2 together.
Question 3
We organize techniques for reducing FP4 computation error into several design dimensions: matrix preprocessing, such as RHT and SmoothQuant; quantization methods, such as MS-EDEN and Four Over Six; data-element formats, such as INT4/E1M2 and E2M1; and techniques for preserving unbiasedness, such as SR and MS-EDEN.
Techniques across these dimensions can often be combined to improve numerical accuracy. In Appendix B, we ablate factors including RHT and SR and select the E2M1 configuration with the best observed training accuracy as the E2M1 reference throughout the paper.
Our paper aims to investigate the FP4 format dimension. Format choices can be combined with techniques such as MS-EDEN and SmoothQuant to obtain further accuracy improvements.
The additional experimental results below illustrate these relationships: Four Over Six is specific to E2M1, whereas MS-EDEN can improve both E2M1 and E1M2.