acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe

Abstract

FP4 training promises substantial reductions in memory and computation cost for LLM pretraining, yet current FP4 hardware paths and recipes, including NVIDIA Blackwell/Rubin-class systems and AMD MI350-series GPUs, remain centered on E2M1 data elements. In this study, we identify a fundamental limitation of that choice: non-uniform formats such as E2M1 inherently suffer from Shrinkage Bias, a systematic negative rounding error caused by the geometric asymmetry of their representable bins. We show that this bias accumulates multiplicatively across layers and is amplified by the Random Hadamard Transform (RHT), providing a unified explanation for the training instability observed in existing E2M1-based FP4 recipes. In contrast, uniform grids (E1M2/INT4) bypass this grid-geometry error and better convert the improved bucket utilization from RHT into higher quantization quality. Based on this finding, we propose UFP4, a uniform 4-bit training recipe that applies RHT to all three training GEMMs while restricting stochastic rounding to dY alone. On Dense 1.5B, MoE 7.9B, and MoE 124B long-run pretraining, UFP4 consistently achieves lower BF16-relative loss degradation than strong E2M1-based baselines, supported by scaling-law analysis and ablation studies. Our results suggest that future accelerators should support E1M2/INT4-style uniform 4-bit grids as first-class training primitives alongside E2M1.

Primary area:
foundation or frontier models, including LLMs
Keywords:
FP4, Full Quantization Training, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_DjrAOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes the compounding effect of quantization-induced gradient norm contraction during backpropagation in quantized training. It describes grid geometry, measures quantization errors and analyzes interactions with the commonly used RHT-based normalization techniques. From all of it, it arrives at the conclusion that uniform-grid (INT4/E1M2) 4-bit formats are preferable for quantized training over the commonly-used non-uniform (FP4/E2M1) formats. As the result, the authors propose a fully-E1M2 computation scheme for fully-quantized (both forward and backward passes) training of LLMs. Experimental validation includes dense and MoE models of reasonable sizes for an academic paper.

Critical strengths

The paper is well written.

The core experimental validation is solid and supports the narrative.

The results agree with existing work.

Critical weaknesses

The paper misses important related works and generally lacks novelty.

Missing shrinkage bias related works

The paper crucially omits certain related work on shrinkage bias in backpropagation and any in-depth discussion of related work that explicitly solves it.

Quartet (NeurIPS 2025, Castro et al.) explicitly defined "projection magnitude misalignment" as norm contraction due to quantization and explicitly demonstrated how it compounded and grew during LLM backward pass. Quartet II (ICML 2026, Panferov et al.), a follow-up work, effectively addressed the same problem by explicitly correcting for the shrinkage via the EDEN scaling routing, showing how it fully removes the bias.

In the manuscript, the former related work is not mentioned at all while the latter is only briefly cited on the last page of the appendix.

RHT usage misrepresentation

The paper states that "E2M1 recipes typically restrict RHT to the weight-gradient path to avoid compounding geometric errors", and presents applying RHT to all three GEMM operations as something novel. This is incorrect as  there exist numerous work that apply RHT to all backward pass GEMMs: (1) "Training LLMs with MXFP4" by Tseng et al. at AISTATS 2025, Quartet by Castro et al. at NeurIPS 2025, TetraJet-v2 by Chen et al. at ICML 2026, Quartet II by Panferov et al. at ICML 2026.

The "avoid compounding geometric errors" part is solved by using an unbiased quantization technique, such as stochastic rounding (SR) for most or MS-EDEN for Quartet II. Debiasing guarantees that no shrinkage happens. The same debiasing via SR is applied in authors own UFP4 on dY tensors, eliminating the shrinkage there and undermining the usefulness of using the E1M2 format for it. The interaction of SR with various grids is under-explored.

No baseline comparisons

The experimental comparisons included in the paper only include performance breakdowns and self-comparisons. Any comparisons with existing computation schemes, such as Quartet, FP4 All the Way, TetraJet-v2, Four Over Six or Quartet II are missing. As the result, it's hard to gauge how useful the improvements are in relation to alternative approaches and whether the setup is compatible with any of the orthogonal directions such as scale search (Four Over Six) or alternative ubiased quantization primitives (Quartet II).

Section 5.6 asks a question with a commonly-known answer

Fusing RHT with FP4 quantization has been an explicit point of numerous works (QuEST, "Training LLMs with MXFP4", Quartet, MR-GPTQ, Quartet II) on FP4 training and inference, some of which consider much deeper implications for micro-scaling formats such as max reduction pre-fusing and post-op re-rounding. The section does not even touch upon these subjects and has no substance over well-established methods.

Summary

While the paper presents a coherent narrative and reasonable experimental validation, the discussion does not fully reflect the current state of the field and tends to slightly overstate the work's novelty. Overall, the technical contributions appear incremental and, in my opinion, below the acceptance threshold for a top-tier ML conference.

Questions for authors

Questions

How do motivate the choice of using stochastic rounding for dY but nowhere else?

2 comments

Hi, thank you very much for your valuable feedback. We would like to clarify the following points.

Question 1

How do you motivate the choice of using stochastic rounding for dY but nowhere else?

At FP8 precision, reducing the quantization block size can lower quantization error and delay the deterioration of the loss gap relative to BF16. At FP4 precision, however, having only 15 distinct representable values makes individual rounding decisions particularly consequential for long-run training. Biased rounding can cause the loss gap relative to BF16 to grow continuously.

For values sampled from a uniform distribution, RTNE (round-to-nearest-even) can be unbiased, while bias in the quantized output can arise from the data distribution itself. SR addresses bias in rounding decisions, but introduces higher variance. Although this strategy preserves expectations, the additional noise makes SR less suitable for the forward pass.

Question 2

RHT usage misrepresentation

Our statement that “E2M1 recipes typically restrict RHT to the weight-gradient path to avoid compounding geometric errors” was not intended to rule out alternative configurations. For example, TetraJet-v2, Quartet II, and NVIDIA’s NVFP4 recipe all apply RHT to WGRAD, while their choices regarding DGRAD differ.

Most FP4 E2M1 training recipes do not introduce RHT in FPROP. Our paper provides an explanation for this observation: RHT can exacerbate Shrinkage Bias under E2M1, leading to degraded training quality.

MS-EDEN compensates for quantization-induced signal attenuation and, under its stated assumptions, produces unbiased vector estimates in expectation. We acknowledge that this guarantee concerns the vector expectation, rather than merely correcting its norm. Nevertheless, unbiasedness does not imply lossless reconstruction: individual quantized realizations can still exhibit magnitude and angular errors, and quantization error and variance remain relevant. The underlying grid can therefore still affect numerical quality, leaving room to investigate whether uniform formats offer a more favorable trade-off.

Through UFP4, we hope to draw attention to the additional correction mechanisms that may accompany continued development around E2M1, and to encourage the community and hardware vendors to explore uniform formats such as INT4 or E1M2 together.

Question 3

No baseline comparisons

We organize techniques for reducing FP4 computation error into several design dimensions: matrix preprocessing, such as RHT and SmoothQuant; quantization methods, such as MS-EDEN and Four Over Six; data-element formats, such as INT4/E1M2 and E2M1; and techniques for preserving unbiasedness, such as SR and MS-EDEN.

Techniques across these dimensions can often be combined to improve numerical accuracy. In Appendix B, we ablate factors including RHT and SR and select the E2M1 configuration with the best observed training accuracy as the E2M1 reference throughout the paper.

Our paper aims to investigate the FP4 format dimension. Format choices can be combined with techniques such as MS-EDEN and SmoothQuant to obtain further accuracy improvements.

The additional experimental results below illustrate these relationships: Four Over Six is specific to E2M1, whereas MS-EDEN can improve both E2M1 and E1M2.

FormatscaleRHT scopeMS-EDEN4over6Relative loss diff vs BF16
E2M11x16 blocksize + per-token fp32 scaleWGRAD❌❌+1.8706%
E2M11x16 blocksize + per-token fp32 scaleWGRAD❌✅+1.2708%
E2M11x16 blocksize + per-token fp32 scaleWGRAD✅✅+1.0686%
E1M21x16 blocksize + per-token fp32 scaleall-RHT❌❌+1.0199%
E1M21x16 blocksize + per-token fp32 scaleall-RHT✅❌+0.7448%

Question 4

The paper misses important related works and generally lacks novelty.

As the reviewer observes, the individual components used in our work are established techniques. RHT and SR have been widely used in prior work, and neither E2M1 nor INT4/E1M2 is a newly introduced format.

Our motivation is that the range–resolution trade-off changes as quantization becomes more fine-grained. With per-tensor scaling, the data-element format must accommodate substantial magnitude variation within a tensor. With fine-grained block scaling, part of this responsibility shifts to the block scales. Combined with RHT-based outlier mitigation, this makes local resolution increasingly important and motivates revisiting the allocation of the limited four-bit budget.

An important observation motivating our work is that mainstream accelerators, including NVIDIA Blackwell/Rubin and AMD MI350X, have moved away from INT4 acceleration paths and concentrated their four-bit support on E2M1. We question whether this is the right direction. As low-precision training and inference continue to develop, new techniques will need to address issues associated with the E2M1 codebook, including norm attenuation and directional bias. The additional complexity of corrective operations can, in turn, limit the acceleration benefits of FP4.

We therefore encourage greater attention to fine-grained quantization with uniform formats such as E1M2/INT4. Combined with efficient RHT, we believe this direction offers the potential to raise the performance ceiling of future FP4 systems.

Reviewer_kTXrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper shows that RTN to E2M1 for uniform distributed data within-a-cell is biased and that uniform grids (E1M2) do not suffer from this. This paper contains experiments that show that INT4 (E1M2 is basically equivalent to INT4) is better for training than FP4.

Critical strengths

The paper has a reasonable amount of empirical results that support the claims.

Critical weaknesses

This paper is pretty misguided. First, it should be immediately obvious that RTN is biased by construction. There are very few cases for which RTN is unbiased (namely, symmetric within-cell distributions for a uniform grid), so the entire shrinkage bias section of this paper is largely not useful. In fact, the fact that the authors used E1M2 with unbiased stochastic rounding in the backwards pass and attributed this to removing shrinkage bias, when SR has no bias to begin with. Second, the E1M2 format that the authors spend a large amount of time on is just the INT4 format, which is well known to have lower distortion on post-RHT distributions that are close to Gaussian (see WUSH, QuIP#, https://arxiv.org/abs/2605.13768, etc.). The empirical improvements the authors observe from E1M2 are almost certainly just because INT4 is better. This paper is also missing key citations to https://arxiv.org/abs/2502.20586, which was original paper that applied RHT's and SR to FP4 training, and https://arxiv.org/abs/2306.11987, which proposed using RHTs and INT4 for training.

Questions for authors

See above

1 comment

Hi, thank you very much for your valuable feedback. We would like to clarify the following points.

Question 1

First, it should be immediately obvious that RTN is biased by construction. There are very few cases for which RTN is unbiased (namely, symmetric within-cell distributions for a uniform grid), so the entire shrinkage bias section of this paper is largely not useful. In fact, the fact that the authors used E1M2 with unbiased stochastic rounding in the backwards pass and attributed this to removing shrinkage bias, when SR has no bias to begin with.

We agree that, for real training data and numerical distributions, RTN generally has difficulty preserving the original statistics without bias. However, we believe it is important to distinguish between bias arising from the data distribution and bias arising from the rounding method itself.

For values sampled from a uniform distribution, round-to-nearest-even can be unbiased; deviations from this unbiased behavior can arise from the data distribution. In E2M1, however, the codebook itself allows values within rounding intervals to incur different rounding errors. For example, rounding 4.9 to 4.0 produces an error magnitude of 0.9, whereas rounding 3.6 to 4.0 produces an error magnitude of 0.4. We find that Shrinkage Bias arising from this intrinsic property of the E2M1 codebook has a substantial impact on FP4 training quality. In contrast, E1M2/INT4 avoids this type of error, although E1M2 still cannot accommodate bias arising from the data distribution itself.

SR is therefore introduced in the backward pass to eliminate the impact of data-distribution bias on FP4 training. SR cannot fundamentally resolve the Shrinkage Bias described above, which stems from an intrinsic limitation of the E2M1 codebook.

Question 2

Second, the E1M2 format that the authors spend a large amount of time on is just the INT4 format, which is well known to have lower distortion on post-RHT distributions that are close to Gaussian (see WUSH, QuIP#, https://arxiv.org/abs/2605.13768, etc.). The empirical improvements the authors observe from E1M2 are almost certainly just because INT4 is better.

You are right. Our paper compares FP4 training quality between E2M1 and uniform E1M2/INT4-style grids. INT4 does indeed exhibit lower distortion on post-RHT distributions that are close to Gaussian. However, this does not establish that prior INT4 training approaches outperform E2M1. The fact that GPU vendors such as NVIDIA and AMD support only E2M1 computation in their latest accelerators, including Blackwell and Rubin, also reflects this point.

Our motivation is that the range–resolution trade-off changes as quantization becomes more fine-grained. With per-tensor scaling, the data-element format must accommodate substantial magnitude variation within a tensor. With fine-grained block scaling, part of this responsibility shifts to the block scales. Combined with RHT-based outlier mitigation, this makes local resolution increasingly important and motivates revisiting the allocation of the limited four-bit budget.

Our measurements illustrate this interaction. For the linear_fc2/fwd_x tensors, E2M1 achieves higher SQNR before RHT (21.90 versus 19.94 dB), while E1M2 achieves higher SQNR after RHT (23.19 versus 20.00 dB). This reversal shows that the preferred grid depends on preprocessing. Our investigation examines how this interaction translates to GEMM outputs, the appropriate scope of RHT, and long-run training quality across dense and MoE models.

Question 3

This paper is also missing key citations to https://arxiv.org/abs/2502.20586, which was original paper that applied RHT's and SR to FP4 training, and https://arxiv.org/abs/2306.11987, which proposed using RHTs and INT4 for training.

Thank you for pointing out these missing references. We will add them in the next revision of the paper.

Reviewer_ixzfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies and anaylzes shrinkage bias in E2M1, a widely used FP4 format, when combined with RHT. It then proposes UFP4, a uniform-grid FP4 recipe that mitigates this bias.

Critical strengths
  • The paper provides a clear explanation of shrinkage bias from the geometry of E2M1’s rounding bins. It connects this format-level bias to signal attenuation across layers and explains why RHT can exacerbate the problem.
  • The paper validates its analysis through a broad set of experiments. It combines tensor- and GEMM-level quantization diagnostics with long-run training across dense and MoE models, scaling-law analysis, and ablations.
Critical weaknesses
  • Figure 2 does not appear to isolate shrinkage bias from differences in representable range.\textbf{Figure 2 does not appear to isolate shrinkage bias from differences in representable range.}
    A higher NMSE for E2M1 may reflect differences in clipping or dynamic-range coverage, rather than skrinkage bias. It would be helpful to complement NMSE with histograms of the normalized values overlaid with the quantization-bin boundaries, as well as the mean signed quantization error for each bin.

  • The explanation that RHT moves values into E2M1’s most asymmetric rounding bins needs more direct evidence.\textbf{The explanation that RHT moves values into E2M1’s most asymmetric rounding bins needs more direct evidence.}
    Figures 2 and 4 show changes in bucket utilization and SQNR, but do not appear to show which rounding bins receive more mass or the resulting signed error in those bins. A histogram of normalized values overlaid with the rounding-bin boundaries, or per-bin signed-error statistics, would make this mechanism easier to verify.

  • The reported layer-wise bucket-ratio and Delta SQNR results do not directly demonstrate the accumulation of shrinkage bias across layers claimed in Section 3.2.\textbf{The reported layer-wise bucket-ratio and Delta SQNR results do not directly demonstrate the accumulation of shrinkage bias across layers claimed in Section 3.2.}
    The paper argues in Section 3.2 that shrinkage bias compounds across layers, but the reported layer-wise bucket-ratio and ΔSQNR results do not directly demonstrate this accumulation. Layer-wise measurements of signed quantization error or signal attenuation would help support this claim.

  • The characterization of the tensors as “outlier-heavy” would be more convincing with distributional evidence.\textbf{The characterization of the tensors as “outlier-heavy” would be more convincing with distributional evidence.}\ Histograms or other distribution plots could show how RHT changes the value distribution and how the resulting mass falls into each format’s rounding bins.

  • The transition from the Section 3 analysis to UFP4’s RHT scope is not fully clear. \textbf{The transition from the Section 3 analysis to UFP4’s RHT scope is not fully clear. }
    The analysis discusses tensors such as fwd_x and bwd_dy, while UFP4 applies RHT to the operands of all three GEMMs: fwd_y, bwd_dx, and bwd_dw.The paper should explain more explicitly how the earlier tensor-level findings motivate applying RHT to each of these paths, particularly bwd_dx and bwd_dw.

  • The Q2 evaluation relies on language-modeling loss.\textbf{The Q2 evaluation relies on language-modeling loss.}
    While language-modeling loss is a relevant pretraining metric, it does not always translate into better downstream task performance. Evaluating downstream tasks and comparing UFP4 with the E2M1 baseline, while reporting statistical significance, would help establish whether the observed loss improvements lead to practical gains.

  • The statistical significance of the Q3 gap is difficult to assess from the current presentation.\textbf{The statistical significance of the Q3 gap is difficult to assess from the current presentation.}\ The reported loss differences appear modest, and the paper does not provide uncertainty estimates or statistical tests. Confidence intervals across runs or seeds, or another assessment of fitting and measurement uncertainty, would help determine whether the observed gap is robust and practically meaningful.

Minor

  • Lines 110–111: The indices ii and jj are used without definition. Please clarify what they index and define xjx_j.

  • Line 193: Please define ZZ, αA\alpha_A, and αB\alpha_B when they first appear, or briefly summarize their definitions from Appendix E. It would also help to clarify the conditions under which αAαB<1\alpha_A\alpha_B<1 holds.

  • Lines 259–260: The phrase “consistent with the outlier-amplifying behavior of SwiGLU” could be read as attributing this behavior to RHT. Please clarify the relationship between SwiGLU and the observed outlier-heavy tensors.

  • The motivation for applying RHT to bwd_dw could be better connected to the preceding quantization analysis. Although the ablation shows that this choice improves performance, it is unclear how the earlier analysis motivates its use on this path.

  • The paper reports GEMM-level SQNR results for applying RHT to non-leaf paths under E2M1 (line 265), but does not evaluate the corresponding long-run training loss. It would be helpful to clarify whether applying RHT to these paths also affects end-to-end training with E2M1.

  • The model or source of the tensors used in Q1 should be stated more clearly in the main text. The paper says the tensors are collected from real training, but it is difficult to tell from the Q1 description which model and training setting produced them. Making this explicit would help readers assess how representative the analysis is.

Questions for authors
  • In Figure 4, E1M2 has a lower effective bucket ratio than E2M1 before RHT. Since bucket utilization alone does not establish the presence or magnitude of shrinkage bias, could the authors clarify how Figure 4 supports their claim about E2M1’s skrinkage bias?
  • Could the observed degradation reflect an incompatibility between RHT and E2M1, rather than an inherent limitation of E2M1? What do the authors think about this interpretation?
  • Relationship b/w non-leaf paths and outlier-heavy tnesors is unclear. The motivation for applying RHT to the non-leaf paths (fwd_y and bwd_dx) is not clearly established by the preceding analysis, which focuses on outlier-heavy fwd_x and bwd_dy tensors. The paper should clarify how these experimental results motivate applying RHT to the non-leaf paths.
  • Is there a rationale for choosing E1M2 over INT4, given that both use uniformly spaced quantization levels?
  • Do the authors expect shrinkage bias to be specific to FP4 training, or could it also arise in other quantization settings, such as FP8 or when converting FP32 values to BF16?
  • Could the authors compare UFP4 with Four Over Six[1], which also aims to improve E2M1-based NVFP4 quantization through adaptive block scaling?

[1] Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.