acceptodds
Under review as a conference paper at ICLR 2027

The Double-Edged Role of Mean Bias in FP4 LLM Pretraining

12 views

Abstract

FP4 training promises substantial memory and compute savings for large language models, but remains fragile because blockwise quantization is dictated by extreme activation magnitudes, which inflate dynamic range and compress long-tail signals. We identify a counterintuitive source of this failure: dominant activation outliers are not merely arbitrary sparse events, but are largely induced by a coherent rank-one mean bias, whose direction aligns with the leading anisotropic spectral component. This mean component strengthens during training, is amplified and reshaped by attention and FFN operators, and increasingly dominates top activation magnitudes. Crucially, this discovery reveals that a seemingly complex outlier-suppression problem admits a truly simple solution: isolate the coherent mean before quantization. We therefore propose Averis, a mean–residual splitting quantization method that separates the mean component using only reductions and elementwise subtractions before FP4 quantization. Across Qwen3-0.6B Dense and Qwen3-7B-A1.5B MoE, each trained on 100B tokens, Averis enables robust W4A4G4 FP4 training, reducing BF16 loss gaps to 1.19%/0.479% versus 2.05%/0.737% for NVIDIA's recently released Hadamard-based outlier-smoothing method, while limiting downstream gaps to 0.89/0.68 points. With only 2.20% end-to-end overhead over vanilla NVFP4, about 30% of NVIDIA's Hadamard-based design, Averis provides a hardware-efficient path to stable low-bit LLM training. Code is available at: https://anonymous.4open.science/r/averis-504D.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Quantization, FP4, Pretraining

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YJWROpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper is targeting to mitigate the outlier problem during low-precision training. To do so, the authors study the mean of each activaiton's channel, and find that the mean direction has a consistent correlation with each token during training. They name this phenomenon "mean bias" and they further analyze how the mean bias causes the activation outlier. Based on this idea, the authors propose Averis which quantize activation mean and residual separately. This decomposition also works on gradient as well. To make the training efficient with the mean and residual separate quantization, the authors further introduce a single GeMM implementation.

Experiments are based on NVFP4 and include both dense model (Qwen3-0.6B) and MoE model (Qwen3-7B-A1.5B), each trained on 100B tokens. The authors report lower training loss, better accuracy, and system improvement and tradeoff. In detail, the training curves show lower training loss compared to NVFP4 baseline, the evaluations on downstream tasks show improvement but not consistent, and the system evaluation show a marginal overhead as a trade-off for better accuracy.

Critical strengths
  • The single-GeMM implementation is clean, clear and making sense under the proposed mean and residual decomposition method. By appending the mean as additional row to the residual, it can avoid a separte matmul for the mean term. Also the reported step-wise overhead is relatvly low (~2% for both models), and it can support potential practical usage.
Critical weaknesses
  • The experiments do not establish a robust advantage under comparable resource budgets. First, the reported results in Table 1 seem not significant enough. FP noise and distributed training could both contribute to the marginal difference between Averis and NVFP4. The authors should at least provide either a confidience interval or standard deviation for each evaluation tasks across more trained checkpoints or train Averis at least twice using exactly same configuration and report the relative error. Otherwise the current evaluation gaps are not supportive enough. Second, the comparision is not fair enough. There are two concerns here. (1) the average bit per activation of this method is higher than NVFP4 and the authors do not discuss it. Given the experiment setup, this method is based on NVFP4 but since it quantizes mean and residual, the mean will cost additional bit storage meaning more information provided actually. (2) this method is slower than NVFP4 and thus for a given time limit, NVFP4 can actually train more tokens. Because of this, the NVFP4 in practice can have a very high chance to have a better performance. These two concerns will limit the practical value of this method and the unfair comparision cannot let me trust the effectiveness of the method.
  • Mean bias causing outlier does not seem convincing to me. The analysis in Section 2.3 is firstly not causual analysis so saying "MEAN BIAS AS THE DOMINANT SOURCE OF ACTIVATION OUTLIERS" is not accurate. Also, the analysis seems mess up with the causal effect or at lease may be using the result as the cause based on my understanding of activation outliers. LLM.int8() has already report that the outliers on activation are mostly channel wise outlier. As a result, it first explan why the squared contribution defined on row 188 can be high when the outliers in the same channel have the same sign and similar magnitudes across tokens. It secondly explain the high cosine similarity if I understand correctly how you compute them. The reason is because of this channel-wise pattern that different rows in the activation are having the outlier at the same position. For example, [0.1, 0.2, 1, -0.1] and [0.2, 0.3, 1.2, 0.1], in this case the same positive channel dominates both vectors, so the direction is bascially the same meaning high cosine similarity. As a result, I think the mean bias contribution is being over-stated, and unless the authors provide strong evidence that this mean bias is indeed the root cause, I will stay negative to this contribution.
Questions for authors

The main issues affecting my judgment are already discussed in the weaknesses. In addition, I have a few questions to make sure I understand the analysis and algorithm correctly.

  1. In Figure 1(B), the text says the “other direction” is the second right singular vector (v_2). Is this what the authors mean by “ordinary variance” in the next sentence? I do not understand why these are equivalent. Why choose this direction as the comparison? There are other directions and statistics that could be compared, so why only these?

  2. At the left end of Figure 1(B), the cosine similarity magnitude for the other direction seems larger than that for the mean direction. Is there an explanation for this? Also, what is the sorting rule for “Sorted token index”?

  3. For “Operator-Level Regeneration and Amplification”, have the authors plotted the same analysis across more layers and training checkpoints? How were the current examples selected? From the current examples, it is hard for me to tell whether this is a general pattern or something specific to the selected cases.

  4. What is the motivation for applying the same mean–residual split to gradients? The main analysis seem focuses on activations only and I do not see clearly why the same decomposition is needed for gradients or how much it contributes to the final results.

The following points by themselves would not change my current judgment, but I still hope the authors can answer them and improve the presentation in a later version.

  1. I do not think the insight from the spectrum analysis in Figure 1 is explained clearly. Could the authors briefly explain why this analysis is needed and what we should learn from it? I suggest stating this more clearly in the paper.

  2. Figure 1 only shows results from Qwen3-0.6B. Could the authors provide the same analysis on more models, especially larger models, to show whether this phenomenon generalizes?

  3. After reading the paper, I am still not clear which mark or equation exactly corresponds to “mean bias”. Or, how "mean bias" connects to other marks.

  4. On line 139, why track the alignment with the top right singular vector? Seems like the earlier discussion talks about token-wise alignment, but it suddenly change to top right singular vector with no reason.

  5. Some notation is used before it is explained. For example, μD\mu_D around line 301 and G in W4A4G4 around line 362. I feel hard to follow the logic so please define these at first use.

Reviewer_c3nbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the author analysis the source of outlier in LLMs training and shown that the key effect is mean of feature maps. In addition they relate the mean bias to the large singular value. Moreover they have shown that operation level analysis which the key point when outlier start. In the finally, part of the analysis the paper discussed the relation of the mean bias to outlier. Base on this analysis the author suggest a simple method that split the quantization into mean quantization and residual quantization. In the experiment the paper compare to single baseline and shown improvement in loss and down steam task.

Critical strengths

(1) The author presented a detailed analysis and relate the means to singular values and outlair which is very informative.

(2) The suggest method is very simple, and easy to implement with a low overhead compare to the select baseline.

Critical weaknesses

(1) A key weakness of this paper is missing related works section and the missing relation between the suggested analysis and other papers,

1.1 The operation level analysis was done by [R.1], which showed the same phenomena that outlier start in the FFN. Moreover they showed that reason for outlier is contributed to the attentions layer.

1.2 Block quantization for training is highly related to micro-scaling data format and there is a new line of works for both training and inference using micro-scaling data format [R.2, R.3] .

1.3 The relation of outlier to signaler value was already presented in [R.4] the author should highlight that the key contribution here which is the relation to the mean vector.

1.4 The effect of mean bias and its relation to outliers was shown in [R.5] in the context of kv-cache, the relation between this work and R.5 is interesting.

(2) Experiment limitation: The paper is comparing with a single baseline. Adding some baseline would help this paper and more specifically [R.4] as they use special decommission and related to anisotropic spectral direction.

(3) Novelty: the key novelty of this method is the relation between the mean bias and outlier. However such relation has been shown in other work in different aspect of quantization such as in NSN [R.5] which removes the bias as well. Moreover the relation to spectral decomposition also has limited novelty as it was shown in this application in [R.4].

[R.1]Bondarenko, Yelysei, Markus Nagel, and Tijmen Blankevoort. "Quantizable transformers: Removing outliers by helping attention heads do nothing." Advances in Neural Information Processing Systems 36 (2023): 75067-75096.

[R.2] Rouhani, Bita Darvish, et al. "Microscaling data formats for deep learning." arXiv preprint arXiv:2310.10537 (2023).

[R.3] Zhang, Yu, et al. "MOSS: Efficient and accurate FP8 LLM training with microscaling and automatic scaling." International Conference on Learning Representations. Vol. 2026. 2026.

[R.4] Cao, Hengjie, et al. "Metis: Training llms with fp4 quantization." International Conference on Learning Representations. Vol. 2026. 2026.

[R.5] Son, Donghyun, Euntae Choi, and Sungjoo Yoo. "NSNQuant: A double normalization approach for calibration-free low-bit vector quantization of KV cache." Advances in Neural Information Processing Systems 38 (2026): 43124-43159.

Minor comments:

M1. Several reference use arXiv instead of conference references for example:

M1.1 Metis: Training llms with fp4 quantization

M1.2 Fp4 all the way: Fully quantized training of llms.

M2. Some notation would be clear for example: M2.1 I would suggest to define Y_{j}^{(0)}=Y_{j}-m_j explicitly as equation. Also why three are two symbols for the same distribution \eta_{I_j} and Y_{j}^{(0)}?

Questions for authors

(1) Can the authors compare to the suggest approach to [R.5] and highlight the benefits of the suggest method ? in terms of accuracy/loss and runtime?

(2) Can the author provide the position of this paper to compare to prior and what is the novelty of the analysis compare to prior art?

(3) Regrading Theorem 1, is there way to relate this to quantization metric such RMSE/MSE to highlight the benefit in terms of precision and not only outliers ? maybe even compare the quantization error of the suggest method compare to naive baseline.

1 comment

Thanks the author for the detailed responses and indeed for the time of the rebuttal they address my points. However by looking at the numbers it looks like NSN style correction would outperform the suggested approach (this is base on table FPROP GeMM-output relative error in the response to review jPhX). I would at less suggest exploring this as future work, if not use it to extend the current method. Overall I think that the observation of the paper is good but maybe incomplete (base on NSN analysis). Despite that I think that the NSN-style correction can provide a more complete approach, I have decided to raise my score.

Reviewer_jPhXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies a shared rank-one mean component as an important source of large activation magnitudes. It proposes Averis, which separately quantizes the mean and residual of activations and output gradients for FP4 pretraining. Experiments on dense and MoE models show smaller loss gaps than vanilla NVFP4 and the Hadamard baseline, with modest additional training overhead.

Critical strengths
  • Simple and interpretable design that avoids explicit SVD.
  • The mean contribution can be computed once and shared across tokens, keeping additional matrix-multiplication cost low.
  • Consistent loss improvements across two model settings trained on 100B tokens, with further gains when combined with Hadamard.
Critical weaknesses
  • Related mean-centering and SVD-based approaches are discussed but not directly compared.
  • The combined Hadamard pipeline and its relationship to the released simulation implementation need clearer specification.
Questions for authors
  • In Averis-Hadamard, is the Hadamard transform applied only to the residual? If so, could outliers in the mean vector still introduce substantial FP4 quantization error? Please quantify this error and its impact on training quality, ideally through an ablation using higher precision for the mean branch.

  • The released simulator appears to quantize the mean before subtraction, whereas the equations center using the unquantized mean. Which implementation produced the reported results, and how does this affect the two-term weight-gradient approximation?

  • Can you provide matched FP8 comparisons and a runtime breakdown covering mean reduction, subtraction, quantization, and mean restoration at the actual per-GPU training shapes?

  • If used for inference, how does the method behave at small batch sizes, especially single-token decoding where the residual becomes zero?

1 comment
AuthorsOpenReview ↗

Gentle Follow-up on Our Rebuttal

Dear Reviewer jPhX,

Thank you again for the time and effort you have put into reviewing our paper. We have responded to each of your questions, quoting and addressing them in turn, and have revised the manuscript accordingly.

We would appreciate it if you could review our responses at your convenience. If any questions remain, we would be happy to discuss them further. If our responses have addressed your concerns, we would be grateful if you would consider updating your assessment.

Thank you once more for your time and constructive feedback.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.