- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper is targeting to mitigate the outlier problem during low-precision training. To do so, the authors study the mean of each activaiton's channel, and find that the mean direction has a consistent correlation with each token during training. They name this phenomenon "mean bias" and they further analyze how the mean bias causes the activation outlier. Based on this idea, the authors propose Averis which quantize activation mean and residual separately. This decomposition also works on gradient as well. To make the training efficient with the mean and residual separate quantization, the authors further introduce a single GeMM implementation.
Experiments are based on NVFP4 and include both dense model (Qwen3-0.6B) and MoE model (Qwen3-7B-A1.5B), each trained on 100B tokens. The authors report lower training loss, better accuracy, and system improvement and tradeoff. In detail, the training curves show lower training loss compared to NVFP4 baseline, the evaluations on downstream tasks show improvement but not consistent, and the system evaluation show a marginal overhead as a trade-off for better accuracy.
Critical strengths
- The single-GeMM implementation is clean, clear and making sense under the proposed mean and residual decomposition method. By appending the mean as additional row to the residual, it can avoid a separte matmul for the mean term. Also the reported step-wise overhead is relatvly low (~2% for both models), and it can support potential practical usage.
Critical weaknesses
- The experiments do not establish a robust advantage under comparable resource budgets. First, the reported results in Table 1 seem not significant enough. FP noise and distributed training could both contribute to the marginal difference between Averis and NVFP4. The authors should at least provide either a confidience interval or standard deviation for each evaluation tasks across more trained checkpoints or train Averis at least twice using exactly same configuration and report the relative error. Otherwise the current evaluation gaps are not supportive enough. Second, the comparision is not fair enough. There are two concerns here. (1) the average bit per activation of this method is higher than NVFP4 and the authors do not discuss it. Given the experiment setup, this method is based on NVFP4 but since it quantizes mean and residual, the mean will cost additional bit storage meaning more information provided actually. (2) this method is slower than NVFP4 and thus for a given time limit, NVFP4 can actually train more tokens. Because of this, the NVFP4 in practice can have a very high chance to have a better performance. These two concerns will limit the practical value of this method and the unfair comparision cannot let me trust the effectiveness of the method.
- Mean bias causing outlier does not seem convincing to me. The analysis in Section 2.3 is firstly not causual analysis so saying "MEAN BIAS AS THE DOMINANT SOURCE OF ACTIVATION OUTLIERS" is not accurate. Also, the analysis seems mess up with the causal effect or at lease may be using the result as the cause based on my understanding of activation outliers. LLM.int8() has already report that the outliers on activation are mostly channel wise outlier. As a result, it first explan why the squared contribution defined on row 188 can be high when the outliers in the same channel have the same sign and similar magnitudes across tokens. It secondly explain the high cosine similarity if I understand correctly how you compute them. The reason is because of this channel-wise pattern that different rows in the activation are having the outlier at the same position. For example, [0.1, 0.2, 1, -0.1] and [0.2, 0.3, 1.2, 0.1], in this case the same positive channel dominates both vectors, so the direction is bascially the same meaning high cosine similarity. As a result, I think the mean bias contribution is being over-stated, and unless the authors provide strong evidence that this mean bias is indeed the root cause, I will stay negative to this contribution.
Questions for authors
The main issues affecting my judgment are already discussed in the weaknesses. In addition, I have a few questions to make sure I understand the analysis and algorithm correctly.
-
In Figure 1(B), the text says the “other direction” is the second right singular vector (v_2). Is this what the authors mean by “ordinary variance” in the next sentence? I do not understand why these are equivalent. Why choose this direction as the comparison? There are other directions and statistics that could be compared, so why only these?
-
At the left end of Figure 1(B), the cosine similarity magnitude for the other direction seems larger than that for the mean direction. Is there an explanation for this? Also, what is the sorting rule for “Sorted token index”?
-
For “Operator-Level Regeneration and Amplification”, have the authors plotted the same analysis across more layers and training checkpoints? How were the current examples selected? From the current examples, it is hard for me to tell whether this is a general pattern or something specific to the selected cases.
-
What is the motivation for applying the same mean–residual split to gradients? The main analysis seem focuses on activations only and I do not see clearly why the same decomposition is needed for gradients or how much it contributes to the final results.
The following points by themselves would not change my current judgment, but I still hope the authors can answer them and improve the presentation in a later version.
-
I do not think the insight from the spectrum analysis in Figure 1 is explained clearly. Could the authors briefly explain why this analysis is needed and what we should learn from it? I suggest stating this more clearly in the paper.
-
Figure 1 only shows results from Qwen3-0.6B. Could the authors provide the same analysis on more models, especially larger models, to show whether this phenomenon generalizes?
-
After reading the paper, I am still not clear which mark or equation exactly corresponds to “mean bias”. Or, how "mean bias" connects to other marks.
-
On line 139, why track the alignment with the top right singular vector? Seems like the earlier discussion talks about token-wise alignment, but it suddenly change to top right singular vector with no reason.
-
Some notation is used before it is explained. For example, around line 301 and G in W4A4G4 around line 362. I feel hard to follow the logic so please define these at first use.
Thanks the author for the detailed responses and indeed for the time of the rebuttal they address my points. However by looking at the numbers it looks like NSN style correction would outperform the suggested approach (this is base on table FPROP GeMM-output relative error in the response to review jPhX). I would at less suggest exploring this as future work, if not use it to extend the current method. Overall I think that the observation of the paper is good but maybe incomplete (base on NSN analysis). Despite that I think that the NSN-style correction can provide a more complete approach, I have decided to raise my score.