- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1(−1)
- Soundness
- 2
Read review
Paper summary
This paper introduces a regularization method in PTQ for penalising the activation outliers through ||W_F1 W_F2 ||_F where W_F1 and W_F2 are the consecutive matrices in FFN of Transformers. This regularizer comes from existing literature TWEO and the existing Colinearity Decay literature.
Critical strengths
- The topic this paper introduced is interesting.
- The experiments make sense to me.
- The bibliography is good.
Critical weaknesses
I carefully read this work for managing to understand its logic, evidences, and method.
The presentation quality is below the bar. There are rooms to be further improved. Please see the "Questions for authors".
This work lacks experiments to defense the arguments/motivations/claims made by authors.
As far as my understanding, it has limited novelty. This regularizer comes from existing literature TWEO and the existing Colinearity Decay literature.
Questions for authors
I first highlight the points I found. I'll state details after these.
- Presentation Quality. Presentation is very limited and I believe there is a significant room to be improved. I carefully read the Abstract, Introduction, related work, and Method. Despite the topic is interesting, the presentation.
For example
"We empirically show that excessive suppression of activation magnitudes can degrade both unquantized and quantized accuracy. Quantization-friendly training should instead target harmful structural artifacts that cause extreme activations".
These claims are crucial to justify/motivate this work, however they are never supported by evidences.
- Motivation Logic.
There is a logic problem in the introduction. In the first paragraph, it argues quantization is crucial for deploying transformer-based models.
In the second paragraph, the work argues PTQ/QAT are a way to reverse the performance degeneration due to quantization.
However, in the third paragraph, the work introduces the attention sink/artefacts/activation outliers. I don't understand how this correlates to quantization and how the activations are not represented by quantized values. Can the authors clarify and improve the presentation in future?
Also the attention sink should cite the Meta's paper that firstly introduced the concept.
- Setting May Not Hold. Another question is, since nowadays attention sink is solved by QWen's group in [1]. What is the significance of the proposed method?
[1] Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free, NIPS25, https://openreview.net/forum?id=1b7whO4SfY
-
The work should clarify/present that why quantization can't represent outliers is a culprit.
-
Over claims or claims not supported by evidences.
For example
"in this paper, we argue that the latter strategy offers a better trade-off between full-precision and quantized accuracy."
this is not supported by evidence or ant experiments.
For example
L86-88
"Our key argument is that quantization- friendly training should reduce harmful structural artifacts that cause extreme activations, but should not suppress activation magnitudes to a level that potentially degrades task performance."
Claims. not supported by evidence or experiments.
I don't check more claims. But regarding the claims I read, they are not supported by evidence.
- L81
"Figure 1 compares TWEO with our Colinearity Decay (CD) and provides task and block-level".
The work doesn't introduce CD yet. This confuses me in reading. The engagement is very low.
- Notations are not introduced.
for example L179-180. The mathematical notations must be properly introduced.
- Typos.
"Figure 1: Smaller activations do not necessarily improve quantization."
improve quantization?
- Limited novelty. This work contributes little to what we already knew. This regularizer comes from existing literature TWEO and the existing Colinearity Decay literature.
4 comments
I thank the authors for the replies.
I thank the authors for the replies. Unfortunately, the replies appear just repeating for each reply and fails to answer my concerns.
Respond to Your Feedback
Thank you for your feedback. We remain open to further technical discussion of the paper and would sincerely welcome specific evidence supporting your concerns, so that we can address them directly.
Thanks
Dear Reviewer mUSi,
As the discussion period draws to a close, we would like to thank you for your time and consideration.
We hope that our responses have clarified CD's contribution and the experimental support for our claims. We would appreciate your consideration of these clarifications and additional results in your final assessment.
We wish you all the best.
Best regards,
The Authors
Thank you for your reviewing.
We first clarify two key misunderstandings concerning the originality and scope of our method stated in the paper summary:
We have devoted substantial effort to making the presentation clear. Given the conference page limit, we cannot discuss every background detail and assume a working familiarity with model quantization. We address your specific questions below:
Experimental support for the accuracy claims: Table 1 compares CD with TWEO and KurtReg under matched training and evaluation settings. Despite substantially reducing maximum activations, these activation regularizers lower FP accuracy in most comparisons and yield inconsistent quantized gains, whereas CD offers a better overall trade-off. Appendices A.3 and A.4 report implementation details and hyperparameter sweeps (Table 5), documenting the selection of comparison settings. Section 4.2 explicitly connects these observations to our claims about activation suppression and FP/quantized accuracy.
Attention sinks and quantization difficulty: We would appreciate clarification of the criterion for considering attention sinks “solved”. In our view, attention sinks and quantization difficulties are overlapping issues, with neither subsuming the other. Table 1 already shows that CD further improves quantized accuracy with AttnBias. To address the cited method directly, we additionally reproduced headwise gated attention. The results below show further gains from CD under both quantization pipelines while preserving FP performance (accuracy in %; gains in percentage points).
We appreciate the suggestions for improving clarity and completeness. We address the specific presentation and citation comments below: