- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Many of post-training quantization methods apply the (randomized) Hadamard transform to the weights to mitigate outliers and then GPTQ[1] minimization problem is solved: $ || Q(W H) H^{-1} X - W X ||_F^2 \rightarrow \min$. Here is the weight matrix and are input activations of a linear layer. Equivalently, the objective is the classic GPTQ minimization with a change of variables to .
This quantizes the weights offline. During inference activations are transformed online as .
The paper investigates a failure mode when activations supplied to GPTQ are kept unchanged ( instead of ), so the minimization becomes $ || Q(W H) X - W H X ||_F^2 \rightarrow \min$. Naturally, without the paired inverse Hadamard transform applied to the activations passed to GPTQ the minimization objective shifts and model quality degrades.
The problem the paper is solving is not substantial. As the authors point out "the failure is one configuration line away". Then the solution is one line away also. Concretely, the fix is to ensure that the inverse Hadamard transform is being applied to activations passed to GPTQ offline and to input activations online (online rotations). In existing codebases this is done by setting an argument online_rotate=True (an example from ModelTC[2]).
Therefore I believe the main question of the paper is 1) not significant and 2) already solved.
Experimental results I find doubtful, because authors report that for Qwen2.5-7B RTN performs better than GPTQ (both without any transform). Theoretical results provided by the authors I find not very relevant, since the authors themselves describe a clear fix to the failure mode by changing a configuration.
[1] https://arxiv.org/abs/2210.17323
[2] https://github.com/ModelTC/LightCompress/blob/main/docs/en/source/practice/quarot_gptq.md
Critical strengths
- Interaction between different quantization related techniques is an important question to study. Applying rotation transforms before quantization introduces an additional error: .
Critical weaknesses
-
Contribution of the paper is marginal. A simple solution already exists: pass the online_rotate flag equal to True. Defaulting it to True is an engineering fix but it is overall not a scientific problem. Constructing transforms that commute with quantization I believe is excessive in this case.
-
Experimental results provided do not appear to be reliable. In Table 1 authors show that for 2 models under no transform RTN quantization outperforms GPTQ, which I find hard to believe.
Questions for authors
- How would you explain that for Qwen2.5-7B and Llama-3.1-8B under no transform PPL with RTN is lower than with GPTQ? (Table 1)
Dear Reviewers, Area Chairs,
If there are any other concerns or comments, we are happy to clarify and answer specific questions, including any additional concerns you may have after our rebuttal. We really enjoyed some of the rebuttal exchanges and look forward to updating the paper with the suggestions and comments.
Best regards, Authors