- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates optimization imbalance in PTQ. The authors observe that MSE optimization will leads to sharp gradient and weights update, so they propose to replace MSE with RMSE, which normalizes the output-space gradient magnitude independently of the reconstruction loss scale. Extensive experiments across multiple LLM families, quantization settings, and PTQ methods demonstrate consistent improvements.
Critical strengths
- Replacing MSE with RMSE is straightforward and requires minimal implementation changes. And the theoretical derivation clearly explains its output-gradient normalization property.
- The analysis of reconstruction losses across different quantization stages provides a useful perspective on existing methods development.
- The consistent improvements are encouraging and the reported computation overheads are also negligible.
Critical weaknesses
- The central argument is that MSE will cause gradient magnitude differences and imbalanced parameter updates, which RMSE mitigates this issue through gradient normalization. However, adaptive optimizers such as Adam are approximately invariant to uniform gradient rescaling under certain conditions. Therefore, normalizing the output-space gradient is not equal to more balanced weight updates.
- While RMSE consistently improves average accuracy, Table 4 shows that under W2A16, HumanEval accuracy remains at 0%, while MBPP improves only from 0% to 0.26%. This indicates that balancing reconstruction gradients may be insufficient to preserve downstream capabilities under extreme low-bit.
- The authors measures optimization strength through relative weight updates and interprets smaller updates as insufficient optimization. However, smaller gradients or parameter updates may also indicate smaller reconstruction errors or closer to a local optimum. Average gradient magnitudes across stages is not equal to more effective optimization.
Questions for authors
- Can you clarify the role of optimizer scale invariance, moment normalization and gradient clipping in the observed improvements?
- Why does RMSE provide limited recovery on downstream tasks under 2-bit despite the reported improvement in gradient balance?
- Can you provide convergence curves or measurements of relative reconstruction error reduction across quantization stages to validate that a smaller relative weight update is equal to insufficient optimization?