- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper analyzes the impact of quantization error on looped models. Based on the analysis and the evaluation across 33 looped models, the paper shows that the fixed weight noise only shift the settled state, rather than fully distorting the settle point due to repeatedly accumulated error. Also, the paper argues that the accuracy failures occur only when this shift exceeds the permitted "room". Therefore, this paper argues that only a few higher-bit loops (e.g., 8-bit) can recover the quantized looped model's accuracy. Based on these findings, the paper proposes a controller that enables adaptive stopping, followed by few 8-bit loops to improve accuracy.
Critical strengths
- This paper first provides the interesting hypothesis and analysis on quantization collapse in looped models. I think this work's insight can be extended to other compression method beyond quantization for looped models.
- Provided analysis is comprehensive and interesting. Especially, different behaviors between the fresh activation noise and the fixed weight noise across iterations are interesting.
- Beyond the analysis, proposing the Controller based on the observation provides a guideline for the deployment.
Critical weaknesses
- While explanation with "Settling-point" is intuitive, I think there is a need of more direct ablation. Under the fixed number of total loops and the 8-bit (high-precision) loops, I wonder how final accuracy differs across the positions of 8-bit loops. For example, while comparing 8-bit loops first then 4-bit loops vs. 4-bit loops then 8-bit loops (default), does the later consistently achieve higher accuracy?
- It seems that activation quantization can be more robust than the weight quantization, as activation quantization error can be acted as fresh noise (if the magnitude is not too big). However, the main results are focused on weight-only quantization. Discussion on weight-activation quantizaiton would be beneficial.
- Currently, the main analysis and results are focused on the round-to-nearest quantization. Extending the observation to calibration-based method would be beneficial. For example, GPTQ targets to minimize output distortion, while increasing the weight distance itself. In this case, I wonder that GPTQ hurts looped model's accuracy due to increased fixed weight noise.
Questions for authors
Please see Weaknesses.
2 comments
Thank you for the rebuttal
Thank you for the rebuttal and the additional experiments. I find the new results interesting and helpful, particularly the loop-ordering ablation, and I would welcome their inclusion in the revised paper. I maintain my positive assessment of the work.
Reply to Reviewer LXVH
Thank you for the review. We ran the two experiments you proposed.
8-bit loops last are better on four of the five collapsed formats, and the order does not matter on surviving formats. We fixed the total at 24 loops (16 compressed loops and eight 8-bit loops) and changed only the position of the 8-bit loops (Sudoku-Extreme, n=1000):
The exception is HRM w2g32, where 8-bit loops first are better. On the four surviving formats we tested, the two orders are within 1 point. The pattern is the same with 4 and with 16 8-bit loops.
On MLP w3c and HRM w2g32, 8-bit loops first also recover much of the accuracy. If rounding error accumulated, this gain would fade as more compressed loops follow. It does not:
The gain keeps 95 to 98% of its size after 256 compressed loops. The loop settles in the basin where the 8-bit loops placed it, which settling can explain and accumulation cannot.
This matches Figure 2: fresh, zero-mean noise produces the same cliff as fixed noise, only later. Fresh error is therefore more forgiving, as you suggest, but it still collapses the model once it is large enough. The weight-activation formats (W4A4 with per-group and MXInt4 scales, and W8A8) are in Appendix C, Table 8; their gap to full precision does not widen along supervision steps or outer cycles.
GPTQ does not hurt. It lowers the push even where it increases the weight distance, and the push is what decides collapse:
GPTQ lowers the push on all 7 formats we measured (by 1.3 to 6.2x; on the other four it lowers the weight distance too), and the push-versus-room law classifies all 7 GPTQ models correctly. Accuracy follows the push: GPTQ raises fixed-depth accuracy, and the controller improves GPTQ on 10 of 10 cells (GPTQ table in the general response).