- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper studies the effect of quantization scale on layer-wise reconstruction loss. The authors show that the normalized reconstruction loss can be approximated by a scalar Gaussian quantization objective under the assumptions that the weights follow a Gaussian distribution and Gram matrices of activations have sufficiently high effective ranks. They further analyze the loss landscape and show that scale sensitivity is significantly higher at low bit-widths. Experimental results on pretrained LLMs are consistent with the theoretical analysis.
Critical strengths
(+) The authors provide rigorous theoretical support for approximating the layer-wise normalized reconstruction loss with a Gaussian quantization objective under their assumptions.
(+) The curvature analysis of the Gaussian objective provides a theoretical explanation for why quantization performance is more sensitive to scale selection at lower bit-widths.
Critical weaknesses
[Contribution]
(-) While the paper provides a rigorous analysis of scale sensitivity under Gaussian weights and a high effective-rank condition, the main conclusion—that scale selection is more critical at very low bit-widths—appears largely intuitive and empirically expected. It is unclear whether the analysis provides a sufficiently strong new understanding of practical PTQ beyond confirming an already familiar trend.
[Validity]
(-) The authors have demonstrated that converges to for infinite values of . Nevertheless, it remains unclear how accurate this approximation is at practical model dimensions. Therefore, I think the authors need to measure the empirical differences between and on real model, e.g., llama, qwen, gemma.
(-) There is a substantial gap between the setting covered by the theoretical analysis and the pretrained LLM experiments. In particular, the effective-rank condition (H2) is verified only for randomly initialized MLPs under isotropic Gaussian inputs, whereas the experiments are conducted on pretrained transformers. As also acknowledged by the authors, it is not established whether the Gaussian-weight and effective-rank assumptions continue to hold after training. An empirical verification of (H2) on the evaluated pretrained models would strengthen the connection between the theory and experiments.
(-) As acknowledged by the authors, the theoretical results are derived only for RTN, whereas the main practical experiments use GPTQ, which performs sequential error compensation and is not covered by the analysis.
(-) The theoretical analysis is restricted to weight-only quantization. It remains unclear whether the derived scale-sensitivity behavior extends to weight-activation quantization settings, where activation quantization errors may substantially alter the reconstruction landscape.
[Presentation]
(-) While the main intuition of the paper is relatively simple, the presentation relies on a large number of propositions and theorems, making the paper unnecessarily difficult to follow. For example, the continuous extension from the discrete quantization level to a real-valued appears primarily technical and could potentially be moved to the appendix, while the main text could focus directly on the bit-dependent Gaussian objective and its curvature.
Questions for authors
Please refer to the weaknesses. My primary concern is whether the paper offers sufficient novelty beyond formalizing and confirming the familiar trend that quantization scale selection is more critical at lower bit-widths. I would consider raising my rating if the authors can address my concerns.
3 comments
- Extension to GPTQ. Our rigorous convergence theorem concerns RTN, we provide now an heuristic argument justifying why the Gaussian landscape transfers to GPTQ after HIP. To address sequential compensation, Section 4.2 now develops a heuristic extension to GPTQ after HIP. Under structural assumptions on the Gram matrix, together with block self-averaging, the predicted limiting landscape is
Request to engage.
We kindly ask the reviewer to let us know if their concern has been resolved. If not, we are happy to discuss any remaining concerns. If there are no remaining doubts, we would appreciate the reviewer to consider updating their assessment.
We thank the reviewer for the careful reading and constructive feedback, and for recognizing the rigor of our theoretical analysis and its explanation of scale sensitivity. Below, we clarify the reviewer's concerns.
-
-
-
=\frac{(\sum_i p_i)^2}{\sum_i p_i^2},$$ where $p_i$ are the coefficients weighting the squared compensated residuals in the GPTQ loss. Larger effective counts and smaller maximum normalized pivot weights are associated with smaller variance-adjusted landscape errors (Table 11). The effective count measures how evenly rounding steps are weighted, and these observations are consistent with the self-averaging motivation in Appendix A.7.Contribution. We agree that the greater difficulty of low-bit quantization, and the importance of carefully selecting quantization parameters in this regime, are familiar observations. However, greater quantization error at low precision does not by itself determine how strongly the reconstruction loss responds to a relative perturbation of the scale. Our contribution is to provide a mathematically precise description of this dependence and identify conditions under which it is justified. The central result rigorously establishes that a high-dimensional, random, Hessian-weighted reconstruction landscape converges uniformly over scales to a deterministic scalar Gaussian objective under the stated assumptions. Identifying Gaussian scalar quantization as the uniform limit of this reconstruction landscape requires justification. This derivation provides an explicit object on which to study scale sensitivity. We characterize its optimal scale and derive its curvature with respect to relative scale perturbations, whose numerical evaluation reveals an approximately exponential decrease with bit-width. The analysis complements the qualitative intuition with concrete, quantitative predictions that go beyond the intuition alone and can be tested at finite widths. In particular, the Gaussian limiting description motivated us to investigate HIP preprocessing as a way to improve agreement with the theoretical prediction, which our experiments support. In the updated version, we further develop heuristics to examine this connection beyond initialization (Appendix A.7) and add quantitative experiments assessing when empirical GPTQ landscapes agree with the Gaussian prediction. See also our responses below on applicability after training and finite-dimensional accuracy.
Finite-dimensional accuracy on trained models. We have added direct measurements of the discrepancy between empirical reconstruction landscapes and the Gaussian prediction in Appendix B.5.2. The experiments use Llama-3.2-1B and Llama-3.1-8B at W2 and W3, and they cover different depths and projection types, with input dimensions ranging from 2048 to 14336. We compare the normalized GPTQ landscapes with the fixed Gaussian objective and variance-adjusted references motivated by the HIP heuristics in Appendix A.7. Figure 13 shows that HIP substantially reduces the measured landscape errors. After additionally accounting for GPTQ compensation-induced variance changes, the remaining discrepancies can be comparable to finite-Gaussian sampling errors (Table 9). These measurements provide empirical support for the approximation at practical dimensions beyond what the theoretical bound certifies. See our answer to Question 5 of Reviewer dPUS for the detailed setup and further results.
Effective rank and applicability after training. We agree that verifying the assumptions for randomly initialized MLPs does not establish their validity for pretrained transformers. We have therefore added measurements of effective rank and its relationship with Gaussian approximation accuracy in Appendix B.5.3. However, the absolute effective-rank values should not be interpreted as a sharp finite-width criterion for Gaussian approximation. Condition (H2) is a sufficient asymptotic growth condition, and our estimates were designed to identify the limiting landscape rather than optimize finite-width constants. The reported mean effective ranks are 3.44/5.34 for Llama-1B and 4.07/8.62 for Llama-8B at W2/W3, respectively. HIP preserves these ranks for fixed incoming activations. Higher effective rank is associated with smaller RTN Gaussian approximation errors (Table 10). Diagonal rescaling and whitening also reduce the median landscape errors (Table 12), although these transformations change both the Gram spectrum and the weight coordinates, so they do not isolate the effect of effective rank. Together with the correlations in Table 10, these observations support effective rank as a relevant feature of the Gram matrix for understanding Gaussian approximation accuracy, even though its numerical value alone does not provide a sharp finite-width accuracy criterion. For GPTQ, we additionally measure the effective number of pivot weights,