- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies why PTQ can fail at low bit widths and how QAT recovers the lost performance. The authors propose a river-valley-basin loss landscape to explain this phenomenon, where quantization may move pretrained weights outside the low-loss region, while QAT uses gradients at quantized weights to guide them back. The paper constructs a Hessian-based PTQ failure example and proves finite-time QAT recovery under specific assumptions. Experiments cover ResNet, DeiT, and a 213M Llama model, including loss landscape analysis and comparisons with continued full-precision training.
Critical strengths
- This paper provides an interesting geometric explanation for PTQ failure and QAT recovery, rather than simply showing that QAT performs better. The theoretical analysis helps explain the different optimization behaviors of these two approaches.
- The experiments cover both vision and language models, multiple bit widths, and different PTQ methods. In particular, the comparison with continued full-precision training under the same additional training budget helps demonstrate that QAT's improvement is not simply due to more training steps.
Critical weaknesses
- My main concern is that the theoretical assumptions are not sufficiently connected to the actual experiments. Theorem 1 requires specific conditions on the basin width, quantization error, and loss geometry, but the paper does not verify whether these conditions hold in the tested neural networks. This makes it difficult to determine how much the theoretical result explains practical QAT recovery.
- The evidence for the proposed inward-gradient mechanism is still somewhat indirect. The loss landscape visualizations only examine a few sampled directions, while the Llama gradient analysis uses the quantization residual as an approximation of the outward normal direction. Can the authors provide more direct evidence that QAT recovery is actually driven by the proposed mechanism, rather than general optimization effects?
- The Hessian-based PTQ failure example is based on a simplified quadratic objective, which does not fully represent practical methods such as GPTQ. Although the experiments show PTQ degradation, it remains unclear whether the observed failures are caused by the same mechanism described in the theoretical construction.
Questions for authors
Please see the weaknesses above