- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper starts from the observation that quantization can preserve aggregate accuracy while still changing substantial fraction of individual predictions. Based on this observation, the authors propose intervention-spectrum analysis. It projects the residual-state difference between a fp model and its quantized version onto successive prefix of a cross-fitted PCA basis. Each projection is then patched into one model, and the analysis measures how much of the full-basis decision transfer a given number of leading direction recovers. In fixed-choice tasks, the top 16 directions recover of the full-basis transfer, whereas in full-vocabulary prediction they recover only about 15%, and transfer accumulates over many directions. Based on this contrast, the authors argue that directional advantage and concentration are distinct properties.
Critical strengths
- The motivation is intuitive and convincing.
- The authors comprehensively evaluate their method across various tasks.
- The paper is clearly written and easy to follow.
Critical weaknesses
- The novelty of this paper is somewhat limited.
- The components are established methods, and the observation that similar accuracy can hide prediction changes was previously studied [1]
- It is unclear what the analysis reveals about quantization. Specifically, the method itself is agnostic to quantization. It applies to any pair of models with a state difference, and the results are nearly identical across GPTQ, AWQ, and RTN.
- The comparison with quantization methods is insufficient.
- The models used by the authors for their analysis are relatively outdated. By failing to include more recently cited papers—such as QuaRot [2], SpinQuant [3], and KVQuant [4]—in their analysis, the authors have not demonstrated whether their proposed method applies to state-of-the-art models; consequently, their analytical approach lacks sufficient validation.
[1] Dutta et al., "Accuracy is Not All You Need," 2024.
[2] Ashkboos et al., "QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs," 2024.
[3] Liu et al., "SpinQuant: LLM quantization with learned rotations," 2024.
[4] Hooper et al., "KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization," 2024.
Questions for authors
The directions are ordered by PCA on full-precision states. In Section 4, Fisher-based directions transfer noticeably more than PCA directions at the same rank. Does this mean the “broadly distributed” result could change with a different ordering? Why is PCA the right choice for measuring concentration?