- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes ShamAN-Q, which is a post-training quantization method that quantizes an LLM to less than 1 bit per weight. ShamAN-Q is an extension to an existing work called NanoQuant. The major contributions of ShamAN-Q are as follows:
- ShamAN-Q adapts KL-Shampoo’s curvature estimator to NanoQuant’s LB-ADMM step.
- The authors proposes non-uniform rank allocation
- The authors propose periodic curvature refresh to NanoQuant’s pipeline
- The authors propose to apply spectral tempering to the estimated curvature, which reduces trust in the dominant curvature directions to prevent inaccuracy.
Experimental evidence shows ShamAN-Q improves model performance compared to NanoQuant with a minor increase in wall-clock time. Ablation studies validate that tempering and curvature refresh can improve performance.
Critical strengths
The paper advances the sub-field of sub-1-bit post-training quantization by adapting techniques (Kronecker curvature estimators and allocating different compression budgets to different layers) from other sub-fields (above 1-bit) of PTQ. These novel adaptations (tempering and dense-curvature LB-ADMM) are technically challenging due to the problem structure of sub-1-bit PTQ. Empirical improvements upon the baseline method NanoQuant is significant.
Critical weaknesses
- The impacts of key contributions should be verified in isolation. In its current state, the paper reads more like a technical documentation, whose main goal is to present every technical detail about a product, and less like a research paper, whose main goal is to present the few key ideas/insights that enable/contribute to the success of the product.
a. Spectral tempering, or reducing the trust in the dominant curvature directions, is presented as a useful technique since the empirical Fisher is inaccurate when reconstruction error is large, which often is the case for sub-1-bit quantization. I think this explanation is believable, but an ablation study would make it more convincing. Consider sweeping reconstruction error from small to large (perhaps by varying bits per weight) while comparing the model perplexity of tempering versus no tempering. While Table 4 contains ablation studies involving tempering, it does not show why tempering is useful, only that it does. Sweeping reconstruction error, on the other hand, would reflect the “why”.
b. The idea of periodic curvature refresh is intuitive, but its impact on model perplexity and end-to-end runtime is unclear. Consider sweeping through the hyperparameter Delta from small to large and show perplexity versus runtime. Are extremely frequent refreshes impractical? When does the performance benefit of increasing refresh frequency saturate?
c. Why do both allocators in Section 4.2 allocate more bits to later layers? Consider adding more details. - The paper is difficult to read.
a. Not every reader is an expert in NanoQuant. When presenting a new technical detail about ShamAN-Q (in Section 4), consider reminding the reader why NanoQuant implemented that technical detail the way it does, and why it is technically challenging to apply NanoQuant’s implementation directly to ShamAN-Q. I understand that Section 3.1 already contains all relevant details about NanoQuant, but a gentle reminder can reduce the load on the reader’s short-term memory and help them absorb the information faster.
b. Consider using the first sentence in a paragraph to motivate the reader to read that particular paragraph. This can be especially helpful in regions of the paper that contain a lot of technical details. First-time readers often get lost in such regions and forget why they are reading. For example, line 362 could summarize what is happening in the paragraph from line 362 to line 377.
Questions for authors
See the weakness section above.
3 comments
Weaknesses, cont'd
- We thank the reviewer for this helpful perspective. In trying to balance exposition with the space constraints of the current format, we necessarily had to make a tradeoff between background material and method description. However, we can certainly improve our clarity as outlined below, following your suggestions:
- a) We can motivate our techniques by more directly tying them back to the baseline NanoQuant method.
- b) We appreciate this candid assessment. We can provide stronger thesis statements for subsections and paragraphs where they are weak. For example, also addressing the previous point (a), we could start the paragraph you identified in L366-372 with “Now, we provide the continuous and updates that optimize Eq (14), which are closed-form solutions to Sylvester equations instead of the simpler normal equations found in NanoQuant.”
I appreciate the authors for the hard work. My concerns have been addressed.
I intend to keep my already positive score.
Please treat my comments about readability as general suggestions for post-rebuttal edits. I really hope the paper can be accessible to a broader range of audiences (e.g. people who are not experts on PTQ).
It would be great if the addtional results above are included in the paper (main text or appendix) so readers who are interested can find the information.
We thank the reviewer for the careful read of our work and the direct comments. We are glad that you recognize the challenge of adapting the techniques to this difficult setting.
Weaknesses
We can add this table and explanation in the main text.
At every bpw level, while tempering does make the proxy error of the untempered metric worse, perplexity actually improves. The spread generally grows as the error grows, but it also generally gets noisier. We can plot this table as a figure with the accompanying text as part of the caption in the main text, and include the table in the appendix for completeness.
Runtime grows with refresh frequency as expected, from 3h 38m at the default setting from the paper (Δ=20) up to 5h 19m at Δ=3; Δ=0 is cheaper than Δ=3/9/18 since it does not actually need an extra forward/backward pass for the refresh.
We can update the text to include this explanation and include this table in the appendix.