- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper build upon SPHQuant: a weight-only vector quantization method that represents each group of weights with a vector, sign, and radius. The paper improves it by applying 2 techniques:
- Energy-Stratified Grouping: permutes channels into groups of similar importance (sorted within each group). This ensures that we don't represent multiple channels with wide range of importance using the same vector look up table (LUT)
- Adaptive Bit Split: finding the best bit width for radius (i.e., precision of radius) and vector LUT (i.e., how many vectors)
The authors apply the approach on a wide range of tasks and models and show that they beat SoTA in 2-bit quantization
Critical strengths
- Clear Figure 2 summarizing the solution
- To the best of my knowledge: permuting channel weights (done in ESG) is novel
- Evaluated a wide range of model sizes and 2 model families on a wide range of tasks.
- Compared with a wide range of tasks and consistently beats them on most tasks.
- Ablation studies that clearly show the contribution of each component of the solution
- detailed systems analysis: TTFT, TPS, Speedup, Model Size, even showing when the model is slower on TTFT compared to some techniques like AWQ but faster than all other 2-bit VQ techniques
- highest end-to-end speedup after comparing with wide range of quantization approaches
Critical weaknesses
- Don't really have a critical weakness in mind.
Questions for authors
Suggestions:
- Figure 1: Please add to each plot, a horizontal dashed line showing FP16 accuracy
- Figure 2: Inside the "Reconstruction" box and under "Three Candidates", the font of "w" is consistent with the rest of the Figure, and "cz" should be "c_{z}"
- Line 196: Please define v
- Line 198: here, |w|, denotes absolute values, rather than norm, right? Clarifying would be helpful.
- Line 202: please add citation to the paper that proposes spherical k-means as it seems to be a seminal paper. when I searched online, it seems to be this:
Dhillon, Inderjit S., and Dharmendra S. Modha. "Concept Decompositions for Large Sparse Text Data Using Clustering." Machine Learning, vol. 42, no. 1/2, 2001, pp. 143–175. SpringerLink, doi:10.1023/A:1007612920971.
- Line 205: can you clarify how many bits are used to store offset and step?
- Equation 3: "v" here is to store a sign bit for each element in a subvector
- Line 300: for those who are not familiar with SPHQuant, please briefly describe how you optimize. Do you do some training?
3 comments
I have read all the reviews and the rebuttals. I believe that the authors have addressed the reviewers concerns. The results in the rebuttal for larger models or applying ESG on other quantization approaches are impressive. I would like to keep my score.
Dear Reviewer iphk,
Thank you for your thoughtful review and positive feedback on our rebuttal and additional experiments. We sincerely appreciate your time and careful consideration throughout the review process.
Best regards, The Authors
Response to Reviewer iphk (denoted as R3)
A3-1: Thank you for the suggestion. We have added a horizontal dashed line indicating the full-precision (BF16) reference score to each accuracy plot in Figure 1 in the revised PDF.
A3-2: We have standardized the font of the weight symbols and corrected the notation to cz in Figure 2 in the revised PDF.
A3-3: We have explicitly defined v as the subvector dimension, with v=8 in our experiments, at its first occurrence in the revised PDF.
A3-4: Yes, ∣w∣ denotes the elementwise absolute value, rather than a norm. We have explicitly clarified this distinction in Section 3.1 of the revised PDF.
A3-5: We have added the citation to Dhillon and Modha (2001) [1] when introducing spherical k-means in Section 3.1 of the revised PDF.
A3-6: The offset and step are each stored in FP16 (16 bits each), contributing an additional 0.25 BPW at a group size of 128 weights. We have explicitly stated this in Section 3.1 of the revised PDF.
A3-7: Yes, that is correct. Each weight in a subvector has one sign bit, requiring v sign bits in total. We have clarified this alongside Equation (3) in the revised PDF.
A3-8: Yes, following SPHQuant, we perform blockwise optimization on calibration data, updating radius and codebook parameters to minimize the MSE between quantized and full-precision block outputs while keeping discrete assignments fixed. We have expanded Section 3.4 in the revised PDF to make this procedure clearer and self-contained for readers unfamiliar with SPHQuant.
[1] Dhillon, Inderjit S., and Dharmendra S. Modha. "Concept Decompositions for Large Sparse Text Data Using Clustering." Machine Learning, vol. 42, no. 1/2, 2001, pp. 143–175. SpringerLink, doi:10.1023/A:1007612920971.