- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper empirically compares various existing 2-bit quantization methods and provides a recipe for how to utilize these methods to quantize existing LLMs to get good performance. Some of the methods that the paper looked at are as follows: SpinQuant, QuaRot, VPTQ, YAQA, AQLM, GSQ, and BitDistiller. The general recommendation of the paper is to use vector representations instead of scalar representations, YAQA quantization for dense models, and VPTQ for MOEs.
Critical strengths
None.
Critical weaknesses
- The paper reads more like a survey of existing methodology. It is unclear why the same conclusion cannot be reached by reading the original methods.
- The empirical suggestions provided in the paper do not offer a proper theoretical explanation of why the recipe works. The results are also not surprising.
Questions for authors
None.
2 comments
Reminder
Dear Reviewer viTJ,
A gentle reminder that the discussion closes on November 18. Our response explains what the unified protocol adds beyond the original method papers and adds a rate-distortion explanation of the 2-bit collapse (Sec. 3.2). We would be happy to answer any further questions you may have. If your concerns are resolved, we would appreciate it if you would consider updating your assessment.
Best regards,
The authors
Response to Reviewer viTJ
Thank you for reviewing our paper.
W1 (reads like a survey; conclusions available from the original papers). Our work goes well beyond a survey by providing a unified empirical evaluation of 2-bit quantization methods under unified protocol, practically relevant evaluation methods and a practical actionable recipe for researchers. Each method paper evaluates one quantizer under its own protocol, bit accounting and models, so their results cannot be compared fairly. Our findings come from experiments on each part of the quantization pipeline, including representation, calibration, recovery, and evaluation, under a unified protocol. We summarize our key contributions below. They are now supported by additional experiments that other reviewers recommended.
We believe that our results set a new standard for how 2-bit PTQ pipelines should be built and evaluated, which no single method paper can provide. Our comparisons show that bit rates must count codebooks and scales, especially on MoEs, that QA and generation must be reported separately, and that calibration and recovery data must be chosen jointly. With thorough experiments in a unified framework, we present a unified recipe for choosing the appropriate representation, calibration and recovery data and evaluation protocol for practitioners and researchers.
W2 (no theoretical explanation; results not surprising). To explain what we observe in practice, we turn to rate-distortion theory for a theoretical justification (Section 3.2). At 2 bits, a scalar grid has a fixed gap to the rate-distortion bound that a vector codebook can partly close. Measured errors on rotated real weights follow this ordering (Appendix B.1, Table 1). Equations 2-3 explain the calibration result, because data enter only through a mixture of per-domain Hessians. Held-out-domain perplexity shows the calibration-recovery coupling directly (Appendix C.2). Furthermore, we show that tuning calibration and recovery data separately, the usual practice, picks the wrong recovery data, which we predicted before running the held-out model (Table 27). We also show that evaluation on generation tasks is needed to capture the quality degradation at 2 bits. Based on our thorough and extensive experiments, practitioners now have an actionable recipe for choosing calibration and fine-tuning data and for evaluating 2-bit quantization methods.
Thank you again for your review. We hope we have addressed your concerns. If you have any further questions or comments, we would be happy to address them and engage in further discussion.