- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors evaluate the effect of changing calibration dataset for the GPTQ quantizer.
Critical strengths
None
Critical weaknesses
-
Lack of Novelty
- The key results are well known in the quantization community (e.g. "Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox" 2024 Liu et al.) They are also inuitively fairly obvious: When the quantization problem is ~harder mistakes are more costly.
- The practical relevance of the observation that the calibration gap is larger at lower bits is limited or at least not well elaborated.
-
Study design is flawed
- As the authors note "The studied regime. The budget of 32 × 128 tokens is 64 times smaller than GPTQ’s reference setup of 128 × 2048 tokens, and short sequences limit the context seen by each activation." -- This is too far outside the practically relevant regime, especially when the main contribution are empirical results.
- "Perplexity selects the wrong source where the choice matters." is not well supported. The perplexity for selection should be evaluated OOD.
Questions for authors
None