- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the potentials of LLM weight compression/quantization by looking at the weight scalar distribution while neglecting all positional information. The experiment shows that Metalog quantile distribution can fit these sorted weight values precisely in a low-dimension characterization. However, the positional information remains the bottleneck for compression/quantization.
Critical strengths
- The paper demonstrates that the empirical distribution of LLM weights can be precisely approximated using a low-dimensional Metalog parameterization.
Critical weaknesses
- Metalog distributions are inherently flexible in approximating probability distributions. Therefore, a good fit to sorted LLM weights is not particularly surprising. A critical question that the paper does not address is how this property can be leveraged for practical model quantization or compression, such as reducing storage requirements or memory bandwidth for LLMs.
- The evaluation is limited to fitting sorted weights using local error metrics (MAE and rel-MSE). More global, task-relevant evaluations are needed to determine whether separating positional information from scalar weight values can provide practical benefits.
Questions for authors
- Beyond the low fitting errors achieved by Metalog distributions, could you explain how the proposed method could provide practical benefits for LLMs, given that positional information must also be preserved?
- Could you provide extra evaluations showing end-to-end improvements in LLM quantization/compression, beyond local fitting metrics? For example, could the proposed method compress a model to an average of X bits per weight while limiting the downstream accuracy degradation to Y%?