- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors study verbatim regurgitation as a function of PTQ bit widths. The findings are that larger models have more regurgitation, and that aggressive quantization reduces regurgitation. The experiments are done on the outdated pythia suite, why not OLMo2 or 3? The experiments do not control for general degradation: if low-bit PTQ degrades general capabilities of a model, it does not mean anything that verbatim regurgitation is 0. Overall, I do not see any verbatim regurgitation specifics.
Critical strengths
- The paper is easy to follow along, and the flip-rate analysis is interesting.
Critical weaknesses
- Study limited to the Pythia family, which is outdated for all of the models of the family are trained on the exact training corpus of ~300B tokens. This means that all of the models are in a heavily under-trained regime relative to modern LLMs. Moreover, if all of the model sizes are trained on the same number of tokens, the relative under-training confounds the model size experiments. Why not OLMo2 or 3?
- There is no control for regurgitation-specific behavior in the main body. I would expect very similar analysis to hold for general PTQ-induced degradation. The same confound appears in the comparison between attn vs MLP, moreover, these do not have the same number of parameters.
- I wonder if the analysis that the later layers are more sensitive to PTQ is an artifact of Pythia, or whether it holds as a general statement. Is this true for other models?
- The setting under study is particularly narrow, and it is unclear why the relationship between PTQ and verbatim regurgitation is of particular interest.
- An idea to isolate regurgitation specific degradation versus general degradation, with robustness on a per-layer basis would be to use a mixed-precision allocator à la dynamic HIGGS [malinovskii] twice under the same bit budget, once with per-layer coefficients estimated on the NLL of memorized sequences and once on general language modelling NLL. Analysing whether and where bit allocations differ would resolve to confound for me.
[malinovskii]: Pushing the Limits of Large Language Model Quantization via the Linearity Theorem, Malinovskii et al.; https://arxiv.org/abs/2411.17525
Questions for authors
Above