- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose a unifying framework for activation denoising in LLM layerwise compression. They recover prior methods that improve the statistic collection used in layerwise compression in special cases, and find their more general setting to improve over the special cases in many quantization configurations.
Critical strengths
1/ Authors provide a general framework for activation denoizing for LLM which recovers some prior methods, this type of research is valuable for a deeper understanding of this problem.
2/ The paper is well-written and easy to follow
3/ Authors provide experiments that supports their claims. Some of the perplexity gains are quite significant given we are only changing how activations are being collected.
Critical weaknesses
1/ It remains quite unclear to what extent practictioners would adopt such method.
2/ This framework studies activation denoising but recent works additionally considers downstream gradient covariance statistic collection as well in YAQA [1] .
3/ The framework should be generalizable to include pruning. It would largely strengthen the paper to consider a couple of algorithms like SparseGPT [2] and ALPS [3] at 50% and 2:4 sparsity settings for instance.
[1] Model-Preserving Adaptive Rounding. Albert Tseng, Zhaofeng Sun, Christopher De Sa [2] SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. Elias Frantar, Dan Alistarh [3] ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models. Xiang Meng, Kayhan Behdin, Haoyue Wang, Rahul Mazumder
Questions for authors
1/ Can the authors extend this framework and results to pruning and/or low-rank compression methods?
2/ Can the authors extend this framework and results to the gradient covariance statistic used in YAQA?
3/ Could you provide more context on how to select hyperparameters and the robustness to such selection?
Thank you for your review!
Regarding adoption by practitioners (Comment 1). We designed activation denoising to be easy for practitioners to adopt: (i) it is a per-layer weight pretransform followed by the practitioner's existing rounding solver, which remains unchanged, (ii) the pretransform is absorbed into the weights, so it introduces no inference overhead, and (iii) it requires only one additional full-precision forward pass for calibration, which is negligible compared to rounding. Moreover, our new results show that the framework also improves sequential quantization (updated Table 1), so practitioners benefit regardless of which quantization schedule they use. In response to your comment, we added this discussion in Section 7.
Regarding extending our framework to additional quantization algorithms (Comment 2 and Question 2). Adapting our robustness regularization to quantization schemes that use gradient-covariance statistics (such as YAQA [1]) is an interesting extension with several open research questions, for example, whether and how to inject noise into the gradients. We agree this is a promising direction, and in response to your question we expanded the future work discussion in Section 7 to highlight these open questions.
Beyond this, we would like to emphasize again that our approach can be applied to any existing quantization algorithm that solves the metric-weighted rounding problem in the background section (Section 3). To demonstrate this further, we provide additional experimental results with QTIP [2] (the vector quantization algorithm that YAQA [1] builds upon) in Table 3 in Appendix A.1, where we find that our framework improves the performance of QTIP as well. We believe this demonstrates that our framework indeed provides a general and effective approach to improve many existing quantization algorithms. We will provide further empirical results for QTIP in the camera-ready version.
Regarding extending our framework to other applications (Comment 3 and Question 1). Thank you for pointing out the potential to extend our approach to other compression methods such as pruning or low-rank factorization. We agree, and in principle our framework extends directly to these settings: the pretransform equivalence (Theorem 1) holds for any feasible set, not just the quantization grid. Replacing it with a sparsity (pruning) or low-rank constraint preserves the equivalence, so our robustness regularization transfers unchanged. Please note that the scope of this paper is to advance state-of-the-art LLM quantization, and we therefore focus on studying quantization errors from a robustness perspective in depth rather than extending the idea in breadth to other application areas. We thus leave the concrete solver integration and empirical validation to future work. In response to your comment, we have added this discussion (together with the suggested references) as a promising direction for future research (in Section 7). Thank you for helping us strengthen the discussion of our paper.
Regarding hyperparameter selection and robustness (Question 3). Thank you for your question. While our framework introduces the noise scales γ, λ and ρ, we find that in practice most of them can be fixed. First, we find γ=0.05 to be a robust choice across all settings we consider. Second, we parameterize the input noise scale as λ=ρ+δ with a small offset δ, since the best configurations consistently lie slightly below the diagonal (ρ<λ, see Figure 9 in Appendix A.3). Practitioners can therefore fix δ to a small value and mainly need to sweep the target-noise scale ρ, which requires only a handful of parallel quantization runs. Most hyperparameters also transfer to the Qwen-3.5 model family. In particular, the noise scale parameter γ remains effective at the same value. The main difference is the target-noise scale ρ: Qwen models favor smaller ρ than Llama models. We additionally find that parameters largely transfer across scale within a model family. We demonstrate transfer across model families and scales in additional ablations in Figures 5, 6 and 7 (Appendix A.1).
We hope that we could address all your questions to your satisfaction. Please let us know if you have any additional comments or questions.
References
[1] Albert Tseng, Zhaofeng Sun, Christopher De Sa. Model-Preserving Adaptive Rounding. 2025.
[2] Albert Tseng, Qingyao Sun, David Hou, Christopher De Sa. QTIP: Quantization with Trellises and Incoherence Processing. NeurIPS 2024.