acceptodds
Under review as a conference paper at ICLR 2027

Activation Denoising: A Robustness View on Parallel vs Sequential LLM Quantization

Abstract

Post-training quantization is a powerful tool for compressing large language models. The most scalable methods quantize every layer in parallel, but quantization errors then compound through the residual stream, as no layer corrects for the errors of the layers before it. Sequential quantization accounts for this error compounding by re-calibrating each layer on the already-quantized outputs of its predecessors, yielding stronger results but at the cost of a serial schedule that becomes a bottleneck at scale. As a solution, we propose parallel quantization with activation denoising, which recovers much of the sequential benefit while keeping quantization fully parallel. Rather than re-calibrating layer-by-layer, we take a robustness perspective and model the upstream error as noise, regularizing to be robust to it through a preprocessing step followed by metric-weighted rounding. Applied at every layer, this regularization forms a depth-compounding smoothness penalty that dampens how strongly quantization errors amplify through the model. Unlike orthogonal rotations commonly used in quantization, which must preserve the model's function, we multiply the weights by a more general linear transformation. We find that the two are complementary and their effects compound. Empirically, our robustness regularization recovers a significant part of sequential quantization's benefit in a single parallel pass, at a fraction of its cost. Overall, by treating compounding quantization errors as a robustness problem, we offer a principled foundation for more efficient and accurate LLM quantization at scale.

Primary area:
foundation or frontier models, including LLMs
Keywords:
post-training quantization, robustness regularization, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DjvuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose a unifying framework for activation denoising in LLM layerwise compression. They recover prior methods that improve the statistic collection used in layerwise compression in special cases, and find their more general setting to improve over the special cases in many quantization configurations.

Critical strengths

1/ Authors provide a general framework for activation denoizing for LLM which recovers some prior methods, this type of research is valuable for a deeper understanding of this problem.

2/ The paper is well-written and easy to follow

3/ Authors provide experiments that supports their claims. Some of the perplexity gains are quite significant given we are only changing how activations are being collected.

Critical weaknesses

1/ It remains quite unclear to what extent practictioners would adopt such method.

2/ This framework studies activation denoising but recent works additionally considers downstream gradient covariance statistic collection as well in YAQA [1] .

3/ The framework should be generalizable to include pruning. It would largely strengthen the paper to consider a couple of algorithms like SparseGPT [2] and ALPS [3] at 50% and 2:4 sparsity settings for instance.

[1] Model-Preserving Adaptive Rounding. Albert Tseng, Zhaofeng Sun, Christopher De Sa [2] SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. Elias Frantar, Dan Alistarh [3] ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models. Xiang Meng, Kayhan Behdin, Haoyue Wang, Rahul Mazumder

Questions for authors

1/ Can the authors extend this framework and results to pruning and/or low-rank compression methods?

2/ Can the authors extend this framework and results to the gradient covariance statistic used in YAQA?

3/ Could you provide more context on how to select hyperparameters and the robustness to such selection?

1 comment

Thank you for your review!

Regarding adoption by practitioners (Comment 1). We designed activation denoising to be easy for practitioners to adopt: (i) it is a per-layer weight pretransform followed by the practitioner's existing rounding solver, which remains unchanged, (ii) the pretransform is absorbed into the weights, so it introduces no inference overhead, and (iii) it requires only one additional full-precision forward pass for calibration, which is negligible compared to rounding. Moreover, our new results show that the framework also improves sequential quantization (updated Table 1), so practitioners benefit regardless of which quantization schedule they use. In response to your comment, we added this discussion in Section 7.

Regarding extending our framework to additional quantization algorithms (Comment 2 and Question 2). Adapting our robustness regularization to quantization schemes that use gradient-covariance statistics (such as YAQA [1]) is an interesting extension with several open research questions, for example, whether and how to inject noise into the gradients. We agree this is a promising direction, and in response to your question we expanded the future work discussion in Section 7 to highlight these open questions.

Beyond this, we would like to emphasize again that our approach can be applied to any existing quantization algorithm that solves the metric-weighted rounding problem in the background section (Section 3). To demonstrate this further, we provide additional experimental results with QTIP [2] (the vector quantization algorithm that YAQA [1] builds upon) in Table 3 in Appendix A.1, where we find that our framework improves the performance of QTIP as well. We believe this demonstrates that our framework indeed provides a general and effective approach to improve many existing quantization algorithms. We will provide further empirical results for QTIP in the camera-ready version.

Regarding extending our framework to other applications (Comment 3 and Question 1). Thank you for pointing out the potential to extend our approach to other compression methods such as pruning or low-rank factorization. We agree, and in principle our framework extends directly to these settings: the pretransform equivalence (Theorem 1) holds for any feasible set, not just the quantization grid. Replacing it with a sparsity (pruning) or low-rank constraint preserves the equivalence, so our robustness regularization transfers unchanged. Please note that the scope of this paper is to advance state-of-the-art LLM quantization, and we therefore focus on studying quantization errors from a robustness perspective in depth rather than extending the idea in breadth to other application areas. We thus leave the concrete solver integration and empirical validation to future work. In response to your comment, we have added this discussion (together with the suggested references) as a promising direction for future research (in Section 7). Thank you for helping us strengthen the discussion of our paper.

Regarding hyperparameter selection and robustness (Question 3). Thank you for your question. While our framework introduces the noise scales γ\gamma, λ\lambda and ρ\rho, we find that in practice most of them can be fixed. First, we find γ=0.05\gamma{=}0.05 to be a robust choice across all settings we consider. Second, we parameterize the input noise scale as λ=ρ+δ\lambda=\rho+\delta with a small offset δ\delta, since the best configurations consistently lie slightly below the diagonal (ρ<λ\rho<\lambda, see Figure 9 in Appendix A.3). Practitioners can therefore fix δ\delta to a small value and mainly need to sweep the target-noise scale ρ\rho, which requires only a handful of parallel quantization runs. Most hyperparameters also transfer to the Qwen-3.5 model family. In particular, the noise scale parameter γ\gamma remains effective at the same value. The main difference is the target-noise scale ρ\rho: Qwen models favor smaller ρ\rho than Llama models. We additionally find that parameters largely transfer across scale within a model family. We demonstrate transfer across model families and scales in additional ablations in Figures 5, 6 and 7 (Appendix A.1).

We hope that we could address all your questions to your satisfaction. Please let us know if you have any additional comments or questions.


References

[1] Albert Tseng, Zhaofeng Sun, Christopher De Sa. Model-Preserving Adaptive Rounding. 2025.

[2] Albert Tseng, Qingyao Sun, David Hou, Christopher De Sa. QTIP: Quantization with Trellises and Incoherence Processing. NeurIPS 2024.

Reviewer_b3muOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes activation denoising to mitigate error accumulation in parallel quantization. It injects and propagates noise through the full-precision model, uses the resulting statistics to adjust each layer’s weights, and then applies existing rounding algorithms independently to each layer. Experiments with GPTQ and QuIP# show improvements in perplexity over parallel baselines, with broadly comparable commonsense reasoning performance. The results suggest that some of the accuracy benefits of sequential quantization can be recovered while preserving parallelism.

Critical strengths
  1. Clear problem formulation and presentation. The paper clearly formulates the goal of recovering the accuracy benefits of sequential quantization without introducing sequential dependencies between layer-wise rounding steps. The method is well explained and easy to follow.

  2. Compatibility with existing methods. The proposed approach can use existing rounding algorithms such as GPTQ. The denoising transformation is absorbed into the weights, making the method straightforward to integrate without additional inference operations.

  3. Thorough ablation studies. The paper examines noise propagation, noise magnitude, and interactions with rotations. These analyses help establish the contribution of the method’s individual components.

Critical weaknesses
  1. The practical motivation needs stronger justification. Quantization is an offline operation that may be performed only once for a given deployment, whereas the resulting model’s quality affects every subsequent inference. For the 13B QuIP# setting, the paper compares 306 minutes of sequential quantization on one GPU with a projected 41 minutes for the proposed method on eight GPUs. Given the substantial costs of pretraining and repeated inference, I am not fully convinced that quantization time is a sufficiently important bottleneck in the settings considered. Although the method is interesting, the paper would benefit from concrete use cases that justify prioritizing faster quantization and clarify the trade-offs among elapsed time, GPU resources, and final model quality.

  2. Limited evaluation across model families. All evaluated models belong to the Llama family. Since the method appears broadly applicable, experiments on another family would strengthen the evidence that its benefits extend beyond the architectures currently evaluated.

Questions for authors
  1. Could the authors identify concrete workflows in which sequential quantization time is a significant bottleneck? In those settings, how should users weigh the reduction in elapsed time against remaining quality gap relative to sequential quantization?

  2. Do the improvements extend to other model families, particularly Qwen? Would the same noise configuration and hyperparameter selection procedure remain effective?

2 comments

Thank you for your review!

Regarding practical motivation (Comment 1 and Question 1). We agree that quantization is an offline operation that may be performed only once. The problem we address is that while sequential quantization provides better quality, it is highly inefficient at frontier scale, where the most expensive vector quantization methods would already take multiple weeks to run sequentially. For example, the authors of AQLM [1, Appendix D] report that quantizing a 70B model on a single GPU would take 10 to 14 days. Similarly, extrapolating from our own QuIP# Llama quantization runtimes to a 2-trillion-parameter model, we expect sequential quantization to take on the order of two to three weeks. While our extrapolation is certainly simplified (for example, it ignores that expert quantization in MoE models can be parallelized), the quantization time remains substantial even if performed only once, motivating objectives that parallelize well. We believe this is also why most state-of-the-art vector quantization techniques (including QuIP#) only implement parallel quantization. Note that we implement a sequential version of QuIP# solely for demonstration purposes in our experimental evaluation.

Please note that it is not feasible for us to provide experiments at frontier scales to further motivate this. Instead, we would like to highlight that our quantization framework (Section 4.1) also improves the quality of sequential quantization. To demonstrate this, we provide additional experiments where we sweep ρ\rho and λ\lambda using the upstream error ηseq\eta_{seq} under sequential quantization. We provide results in the updated Table 1. Critically, we find that our approach consistently improves over the existing sequential quantization baselines as well. In particular, our framework improves sequential quantization most at 2-bit vector quantization, where the sequential asymmetric correction (the sequential baseline) offers only limited improvements.

We further believe our additional experiments demonstrate that our contributions yield meaningful and significant performance improvements for both parallel and sequential quantization and in particular hold beyond the discussion on their efficiency. In response to your comment, we expanded the motivation in Section 6.1 and added a corresponding discussion to Section 7. Thank you for helping us improve the motivation of our work.

Regarding additional model families (Comment 2 and Question 2). Thank you for proposing to expand our evaluation to additional model families. In response to your comment, we provide additional experiments for the Qwen-3.5 model family (0.8B, 2B, 4B, 9B) in Table 2 in Appendix A.1. Specifically, we follow the experimental setup for Llama models and run a hyperparameter grid search at two bit widths, quantizing each model 250 times in total. Our results consistently demonstrate substantial perplexity improvements across all Qwen models, especially at 2-bit weight quantization. We believe the additional empirical results strengthen our main findings. Thank you for helping us improve the experimental evaluation of our paper.

Regarding robustness of hyperparameters across model families (Question 2). Thank you for your question. We find that most hyperparameters transfer to the Qwen-3.5 model family. In particular, the noise scale parameter γ\gamma remains effective at the same value. The main difference is the target-noise scale ρ\rho: Qwen-3.5 models favor smaller ρ\rho than Llama, and we adapted the ρ\rho grid toward lower values for this family. We additionally find that parameters largely transfer across scale within a model family. We demonstrate transfer across model families and scales in additional ablations in Figures 5, 6 and 7 (Appendix A.1).

We hope that we could address all your questions to your satisfaction. Please let us know if you have any additional comments or questions.


References

[1] Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, Dan Alistarh. Extreme Compression of Large Language Models via Additive Quantization. ICML 2024.

Thank you for the detailed response and additional experiments, which have fully addressed my concerns. I still think this is a solid paper and will keep my weak acceptance rating.

Reviewer_N1jgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a parallel post-training quantization method with activation denoising to address the accumulation of quantization errors across layers while avoiding the computational overhead of sequential quantization. The method models upstream quantization errors as input noise and improves robustness through a preprocessing step and metric-weighted rounding. It further introduces a general linear transformation that complements existing orthogonal rotation techniques. Experimental results show that the proposed method recovers a substantial portion of the accuracy benefits of sequential quantization while maintaining the efficiency of parallel quantization.

Critical strengths
  1. This paper focuses on post-training quantization, which is efficient in the calibration stage and easy to deploy. The motivation of this work is interesting, as it addresses the efficiency issue of sequential quantization.

  2. This paper is clear in writing and easy to follow.

  3. In the experimental results, this paper improves results in Llama-2 / 3.2 compared with the baselines.

Critical weaknesses
  1. My main concern is the performance of the proposed quantization method. Current frontier quantization methods typically employ end-to-end optimization, such as SpinQuant. These methods generally achieve substantially better performance than simply sequential or parallel quantization, while requiring only a few GPUs for optimization. As a result, training cost is no longer a major bottleneck in quantization, and further reducing this cost may offer only limited practical benefits.

  2. The effectiveness of the quantization method on modern long-context tasks deserves further investigation. For example, modern LLMs have demonstrated strong reasoning capabilities. However, this work primarily focuses on PPL and simple zero-shot tasks and lacks further exploration of more challenging long-context / response tasks, particularly those requiring complex reasoning. This can be written in the limitations.

Questions for authors

N/A

1 comment

Thank you for your review!

Regarding end-to-end optimization methods (Comment 1). Please note that end-to-end optimization methods such as SpinQuant are orthogonal approaches, since they optimize a weight transformation in a preprocessing step separate from the actual weight quantization (which we focus on). In particular, we find that our approach also yields perplexity improvements when using end-to-end learned SpinQuant rotations (see ablation in Figure 3, and Table 4 in Appendix A.4.1).

Agnostic to the use of end-to-end optimization, we tackle the problem that sequential weight quantization improves performance but constitutes a time bottleneck in practice. This problem exists in particular at frontier scale, where the most expensive vector quantization methods alone would already take multiple weeks to run sequentially. For example, the authors of AQLM [1, Appendix D] report that quantizing a 70B model on a single GPU would already take 10 to 14 days.

If efficiency is not a concern, we show in additional experiments that our framework (Section 4.1) offers significant performance improvements for sequential quantization as well (see updated Table 1). To demonstrate this, we sweep ρ\rho and λ\lambda using the upstream error ηseq\eta_{seq} under sequential quantization. Critically, we find that our approach consistently improves over the existing sequential quantization baselines as well. In particular, our framework improves sequential quantization most at 2-bit vector quantization, where the sequential asymmetric correction (the sequential baseline) offers only limited improvements. We believe our additional experiments demonstrate that our contributions yield meaningful and significant performance improvements for both parallel and sequential quantization and in particular hold beyond the discussion on their efficiency.

In response to your comment, we added a corresponding discussion to Section 7. Overall, our method significantly expands the Pareto-frontier of efficient and accurate LLM quantization, as also highlighted by other reviewers. Beyond this, we believe that our robustness perspective (treating quantization errors as a robustness problem and making LLMs robust to them) constitutes a meaningful contribution, and provides valuable insights for the quantization community with impact beyond empirical performance improvements.

Regarding long-context tasks (Comment 2). We would like to point out that perplexity and zero-shot accuracy are standard evaluation metrics in the LLM quantization literature, and reporting them allows us to effectively compare against existing works. In particular, please note that LLM quantization has not been solved so far and remains a challenging optimization problem under the metrics we consider. Nevertheless, we agree that long-context and complex reasoning tasks constitute additional, more challenging problems, and we added this discussion in Section 7 as suggested. Thank you for helping us strengthen the discussion of our paper.

We hope that we could address all your questions to your satisfaction. Please let us know if you have any additional comments or questions.


References

[1] Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, Dan Alistarh. Extreme Compression of Large Language Models via Additive Quantization. ICML 2024.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We want to thank all reviewers for their valuable feedback and for acknowledging the novelty of our work.

We summarize the core points addressed in our rebuttal:

  • Is sequential quantization time a practical bottleneck? (N1jg, b3mu) Yes. Although quantization is done only once, running state-of-the-art vector quantization sequentially takes days to weeks at frontier scale (e.g., 10 to 14 days for a 70B model with AQLM [1, Appendix D]). This is why most such methods only implement parallel quantization. Our method stays fully parallel while recovering much of the sequential benefit.

  • Are the benefits limited to quantization efficiency? (N1jg, b3mu, Djvu) No. Our framework also consistently improves sequential quantization, and even makes sequential 2-bit vector quantization effective (updated Table 1). Practitioners therefore benefit under either (parallel or sequential) schedule.

  • Does the framework extend beyond the quantization settings we consider? (Djvu) Yes. Our objective does not depend on how the compressed weights are constrained (the quantization grid is just one choice of feasible set), so it applies in principle to pruning and low-rank compression. Extending it further to gradient-covariance statistics as in YAQA [2] raises interesting open questions for future research. We discuss both in the extended future work section (Section 7).

  • Do results generalize across models and algorithms? (b3mu, Djvu) Yes. New experiments on the Qwen-3.5 family (0.8B, 2B, 4B, 9B) and with QTIP [3] show consistent improvements, and hyperparameters largely transfer across model families and scales (Appendix A.1).

We changed the initial submission in response to their comments as follows (revisions marked in blue in the updated manuscript):

  • Additional experiments demonstrating that our framework also improves over the sequential quantization baselines (if quantization time is not a concern) in response to Reviewers N1jg, b3mu and Djvu (updated Table 1 and Paragraph 1 in 6.1).
  • Additional experiments on the Qwen-3.5 model family (0.8B, 2B, 4B, 9B) in response to Reviewer b3mu (Table 2 in Appendix A.1).
  • Additional experiments using an additional quantization algorithm (QTIP) in response to Reviewer Djvu (Table 3 in Appendix A.1).
  • Additional ablation studies on hyperparameter transfer across model families and scales in response to Reviewers b3mu and Djvu (Figures 5, 6 and 7 in Appendix A.1).
  • We expanded the practical motivation in Section 6.1 (second paragraph) and added an efficiency discussion to Section 7 in response to Reviewers N1jg and b3mu.
  • We expanded the discussion section in response to Reviewers N1jg and Djvu (Section 7).

We believe the additional empirical evidence and discussion strengthens our main contributions and demonstrates the effectiveness of our proposed quantization framework.


References

[1] Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, Dan Alistarh. Extreme Compression of Large Language Models via Additive Quantization. ICML 2024.

[2] Albert Tseng, Zhaofeng Sun, Christopher De Sa. Model-Preserving Adaptive Rounding. 2025.

[3] Albert Tseng, Qingyao Sun, David Hou, Christopher De Sa. QTIP: Quantization with Trellises and Incoherence Processing. NeurIPS 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.