acceptodds
Withdrawn from ICLR 2027

The Devil Is in the Reconstruction Loss Scale: Rethinking Optimization in LLM Quantization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Post-training quantization (PTQ) methods typically quantize a pre-trained large language model (LLM) sequentially using a small set of calibration samples, achieving high data and computational efficiency. Specifically, the model is partitioned into a series of units, such as transformer blocks, with one unit quantized at each stage. State-of-the-art PTQ methods are predominantly learning-based, optimizing auxiliary quantization parameters (e.g., scaling factors, rotation matrices, clipping thresholds, and adapters) via gradient descent to minimize a reconstruction loss, thereby mitigating the impact of outliers and reducing quantization errors. In learning-based PTQ, a common practice is to use mean squared error (MSE) as the reconstruction loss function, yet its induced optimization behavior remains largely unexplored. In this work, we take a holistic view of sequential quantization and systematically investigate how optimization evolves from the first quantization stage to the last, aiming for a deep understanding of optimization in learning-based PTQ schemes. Through extensive empirical studies spanning representative learning-based PTQ methods, model families, model scales, architectures, and quantization settings, we consistently uncover *Optimization Imbalance*: reconstruction loss magnitudes vary dramatically across stages, accompanied by highly uneven gradient magnitudes and parameter updates under MSE. We term the cross-stage range of loss magnitudes the *reconstruction loss scale*. Our analysis reveals that MSE translates the unexpectedly large reconstruction loss scale into highly uneven gradient magnitudes, which in turn lead to uneven optimization strength across quantization stages. This finding suggests a general principle for improving learning-based PTQ: optimization strength across stages should be decoupled from the reconstruction loss scale. Theoretically, we show that root mean squared error (RMSE) naturally realizes this principle through implicit gradient normalization, thereby enabling balanced optimization across quantization stages. Moreover, RMSE can be naturally defined over different feature groupings, yielding four variants at the sample, channel, token, and element levels. As a drop-in replacement for MSE, RMSE consistently improves quantization performance across language modeling, commonsense reasoning, mathematical reasoning, and coding tasks, with all four variants outperforming MSE and the gains becoming increasingly pronounced under more aggressive quantization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, post-training quantization, learning-based quantization, sequential quantization, optimization imbalance

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9T6dOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates optimization imbalance in PTQ. The authors observe that MSE optimization will leads to sharp gradient and weights update, so they propose to replace MSE with RMSE, which normalizes the output-space gradient magnitude independently of the reconstruction loss scale. Extensive experiments across multiple LLM families, quantization settings, and PTQ methods demonstrate consistent improvements.

Critical strengths
  1. Replacing MSE with RMSE is straightforward and requires minimal implementation changes. And the theoretical derivation clearly explains its output-gradient normalization property.
  2. The analysis of reconstruction losses across different quantization stages provides a useful perspective on existing methods development.
  3. The consistent improvements are encouraging and the reported computation overheads are also negligible.
Critical weaknesses
  1. The central argument is that MSE will cause gradient magnitude differences and imbalanced parameter updates, which RMSE mitigates this issue through gradient normalization. However, adaptive optimizers such as Adam are approximately invariant to uniform gradient rescaling under certain conditions. Therefore, normalizing the output-space gradient is not equal to more balanced weight updates.
  2. While RMSE consistently improves average accuracy, Table 4 shows that under W2A16, HumanEval accuracy remains at 0%, while MBPP improves only from 0% to 0.26%. This indicates that balancing reconstruction gradients may be insufficient to preserve downstream capabilities under extreme low-bit.
  3. The authors measures optimization strength through relative weight updates and interprets smaller updates as insufficient optimization. However, smaller gradients or parameter updates may also indicate smaller reconstruction errors or closer to a local optimum. Average gradient magnitudes across stages is not equal to more effective optimization.
Questions for authors
  1. Can you clarify the role of optimizer scale invariance, moment normalization and gradient clipping in the observed improvements?
  2. Why does RMSE provide limited recovery on downstream tasks under 2-bit despite the reported improvement in gradient balance?
  3. Can you provide convergence curves or measurements of relative reconstruction error reduction across quantization stages to validate that a smaller relative weight update is equal to insufficient optimization?
Reviewer_LH7xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates optimization imbalance in sequential learning-based post-training quantization. It observes substantial differences in reconstruction losses, gradients, and parameter updates across stages, and proposes replacing MSE with four grouped RMSE variants. The analysis establishes their output-gradient normalization property. Experiments across multiple quantization frameworks, model families, and bit widths report improved perplexity and downstream performance with minimal additional calibration cost.

Critical strengths
  • A useful diagnostic perspective. Examining reconstruction losses, output gradients, parameter gradients, and relative updates together highlights optimization behavior as a relevant consideration in sequential quantization.
  • Promising results from a simple modification. The evaluation covers multiple frameworks and model families with little additional calibration cost. For example, under OmniQuant W2A16, Qwen3.5-27B improves from 60.45% to 70.96% average commonsense accuracy, suggesting meaningful practical value.
Critical weaknesses
  • The proposed mechanism is insufficiently isolated. Constant output-gradient norms do not establish adequate parameter optimization, which also depends on the network Jacobian and optimizer. Moreover, grouped RMSE changes the relative weighting of reconstruction errors. Without targeted scaling controls and within-stage convergence analysis, the improvements cannot be confidently attributed to reduced cross-stage optimization imbalance.

  • The FlatQuant baseline requires clarification. Its official implementation applies loss = loss / loss.clone().detach() before backpropagation, yielding different gradient scaling from the ordinary MSE analyzed in Eq. (5). If retained, this operation must be incorporated into the analysis of Figure 3; if removed, the baseline departs from the official optimization procedure. This ambiguity affects both baseline fidelity and the claimed generality across methods.

Questions for authors
  • Was FlatQuant’s detached loss normalization retained for both MSE and RMSE? Does Figure 3 measure gradients of the actual training objective, and do the improvements persist against the original official configuration?

  • Do the gains persist against MSE with matched tuning budgets for learning rates and loss scaling? Can targeted ablations distinguish cross-stage scaling from group reweighting and demonstrate that the early stages were insufficiently optimized?

  • How many independent calibration and optimization runs support the principal results, and how were the grouping defaults selected? Please report uncertainty for the paired performance differences.

Reviewer_LorXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the optimization behavior of reconstruction-based post-training quantization (PTQ) for large language models. The authors identify a phenomenon termed Optimization Imbalance, where reconstruction losses, gradient magnitudes, and relative parameter updates vary substantially across sequential quantization stages under the commonly used mean squared error (MSE) objective.

The paper argues that MSE couples the magnitude of the reconstruction error with the optimization signal, resulting in uneven optimization across quantization stages. To mitigate this issue, the authors propose replacing MSE with root mean squared error (RMSE), whose output-space gradient norm is independent of the reconstruction loss magnitude. The authors further generalize this formulation into four variants based on sample-, channel-, token-, and element-level grouping, with the element-level formulation reducing to the L1 loss.

The proposed approach is evaluated across five LLM configurations, four representative learning-based PTQ methods (OmniQuant, DuQuant, FlatQuant, and SliderQuant), and multiple weight-only and weight-activation quantization settings. Experiments demonstrate consistent improvements in perplexity and average downstream accuracy, particularly under aggressive quantization, with negligible additional quantization time and memory overhead.

Critical strengths

S1. A systematic and practically relevant investigation of reconstruction optimization in learning-based PTQ.

The paper examines optimization dynamics throughout the complete sequential quantization process and identifies substantial cross-stage variation in reconstruction losses, gradients, and parameter updates. Importantly, the observations are replicated across multiple model families, architectures, quantization settings, and PTQ frameworks.

This analysis is valuable because reconstruction objectives are central to many learning-based PTQ algorithms, yet their optimization behavior is often treated as an implementation detail. The paper highlights a potentially general issue that extends beyond any particular quantization parameterization.

S2. Consistent empirical improvements with minimal additional complexity.

The proposed RMSE objectives achieve consistent improvements over MSE across a diverse set of evaluations. The gains are particularly pronounced under aggressive quantization, where improving reconstruction optimization is most consequential.

The experimental evidence also extends beyond perplexity to commonsense reasoning, mathematical reasoning, and coding tasks. Furthermore, RMSE can be incorporated into existing PTQ frameworks without introducing additional trainable parameters or substantial runtime and memory overhead.

The combination of broad empirical effectiveness and implementation simplicity represents a meaningful practical contribution.

Critical weaknesses

W1. The causal explanation connecting output-space gradient normalization to improved optimization is insufficiently established.

The central theoretical argument is that RMSE eliminates the dependence of the output-space gradient norm on the reconstruction loss magnitude, thereby balancing optimization across sequential quantization stages.

While this property is mathematically correct, the actual optimization trajectory is determined by gradients with respect to the trainable quantization parameters and their interaction with the optimizer.

For the ungrouped RMSE objective, the parameter gradient is a positive scalar multiple of the corresponding MSE gradient.

The effect of this scaling on parameter updates depends on the optimization algorithm. In particular, adaptive optimizers such as Adam exhibit partial invariance to gradient rescaling, although this invariance is not exact for time-varying scaling factors and in the presence of optimizer regularization or numerical stabilization.

Therefore, a constant output-space gradient norm does not by itself establish more effective or appropriately balanced optimization in parameter space.

Although Figures 2 and 3 report parameter-space gradients and relative parameter updates, the experiments do not sufficiently isolate gradient normalization from alternative explanations involving effective learning rates, optimizer dynamics, or loss scaling.

More targeted comparisons with MSE under gradient normalization, learning-rate adjustment, and alternative optimizers would be important for substantiating the proposed mechanism.

W2. The experimental analysis does not adequately distinguish cross-stage gradient balancing from alternative loss-function effects.

The paper introduces four RMSE variants and attributes their effectiveness to a shared gradient normalization principle. However, the variants differ in more than their overall gradient scale.

In particular, channel- and token-level RMSE introduce different relative weighting across reconstruction-error groups, while element-level RMSE reduces to the L1 loss. These formulations can change the relative influence of outliers, channels, and tokens, potentially improving quantization through mechanisms other than cross-stage balancing.

The reported differences among RMSE variants further suggest that the grouping-dependent weighting of reconstruction errors has a substantial effect on performance.

However, the experiments do not compare RMSE with sufficiently informative alternatives such as MSE with explicit gradient normalization, Huber loss, Smooth L1, or other normalized reconstruction objectives. As a result, it remains unclear how much of the observed gain is specifically attributable to the proposed cross-stage optimization principle, rather than more general properties of alternative reconstruction losses.

These controls would help establish whether the proposed formulation is uniquely effective or whether the observed improvements primarily reflect a broader loss-design or optimization phenomenon.

Questions for authors

Q1. Optimizer dependence and the proposed balancing mechanism (related to W1).

Could the authors clarify how the proposed output-space gradient normalization translates into parameter updates under the optimizers used in the evaluated PTQ frameworks?

In particular, given that the ungrouped RMSE gradient is a scalar multiple of the MSE gradient, how does the proposed explanation account for the partial scale invariance of adaptive optimizers such as Adam?

A comparison with MSE using explicit gradient normalization or adjusted optimization hyperparameters would help determine whether the improvements arise specifically from the RMSE objective or from effective gradient rescaling.

Q2. Distinguishing cross-stage balancing from alternative reconstruction objectives (related to W2).

Could the authors compare RMSE against MSE with explicit gradient normalization and representative robust reconstruction losses, such as L1, Smooth L1, or Huber loss, under matched optimization settings?

Since RMSE-Element is equivalent to L1 and the different grouping variants assign different weights to reconstruction errors, these comparisons would help clarify whether cross-stage balancing is the primary source of improvement.

Q3. Hyperparameter sensitivity and robustness.

The manuscript states that MSE and RMSE use identical optimizer settings and reconstruction budgets. Have the authors investigated whether the performance gap persists after tuning the learning rate or related optimization hyperparameters for the MSE baseline?

Additionally, could the authors provide evidence of robustness across different calibration subsets or random seeds, especially in relatively mild quantization settings where the observed gains are small?

These results would help establish whether the improvements reflect a robust advantage of the proposed objective rather than sensitivity to the original optimization configuration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.