acceptodds
Withdrawn from ICLR 2027

Understanding Quantization-Aware Training: Gradients at Quantized Weights Bias to the Low-Loss Basin

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Post-training quantization (PTQ) converts a trained full-precision model into low-bit weights without task-level retraining, while quantization-aware training (QAT) incorporates quantization into the training loop. Although PTQ is efficient and often accurate at moderate bitwidths, it can fail sharply at aggressive bitwidths; QAT is more expensive but can often recover the lost accuracy. We propose a unified geometric framework that explains both PTQ failure and QAT recovery. We model full-precision training as following a low-loss river inside a wider valley: a normal neighborhood of the river forms a nearly flat basin, while leaving this basin incurs a sharp loss increase. When the quantization grid is comparable to the basin width, local PTQ objectives, including rounding and Hessian-based second-order reconstruction, can select a high-loss deployed quantized point outside the basin even when nearby low-loss quantized points exist. In this regime, straight-through-estimator-based QAT has a useful bias: it evaluates gradients at the deployed quantized weights while updating latent full-precision weights, causing the gradient to sense the valley wall and acquire an inward component that steers subsequent quantized iterates back into the basin. We formalize this mechanism through a local landscape model, construct a geometric PTQ failure mode, and prove finite-time QAT recovery under the landscape and quantizer-compatibility assumptions. Experiments across vision and language models under multiple quantization schemes show PTQ degradation followed by QAT recovery, with landscape diagnostics consistent with this mechanism.

Primary area:
optimization
Keywords:
Quantization-aware training, loss landscape, LLM training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_y8oJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why PTQ can fail at low bit widths and how QAT recovers the lost performance. The authors propose a river-valley-basin loss landscape to explain this phenomenon, where quantization may move pretrained weights outside the low-loss region, while QAT uses gradients at quantized weights to guide them back. The paper constructs a Hessian-based PTQ failure example and proves finite-time QAT recovery under specific assumptions. Experiments cover ResNet, DeiT, and a 213M Llama model, including loss landscape analysis and comparisons with continued full-precision training.

Critical strengths
  1. This paper provides an interesting geometric explanation for PTQ failure and QAT recovery, rather than simply showing that QAT performs better. The theoretical analysis helps explain the different optimization behaviors of these two approaches.
  2. The experiments cover both vision and language models, multiple bit widths, and different PTQ methods. In particular, the comparison with continued full-precision training under the same additional training budget helps demonstrate that QAT's improvement is not simply due to more training steps.
Critical weaknesses
  1. My main concern is that the theoretical assumptions are not sufficiently connected to the actual experiments. Theorem 1 requires specific conditions on the basin width, quantization error, and loss geometry, but the paper does not verify whether these conditions hold in the tested neural networks. This makes it difficult to determine how much the theoretical result explains practical QAT recovery.
  2. The evidence for the proposed inward-gradient mechanism is still somewhat indirect. The loss landscape visualizations only examine a few sampled directions, while the Llama gradient analysis uses the quantization residual as an approximation of the outward normal direction. Can the authors provide more direct evidence that QAT recovery is actually driven by the proposed mechanism, rather than general optimization effects?
  3. The Hessian-based PTQ failure example is based on a simplified quadratic objective, which does not fully represent practical methods such as GPTQ. Although the experiments show PTQ degradation, it remains unclear whether the observed failures are caused by the same mechanism described in the theoretical construction.
Questions for authors

Please see the weaknesses above

Reviewer_tNM6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This is an analysis paper that proposes a geometric interpretation explaining both why low-bit PTQ fails and why STE-based QAT recovers from it. The authors start from the question of what it means if equal-budget "continued FP training + PTQ" fails to reproduce the gains of QAT. Their answer is that QAT does more than add optimization steps: it corrects quantization-specific errors that FP training cannot see.

To this end, the authors assume that the local loss landscape around a pretrained checkpoint has a river–valley–basin structure. The river is a low-loss manifold along which FP training moves. The basin is a tube around the river in which normal perturbations barely change the loss. Outside the basin are steep valley walls where the gradient points outward.

Within this framework, the authors make two claims. First, if the FP weights (w_fp) lie inside the flat basin, the Hessian at w_fp is nearly flat in the normal directions. Hessian-based PTQ methods can therefore underestimate the cost of crossing the basin boundary and select a high-loss codeword outside the basin. Second, STE-QAT evaluates gradients at the quantized weights while updating the latent FP weights. As a result, the update (negative gradient) has an inward normal component while the quantized iterate is on a valley wall, and under a fixed quantizer the quantized iterate enters the basin in finite time. The authors' main claim is that this explains why QAT recovers performance while PTQ does not.

The experiments compare RTN, PTQ, QAT on the same grid, and equal-budget continued FP training + PTQ on ResNet-20/56 (CIFAR-10), DeiT-Tiny (ImageNet), and a 213M Llama-style model (SlimPajama). PTQ increases the loss and QAT recovers it. QAT's advantage over the control is large at 2-bit and narrows at higher bitwidths, vanishing at 4-bit for the vision models. The authors interpret these results as evidence consistent with the proposed mechanism.

Critical strengths
  1. An attempt to explain the cause from a geometric perspective. The paper does not stop at the simple explanation that QAT is "more robust because it trains with awareness of quantization error." It instead tries to explain both the failure of PTQ and the recovery of QAT through the geometric structure of neural-network loss landscapes studied in recent work (river–valley–basin). It formalizes this view with explicit assumptions and theorems, which gives a framework for discussing the cause of the improvement in structural terms.

  2. Experiments across diverse architectures. The experiments cover not only small CNNs (ResNet-20/56) but also recent architectures such as DeiT and Llama. This is a meaningful effort to show that the proposed interpretation is not specific to a particular model architecture.

Critical weaknesses

Insufficient justification for the geometric explanation.

In the current version, it is unclear why the river–valley–basin geometry is needed to explain the PTQ/QAT gap, given simpler explanations that are already known.

  1. Simpler explanations already account for the main results:

    • (i) QAT directly minimizes the loss of the deployed (quantized) model, while PTQ and FP training do not.
    • (ii) Evaluating gradients at the quantized weights acts as sharpness minimization against the quantization noise, which pushes the weights toward regions that are robust to it.
    • (iii) Sharpness tends to increase late in training as the learning rate decays, which makes FP checkpoints and continued FP training more sensitive to quantization.

    Together, these predict the paper's main results:

    • PTQ raises the loss and QAT recovers it, because only QAT optimizes the loss at the quantized weights (i).
    • QAT beats equal-budget FP training + PTQ, because FP training optimizes a different objective and does not seek robustness to quantization noise (i, ii).
    • The gap is large at 2-bit and narrows at higher bitwidths (vanishing at 4-bit for the vision models), because the two objectives nearly coincide when the quantization error is small (i).
    • Continued FP training can make PTQ worse (e.g., ResNet-20 at 2-bit in Table 1: 0.497 → 0.576), because the annealing tail of the schedule increases sharpness (iii).
    • Re-applying GPTQ to the QAT latents does not reproduce the gain, because QAT is fitted to its fixed grid (i).

    The paper offers no prediction that follows from the geometric account but not from these explanations.

  2. The theory does not separate the proposed mechanism from (ii) either. Theorem 1 says that if the negative gradient points inward outside the basin and the quantization error is small relative to the basin, STE updates bring the quantized iterate into the basin. This is essentially the SAM argument: the gradient is evaluated at a perturbed point and applied to the unperturbed weights. The proof uses only ‖Q(w) − w‖ ≤ ρ, so the same result holds for any bounded perturbation, and nothing in it is specific to quantization.

  3. The strongest results are at 2-bit, where QAT's starting point (RTN on the shared grid) effectively collapses (10.4% on ResNet-20, 0.16% on DeiT-Tiny). Recovering from this is closer to retraining than to the local setting of Theorem 1, so the largest effects lie outside the regime the theory covers.

  4. The landscape diagnostics do not discriminate either.

    • The 2D plots set the direction toward the QAT solution, which QAT has optimized to have low loss, against a random direction. A "low along x, steep along y" shape is therefore expected regardless of the underlying geometry.
    • A positive gradient pairing s_k is typically expected for a locally convex loss whenever the quantized loss exceeds the latent loss.
    • Near-orthogonality is the default in high dimensions, so the ~92° angle alone does not indicate a distinct adaptation direction without a baseline

Theorem 1 is self-consistent, but it only shows that the mechanism is sufficient under its assumptions, and the experiments do not distinguish it from the simpler alternatives above. As a result, I find it hard to see what new understanding of quantization error the paper brings to the community.

Questions for authors

Q1. Prior river–valley analyses describe training dynamics, where the gradient along the river is nonzero and the directions across it are sharp. Here the structure is assumed at a trained checkpoint, with a flat normal basin and a near-stationary river. What evidence supports this at the end of training, where sharpness typically increases?

Q2. Xu et al. (2025) already show that PTQ leaves the basin when the quantization perturbation exceeds the basin radius. What does this paper add beyond that? Also, Xu et al. find that QAT lands in a separate basin. How is this consistent with the claim that QAT returns to the original basin?

Q3. Corollary 1 bounds the QAT loss only by the FP loss plus O(ε_flat), so it does not explain QAT ending below FP (Llama: 2.8430 vs. 2.8525). Only Corollary 2 covers this, and only by assuming an undertrained checkpoint, i.e., ordinary training progress. Is this assumption natural, and how do the authors separate the quantization-specific effect from it?

Reviewer_kheKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper uses a local river–valley–basin loss geometry to explain PTQ failure and QAT recovery. It shows that PTQ can place quantized weights outside a low-loss basin and proves that STE-based QAT can bring them back in finite time under specific landscape and quantizer-compatibility assumptions.

Critical strengths
  1. The paper is clearly written, and its intuitive geometric interpretation helps explain the differences between PTQ and QAT and their underlying optimization mechanisms.
  2. The paper explains when and why QAT can correct PTQ failure. Gradients evaluated at quantized weights can detect steep valley walls and guide the quantized model back toward a low-loss region.
Critical weaknesses
  1. A gap remains between the theoretical assumptions and the evidence from real models. The true normal direction depends on an unknown manifold, projection, and metric H, whereas directions orthogonal to the QAT trajectory may not correspond to the normal directions required by the theory.
  2. The language model experiments mainly use a 213M model, leaving the applicability of the proposed mechanism to larger LLMs insufficiently established.
Questions for authors
  1. Can the authors provide stronger evidence that these theoretical assumptions approximately hold in practical neural networks? The current low-dimensional visualizations do not sufficiently establish the proposed mechanism in high-dimensional parameter spaces.
  2. Can the authors provide stronger evidence that the proposed PTQ failure mechanism also explains the behavior of practical methods such as GPTQ, rather than only the idealized Hessian-based objective analyzed in the theory?
  3. Can the authors provide results on larger LLMs (~7B parameters) to demonstrate whether the proposed PTQ failure and QAT recovery mechanisms generalize to practically relevant model scales?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.