acceptodds
Under review as a conference paper at ICLR 2027

Why does Post-Training Quantization Work?

Abstract

Post-training quantization (PTQ) compresses large language models (LLMs) by reducing weight precision and can largely preserve downstream task performance, even though they are never exposed to quantization error during pretraining. A common intuition is that low-bit formats such as NVFP4 induce only small errors in each weight matrix, so errors accumulate slowly through the model. However, we show that this explanation alone does not fully explain PTQ's empirical effectiveness. Specifically, when quantizing randomly initialized and pretrained models to the same precision, we observe that their weight quantization errors are nearly identical, yet randomly initialized models exhibit substantially larger hidden-state error than pretrained models. We attribute this discrepancy to a counteracting mechanism present in pretrained networks: the error newly introduced by a layer tends to oppose the error propagated from the preceding layer. These two components partially cancel, substantially slowing error accumulation across layers. Moreover, at the output layer, the probabilities of high-ranked tokens are more robust to the propagated errors. We quantify these mechanisms and validate our findings across multiple model families and quantization settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Quantization, Post-Training Quantization, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iDwjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors discussed "How post-training quantization works for pre-trained models". To this end, the authors compared pre-trained and randomly initialized modelsm and showed that even with comparable weight deviation, the final results are very different. The authors analyzed the reasons behind this phenomenon.

Critical strengths

The authors' view on post-training quantization is interesting. While most of papers tackle improving the performance of quantization, the authors considered one essential problem: why post-training quantization works.

Critical weaknesses

While the point of view is very interesting, I'm not sure whether this view is worthwhile to investigate, so I cannot judge the value of this study.

To my knowledge, the different behaviors of pre-trained and randomly initialized models are somewhat expected. Pre-trained models are trained to convergence, possibly with sharpness-aware minimization. Then, the task loss degradation incurred by the weight perturbation, which can be approximated as ΔwTHΔw\Delta \mathbf{w}^{T} \mathbf{H} \Delta \mathbf{w} using second-order Taylor expansion, should be small with the small weight perturbation. Whereas, this approximation does not hold for randomly initialized models.

So, while I acknowledged some interesting findings (e.g., counteraction), I think the authors tried to solve some trivial problem in a very difficult way, and I am uncertain whether this direction has merits.

Questions for authors

See weaknesses above.

6 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer iDwj

We thank you for your time and for highlighting that asking why PTQ works is an essential question. Below, we would like to clarify a possible misunderstanding regarding our setup and provide new experimental results to address your concerns.

First, we clarify a possible fundamental misunderstanding: second-order Taylor expansion cannot quantitatively explain 4-bit PTQ. The breakdown of this intuition directly supports the necessity of our study: To understand PTQ robustness, we must investigate the internal dynamics of the model, rather than relying on an end-to-end loss approximation.

Moreover, comparing random and pretrained models is not our goal, but only a motivating example showing that active error counteraction is a learned property acquired during training; our research goal is to understand the internal mechanism of PTQ robustness rather than only explaining the loss change.

1. Open-recipe models in our study are trained without SAM (Sharpness-Aware Minimization)

All open-recipe models in our study (Pythia [1], OLMoE [2], and OLMo 3 [3]) are pretrained using Adam/AdamW without SAM, yet 4-bit PTQ works effectively across all of them. Therefore, SAM cannot explain why PTQ works in our study.

2. Second-order Taylor expansion cannot predict the 4-bit loss change

Even when we compute the exact second-order Taylor expansion directly on each evaluated text and token itself, it still fails to approximate the actual 4-bit loss change ΔCEi\Delta{\rm CE}_i (measured via WAPE:=∑i∣ΔCE^i−ΔCEi∣∑i∣ΔCEi∣{\rm WAPE} := \frac{\sum_i |\widehat{\Delta{\rm CE}}_i - \Delta{\rm CE}_i|}{\sum_i |\Delta{\rm CE}_i|}):

Table: Prediction error (WAPE↓{\rm WAPE} \downarrow) of exact 2nd-order Taylor approximations for actual 4-bit loss change ΔCE\Delta{\rm CE} on WikiText-103 (Qwen3-4B, weight-only NVFP4, 128 texts / 65,536 positions).

ΔCE\Delta{\rm CE} Taylor Approximation Method4-bit (NVFP4) Pred. Error (WAPE↓{\rm WAPE} \downarrow)
2nd-order only (12Δw⊤HΔw\frac{1}{2}\Delta w^\top H\Delta w)690.0%
Full 2nd-order Taylor (∇L⊤Δw+12Δw⊤HΔw\nabla \mathcal{L}^\top\Delta w + \frac{1}{2}\Delta w^\top H \Delta w)593.4%

Even when computing a token's own exact Hessian on WikiText-103, it can completely miss severe actual degradations: For example, on token "lt", the actual CE{\rm CE} jumps from 0.0070.007 (BF16) to 2.39 (W4), yet the 2nd-order and full Taylor expansions predict only ΔCE^=+0.16\widehat{\Delta{\rm CE}} = +0.16 and +0.21+0.21, respectively. Similarly, on token "9", the actual CE{\rm CE} jumps from 2.9×10−92.9\times 10^{-9} (BF16) to 1.89 (W4), while the two Taylor estimates predict only +1.2×10−8+1.2\times 10^{-8} and +1.5×10−8+1.5\times 10^{-8}.

Therefore, the second-order Taylor expansion fails severely in the 4-bit regime: such a massive WAPE error (593.4593.4%–690.0690.0% and up to >108×>10^{8}\times underestimation for specific tokens) directly supports that 4-bit perturbations exceed the effective range of the second-order Taylor expansion. This aligns with [4], which also shows that second-order approximations inaccurately estimate quantization loss changes, making why 4-bit PTQ works genuinely non-trivial.

3. Activation quantization has no weight Taylor expansion, yet shares the same counteraction

Under activation-only quantization (A4), all weights remain unchanged (Δw=0\Delta w = 0), so a weight-based Taylor or Hessian analysis predicts zero loss change and cannot be applied at all. Yet 4-bit activation-only quantization exhibits the exact same cross-layer error counteraction behavior as weight-only quantization (Figure 6 in Section 5 of our paper). This proves that PTQ robustness arises from how the network internally processes hidden-state errors across layers, which a weight-level Hessian cannot capture.

4. Why the Taylor expansion argument underscores the necessity of our mechanism study

The flat loss landscape intuition that a second-order Taylor expansion easily explains PTQ is a natural expectation, but its quantitative breakdown above demonstrates precisely why a deeper mechanistic study is necessary. In summary, understanding why low-bit PTQ works cannot stop at a black-box, end-to-end scalar loss approximation; we must dive into the inner properties of the network to reveal how errors cancel across layers and why top-token predictions remain stable.

We hope our responses and new experimental results clarify our methodology and core findings, and fully address your points. We will integrate all updates into the final version of the manuscript and are happy to discuss further.

References:
[1] Biderman et al., Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling, ICML 2023.
[2] Muennighoff et al., OLMoE: Open Mixture-of-Experts Language Models, ICLR 2025.
[3] Team Olmo et al., OLMo 3, arXiv:2512.13961, 2025.
[4] Hu et al., Identifying Sensitive Weights via Post-Quantization Integral, arXiv:2503.01901, 2025.

Thank you for the detailed response and the additional experiments.

While I appreciate the authors' efforts, my main concern about the practical significance still remains. In my initial review, I questioned whether this direction has practical merit; below, I elaborate on this concern more concretely (Given the other reviewers' positive assessments, I carefully re-read the revised manuscript, but my concerns remain):

The first point is that the analysis mainly covers 4-bit rounding-to-nearest (RTN), where PTQ already works well without any calibration. I think it is important to analyze the regimes where PTQ actually fails (e.g., low-bit settings such as W2 or W3 without any group-wise parameters).

Also, to have practical value, the authors' findings, such as counteraction in Proposition 1 and the large angular discrepancy in Proposition 2, should be incorporated into the loss functions of existing PTQ methods (e.g., training-based methods such as OmniQuant, FlexRound, EfficientQAT) and shown to improve the quantization performance in sub 4-bit regimes.

In summary, I am still uncertain about the value of this work, so I keep my score unchanged.

AuthorsOpenReview ↗

Follow-up Response to Reviewer iDwj (Part 1/2)

We thank you for re-reading our revised manuscript and elaborating on these two concerns. First, we would like to clarify that while 4-bit RTN works well empirically without calibration, knowing that 4-bit RTN works does not explain how pretrained networks internally suppress quantization errors, or when that suppression breaks down. Following your two suggestions, we conducted additional experiments showing that our mechanism directly accounts for sub-4-bit failure regimes (2-bit/3-bit weight quantization without group-wise parameters) and improves training-based sub-4-bit PTQ (OmniQuant [1]).

1. Analyzing Regimes Where PTQ Fails (W2 and W3 Without Group-Wise Parameters)

First, regarding why our main analysis examines calibration-free 4-bit RTN: in this setting, no external calibration procedure has been applied, yet the network still substantially suppresses error growth across layers. Studying this setting therefore directly reveals the inherent error-suppressing mechanism of the pretrained model itself, which end-to-end performance comparisons alone cannot uncover.

Second, to analyze regimes where PTQ actually fails (W2 and W3 without group-wise parameters), we evaluated per-channel affine RTN from INT8 to INT2 (one scale and zero-point per output channel, no group-wise parameters) on OLMo3-7B (16 C4 texts, 2 seeds, mean ± SD, unfiltered; unquantized ref. CE = 3.159±0.5883.159 \pm 0.588). Alongside full-model CE, relative ΔCE\Delta{\rm CE}, and KL divergence, we measure the single-block (counteraction) cancellation ratio ηnorm\eta_{\rm norm} (%) at a representative counteracting block (block 16) under the actual prefix input error Δh\Delta h and block response Δu=F(h+Δh)−F(h)\Delta u = F(h + \Delta h) - F(h) (we also introduce this metric in Rebuttal to Reviewer b3Yh, Q2): ηnorm:=1−∥Δh+Δu∥2∥Δh∥22+∥Δu∥22.\eta_{\rm norm} := 1 - \frac{\lVert \Delta h + \Delta u\rVert_2}{\sqrt{\lVert \Delta h\rVert_2^2 + \lVert \Delta u\rVert_2^2}}.

Metric (Per-Channel RTN, No Group Parameters)INT8INT6INT4INT3INT2
Full Model CE ↓3.160±0.5893.162±0.5883.277±0.5913.895±0.55712.990±0.540
Full Model ΔCE\Delta{\rm CE} (%) ↓+0.04±0.08+0.11±0.36+3.84±1.93+24.54±9.33+325.47±84.45
Full Model KL Divergence ↓0.0002±0.00010.0038±0.00100.1269±0.03430.7869±0.16169.903±0.647
Block-16 Cancellation Ratio ηnorm\eta_{\rm norm} (%) ↑11.91±8.1612.99±3.6813.32±3.5610.73±2.455.27±1.21

Observed failure boundary in sub-4-bit regimes: From INT8 to INT4, we observe a stable cancellation ratio (ηnorm≈12\eta_{\rm norm} \approx 12%–1313%) alongside minimal output degradation. Moving to W3 and W2 without group-wise parameters, we observe three clear transitions:

  • (i) Onset of degradation in INT3 per-channel: As per-layer quantization error grows, the cancellation ratio begins to weaken (ηnorm=10.73\eta_{\rm norm} = 10.73%) and noticeable output degradation emerges (ΔCE=+24.54\Delta{\rm CE} = +24.54%).
  • (ii) Severe breakdown in INT2 per-channel: In INT2, the cancellation ratio drops by more than half to 5.275.27% as accumulated errors exceed the local effective regime (aligning with our finding in Rebuttal to Reviewer b3Yh, Q2, that counteraction similarly weakens under input perturbations with 10×10\times the measured NVFP4 hidden-error norm), and the model suffers severe output failure (ΔCE=+325.47\Delta{\rm CE} = +325.47%).
  • (iii) Contrast across scaling granularities: Removing channel scales as well (per-tensor INT3 / INT2) further reduces ηnorm\eta_{\rm norm} to 3.343.34% / 1.711.71% (ΔCE=+372.6\Delta{\rm CE} = +372.6% / +1058.6+1058.6%), whereas introducing group-wise scaling (group128 INT3 / INT2) keeps per-layer errors smaller and maintains ηnorm\eta_{\rm norm} at 10.8310.83% / 10.3510.35% (ΔCE=+7.38\Delta{\rm CE} = +7.38% at INT3). This comparison also shows that, at the same weight bit width, scaling granularity substantially changes both the measured error cancellation and output degradation.

These results extend our analysis from identifying counteraction to examining whether it is sufficient to preserve output quality. Because the same mechanism that limits error growth in successful 4-bit PTQ becomes insufficient under more aggressive quantization, our mechanism analysis connects working and failing regimes and supports the diagnostic value of our counteraction analysis.

Reviewer_b3YhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper aims to explain, both theoretically and empirically, why post-training quantization (PTQ) works effectively for large language models. The authors propose two main mechanisms:

  1. Counteraction during error propagation. Pretrained Transformer blocks tend to respond to input hidden-state errors by producing residual-update errors that point in the opposite direction, thereby partially canceling accumulated errors and slowing their growth across layers. However, why pretraining leads to this property is not yet fully understood.
  2. Preferential preservation of top-ranked token predictions. The hidden-state errors introduced by quantization mainly manifest as rotations rather than changes in vector magnitude. The high-dimensional geometry of the LM head attenuates the effects of these rotations on token scores. In particular, the scores and probabilities of top-ranked tokens are less affected, helping preserve overall output quality.
Critical strengths
  1. A coherent explanation linking hidden-state errors to output stability. The paper establishes a clear connection between hidden-state error propagation and output quality, explaining why quantization errors accumulate slowly across layers and why the remaining errors have limited effects on final predictions.

  2. Solid mathematical analysis supported by experiments. The paper provides mathematical derivations and experimental evidence to support its proposed mechanisms. In particular, Section 3 presents a detailed analysis of counteraction, including exact error decompositions and intervention experiments that provide evidence for its contribution to limiting error growth.

  3. Evaluation across different models and architectures. The authors validate their findings across multiple model families, including Qwen, OLMo, and Gemma, as well as both dense and mixture-of-experts (MoE) architectures. These experiments support the generality of the proposed mechanisms.

Critical weaknesses
  1. Potential confounding factors in the comparison between randomly initialized and pretrained models. The paper compares randomly initialized models with pretrained models to investigate whether pretraining leads to stronger counteraction and improved quantization robustness. However, pretraining changes many properties of a model beyond counteraction, including weight distributions, activation statistics, and sensitivity to perturbations. Therefore, this comparison alone may not be sufficient to isolate the contribution of counteraction. Although the authors provide additional evidence through comparisons across training checkpoints, mathematical decompositions, and intervention experiments, these experiments may not fully disentangle counteraction from other changes introduced during pretraining.

  2. Assumptions in the LM-head analysis and limited evaluation of autoregressive generation. The theoretical analysis of LM-head geometry relies on a relatively strong assumption about the distribution of hidden-state rotation directions. Moreover, the paper primarily focuses on individual next-token predictions rather than complete autoregressive generation. It remains unclear to what extent the observed output stability extends to long-form generation, where small prediction differences may accumulate and lead to divergent generation trajectories.

Questions for authors
  1. Introduce more carefully controlled comparison experiments. The current analysis primarily compares randomly initialized models with pretrained models and checkpoints at different training stages. I suggest introducing additional controls for factors such as the relative norms of per-layer weight quantization errors and input activation distributions, rather than relying mainly on weight reconstruction cosine similarity. This could help clarify whether the observed differences in quantization robustness are primarily associated with counteraction or other changes introduced by pretraining.

  2. Investigate counteraction using controlled hidden-state perturbations. The authors could directly inject controlled perturbations into the hidden states of pretrained models, independently of the quantization process, and analyze how subsequent Transformer blocks respond. By systematically varying the magnitude and direction of these perturbations, and comparing the responses of pretrained and randomly initialized models, the authors could further investigate whether counteraction reflects a general ability to suppress input perturbations or depends on the specific structure of quantization-induced errors.

3 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer b3Yh (Part 1/2)

We thank you for your time and constructive feedback, and for recognizing our coherent explanation of PTQ output stability, our solid mathematical and experimental analysis, and our broad evaluation across model families and architectures. Below, we address your concerns and present new controlled perturbation and long-form generation experiments following your suggestions.

(W1 & Q1) Controlled comparisons across weight, GEMM, block-output, and final hidden-state levels

Following your suggestion, we control for both per-layer weight quantization error norms and input activation distributions by feeding the same activations XX and block inputs hh from the pretrained model into both the randomly initialized and pretrained weights:

Table: Relative L2 errors (mean ± SD) under shared real inputs (Qwen3-4B, NVFP4, 16 C4 sequences).

Evaluated Rel. L2 ErrorRandom Init.Pretrained
1. Weight: ∥Q(W)−W∥F/∥W∥F\lVert Q(W)-W\rVert_F/\lVert W\rVert_F0.0951±0.00010.0949±0.0002
2. GEMM Output (same XX): ∥XQ(W)⊤−XW⊤∥F/∥XW⊤∥F\lVert X Q(W)^\top-X W^\top\rVert_F/\lVert X W^\top\rVert_F0.0951±0.00000.0947±0.0006
3. Block output (same hh): ∥F^(h)−F(h)∥2/∥F(h)∥2\lVert\hat{F}(h)-F(h)\rVert_2/\lVert F(h)\rVert_20.1857±0.00070.1706±0.0018
4. Final Hidden State (same h(0)h^{(0)}): ∥h^(L)−h(L)∥2/∥h(L)∥2\lVert\hat{h}^{(L)}-h^{(L)}\rVert_2/\lVert h^{(L)}\rVert_20.6946±0.02480.3186±0.0405

Under the same input activations XX and hh, the single-layer weight, GEMM, and block-output errors of the two models differ by only 0.2%–8.1%, whereas the multi-layer final hidden-state error is 2.18$\times$ larger at random initialization than in the pretrained model. This supports that the geometric counteraction between block errors along the residual stream is an important contributor to the quantization robustness in pretrained models.

(Q2) Controlled hidden-state perturbations across magnitudes, directions, and multi-block propagation

As shown in our Jacobian analysis for Reviewer Ww8L (W1 & Q1), pretraining shifts the symmetric block Jacobian spectrum to be predominantly negative (tr(S)<0{\rm tr}(S)<0), giving pretrained blocks a learned general ability to suppress input perturbations.

Following your suggestion, we directly inject controlled perturbations Δhα=α⋅δ\Delta h_\alpha=\alpha\cdot\delta into clean hidden states hh without weight quantization, varying the magnitude scale α\alpha (α=1\alpha=1 matches the actual NVFP4 hidden-error norm) and direction δ\delta (actual quantization error direction vs. equal-norm isotropic random direction).

(1) Single-block response across magnitudes and directions: For a single unquantized block with response Δu=F(h+Δhα)−F(h)\Delta u=F(h+\Delta h_\alpha)-F(h), we measure the error-norm reduction ratio relative to orthogonal accumulation, ηnorm(α):=1−∥Δhα+Δu∥2∥Δhα∥22+∥Δu∥22\eta_{\rm norm}(\alpha):=1-\frac{\lVert\Delta h_\alpha+\Delta u\rVert_2}{\sqrt{\lVert\Delta h_\alpha\rVert_2^2+\lVert\Delta u\rVert_2^2}}, showing how much counteraction reduces:

Table: Single-block error-norm reduction ηnorm(α)\eta_{\rm norm}(\alpha) (%; mean ± SD) on unquantized OLMo3-7B (block 16, 16 C4 texts).

Model & Directionα=0.03\alpha=0.03α=0.1\alpha=0.1α=0.3\alpha=0.3α=1\alpha=1 (Actual)α=3\alpha=3α=10\alpha=10
Pretrained (Actual Error Dir.)13.74±2.2513.75±2.2813.75±2.3613.67±2.5513.11±1.545.83±1.07
Pretrained (Random Dir.)16.73±1.3016.73±1.3016.71±1.3016.51±1.3015.00±1.286.70±1.02
Random Init. (Actual Error Dir.)0.01±0.050.01±0.050.01±0.050.01±0.050.01±0.060.02±0.09
Random Init. (Random Dir.)0.00±0.020.00±0.020.00±0.020.00±0.020.00±0.030.00±0.03

(2) Multi-block propagation of a single injected perturbation: We inject the perturbation once at block 1 (α=1\alpha=1) and propagate it through the unquantized network, measuring the final relative hidden error ∥h~(L)−h(L)∥2/∥h(L)∥2\lVert\tilde{h}^{(L)}-h^{(L)}\rVert_2/\lVert h^{(L)}\rVert_2:

Table: Final relative hidden error (%; mean ± SD) at block 32 after a single perturbation injection at block 1 (unquantized OLMo3-7B, 16 C4 texts).

Injected DirectionPretrained (Natural)Pretrained (Counteraction Removed)Random Init. (Natural)
Actual Quantization Error Dir.1.18±0.70%5.79±1.22% (4.91$\times$)10.41±0.95% (8.82$\times$)
Isotropic Random Dir.0.94±0.27%5.18±1.05% (5.49$\times$)8.64±0.74% (9.15$\times$)

These results directly support our core findings: (1) Counteraction reflects a general learned ability to suppress input perturbations, as pretrained blocks suppress both random-direction and quantization-direction errors while randomly initialized blocks do not; and (2) the pretrained model provides a stable self-correcting counteracting mechanism under typical 4-bit quantization error scales (α∈[0.03,3]\alpha\in[0.03,3]), whereas under more extreme perturbations (α=10\alpha=10), it can no longer cancel a sufficient fraction of the error, reflecting the local, magnitude-bounded nature of counteraction.

AuthorsOpenReview ↗

Rebuttal to Reviewer b3Yh (Part 2/2)

(W2 Part 1) Empirical verification and role of the LM-head rotation-direction assumption

Our assumption in Thm. 2 is made to capture the high-dimensional geometric property that the ratio between the LM-head projection-angle shift and the hidden-state rotation angle shrinks as the dimension dd grows. Under this condition, projecting the unit hidden-state rotation direction onto unit LM-head weight vectors decays at rate O(1/d)O(1/\sqrt{d}). We directly verify this property on actual quantization errors across six Qwen3 scales (0.6B0.6\text{B} to 32B32\text{B}, 512 C4 documents per model):

Qwen3 Scale (dd)0.6B (1024)1.7B (2048)4B (2560)8B (4096)14B (5120)32B (5120)C/dC/\sqrt{d} Fit R2R^2
All-vocab Projection RMS0.03210.02460.02450.01840.01810.01660.873
Top-10k Projection RMS0.03260.02390.02380.01720.01690.01610.947

These results confirm that actual quantization errors closely follow the O(1/d)O(1/\sqrt{d}) dimensional attenuation used in Thm. 2.

Moreover, this assumption is not a prerequisite for our core mechanism findings in Sec. 4 (and Thm. 3 does not require it at all); Thm. 2 serves only to provide an interpretable high-dimensional geometric explanation for the observed order-of-magnitude angle attenuation and rank-dependent trend, whereas all core findings in Sec. 4 are empirical observations directly measured across all evaluated models and datasets.

(W2 Part 2) Stability and role of counteraction under multi-step autoregressive (AR) generation

Analyzing internal hidden-state errors requires feeding identical decoding tokens to both models (full-precision BF16 and W4 quantized variant). The models can sample different tokens in each AR generation step. Comparing hidden states after token trajectories diverge is beyond our scope, as we acknowledge in Sec. 5.4.

Therefore, to evaluate multi-step AR error accumulation, we evaluate 256256 incremental decoding steps under teacher forcing. We feed the same ground-truth token from sample texts of the C4 dataset to both models at each step to keep the token trajectory the same, while retaining each model's own independent KV cache so that historical KV errors accumulate across steps:

Table: Cumulative mean CE and final relative hidden error ∥Δh(L)∥/∥h(L)∥\lVert \Delta h^{(L)}\rVert / \lVert h^{(L)}\rVert over NN AR steps (Qwen3-8B on C4, 32 texts, 256-token prefix; counteraction removal/reversal applied at blocks 6--23).

Tokens NNCE: BF16CE: W4CE: RemovalCE: ReversalFinal Hidden Error: W4Final Hidden Error: RemovalFinal Hidden Error: Reversal
161.8811.916 (+1.9%)2.203 (+17.2%)6.166 (+227.9%)12.22%21.96%72.57%
641.8421.886 (+2.4%)2.132 (+15.8%)6.202 (+236.7%)11.87%21.61%71.72%
1281.8281.873 (+2.5%)2.112 (+15.6%)6.053 (+231.2%)11.70%22.05%72.12%
2561.7781.821 (+2.4%)2.057 (+15.7%)5.992 (+237.1%)11.55%22.87%73.66%

Across 256256 AR steps with accumulating KV-cache errors, both the average CE degradation and the internal hidden-state error remain stable as the number of generated tokens increases (W4 ΔCE\Delta{\rm CE} stays at +1.9+1.9%–2.52.5% and final relative hidden error ∥Δh(L)∥/∥h(L)∥\lVert \Delta h^{(L)}\rVert / \lVert h^{(L)}\rVert stays at 12.2212.22% →\to 11.5511.55%), whereas removing or reversing counteraction sharply degrades both metrics throughout generation. In summary, the error counteraction remains stable during long-form generation with shared token history, which further strengthens our findings.

Moreover, for generative tasks with free AR decoding, 4-bit PTQ has already been evaluated in prior work [1,2]. Token-trajectory divergence is not equivalent to task failure: different generated reasoning sequences can still reach the same correct answer. NVIDIA's NVFP4 report demonstrates near-FP8 performance on mathematical reasoning and code-generation benchmarks [1]. Understanding how counteraction operates after generation trajectories diverge remains an open problem beyond our matched-prefix analysis, as we explicitly acknowledge in Sec. 5.4. Our controlled AR results and error-decomposition framework provide a concrete starting point for this investigation.

Thank you again for your insightful feedback. The additional results presented above further validate our core finding that counteraction is a learned property that actively suppresses both random and quantization-induced perturbations within the network, and our extended analysis confirms that it remains stable during long-form generation. All of these findings will be included in our final revised manuscript. We hope our responses have fully addressed your questions, and we are happy to discuss further.

References:
[1] NVIDIA. Introducing NVFP4 for Efficient and Accurate Low-Precision Inference, NVIDIA Technical Blog, 2025.
[2] Liu et al., Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models, COLM 2025.

Thank you for the responses and additional experiments. My major concerns have been addressed, and I will maintain my positive rating.

Reviewer_Ww8LOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why post-training quantization (PTQ) can preserve the performance of pretrained large language models despite introducing errors into their weights. The authors show that similar weight quantization errors lead to much larger hidden-state errors in randomly initialized models than in pretrained models. They identify an important mechanism called error counteraction: errors introduced by a layer tend to oppose errors propagated from earlier layers, reducing the overall accumulation of quantization errors.

The paper develops a mathematical analysis of this effect and uses intervention experiments to demonstrate its importance. It also examines the robustness of output token probabilities to quantization errors. The findings are validated across multiple model architectures, quantization methods, and precision levels.

Critical strengths

1. Novel perspective on PTQ robustness.

The paper provides an interesting explanation for why pretrained models are robust to quantization errors. Rather than focusing only on the size of weight quantization errors, it studies how errors propagate and partially cancel across layers. This offers a useful perspective on a fundamental question in model quantization.

2. Strong experimental evidence for error counteraction.

The intervention experiments show that disrupting error counteraction substantially increases hidden-state errors and output distribution differences. The consistent findings across different model families and quantization settings further support the importance and generality of this phenomenon.

Critical weaknesses

1. Limited explanation of why error counteraction emerges

Although the paper demonstrates that error counteraction develops during pretraining, it does not fully explain which learned properties of the model produce this behavior. The analysis is mainly descriptive and provides limited guidance for predicting when counteraction will be effective or when it might fail. Furthermore, the interventions used to study counteraction also affect later hidden states and error magnitudes, making it difficult to isolate the direct effect of error cancellation.

This limits the paper's ability to provide a complete mechanistic explanation of PTQ robustness.

2. Theoretical assumptions and evaluation limitations

The analysis of the output layer relies on an assumption about the directions of hidden-state perturbations, but the paper does not directly verify whether this assumption holds for actual quantization errors.

In addition, the evaluation focuses on next-token prediction, leaving the effect on autoregressive generation unclear. The estimated contribution of error counteraction also changes substantially, from 30.4% to 50.4%, when approximately 0.28% of abnormal trajectories are excluded. This raises questions about the robustness of the quantitative conclusions. These limitations make it difficult to determine how broadly the proposed explanation applies.

Questions for authors
  1. What learned properties of pretrained Transformers lead to error counteraction? Could the authors examine how individual Transformer layers respond to small input perturbations, for example through local sensitivity or Jacobian analysis? Such evidence could help explain why counteraction emerges during pretraining and whether its effectiveness can be predicted.

  2. Can the effects of error counteraction be isolated more precisely? Since the intervention experiments also change subsequent hidden states and error magnitudes, could the authors provide additional controls to distinguish direct error cancellation from these downstream effects?

  3. How well does the output-layer assumption match actual quantization errors? Can the authors empirically examine the directions of quantization-induced perturbations and evaluate how sensitive the theoretical conclusions are to this assumption?

  4. How robust are the findings under broader evaluation settings? In particular, does error counteraction remain effective during autoregressive generation? Can the authors also clarify why excluding a very small fraction of abnormal trajectories substantially changes the estimated contribution of counteraction?

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer Ww8L (Part 1/3)

We thank you for your time, constructive comments, and insightful suggestions, as well as for recognizing our novel perspective on PTQ robustness and our strong experimental evidence for error counteraction.

Based on your suggestions, we extended our analysis with several new valuable experiments, which all confirm and strengthen the core findings of our paper. Before addressing each point in detail below, we briefly summarize the main takeaways from our extended analysis:

  • Learned properties behind counteraction (W1, Q1): A newly suggested Jacobian analysis shows that pretraining shifts the symmetric Jacobian spectrum of counteracting blocks to be predominantly negative, which naturally induces counteraction when errors project across eigenspaces.
  • Isolating direct error cancellation (W1, Q2): Even when downstream hidden states and per-layer error magnitudes are held strictly fixed, direct directional cancellation alone eliminates 44.2% of single-block error energy and reduces multi-block relative error by 31.9% to 40.5%.
  • Empirical verification of the LM-head assumption (W2, Q3): Actual quantization errors closely follow the O(1/d)O(1/\sqrt{d}) dimensional attenuation predicted by Thm. 2, and we clarify that our core empirical findings in Sec. 4 and Thm. 3 hold independently of this assumption.
  • Autoregressive generation and outlier robustness (W2, Q4): Across 256 AR steps with independent KV caches, hidden-state errors and cross-entropy degradation remain stable due to counteraction. Furthermore, we show that the shift when excluding 0.28% of trajectories is primarily due to the sensitivity of squared statistics to extreme outliers.

Below, we provide the full details and experimental results for each question.

(W1 Part 1 & Q1) Jacobian analysis can explain the learned properties behind counteraction and when it works or fails

We expand our analysis according to your suggestion. In summary, pretraining shifts the counteracting block's symmetric Jacobian spectrum to be predominantly negative, and because propagated quantization errors project almost evenly across eigenspaces, this negative spectrum naturally induces counteraction. We show the analysis as follows:

In Sec. 5.2 and App. E.6 (Eq. (22), Fig. 16), we showed that >99>99% of the negative error interaction comes from the block's response to its inherited input error rather than direct weight perturbation at a fixed input. Following your suggestion, we examine this mechanism in a controlled same-token experiment (holding preceding token states and block weights fixed and perturbing only the current token's state hh) via the local block Jacobian JF=∂F(h)∂hJ_F = \frac{\partial F(h)}{\partial h}, giving the local response Δuresponse=F(h+Δh)−F(h)≈JFΔh\Delta u_{\rm response} = F(h + \Delta h) - F(h) \approx J_F \Delta h. Because Δh⊤(JF−JF⊤)Δh=0\Delta h^\top (J_F - J_F^\top)\Delta h = 0, the first-order interaction depends only on the symmetric part S:=12(JF+JF⊤)S := \frac{1}{2}(J_F + J_F^\top) and its orthonormal eigenbasis {qiq_i}i=1d_{i=1}^d:

⟨Δh,Δuresponse⟩≈⟨Δh,JFΔh⟩=Δh⊤JF+JF⊤2Δh=Δh⊤SΔh=∑i=1dλi(S)⟨qi,Δh⟩2.\langle \Delta h, \Delta u_{\rm response}\rangle \approx \langle \Delta h, J_F\Delta h \rangle = \Delta h^\top \frac{J_F + J_F^\top}{2} \Delta h = \Delta h^\top S \Delta h = \sum_{i=1}^d \lambda_i(S)\langle q_i, \Delta h\rangle^2.

Evaluating this spectral decomposition (see the last two pages of our revised PDF paper) yields two key quantitative findings and predictions:

  • Pretraining learns a negatively skewed spectrum of SS: At random initialization, λi(S)\lambda_i(S) is nearly symmetric around 00 (∼50\sim 50% negative, tr(S)≈0{\rm tr}(S) \approx 0). Pretraining shifts the spectrum so that ∼77\sim 77% of λi(S)\lambda_i(S) become negative (tr(S)<0{\rm tr}(S) < 0), with the normalized response Δh⊤SΔh/∥Δh∥22\Delta h^\top S\Delta h / \lVert \Delta h\rVert_2^2 decreasing monotonically from ≈0\approx 0 at initialization to −0.31-0.31 at the final checkpoint for Pythia-410M block 12.
  • Near-isotropic error projection triggers counteraction (and predicts when it works or fails): Whenever a perturbation projects evenly across {qi}\{q_i\}, the interaction simplifies to Δh⊤SΔh=∑i=1d⟨qi,Δh⟩2⋅λi(S)∼∥Δh∥22d⋅tr(S)<0,\Delta h^\top S \Delta h = \sum_{i=1}^{d} \langle q_i, \Delta h \rangle^2 \cdot \lambda_i(S) \sim \frac{\lVert \Delta h\rVert_2^2}{d} \cdot {\rm tr}(S) < 0, yielding counteraction; conversely, counteraction fails if a perturbation is deliberately concentrated in the λi(S)>0\lambda_i(S) > 0 subspace. Empirically, we find that the actual propagated quantization error Δh\Delta h in Pythia-2.8B, Qwen3, and OLMo3 closely matches the isotropic baseline across eigenspaces (isotropic trace reference ∥Δh∥22d⋅tr(S)\frac{\lVert \Delta h\rVert_2^2}{d} \cdot {\rm tr}(S) differs from the measured local interaction by only 1.101.10%–3.753.75% on average relatively). Also, the first-order Jacobian prediction of the local interaction achieves 2.012.01%–3.063.06% relative error. This confirms that counteracting blocks induce counteraction through their learned negatively skewed spectrum naturally.
AuthorsOpenReview ↗

Rebuttal to Reviewer Ww8L (Part 2/3)

(W1 Part 2 & Q2) Isolating direct error cancellation from downstream trajectory effects

We address your concern that online interventions alter subsequent hidden states and later error magnitudes ∥Δu(ℓ)∥2\lVert \Delta u^{(\ell)}\rVert_2 as follows: To eliminate this downstream change, we keep the original unmodified forward trajectory and exact error magnitudes strictly fixed, and compare the actual error against the orthogonal-accumulation reference where the update errors accumulate without directional cancellation (⟨Δh,Δu⟩=0\langle \Delta h, \Delta u \rangle = 0).

At a single block with fixed weights (Pythia-410M, block 12), the response Δuresponse=F(h+Δh)−F(h)\Delta u_{\rm response} = F(h + \Delta h) - F(h) yields a normalized interaction ⟨Δh,Δuresponse⟩/∥Δh∥22=−0.303\langle \Delta h, \Delta u_{\rm response} \rangle / \lVert \Delta h\rVert_2^2 = \mathbf{-0.303} (matching our Jacobian prediction of −0.310-0.310), directly canceling 44.244.2% of the orthogonal-sum energy ∥Δh∥22+∥Δuresponse∥22\lVert \Delta h\rVert_2^2 + \lVert \Delta u_{\rm response}\rVert_2^2.

Across prefix blocks 1…M1 \dots M with the counteraction phenomenon, unrolling Prop. 1 on the unmodified W4 trajectory gives ∥Δh(M)∥22=∑ℓ=1M∥Δu(ℓ)∥22+2∑ℓ=1M⟨Δh(ℓ−1),Δu(ℓ)⟩.\lVert \Delta h^{(M)}\rVert_2^2 = \sum_{\ell=1}^M \lVert \Delta u^{(\ell)}\rVert_2^2 + 2\sum_{\ell=1}^M \langle \Delta h^{(\ell-1)}, \Delta u^{(\ell)} \rangle. Holding the measured Δu(ℓ)\Delta u^{(\ell)} vectors completely unchanged, direct counteraction (⟨Δh(ℓ−1),Δu(ℓ)⟩<0\langle \Delta h^{(\ell-1)}, \Delta u^{(\ell)} \rangle < 0) substantially reduces the relative hidden error ∥Δh(M)∥2/∥h(M)∥2\lVert \Delta h^{(M)}\rVert_2 / \lVert h^{(M)}\rVert_2 compared with the orthogonal reference ∑ℓ=1M∥Δu(ℓ)∥22/∥h(M)∥2\sqrt{\sum_{\ell=1}^M \lVert \Delta u^{(\ell)}\rVert_2^2} / \lVert h^{(M)}\rVert_2 without counteraction (128 C4 texts, unfiltered):

Model (Prefix 1–MM)Qwen3-8B (1–18)Qwen3-30B-A3B (1–24)OLMo3-32B (1–32)
Orthogonal Reference27.23%18.05%18.46%
Actual Relative Error16.19%12.28%11.29%
Direct Cancellation-40.5%-31.9%-38.9%

In conclusion, even when downstream hidden states and per-layer error magnitudes ∥Δu(ℓ)∥2\lVert \Delta u^{(\ell)}\rVert_2 are held strictly fixed, direct directional cancellation alone eliminates 44.244.2% of single-block error energy and reduces multi-block relative error by 31.931.9%–40.540.5% compared to the zero-interaction reference constructed from the same measured update-error norms.

Moreover, the downstream effect captured by our Sec. 3.3 interventions is not a confounding artifact, but a core part of how counteraction operates in deep networks: by shrinking Δh(ℓ)\Delta h^{(\ell)} at block ℓ\ell, counteraction delivers a smaller input perturbation to block ℓ+1\ell+1, preventing compound error amplification across depth.

(W2 Part 1 & Q3) Empirical verification of LM-head perturbation directions, sensitivity, and clarifying the role of the theoretical assumption

First, we provide a key test on the prediction of the assumption, namely the dimensional attenuation underlying the LM-head analysis. Our assumption in Thm. 2 is made to capture the high-dimensional geometric property that the ratio between the LM-head projection-angle shift and the hidden-state rotation angle shrinks as the dimension dd grows. If this assumption holds, projecting the unit hidden-state rotation direction onto unit LM-head weight vectors should decay at rate O(1/d)O(1/\sqrt{d}). We therefore directly test actual errors using this property across six Qwen3 scales (0.6B0.6\text{B} to 32B32\text{B}, using 512 C4 documents per model):

Qwen3 Scale (dd)0.6B (1024)1.7B (2048)4B (2560)8B (4096)14B (5120)32B (5120)C/dC/\sqrt{d} Fit R2R^2
All-vocab Projection RMS0.03210.02460.02450.01840.01810.01660.873
Top-10k Projection RMS0.03260.02390.02380.01720.01690.01610.947

These results confirm that actual quantization errors closely follow the O(1/d)O(1/\sqrt{d}) dimensional attenuation predicted in Thm. 2.

Besides, the empirical findings and Thm. 3 do not depend on this assumption. For Thm. 2 (which uses the directional assumption), the analytical approximation consistently captures the rank-monotonic trend across all evaluated models, with 17.517.5%–19.819.8% averaged relative error across Qwen3, Gemma3, and OLMoE models. For Thm. 3 (log-probability changes ∣Δlog⁡p∣|\Delta \log p|, which does not require the directional assumption), the 1st- and 2nd-order expansions match empirical rank curves with only 0.300.30%–6.546.54% and 0.200.20%–2.042.04% error across models.

Finally, we clarify that the assumption is not a prerequisite for our core mechanism findings in Sec. 4; Thm. 2 serves only to provide an interpretable high-dimensional geometric explanation for the observed order-of-magnitude angle attenuation and rank-dependent trend. Core findings in Sec. 4 are empirical observations directly measured across all evaluated models and datasets and hold independently of this theoretical assumption.

AuthorsOpenReview ↗

Rebuttal to Reviewer Ww8L (Part 3/3)

(W2 Part 2 & Q4 Part 1) Autoregressive (AR) generation with own independent KV cache.

Analyzing internal hidden-state errors requires feeding identical decoding tokens to both models (full-precision BF16 and W4 quantized variant). The models can sample different tokens in each AR generation step. Analyzing hidden states after token trajectories diverge is beyond our scope. Therefore, to test multi-step AR error accumulation, we iteratively feed the ground-truth token from sample texts of the C4 dataset to both models at each step to keep the token trajectory the same while retaining each model's own independent KV cache so that historical KV errors accumulate across steps:

Table: Cumulative mean CE and final relative hidden error ∥Δh(L)∥/∥h(L)∥\lVert \Delta h^{(L)}\rVert / \lVert h^{(L)}\rVert over NN AR steps (Qwen3-8B on C4, 32 texts, 256-token prefix; counteraction removal/reversal applied at blocks 6--23).

Tokens NNCE: BF16CE: W4CE: RemovalCE: ReversalFinal Hidden Error: W4Final Hidden Error: RemovalFinal Hidden Error: Reversal
161.8811.916 (+1.9%)2.203 (+17.2%)6.166 (+227.9%)12.22%21.96%72.57%
641.8421.886 (+2.4%)2.132 (+15.8%)6.202 (+236.7%)11.87%21.61%71.72%
1281.8281.873 (+2.5%)2.112 (+15.6%)6.053 (+231.2%)11.70%22.05%72.12%
2561.7781.821 (+2.4%)2.057 (+15.7%)5.992 (+237.1%)11.55%22.87%73.66%

In summary, across 256256 AR steps with accumulating KV-cache errors, both the average CE degradation and the internal hidden-state error remain stable as the number of generated tokens increases (W4 ΔCE\Delta{\rm CE} stays at +1.9+1.9%–2.52.5% and final relative hidden error ∥Δh(L)∥/∥h(L)∥\lVert \Delta h^{(L)}\rVert /\lVert h^{(L)}\rVert stays at 12.2212.22% →\to 11.5511.55%), whereas removing or reversing counteraction sharply degrades both metrics throughout generation. Thus, the results show that the error counteraction remains effective during AR generation with shared token history, which further strengthens our findings.

Our matched-token AR experiments have directly demonstrated that counteraction remains effective over multiple decoding steps with model-specific KV caches. As for free generation task performance: Although evaluation has already been studied showing the NVFP4 model demonstrates near-FP8 performance on mathematical reasoning and code-generation benchmarks [1], analyzing internal errors after tokens diverge is harder because hidden-state differences reflect both quantization and different conditioning inputs. Characterizing counteraction under independently evolving token histories remains an open problem, as acknowledged in Sec. 5.4; our controlled AR results and error-decomposition framework provide a concrete starting point for this investigation.

(W2 Part 3 & Q4 Part 2) Why does excluding 0.280.28% abnormal trajectories change the estimated counteraction contribution?

First, this change is due to the outlier sensitivity of squared statistics, whereas our tested first-order statistics are insensitive: with default filtering, the mean final relative error E[∥Δh(L)∥2/∥h(L)∥2]\mathbb{E}[\lVert \Delta h^{(L)}\rVert_2/\lVert h^{(L)}\rVert_2] changes by only 0.740.74% (0.1480.148 vs. 0.1470.147, Tab. 2 in paper), and such shift appears only in the decomposition terms of the squared relative-error recurrence (Thm. 1).

Second, trimming extreme tails is standard statistical practice [2], and it allows us to genuinely separate typical behavior from rare abnormal cases in our study. For instance, because squaring amplifies extreme outliers, 0.040.04% of abnormal positions can dominate 87.387.3% of the cumulative squared term TaddT_{\rm add} (Qwen3-8B on C4). The estimated magnitude is tail-sensitive, while the qualitative finding of counteraction persists and remains stable for typical cases. Once these rare abnormal cases are filtered out, tightening the threshold further (0.5→0.250.5 \to 0.25) barely changes the estimate (50.450.4% vs. 50.250.2%, Tab. 2 in the paper), confirming our characterization of typical trajectories (>99.7>99.7%) is stable and accurate.

In summary, our core findings do not rely on this filter. Meanwhile, the excluded tail reflects genuine rare failure cases in low-bit quantization (analyzed in Sec. 5.3 and App. E.3). We do not claim that PTQ is uniformly stable for every individual token; rather, our mechanism characterizes the typical behavior of the vast majority of tokens (>99.7>99.7%), rather than these rare abnormal cases.

Thank you again for your valuable feedback, which led to this extended analysis. The additional results further validate our existing core findings and will be included in the final revision. We hope we have answered all of your questions and are happy to discuss further if needed.

References:
[1] NVIDIA. Introducing NVFP4 for Efficient and Accurate Low-Precision Inference, NVIDIA Technical Blog, 2025.
[2] Huber and Ronchetti, Robust Statistics, 2nd ed., Wiley, 2009.

Reviewer_Ww8LOpenReview ↗

Reply

The authors have partially addressed my concerns, so I will raise my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.