- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors discussed "How post-training quantization works for pre-trained models". To this end, the authors compared pre-trained and randomly initialized modelsm and showed that even with comparable weight deviation, the final results are very different. The authors analyzed the reasons behind this phenomenon.
Critical strengths
The authors' view on post-training quantization is interesting. While most of papers tackle improving the performance of quantization, the authors considered one essential problem: why post-training quantization works.
Critical weaknesses
While the point of view is very interesting, I'm not sure whether this view is worthwhile to investigate, so I cannot judge the value of this study.
To my knowledge, the different behaviors of pre-trained and randomly initialized models are somewhat expected. Pre-trained models are trained to convergence, possibly with sharpness-aware minimization. Then, the task loss degradation incurred by the weight perturbation, which can be approximated as using second-order Taylor expansion, should be small with the small weight perturbation. Whereas, this approximation does not hold for randomly initialized models.
So, while I acknowledged some interesting findings (e.g., counteraction), I think the authors tried to solve some trivial problem in a very difficult way, and I am uncertain whether this direction has merits.
Questions for authors
See weaknesses above.
6 comments
Thank you for the detailed response and the additional experiments.
While I appreciate the authors' efforts, my main concern about the practical significance still remains. In my initial review, I questioned whether this direction has practical merit; below, I elaborate on this concern more concretely (Given the other reviewers' positive assessments, I carefully re-read the revised manuscript, but my concerns remain):
The first point is that the analysis mainly covers 4-bit rounding-to-nearest (RTN), where PTQ already works well without any calibration. I think it is important to analyze the regimes where PTQ actually fails (e.g., low-bit settings such as W2 or W3 without any group-wise parameters).
Also, to have practical value, the authors' findings, such as counteraction in Proposition 1 and the large angular discrepancy in Proposition 2, should be incorporated into the loss functions of existing PTQ methods (e.g., training-based methods such as OmniQuant, FlexRound, EfficientQAT) and shown to improve the quantization performance in sub 4-bit regimes.
In summary, I am still uncertain about the value of this work, so I keep my score unchanged.
Follow-up Response to Reviewer iDwj (Part 1/2)
We thank you for re-reading our revised manuscript and elaborating on these two concerns. First, we would like to clarify that while 4-bit RTN works well empirically without calibration, knowing that 4-bit RTN works does not explain how pretrained networks internally suppress quantization errors, or when that suppression breaks down. Following your two suggestions, we conducted additional experiments showing that our mechanism directly accounts for sub-4-bit failure regimes (2-bit/3-bit weight quantization without group-wise parameters) and improves training-based sub-4-bit PTQ (OmniQuant [1]).
1. Analyzing Regimes Where PTQ Fails (W2 and W3 Without Group-Wise Parameters)
First, regarding why our main analysis examines calibration-free 4-bit RTN: in this setting, no external calibration procedure has been applied, yet the network still substantially suppresses error growth across layers. Studying this setting therefore directly reveals the inherent error-suppressing mechanism of the pretrained model itself, which end-to-end performance comparisons alone cannot uncover.
Second, to analyze regimes where PTQ actually fails (W2 and W3 without group-wise parameters), we evaluated per-channel affine RTN from INT8 to INT2 (one scale and zero-point per output channel, no group-wise parameters) on OLMo3-7B (16 C4 texts, 2 seeds, mean ± SD, unfiltered; unquantized ref. CE = ). Alongside full-model CE, relative , and KL divergence, we measure the single-block (counteraction) cancellation ratio (%) at a representative counteracting block (block 16) under the actual prefix input error and block response (we also introduce this metric in Rebuttal to Reviewer b3Yh, Q2):
| Metric (Per-Channel RTN, No Group Parameters) | INT8 | INT6 | INT4 | INT3 | INT2 |
|---|---|---|---|---|---|
| Full Model CE ↓ | 3.160±0.589 | 3.162±0.588 | 3.277±0.591 | 3.895±0.557 | 12.990±0.540 |
| Full Model (%) ↓ | +0.04±0.08 | +0.11±0.36 | +3.84±1.93 | +24.54±9.33 | +325.47±84.45 |
| Full Model KL Divergence ↓ | 0.0002±0.0001 | 0.0038±0.0010 | 0.1269±0.0343 | 0.7869±0.1616 | 9.903±0.647 |
| Block-16 Cancellation Ratio (%) ↑ | 11.91±8.16 | 12.99±3.68 | 13.32±3.56 | 10.73±2.45 | 5.27±1.21 |
Observed failure boundary in sub-4-bit regimes: From INT8 to INT4, we observe a stable cancellation ratio (%–%) alongside minimal output degradation. Moving to W3 and W2 without group-wise parameters, we observe three clear transitions:
- (i) Onset of degradation in INT3 per-channel: As per-layer quantization error grows, the cancellation ratio begins to weaken (%) and noticeable output degradation emerges (%).
- (ii) Severe breakdown in INT2 per-channel: In INT2, the cancellation ratio drops by more than half to % as accumulated errors exceed the local effective regime (aligning with our finding in Rebuttal to Reviewer b3Yh, Q2, that counteraction similarly weakens under input perturbations with the measured NVFP4 hidden-error norm), and the model suffers severe output failure (%).
- (iii) Contrast across scaling granularities: Removing channel scales as well (per-tensor INT3 / INT2) further reduces to % / % (% / %), whereas introducing group-wise scaling (
group128INT3 / INT2) keeps per-layer errors smaller and maintains at % / % (% at INT3). This comparison also shows that, at the same weight bit width, scaling granularity substantially changes both the measured error cancellation and output degradation.
These results extend our analysis from identifying counteraction to examining whether it is sufficient to preserve output quality. Because the same mechanism that limits error growth in successful 4-bit PTQ becomes insufficient under more aggressive quantization, our mechanism analysis connects working and failing regimes and supports the diagnostic value of our counteraction analysis.
Rebuttal to Reviewer iDwj
We thank you for your time and for highlighting that asking why PTQ works is an essential question. Below, we would like to clarify a possible misunderstanding regarding our setup and provide new experimental results to address your concerns.
First, we clarify a possible fundamental misunderstanding: second-order Taylor expansion cannot quantitatively explain 4-bit PTQ. The breakdown of this intuition directly supports the necessity of our study: To understand PTQ robustness, we must investigate the internal dynamics of the model, rather than relying on an end-to-end loss approximation.
Moreover, comparing random and pretrained models is not our goal, but only a motivating example showing that active error counteraction is a learned property acquired during training; our research goal is to understand the internal mechanism of PTQ robustness rather than only explaining the loss change.
1. Open-recipe models in our study are trained without SAM (Sharpness-Aware Minimization)
All open-recipe models in our study (Pythia [1], OLMoE [2], and OLMo 3 [3]) are pretrained using Adam/AdamW without SAM, yet 4-bit PTQ works effectively across all of them. Therefore, SAM cannot explain why PTQ works in our study.
2. Second-order Taylor expansion cannot predict the 4-bit loss change
Even when we compute the exact second-order Taylor expansion directly on each evaluated text and token itself, it still fails to approximate the actual 4-bit loss change ΔCEi (measured via WAPE:=∑i∣ΔCEi∣∑i∣ΔCEi−ΔCEi∣):
Table: Prediction error (WAPE↓) of exact 2nd-order Taylor approximations for actual 4-bit loss change ΔCE on WikiText-103 (Qwen3-4B, weight-only NVFP4, 128 texts / 65,536 positions).
Even when computing a token's own exact Hessian on WikiText-103, it can completely miss severe actual degradations: For example, on token
"lt", the actual CE jumps from 0.007 (BF16) to 2.39 (W4), yet the 2nd-order and full Taylor expansions predict only ΔCE=+0.16 and +0.21, respectively. Similarly, on token"9", the actual CE jumps from 2.9×10−9 (BF16) to 1.89 (W4), while the two Taylor estimates predict only +1.2×10−8 and +1.5×10−8.Therefore, the second-order Taylor expansion fails severely in the 4-bit regime: such a massive WAPE error (593.4%–690.0% and up to >108× underestimation for specific tokens) directly supports that 4-bit perturbations exceed the effective range of the second-order Taylor expansion. This aligns with [4], which also shows that second-order approximations inaccurately estimate quantization loss changes, making why 4-bit PTQ works genuinely non-trivial.
3. Activation quantization has no weight Taylor expansion, yet shares the same counteraction
Under activation-only quantization (A4), all weights remain unchanged (Δw=0), so a weight-based Taylor or Hessian analysis predicts zero loss change and cannot be applied at all. Yet 4-bit activation-only quantization exhibits the exact same cross-layer error counteraction behavior as weight-only quantization (Figure 6 in Section 5 of our paper). This proves that PTQ robustness arises from how the network internally processes hidden-state errors across layers, which a weight-level Hessian cannot capture.
4. Why the Taylor expansion argument underscores the necessity of our mechanism study
The flat loss landscape intuition that a second-order Taylor expansion easily explains PTQ is a natural expectation, but its quantitative breakdown above demonstrates precisely why a deeper mechanistic study is necessary. In summary, understanding why low-bit PTQ works cannot stop at a black-box, end-to-end scalar loss approximation; we must dive into the inner properties of the network to reveal how errors cancel across layers and why top-token predictions remain stable.
We hope our responses and new experimental results clarify our methodology and core findings, and fully address your points. We will integrate all updates into the final version of the manuscript and are happy to discuss further.
References:
[1] Biderman et al., Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling, ICML 2023.
[2] Muennighoff et al., OLMoE: Open Mixture-of-Experts Language Models, ICLR 2025.
[3] Team Olmo et al., OLMo 3, arXiv:2512.13961, 2025.
[4] Hu et al., Identifying Sensitive Weights via Post-Quantization Integral, arXiv:2503.01901, 2025.