- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes FR-PTQ, which improves layer-wise reconstruction objectives with a Fisher-weighted regularization term to better preserve downstream task performance. The authors introduce two variants, where Task Fisher is estimated from labeled task data and Generic Fisher is estimated from generic calibration data. Experiments demonstrate improvements in both perplexity and downstream accuracy.
Critical strengths
- The paper addresses the mismatch between local reconstruction objectives and downstream task performance.
- The proposed method can be incorporated into existing methods without changing the model architecture or inference pipeline.
- The experiments are extensive, including model sizes from 1B to 70B and diverse bitwidth.
Critical weaknesses
- Adding a channel-wise Fisher diagonal to the reconstruction Hessian is the core contribution of this paper. However, Hessian regularization and Fisher-based sensitivity estimation have already been extensively studied [1][2][3].
- The Fisher computation requires additional calibration sets and backward propagation, which will introduce additional overheads. Its practical capability is not sufficiently discussed.
- For Hessian-based methods, the regularizer is implemented through diagonal loading. But for AWQ and SliderQuant, the authors incorporate the Fisher penalty using additional adaptive normalization factors. The paper does not sufficiently analyze whether the different implementations preserve the same optimization interpretation.
- The main paper contains only three figures and two tables. Although more analysis and results are summarized into the appendix, the limited visual and experimental analysis in the main paper weakens the presentation of the paper. In addition, several figures and tables are too small and dense to clearly convey their findings.
[1] Kim S, Hooper C, Gholami A, et al. Squeezellm: Dense-and-sparse quantization[J]. arXiv preprint arXiv:2306.07629, 2023.
[2] Li Y, Gong R, Tan X, et al. Brecq: Pushing the limit of post-training quantization by block reconstruction[J]. arXiv preprint arXiv:2102.05426, 2021.
[3] Wu Z, Wang S, Zhang J, et al. FIMA-Q: Post-training quantization for vision transformers by fisher information matrix approximation[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 14891-14900.
Questions for authors
- Please provide more discussion about the differences among this paper and the listed related works.
- Please provide the computational overhead comparison among FR-PTQ and existing baselines.
- How does the adaptive normalization in AWQ and SliderQuant relate to the fixed regularization objective in Eq3?
5 comments
W2 (Q2). The Fisher computation requires additional calibration data and backpropagation, and the overhead compared with existing baselines is not provided.
Compared with the baselines (GPTQ, AWQ, and QuIP alone), the only additional cost is a single Fisher computation. The Fisher computation runs one forward and one backward pass per sample and accumulates per-input-channel values in each layer, without any training. The computational overhead is also discussed in Sec. 5.5 and Appendix D.3, and the measurements in Appendix Table 7 are as follows.
| Model | Fisher (1,024 sequences) | One quantization run (GPTQ / AWQ / QuIP) |
|---|---|---|
| Llama-3-8B | about 5 min | about 7–14 min |
| Llama-3-70B (4 GPUs) | about 30 min | about 56 min – 3.2 h |
The Fisher computation costs no more than about one quantization run and does not change the order of the overall quantization time (minutes for 8B and hours for 70B). The Fisher depends only on the model and the estimation data, so it is shared across solvers, bit widths, and λ, and is further amortized when several settings are quantized. FR-PTQ stores only one value per channel, 4.8 MB for Llama-3-8B. The quantized weight format is the same as the baseline, so there is no additional inference cost.
Regarding additional calibration data, estimating the Fisher on the same data as the calibration still yields the gain (Table R1 in our response to Reviewer 1jfn), so additional data is not required.
W3 (Q3). How does the adaptive normalization in AWQ and SliderQuant relate to the fixed regularization objective in Eq. 3?
The integration into each solver is described in Appendix B.3 and Algorithm 2. Here we clarify the relation to Eq. 3.
The normalization factor of AWQ, γ = mean(L_recon) / mean(L_F), is a constant computed once per block.
AWQ therefore solves, in each block, exactly Eq. 3 with the effective coefficient λγ, restricted to the AWQ scaling family.
The role of γ is to make λ scale-invariant across layers, the same role as the per-layer mean-1 normalization of the Fisher in the Hessian-based solvers (Eq. 14).
The effect of the restriction to the scaling family is covered by Prop. 4, which shows that the optimum over the restricted family can be no better than the optimum of the Hessian-based solvers.
SliderQuant recomputes γ at every optimization step, so its correspondence to Eq. 3 is approximate.
We will state that AWQ solves Eq. 3 with the effective coefficient λγ restricted to its scaling family, together with its relation to Prop. 4, in Appendix B.3.
W4. The main paper contains only three figures and two tables, much of the experimental analysis is in the appendix, and several figures and tables are too small and dense.
While keeping the structure of the main text, we will further strengthen its figures and experimental analysis following the reviewer's suggestion. We will add a figure that illustrates the overall method to the main text to improve readability. The additional experimental analyses conducted for this rebuttal will be added to the appendix and referenced from the main text. We will also revise the small and dense figures and tables into less crowded formats that are easier to read.
Dear Reviewer,
Thank you again for your time and for the constructive review. We have posted our response, which addresses each of your concerns and includes additional experimental results that support our answers. As the discussion period is coming to an end, we would like to ask whether our response has resolved your concerns. If any points remain unclear, we would be glad to clarify them within the remaining time.
Best regards, The Authors
We thank the reviewer for the careful review, and for noting the problem setting of the mismatch between local reconstruction objectives and downstream performance, the ease of integration into existing methods, and the broad experiments from 1B to 70B.
Before our responses, we report one issue. Prompted by Reviewer 4aTP's Q3, we found a bug in our GPTQ implementation for Llama-family models, in which only one calibration sequence was effectively used. The bug is limited to GPTQ on Llama-family models, and GPTQ on Qwen3-8B, AWQ, and QuIP are not affected. We re-ran GPTQ on Llama-3-8B, our main evaluation model, after the fix. Unless otherwise noted, all GPTQ numbers below are after the fix.
Unless otherwise noted, all tables report Llama-3-8B results as mean ± SEM over 5 seeds. MC = average of the six multiple-choice tasks, Non-MC = average of LAMBADA, GSM8K, and TriviaQA, PPL = average of WikiText-2 and PTB. "w/Generic" is FR-PTQ with the generic (C4) Fisher, with λ selected from {0.1, 1, 2} by C4 perplexity. Bold marks the best value in each comparison, and "–" means not evaluated or PPL > 1000.
W1 (Q1). Hessian regularization and Fisher-based sensitivity have been studied in SqueezeLLM, BRECQ, and FIMA-Q, and the differences should be discussed.
The novelty lies not in using the Fisher itself but in defining an objective that goes beyond the reconstruction error.
FR-PTQ starts from the preliminary experiment in Figure 1, where calibrating on task data improves the perplexity on the task text, which the reconstruction error approximates, but does not improve task accuracy. The problem is thus not the calibration data but the reconstruction objective itself. FR-PTQ therefore adds a Fisher term, a second-order approximation of the loss, to the reconstruction error (Eq. 3, Prop. 2).
BRECQ and GuidedQuant weight the reconstruction error per token or per output using loss gradients. Their Hessian has the form
Σ_n s_n x_n x_nᵀ, which remains a weighted sum of second moments of the calibration activations for any choice of weights. They are therefore refinements of the reconstruction error, and directions that the calibration activations rarely excite are not protected. The Fisher term of FR-PTQ is an input-channel diagonal outside this form, and it adds loss information from outside the reconstruction error (see also our response to Reviewer 4aTP, Q6).The differences from the individual methods are also discussed in Sec. 2.3.
We also confirmed that this input-channel diagonal has an effect of its own, through comparisons with damping of the same total diagonal and with a shuffled Fisher of the same value distribution (Tables R3 and R4 in our response to Reviewer 1jfn). We also compared FR-PTQ with GuidedQuant (Kim et al., 2025), a recent Fisher-based PTQ method that weights the layer-wise reconstruction with end-loss gradients, and evaluated their combination under matched models, bit widths, tasks, and calibration data (Table R10 in our response to Reviewer 4aTP). FR-PTQ outperforms GuidedQuant on GPTQ, and the combination gives the best Non-MC for AWQ at INT3 and for QuIP at INT3 and INT4, showing that the two are complementary.