acceptodds
Under review as a conference paper at ICLR 2027

Beyond Reconstruction: Fisher-Regularized Post-Training Quantization for LLMs

Abstract

Large language models must be compressed to be deployed under realistic memory and latency budgets, and post-training quantization (PTQ) is dominant because it needs no retraining. Existing PTQ methods minimize a per-layer reconstruction error that approximates perplexity, but this proxy is not aligned with downstream task loss. Since a deployed model is typically dedicated to one task, the most direct way to preserve its task accuracy is to draw the calibration data from that task, yet this does not close the gap. We argue that the mismatch lies in the objective rather than in the data, and we introduce a simple yet effective modification that augments the PTQ objective with a Fisher-information regularizer preserving the weight directions to which the loss is most sensitive. We instantiate the regularizer in two forms, a Task Fisher computed from a labeled task training split and a Generic Fisher computed from the generic calibration data already used by PTQ, the latter removing any dependence on task labels and yielding a single quantized model that serves all tasks rather than a separate quantization per task. The regularizer requires no architectural changes and integrates into any per-layer PTQ solver as a diagonal loading of the per-layer curvature. Across five LLMs from 1B to 70B parameters and 2–4 bit widths, Task Fisher and Generic Fisher improve zero-shot accuracy and perplexity to a comparable degree, showing that neither task labels nor per-task quantization is necessary. Adding the label-free Generic Fisher to standard PTQ solvers raises multiple-choice accuracy by up to 6.3 points and generative-task accuracy by up to 9.4 points over the unregularized baselines, and brings the same improvement to a state-of-the-art solver.

Primary area:
foundation or frontier models, including LLMs
Keywords:
post-training quantization, large language models, Fisher information, model compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VLMqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes FR-PTQ, which improves layer-wise reconstruction objectives with a Fisher-weighted regularization term to better preserve downstream task performance. The authors introduce two variants, where Task Fisher is estimated from labeled task data and Generic Fisher is estimated from generic calibration data. Experiments demonstrate improvements in both perplexity and downstream accuracy.

Critical strengths
  1. The paper addresses the mismatch between local reconstruction objectives and downstream task performance.
  2. The proposed method can be incorporated into existing methods without changing the model architecture or inference pipeline.
  3. The experiments are extensive, including model sizes from 1B to 70B and diverse bitwidth.
Critical weaknesses
  1. Adding a channel-wise Fisher diagonal to the reconstruction Hessian is the core contribution of this paper. However, Hessian regularization and Fisher-based sensitivity estimation have already been extensively studied [1][2][3].
  2. The Fisher computation requires additional calibration sets and backward propagation, which will introduce additional overheads. Its practical capability is not sufficiently discussed.
  3. For Hessian-based methods, the regularizer is implemented through diagonal loading. But for AWQ and SliderQuant, the authors incorporate the Fisher penalty using additional adaptive normalization factors. The paper does not sufficiently analyze whether the different implementations preserve the same optimization interpretation.
  4. The main paper contains only three figures and two tables. Although more analysis and results are summarized into the appendix, the limited visual and experimental analysis in the main paper weakens the presentation of the paper. In addition, several figures and tables are too small and dense to clearly convey their findings.

[1] Kim S, Hooper C, Gholami A, et al. Squeezellm: Dense-and-sparse quantization[J]. arXiv preprint arXiv:2306.07629, 2023.

[2] Li Y, Gong R, Tan X, et al. Brecq: Pushing the limit of post-training quantization by block reconstruction[J]. arXiv preprint arXiv:2102.05426, 2021.

[3] Wu Z, Wang S, Zhang J, et al. FIMA-Q: Post-training quantization for vision transformers by fisher information matrix approximation[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 14891-14900.

Questions for authors
  1. Please provide more discussion about the differences among this paper and the listed related works.
  2. Please provide the computational overhead comparison among FR-PTQ and existing baselines.
  3. How does the adaptive normalization in AWQ and SliderQuant relate to the fixed regularization objective in Eq3?
5 comments

We thank the reviewer for the careful review, and for noting the problem setting of the mismatch between local reconstruction objectives and downstream performance, the ease of integration into existing methods, and the broad experiments from 1B to 70B.

Before our responses, we report one issue. Prompted by Reviewer 4aTP's Q3, we found a bug in our GPTQ implementation for Llama-family models, in which only one calibration sequence was effectively used. The bug is limited to GPTQ on Llama-family models, and GPTQ on Qwen3-8B, AWQ, and QuIP are not affected. We re-ran GPTQ on Llama-3-8B, our main evaluation model, after the fix. Unless otherwise noted, all GPTQ numbers below are after the fix.

Unless otherwise noted, all tables report Llama-3-8B results as mean ± SEM over 5 seeds. MC = average of the six multiple-choice tasks, Non-MC = average of LAMBADA, GSM8K, and TriviaQA, PPL = average of WikiText-2 and PTB. "w/Generic" is FR-PTQ with the generic (C4) Fisher, with λ selected from {0.1, 1, 2} by C4 perplexity. Bold marks the best value in each comparison, and "–" means not evaluated or PPL > 1000.

W1 (Q1). Hessian regularization and Fisher-based sensitivity have been studied in SqueezeLLM, BRECQ, and FIMA-Q, and the differences should be discussed.

The novelty lies not in using the Fisher itself but in defining an objective that goes beyond the reconstruction error.

FR-PTQ starts from the preliminary experiment in Figure 1, where calibrating on task data improves the perplexity on the task text, which the reconstruction error approximates, but does not improve task accuracy. The problem is thus not the calibration data but the reconstruction objective itself. FR-PTQ therefore adds a Fisher term, a second-order approximation of the loss, to the reconstruction error (Eq. 3, Prop. 2).

BRECQ and GuidedQuant weight the reconstruction error per token or per output using loss gradients. Their Hessian has the form Σ_n s_n x_n x_nᵀ, which remains a weighted sum of second moments of the calibration activations for any choice of weights. They are therefore refinements of the reconstruction error, and directions that the calibration activations rarely excite are not protected. The Fisher term of FR-PTQ is an input-channel diagonal outside this form, and it adds loss information from outside the reconstruction error (see also our response to Reviewer 4aTP, Q6).

The differences from the individual methods are also discussed in Sec. 2.3.

  • SqueezeLLM uses only the per-weight Fisher diagonal as its objective, without a reconstruction term, in a weighted k-means for a non-uniform quantizer. FR-PTQ keeps the reconstruction term and adds an input-channel Fisher term, so it exactly recovers the original solver at λ = 0 (Prop. 1). For Hessian-based solvers, a Fisher term alone would make the Hessian diagonal and disable the error compensation across columns. FR-PTQ also keeps the uniform quantizer, so existing quantized kernels can be used for inference as they are.
  • BRECQ weights the block reconstruction per output using the output-side Fisher. This weighting breaks the shared Hessian that closed-form solvers factorize only once. FR-PTQ only adds an input-channel diagonal, so it preserves the shared Hessian and plugs directly into GPTQ and QuIP.
  • FIMA-Q targets ViTs and replaces the reconstruction loss with an approximation of the FIM. FR-PTQ differs in that it keeps the reconstruction term and adds the Fisher term, exactly recovering the original solver at λ = 0 (Prop. 1). We will add FIMA-Q to the related work.

We also confirmed that this input-channel diagonal has an effect of its own, through comparisons with damping of the same total diagonal and with a shuffled Fisher of the same value distribution (Tables R3 and R4 in our response to Reviewer 1jfn). We also compared FR-PTQ with GuidedQuant (Kim et al., 2025), a recent Fisher-based PTQ method that weights the layer-wise reconstruction with end-loss gradients, and evaluated their combination under matched models, bit widths, tasks, and calibration data (Table R10 in our response to Reviewer 4aTP). FR-PTQ outperforms GuidedQuant on GPTQ, and the combination gives the best Non-MC for AWQ at INT3 and for QuIP at INT3 and INT4, showing that the two are complementary.

W2 (Q2). The Fisher computation requires additional calibration data and backpropagation, and the overhead compared with existing baselines is not provided.

Compared with the baselines (GPTQ, AWQ, and QuIP alone), the only additional cost is a single Fisher computation. The Fisher computation runs one forward and one backward pass per sample and accumulates per-input-channel values in each layer, without any training. The computational overhead is also discussed in Sec. 5.5 and Appendix D.3, and the measurements in Appendix Table 7 are as follows.

ModelFisher (1,024 sequences)One quantization run (GPTQ / AWQ / QuIP)
Llama-3-8Babout 5 minabout 7–14 min
Llama-3-70B (4 GPUs)about 30 minabout 56 min – 3.2 h

The Fisher computation costs no more than about one quantization run and does not change the order of the overall quantization time (minutes for 8B and hours for 70B). The Fisher depends only on the model and the estimation data, so it is shared across solvers, bit widths, and λ, and is further amortized when several settings are quantized. FR-PTQ stores only one value per channel, 4.8 MB for Llama-3-8B. The quantized weight format is the same as the baseline, so there is no additional inference cost.

Regarding additional calibration data, estimating the Fisher on the same data as the calibration still yields the gain (Table R1 in our response to Reviewer 1jfn), so additional data is not required.

W3 (Q3). How does the adaptive normalization in AWQ and SliderQuant relate to the fixed regularization objective in Eq. 3?

The integration into each solver is described in Appendix B.3 and Algorithm 2. Here we clarify the relation to Eq. 3.

The normalization factor of AWQ, γ = mean(L_recon) / mean(L_F), is a constant computed once per block. AWQ therefore solves, in each block, exactly Eq. 3 with the effective coefficient λγ, restricted to the AWQ scaling family. The role of γ is to make λ scale-invariant across layers, the same role as the per-layer mean-1 normalization of the Fisher in the Hessian-based solvers (Eq. 14). The effect of the restriction to the scaling family is covered by Prop. 4, which shows that the optimum over the restricted family can be no better than the optimum of the Hessian-based solvers. SliderQuant recomputes γ at every optimization step, so its correspondence to Eq. 3 is approximate. We will state that AWQ solves Eq. 3 with the effective coefficient λγ restricted to its scaling family, together with its relation to Prop. 4, in Appendix B.3.

W4. The main paper contains only three figures and two tables, much of the experimental analysis is in the appendix, and several figures and tables are too small and dense.

While keeping the structure of the main text, we will further strengthen its figures and experimental analysis following the reviewer's suggestion. We will add a figure that illustrates the overall method to the main text to improve readability. The additional experimental analyses conducted for this rebuttal will be added to the appendix and referenced from the main text. We will also revise the small and dense figures and tables into less crowded formats that are easier to read.

Dear Reviewer,

Thank you again for your time and for the constructive review. We have posted our response, which addresses each of your concerns and includes additional experimental results that support our answers. As the discussion period is coming to an end, we would like to ask whether our response has resolved your concerns. If any points remain unclear, we would be glad to clarify them within the remaining time.

Best regards, The Authors

Reviewer_4aTPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FR-PTQ, which adds a Fisher-weighted quadratic penalty to the per-layer objective of existing layer-wise PTQ solvers. The motivation is a preliminary study on Llama-3-8B showing that switching the calibration data to the target task improves perplexity on that task but leaves accuracy unchanged, which the paper reads as locating the mismatch in the objective rather than the data. The penalty weights each input channel by the Fisher diagonal of a surrogate loss, so for solvers that expose the per-layer second moment it acts as a diagonal loading of the Hessian. Two estimators are studied, a Task Fisher computed from a labeled task split and a label-free Generic Fisher computed from the generic calibration data under a last-token cross-entropy. Experiments cover five models from 1B to 70B, four solvers, and 2 to 4 bits, with five seeds per cell. Generic Fisher is reported to match Task Fisher and to improve mean multiple-choice accuracy on every model, by up to 6.3 points, and generative accuracy by up to 9.4 points. The appendix adds proofs, a channel-overlap analysis of the Fisher estimators, and hyperparameter sweeps.

Critical strengths
  1. The paper isolates its own contribution from the calibration data. Table 9 runs the same task-matched calibration with and without the regularizer, so the difference between the two rows comes from the regularizer alone.
  2. The paper tests why the label-free estimator works as well as the task-specific one, rather than only reporting that it does. Appendix D.6 compares the two directly and shows they agree on which channels matter.
Critical weaknesses
  1. The method helps mainly where the baseline has already failed. The paper reports this and treats it as benign, noting that the regularizer helps where the baseline leaves room and otherwise stays within seed noise (Sec. 5.2). Both headline numbers come from baselines far below the unquantized model. In the settings a practitioner would deploy, where the baseline is close to the unquantized model, the gain is within noise, and 12 of the 45 cells in Table 2 are regressions.
  2. The improvement is not shown to come from the Fisher information. The regularizer adds a correction far larger than the damping the solver already applies, and nothing separates the size of that correction from the Fisher weighting it carries. Proposition 1(ii) names the uniform case but it is never run. The paper's own evidence points the same way, since the solver with the largest gains lowers the error evenly across all channels rather than shifting it toward the important ones, which the paper attributes only to a solver with smaller gains (Sec. 5.4). The task-specific and generic estimators are interchangeable, so no task information is needed. GuidedQuant [1], which also brings end-loss gradients into this objective and works with any solver, is neither cited nor compared, and QEP [2] is cited but not evaluated.

References

[1] Jinuk Kim, Marwa El Halabi, Wonpyo Park, Clemens J. S. Schaefer, Deokjae Lee, Yeonhong Park, Jae W. Lee, and Hyun Oh Song. GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance. ICML, 2025.

[2] Yamato Arai and Yuma Ichikawa. Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization. NeurIPS, 2025.

Questions for authors

Clarification requests:

  1. Report the unquantized reference accuracy and perplexity for every model in Tab. 2 and Tab. 6, which currently give no reference outside Llama-3-8B.
  2. Report the lambda selected in each cell of Tab. 1 and Tab. 2.
  3. Explain why the GPTQ baseline collapses at 3 bits on Llama-3-8B and stays near chance on Llama-3-70B at 3 and 4 bits, and state whether the gains hold against a grouped GPTQ configuration.
  4. Explain why the QuIP baseline falls far below FP16 on Qwen3-8B and Llama-3-70B at 4 bits.
  5. Reconcile Sec. 5.4 with the posited mechanism, given that the solver with the largest gains shows a uniform error reduction rather than reallocation toward high-Fisher channels.
  6. Position FR-PTQ against GuidedQuant [1], which already incorporates end-loss gradient information into the layer-wise objective.

Requested experiments:

  1. Run the regularizer with the Fisher diagonal replaced by a uniform diagonal of the same mean and by a shuffled Fisher diagonal, under the same lambda selection.
  2. Compare against GuidedQuant [1] and QEP [2] under matched models, bit widths, and tasks.
  3. Report the generative tasks on models other than Llama-3-8B.
7 comments

We thank the reviewer for the careful review, and for noting that Table 9 isolates the regularizer's contribution from the calibration data and that App. D.6 directly tests why the label-free estimator works.

Before our responses, we report one issue. Prompted by the reviewer's Q3, we found a bug in our GPTQ implementation for Llama-family models, in which only one calibration sequence was effectively used. The bug is limited to GPTQ on Llama-family models, and GPTQ on Qwen3-8B, AWQ, and QuIP are not affected. We re-ran GPTQ on Llama-3-8B, our main evaluation model, after the fix. Unless otherwise noted, all GPTQ numbers below are after the fix.

Unless otherwise noted, all tables report Llama-3-8B results as mean ± SEM over 5 seeds. MC = average of the six multiple-choice tasks, Non-MC = average of LAMBADA, GSM8K, and TriviaQA, PPL = average of WikiText-2 and PTB. "w/Generic" is FR-PTQ with the generic (C4) Fisher, with λ selected from {0.1, 1, 2} by C4 perplexity. Bold marks the best value in each comparison, and "–" means not evaluated or PPL > 1000.

W1. The method helps mainly where the baseline has already failed, while elsewhere the gains are within noise and 12 of 45 cells in Table 2 degrade.

We view this behavior as a strength rather than a weakness. Whether layer-wise reconstruction degrades depends on the bit width, the model, and the solver, and is not known in advance. FR-PTQ mitigates the degradation where it occurs and barely affects healthy settings, so it can be applied without knowing in advance which settings will fail. Even where the baseline is close to FP16, it does not hurt. For grouped GPTQ (Table R2), MC is on par at INT4 (+0.01) with a small PPL change (+0.08), and at INT3 it still lowers PPL from 12.92 to 11.27. Of the 12 degraded cells in Table 2, 9 are within 0.31 points, and the mean change is positive for every model (+0.74 to +1.08).

W2. The improvement is not shown to come from the Fisher information, since the correction magnitude and the Fisher weighting are not separated, the uniform case of Prop. 1(ii) is not run, and GuidedQuant and QEP are not compared.

We ran the uniform case of Prop. 1(ii) (damping) and a shuffled Fisher (Q7, Tables R3 and R4). We acknowledge that damping itself is effective. However, with the total diagonal matched, FR-PTQ outperforms damping and shuffle in INT3 PPL, and combining it with damping improves further. Hence the Fisher weighting has an effect distinct from the correction magnitude. The comparison with GuidedQuant and the relation to QEP are discussed in Q6 and Q8.

Q1. Report the FP16 reference for every model in Tables 2 and 6.

We will add the FP16 results for all models in Tables 2 and 6.

Table R5: FP16 references

ModelMC (↑)Non-MC (↑)PPL (↓)
Llama-3.2-1B52.9027.6613.16
Llama-3.2-3B59.8141.9310.23
Llama-3-8B63.8549.268.36
Llama-3-70B69.6869.385.51
Qwen3-8B63.7151.1713.33

Q2. Report the λ selected in each cell of Tables 1 and 2.

λ is chosen from {0.1, 1, 2} as the value that minimizes the seed-averaged C4 perplexity of the calibration data (ties are broken toward 1). Evaluation-task results are not used for the selection. We will add the selected λ for every cell to the appendix.

Table R6: Selected λ (GPTQ after the fix with "–" for models not re-run, and AWQ and QuIP as in Tables 1 and 2)

ModelGPTQ INT3GPTQ INT4AWQ INT3AWQ INT4QuIP INT3QuIP INT4
Llama-3.2-1B––120.10.1
Llama-3.2-3B––210.11
Llama-3-8B0.112110.1
Llama-3-70B––1210.1
Qwen3-8B0.10.1210.11

Q3. Explain why the GPTQ baseline collapses at 3 bits on Llama-3-8B and stays near chance on Llama-3-70B, and whether the gains hold for grouped GPTQ.

The collapse was caused by the implementation bug reported above.

Table R7: GPTQ before and after the fix

MethodINT3 MC (↑)INT3 Non-MC (↑)INT3 PPL (↓)INT4 MC (↑)INT4 Non-MC (↑)INT4 PPL (↓)
FP1663.8549.268.3663.8549.268.36
GPTQ (before fix)33.12 ± 0.080.16 ± 0.06–58.23 ± 0.8630.30 ± 1.7612.05 ± 0.37
w/Generic (before fix)38.14 ± 0.172.02 ± 0.23109.34 ± 12.7061.57 ± 0.0739.69 ± 0.3410.23 ± 0.04
GPTQ (after fix)43.36 ± 1.236.78 ± 0.5135.42 ± 1.8261.71 ± 0.2341.33 ± 0.369.68 ± 0.07
w/Generic (after fix)46.49 ± 0.7610.55 ± 0.3927.72 ± 0.5861.83 ± 0.0641.65 ± 0.249.68 ± 0.01

After the fix, GPTQ no longer collapses. We attribute the near-chance GPTQ results on Llama-3-70B at INT3 and INT4 (MC 32.77 / 32.79 in Table 2, where the chance level of the six tasks is about 33.3) to the same bug. We could not re-run Llama-3-70B within the rebuttal period. After the fix, FR-PTQ still lowers INT3 PPL from 35.42 to 27.72 and raises INT3 Non-MC from 6.78 to 10.55. For grouped GPTQ (Table R2), the gain remains at INT3. At INT4, where the baseline is close to FP16, MC is on par (+0.01) and the PPL difference is small (+0.08).

Q4. Explain why the QuIP baseline falls far below FP16 on Qwen3-8B and Llama-3-70B at 4 bits.

The low QuIP accuracy is not specific to our implementation. On Llama-3-70B, our QuIP numbers are higher than those reported in PTQ-Bench (Zhao et al., 2025).

Table R8: QuIP on Llama-3-70B, MC (↑), compared with PTQ-Bench

MethodINT3 PTQ-BenchINT3 OursINT2 PTQ-BenchINT2 Ours
QuIP37.7345.82 ± 2.5534.5437.18 ± 0.59

QuIP also falls far below FP16 on Llama-3-70B, so the issue is not specific to Qwen3-8B. Given the large seed variance, we believe that the seed dependence of the random orthogonal transform is one cause. We will continue to investigate and report the findings in the appendix of the final version.

Q5. Reconcile Sec. 5.4 with the posited mechanism, given the uniform error reduction of the solver with the largest gains.

The GPTQ panel of Figure 3 was produced before the fix. In GPTQ, adding any diagonal weakens the error compensation and lowers the weight error in all channels regardless of the Fisher (the damping component), and we believe the uniform reduction in Figure 3 included this effect. We therefore re-ran the analysis on the fixed GPTQ, using w/damping with the same uniform damping as the reference, so that the uniform damping component cancels (Table R9).

Table R9: Change in weight MSE per Fisher-importance decile (GPTQ INT4, w/damping + Generic vs. w/damping, λ_iso = λ_F = 0.1, seed 42)

Fisher rankChange
0–10%−0.5%
10–20%−2.0%
20–30%−2.4%
30–40%−3.0%
40–50%−3.5%
50–60%−3.8%
60–70%−4.1%
70–80%−4.9%
80–90%−6.1%
90–100%−8.4%
All−3.7%

The Fisher term still adds some diagonal mass on average, which contributes a uniform part of the reduction, but the reduction grows monotonically with the Fisher importance (−0.5% in the bottom decile and −8.4% in the top decile), which reflects how the Fisher allocates its weight across channels. This is consistent with the posited mechanism of selectively suppressing the error of important channels. We will replace Figure 3 with this comparison. The shuffle comparison (Table R3) also shows that the channel-wise placement of the Fisher matters. Destroying the assignment of values to channels while keeping their distribution raises INT3 PPL from 27.72 to 40.43.

Q6. Position FR-PTQ against GuidedQuant.

FR-PTQ and GuidedQuant differ in their starting point and in the form of the objective.

FR-PTQ starts from the preliminary experiment in Figure 1, where calibrating on task data improves the perplexity on the task text, which the reconstruction error approximates, but does not improve task accuracy. The problem is thus not the calibration data but the reconstruction objective itself. FR-PTQ therefore adds a Fisher term, a second-order approximation of the loss, to the reconstruction error (Eq. 3, Prop. 2).

GuidedQuant minimizes a reconstruction error weighted per token and output channel by end-loss gradients. Its Hessian has the form H̄_k = Σ_n s_{n,k} x_n x_nᵀ, which remains a weighted sum of second moments of the calibration activations for any choice of weights. GuidedQuant is therefore a refinement of the reconstruction error, and directions that the calibration activations rarely excite are not protected. The Fisher term of FR-PTQ is an input-channel diagonal outside this form, and it adds loss information from outside the reconstruction error.

There are two further structural differences.

  1. FR-PTQ does not require token alignment, so the data and the number of samples for Fisher estimation can be chosen independently of calibration. The calibration can stay on generic text while the Fisher is estimated on task data with a task loss, such as a multiple-choice loss (Task Fisher).
  2. FR-PTQ stores only one value per channel, 4.8 MB for Llama-3-8B, independent of the number of samples (GuidedQuant's per-token weights with g = 1 take 112.6 MB for 128 sequences and 900.1 MB for 1,024 sequences).

The two act in orthogonal directions and can be combined (Table R10). We will add GuidedQuant to the related work.

Q7. Run the regularizer with a uniform diagonal of the same mean and with a shuffled Fisher diagonal, under the same λ selection.

Please see Tables R3 and R4 in our response to Reviewer 1jfn (W3). Under the same λ selection, FR-PTQ outperforms both controls in INT3 PPL and is clearly more robust to λ, and combining it with damping improves further.

Q8. Compare against GuidedQuant and QEP under matched models, bit widths, and tasks.

We evaluated GuidedQuant with the same models, bit widths, tasks, and calibration data (g = 4 is the setting of the GuidedQuant paper).

Table R10: Comparison with GuidedQuant

MethodINT3 MC (↑)INT3 Non-MC (↑)INT3 PPL (↓)INT4 MC (↑)INT4 Non-MC (↑)INT4 PPL (↓)
GPTQ43.36 ± 1.236.78 ± 0.5135.42 ± 1.8261.71 ± 0.2341.33 ± 0.369.68 ± 0.07
w/GuidedQuant (g=1)41.31 ± 0.54–108.93 ± 23.9060.80 ± 0.32–9.73 ± 0.13
w/GuidedQuant (g=4)42.39 ± 0.96–123.53 ± 58.8661.42 ± 0.42–9.93 ± 0.19
w/Generic46.49 ± 0.7610.55 ± 0.3927.72 ± 0.5861.83 ± 0.0641.65 ± 0.249.68 ± 0.01
w/GuidedQuant (g=1) + Generic44.22 ± 0.347.72 ± 0.3043.90 ± 6.4062.00 ± 0.1241.14 ± 0.489.69 ± 0.04
AWQ59.96 ± 0.0037.68 ± 0.0010.63 ± 0.0063.52 ± 0.0047.59 ± 0.008.81 ± 0.00
w/GuidedQuant (g=1)59.58 ± 0.0037.35 ± 0.0010.65 ± 0.0063.58 ± 0.0048.04 ± 0.008.79 ± 0.00
w/Generic59.94 ± 0.0437.98 ± 0.0410.63 ± 0.0063.27 ± 0.0647.43 ± 0.218.79 ± 0.00
w/GuidedQuant (g=1) + Generic59.92 ± 0.0538.75 ± 0.0810.64 ± 0.0063.22 ± 0.0147.63 ± 0.118.80 ± 0.00
QuIP59.21 ± 0.3230.63 ± 0.8210.87 ± 0.0962.45 ± 0.1842.84 ± 0.819.20 ± 0.06
w/GuidedQuant (g=1)59.86 ± 0.1933.54 ± 0.9610.41 ± 0.0362.62 ± 0.2345.20 ± 0.629.06 ± 0.07
w/Generic59.46 ± 0.2333.18 ± 1.5510.41 ± 0.0662.50 ± 0.2642.01 ± 1.149.13 ± 0.07
w/GuidedQuant (g=1) + Generic59.80 ± 0.2833.92 ± 0.9910.35 ± 0.0562.86 ± 0.1645.37 ± 0.269.03 ± 0.02

For GPTQ, GuidedQuant falls below the baseline, while FR-PTQ is the best or tied in all metrics except INT4 MC. For QuIP, the combination gives the best PPL and Non-MC at both bit widths and the best MC at INT4. For AWQ, all methods are within 0.4 points in MC, and GuidedQuant is slightly better at INT4.

QEP corrects the reconstruction target of each layer by accounting for the quantization error propagated from the preceding layers. Like GuidedQuant, it is a refinement of the reconstruction error. Since FR-PTQ keeps the reconstruction term and adds the Fisher term, it does not compete with QEP and can be added directly to the QEP-corrected reconstruction error.

Reviewer_1jfnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Layer-wise post-training quantization usually minimizes reconstruction error, but better reconstruction does not always preserve downstream accuracy. This paper adds a Fisher-weighted penalty to protect weight directions that have a larger effect on the loss. The Fisher scores can be estimated from labeled task data or ordinary text. The authors evaluate five models from 1B to 70B at 2–4 bits, applying the penalty to GPTQ, AWQ, QuIP, and, in a smaller study, SliderQuant. The results show improvements in accuracy and perplexity in many settings. Generic-text estimates perform similarly to task-specific estimates, although the gains vary considerably across quantizers.

Critical strengths
  1. The method is simple and can be added to existing quantizers without changing the final weight format or adding inference cost. Using generic text instead of task labels also makes it easier to apply.
  2. The evaluation covers several model sizes and bit widths, reports results over five seeds, and includes sensitivity studies. Comparing each quantizer with and without the penalty is a reasonable way to test the proposed change.
Critical weaknesses
  1. The proposed method uses more data than the baseline. Standard calibration uses 128 sequences, while Fisher estimation uses another 1,024 sequences. Sharing the same source dataset does not mean sharing the same data budget. A baseline using more calibration data would help separate the benefit of the Fisher objective from the benefit of additional statistics.
  2. Some large gains come from very weak configurations. GPTQ uses no finer grouping and remains near chance accuracy on Llama-3-70B even at 4 bits. These results do not invalidate the within-quantizer comparisons, but leave open whether the gains remain useful with stronger grouped configurations. SliderQuant provides a recent baseline, but is tested on only one model, and broader cross-model evaluation mainly covers multiple-choice tasks.
  3. The paper lacks a direct comparison against simpler regularization choices. Since the method adds a diagonal term to the curvature matrix, some improvements may come from stronger damping or better numerical stability. The authors should compare against tuned ordinary damping and shuffled Fisher scores under the same data and search budgets.
Questions for authors

See weakness above.

5 comments

We thank the reviewer for the careful review, and for noting that the method can be added to existing quantizers without changing the weight format or inference cost, that it works with generic text, and that the evaluation includes five seeds and sensitivity studies.

Before our responses, we report one issue. Prompted by Reviewer 4aTP's Q3, we found a bug in our GPTQ implementation for Llama-family models, in which only one calibration sequence was effectively used. The bug is limited to GPTQ on Llama-family models, and GPTQ on Qwen3-8B, AWQ, and QuIP are not affected. We re-ran GPTQ on Llama-3-8B, our main evaluation model, after the fix. Unless otherwise noted, all GPTQ numbers below are after the fix.

Unless otherwise noted, all tables report Llama-3-8B results as mean ± SEM over 5 seeds. MC = average of the six multiple-choice tasks, Non-MC = average of LAMBADA, GSM8K, and TriviaQA, PPL = average of WikiText-2 and PTB. "w/Generic" is FR-PTQ with the generic (C4) Fisher, with λ selected from {0.1, 1, 2} by C4 perplexity. Bold marks the best value in each comparison, and "–" means not evaluated or PPL > 1000.

W1. The method uses more data than the baseline (128 calibration sequences plus 1,024 Fisher sequences).

We compared a baseline calibrated on 1,024 sequences with FR-PTQ that uses the same 1,024 sequences for both calibration and Fisher estimation. The two therefore use an identical data budget.

Table R1: Matched data budget (calibration = Fisher = 1,024 sequences)

MethodINT3 MC (↑)INT3 PPL (↓)INT4 MC (↑)INT4 PPL (↓)
GPTQ47.55 ± 0.8329.40 ± 1.7661.83 ± 0.209.75 ± 0.12
w/Generic48.68 ± 0.3922.90 ± 0.8662.10 ± 0.159.64 ± 0.01
AWQ60.35 ± 0.0010.67 ± 0.0063.64 ± 0.008.80 ± 0.00
w/Generic60.36 ± 0.0510.68 ± 0.0063.79 ± 0.058.80 ± 0.00
QuIP59.07 ± 0.3610.64 ± 0.0762.50 ± 0.209.25 ± 0.02
w/Generic60.43 ± 0.1710.30 ± 0.0963.03 ± 0.219.11 ± 0.02

Under the same data budget, FR-PTQ improves both MC and PPL for GPTQ and QuIP at INT3 and INT4 (MC 47.55 → 48.68 and PPL 29.40 → 22.90 for GPTQ INT3, and MC 59.07 → 60.43 and PPL 10.64 → 10.30 for QuIP INT3), and AWQ is on par with the baseline. The gain therefore does not come from additional data. Using the same data also for Fisher estimation is more effective than using it only for calibration. We will add this comparison to the appendix and refer to it from the main text.

W2. Large gains may come from weak configurations (ungrouped GPTQ), and SliderQuant and the cross-model evaluation are limited.

We evaluated GPTQ with group size 128.

Table R2: Grouped GPTQ (group size 128)

MethodINT3 MC (↑)INT3 PPL (↓)INT4 MC (↑)INT4 PPL (↓)
FP1663.858.3663.858.36
GPTQ (g128)57.88 ± 0.4212.92 ± 0.6763.10 ± 0.088.75 ± 0.01
w/Generic58.64 ± 0.1311.27 ± 0.0763.11 ± 0.078.83 ± 0.02

INT3 g128 is a configuration in which the baseline does not collapse from FP16. FR-PTQ still lowers PPL from 12.92 to 11.27 and greatly reduces the seed variance (PPL SEM 0.67 → 0.07). At INT4, where the baseline is already close to FP16, MC is on par (+0.01) and the PPL difference is small (+0.08), so FR-PTQ barely affects performance. We attribute the near-chance GPTQ results on Llama-3-70B to the implementation bug reported above. After the fix, per-channel GPTQ on Llama-3-8B no longer diverges at INT3 (see our response to Reviewer 4aTP, Q3).

For generative tasks beyond Llama-3-8B, we evaluated Non-MC on Qwen3-8B for all three solvers at INT3 and INT4 (Table R11 in our response to Reviewer 4aTP). At INT4, Non-MC improves for all three solvers (GPTQ 34.11 → 37.94, AWQ 47.24 → 49.71, QuIP 9.29 → 11.72).

SliderQuant learns quantization parameters by gradient descent over sliding windows, so a single quantization run is far more expensive than GPTQ, AWQ, or QuIP, and we could not extend it to other models within the rebuttal period. The SliderQuant result is intended to show that FR-PTQ plugs into a recent learning-based solver without modification.

W3. No comparison with tuned ordinary damping and shuffled Fisher scores under the same data and search budgets.

For GPTQ, we compared ordinary damping and a shuffled Fisher under the same search budget as FR-PTQ (three candidates λ ∈ {0.1, 1, 2}, selected by C4 perplexity). We also tested combining damping with FR-PTQ.

  • w/damping uses Σx + λh̄I. Because the Fisher is normalized to mean 1 in each layer, the total added diagonal exactly matches FR-PTQ.
  • w/shuffle uses Σx + λh̄·Perm(F). The value distribution is identical to FR-PTQ, and only the assignment of values to channels is destroyed.
  • w/damping + Generic uses Σx + λ_iso h̄I + λ_F h̄F.

Table R3: Comparison with damping and shuffled Fisher (GPTQ)

MethodINT3 MC (↑)INT3 PPL (↓)INT4 MC (↑)INT4 PPL (↓)
GPTQ43.36 ± 1.2335.42 ± 1.8261.71 ± 0.239.68 ± 0.07
w/damping47.07 ± 0.3736.85 ± 7.0561.97 ± 0.159.61 ± 0.16
w/shuffle45.78 ± 0.7140.43 ± 1.7561.53 ± 0.259.61 ± 0.09
w/Generic46.49 ± 0.7627.72 ± 0.5861.83 ± 0.069.68 ± 0.01
w/damping + Generic47.25 ± 0.8122.39 ± 0.5762.28 ± 0.139.63 ± 0.09

Table R4: Robustness to λ (INT3, C4 PPL)

Methodλ = 0.1λ = 1λ = 2
w/damping30.79 ± 3.1640.95 ± 2.33146.70 ± 39.93
w/shuffle32.22 ± 2.5538.23 ± 3.8166.41 ± 17.52
w/Generic19.28 ± 0.2723.99 ± 0.1627.94 ± 0.12

As the reviewer suspected, damping alone is effective, and its MC is higher than that of FR-PTQ. In INT3 PPL, however, FR-PTQ clearly outperforms both damping and shuffle, and it remains stable as λ increases while both controls degrade sharply (Table R4). At INT4, the PPL differences are within the seed variation. Since the INT3 PPL gap remains even when the total diagonal and the value distribution are matched, this gain comes from which channels the Fisher emphasizes, not from the amount of damping. Combining the two lowers INT3 PPL further to 22.39 and gives the best MC, indicating that damping and the Fisher term have complementary effects. The combination selects (λ_iso, λ_F) from nine pairs, a larger search budget than the single methods. With the same three-candidate budget, FR-PTQ still outperforms both controls in INT3 PPL and in robustness to λ.

Dear Reviewer,

Thank you again for your time and for the constructive review. We have posted our response, which addresses each of your concerns and includes additional experimental results that support our answers. As the discussion period is coming to an end, we would like to ask whether our response has resolved your concerns. If any points remain unclear, we would be glad to clarify them within the remaining time.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.