acceptodds
Under review as a conference paper at ICLR 2027

A Unified Hessian-Guided Framework for Post-Training Quantization

Abstract

Low-bit quantization enables efficient deployment of large language models by reducing memory footprint and computational cost. Achieving high accuracy at low precision requires selecting quantization ranges, linear transformations, and weight rounding decisions. Existing approaches typically optimize these components independently using local reconstruction objectives, which fail to account for the varying impact of quantization errors on downstream predictions. Direct optimization of task-level metrics would provide a more faithful objective, but is challenging due to the discontinuous nature of quantization operations and the high computational cost of evaluating their downstream effects. In this work, we derive a Hessian-guided approximation of the task loss induced by joint weight and activation quantization. Based on a tractable Fisher approximation, we formulate a unified post-training quantization framework that enables the joint optimization of quantization ranges, linear transformations, and weight rounding under a common output-sensitivity objective. The resulting method remains practical for modern large language models and consistently outperforms state-of-the-art transformation and rounding methods across both channel-wise and block-wise quantization settings.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Quantization, Compression, Model Efficiency

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_QtFLOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a unified Hessian-guided framework for LLMs PTQ named HZ-PTQ. HZ-PTQ introduces a novel approximation method to estimate the downstream impact of weight and activation quantization errors, and then develop a common optimization method based on this approximation.

Critical strengths
  1. Using output sensitivity to guide quantization optimization is theoretically meaningful.
  2. The output Kronecker approximation makes the downstream-sensitivity task computationally tractable.
  3. The method achieves consistent competitive performance on four model families and diverse configurations.
Critical weaknesses
  1. The derivations rely on several approximations, including rank-one Fisher factorization, independence between weight and activation quantization errors and negligible clipping effects. However, these assumptions may not hold in some cases. The paper does not sufficiently evaluate how accurately these approximations reflect the actual downstream loss.
  2. Eq11 derives a quantization-error objective that depends on both transformation-dependent quantization ranges and gradient sensitivities. However, Eq. (12) optimizes only the gradient terms. Therefore, the transformation is not necessarily optimal for the objective in Eq11. The paper lacks of justification or empirical validation of this simplification.
  3. Calibration cost is an important metric to evaluate a PTQ method. However, this paper lacks of reporting the calibration time, peak memory consumption and computational overhead compared with existing PTQ method.
Questions for authors
  1. Can the authors evaluate the approximation error in W1?
  2. Can the authors evaluate whether the transformations actually minimize the original objective in Eq11 in W2?
  3. Please provide the calibration cost of the proposed method compared with baselines in W3?
1 comment
AuthorsOpenReview ↗

Response to Reviewer QtFL

We thank the reviewer for the precise questions, for finding our use of output sensitivity "theoretically meaningful" and the Output Kronecker approximation "computationally tractable", and for noting the "consistent competitive performance".

1. Approximation Validation

We extend the comparison in Figure 1 to more models and lower bit widths, and add the decomposition in Eq. 8, which assumes independent weight and activation errors (see Table A in our General Response). The factorized proxy (Eq. 6) tracks the full output-Fisher proxy (Eq. 5) closely, and the decomposition (Eq. 8) leaves the correlation with output KL unchanged. This holds across models (0.97–0.99) and down to W2A8 (0.71), where the commonly used MSE shows no correlation (−0.02). The decomposition of activation and weight quantization error is further motivated by existing work in the literature [1].

We recognize that our transform derivation neglects the effect of clipping, however, the additional results reported in Table D confirm that our approximate solution contributes to the reduction of the Fisher-weighted quantization error. We will add these additional references and results to the paper.

2. Transform Optimality

Eq. 12 optimizes only the gradient terms of Eq. 11, but those terms are invariant to orthogonal rotations. We therefore first find the transform M^\hat{\mathbf{M}} that minimizes Eq. 12, then apply a Hadamard rotation. Being orthogonal, it leaves the value of Eq. 12 unchanged while reducing the quantization ranges in Eq. 11. To check that this approximately minimizes the target objective, we compare the value of the factorized Fisher-weighted error under our transform, the Hadamard transform (QuaRot) and the CAT [1] transform (Table D). Our transform attains the lowest average Fisher-weighted error. We will add these results to the paper.

Method∥ΔZT∥22\Vert\Delta Z_{T}\Vert_2^2 [dB]∥ΔZT∥FZ2\Vert\Delta Z_{T}\Vert_{\bf F_Z}^2 [dB]∥ΔZT∥⊗FZ2\Vert\Delta Z_{T}\Vert_{\otimes{\bf F_Z}}^2 [dB]
RTN45.015.215.2
QuaRot41.312.212.3
CAT39.010.410.5
Hz-PTQ (transform only)39.210.010.1

Table D: Average layer-output error under each transform on Llama 3.2 1B Instruct with W4A4 rounding after the transform.

The values are reported in decibels [dB] (10⋅log⁡1010\cdot\log_{10} value) for readability. Note that while CAT attains the lowest L2 error., our transform reaches lower values for the Fisher-weighted proxy.

3. Calibration Time and Memory

The memory and runtime costs of our method are reported in Table B of our General Response. Our method takes 158 min and 32 GB on Llama 3.2 1B Instruct, versus 96 min and 32 GB for YAQA, 127 min and 25 GB for FlatQuant+GPTQ, and 74 min and 25 GB for CAT+GPTQ. We will include them in the paper. Our method extends YAQA's [2] Kronecker-factored curvature to activation quantization, transform selection and range setting. It therefore inherits the overhead of estimating the Kronecker-factored curvature, here with respect to the layer outputs, which takes time and compute that are essentially equivalent to YAQA. Transform and range selection are performed layer by layer and have low peak memory. Concurrent and complementary work, [3], speeds up two-sided LDLQ and proposes a lower-memory procedure for computing the Kronecker-factored Hessians. These improvements apply directly to our method and would likely reduce this overhead.

References

[1] M. Federici, B. van Breugel, P. N. Whatmough, M. Nagel. "Dissecting Quantization Error: A Concentration-Alignment Perspective". ICML 2026.

[2] A. Tseng, Z. Sun, C. De Sa. "Model-Preserving Adaptive Rounding". ICML 2026.

[3] J. Birnick, R. Saab. "FastKron: Efficient Quantization with Kronecker-Factored Hessians". arXiv:2608.06291, 2026.

Reviewer_c3mGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies post-training quantization for language models. This paper approximates the output KL divergence between the full model and the quantized model with a quadratic function of each layer's output error. It then factorizes the output Fisher matrix across token and feature dimensions. The proposed method uses this proxy to choose a transformation, weight scales and rounding, and activation clipping in sequence. In experiments, this paper evaluates the proposed method on four language models under channel-wise and block-wise 4-bit quantization. Baselines include round-to-nearest, QuaRot, CAT, FlatQuant, and WUSH. Evaluation uses WikiText perplexity and average accuracy over six reasoning tasks: PIQA, WinoGrande, HellaSwag, ARC-e, ARC-c, and LAMBADA. In W4A4KV4, WikiText perplexity is 1.6–6.9% lower than the strongest baseline. Compared to this baseline, the proposed method improves reasoning accuracy by up to 1.3 points on three models and is lower on Llama 3 8B. The proposed method also reports the lowest perplexity in the block-wise settings.

Critical strengths
  • This paper compares the proposed output-error proxy with downstream prediction divergence. This reports a strong correlation for the proposed factorization, which justifies the method.
  • In experiments, this paper evaluates four models from Llama, Ministral, and Qwen across channel-wise and block-wise settings. The proposed method reduces WikiText perplexity relative to the baselines in these settings.
Critical weaknesses
  • The contribution over related work is unclear. For example, YAQA has used Kronecker-factored curvature for rounding. This paper applied these ideas to the Fisher matrix approximation. The contribution to existing methods remains uncertain.

  • This method proposes a procedure to sequentially determine the transformation matrix, weight scales, rounding, and activation clipping. However, the original problem is a joint optimization problem. The reason why the proposed method follows such a process is unclear.

  • The proposed method involves many components, such as the Hessian approximation via the Kronecker factorization, optimal transport for the transformation matrix, and solving problems for other quantization parameters. How each part affects the final performance and their relative importance is understudied. For example, how does the downstream performance relate to the approximation accuracy of the Hessian matrix?

  • Further, the notations are generally hard to parse. Figure 1 uses G_W, G_Z, Vec, an expectation, and dB scales without defining them. Since Section 2 defines ΔZ = Z − Z̃, the first-order term in Equation (3) appears to have the opposite sign. This term disappears in the KL objective in Equation (4).

Questions for authors
  • What are the contributions of the output-Fisher approximation, transformation, scale/rounding, and clipping?
  • Why is the transformation chosen before the other variables in Equation (2)? How is the order of the optimization determined?
3 comments
AuthorsOpenReview ↗

Response to reviewer c3mG

We thank the reviewer for recognizing that the strong correlation between our proxy and the output KL justifies the method, as well as our consistent perplexity improvements across four models in channel and block-wise settings.

1. Comparison with related Work

YAQA [1] uses Kronecker-factored curvature, but only for weight rounding, and its weight Fisher cannot capture activation quantization error (Fig. 1, col. 2). We instead derive an output-Fisher objective that covers weights and activations, and show how to tractably optimize every PTQ decision under it: a closed-form transform (Eq. 12) that generalizes CAT[2]/WUSH[3], Fisher-weighted scale selection that matches or beats the best-tuned LpL_p without tuning pp (Fig. 2) and activation clipping. A single curvature estimate works for all components. Our rounding procedure also improves YAQA with YAQA's own Hessian (18.81 → 15.34 PPL, Table 5), and HZH_Z-PTQ outperforms Had.+YAQA on all models (Table 3).

2. Sequential Optimization

We agree that fully joint optimization would be ideal, but it is intractable at LLM scale. We therefore optimize the same output-Fisher objective in sequence (Section 3 of our General Response). The transform comes first because it changes weights, activations and gradients alike, while its solution (Eq. 12) does not depend on the quantization error. Weight scales and rounding are then chosen together, since each group's scale is set after LDLQ compensation. Clipping comes last and uses the rounded weights. Optimizing weights and activations separately relies on the Eq. 8 decomposition, which leaves the correlation with the output KL unchanged in all six settings of Table A in our General Response. We will add these clarifications to Section 3.3.

3. Effect of HZH_Z-PTQ components

Table 4 ablates each component in turn. Removing the transform causes the largest degradation (up to 10.43 → 25.19 PPL on Qwen3 8B), followed by activation clipping, then the Hessian approximation and rounding. Replacing our Hessian approximation with a GPTQ-style one significantly degrades performance on three of four models.

The full output Fisher is too large to compare with our approximation directly, but we can compare the error proxies they induce: our Kronecker factorization (Eq. 6) predicts the output KL as well as the full Fisher (Eq. 5), staying within 0.03 in every setting (Figure 1; Table A in our General Response). Table 5 fixes the rounding algorithm and varies the curvature estimate. With our rounding, our approximation performs on par with or better than YAQA's, and it outperforms GPTQ's under every rounding algorithm. Figure 2 shows the contribution of our scale selection: it matches or beats the best-tuned LpL_p objective without tuning pp.

4. Notation

Thank you for the feedback on the notation. We will include the definitions to ensure that the quantities are clear and ensure consistency in the definition of the second-order Taylor expansion, correcting the sign in Equation 3. As you note, this does not affect Eq. 4 and onward, since the term disappears.

References

[1] A. Tseng, Z. Sun, C. De Sa. "Model-Preserving Adaptive Rounding". ICML 2026.

[2] M. Federici, B. van Breugel, P. N. Whatmough, M. Nagel. "Dissecting Quantization Error: A Concentration-Alignment Perspective". ICML 2026.

[3] J. Chen, V. Egiazarian, R. L. Castro, T. Hoefler, D. Alistarh. "WUSH: Near-Optimal Adaptive Transforms for LLM Quantization". ICML 2026.

Thanks to the authors for their detailed responses. My question about the difference from the existing work, YAQA, has been addressed. However, I found that the design for solving the joint optimization problem in multiple phases is not well justified. For example, it is unclear why this sentence holds: "The transform comes first because it changes weights, activations, and gradients alike, while its solution (Eq. 12) does not depend on the quantization error." Thus, I would like to maintain my original evaluations.

We thank the reviewer for their comments and engagement in the rebuttal process.

First, we want to highlight that the proposed minimization procedure is still an approximation that is designed to be as tractable as possible. Given this we can further motivate our sequential optimization procedure:

1. Fixing the transform before rounding is standard. QuIP#, CAT [2], WUSH [3] and all baselines in Tables 2 and 3 compute a transform and apply it before weight rounding, and so do we. Even FlatQuant (Sun et al., 2025), which learns its transform and clipping with quantization in the loop, rounds only afterwards. HZ\mathbf{H_Z}-PTQ, with a closed-form transform, still has lower perplexity on all four models (Table 2).

2. Our derivation supports this order. With WT=T−1WW_T=T^{-1}W, we replace the joint problem (Eq. 7) by T^=arg⁡min⁡T cxRx(T)E∥GXT−⊤∥F2+cwRw(T)E∥T⊤GW∥F2,\hat T=\arg\min_T\ c_xR_x(T)\mathbb{E}\Vert G_XT^{-\top}\Vert_F^2+c_wR_w(T)\mathbb{E}\Vert T^\top G_W\Vert_F^2, s^w,W^∗=arg⁡min⁡sw,W∗∥WT^−Q(W∗;sw)∥T^⊤E[GWGW⊤]T^⊗E[GZ⊤GZ]2,\hat s_w,\hat W^{\ast}=\arg\min_{s_w,W^{\ast}}\big\Vert W_{\hat T}-Q(W^{\ast};s_w)\big\Vert^2_{\hat T^\top\mathbb{E}[G_WG_W^\top]\hat T\otimes\mathbb{E}[G_Z^\top G_Z]}, with grid ranges Rx,RwR_x,R_w and bit- width constants cx,cwc_x,c_w.

The scale and rounding search depends on TT: its objective (Sec. 3.5) contains WT^W_{\hat T} and the curvature T^⊤E[GWGW⊤]T^\hat T^\top\mathbb{E}[G_WG_W^\top]\hat T, so it needs T^\hat T first.

The transform does not depend on the scale and rounding search: the minimum over sw,W∗s_w,W^{\ast} is an integer program with no closed form in TT. Ideally TT would minimize the error after LDLQ, but that couples TT to the discrete rounding, a much harder problem. We therefore use a tractable surrogate. Prior work shows that minimizing outliers and balancing expected activation and weight errors works well [2, 3], and we extend this objective to the downstream loss through the Hessian. The first line is this expected error for round-to-nearest under a uniform noise model (Eq. 11), which averages out rounding errors and leaves TT only in ranges and gradient norms. Since the expected LDLQ error is at most that of round-to-nearest (Chee et al., 2023, Thm. 1), the first line also bounds the LDLQ error. Eq. 12 and the Hadamard rotation approximately minimize the first line from full-precision statistics.

We further thank the reviewer in underlining the lack of clarity in our statement, we will update Sec. 3.3 to underline that our solution is still approximate in nature and add a summary of our answer to the main text.

Reviewer_RSWVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HZ-PTQ, a unified Hessian-guided framework for post-training quantization that aims to minimize the impact of quantization errors on downstream model predictions rather than relying on local reconstruction errors. The key contribution is a tractable Kronecker-factored approximation of the layer-output Fisher information matrix, which captures sensitivity to both weight and activation quantization errors. Based on this approximation, the authors develop a unified objective to guide transformation selection, weight quantization scale optimization, adaptive rounding, and activation clipping.

The authors validate their approach by showing a strong correlation between the proposed Fisher-weighted error proxy and the actual downstream KL divergence. They further evaluate HZ-PTQ on several LLMs across several 4-bits quantization settings, comparing against established PTQ methods including GPTQ, QuaRot, CAT, FlatQuant, WUSH, and YAQA. Experimental results demonstrate generally improved perplexity and competitive or improved reasoning accuracy. Ablation studies further examine the contributions of the individual optimization components.

Critical strengths
  1. Well-motivated and principled objective. The paper addresses an important limitation of reconstruction-based PTQ methods by considering the downstream impact of quantization errors. The proposed Output Kronecker Fisher approximation provides a tractable, theoretically motivated sensitivity metric that accounts for both weight and activation quantization errors.

  2. Coherent and unified optimization framework. The authors derive transformation selection, weight scale optimization, adaptive rounding, and activation clipping from a common output-sensitivity objective. Although these components are optimized sequentially rather than jointly, the framework provides a systematic approach to coordinating multiple quantization decisions.

Critical weaknesses
  1. Insufficient validation of the Fisher approximation. The proxy is theoretically exact only under a rank-one gradient assumption, while its correlation with downstream KL divergence is evaluated on just one model under W4A4 quantization. Its general applicability remains unclear.

  2. Limited experimental coverage. The experiments cover only four models (1B–8B) and 4-bit integer quantization, without evaluating larger models, lower-bit precision, or mixed-precision schemes.

  3. Missing computational overhead analysis. The method requires additional backward passes and Fisher estimation, but provides no detailed comparison of calibration time, memory consumption, or quantization cost against existing methods.

  4. Reliance on simplifying assumptions. The framework uses sequential optimization and assumes independent quantization errors and negligible covariance terms. The impact of these approximations is not sufficiently investigated.

Questions for authors
  1. Generalization to other quantization schemes: Can the proposed Fisher proxy be applied to other quantization formats, such as FP8, MXFP8, lower-bit quantization, or mixed-precision settings? Would the underlying assumptions and optimization procedures still hold?

  2. Separating weight and activation quantization errors: Can the proposed proxy reliably isolate and quantify the individual contributions of weight and activation quantization to downstream model degradation? How accurate is the decomposition in Eq. 8, particularly when interactions between weight and activation quantization errors are significant?

3 comments
AuthorsOpenReview ↗

Response to Reviewer RSWV

We thank the reviewer for the feedback and recognition of the method as a principled unified objective.

1. Validating the Approximations

In order to strengthen our analysis, we include the correlation between our error proxy and the KL between the model's output for 2 additional LLMs and 3 quantization settings. Table A in our general response reports the Pearson Correlation coefficients for the approximations reported in Equation 5, Equation 6 and the addition of Equation 8 to justify our statement of independence between weight and activation quantization error. Even at W2A8, our proxy keeps a 0.71 correlation with the output KL where the L2 error drops to −0.02, and the Eq. 8 decomposition maintains the same correlation (Table A, General Response). Theoretically, the separation of activation and weight quantization error has been described in previous work [1]. Although Eq. 6 is exact only for rank-one gradients, its correlation stays within 0.03 of Eq. 5 in all six settings.

We plan to include this new data and clarifications into the paper to further support our claims.

2. Additional Quantization Settings

In order to strengthen our experimental section, we include the results for Table 2 in 2 additional quantization settings: W3A3KV3 and W2A4KV4.

QuantMethodLlama 3.2 1B Inst. Wiki ↓Llama 3.2 1B Inst. CSR ↑Qwen3 8B Wiki ↓Qwen3 8B CSR ↑
W3A3KV3RTN1e5292e730
QuaRot+GPTQ3e4302e330
CAT+GPTQ423311847
FlatQuant+GPTQ59401655
Hz-PTQ49411358
W2A4KV4RTN6e5291e630
QuaRot+GPTQ749304e730
CAT+GPTQ1e3314449
FlatQuant+GPTQ2693455930
Hz-PTQ126353147

Table C: WikiText perplexity and CSR accuracy in two additional quantization settings (single seed).

This demonstrates that our method improves over the strong baselines in 7 of 8 entries even at more difficult quantization settings.

We will further include results for larger architectures such as Qwen 3 14B or Qwen 3 32B in Table 2 to further demonstrate the applicability of our method. Due to the limited rebuttal period, these evaluations, as well as standard deviations over multiple seeds, could not be completed in time. We will include them, together with the full tables, in the camera-ready version.

The proxy objective introduced in this work is independent of the type of quantization error such as MXFP8, so, in principle, it can be extended to other data-types and quantization strategies (e.g. Vector Quantization). The transform resulting from Equation 11 relies on the assumption of integer quantization. However, previous work has shown similar optimality for floating-point quantization for non-Hessian-weighted local error [2]. Because the proxy measures each layer's contribution to the output KL, it could also guide mixed-precision bit allocation, which we leave to future work.

3. Calibration Time and Memory

Table B in Section 2 of our General Response reports the end-to-end calibration time (Hessian estimation + quantization) and peak memory of our method. Our method inherits the runtime properties of YAQA [3], which requires collecting Kronecker-factored Hessians followed by the two-sided LDLQ routine: our Hessian estimation is computationally equivalent to YAQA's and has the same peak memory (32 GB). The additional cost comes from computing the transform, the weight scales and the activation clipping factor, which brings the total runtime to 158 min, about 25% more than FlatQuant+GPTQ (127 min).

Concurrent work [4] proposes faster two-sided LDLQ and a memory-optimized procedure for Hessian collection, which could be integrated directly into our approach. The primary contribution of this work is the unified approach for solving multiple components of PTQ under one objective; production-level optimization remains future work.

4. Sequential Optimization

The transform is chosen first because its solution (Eq. 12) does not depend on the quantization error, and weights and activations are optimized separately because neglecting their covariance (Eq. 8) leaves the correlation with the output KL unchanged in all six settings (Table A). Section 3 of our General Response gives more details regarding the full ordering, while Table 4 ablates each stage.

References

[1] M. Federici, et al. "Dissecting Quantization Error: A Concentration-Alignment Perspective". ICML 2026.

[2] J. Chen, V. Egiazarian, R. L. Castro, T. Hoefler, D. Alistarh. "WUSH: Near-Optimal Adaptive Transforms for LLM Quantization". ICML 2026.

[3] A. Tseng, Z. Sun, C. De Sa. "Model-Preserving Adaptive Rounding". ICML 2026.

[4] J. Birnick, R. Saab. "FastKron: Efficient Quantization with Kronecker-Factored Hessians". arXiv:2608.06291, 2026.

Thanks for the response and additional experiments. The new results help address some of my concerns, particularly regarding lower-bit quantization settings and calibration overhead.

However, I believe the main limitations remain. The validation of the proposed Fisher proxy is still relatively limited, and the unchanged correlation after Eq. 8 does not fully establish that the individual contributions of weight and activation quantization errors can be accurately isolated. Additionally, the evaluation on larger models and other quantization formats remains incomplete.

Overall, while I appreciate the authors' efforts and the additional evidence, I would like to maintain my original evaluation.

We thank the reviewer for appreciating our additional data and for engaging with our work.

On Validity of the proposed approximations

We acknowledge the reviewer's observation that correlation alone is insufficient to establish the accuracy of the approximation chain. To directly address this concern, we performed an additional analysis that measures the approximation error with respect to the true output divergence.

Specifically, we quantize each layer independently and compute:

  • the true output KL divergence;
  • the weight-only Fisher estimate;
  • the output-Fisher approximation (Eq. 5);
  • the factorized output-Fisher approximation (Eq. 6);
  • the final approximation used by Hz-PTQ (Eq. 8).

We then report the absolute error of each approximation relative to the true output KL.

Tables E and F show the results across multiple layers and quantization settings. The main finding is that the successive approximations introduced in the paper incur only a small additional error relative to the true output KL, while substantially improving over a weight-only Fisher approximation.

In particular:

  • Eq. 5 already reduces the error dramatically compared to the weight-only Fisher estimate in most settings.
  • The factorization introduced in Eq. 6 generally preserves this accuracy.
  • The final decomposition between weight and activation quantization (Eq. 8) introduces only a minor additional approximation error.

This trend is especially pronounced in challenging low-bit settings (W3A3 and W4A4), where the weight-only Fisher estimate can deviate substantially from the true output KL, while the proposed approximations remain much closer.

Importantly, the compounding approximation errors along the sequence Eq.5$\to$Eq.6$\to$Eq.8 are small compared to the alternative weight-Fisher. The final objective remains comparably accurate to the intermediate formulations, indicating that the practical benefits of the additional simplifications are achieved at the expense of relatively small approximation error.

Overall, these results provide direct empirical support for the central assumptions underlying HzH_z-PTQ and demonstrate that the approximation chain remains accurate under practical quantization regimes.

Quant.ComponentOutput KLError Weight FisherError Eq. 5Error Eq. 6Error Eq.8
W4A4v_proj[7]10.396.160.440.340.61
W3A3v_proj[7]58.8340.7815.2718.099.25
W4A8v_proj[7]4.540.640.600.310.32
W2A8v_proj[7]109.7831.8232.1532.3532.31
W4A4down_proj[4]82.0370.1412.955.045.34
W3A3down_proj[4]198.37159.4430.302.2013.76
W2A8down_proj[4]481.21257.07257.35158.32158.19
W4A8down_proj[4]10.360.290.190.010.00

Table E: Output KL and absolute error |KL-Approximation| of the weight-only Fisher, Output Fisher (Eq.5), Factorized Output Fisher (Eq.6) and independent weight and activation quantization (Eq.8), for two linear layers and four quantization settings.

LayerOutput KLError Weight FisherError Eq. 5Error Eq. 6Error Eq.8
down_proj120.0491.6517.7222.4921.96
gate_proj10.185.601.920.760.85
k_proj22.7019.863.605.256.13
o_proj17.6711.281.810.851.02
q_proj8.095.891.180.290.33
up_proj16.109.731.130.410.62
v_proj13.499.100.980.950.96

Table F: Aggregated average and (absolute) error for various layer types on a LLama 3.2 1B architecture on the W4A4 setting.

Evaluation of larger models

To address this point, we extended Table 2 with additional experiments on Qwen 3 14B. The results are reported in Table G.

HzH_z-PTQ continues to achieve the strongest results among PTQ baselines, obtaining the best CSR score (68.96) while maintaining competitive lowest perplexity. These results suggest that the benefits of the proposed Hessian-guided formulation are not limited to small models and transfer to larger LLMs.

MethodWikiCSR
FP8.6570.75
RTN7e529.41
QuaRot+GPTQ16.3242.66
CAT+GPTQ9.4068.31
FlatQuant+GPTQ9.6466.62
HZ\mathbf{H_Z}-PTQ9.3368.96

Table G: Results for Qwen 3 14B architecture on the W4A4KV4 setting reported in Table 2

We thank the reviewer for the constructive feedback. The additional analyses above were performed specifically to address the concerns regarding (i) the validity of the approximation chain and (ii) the evaluation on larger-scale models. We believe these new results significantly strengthen the empirical support for the proposed method and provide direct evidence that the key assumptions underlying HzH_z-PTQ hold in practice.

Reviewer_sShgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript under review is on Hessian-based quantization. The authors first present PTQ as the minimization of a loss-function which measures divergence induced by replacing full-precision weights and activations with their quantized versions. Using Taylor's expansion, they approximate the divergence measure by a quadratic form in the output change. The quadratic form involves the Hessian matrix, hence the reason for the name. The Hessian matrix is, however, intractably large, at least for real models. To this end, the authors suggest an intractable proxy. Optimizing the proxy, they can find the necessary parameters for joint quantization of weights and activations. The manuscript also includes experiments on Llama with different sizes and Qwen where the authors compare their proposed method with some other methods.

Critical strengths
  1. The idea of optimizing transformation, weight scales and rounding, and activation clipping simultaneously is interesting and to me seems the right approach.

  2. The mathematical aspects of the work is sound.

Critical weaknesses
  1. The computational aspects of your experiment presented in Table 1 is not clear. It seems to me that you solve Equation (7) for different weight matrices for 4-bit quantization of both weights and activations. Then having solved the optimization problem, you can find \tilde{Z} and hence its distribution using which you can find the KL divergence between the distribution of Z's and that of \tilde{Z}. If this is the case, then it is clear that you should get good correlation between KL-divergence and output Fisher-weighted and your proposed method, and the first two should clearly be off, actually quite a bit.

  2. While the proposed method seems to work, I think it is more or less on par with some other existing methods, at least based on the figures I see in your tables. Or even if there is improvement, it is often miniscule. So, I think some of your statements should be toned down a bit.

  3. I am not clear if your activation quantization is static or dynamic and which one you actually used for your experiments. On page 2, line 8 from below, I see signs that indicate your activation quantization is dynamic, but then I also see in Section 3.6, on page 6, that for finding \gamma_{x} you need a calibration set, and as such I think it is static. It seems that you are using a hybrid approach. This should be stated clearly and pros and cons of the method needs to be discussed.

  4. I think an E ( for Expectation ) is missing in equation 4.

Questions for authors

Please address the points raised in "Critical Weaknesses" section.

2 comments
AuthorsOpenReview ↗

Response for sShg

We thank the reviewer for appreciating our framework and for their comments and questions.

1. Clarification for Figure 1

The experiment in Figure 1 does not involve any optimization but simple Round-to-Nearest quantization with MinMax scales under a W4A4 regime. Hence, the correlation between the output KL and the local output-Kronecker Fisher-weighted error is not a given. We will include additional experimental description to the main text to clarify this aspect. We further point the reviewer to Table A in our general response for additional results validating the approximation reported in Figure 1 for different models and quantization settings.

2. Experimental Results

We will carefully inspect the wording in our text to make sure we are not overclaiming. However, we want to underline that in W4A4KV4, the performance of 8B models is quite close to full precision, hence, the margin for improvement is small. Despite that, we observe improvements larger than a standard deviation in perplexity in all but one case across Table 2 and Table 3.

We further include results for the more challenging W3A3KV3 and W2A4KV4 settings in our response to reviewer RSWV, where the gaps are larger.

3. Clarifications on Activation Quantization

We understand the confusion arising from the compressed wording used in Section 3.6. To clarify: we use dynamic activation quantization, although the clipping coefficient γ\gamma is determined globally. This is because using a different value of γ\gamma for each token and sequence would result in noticeable runtime overhead. The use of a unique static γ\gamma for each activation quantization operation, as in the paper, results in essentially no overhead when compared to traditional dynamic quantization since the scale sjs_j for the j-th token is determined with γ⋅range(Xj)\gamma \cdot \mathrm{range}(X_j) instead of just range(Xj)\mathrm{range}(X_j). This is indeed a hybrid scheme: per-token scales are dynamic, while γ\gamma is static and calibrated. Its limitation is that one γ\gamma per quantizer cannot adapt to individual tokens and relies on a representative calibration set. We will modify Section 3.6 to clarify this aspect.

4. Missing Expectation

We thank the reviewer for noticing the typo. Indeed, Equation 4 holds in expectation over inputs, we will correct it in the main text.

Thank you for your responses to my comments. I will consider your responses when discussing with other reviewers and the AC.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all reviewers for their constructive feedback. Reviewers find our objective "well-motivated and principled" (RSWV) and "theoretically meaningful" (QtFL), consider optimizing the transformation, scales, rounding and clipping together "the right approach" (sShg), and note that the strong correlation between our proxy and the output KL "justifies the method" (c3mG). QtFL notes "consistent competitive performance on four model families and diverse configurations".

The shared concerns are (1) how well our approximations track the downstream loss beyond Figure 1, (2) calibration cost and (3) the sequential optimization. We address these below with new results, and respond to all other points individually.

1. Validation of the approximations (RSWV, QtFL, c3mG, sShg)

Figure 1 uses round-to-nearest quantization with min-max scales and does not optimize any component of our method. Thus, the observed correlation does not arise from optimizing the same proxy being evaluated. We extend this analysis along two dimensions: across three models at W4A4 and across four quantization settings on Llama 3.2 1B Instruct. We also evaluate the decomposition of Eq. 8. Table A reports the Pearson correlation between each proxy and the output KL divergence.

ModelQuant.L2 errorFisher (Eq. 5)Kronecker (Eq. 6)Weight + act. split (Eq. 8)
Llama 3.2 1B Inst.W4A4−0.310.980.990.99
Qwen3 0.6BW4A40.120.970.970.97
Llama 3 8BW4A4−0.340.970.980.98
Llama 3.2 1B Inst.W3A3−0.250.940.910.91
Llama 3.2 1B Inst.W4A8−0.380.980.980.98
Llama 3.2 1B Inst.W2A8−0.020.700.710.71

Table A: Pearson correlation with output KL divergence. L2 error is E[∥ΔZ∥22]\mathbb{E}[\Vert\Delta Z\Vert_2^2]; Eq. 5 is the output-Fisher proxy E[∥ΔZ∥FZ2]\mathbb{E}[\Vert\Delta Z\Vert^2_{F_Z}] and Eq. 6 its Kronecker-factorized approximation E[∥ΔZ∥F⊗Z2]\mathbb{E}[\Vert\Delta Z\Vert^2_{F_{\otimes Z}}]. Eq. 8 splits Eq. 6 into a weight-error term E[∥XΔW∥F⊗Z2]\mathbb{E}[\Vert X\Delta W\Vert^2_{F_{\otimes Z}}] plus an activation-error term E[∥ΔXW∥F⊗Z2]\mathbb{E}[\Vert\Delta X W\Vert^2_{F_{\otimes Z}}], dropping their cross term.

The factorization (Eq. 6) stays within 0.03 of Eq. 5, and the independence assumption (Eq. 8) leaves the correlation unchanged in every setting. At W2A8, the correlation decreases to 0.71, which is consistent with the second-order approximation becoming less accurate as the quantization perturbation increases. Nevertheless, the L2 error is uncorrelated with the output KL in this setting (−0.02).

2. Calibration cost (RSWV, QtFL)

Table B reports peak memory and total runtime, including Hessian estimation, on Llama 3.2 1B Instruct with per-channel quantization.

MethodPeak Memory [GB]Runtime [min]
CAT+GPTQ2574
FlatQuant+GPTQ25127
YAQA3296
Hz-PTQ32158

Table B: Runtime and peak memory measured on an A100 GPU.

Our Hessian estimation is computationally equivalent to YAQA's (about 1 h on 1B models), and one estimate serves all components and the additional overhead arises from the computation of transforms, hessian-based weight scale and activation clipping selection. The full pipeline quantizes 8B models in under 8 h.

Although the current implementation is not optimized, its main computational components can directly benefit from faster two-sided LDLQ and Hessian collection [1], as well as custom CUDA kernels, improved offloading, and parallelization. Our contribution is to demonstrate that the proposed output-sensitivity objective provides an effective unifying principle for post-training quantization. The reported run times establish the practical feasibility of the approach, while production-level optimization remains future work.

3. Sequential optimization (RSWV, c3mG, QtFL)

Joint optimization is intractable at LLM scale; rounding alone is an integer quadratic program. We minimize the same output-Fisher objective in stages ordered by dependency: the transformation first, as it changes the weights, activations and gradients; then scales and rounding together, as each group's scale is set after previous LDLQ compensation; and clipping last, given the rounded weights. Every stage targets the same downstream objective rather than a separate reconstruction loss.

Table 4 ablates the contribution of each component, while Table A shows that the Eq. 8 decomposition preserves the proxy’s correlation with output KL when weight and activation error contributions are treated separately. We will revise Section 3.3 to clarify both the dependency structure and the rationale for this sequential procedure.

We thank the reviewers again for the quality of their reviews.

References

[1] J. Birnick, R. Saab. "FastKron: Efficient Quantization with Kronecker-Factored Hessians". arXiv:2608.06291, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.