acceptodds
Under review as a conference paper at ICLR 2027

WaterKron and FlipFlop Hessian: Information-Theoretically Grounded Quantization with Kronecker-Factored Hessians

Abstract

How should a Kronecker-factored Hessian approximation be chosen for post-training quantization? We address this question through WaterKron, which combines two-sided GPTQ with row- and column-dependent waterfilling scales and entropy coding. We derive its high-rate distortion with respect to the full Hessian using an explicit Kronecker-Hessian mismatch factor . This factor quantifies the asymptotic distortion penalty due to the Kronecker Hessian approximation and provides a criterion for selecting the factors optimally. Minimizing leads to a Gaussian covariance-fitting problem with classical flip-flop updates. We thus give a rate-distortion justification for using the resulting FlipFlop Hessian in quantization. We evaluate it empirically, finding that the FlipFlop Hessian consistently improves KL divergence and perplexity over input-only, marginal, and Frobenius-based Hessian choices.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
quantization, gptq, watersic, kronecker factorization, rate-distortion

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_mdM2OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

WaterKron extends waterfilling scale selection to two-sided GPTQ with a Kronecker-factored Hessian. Its main result derives an asymptotic distortion penalty for approximating a full Hessian by Kronecker factors. Minimizing this penalty leads to the classical flip-flop covariance updates, providing a quantization-based reason to prefer them over Frobenius fitting. The analysis assumes iid Gaussian weights, positive-definite factors, high-rate quantization, and efficient entropy coding. Controlled experiments on 6 Llama and Qwen models report improved KL and perplexity at a target rate of 2 bits per quantized weight. Improvements vary substantially by model, and the paper does not establish realized encoded storage or inference performance.

Critical strengths
  1. The full-Hessian mismatch result gives an interpretable criterion for selecting Kronecker factors. Its arithmetic-to-geometric-mean expression measures distortion from the generalized eigenvalue spread, and its Gaussian-KL interpretation explains the connection to covariance fitting (Thm. 4.1 and Secs. 4-5).
  2. The paper distinguishes existing algorithms from its contribution. It credits prior two-sided GPTQ, waterfilling, and flip-flop updates, then derives why the latter optimize the proposed quantization criterion. The side-information convention and order of limits are stated in Appendix C.
  3. The experiments isolate factor selection through input-only, independent-marginal, Frobenius, and FlipFlop alternatives. At the stated 2-bit target, a tested FlipFlop variant gives the best WikiText-2 perplexity in all 6 model tables. For example, Llama-3.2-1B improves from 49.3 with WaterSIC to 21.0, while Qwen3-8B improves more modestly from 13.0 to 12.7 (Tbls. 2-7).
  4. The iteration comparisons are informative. Additional Frobenius updates often worsen the language-model metrics, whereas FlipFlop generally improves or levels off after 2 updates. The paper also acknowledges that its theory concerns high rates and that efficient entropy decoding remains a practical limitation.
Critical weaknesses
  1. The operational rate is insufficiently specified. Sec. 3 explains why the earlier per-column Huffman scheme is no longer adequate, but Appendix B does not identify the coder or entropy estimator, how scales are selected to reach the target rate, or whether any reported rate includes scale and coding overhead. Consequently, the plots support comparisons at a stated target, but their storage interpretation and rate matching cannot be checked. A precise rate calculation and scale-selection procedure are needed.
  2. The experiments do not directly test the proposed mismatch criterion. They report model KL and perplexity at low rates, while Thm. 4.1 concerns high-rate quadratic distortion with Gaussian weights. No measured mismatch factors or predicted-versus-observed distortion results connect these regimes. The acknowledged asymptotic scope is reasonable, but a small finite-rate diagnostic would help establish how much of the observed benefit the theory explains.
  3. The implemented updates differ from the ideal derivation. Sec. 5 motivates curvature using model-sampled labels, whereas Appendix B uses observed labels for an empirical Fisher and repeatedly damps both factors by 0.1 times their mean diagonal. The outer-product formulation still applies to that surrogate, but its interpretation as model KL curvature changes, and the stated ideal minimization/convergence result does not directly characterize the damped iteration. The current results do not separate the factor-fitting objective from regularization effects.
  4. No substantive reproduction code is supplied or verified. The paper promises a release with publication, but the submission is not really reproducible from the stated implementation and rate-setting details. It also lacks repeated-calibration uncertainty and factor-fitting time or peak-memory measurements, which limits interpretation of the smaller gains and the practical cost.
Questions for authors
  1. How is each experimental target rate computed and reached? Please specify the scale search, entropy estimator or actual codec, and whether rates include row/column scales and coding overhead. If a bitstream is not produced, please distinguish the estimated rate from measured storage.
  2. Does the damped FlipFlop implementation reduce the proposed objective on the empirical curvature surrogate? An objective trace and a small damping sensitivity check on an existing model would help distinguish the choice of fitting objective from the effect of regularization.
  3. Can a small matrix or selected-layer experiment compare the predicted full-Hessian quadratic distortion with measured distortion across rates, including the low-rate regime used in the model experiments? This would test the practical reach of the mismatch factor without requiring a new model benchmark.
1 comment

We thank the reviewer for their feedback. We address all weaknesses and questions below.

W1: The coder or entropy estimator is insufficiently specified.

You are right, we should have clarified this better. We use the empirical entropy of ZZ to measure the rate in practice. While this does not perfectly match the theory of how ZZ should optimally be encoded, it is a useful proxy, please see the comments to reviewer SPsS where we explained this in detail and also commented on using the Zstd encoder.

We have written a new appendix C that discusses all encoding issues in detail.

W2: The experiments do not directly test the proposed mismatch criterion.

We have now added three module-level metrics to the experiment tables:

  • The measured distortion DHD_H.
  • The predicted distortion DHpredD_H^{\mathrm{pred}} according to our theorem, involving Φ\Phi as a factor.
    Comparing it with DHD_H shows that DHpredD_H^{\mathrm{pred}} accurately predicts (up to about 1% error) the measured distortion DHD_H for two-sided Hessians. Only for the "Input" Hessian the prediction can be off by more (usually about 10% but sometimes more).
  • A measure we call "rate gain", which is essentially −12log⁡Φ-\frac{1}{2} \log \Phi (plus some constant). This measures directly how many bits the respective Kronecker approximation saves you according to the theory, and it can be seen that this follows the same ranking as the measured distortion.
    (The reason we cannot directly measure the absolute −12log⁡Φ-\frac{1}{2} \log \Phi is that we don't have enough calibration data to compute ∣H∣|H|, but the rate gain is a relative measure which we can compute.)

W3: The implemented updates differ from the ideal derivation. (Empirical Fisher, damped matrices)

Regarding empirical Fisher vs real Fisher (with sampled labels), first note that on a perfectly fitted model the two actually agree. We have tested using (a sampled approximation of) the real Fisher in the experiments, but there is no significant difference: The real Fisher can bring a gain of about 0.01 to 0.05 bits; otherwise the results, in particular the ranking of Hessians, stay the same.

As for the damping factors, we did not optimize them, but just chose them large enough so that there are no numerical issues when taking the Cholesky decomposition in the rounding algorithm or inverses in the FlipFlop algorithm. We did observe that using very small damping factors (e.g. 0.01 instead of 0.1) does produce worse results, and we are not entirely sure why. We have added this as an additional limitation to the paper. However, empirically, even with small damping factors, the ranking of the Hessian approximations on the calibration data, in particular the advantage of FlipFlop over the other Hessian approximations, is always maintained.

W4: Reproduction code is not supplied.

We have now attached the experiment code.

Q1: How is each experimental target rate computed and reached?

It is predicted with a non-trivial formula that involves the predicted rate from our theory, where we measure γ\gamma, ∣A∣|A|, ∣B∣|B|, the "empirical" variance of WW, and the empirical entropy of ZZ, as well as a correction term to account for the spread introduced by the (two-sided) GPTQ rounding algorithm. Please see the experiment code for the exact formula. We confirm that this accurately predicts the empirical entropy of ZZ after running the two-sided GPTQ algorithm. The prediction is worse for larger models, but by at most about 0.03 bits, and by about the same amount for all Hessian approximations, which is why we just report the target rate instead of the measured empirical entropy (which would just be another proxy anyway).

Also please see the new appendix C for a detailed discussion of the encoding issues. (more focused on theory instead of experiments)

Q2: Does the damped FlipFlop implementation reduce the proposed objective on the empirical curvature surrogate?

Yes, please see the answer to W2; we have added module-level metrics to the experiments that confirm the reduction of Φ\Phi and DHD_H as predicted by the theory.

Q3: Compare predicted full-Hessian distortion with measured distortion.

Again, please see the answer to W2. The newly reported module-level metrics in particular contain the measured DHD_H and the predicted DHpredD_H^{\mathrm{pred}}, which match very well for the two-sided Hessian approximations.

We hope that we have addressed all weaknesses and questions to the reviewer's satisfaction, so that the reviewer can hopefully recommend clear acceptance of the paper.

Reviewer_SPsSOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper deals with weight-only quantization of LLMs. It extends the WaterSIC quantization algorithm, which aims to minimize the "local" MSE distortion for a single linear projection WX with a more global distortion measure that is based on the hessian of the loss matrix. In the single linear projection setup, the losses that correspond to different rows of W are decoupled, and each row can therefore be quantized independently of the others. In the "global hessian case" the rows of W are coupled in the loss function, and therefore all entries of W should be quantized together. If W as dimensions mxn, the hessian has dimensions (mn)x(mn). For the special case where the hessian if of the form of a Kronecker product of a n x n and m x m matrices, the loss function takes a two-sided weighted form. A natural extension of GPTQ to this form can be used to sequentially compensate for quantization errors in proceeding steps. This was known before this work, e.g. in YAQA.

The innovation in this works consists of: 1)Deriving the optimal left-sided and right-sided scaling matrices for two-sided GPTQ to the integers, followed by entropy coding. This is a rather straightforward extension of WaterSIC. 2)Proving that whenever the hessian is exactly a Kronecker product this scheme achieves the information-theoretic limit to within 0.255 bits at high-resolution (limit of small distortion). This is also a rather straightforward extension of the theory developed in WaterSIC. 3)Deriving an expression \Phi(H,A\otimes B) for the loss of optimality in treating the hessian H as a Kronecker product of matrices A,B (under WaterKron with entropy coding) 4)Deriving an algorithm for optimizing A,B with the objective of minimizing \Phi(H,A\otimes B).

Critical strengths

The paper is well-written and easy to follow. I appreciated the authors effort to provide intuition and understanding. However, in several places I felt the strive for intuition resulted is a bit of oversimplification. Contributions 1+2 form a pretty obvious step in light of WaterSIC. Nevertheless, I think this work is an important contribution to the literature, as it provides fundamental information-theoretic guidelines to how "two-sided GPTQ" should be performed, and shows that with appropriate scaling and entropy coding it is almost optimal under the Kronecker setup. I think the fact that two-sided GPTQ algorithms were already studied in different works under a less fundamental lens, and are sometimes used in practice, makes this work even more important.

Contribution 3 is also significant, as it allows to identify what we lose for insisting on the two-sided GPTQ algorithm instead of using more general quantizer (eg, GPTQ on vector(W) using the full-hessian H, with optimized rank-1 scaling), and contribution 4 is the bridge between the theory and practical algorithms.

Given all the above, I think that this paper is an excellent contribution to ICLR, which will inspire further developments in LLM quantization.

Critical weaknesses

-As pointed above, many of the contributions are somewhat natural/incremental extensions of previous results. -The connection to prior work can and should be improved. The main difference between WaterKron and YAQA seems to be the criterion for optimizing the Kronecker factors. Under entropy coding and optimal scaling, it is clear that the criterion of this paper is more appropriate. However, YAQA did not use entropy coding, and it is therefore less clear that their criterion is less appropriate. This should be stated explicitly. Also, the relation to YAQA should be made more explicit in the intro and not just in the end. What about Chen 26a? How does it relate to this work? Are those the only two references that did two-sided GPTQ? -In the experiments, it is not really clear how the hessian was estimated. Please provide more details. -In general, the code that was used for all experiments will be useful.

Comments about presentation: -Is Algorithm 1 the traditional GPTQ algorithm, or is it the SIC version? Conceptually the two are the same, and this is now widely understood, but calling it GPTQ may be confusing to readers familiar with the "classic" GPTQ form. Same for algorithm 2. -In Theorem 2.2, I think the assumption needs to be that we can encode Z with expected length \sum_{i,j} H(Z_{i,j})+o(mn). This is more reasonable than assuming we can achieve H(Z), as entries of Z should not be independent due to fed-back quantization noise. -I am not sure that the "high-rate waterfilling rule" is a good (nor common) name for \alpha_j\propto 1/L_{i,j}. -The content of the intuition paragraph in the end of page 5 and beginning of page 6 is covered in depth in Ordentlich&Polyanskiy 2026. Refer the interested reader there for more details that are missing in the current explanation (eg, why R is dictated by the volume of E). -It should be emphasized that the lower bound in Theorem 2.1 is non-asymptotic and holds for all dimensions. This is why we can extend it to Corollary 3.2. -I would wrap the first sentence in page 6 ("Now not that calling Algorithm 1...") in a proposition with a proof in the appendix. This is the key for extending WaterSIC to WaterKron. -The remark in the end of section 3 is very important and should be more emphasized. 1)Lossless compression of a source with unkown statistics is generally called "universal compression". The rule of thumb is that if the numbers of parameters in the distribution are sufficiently small wrt to the length of the compressed sequence, we can approach the source entropy using universal compressors. For example, for an iid source over a finite alphabet, we can always compress a long enough sequence with rate approaching its entropy. In fact, of the shelf compressors like zsdt will achieve this. In WaterSIC all entries in columns of Z have the same distribution, so compressing to the source entropy is indeed reasonable for large n. In WaterKron, each Z_{ij} may have a different distribution. Thus, it is not at all clear that the assumption of Corollary 3.2 makes sense. Also, the last sentence in the remark looks wrong to me (but maybe I misunderstood something). 2)I did not find an account for what entropy coding algorithm was used for the experiments. Just calculating rates according to \sum_{ij}H(Z_{ij}) is not a valid proxy for rate here! I would guess that applying a universal compressor (eg zstd) on the entire matrix Z would work quite well (it did for WaterSIC), but this must be checked empirically. -I would expect a more rigorous proof of Theorem 4.1. The current proof is too hand-waved. -It would be helpful to state that under the full-hessian the fundamental limit is D^_H(R)=|H|^{1/n} 2^{-2R} and under WaterKron the high-rate limit is [(2\pi e)/12]\Phi(H,A\otimes B) D^*_H(R). Thus, \Phi(H,A\otimes B) is the gap-to-optimality up to the unavoidable 1D loss of 0.255 bits. -The D_KL interpretation of log \Phi is very important and must be established in the paper (appendix is fine of course). -I got completely lost in section 5. What is the role of g and x, and why is modeling H as in the displayed equation without loss of generality?

Questions for authors

Q1)How is WaterKron related to YAQA and GPTQ-2D? Q2)How was the hessian computed in the experiments? Q3)What entropy coding algorithm was used for the experiments? Q4)What is the role of g and x, and why is modeling H as in the displayed equation without loss of generality?

3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for their detailed feedback which has been very helpful to improve the paper. We address all weaknesses and questions below.

The scaling factor choice is a natural/incremental extension of a previous result.

That is right. We try to emphasize this by phrasing the results in section 3 as corollaries (not theorems), and by saying that we extend the results of WaterSIC and unify it with the two-sided algorithm. It is just a necessary prerequisite for the following sections.

The core contribution is the theory about Kronecker-Hessian mismatch involving Φ\Phi and the resulting FlipFlop approximation algorithm, which is new.

The connection to prior work should be improved.

We have added appendix D which discusses this.

How exactly was the Hessian estimated in the experiments?

We adapted the "recursive halving flow" from the FastKron paper, this is also mentioned in appendix D now.

The code of the experiments would be useful.

This has been provided now.

Are the algorithms the GPTQ or SIC formulation?

It is the GPTQ formulation with the following differences to the original GPTQ paper: First, we omit block-wise processing (which is just an "implementation" technique), and second, we normalize L~\tilde{L} in the beginning instead of dividing by L~i,i\tilde{L}_{i,i} in every step. This seems to result in essentially the SIC formulation (but we are not very familiar with the SIC formulation).

Not sure that waterfilling is a good/common name for the 1/Li,i1/L_{i,i} scaling factors.

We actually agree: The reason "waterfilling" has its name is because of "clipping" certain eigenvalues which has the nice visual image of a "waterline" (the boundary where eigenvalues are clipped). Since we work in the high rate limit, this clipping doesn't occur, so there is no "waterfilling". However, we are not sure what a good alternative name would be, and we wanted to make the connection to WaterSIC clear.

WaterSIC explanation available in detail in Ordentlich & Polyanskiy.

We now refer the reader to that reference at the end of the intuition paragraph you mentioned.

It should be emphasized that the lower bounds are non-asymptotic.

We have added "for all m,n,Rm,n,R" to the theorem statements to make this clearer.

"Encoder assumptions should be based on H(Zi,j)H(Z_{i,j}) not H(Z)H(Z)", the remark in section 3, and various comments about encoding.

It is true that the Zi,jZ_{i,j} have different distributions. We have changed the theorems so that they work directly with H(Z)H(Z) in the rate definition, not using any encoder, and added a whole encoder discussion to appendix C. This appendix hopefully clarifies all questions regarding the encoding.

It is not clear that the assumptions make sense here; in WaterSIC at least all entries in a column have the same distribution, with WaterKron that's not the case anymore.

If we understand correctly, your worry is that we only have a "single sample" of the whole matrix ZZ, so it's unclear how this can be entropy-coded or why an expectation makes sense. We understand this worry, however, the setting seems necessary/clean for the theory, and in practice it turns out to be not nearly as harmful as one might worry. Let us share three perspectives on that.

First, in theory, if the decoder has access to A,B,γ,σWA,B,\gamma,\sigma_W, then it can determine the distribution of each Zi,jZ_{i,j} and do arithmetic encoding on that, see the new appendix C. Or it can build a simplified model of the distribution of Zi,jZ_{i,j}. Or in the high rate regime, the distributions of Zi,jZ_{i,j} should actually become essentially equal. In any case, there are theoretical solutions for dealing with the different distributions of the entries.

Second, since we want to apply information theory, there must be a source of randomness. If one does not want take randomness over WW, then, for example, one could also consider randomness in the quantizer, e.g. randomized rounding within GPTQ (see e.g. Babai-Klein algorithm), and if we're not mistaken then one can use this route to similarly arrive at a point where ε\varepsilon is uniformly distributed within the rectangle E\mathcal{E}, which is all that's needed to derive Φ\Phi and the FlipFlop algorithm (assuming the rate doesn't depend directly on A,BA,B). Or perhaps there are other reasons/motivations to consider E[εεT]∝I\mathbb{E}[\varepsilon \varepsilon^T] \propto I; the point is that Φ\Phi and the FlipFlop Hessian approximation may be useful in other settings too, we just used the entropy coding setting because it is theoretically clean and allows to compare with lower bounds.

Third, we have added new columns in the experiments that list the measured DHD_H along with the prediction DHpredD_H^{\mathrm{pred}} (for the expected DHD_H according to the theory). They match very well for the two-sided Hessian approximations, justifying the assumptions of the theory.

AuthorsOpenReview ↗

Rebuttal (2/2)

How was entropy measured in the experiments?

We simply used the empirical entropy of the entries of ZZ seen as one empirical distribution over integers. We are aware that this does not directly correspond to the theory, in particular this assumes that all entries Zi,jZ_{i,j} have the same distribution, which is not the case.

However, we do think it is a useful proxy for practical encoding schemes, since an encoder used in practical quantization would probably assume the same distribution for each Zi,jZ_{i,j} anyway.

We have also measured compression with Zstd as you suggested, and at level 19 we found it to be about 0.1 bits off from the empirical entropy measure at 2 bits. The difference increases at low rate and becomes smaller at high rate, but it is consistent across the Hessian approximations, so that if we were to compress with Zstd, the ranking of Hessian approximations would stay the same. At Zstd level 3 (default), however, the gap is much larger.

More rigorous proof of Theorem 4.2 (previously Theorem 4.1).

We will add one in the camera ready version. But please note that it is essentially the same proof as in the WaterSIC paper, with the additional DHD_H computation that we have provided.

It would be helpful to restate the distortion DHD_H in terms of Φ\Phi and also the lower bound on DHD_H.

Agreed, we did that now.

The Gaussian KL interpretation of Φ\Phi should be proved in the paper.

Added in appendix E.

Section 5 is unclear, what is the role of gg and xx, and why is modeling HH like this without loss of generality?

There are two unrelated steps:

  1. Deriving the FlipFlop algorithm by fixing BB and minimizing Φ(H,A⊗B)\Phi(H, A \otimes B) over AA (and then the other way around).
  2. Plugging in H=Ex,g[xxT⊗ggT]H = \mathbb{E}_{x,g} [xx^T \otimes gg^T], which is the form HH has in the case where HH is the Fisher information matrix.
    (If vec(G)\mathrm{vec}(G) is the appropriate gradient of vec(W)\mathrm{vec}(W) then the Fisher is H=E[vec(G)vec(G)T]H = \mathbb{E}[\mathrm{vec}(G) \mathrm{vec}(G)^T], and since G=gxTG = gx^T where xx is the module input and gg is the module output gradient we arrive at the specified form.)

Due to lack of space, and to focus on the most important case without introducing more notation, we had unified these steps and made the derivation of step 1 directly with the Hessian form of step 2. Now we have reorganized that section and we split the two steps apart, hopefully that clarifies things.

Q1: How is WaterKron related to YAQA and GPTQ-2D?

This is discussed in the new appendix D. In short, the core algorithm in YAQA is exactly the two-sided GPTQ algorithm listed in our paper. FastKron and GPTQ-2D are just fast implementations of this algorithm. The FastKron paper contains a thorough treatment of the algorithm including various Hessian approximations that have been used with it (e.g. also BoA which uses a local Hessian instead of the global Fisher information matrix).

Questions Q2, Q3, Q4 have already been addressed above.

We thank the reviewer again for their helpful input and for recommending the paper for acceptance.

I thank the authors for their detailed response, which fully addressed the few minor concerns I had with the paper.

Reviewer_xHmGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WaterKron, an information-theoretically motivated extension of waterfilling-based quantization to two-sided GPTQ with Kronecker-factored Hessians. The authors derive a Hessian mismatch factor that characterizes the asymptotic distortion penalty caused by Kronecker approximation. By minimizing this factor, they introduce the FlipFlop Hessian approximation based on alternating covariance estimation. Experiments on Llama and Qwen models demonstrate improvements in KL divergence and perplexity over input-only, marginal, and Frobenius-based Hessian approximations.

Critical strengths
  1. The paper provides a well-motivated theoretical framework connecting Kronecker-factored Hessian approximation with quantization distortion.

  2. The proposed FlipFlop Hessian offers a principled alternative to Frobenius-based approximation, with a clear information-theoretic justification.

  3. Comprehensive experiments.

Critical weaknesses
  1. The proposed Hessian mismatch factor (Φ\Phi) is not directly compared with empirical quantization distortion. Although the experiments demonstrate improvements in KL divergence and perplexity, it would be helpful to examine whether Φ\Phi accurately predicts quantization distortion in the low-bit regime.

  2. The experiments evaluate FlipFlop only within the WaterKron framework. Since the main contribution is an improved Kronecker-factored Hessian approximation, it would be valuable to investigate whether replacing the existing Hessian approximation in YAQA [1] with FlipFlop also improves quantization performance.

  3. FlipFlop requires gradient-based statistics and iterative Hessian estimation, but the paper does not report calibration time or memory overhead.

[1] Tseng, Albert, Zhaofeng Sun, and Christopher De Sa. "Model-preserving adaptive rounding." arXiv preprint arXiv:2505.22988 (2025).

Questions for authors
  1. Could the authors provide downstream task evaluations (e.g., MMLU or GSM8K) to demonstrate whether the improvements in KL divergence and perplexity translate into better task accuracy?

  2. I believe the Frobenius-based Hessian approximation is also used in YAQA, and I found that KronQ [1] uses a marginal Kronecker-factored Hessian. It would be helpful to clarify these connections in the related work discussion.

[1] Lee, Donghyun, et al. "KronQ: LLM Quantization via Kronecker-Factored Hessian." arXiv preprint arXiv:2607.07964 (2026).

1 comment

We thank the reviewer for the feedback. We address all weaknesses and questions below.

W1: Φ\Phi is not directly compared with empirical quantization distortion.

We have now added three module-level metrics to the experiment tables:

  • The measured distortion DHD_H.
  • The predicted distortion DHpredD_H^{\mathrm{pred}} according to our theorem, involving Φ\Phi as a factor.
    Comparing it with DHD_H shows that DHpredD_H^{\mathrm{pred}} accurately predicts (up to about 1% error) the measured distortion DHD_H for two-sided Hessians. Only for the "Input" Hessian the prediction can be off by more (usually about 10% but sometimes more).
  • A measure we call "rate gain", which is essentially −12log⁡Φ-\frac{1}{2} \log \Phi (plus some constant). This measures directly how many bits the respective Kronecker approximation saves you according to the theory, and it can be seen that this follows the same ranking as the measured distortion.
    (The reason we cannot directly measure the absolute −12log⁡Φ-\frac{1}{2} \log \Phi is because we don't have enough calibration data to compute ∣H∣|H|, but the rate gain is a relative measure which we can compute.)

W2: FlipFlop is only evaluated in WaterKron framework, would be valuable to test it in YAQA.

Please note that Algorithm 2, i.e. the two-sided GPTQ algorithm, is exactly equivalent to YAQA. In particular, when we report the "Frobenius" Hessian experiments, that is essentially YAQA, the only difference being that YAQA clamps the integers and therefore must also use different scaling factors (but this makes experiments more noisy, unbounded integers are cleaner in theory and experiments). See e.g. the "FastKron" paper by Birnick and Saab which discusses the algorithm in more detail and explains that it's equivalent to YAQA. We therefore essentially already compare to YAQA (in fact YAQA is the biggest reason why we measure the "Frobenius" Hessian in the first place).

W3: FlipFlop requires gradient-based statistics and iterative Hessian estimation, but the paper does not report time/memory overhead.

Since we just discussed YAQA, please first note that the "Frobenius" Hessian (i.e. YAQA) has the same cost, so FlipFlop has no additional cost compared to YAQA.

Now in comparison to the "Input" Hessian, the biggest difference is that we need gradients. There are multiple "quantization flows" that allow you to get these, and this has been discussed in the "FastKron" paper: In summary, if you have enough fast memory, there is only an about 2x compute overhead (for doing an additional backward instead of just forward pass of the calibration data, and for accumulating BB on top of AA). However, usually one would not have so much memory, and one can use their "recursive halving flow", which we have also adapted to our experiments. This practically multiplies the compute requirement by about 1.5. Additionally, one must account for the the number of iterations for FlipFlop/Frobenius approximations. This multiplies the compute requirement by about the number of iterations, although slightly less so for the recursive halving flow because only the leaves of the recursion carry the additional multiplicative factor. For example, for 2 iterations of FlipFlop/Frobenius with the recursive halving flow, that would be about a 5x factor compared to the the Input Hessian approximation.

Q1: Could the authors provide downstream task evaluations, e.g. MMLU?

Yes, we have added MMLU as well as common zero-shot tasks to the experiment tables.

Q2: It would be helpful to clarify the connection to YAQA and other quantization algorithms.

We clarify this in the newly added appendix D.

We thank the reviewer again for the feedback which has been helpful to improve the paper. We hope that we have addressed all weaknesses and questions to their satisfaction, so that the reviewer can hopefully recommend clear acceptance of the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Paper Changes

We thank all reviewers again for their valuable feedback, which has helped to improve the paper. Below we summarize the changes we made to the paper.

  • We have added addition module-level metrics to the experiment tables. (DHD_H, DHpredD_H^{\mathrm{pred}}, rate gain) This allows to verify that the practical gains are indeed explained by our theory.
  • We have added additional downstream task evaluations.
  • We have added another figure to the experiments, breaking down which modules benefit most from which Kronecker approximation.
  • The theorems 2.2, 3.2, 4.2 now use a rate definition directly in terms of the entropy H(Z)H(Z). The whole coding scheme discussion has been moved to appendix C and expanded, clarifying in detail the questions/nuances that were raised regarding the encoding.
  • Section 4 contains a new remark that discusses the joint optimization of scaling factors α,β\alpha, \beta and Kronecker factors A,BA,B
  • Section 5 is now split into two steps: First a general derivation of FlipFlop, followed by the special case of the Fisher matrix.
  • Additional appendices that discuss the relation to other papers / quantization methods, and a proof for the Gaussian KL interpretation of Φ\Phi.
  • Other smaller edits.
  • We added the experiment code as supplementary material.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.