- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
WaterKron extends waterfilling scale selection to two-sided GPTQ with a Kronecker-factored Hessian. Its main result derives an asymptotic distortion penalty for approximating a full Hessian by Kronecker factors. Minimizing this penalty leads to the classical flip-flop covariance updates, providing a quantization-based reason to prefer them over Frobenius fitting. The analysis assumes iid Gaussian weights, positive-definite factors, high-rate quantization, and efficient entropy coding. Controlled experiments on 6 Llama and Qwen models report improved KL and perplexity at a target rate of 2 bits per quantized weight. Improvements vary substantially by model, and the paper does not establish realized encoded storage or inference performance.
Critical strengths
- The full-Hessian mismatch result gives an interpretable criterion for selecting Kronecker factors. Its arithmetic-to-geometric-mean expression measures distortion from the generalized eigenvalue spread, and its Gaussian-KL interpretation explains the connection to covariance fitting (Thm. 4.1 and Secs. 4-5).
- The paper distinguishes existing algorithms from its contribution. It credits prior two-sided GPTQ, waterfilling, and flip-flop updates, then derives why the latter optimize the proposed quantization criterion. The side-information convention and order of limits are stated in Appendix C.
- The experiments isolate factor selection through input-only, independent-marginal, Frobenius, and FlipFlop alternatives. At the stated 2-bit target, a tested FlipFlop variant gives the best WikiText-2 perplexity in all 6 model tables. For example, Llama-3.2-1B improves from 49.3 with WaterSIC to 21.0, while Qwen3-8B improves more modestly from 13.0 to 12.7 (Tbls. 2-7).
- The iteration comparisons are informative. Additional Frobenius updates often worsen the language-model metrics, whereas FlipFlop generally improves or levels off after 2 updates. The paper also acknowledges that its theory concerns high rates and that efficient entropy decoding remains a practical limitation.
Critical weaknesses
- The operational rate is insufficiently specified. Sec. 3 explains why the earlier per-column Huffman scheme is no longer adequate, but Appendix B does not identify the coder or entropy estimator, how scales are selected to reach the target rate, or whether any reported rate includes scale and coding overhead. Consequently, the plots support comparisons at a stated target, but their storage interpretation and rate matching cannot be checked. A precise rate calculation and scale-selection procedure are needed.
- The experiments do not directly test the proposed mismatch criterion. They report model KL and perplexity at low rates, while Thm. 4.1 concerns high-rate quadratic distortion with Gaussian weights. No measured mismatch factors or predicted-versus-observed distortion results connect these regimes. The acknowledged asymptotic scope is reasonable, but a small finite-rate diagnostic would help establish how much of the observed benefit the theory explains.
- The implemented updates differ from the ideal derivation. Sec. 5 motivates curvature using model-sampled labels, whereas Appendix B uses observed labels for an empirical Fisher and repeatedly damps both factors by 0.1 times their mean diagonal. The outer-product formulation still applies to that surrogate, but its interpretation as model KL curvature changes, and the stated ideal minimization/convergence result does not directly characterize the damped iteration. The current results do not separate the factor-fitting objective from regularization effects.
- No substantive reproduction code is supplied or verified. The paper promises a release with publication, but the submission is not really reproducible from the stated implementation and rate-setting details. It also lacks repeated-calibration uncertainty and factor-fitting time or peak-memory measurements, which limits interpretation of the smaller gains and the practical cost.
Questions for authors
- How is each experimental target rate computed and reached? Please specify the scale search, entropy estimator or actual codec, and whether rates include row/column scales and coding overhead. If a bitstream is not produced, please distinguish the estimated rate from measured storage.
- Does the damped FlipFlop implementation reduce the proposed objective on the empirical curvature surrogate? An objective trace and a small damping sensitivity check on an existing model would help distinguish the choice of fitting objective from the effect of regularization.
- Can a small matrix or selected-layer experiment compare the predicted full-Hessian quadratic distortion with measured distortion across rates, including the low-rate regime used in the model experiments? This would test the practical reach of the mismatch factor without requiring a new model benchmark.
Rebuttal
We thank the reviewer for their feedback. We address all weaknesses and questions below.
W1: The coder or entropy estimator is insufficiently specified.
You are right, we should have clarified this better. We use the empirical entropy of Z to measure the rate in practice. While this does not perfectly match the theory of how Z should optimally be encoded, it is a useful proxy, please see the comments to reviewer SPsS where we explained this in detail and also commented on using the Zstd encoder.
We have written a new appendix C that discusses all encoding issues in detail.
W2: The experiments do not directly test the proposed mismatch criterion.
We have now added three module-level metrics to the experiment tables:
Comparing it with DH shows that DHpred accurately predicts (up to about 1% error) the measured distortion DH for two-sided Hessians. Only for the "Input" Hessian the prediction can be off by more (usually about 10% but sometimes more).
(The reason we cannot directly measure the absolute −21logΦ is that we don't have enough calibration data to compute ∣H∣, but the rate gain is a relative measure which we can compute.)
W3: The implemented updates differ from the ideal derivation. (Empirical Fisher, damped matrices)
Regarding empirical Fisher vs real Fisher (with sampled labels), first note that on a perfectly fitted model the two actually agree. We have tested using (a sampled approximation of) the real Fisher in the experiments, but there is no significant difference: The real Fisher can bring a gain of about 0.01 to 0.05 bits; otherwise the results, in particular the ranking of Hessians, stay the same.
As for the damping factors, we did not optimize them, but just chose them large enough so that there are no numerical issues when taking the Cholesky decomposition in the rounding algorithm or inverses in the FlipFlop algorithm. We did observe that using very small damping factors (e.g. 0.01 instead of 0.1) does produce worse results, and we are not entirely sure why. We have added this as an additional limitation to the paper. However, empirically, even with small damping factors, the ranking of the Hessian approximations on the calibration data, in particular the advantage of FlipFlop over the other Hessian approximations, is always maintained.
W4: Reproduction code is not supplied.
We have now attached the experiment code.
Q1: How is each experimental target rate computed and reached?
It is predicted with a non-trivial formula that involves the predicted rate from our theory, where we measure γ, ∣A∣, ∣B∣, the "empirical" variance of W, and the empirical entropy of Z, as well as a correction term to account for the spread introduced by the (two-sided) GPTQ rounding algorithm. Please see the experiment code for the exact formula. We confirm that this accurately predicts the empirical entropy of Z after running the two-sided GPTQ algorithm. The prediction is worse for larger models, but by at most about 0.03 bits, and by about the same amount for all Hessian approximations, which is why we just report the target rate instead of the measured empirical entropy (which would just be another proxy anyway).
Also please see the new appendix C for a detailed discussion of the encoding issues. (more focused on theory instead of experiments)
Q2: Does the damped FlipFlop implementation reduce the proposed objective on the empirical curvature surrogate?
Yes, please see the answer to W2; we have added module-level metrics to the experiments that confirm the reduction of Φ and DH as predicted by the theory.
Q3: Compare predicted full-Hessian distortion with measured distortion.
Again, please see the answer to W2. The newly reported module-level metrics in particular contain the measured DH and the predicted DHpred, which match very well for the two-sided Hessian approximations.
We hope that we have addressed all weaknesses and questions to the reviewer's satisfaction, so that the reviewer can hopefully recommend clear acceptance of the paper.