- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes KLIQ, a backpropagation-free post-training quantization method for low-bit LLM quantization. KLIQ focuses on the query and key projections and uses KL divergence between full-precision and quantized attention distributions to construct Fisher-Kronecker curvature approximations. It further introduces staged functional calibration, in which query quantization parameters are selected first and key parameters are then selected conditioned on the quantized queries, as well as KL-informed propagated-error compensation.
Critical strengths
Interesting observation. The central observation, that equal-magnitude score perturbations can have different effects depending on the local attention distribution, is well motivated. Using post-softmax attention-distribution KL rather than only pre-softmax or layer-output reconstruction is a meaningful direction.
Critical weaknesses
-
The Hessian of the forward KL with respect to the score vector is indeed the categorical Fisher evaluated at the current quantized distribution. However, the Taylor expansion in Appendix B contains a nonzero first-order term whenever the current quantized attention distribution differs from the full-precision target. KLIQ retains only the positive-semidefinite quadratic term as its curvature metric. This can be a reasonable approximation or preconditioner, but the paper does not clearly justify why dropping the first-order term is appropriate for the actual quantization perturbation, nor does it clearly explain at which Q/K state the Fisher is evaluated in the sequential procedure.
-
The paper shows diagonal-versus-full Fisher and calibration-strategy ablations, but does not isolate the effect of using the KL-derived curvature factors versus the KL search objective, or the effect of the proposed compensation statistics. In particular, the unconstrained stationary correction for the query projection appears to reduce to the standard propagated-error correction because its column factor and cross-covariance do not include the Fisher weighting; the KL dependence enters through the row factor and the discrete sequential procedure.
Questions for authors
-
Please clarify precisely at which state the score Fisher is evaluated during KLIQ, why the first-order term can be omitted for the proposed sequential quantization updates, and whether the resulting objective should be interpreted as a local approximation to the original KL or as a Fisher-based surrogate/preconditioner.
-
Please provide matched component ablations that isolate: (a)KL-based Q/K parameter search versus the existing TurboBoA search procedure; (b) KL-derived curvature versus the TurboBoA curvature under the same search procedure; and (c) KL-derived propagated-error compensation versus standard propagated-error compensation or no propagated-error compensation.
-
Can the authors provide a clearer aggregate analysis of when staged calibration helps, including variability across calibration subsets or random calibration seeds?
KLIQ’s Fisher surrogate clarification, component ablations, and calibration across seeds
Dear Reviewer,
Thank you for your very constructive feedback. We clarified the Fisher surrogate, added component ablations, and evaluated staged calibration across three calibration runs.
Q1. Fisher evaluation and the first-order term clarification
Q1.1. Fisher evaluation point
Before quantizing Q or K, we compute the categorical Fisher diagonal using full-precision Q/K weights and activations from preceding quantized layers. Both updates use the resulting Kronecker factors.
Revisions marked in red. Line 2 and 5 of Algorithm 1 (p.4) and §3.2.1 (before Eq.3, p.5) specify the Fisher evaluation point.
Q1.2. Local quadratic surrogate
Our original draft was too brief about the update objective. As you point out, the forward-KL gradient q−r can be nonzero. The weight updates use the local Fisher quadratic of DKL(q∥q′). with q fixed, and q′ is the attention distribution after a score perturbation. At zero perturbation (q′=q), the first derivative vanishes, and the second-order term is the Fisher quadratic. Staged parameter search directly evaluates KL to the full-precision target r.
Figure 2a (p. 9) compares the quadratic predictions with measured attention KL layer by layer, using the same held-out Q/K perturbations. After averaging over query positions, the diagonal Fisher yielded median predicted-to-measured KL ratios closer to 1 across all three models (0.79–1.28 versus 0.51–0.64 for the full Fisher). This comparison supports the diagonal Fisher quadratic as a local surrogate for the tested perturbations. In the quantization comparison (Figure 2b), the diagonal variant also yielded lower measured attention KL than the full variant in all tested model layers.
Revisions marked in red. Appendix B.1 (after Eq. 20, pp. 14–15) clarifies the derivation of the surrogate. Section 4.3 (pp. 7–9, Figure 2) adds the supportive layerwise analysis.
Q2. Component ablations
We add the three KLIQ components cumulatively under 2-bit weight-only quantization with QuaRot. The comparison below supplements Table 4 with existing TurboBoA runs that match KLIQ’s activation ordering for each model. Entries show C4 perplexity ↓ / mean zero-shot accuracy (%) ↑.
With TurboBoA’s search, changing the curvature alone gave mixed perplexity results. Adding staged KL search improved both metrics on all three models. Enabling KL-derived compensation further lowered perplexity on all three models. Zero-shot accuracy rose on 3B and 8B and fell on 1B.
As you noted, the row factor cancels in the unconstrained query correction, reducing the query to standard form. The Fisher-weighted query row factor still governs the discrete updates across row blocks. For keys, Fisher weighting enters both the column factor and the propagated-error cross-covariance (Appendix C, pp. 17–18, Eqs. 45–48).
Revisions marked in red. §4.3 (pp. 9–10, Table 4) adds the component ablation and quantization costs.
Q3. When staged calibration helps
Q-first outperformed independent calibration in 8 of 12 model–metric comparisons when averaged across three calibration runs. In grouped-query attention, Q-first calibrates each shared key head against the quantized queries in its group.
Each cell shows Q-first / independent calibration, with the better result in bold.
Across individual calibration runs, Q-first led in 5 to 8 of 12 comparisons.
Revisions marked in red. Appendix E.5 (p. 26, Table 11b) reports the comparison averaged across three calibration runs. Section 4.3 (p. 10) points to this analysis.
Thank you again for your feedback. It helped us clarify the role of the Fisher surrogate and staged calibration. The revised paper also extends the 2-bit evaluation to dense models up to 32B and to attention-only quantization of MoE models, including one with MLA (Appendix E.2, p. 23, Table 8), and presents consistent gains. We added inference memory, latency, and throughput benchmarks (Appendix E.4, p. 25, Table 10b) and a staged-search ablation on LongBench (§4.3, p. 10, Table 5). The new results further strengthen the case for KLIQ as an effective method for 2-bit post-training quantization across model architectures.