acceptodds
Under review as a conference paper at ICLR 2027

KLIQ: KL-Informed Low-Bit Quantization via Attention-Distribution-Aware Fisher-Kronecker Curvature

Abstract

Post-training quantization (PTQ) reduces the memory and computation costs of large language models (LLMs), but preserving accuracy at very low bit-widths remains challenging. Existing reconstruction-based curvature approximations in second-order PTQ can overlook the distribution-dependent sensitivity of attention probabilities to query and key quantization errors. We introduce KLIQ, a backpropagation-free PTQ method that uses KL divergence between full-precision and quantized attention distributions to guide curvature estimation, quantization parameter selection, and error compensation. KLIQ introduces three key innovations: (1) a Fisher-Kronecker curvature surrogate that maps a diagonal approximation of the attention-score categorical Fisher to query and key weight spaces; (2) staged functional calibration that conditions key quantization parameter selection on already quantized and compensated queries to account for query-key error interactions; and (3) KL-derived compensation statistics for compensating quantization errors propagated from preceding layers. Extensive experiments across multiple model families show that, when combined with conventional outlier-suppression techniques, KLIQ achieves state-of-the-art performance on language modeling and downstream tasks under low-bit weight-only and weight-activation quantization. The source code is available at https://anonymous.4open.science/r/KLIQ-7440/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
post-training quantization, large language models, attention-distribution-aware quantization, KL divergence, Fisher information

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kdekOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KLIQ, a backpropagation-free post-training quantization method for low-bit LLM quantization. KLIQ focuses on the query and key projections and uses KL divergence between full-precision and quantized attention distributions to construct Fisher-Kronecker curvature approximations. It further introduces staged functional calibration, in which query quantization parameters are selected first and key parameters are then selected conditioned on the quantized queries, as well as KL-informed propagated-error compensation.

Critical strengths

Interesting observation. The central observation, that equal-magnitude score perturbations can have different effects depending on the local attention distribution, is well motivated. Using post-softmax attention-distribution KL rather than only pre-softmax or layer-output reconstruction is a meaningful direction.

Critical weaknesses
  1. The Hessian of the forward KL with respect to the score vector is indeed the categorical Fisher evaluated at the current quantized distribution. However, the Taylor expansion in Appendix B contains a nonzero first-order term whenever the current quantized attention distribution differs from the full-precision target. KLIQ retains only the positive-semidefinite quadratic term as its curvature metric. This can be a reasonable approximation or preconditioner, but the paper does not clearly justify why dropping the first-order term is appropriate for the actual quantization perturbation, nor does it clearly explain at which Q/K state the Fisher is evaluated in the sequential procedure.

  2. The paper shows diagonal-versus-full Fisher and calibration-strategy ablations, but does not isolate the effect of using the KL-derived curvature factors versus the KL search objective, or the effect of the proposed compensation statistics. In particular, the unconstrained stationary correction for the query projection appears to reduce to the standard propagated-error correction because its column factor and cross-covariance do not include the Fisher weighting; the KL dependence enters through the row factor and the discrete sequential procedure.

Questions for authors
  1. Please clarify precisely at which state the score Fisher is evaluated during KLIQ, why the first-order term can be omitted for the proposed sequential quantization updates, and whether the resulting objective should be interpreted as a local approximation to the original KL or as a Fisher-based surrogate/preconditioner.

  2. Please provide matched component ablations that isolate: (a)KL-based Q/K parameter search versus the existing TurboBoA search procedure; (b) KL-derived curvature versus the TurboBoA curvature under the same search procedure; and (c) KL-derived propagated-error compensation versus standard propagated-error compensation or no propagated-error compensation.

  3. Can the authors provide a clearer aggregate analysis of when staged calibration helps, including variability across calibration subsets or random calibration seeds?

1 comment
AuthorsOpenReview ↗

KLIQ’s Fisher surrogate clarification, component ablations, and calibration across seeds

Dear Reviewer,

Thank you for your very constructive feedback. We clarified the Fisher surrogate, added component ablations, and evaluated staged calibration across three calibration runs.

Q1. Fisher evaluation and the first-order term clarification

Q1.1. Fisher evaluation point

Before quantizing Q or K, we compute the categorical Fisher diagonal using full-precision Q/K weights and activations from preceding quantized layers. Both updates use the resulting Kronecker factors.

Revisions marked in red. Line 2 and 5 of Algorithm 1 (p.4) and §3.2.1 (before Eq.3, p.5) specify the Fisher evaluation point.

Q1.2. Local quadratic surrogate

Our original draft was too brief about the update objective. As you point out, the forward-KL gradient q−rq-r can be nonzero. The weight updates use the local Fisher quadratic of DKL(q∥q′)D_{\rm KL}(q\|q'). with qq fixed, and q′q' is the attention distribution after a score perturbation. At zero perturbation (q′=qq'=q), the first derivative vanishes, and the second-order term is the Fisher quadratic. Staged parameter search directly evaluates KL to the full-precision target rr.

Figure 2a (p. 9) compares the quadratic predictions with measured attention KL layer by layer, using the same held-out Q/K perturbations. After averaging over query positions, the diagonal Fisher yielded median predicted-to-measured KL ratios closer to 1 across all three models (0.79–1.28 versus 0.51–0.64 for the full Fisher). This comparison supports the diagonal Fisher quadratic as a local surrogate for the tested perturbations. In the quantization comparison (Figure 2b), the diagonal variant also yielded lower measured attention KL than the full variant in all tested model layers.

Revisions marked in red. Appendix B.1 (after Eq. 20, pp. 14–15) clarifies the derivation of the surrogate. Section 4.3 (pp. 7–9, Figure 2) adds the supportive layerwise analysis.

Q2. Component ablations

We add the three KLIQ components cumulatively under 2-bit weight-only quantization with QuaRot. The comparison below supplements Table 4 with existing TurboBoA runs that match KLIQ’s activation ordering for each model. Entries show C4 perplexity ↓ / mean zero-shot accuracy (%) ↑.

Configuration1B3B8B
TurboBoA56.97 / 38.7430.12 / 46.5925.10 / 51.78
KLIQ curvature56.41 / 38.8430.87 / 47.1026.18 / 51.97
+ staged KL search54.47 / 39.1729.70 / 47.4525.36 / 52.67
+ KL-derived compensation53.30 / 38.7929.04 / 47.7824.97 / 53.00

With TurboBoA’s search, changing the curvature alone gave mixed perplexity results. Adding staged KL search improved both metrics on all three models. Enabling KL-derived compensation further lowered perplexity on all three models. Zero-shot accuracy rose on 3B and 8B and fell on 1B.

As you noted, the row factor cancels in the unconstrained query correction, reducing the query to standard form. The Fisher-weighted query row factor still governs the discrete updates across row blocks. For keys, Fisher weighting enters both the column factor and the propagated-error cross-covariance (Appendix C, pp. 17–18, Eqs. 45–48).

Revisions marked in red. §4.3 (pp. 9–10, Table 4) adds the component ablation and quantization costs.

Q3. When staged calibration helps

Q-first outperformed independent calibration in 8 of 12 model–metric comparisons when averaged across three calibration runs. In grouped-query attention, Q-first calibrates each shared key head against the quantized queries in its group.

Each cell shows Q-first / independent calibration, with the better result in bold.

ModelWiki2 PPL ↓C4 PPL ↓Mean zero-shot (%) ↑Five-shot MMLU (%) ↑
1B52.51 / 52.0554.14 / 54.2338.94 / 39.3625.70 / 25.21
3B25.98 / 27.3929.14 / 30.4147.51 / 47.1028.33 / 27.34
8B22.55 / 22.3825.05 / 25.5852.52 / 51.8829.60 / 29.93

Across individual calibration runs, Q-first led in 5 to 8 of 12 comparisons.

Revisions marked in red. Appendix E.5 (p. 26, Table 11b) reports the comparison averaged across three calibration runs. Section 4.3 (p. 10) points to this analysis.

Thank you again for your feedback. It helped us clarify the role of the Fisher surrogate and staged calibration. The revised paper also extends the 2-bit evaluation to dense models up to 32B and to attention-only quantization of MoE models, including one with MLA (Appendix E.2, p. 23, Table 8), and presents consistent gains. We added inference memory, latency, and throughput benchmarks (Appendix E.4, p. 25, Table 10b) and a staged-search ablation on LongBench (§4.3, p. 10, Table 5). The new results further strengthen the case for KLIQ as an effective method for 2-bit post-training quantization across model architectures.

Reviewer_g23qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors observe that, under low-bit quantization, existing methods overlook the impact of quantization errors on attention distributions, making accuracy hard to preserve. To address this, they propose KLIQ, which uses the KL divergence between full-precision and quantized attention distributions as a unified objective. Through theoretical derivation, the authors use Fisher information to obtain Kronecker factors that depend on the attention distribution, which guide the subsequent quantization. Building on this, they select Q/K quantization parameters in stages by minimizing the KL divergence: queries are processed with full-precision keys, and keys are then processed using the quantized and compensated queries, capturing query-key error interactions. In addition, the Kronecker factors and the corresponding error cross-covariance are used to compensate for quantization errors propagated from preceding layers. Under 2-bit weight-only quantization on LLaMA, Qwen3, and Mistral models, KLIQ reduces Wiki2 and C4 perplexity by an average of 6.99% and 6.37% relative to TurboBoA and improves average zero-shot accuracy and MMLU by 0.79 and 2.05 pp, respectively, with consistent gains under W2A4 weight-activation quantization as well.

Critical strengths
  1. The paper provides a theoretical derivation: starting from the KL divergence of attention distributions, it derives the Kronecker curvature factors and the error-compensation form for Q/K, and adapts them to RoPE and RMSNorm, giving the method a clear theoretical foundation.
  2. The additional overhead is modest. Compared with TurboBoA, KLIQ increases peak memory by only 5.9% and quantization time by 30.5% on average, striking a reasonable trade-off between computational feasibility and modeling capacity.
  3. The authors release their code, which supports reproducibility.
Critical weaknesses
  1. Limited model scale. The evaluated models range from 1B to 14B, lacking validation on larger models such as 30B or 70B.
  2. Limited architecture coverage. The experiments only involve dense models with standard MHA/GQA attention, lacking validation on Mixture-of-Experts (MoE) architectures. Moreover, since KLIQ's derivation relies on standard softmax attention, the paper does not discuss whether it generalizes to other attention mechanisms such as MLA and DSA, or to hybrid models that combine linear and full attention.
  3. Missing deployment evaluation. As a quantization method, the paper only reports the memory and time costs of offline quantization, without reporting deployment metrics such as quantized model size and inference memory usage, or inference speedup metrics such as throughput, TTFT, and TPOT, making it difficult to assess the practical benefits of the method in real deployment.
Questions for authors
  1. How does KLIQ perform on larger models, MoE architectures, and different attention architectures such as MLA, DSA, and hybrid models?
  2. Could the authors report inference memory usage and inference speedup metrics such as throughput, TTFT, and TPOT?
  3. The calibration sequences contain 2048 tokens, and the parameter search retains only the first 1024 tokens. In long-context scenarios such as LongBench, attention distributions may differ substantially from those of short sequences (e.g., becoming more dispersed). Does the Fisher-Kronecker curvature estimated on short sequences still apply in this case, and how does this discrepancy affect quantization quality? Could the authors report KLIQ's performance on long-context tasks?
1 comment
AuthorsOpenReview ↗

KLIQ gains across architectures (with inference benchmarks and a staged-search ablation on LongBench)

Dear Reviewer,

Thank you for your careful review. In response, we expanded our model evaluation, added deployment benchmarks, and tested staged calibration on LongBench.

Q1. Larger models, MoE, and attention architectures

We added 2-bit weight-only results for dense Qwen3-32B, Qwen3-30B-A3B-Base (MoE), and DeepSeek-V2-Lite (MoE with MLA). Both methods used QuaRot. On MoE models, we quantized only the targeted attention weights and kept experts and routers in FP16.

KLIQ improved all four metrics over TurboBoA on all three models. Five-shot MMLU gains were 1.1–3.7 percentage points.

ModelMethodWiki2 PPL ↓C4 PPL ↓Zero-shot avg. (%) ↑MMLU (%) ↑
DeepSeek-V2-LiteFP167.289.5665.7848.61
TurboBoA8.6411.3461.7838.09
KLIQ8.0910.6363.2741.33
Qwen3-30B-A3B-BaseFP166.0810.0071.6081.06
TurboBoA8.4412.7265.5969.93
KLIQ7.4911.3167.2473.64
Qwen3-32BFP167.6011.2556.6881.88
TurboBoA12.0416.5153.5167.53
KLIQ11.7116.0654.2868.62

Revisions marked in red. Appendix E.2 (p. 23, Table 8) adds the setup and results. The Introduction (p. 2, first sentence of the final paragraph), §4.2 (p. 7, Weight-Only Quantization), and Conclusion (p. 10, final sentence) reflect the expanded evaluation.

Q2. Inference memory and speed

We used one NVIDIA A100-SXM4 (80 GB), batch size 1, a 1,024-token prompt, 64 generated tokens, and a 16-bit KV cache. Both packed W2A16 methods shared the same backend.

KLIQ used 43–74% less peak GPU memory than FP16. Throughput was 11–38% lower. Weight dequantization and QuaRot’s online Hadamard transform can offset savings from reduced weight reads. KLIQ matched packed TurboBoA’s peak memory and checkpoint size, with similar TTFT. TPOT and throughput favored KLIQ on 1B and 3B and TurboBoA on 8B.

ModelMethodCheckpoint (GiB)Peak mem. (GiB)TTFT (ms) ↓TPOT (ms) ↓Throughput (tok/s) ↑
LLaMA 3.2-1BFP162.302.4035.98.11116.96
TurboBoA1.211.3646.610.2991.85
KLIQ1.211.3648.39.56101.39
LLaMA 3.2-3BFP165.986.1786.816.9955.30
TurboBoA2.132.39113.219.5548.17
KLIQ2.132.39113.418.9349.17
LLaMA 3-8BFP1614.9615.2083.617.8153.10
TurboBoA3.593.98115.726.4935.86
KLIQ3.593.98115.828.7933.17

Throughput includes prompt processing.

Revisions marked in red. Appendix E.4 (p. 25, Table 10b) adds the benchmark protocol and results. Section 4.2 (p. 7) points to Appendix E.4.

Q3. Long-context performance

We estimate the Fisher–Kronecker factors using 2,048-token calibration sequences. The 1,024-token truncation applies only to staged parameter search. The same Fisher formulation applies at longer context lengths. Its diagonal weights depend on the local attention probabilities through p(1−p)p(1-p), so shifts in those probabilities can make the calibrated factors less representative of long-context sensitivity.

We varied the staged-search sequence length and count using the same 2,048-token Fisher calibration. We evaluated 2-bit weight-only KLIQ with QuaRot on six LongBench v1 tasks, with prompts capped at 8,192 tokens.

Table 5 excerpt. LongBench scores (↑)

Model8 × 5128 × 1,024 (default)8 × 2,0484 × 1,02416 × 1,024
LLaMA 3.2-1B13.7314.9213.0914.2313.17
LLaMA 3.2-3B17.9618.3115.7920.5418.64
LLaMA 3-8B21.4522.4522.3018.4920.88

Columns show the number of search sequences × tokens per sequence.

The default achieved the highest mean LongBench score (18.56). At a fixed count of eight sequences, doubling the search length lowered scores on all three models and increased quantization time by 14%. Measuring the effect of longer contexts on Fisher estimation would require directly varying the curvature-calibration length.

Revisions marked in red. Section 4.3 (p. 10, Table 5) adds the LongBench results and the staged-search ablation.

In §4.3 (pp. 8–9), the diagonal Fisher yielded lower held-out attention KL in all tested model layers than the full Fisher after both were averaged over query positions. We also measured each component’s incremental contribution.

Thank you again for your thoughtful feedback. You helped us clarify the method and broaden the evaluation. The new results further strengthen the case for KLIQ as an effective approach to 2-bit quantization across model scales and architectures.

Reviewer_F3ZDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a backpropagation-free PTQ methods that improve BoA, TurboBoA by considering the quantization impact on the attention map (post-softmax scores). To this end, the authors developed advanced Hessians for query and key projection weights. Additionally, the authors proposed staged calibration strategy where the key projection weight is quantized by exploiting the quantized query projection weight (not the full-precision one).

Critical strengths

The paper is well-written.

The motivation is clear, and the authors pointed out the limitation of conventional BoA and TurboBoA in terms of the quantization of query and key projection weights.

The approach that the authors took is reasonable, and the mathematical derivation seems to be correct.

Critical weaknesses

The contribution is incremental.

While I acknowledged the authors' efforts on improving BoA and TurboBoA, most of mathematical analyses are based on BoA and TurboBoA.

The most novel contribution is the development of advanced Hessians by incorporating the KL divergence between the quantized and full-precision attention maps. However, this applies only to query and key projection weights, and except these developments, all the extensions of BoA / TurboBoA's formulations are straightforward in my opinion.

That's why my decision leans towards rejection rather than acceptance.

Questions for authors
  1. How does the authors solve Eq. (11)? It is non-trivial to solve this because the objective function is non-linear (see Eq. (10)), which is in contrast to conventional methods where the objective is linear (e.g., L=∣∣ΔWX∣∣F2\mathcal{L} = || \Delta \mathbf{W} \mathbf{X} ||_{F}^{2}).

  2. For the staged quantization parameter search, does the quantization order matter? Specificially, what happened if the key projection weight is quantized first, and then the query projection weight is quantized based on the quantized key projection weight? Is there any reason why the authors quantize the query projection weight first?

  3. Can the authors provide a justification behind the result that the diagonal Fisher approximation leads to much better performance? To my understanding, full Fisher does not involve any approximation and is more accurate than diagonal Fisher, but the performance gap between these two is very large, especially on Llama2-13B.

3 comments
AuthorsOpenReview ↗

KLIQ’s grid search clarification, Q/K calibration order, and diagonal Fisher results

Dear Reviewer,

We highly appreciate your constructive comments. Your questions helped us clarify the parameter search and investigate the effects of calibration order and the Fisher approximation.

Q1. How we solve Eq. (11)

We enumerate shrinkage factors for per-row weight ranges and zero-point rounding choices. We fake-quantize each candidate projection and score it by attention KL on the calibration subset (Eq. 10). We select the lowest-KL candidate per query or shared key head and fix its scales and zero points during sequential updates. Q search uses full-precision keys. K search uses quantized and compensated queries (Algorithm 1, p. 4 and §3.3, p. 6).

Q2. Why calibrate Q first?

In grouped-query attention, several query heads share a key head. We chose Q-first so the shared key parameters minimize attention KL across the group after its individual query heads are quantized and compensated. We also tested K-first.

Entries show Q-first / K-first.

ModelWiki2 PPL ↓C4 PPL ↓Mean zero-shot (%) ↑Five-shot MMLU (%) ↑
1B49.86 / 51.8653.30 / 54.0438.79 / 38.9425.98 / 25.87
3B25.99 / 26.1629.04 / 29.0247.78 / 47.3827.77 / 29.00
8B21.87 / 21.4224.97 / 24.7953.00 / 52.2228.50 / 30.02

Thank you for suggesting that we reverse the calibration order. On the 3B and 8B models, Q-first gave higher mean zero-shot accuracy and K-first gave higher MMLU. The revised paper now discusses K-first as a viable alternative and explains how the preferred order depends on the model and metric.

Revisions marked in red. Appendix E.5 (p. 26, Table 11a) adds the Q-first versus K-first ablation.

Q3. Diagonal versus full Fisher and the 13B collapse

Q3.1. Why did diagonal Fisher perform better?

The full Fisher gives the exact local curvature for each query in attention-logit space. KLIQ averages these matrices over query positions to obtain one shared Fisher per head and make the weight-space curvature computation tractable.

Figure 2a (p. 9) shows that after query averaging, the diagonal quadratic’s median prediction was closer to measured KL on all three models. For the same held-out Q/K perturbations, averaging shifted both estimates downward. The full quadratic went from 103.4–106.9% of measured KL to 50.8–63.9%, while the diagonal medians moved closer to 100% (78.6–128.0%). The averaged off-diagonal terms lowered the full quadratic relative to the diagonal in all 76 layers.

Empirical results in Figure 2b (p. 9) show that the diagonal variant yielded lower measured attention KL after quantization in all 76 tested layers.

Revisions marked in red. Section 4.3 (pp. 7–9, Table 3 and Figure 2) adds the layerwise Fisher diagnostics and downstream results.

Q3.2. Why LLaMA 2-13B collapses?

For LLaMA 2-13B, the collapse occurred with row activation ordering enabled. The loss spiked after layer 28 while Q/K attention KL on common inputs changed little, suggesting possible instability in the value (V) projection. TurboBoA also collapsed with row ordering on this model. With both row and column ordering disabled, the full-Fisher rerun achieved 11.76 Wiki2, 13.37 C4, 57.97% mean zero-shot accuracy, and 34.92% five-shot MMLU. The originally reported diagonal-Fisher results were 10.00, 11.79, 59.47%, and 36.38%, respectively.

Revisions. We removed the LLaMA 2-13B row from Table 3 so the revised ablations use the same three LLaMA 3 models throughout the paper.

KLIQ’s contribution

BoA extends GPTQ-style quantization by deriving curvature from attention-output reconstruction. TurboBoA builds on BoA with joint quantization of multiple output rows and GPTAQ-inspired compensation for errors propagated from earlier layers.

KLIQ derives Q/K weight-space curvature from KL divergence between full-precision and quantized attention distributions. The diagonal of the query-averaged categorical Fisher enters the query row factor and the key column factor. On the key side, it also weights the propagated-error statistics. Staged calibration then uses attention KL to select key quantization parameters against the quantized and compensated queries. Table 4 (p. 10) reports the incremental effect of each component.

KLIQ consistently improves perplexity and downstream accuracy over matched TurboBoA baselines across the evaluated LLaMA, Qwen, and Mistral models. The revised evaluation extends those gains to dense Qwen3-32B and to attention-only quantization of MoE models, including one with MLA (Appendix E.2, p. 23, Table 8). We believe the distinct derivation and the gains across model sizes and architectures make KLIQ a substantive contribution to 2-bit post-training quantization.

I appreciate the authors' response.

However, I still think the contribution over BoA/TurboBoA is marginal. Most of the mathematical derivation is essentially the same as in BoA with incorporating the diagonal Fisher term, derived from the KL diverence between the full-precision and quantized attention maps, into the curvature.

Accordingly, the improvement in downstream accuracy (e.g., MMLU) over TurboBoA is small even at 2-bit, which I believe the improvement at 3-bit and 4-bit would be very small.

Therefore, I keep insisting my score.

AuthorsOpenReview ↗

Same Derivation? Small Gains?

We appreciate the reviewer's prompt response. With respect and scientific rigor, we strongly disagree with the reviewer's opinion that our contribution is marginal and its math is the same as BoA. BoA and TurboBoA build on GPTQ’s squared-error reconstruction approach by applying it to attention outputs. We derive the Q/K curvature from the second-order Taylor expansion of the KL divergence between full-precision and quantized attention distributions. We retain the diagonal of the categorical Fisher in attention-logit space, average it over query positions, and map the resulting quadratic into Q/K weight space. The similar final Kronecker form may give the impression that the derivations are the same. The use of Fisher information matrix was inspired by Shun-ichi Amari's seminal work on natural gradient descent in the 90s (https://proceedings.neurips.cc/paper/1996/file/39e4973ba3321b80f37d9b55f63ed8b8-Paper.pdf) and James Martens and Roger Grosse recent work on Kronecker-factored Approximate Curvature for neural networks in 2015 (https://arxiv.org/abs/1503.05671). Yoshua Bengio et al. showed the difference between Hessian and Fisher (https://arxiv.org/abs/1906.07774v1). Essentially, updating in a distributional manifold is substantially different from updating in the Euclidean space.

We also disagree with the characterization of the 2-bit MMLU gains as small. KLIQ improves five-shot MMLU and mean zero-shot accuracy and lowers perplexity on both WikiText-2 and C4 over matched TurboBoA on all 13 evaluated models. These include 11 dense models and two MoE models with attention-only quantization. The mean MMLU gain is 2.19 percentage points across all 13 evaluated models. The largest gain is 5.35 points on Mistral-7B-v0.3 (29.40% to 34.75%). For context, TurboBoA’s own INT2 results with QuaRot show a mean gain of 2.19 points over BoA across five Llama models on its eight-task zero-shot benchmark (Table 4b, p.8, TurboBoA).

We respectfully request that the reviewer kindly reconsiders the evaluation. By the same reasoning, one could describe BoA as essentially GPTQ with MSE-based attention-output reconstruction, and TurboBoA as essentially BoA with GPTAQ-style propagated-error compensation.

KLIQ consistently improves over matched TurboBoA baselines in W2A16KV16, W2A4KV16, and W2A4KV4. Our weight-only experiments extend beyond the Llama models studied in TurboBoA to dense Qwen and Mistral models, as well as attention-only quantization of MoE Qwen3-30B-A3B and DeepSeek-V2-Lite, the latter using multi-head latent attention (MLA). If sharing a solver and a similar algebraic form is enough to call a contribution marginal despite a distinct curvature derivation and consistent gains across these settings and architectures, the same standard would classify many extensions of established methods as marginal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.