acceptodds
Under review as a conference paper at ICLR 2027

OPQAD: Recovering Capability under KV-Cache Quantization via On-Policy Quantization-Aware Distillation

Abstract

QAD is a common approach for recovering capability loss in post-training quantization (PTQ). However, naive off-policy QAD recovers only 41.5% of the loss for a 1.5B model under TurboQuant K2V3. To understand this bottleneck, we analyze the issue from two perspectives: quantization-induced reasoning degradation and the intrinsic limitations of off-policy QAD. We find that quantization creates a distributional "flattening effect" and suppresses key high-confidence tokens, while off-policy QAD suffers from exposure bias and layer-wise representation drift. Building on these insights, we propose On-Policy Quantization-Aware Distillation (OPQAD), which integrates an on-policy backbone, reverse KL divergence, layer-wise hidden-state alignment, and a gated forward KL to correct suppressed tokens. Evaluated across 1.5B–8B models and long CoT tasks (up to 32K length), OPQAD improves AIME24 math accuracy by 6.0–7.3 percentage points over off-policy QAD under K2V3, raising the 1.5B model's recovery rate to 58.7%, while also yielding modest gains under OSCAR. Trained solely on math data, OPQAD generalizes to out-of-distribution code and instruction-following tasks with 9.7% training overhead. Ablations confirm that OPQAD reduces student-prefix distribution errors, hidden-layer drift, and extreme probability inversions, validating its effectiveness for low-bit KV cache recovery.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, KV cache quantization, Quantization-aware distillation, On-policy distillation, Low-bit inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KfyXOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies capability degradation caused by low-bit KV-cache quantization and proposes OPQAD, an on-policy quantization-aware distillation framework for recovering reasoning performance. The authors identify several failure modes, including distribution flattening, critical-token probability collapse, layer-wise representation drift, and exposure bias in conventional off-policy QAD. OPQAD addresses these issues with student-generated on-policy trajectories, reverse-KL distillation, hidden-state alignment, and a gated forward-KL correction for severely underestimated tokens. Experiments on Qwen-based models from 1.5B to 8B with K2V3 and OScaR show consistent recovery gains on mathematical reasoning, with additional transfer to code and instruction-following tasks. Overall, the paper is technically well motivated and supported by extensive diagnostic analyses and ablations.

Critical strengths

S1. Careful and insightful diagnosis of capability degradation under KV-cache quantization.
The paper goes beyond reporting accuracy degradation and systematically studies how low-bit KV-cache quantization changes model behavior. The analyses on distribution flattening, high-confidence token probability inversion, layer-wise representation drift, and exposure bias are well motivated and supported by multiple complementary measurements, including entropy/KL statistics, Gaussian-noise controls, arithmetic error analysis, hidden-state patching, and prefix-dependent error analysis.

S2. A well-motivated recovery framework that directly follows the diagnosed failure modes.
OPQAD is conceptually coherent: on-policy trajectories address exposure bias, reverse KL targets distribution flattening, hidden-state alignment mitigates accumulated representation drift, and gated forward correction focuses on severely underestimated critical tokens. This mechanism-driven design is more convincing than introducing a generic distillation loss without analyzing the source of quantization errors.

S3. Strong empirical recovery across model scales, quantizers, and downstream tasks.
OPQAD consistently improves over off-policy QAD across 1.5B--8B models and both K2V3 and OScaR quantization. The improvements are particularly substantial in the challenging K2V3 setting; for example, BroRL-1.5B improves from 25.21 to 29.86 Math Avg over reverse-KL QAD, while Qwen3-4B improves from 35.07 to 42.36. Moreover, although training uses only mathematical data, the recovered model generally maintains or improves performance on code generation and instruction-following tasks, suggesting useful cross-task transfer.

S4. Extensive supporting analysis and a reasonably practical on-policy training implementation.
The paper provides substantially more diagnostic and ablation evidence than is typical for a quantization-recovery study, including KL-direction comparisons, exposure-bias measurements, extreme-inversion statistics, hidden-representation analyses, and sensitivity studies. It also implements low-bit student rollouts in the training pipeline and reports that on-policy training incurs only about 9.7% additional total training time over the off-policy baseline, demonstrating that the proposed training paradigm is practically feasible.

Critical weaknesses

W1. Limited component-level analysis and efficiency breakdown.
OPQAD combines several relatively ad-hoc components, including hidden-state alignment and gated probability correction. Although some ablations are provided, they are mainly restricted to BroRL-1.5B with K2V3, making it unclear whether these components are consistently useful across models and quantizers. Moreover, Table 4 only reports a coarse stage-level breakdown (generation, teacher scoring, and actor update), without isolating the additional computation and peak-memory overhead introduced by these OPQAD-specific operations. The different response lengths between on- and off-policy training further make the efficiency comparison less controlled.

W2. Missing practical accuracy--memory Pareto comparison.
The largest improvements are observed in aggressively quantized settings where the unadapted model has already suffered severe accuracy degradation. It is therefore unclear whether OPQAD improves the Pareto frontier in the practically useful regime, where existing KV-cache quantization methods can already retain relatively high accuracy. For example, KIVI [1] demonstrates competitive quality with 2-bit KV-cache quantization, while less aggressive quantization generally provides an even stronger accuracy baseline. An accuracy-versus-effective-KV-bitrate/memory comparison against representative methods such as KIVI [1] and OScaR [2] would better show whether OPQAD can make a lower-bit configuration approach the accuracy of a higher-precision, near-lossless configuration.

W3. Limited model-family generality.
Although the experiments span 1.5B, 4B, and 8B models, all evaluated models are Qwen-based, including BroRL-1.5B. Since the paper makes broader mechanistic claims about quantization-induced distribution flattening, representation drift, and probability collapse, evaluation on an architecturally different family would strengthen the generality of the conclusions. Representative choices could include Llama 3 [3] or Gemma 4 [4].

[1] Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V. Braverman, B. Chen, and X. Hu.
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. arXiv:2402.02750, 2024.

[2] Z. Su, R. Yang, C. Zhang, Y. Liu, Y. Zhang, W. Wu, J. Xiong, D. Du, X. Zhuang, Y. Qian, Y. Xie, Y.-C. Wu, H. Yang, and N. Wong.
OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond. arXiv:2605.19660, 2026.

[3] A. Grattafiori et al.
The Llama 3 Herd of Models. arXiv:2407.21783, 2024.

[4] Gemma Team et al.
Gemma 4 Technical Report. arXiv:2607.02770, 2026.

Questions for authors

How well does OPQAD extend beyond conventional dense KV-cache quantization? Recent models increasingly use linear or sparse attention, where the marginal memory bandwidth benefit of KV quantization can differ substantially from standard attention. Could OPQAD be adapted to quantized recurrent states or sparsely accessed KV caches? In addition, can its on-policy distillation and recovery objectives be incorporated during post-training to make models natively robust to practical FP8/FP4 KV-cache formats, rather than being used only as post-hoc recovery?

Reviewer_jxCAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies recovery of long-form reasoning after low-bit KV-cache quantization. Its diagnostics identify uneven distribution flattening, extreme probability inversions at some high-confidence tokens, hidden-state drift, and differences between teacher and student prefixes. OPQAD combines student-generated rollouts, full-vocabulary reverse KL, half-norm hidden-state alignment, and a gated squared log-ratio penalty. Experiments use K2V3 on BroRL 1.5B and Qwen3-4B/8B, and OSCAR on Qwen3-4B/8B, evaluating three AIME sets, code, and instruction following. Mathematics averages improve over reverse-KL off-policy QAD in all five configurations. For 1.5B, recovery of the quantization-induced accuracy loss increases from 41.5% with forward-KL off-policy QAD to 58.7%. A separate B300 timing experiment reports 9.7% additional total cost.

Critical strengths

The diagnosis goes beyond aggregate accuracy and includes controls for plausible alternative explanations. In Table 12, variance-matched independent Gaussian noise does not reproduce both the entropy increase and KL asymmetry observed under quantization. Full-vocabulary measurements help avoid selection effects from nucleus truncation. Token categories, local arithmetic checks, and hidden-state patching provide complementary evidence. The authors also state the limits of the fixed-suffix arithmetic analysis rather than treating it as a complete causal explanation of free-generation failures. This diagnostic contribution is a substantial part of the paper’s value.

Critical weaknesses
  1. The value of the recovery term is better established than the need for its squared form. Tables 3 and 20 show that removing the term lowers mathematics accuracy and leaves more extreme inversions, and Appendix C.2 explains its stronger gradient at severely underestimated tokens. What is not isolated is whether squaring the gated log ratio improves on an otherwise matched unsquared version with the same teacher support and gate. The current comparisons establish the value of adding targeted recovery to reverse KL, but provide less evidence for this particular weighting of the recovery signal.
  2. The uncertainty of the main task improvements is not quantified. Appendix A.3 specifies 30 problems and 16 responses per problem for each AIME set, but Table 2 does not provide paired intervals for the accuracy differences. These repeated responses share problems, so resampling should preserve that structure. This matters particularly for the 1.17-point mathematics gain on 8B OSCAR and the 0.39-point code-average gain on 8B K2V3. The paper already provides problem-paired exposure-bias intervals and trajectory-paired inversion statistics; those support the diagnostics but do not quantify uncertainty in the main benchmark gains. The larger K2V3 improvements remain encouraging, while the smaller transfer and OSCAR gains warrant more cautious interpretation.
Questions for authors
  1. With the rollout procedure, teacher support, gate conditions, and training budget fixed, how does the squared penalty compare with an unsquared version of the same gated penalty? This would distinguish the benefit of targeting these tokens from the benefit of the squared weighting.
  2. Can you report paired bootstrap intervals for the Table 2 and Table 3 accuracy differences, preserving all responses from each sampled problem? I am particularly interested in the smaller OSCAR/code gains and the 2.50-point loss increment in the on-policy comparison. The existing diagnostic intervals do not answer this benchmark question.
  3. As shown in the weakness.
3 comments
AuthorsOpenReview ↗

Follow-up on our response to Reviewer jxCA

Dear Reviewer jxCA,

We have posted our responses to your comments. As the discussion period is nearing its end, we would appreciate it if you could take a look when you have a chance. Please let us know if any concerns remain or if further clarification would be helpful.

Thank you for your time and consideration!

The authors

Area_Chair_uaMSOpenReview ↗

Please respond to the rebuttal

Dear Reviewer jxCA, could you please respond to the authors' rebuttal at your earliest convenience? Thanks! --AC

AuthorsOpenReview ↗

A short follow-up with additional results

Dear Reviewer jxCA,

Thank you again for your time and for the careful comments on our work. We hope our earlier response has helped address your two questions, on the squared recovery term and on the paired intervals for the main results.

We would also like to briefly share some additional experiments from the discussion with other reviewers, since we think they are closely related to your concern. Following their suggestions, we trained OPQAD and reverse-KL QAD again under other bit widths (K3V2 and K3V3), on a hybrid Mamba2–attention model with quantized recurrent states, and in a quantization-aware post-training setting with NVFP4 KV caches. OPQAD improves Math Avg over reverse-KL QAD in each of them. We also computed the paired bootstrap intervals in the way you suggested, keeping all responses of each sampled question, and under K3V2, K3V3 and the recurrent-state setting the whole interval stays above zero. The component ablation repeated on Qwen3-4B also shows the same trends as on BroRL-1.5B.

As these are independent training runs, we believe they show that the advantage of OPQAD over off-policy QAD is not specific to one model or one quantization setting, but holds consistently, which we hope can further address your concern on statistical significance. These results are now included in the revised manuscript (Table 3b and Appendices E.2–E.4).

We understand this is a busy period, and we would greatly appreciate it if you could take a look when you have time. Please do not hesitate to let us know if you have any further questions.

Best regards, The authors

Reviewer_LQ1NOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents On-Policy Quantization-Aware Distillation (OPQAD), which integrates on-policy distillation with reverse KL, layer-wise hidden states alignment, and a gated KL. It shows that naive off-policy QAD does not recover well under TurboQuant K2V3. It shows that OPQAD enhances the accuracy of AIME24 considerably under K2V3.

Critical strengths
  • The authors clarify the problem of naive off-policy QAD under TurboQuant.
  • To address it, they introduce three components: on-policy distillation, hidden states alignment per layer, a gated squared forward KL.
  • They show experimental results that OPQAD improves math and code capabilities under K2V3.
Critical weaknesses
  • Given that each component the authors introduce is a well-known existing technique, the proposed method seems not novel.
  • The authors focus on K2V3. It would be more instructive if authors also explore K2V2, K3V2, and K3V3.
Questions for authors

NA

1 comment

Dear Reviewer LQ1N

Thank you for your time and thoughtful review. We have posted our detailed responses addressing your comments and questions. With the discussion period wrapping up soon, we would be grateful if you could take a look and let us know if anything remains unclear or if further details would help.

Best regards,

The authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Updated Manuscript

Dear Area Chair and Reviewers,

Thank you for your time and helpful feedback during the review and discussion. We have uploaded a revised version of the manuscript that incorporates the new experiments and analyses from our rebuttal, including paired confidence intervals, expanded ablations, controlled efficiency profiling, Pareto comparisons, and the additional architecture and bit-width results.

We also refined parts of the text, figures, and tables to make the paper easier to follow, while keeping our method and main conclusions unchanged. Please let us know if you have any further questions.

AuthorsOpenReview ↗

Rebuttal Summary

Dear AC, SAC, and PC,

Thank you for coordinating the review and discussion of our submission, and we sincerely thank reviewers LQ1N, jxCA, and KfyX for their thoughtful feedback. We are glad that all three reviewers now hold a positive view of the paper. They found value in our diagnosis of how KV-cache quantization affects reasoning and in the recovery framework built on it. During the discussion, we answered every question raised, in most cases with the exact experiment the reviewer asked for, and these results have been incorporated into the revised manuscript.

Reviewer KfyX (rating 3, confidence 3) The reviewer gave a positive assessment from the start, describing the diagnosis as careful and insightful and the recovery framework as well motivated. The questions were mainly about how well the results generalize. We ran each of the requested experiments, including ablations at a larger scale, a per-component cost breakdown, a Pareto comparison with existing KV-cache quantization methods, and results on other model families and architectures. These show that both the diagnosed failure modes and the gains of OPQAD hold beyond a single model or quantizer. The reviewer confirmed that the concerns are addressed and kept the positive rating.

Reviewer jxCA (rating 2 → 3, confidence 2) The reviewer considered the diagnosis a substantial part of the paper’s value, and asked about the squared recovery term and the statistical uncertainty of the gains. We ran the matched ablation exactly as described together with a gradient analysis, and reported paired bootstrap intervals in the way the reviewer requested. In a follow-up, we also pointed to the replications run for other reviewers. The reviewer replied that the response addressed the concerns and raised the score.

Reviewer LQ1N (rating 3, confidence 3) The reviewer gave a positive rating, recognizing that the paper clearly identifies the problem of off-policy QAD and that OPQAD considerably improves accuracy under K2V3. The two questions, on the novelty of the components and on other bit widths, were answered in our response. We explained how each component is designed for a specific failure mode found in our diagnosis, and added results at the suggested bit widths, where OPQAD outperforms the baseline in each of them.

Overall, all three reviewers now view the paper positively, with two of them confirming that their concerns are addressed and one raising the score. These suggestions have helped make the paper more complete, and we sincerely appreciate the reviewers’ efforts. We would be glad to provide any further results while the discussion is open.

Best regards, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.