- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies capability degradation caused by low-bit KV-cache quantization and proposes OPQAD, an on-policy quantization-aware distillation framework for recovering reasoning performance. The authors identify several failure modes, including distribution flattening, critical-token probability collapse, layer-wise representation drift, and exposure bias in conventional off-policy QAD. OPQAD addresses these issues with student-generated on-policy trajectories, reverse-KL distillation, hidden-state alignment, and a gated forward-KL correction for severely underestimated tokens. Experiments on Qwen-based models from 1.5B to 8B with K2V3 and OScaR show consistent recovery gains on mathematical reasoning, with additional transfer to code and instruction-following tasks. Overall, the paper is technically well motivated and supported by extensive diagnostic analyses and ablations.
Critical strengths
S1. Careful and insightful diagnosis of capability degradation under KV-cache quantization.
The paper goes beyond reporting accuracy degradation and systematically studies how low-bit KV-cache quantization changes model behavior. The analyses on distribution flattening, high-confidence token probability inversion, layer-wise representation drift, and exposure bias are well motivated and supported by multiple complementary measurements, including entropy/KL statistics, Gaussian-noise controls, arithmetic error analysis, hidden-state patching, and prefix-dependent error analysis.
S2. A well-motivated recovery framework that directly follows the diagnosed failure modes.
OPQAD is conceptually coherent: on-policy trajectories address exposure bias, reverse KL targets distribution flattening, hidden-state alignment mitigates accumulated representation drift, and gated forward correction focuses on severely underestimated critical tokens. This mechanism-driven design is more convincing than introducing a generic distillation loss without analyzing the source of quantization errors.
S3. Strong empirical recovery across model scales, quantizers, and downstream tasks.
OPQAD consistently improves over off-policy QAD across 1.5B--8B models and both K2V3 and OScaR quantization. The improvements are particularly substantial in the challenging K2V3 setting; for example, BroRL-1.5B improves from 25.21 to 29.86 Math Avg over reverse-KL QAD, while Qwen3-4B improves from 35.07 to 42.36. Moreover, although training uses only mathematical data, the recovered model generally maintains or improves performance on code generation and instruction-following tasks, suggesting useful cross-task transfer.
S4. Extensive supporting analysis and a reasonably practical on-policy training implementation.
The paper provides substantially more diagnostic and ablation evidence than is typical for a quantization-recovery study, including KL-direction comparisons, exposure-bias measurements, extreme-inversion statistics, hidden-representation analyses, and sensitivity studies. It also implements low-bit student rollouts in the training pipeline and reports that on-policy training incurs only about 9.7% additional total training time over the off-policy baseline, demonstrating that the proposed training paradigm is practically feasible.
Critical weaknesses
W1. Limited component-level analysis and efficiency breakdown.
OPQAD combines several relatively ad-hoc components, including hidden-state alignment and gated probability correction. Although some ablations are provided, they are mainly restricted to BroRL-1.5B with K2V3, making it unclear whether these components are consistently useful across models and quantizers. Moreover, Table 4 only reports a coarse stage-level breakdown (generation, teacher scoring, and actor update), without isolating the additional computation and peak-memory overhead introduced by these OPQAD-specific operations. The different response lengths between on- and off-policy training further make the efficiency comparison less controlled.
W2. Missing practical accuracy--memory Pareto comparison.
The largest improvements are observed in aggressively quantized settings where the unadapted model has already suffered severe accuracy degradation. It is therefore unclear whether OPQAD improves the Pareto frontier in the practically useful regime, where existing KV-cache quantization methods can already retain relatively high accuracy. For example, KIVI [1] demonstrates competitive quality with 2-bit KV-cache quantization, while less aggressive quantization generally provides an even stronger accuracy baseline. An accuracy-versus-effective-KV-bitrate/memory comparison against representative methods such as KIVI [1] and OScaR [2] would better show whether OPQAD can make a lower-bit configuration approach the accuracy of a higher-precision, near-lossless configuration.
W3. Limited model-family generality.
Although the experiments span 1.5B, 4B, and 8B models, all evaluated models are Qwen-based, including BroRL-1.5B. Since the paper makes broader mechanistic claims about quantization-induced distribution flattening, representation drift, and probability collapse, evaluation on an architecturally different family would strengthen the generality of the conclusions. Representative choices could include Llama 3 [3] or Gemma 4 [4].
[1] Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V. Braverman, B. Chen, and X. Hu.
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. arXiv:2402.02750, 2024.
[2] Z. Su, R. Yang, C. Zhang, Y. Liu, Y. Zhang, W. Wu, J. Xiong, D. Du, X. Zhuang, Y. Qian, Y. Xie, Y.-C. Wu, H. Yang, and N. Wong.
OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond. arXiv:2605.19660, 2026.
[3] A. Grattafiori et al.
The Llama 3 Herd of Models. arXiv:2407.21783, 2024.
[4] Gemma Team et al.
Gemma 4 Technical Report. arXiv:2607.02770, 2026.
Questions for authors
How well does OPQAD extend beyond conventional dense KV-cache quantization? Recent models increasingly use linear or sparse attention, where the marginal memory bandwidth benefit of KV quantization can differ substantially from standard attention. Could OPQAD be adapted to quantized recurrent states or sparsely accessed KV caches? In addition, can its on-policy distillation and recovery objectives be incorporated during post-training to make models natively robust to practical FP8/FP4 KV-cache formats, rather than being used only as post-hoc recovery?
Follow-up on our response to Reviewer jxCA
Dear Reviewer jxCA,
We have posted our responses to your comments. As the discussion period is nearing its end, we would appreciate it if you could take a look when you have a chance. Please let us know if any concerns remain or if further clarification would be helpful.
Thank you for your time and consideration!
The authors