- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes On-Policy Quantization (OPQ), which is a framework for improving the performance of low-bit LLMs by training on student-generated trajectories. With the proposed dynamic KL and KV alignment, OPQ demonstrates promising performance over existing methods.
Critical strengths
- Training on student-generated trajectories is well-motivated because quantization errors will lead to distribution shifts.
- OPQ achieves promising results, especially at 2-bit which outperforms the strongest 9.49%.
- The combination of output distribution alignment and intermediate K/V representation alignment is reasonable. The paper also provides theoretical analysis to support its design.
Critical weaknesses
- RL-like QAT has been explored in prior works, such as QeRL[1], QaRL[2] and ReasoningQAT[3]. So it is necessary to summarize their differences.
- As discussed in appendix, the compared baselines use different training or calibration datasets. But the authors ignore to discuss whether this may lead to performance gap or not.
- The theoretical analysis in Lemma 1 demonstrates that incorporating forward KL provides non-vanishing corrective gradients for underrepresented tokens. However, this property also holds for a fixed mixture of forward and reverse KL. Table 3 only compares Dynamic KL against reverse KL but ignores to evaluate fixed KL mixtures. In addition, the proposed coefficient relies on differences in top-k teacher-probability coverage, which cannot capture distributional changes when the top-k token sets are consistent.
[1] Huang W, Ge Y, Yang S, et al. QeRL: Beyond efficiency-quantization-enhanced reinforcement learning for LLMs[C]//International Conference on Learning Representations. 2026, 2026: 99338-99361.
[2] Gu H, Wang H, Liu J, et al. Qarl: Rollout-aligned quantization-aware rl for fast and stable training under training–inference mismatch[C]//Findings of the Association for Computational Linguistics: ACL 2026. 2026: 17051-17064.
[3] Yasuyuki Okoshi, Hikari Otsuka, Daichi Fujiki, Masato Motomura, Towards Quantization-Aware Training for Ultra-Low-Bit Reasoning LLMs[C]// The Fourteenth International Conference on Learning Representations (ICLR 2026).
Questions for authors
- Can the authors provide detailed clarifications about the differences among baselines? (W1)
- Can the authors give the performance comparison under the same training dataset? (W2)
- Can the authors compare Dynamic KL with fixed mixtures of forward and reverse KL, and explain why top-k coverage is an appropriate indicator for adjusting the KL weights? (W3)
4 comments
Response to W3
W3-1&Q3-1: Does Lemma 1 also hold for fixed KL mixtures? How do fixed mixtures perform?
A3-1: Yes. The corrective-gradient property also holds for a fixed positive forward-KL weight. It shows that a positive forward-KL weight provides a corrective gradient for teacher-supported tokens that the student underestimates. Dynamic KL uses the precision-induced coverage gap to select this weight for each token.
We further compare token-level Dynamic KL with five fixed forward-KL weights from 0.1 to 0.9, retaining the same K/V loss. We have updated Section 6.4, Figure 4 and Appendix C.1 with this comparison. Table 6 gives the full dataset results for Qwen3.5-4B W2. The forward-KL weight is and reverse-KL weight is .
| KL mixture | SQA-IMG | PIQA | MMLU-R | IFEval | GSM8K | MATH | LCB | Avg. |
|---|---|---|---|---|---|---|---|---|
| Dynamic (OPQ) | 75.81 | 78.89 | 65.16 | 71.90 | 83.62 | 55.40 | 12.09 | 63.27 |
| 0.1 | 76.50 | 79.60 | 64.86 | 69.13 | 81.27 | 51.00 | 9.89 | 61.75 |
| 0.3 | 75.81 | 79.27 | 64.98 | 68.95 | 82.71 | 53.00 | 10.44 | 62.17 |
| 0.5 | 76.85 | 79.00 | 65.09 | 69.69 | 83.55 | 54.40 | 12.64 | 63.03 |
| 0.7 | 75.81 | 78.29 | 65.28 | 68.95 | 82.87 | 49.80 | 11.54 | 61.79 |
| 0.9 | 75.61 | 78.35 | 65.18 | 68.76 | 81.58 | 49.60 | 9.34 | 61.20 |
Dynamic KL achieves the highest average since it adjusts supervision using each token's precision discrepancy. Forward KL receives more weight when quantization loses teacher-supported alternatives, while reverse KL receives more weight when coverage is retained. Individual tasks favor different ratios. Besides, among all fixed KL ratios, the balanced mixture () performs best, likely because it balances the recovery of teacher-supported alternatives with the alignment of student-preferred predictions.
W3-2&Q3-2: Why use top-k coverage gap, including when the top- token sets are consistent?
A3-2: What the coverage gap measures. On the same student prefix, we compare the teacher probability mass on the top- tokens of the floating and quantized student views, retaining positive decreases. A larger gap indicates that quantization has removed more teacher-supported mass from the student’s leading predictions.
Why it is useful. We compare the coverage gap with the change in teacher-to-student KL after quantization, , where and are the quantized and floating views of the same student weights. Over 64 held-out prompts, the Pearson correlation between these two quantities is 0.73 initially and 0.55 after training (Figure 7). This positive association supports using the gap to adjust the KL mixture.
What changes during training. Figure 9 shows that larger gap groups have higher mean additional KL at both checkpoints. After training, the mean additional KL is lower in each group, and gaps concentrate closer to zero across the different .
When the top- token sets are consistent. The top- coverage gap is zero, so OPQ uses reverse KL ( calculated by Equation 4). It still supervises the full output distribution, including probability differences within and outside the leading set. Top- determines the mixture weight, while both KL losses use the full vocabulary.
We have updated Section 6.6 and Appendix C.2 with this explanation and the diagnostics.
A Kind Reminder for Reading the Response
Dear reviewer,
We have revised the paper and added further results to address your comments. As the discussion period is drawing to a close, we would greatly appreciate it if you could review our response above and let us know whether they address your concerns. We would be happy to discuss any remaining questions and provide further clarification.
Thank you for your time and consideration!
Best regards, The authors of OPQ
A Gentle Follow-up on the Final Day of Discussion
Dear Reviewer,
We apologize for following up once more and sincerely appreciate your time. As the discussion closes on November 18 at 11:59 PM AoE, we wanted to ask whether any key concerns remain unresolved.
Our response covers the distinction from RL-like QAT (Section 2), shared training data comparisons (Appendix F), and fixed-KL comparisons and coverage-gap diagnostics (Sections 6.4 and 6.6, Appendix C). We also clarify that Lemma 1 applies to fixed mixtures with positive forward-KL weights and explain full-vocabulary supervision when the top-k sets coincide.
Please let us know if further clarification would help. If these experiments and explanations resolve your concerns, we would be grateful if you could take them into account when revisiting your assessment and score.
Thank you again for your thoughtful feedback.
Best regards,
The authors of OPQ
Response to W1 & W2
Thank you for your careful review and thoughtful questions!
W1&Q1: How does OPQ differ from RL-like QAT methods?
A1: The main difference is the goal. RL-like QAT, like QeRL and QaRL, uses quantization to make reinforcement learning more efficient. OPQ aims to produce a compact low-bit model that preserves the general capabilities of the original model.
The training methods also differ. QeRL and QaRL use reward-based policy optimization. OPQ instead distills a teacher on the quantized student’s own trajectories. It uses Dynamic KL to adjust output supervision to precision discrepancies and K/V alignment to preserve attention representations.
ReasoningQAT also targets low-bit deployment, but focuses on reasoning through mixed-domain calibration and teacher-weighted supervised fine-tuning. OPQ uses student-generated trajectories and precision-aware supervision to preserve capabilities across the evaluated tasks.
W2&Q2: Why did the baselines use different datasets, and what happens with shared data?
A2: The main comparison follows the data preparation used for each baseline’s purpose.
Updated Section 6.2 and Table 1 provide the updated dataset-level results of BitDistiller.
We additionally evaluate data sensitivity with the same training prompts of different baselines. For Qwen3.5-4B, we replace each baseline’s original inputs with the same new training prompts as OPQ while retaining its quantizer and optimization procedure. We have updated the results in Appendix F and Figure 10.
Some baselines are sensitive to the choice of data, especially LLM-QAT and OmniQuant at W2. OPQ's training prompts improve BitDistiller at both precisions, with a larger gain at W2, while AWQ changes little. Despite this, across the evaluated baseline data recipes, OPQ trained achieves the highest average accuracy at both W2 and W3.