acceptodds
Under review as a conference paper at ICLR 2027

OPQ: On-Policy Quantization for Low-Bit Large Language Models

Abstract

Low-bit quantization reduces the memory required to deploy large language models. However, learning on fixed sequences can miss the contexts reached by a quantized model's own predictions. We propose on-policy quantization (OPQ), a framework that learns directly on student-generated trajectories. On-policy quantization faces two main challenges: correcting output discrepancies introduced by low precision and maintaining consistent internal attention representations. To address the first, Dynamic Kullback–Leibler (KL) divergence adjusts output supervision using floating and quantized views of the student. To address the second, key/value (K/V) alignment supervises their attention memory. Our theoretical analysis explains the corrective gradients of the output objective and how K/V discrepancies affect attention outputs. With 2-bit and 3-bit weights, OPQ achieves the highest average task accuracy among the evaluated low-bit methods across two model families. Its gains are largest at 2-bit precision, reaching 9.49 percentage points over the strongest baseline. Ablations further show how student trajectories, Dynamic KL and K/V alignment contribute to capability preservation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Low-Bit Quantization, Post-Training Quantization, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oGLiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes On-Policy Quantization (OPQ), which is a framework for improving the performance of low-bit LLMs by training on student-generated trajectories. With the proposed dynamic KL and KV alignment, OPQ demonstrates promising performance over existing methods.

Critical strengths
  1. Training on student-generated trajectories is well-motivated because quantization errors will lead to distribution shifts.
  2. OPQ achieves promising results, especially at 2-bit which outperforms the strongest 9.49%.
  3. The combination of output distribution alignment and intermediate K/V representation alignment is reasonable. The paper also provides theoretical analysis to support its design.
Critical weaknesses
  1. RL-like QAT has been explored in prior works, such as QeRL[1], QaRL[2] and ReasoningQAT[3]. So it is necessary to summarize their differences.
  2. As discussed in appendix, the compared baselines use different training or calibration datasets. But the authors ignore to discuss whether this may lead to performance gap or not.
  3. The theoretical analysis in Lemma 1 demonstrates that incorporating forward KL provides non-vanishing corrective gradients for underrepresented tokens. However, this property also holds for a fixed mixture of forward and reverse KL. Table 3 only compares Dynamic KL against reverse KL but ignores to evaluate fixed KL mixtures. In addition, the proposed coefficient relies on differences in top-k teacher-probability coverage, which cannot capture distributional changes when the top-k token sets are consistent.

[1] Huang W, Ge Y, Yang S, et al. QeRL: Beyond efficiency-quantization-enhanced reinforcement learning for LLMs[C]//International Conference on Learning Representations. 2026, 2026: 99338-99361.

[2] Gu H, Wang H, Liu J, et al. Qarl: Rollout-aligned quantization-aware rl for fast and stable training under training–inference mismatch[C]//Findings of the Association for Computational Linguistics: ACL 2026. 2026: 17051-17064.

[3] Yasuyuki Okoshi, Hikari Otsuka, Daichi Fujiki, Masato Motomura, Towards Quantization-Aware Training for Ultra-Low-Bit Reasoning LLMs[C]// The Fourteenth International Conference on Learning Representations (ICLR 2026).

Questions for authors
  1. Can the authors provide detailed clarifications about the differences among baselines? (W1)
  2. Can the authors give the performance comparison under the same training dataset? (W2)
  3. Can the authors compare Dynamic KL with fixed mixtures of forward and reverse KL, and explain why top-k coverage is an appropriate indicator for adjusting the KL weights? (W3)
4 comments
AuthorsOpenReview ↗

Response to W1 & W2

Thank you for your careful review and thoughtful questions!

 

W1&Q1: How does OPQ differ from RL-like QAT methods?

A1: The main difference is the goal. RL-like QAT, like QeRL and QaRL, uses quantization to make reinforcement learning more efficient. OPQ aims to produce a compact low-bit model that preserves the general capabilities of the original model.

The training methods also differ. QeRL and QaRL use reward-based policy optimization. OPQ instead distills a teacher on the quantized student’s own trajectories. It uses Dynamic KL to adjust output supervision to precision discrepancies and K/V alignment to preserve attention representations.

ReasoningQAT also targets low-bit deployment, but focuses on reasoning through mixed-domain calibration and teacher-weighted supervised fine-tuning. OPQ uses student-generated trajectories and precision-aware supervision to preserve capabilities across the evaluated tasks.

 

W2&Q2: Why did the baselines use different datasets, and what happens with shared data?

A2: The main comparison follows the data preparation used for each baseline’s purpose.

  • PTQ: GPTQ, AWQ and OmniQuant retain their calibration setting, using 128 WikiText-103 sequences of 2,048 tokens. RTN requires no calibration data.
  • LLM-QAT: We retain its data-free generation recipe: the teacher generates fixed training sequences from randomly selected vocabulary tokens.
  • BitDistiller and OPQ: We update BitDistiller to use the same training prompts as OPQ, while retaining each method’s response generation, quantizer and loss.

Updated Section 6.2 and Table 1 provide the updated dataset-level results of BitDistiller.

 

We additionally evaluate data sensitivity with the same training prompts of different baselines. For Qwen3.5-4B, we replace each baseline’s original inputs with the same new training prompts as OPQ while retaining its quantizer and optimization procedure. We have updated the results in Appendix F and Figure 10.

MethodW3 OriginalW3 NewW2 OriginalW2 New
GPTQ68.0871.53—23.96
AWQ71.1871.1035.2435.65
OmniQuant70.4369.6514.6317.36
LLM-QAT66.8867.388.1817.15
BitDistiller70.2871.5155.5257.50
OPQ—73.60—63.27

Some baselines are sensitive to the choice of data, especially LLM-QAT and OmniQuant at W2. OPQ's training prompts improve BitDistiller at both precisions, with a larger gain at W2, while AWQ changes little. Despite this, across the evaluated baseline data recipes, OPQ trained achieves the highest average accuracy at both W2 and W3.

AuthorsOpenReview ↗

Response to W3

W3-1&Q3-1: Does Lemma 1 also hold for fixed KL mixtures? How do fixed mixtures perform?

A3-1: Yes. The corrective-gradient property also holds for a fixed positive forward-KL weight. It shows that a positive forward-KL weight provides a corrective gradient for teacher-supported tokens that the student underestimates. Dynamic KL uses the precision-induced coverage gap to select this weight for each token.

We further compare token-level Dynamic KL with five fixed forward-KL weights from 0.1 to 0.9, retaining the same K/V loss. We have updated Section 6.4, Figure 4 and Appendix C.1 with this comparison. Table 6 gives the full dataset results for Qwen3.5-4B W2. The forward-KL weight is α\alpha and reverse-KL weight is 1−α1−\alpha.

KL mixture α\alphaSQA-IMGPIQAMMLU-RIFEvalGSM8KMATHLCBAvg.
Dynamic (OPQ)75.8178.8965.1671.9083.6255.4012.0963.27
0.176.5079.6064.8669.1381.2751.009.8961.75
0.375.8179.2764.9868.9582.7153.0010.4462.17
0.576.8579.0065.0969.6983.5554.4012.6463.03
0.775.8178.2965.2868.9582.8749.8011.5461.79
0.975.6178.3565.1868.7681.5849.609.3461.20

Dynamic KL achieves the highest average since it adjusts supervision using each token's precision discrepancy. Forward KL receives more weight when quantization loses teacher-supported alternatives, while reverse KL receives more weight when coverage is retained. Individual tasks favor different ratios. Besides, among all fixed KL ratios, the balanced mixture (α=0.5\alpha=0.5) performs best, likely because it balances the recovery of teacher-supported alternatives with the alignment of student-preferred predictions.

 

W3-2&Q3-2: Why use top-k coverage gap, including when the top-kk token sets are consistent?

A3-2: What the coverage gap measures. On the same student prefix, we compare the teacher probability mass on the top-kk tokens of the floating and quantized student views, retaining positive decreases. A larger gap indicates that quantization has removed more teacher-supported mass from the student’s leading predictions.

Why it is useful. We compare the coverage gap with the change in teacher-to-student KL after quantization, KL(T∥Q)−KL(T∥F)\mathrm{KL}(T\Vert Q)-\mathrm{KL}(T\Vert F), where QQ and FF are the quantized and floating views of the same student weights. Over 64 held-out prompts, the Pearson correlation between these two quantities is 0.73 initially and 0.55 after training (Figure 7). This positive association supports using the gap to adjust the KL mixture.

What changes during training. Figure 9 shows that larger gap groups have higher mean additional KL at both checkpoints. After training, the mean additional KL is lower in each group, and gaps concentrate closer to zero across the different kk.

When the top-kk token sets are consistent. The top-kk coverage gap is zero, so OPQ uses reverse KL (α=0\alpha=0 calculated by Equation 4). It still supervises the full output distribution, including probability differences within and outside the leading set. Top-kk determines the mixture weight, while both KL losses use the full vocabulary.

We have updated Section 6.6 and Appendix C.2 with this explanation and the diagnostics.

AuthorsOpenReview ↗

A Kind Reminder for Reading the Response

Dear reviewer,

We have revised the paper and added further results to address your comments. As the discussion period is drawing to a close, we would greatly appreciate it if you could review our response above and let us know whether they address your concerns. We would be happy to discuss any remaining questions and provide further clarification.

Thank you for your time and consideration!

Best regards, The authors of OPQ

AuthorsOpenReview ↗

A Gentle Follow-up on the Final Day of Discussion

Dear Reviewer,

We apologize for following up once more and sincerely appreciate your time. As the discussion closes on November 18 at 11:59 PM AoE, we wanted to ask whether any key concerns remain unresolved.

Our response covers the distinction from RL-like QAT (Section 2), shared training data comparisons (Appendix F), and fixed-KL comparisons and coverage-gap diagnostics (Sections 6.4 and 6.6, Appendix C). We also clarify that Lemma 1 applies to fixed mixtures with positive forward-KL weights and explain full-vocabulary supervision when the top-k sets coincide.

Please let us know if further clarification would help. If these experiments and explanations resolve your concerns, we would be grateful if you could take them into account when revisiting your assessment and score.

Thank you again for your thoughtful feedback.

Best regards,
The authors of OPQ

Reviewer_p1h9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a framework of on-policy quantization (OPQ) to address the training-inference distribution mismatch caused by fixed teacher-generated trajectories. OPQ identifies two challenges: (1) the discrepancies between the teacher and student arise from both parameter updates and quantization, which a fixed KL cannot explicitly distinguish. (2) output matching does not directly constrain latent representations. Based on these two challenges, OPQ proposes a systematic framework of on-policy quantization including dynamic KL and key/value alignment. For dynamic KL, OPQ simultaneously considers forward KL and reverse KL. OPQ introduces a coefficient α\alpha which measures the loss of teacher probability mass in the student's top-k predictions after quantization. Using α\alpha, OPQ dynamically balances the contributions of forward KL and reverse KL. For K/V alignment, OPQ introduces a representation alignment loss to align the K/V representations between the quantized student and the floating view. Experiments demonstrate that on-policy quantization improves output fidelity and latent consistency. On two Qwen models with W3/W2 quantization settings, OPQ achieves the highest average accuracy on downstream tasks among the evaluated baselines. Ablation studies also demonstrate the effectiveness of the proposed dynamic KL and K/V alignment loss.

Critical strengths
  1. This paper identifies the training-inference distribution mismatch caused by fixed teacher-generated trajectories and proposes a framework of on-policy quantization. To address two challenges, this paper proposes a systematic framework of on-policy quantization including dynamic KL and key/value alignment.
  2. Experiments and ablation studies demonstrate that OPQ achieves the highest average accuracy on downstream tasks among the evaluated baselines. Ablation studies also demonstrate the effectiveness of the proposed dynamic KL and K/V alignment components.
Critical weaknesses
  1. OPQ proposes dynamic KL, which dynamically combines the forward KL and reverse KL during training. However, table 3 only demonstrates the effectiveness of dynamic KL, it does not demonstrate whether dynamic KL is better than a fixed ratio of forward KL and reverse KL.
  2. Although table 6 in appendix B reports the training time of OPQ with other QAT baselines, it is strongly recommended to report the end-to-end quantization time and inference efficiency results compared with baselines under comparable hardware settings, including QAT methods and PTQ methods.
Questions for authors
  1. Could the authors report the results under fixed ratios of forward KL and reverse KL? For example, the fixed ratios of forward KL and reverse KL could be 0.5/0.5 or 0.7/0.3, or 0.3/0.7.
  2. Could the authors report the end-to-end quantization time compared with baselines (e.g. BitDistiller) under comparable hardware settings? This would help us assess the quantization cost of OPQ.
  3. Could the authors report the inference efficiency of OPQ compared with BitDistiller or GPTQ? For example, under the same inference settings and environment, report the decoding time and peak GPU memory usage of OPQ and BitDistiller.
5 comments

Thank you for your careful review and concrete suggestions!

 

W1&Q1: How does Dynamic KL compare with fixed forward/reverse KL mixtures?

A1: We compare token-level Dynamic KL with five fixed forward-KL weights from 0.1 to 0.9, retaining the same K/V loss. We have updated Section 6.4, Figure 4 and Appendix C.1 with this comparison. Table 6 gives the full dataset results for Qwen3.5-4B W2. The forward-KL weight is α\alpha and reverse-KL weight is 1−α1−\alpha.

KL mixture α\alphaSQA-IMGPIQAMMLU-RIFEvalGSM8KMATHLCBAvg.
Dynamic (OPQ)75.8178.8965.1671.9083.6255.4012.0963.27
0.176.5079.6064.8669.1381.2751.009.8961.75
0.375.8179.2764.9868.9582.7153.0010.4462.17
0.576.8579.0065.0969.6983.5554.4012.6463.03
0.775.8178.2965.2868.9582.8749.8011.5461.79
0.975.6178.3565.1868.7681.5849.609.3461.20

Dynamic KL achieves the highest average since it adjusts supervision using each token's precision discrepancy. Forward KL receives more weight when quantization loses teacher-supported alternatives, while reverse KL receives more weight when coverage is retained. Individual tasks favor different ratios. Besides, among all fixed KL ratios, the balanced mixture (α=0.5\alpha=0.5) performs best, likely because it balances the recovery of teacher-supported alternatives with the alignment of student-preferred predictions.

 

W2&Q2: What is the quantization cost on comparable hardware?

A2: We compare quantization cost on the same host with eight A800 80GB GPUs. We have updated Appendix E and Table 9 with Qwen3.5-4B W2 results at group size 32. Total covers preparation, initialization, recovery and export.

MethodRecovery (h)Total (h)Allocated GPU·h
GPTQ0.090.100.10
OmniQuant0.550.570.57
LLM-QAT0.330.634.95
BitDistiller0.330.755.33
OPQ2.732.7922.24

PTQ methods are faster because they avoid training, but generally retain less downstream accuracy (Table 1). Among the distillation-based methods, BitDistiller is faster, while OPQ takes an additional 2.04 h in this configuration and completes quantization in under three hours. OPQ's quantization includes online generation and teacher, floating-student and quantized-student computations.

Note that OPQ currently uses synchronous training, and tensor and pipeline parallelism have not been tuned. Asynchronous training and further parallelism tuning offer opportunities to reduce runtime.

Besides, this is a one-time cost for OPQ accompanied by stronger downstream accuracy. OPQ outperforms baselines at both precisions on both main models. Once quantized, the model can be reused for subsequent inference, amortizing the upfront cost over deployment.

 

W2&Q3: What are the decoding time and peak GPU memory under identical inference settings?

A3: With the same model, quantization scope and precision, weight format, dequantization kernel and runtime, the quantization method does not add inference computation. OPQ and BitDistiller use the same GPTQ CUDA dequantization path and have closely matched inference costs.

We measure Qwen3.5-4B W2 on the same A800 host with TP4, FP16 activations, a 4 GiB K/V-cache budget per rank and 256 generated tokens. Columns show input tokens / batch size. Latency is averaged over five runs. Memory is the largest sampled NVML peak across ranks and runs. Bold marks the lowest value among quantized models. BF16 is the high-precision reference.

Decoding latency (ms/token)

Method512 / 1512 / 162048 / 12048 / 16
BF163.5114.1923.5094.935
GPTQ3.8255.1253.8265.919
BitDistiller3.2014.8213.2055.603
OPQ3.1974.7943.1955.559

Peak GPU memory (GiB/rank)

Method512 / 1512 / 162048 / 12048 / 16
BF167.698.528.528.68
GPTQ6.637.527.527.67
BitDistiller6.597.497.497.65
OPQ6.607.397.397.54

GPTQ has somewhat higher decoding latency, while OPQ and BitDistiller are close. The remaining differences may reflect implementation, scheduling and measurement variability. The full results are also reported in Appendix E and Table 10.

AuthorsOpenReview ↗

A Kind Reminder for Reading the Response

Dear reviewer,

We have revised the paper and added further results to address your comments. As the discussion period is drawing to a close, we would greatly appreciate it if you could review our response above and let us know whether they address your concerns. We would be happy to discuss any remaining questions and provide further clarification.

Thank you for your time and consideration!

Best regards, The authors of OPQ

AuthorsOpenReview ↗

A Gentle Follow-up on the Final Day of Discussion

Dear Reviewer,

We apologize for reaching out again and fully appreciate that your time is valuable. With discussion ending on November 18 at 11:59 PM AoE, we would welcome any remaining feedback on our response.

We have provided five fixed-KL comparisons (Section 6.4 and Appendix C.1) and end-to-end quantization time, decoding latency and peak GPU memory measurements on the same A800 host (Appendix E), addressing the requested ablation and efficiency comparisons.

We would be happy to clarify any unresolved questions promptly. If the additional evidence addresses your key concerns, we would sincerely appreciate your reconsideration of the assessment and score.

Thank you again for your careful review.

Best regards,
The authors of OPQ

Reviewer_4uAiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes On-Policy Quantization (OPQ), a framework that learns directly on student-generated trajectories. OPQ aims to address two main challenges in quantization: correcting output discrepancies introduced by low precision and maintaining consistent internal attention representations. To address these issues, OPQ utilizes Dynamic Kullback-Leibler divergence to adjust output supervision using floating and quantized views of the student, and uses key/value alignment to supervise the attention memory. Training updates the dense master weights through a straight-through estimator, and the final model is exported into packed integer weights. The teacher and floating reference are required only during training. Empirically, experiments evaluate Qwen3.5-4B and Qwen3-8B with 2-bit and 3-bit weights across knowledge, instruction following, mathematics, and code generation, showing strong empirical results.

Critical strengths
  1. This paper addresses an important training–inference mismatch in low-bit models. The proposed approach uses student trajectories to address the issue that training only on teacher-generated or fixed sequences can therefore miss relevant student states.
  2. Comparing the quantized and floating views of the same updated weights provides a way to examine the immediate effect of quantization separately from changes introduced by parameter adaptation. This is a sensible foundation for designing supervision that responds specifically to precision differences.
  3. The main results are consistently positive across the settings. OPQ achieves strong empirical results, with substantial improvements over baseline approaches.
  4. Detailed ablations support the effectiveness of the proposed approach.
  5. This paper is well written and easy to follow.
Critical weaknesses
  1. All the results were tested on Qwen 4B and 8B models. Generalization to larger models and different VLM family model is uncertain.
  2. The evidence for preserving visual capabilities is limited. The paper evaluates its visual capabilities using only ScienceQA image questions. This does not tell us whether the method preserves capabilities such as OCR, counting, spatial understanding, or detailed image description.
Questions for authors

N/A

5 comments

Thank you for your careful review and constructive suggestions!

W1-1: Does OPQ generalize to larger LLMs?

A1-1: We extend OPQ to Qwen3.8-27B with both 2-bit and 3-bit quantization. We have updated Section 6.5 and Appendix D.1 with the experiments. Figure 5 summarizes average accuracy and Table 7 gives the full dataset results below. Paired cells list W3 / W2 accuracy (%). Bold follows the paper's best quantized scores. BF16 is the high-precision reference.

Qwen3.8-27B — updated Table 7

MethodSQA-IMGPIQAMMLU-RIFEvalGSM8KMATHLCBAvg.
BF1691.0893.2084.9581.8996.5996.8031.8782.34
RTN86.76 / 54.5491.62 / 72.1483.54 / 41.7782.44 / 47.5096.21 / 80.4494.00 / 54.8018.68 / 6.0479.04 / 51.03
GPTQ89.84 / 58.3092.33 / 77.6482.39 / 65.1981.89 / 71.5395.91 / 90.9895.80 / 67.4026.37 / 8.2480.65 / 62.76
OmniQuant82.30 / 62.0292.27 / 82.9283.74 / 67.1979.67 / 58.0496.13 / 88.7895.20 / 72.8028.57 / 11.5479.70 / 63.33
BitDistiller90.48 / 85.5291.24 / 87.5482.98 / 71.9879.85 / 73.5794.92 / 94.6292.80 / 87.6037.91 / 29.1281.46 / 75.71
OPQ90.73 / 84.8892.49 / 91.3584.16 / 79.4681.52 / 77.8296.36 / 95.5395.00 / 87.4039.01 / 32.4282.75 / 78.41

Although a large LLM shows a better robustness against different low-bit quantization methods, OPQ gives a best choice for capability maintenance.

 

W1-2: Does OPQ generalize to other VLM families?

A1-2: We extend OPQ to InternVL3.5-4B with both 2-bit and 3-bit quantization. Specifically, we quantizes its language modules while keeping the visual components at high precision. We have updated Section 6.5 and Appendix D.2 with the experiments. Figure 6 summarizes average accuracy and Table 8 gives the full dataset results below. Paired cells list W3 / W2 accuracy (%).

InternVL3.5-4B — updated Table 8

MethodSQA-IMGPIQAMMLU-RIFEvalGSM8KMATHLCBAvg.
BF1677.9477.8065.0773.5791.8175.8020.3368.90
RTN63.81 / 6.9975.57 / 13.1159.05 / 8.9651.94 / 10.3578.85 / 1.0650.20 / 1.404.40 / 0.0054.83 / 5.98
GPTQ72.98 / 12.3977.48 / 19.7563.26 / 12.1468.02 / 29.7689.31 / 17.8266.40 / 4.6010.44 / 0.5563.98 / 13.86
OmniQuant68.17 / 34.4179.82 / 26.5056.33 / 23.9365.99 / 44.5587.79 / 42.1566.00 / 15.6013.19 / 1.1062.47 / 26.89
BitDistiller74.12 / 48.9372.80 / 32.1066.21 / 29.8968.02 / 51.5788.63 / 69.2966.60 / 36.4017.58 / 8.7964.85 / 39.57
OPQ74.57 / 49.8877.91 / 36.1367.09 / 31.3571.53 / 60.4488.40 / 73.0966.40 / 35.0020.33 / 9.8966.60 / 42.25

The capability maintenance extends to a different VLM family. Both distillation-based methods improve over the evaluated PTQ baselines, and OPQ achieves the strongest performance at both bit quantization.

 

W2: Does OPQ preserve visual capabilities beyond ScienceQA?

A2: We evaluate visual capabilities beyond ScienceQA using three complementary benchmarks (OCRBench, MME perception and CV-Bench). We have updated Appendix G and Table 11.

  • OCRBench tests text recognition and related tasks. We use the original 1,000 questions and report the official total score out of 1,000.
  • MME perception covers existence, counting, position, colour and OCR. The official perception score combines individual-answer accuracy and correctness on both questions in each pair, with a maximum of 2,000.
  • CV-Bench tests counting and spatial relations in 2D, together with depth and distance in 3D. We use 2,638 questions, including 1,438 2D and 1,200 3D items, and report the mean of its 2D and 3D scores.
MethodOCRBench /1000MME perception /2000CV-Bench-Count (%)CV-Bench-Relation (%)CV-Bench-Depth (%)CV-Bench-Distance (%)CV-Bench-Overall (%)
BF168821599.5669.0494.4694.3387.3385.35
RTN374158.9017.5113.0823.5021.0018.82
AWQ7381243.5059.7779.5466.3363.6766.46
BitDistiller8011574.3765.2390.3190.3383.8380.43
OPQ8241632.4965.8692.3192.1784.3382.70

OPQ obtains the highest OCRBench, MME perception and CV-Bench overall scores among the evaluated baselines. The CV-Bench breakdown also shows its gains over BitDistiller in counting, spatial relations, depth and distance.

AuthorsOpenReview ↗

A Kind Reminder for Reading the Response

Dear reviewer,

We have revised the paper and added further results to address your comments. As the discussion period is drawing to a close, we would greatly appreciate it if you could review our response above and let us know whether they address your concerns. We would be happy to discuss any remaining questions and provide further clarification.

Thank you for your time and consideration!

Best regards, The authors of OPQ

AuthorsOpenReview ↗

A Gentle Follow-up on the Final Day of Discussion

Dear Reviewer,

We apologize for following up once more and sincerely appreciate your time. Before the discussion closes on November 18 at 11:59 PM AoE, we wanted to check whether our response has addressed your key concerns.

Our additions cover generalization to Qwen3.8-27B and InternVL3.5-4B at both precisions (Section 6.5 and Appendix D), and visual capabilities beyond ScienceQA through OCRBench, MME perception and CV-Bench (Appendix G).

If any concerns remain, please let us know so we can respond promptly. If these additions resolve your concerns, we would be grateful if you could consider them when revisiting your assessment and score.

Thank you again for your time and consideration.

Best regards,
The authors of OPQ

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers

We sincerely thank all reviewers for their careful reviews and constructive suggestions. We appreciate the recognition of the training–inference mismatch in low-bit models and the value of student-generated trajectories. Your comments have helped us strengthen the evaluation and clarify OPQ.

We address individual comments separately. Below, we summarize the experiments and clarifications incorporated into the revised manuscript, with updated passages highlighted in orange.

  • Generalization across model scales and architectures: We extend OPQ to Qwen3.8-27B and InternVL3.5-4B at 2-bit and 3-bit precision. OPQ achieves the highest average accuracy among the evaluated quantized methods in all four additional settings. We have updated Section 6.5 and Appendix D, with full comparisons in Tables 7–8.
  • Visual capability beyond ScienceQA: We add OCRBench, MME perception and CV-Bench, covering text recognition, counting and spatial understanding. OPQ achieves the highest scores among the evaluated quantized models on all three benchmarks. We have updated Appendix G and Table 11, including official OCRBench/MME scores and the CV-Bench breakdown.
  • Evidence for token-level KL adjustment: We compare Dynamic KL with five fixed forward-KL weights from 0.1 to 0.9, keeping the same K/V loss. Dynamic KL achieves the highest average. We clarify that Lemma 1 also applies to fixed positive forward-KL weights. Coverage diagnostics show that larger top-k gaps are associated with larger increases in teacher-to-student KL after quantization. When the top-k sets coincide, OPQ uses full-vocabulary reverse KL. We have updated Sections 6.4 and 6.6, Figures 4 and 7, and Appendix C.
  • Comparisons using shared prompts: Among the baselines, BitDistiller's results are updated in Section 6.2 and Table 1, using OPQ's training data. Other baselines retain their original results to preserve their established data recipes: WikiText-103 calibration for GPTQ, AWQ and OmniQuant, and teacher generation from random vocabulary tokens for LLM-QAT. We additionally evaluate these baselines with OPQ's training data in Appendix F, retaining their quantizers and optimization procedures. Some baselines are sensitive to the choice of data, especially LLM-QAT and OmniQuant at W2. Despite this, OPQ still achieves the highest average at both precisions across the evaluated data recipes. Appendix A.2 explains the data preparation.
  • Inference cost and one-time quantization cost: For the same model, quantization scope, precision, weight format, kernel and runtime, the quantization method adds no inference computation. OPQ and BitDistiller share the GPTQ CUDA dequantization path and have closely matched latency and memory usage. On the same A800 host, total quantization takes 2.79 h for OPQ and 0.75 h for BitDistiller. OPQ includes online generation and teacher, floating-student and quantized-student computation. Asynchronous training and tensor/pipeline parallelism tuning offer opportunities to reduce this one-time cost, which is amortized over subsequent inference. We have updated Appendix E and Tables 9–10.
  • Distinction from RL-like QAT: QeRL and QaRL use quantization to accelerate reinforcement learning. OPQ preserves general capabilities for low-bit deployment through teacher supervision on student-generated responses. ReasoningQAT also targets low-bit deployment, but focuses on reasoning through mixed-domain calibration and teacher-weighted supervised fine-tuning. OPQ uses student-generated trajectories and precision-aware supervision to preserve capabilities. We have clarified these differences in Section 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.