acceptodds
Under review as a conference paper at ICLR 2027

Beyond Full Imitation: Direction-Guided Quantization-Aware On-Policy Distillation

Abstract

Microscaling floating-point (MXFP) formats have emerged as a promising low-precision representation for efficient large language model (LLM) training and inference, yet quantizing both weights and activations to 4 bits (also known as W4A4 quantization) can still severely degrade performance on complex reasoning tasks. While existing post-training quantization (PTQ) and quantization-aware distillation (QAD) methods struggle to fully recover BF16 performance under aggressive quantization, on-policy distillation (OPD) is naturally suited to this setting because it supervises the quantized student on its own generated trajectories and better matches the states encountered during inference. However, we find that full imitation of the BF16 teacher is unnecessary for effective recovery. Quantization can either weaken or strengthen preferences introduced by post-training, suggesting that not every deviation from the BF16 teacher should be corrected. Based on this insight, we introduce DQ-OPD, a Direction-guided Quantization-aware On-Policy Distillation method that selectively corrects preference-weakening deviations while preserving preference-strengthening ones. Across Qwen3-8B and Qwen3-30B-A3B under MXFP W4A4 quantization, DQ-OPD consistently outperforms all PTQ and QAD baselines. On Qwen3-30B-A3B, it narrows the average gap to BF16 to 0.71 points and even surpasses the BF16 teacher on AIME24. These results demonstrate that near-BF16 performance under W4A4 quantization can be achieved without fully imitating the BF16 teacher, highlighting the importance of deciding which quantization deviations to be corrected rather than applying teacher correction uniformly.

Primary area:
foundation or frontier models, including LLMs
Keywords:
quantization-aware distillation, on-policy distillation, MXFP4 quantization, large language models, low-bit quantization

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_BM7wOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DQ-OPD, a selective online distillation method for quantization recovery. By comparing the directions of post-training drift and quantization drift, it selectively applies teacher correction while retaining a regularization constraint toward the quantized initialization. Experiments on two Qwen3 models show that DQ-OPD improves performance after MXFP4 W4A4 quantization.

Critical strengths

1. Practical and well-motivated problem.
The paper asks an interesting question: should all quantization-induced deviations from the full-precision teacher be corrected? The selective recovery objective is well motivated and practically relevant.

2. Consistent empirical gains.
DQ-OPD improves over the strongest reported baseline on both Qwen3 models, with gains across multiple benchmarks rather than from a single task.

Critical weaknesses

1. The comparison with G-OPD is not fully controlled.
The reward scale and tuning procedure for G-OPD are not clearly specified. This makes it hard to tell how much of the gain comes from directional masking versus reward-scale choices.

2. The HMMT decoding experiment needs a stronger control.
A random-correction baseline with the same token-replacement rate would better isolate the effect of the directional rule. The reported 0.83-point gain over BF16 is also difficult to assess without uncertainty estimates.

Questions for authors

Clarification requests

  1. Could the authors clarify the λ\lambda settings and tuning procedure for G-OPD and DQ-OPD? A matched comparison under the same λ\lambda would help isolate the gain from directional masking.

  2. For HMMT decoding, could the authors add a random-correction control with the same replacement rate and report uncertainty estimates? This would better test whether the directional criterion selects more useful corrections.

Future work

Testing the method across more quantization settings and model families would further strengthen its generality.

2 comments

We thank Reviewer BM7w for the thoughtful review and for finding the problem practical and well motivated. We address each comment below.

[W1 & Q1] Matched-λ comparison with G-OPD

W1: The comparison with G-OPD is not fully controlled. The reward scale and tuning procedure for G-OPD are not clearly specified. This makes it hard to tell how much of the gain comes from directional masking versus reward-scale choices.

Q1: Could the authors clarify the λ settings and tuning procedure for G-OPD and DQ-OPD? A matched comparison under the same λ would help isolate the gain from directional masking.

Thank you for this question. We agree that a matched comparison is the most direct way to isolate the gain from directional masking, and we have added it during the rebuttal.

λ settings and tuning procedure. For DQ-OPD, λ\lambda is chosen from the sweep λ∈{1,1.25,2,4}\lambda \in \{1, 1.25, 2, 4\} in Table 5, which gives λ=1.25\lambda = 1.25 on Qwen3-8B and λ=4\lambda = 4 on Qwen3-30B-A3B. The main-table G-OPD uses default value λ=1.25\lambda = 1.25. All methods follow the same checkpoint selection protocol, reporting the checkpoint with the highest mean accuracy in evaluation of every 20 steps.

Matched-λ comparison. To make the comparison fully controlled, we additionally ran G-OPD over the same λ\lambda sweep on Qwen3-8B. The four-benchmark averages (AIME-24 / AIME-25 / GPQA / MMLU-Pro) are:

Methodλ\lambda = 1λ\lambda = 1.25λ\lambda = 2λ\lambda = 4
G-OPD61.1562.1160.0957.86
DQ-OPD63.1864.4163.3964.55
Δ+2.04+2.30+3.30+6.69

We find DQ-OPD is better at every matched λ, and even its weakest setting (63.18) is above the best G-OPD setting (62.11). And the main-table G-OPD already uses its best tested λ (1.25). Among the four values, λ=1.25\lambda = 1.25 turns out to be the best one for G-OPD, so Tables 1–2 compare DQ-OPD with the strongest G-OPD configuration we tested.

These results confirm gain comes from the mask rather than from λ. At λ=1\lambda = 1, G-OPD reduces to standard OPD and DQ-OPD differs from it only through the mask, yet DQ-OPD still improves the average by +2.04 points. Moreover, G-OPD degrades as λ\lambda grows, while DQ-OPD remains stable: without the mask, a larger λ\lambda also amplifies the teacher pull-back on preference-strengthening tokens, which DQ-OPD leaves uncorrected.

On Qwen3-30B-A3B, the matched comparisons at λ=1\lambda = 1 and 1.251.25 show the same ordering (+0.89 and +0.77 on the four-benchmark average). Due to our limited computational resources, we were unfortunately unable to train G-OPD with λ=2\lambda = 2 and 44 on the 30B model within the rebuttal period. Per-benchmark results are given in our response to Reviewer 3wkZ (W2/Q2). We hope this clarifies the role of λ\lambda in our comparison.

[W2 & Q2] Random-correction control and uncertainty for HMMT decoding

W2: The HMMT decoding experiment needs a stronger control. A random-correction baseline with the same token-replacement rate would better isolate the effect of the directional rule. The reported 0.83-point gain over BF16 is also difficult to assess without uncertainty estimates.

Q2: For HMMT decoding, could the authors add a random-correction control with the same replacement rate and report uncertainty estimates? This would better test whether the directional criterion selects more useful corrections.

We thank the reviewer for this excellent suggestion. For the HMMT decoding experiment, we added a random-correction control with a matched replacement rate and report std across difference trials.

On HMMT Nov. 2025 (30 problems × 4 repeats) with Qwen3-8B W4A4, direction-guided decoding replaces the W4A4 token with the BF16 teacher token at 4.76% of the decoding steps. The random control replaces tokens with the teacher token uniformly at random at almost the same rate (4.41%). Following the paper, accuracy is computed over all 120 (problem, repeat) samplings and reported as mean ± std over the 4 repeats:

MethodReplacement rateAccuracy (%)
W4A4–43.33 ± 4.71
Random replacement4.41%47.50 ± 3.19
Direction-guided4.76%53.33 ± 4.71
BF16 teacher–52.50 ± 4.19

Randomly injecting teacher tokens recovers only about 45% of the W4A4-to-BF16 gap (43.33 →\rightarrow 47.50). At the same replacement rate, direction-guided selection closes the full gap and outperforms random replacement by +5.83 points. This suggests that the directional criterion selects more useful corrections, further benefiting from injected teacher tokens. It is also consistent with the training-time ablation in Figure 5a, where correcting a random subset of tokens at the same rate performs clearly worse than correcting preference-weakening deviations. We will add the new controlled result to the revised paper.

[Future work] More quantization settings and model families

Future work: Testing the method across more quantization settings and model families would further strengthen its generality.

Thank you for this encouraging suggestion. During the rebuttal, we extended the evaluation in both directions:

Another quantization format (NVFP4). DQ-OPD improves over RTN by +8.54 / +6.25 / +1.87 points on AIME-24 / AIME-25 / GPQA, and its four-benchmark average matches the BF16 model (66.64 vs. 66.62). Under this milder format, it is also slightly better than Direct-OPD (66.64 vs. 66.20).

Another model family with SFT-only post-training (Olmo3-7B-Think-SFT). With the pretrained Olmo3-7B as πB\pi_B and Olmo-3-7B-Think-SFT as πT\pi_T, DQ-OPD fully recovers the BF16 teacher on the four-benchmark average (59.17 vs. 59.10). It outperforms RTN by +7.17 points and Direct-OPD by +2.94 points.

In addition, Appendix C.1 reports MXFP4 without MBS on both Qwen3 models. Detailed results are given in our responses to Reviewer 4jqd (Q1) and Reviewer 3wkZ (Q1), and we will include these experiments in the revised paper.

We hope these additional results address your concerns, and we will incorporate them into the revised version. Thank you again for your helpful suggestions.

Reviewer_4jqdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper recovers the accuracy of MXFP4 W4A4-quantized reasoning LLMs with on-policy distillation from the BF16 teacher. Its key idea is to compare, per token, the post-training shift with the quantization deviation. The teacher reward is applied only when quantization undoes a post-training preference, and is masked when quantization pushes further in the post-training direction.

Critical strengths

The figures, tables and writing are good.

Critical weaknesses
  1. The method is a modest modification of existing OPD objectives. The reward-scale view comes from G-OPD, and using a base and post-trained checkpoint pair to build the reward comes from Direct-OPD. The paper should state more directly how the mask differs from Direct-OPD's reward, for example by writing both rewards side by side.
  2. The evaluation is not sufficient. Evaluation covers one model family (Qwen3) and one format (MXFP4 with MBS).
Questions for authors

Does the method generalize beyond Qwen3 and MXFP4 with MBS? Can you evaluate additional model families and quantization formats?

2 comments

We thank Reviewer 4jqd for the constructive feedback and for the kind words on the figures, tables and writing. We address each comment below.

[W1] How the directional mask differs from Direct-OPD’s reward

W1: The method is a modest modification of existing OPD objectives. The reward-scale view comes from G-OPD, and using a base and post-trained checkpoint pair to build the reward comes from Direct-OPD. The paper should state more directly how the mask differs from Direct-OPD’s reward, for example by writing both rewards side by side.

Thank you for this helpful suggestion. We agree that DQ-OPD builds on the OPD family, including G-OPD and Direct-OPD, and that writing the objectives side by side is the clearest way to show how they differ. All three methods sample responses from the current W4A4 student, initialize it from πQ\pi_Q, and use the same KL regularizer toward πQ\pi_Q, computed on the sampled tokens. At each visited prefix st=(x,y^<t)s_t = (x, \hat{y}_{<t}) with sampled token yty_t, following the notation of Eqs. (3)–(4), let ΔPT(v∣st)=log⁡πT(v∣st)πB(v∣st)\Delta_{PT}(v \mid s_t) = \log\frac{\pi_T(v \mid s_t)}{\pi_B(v \mid s_t)} and ΔQ(v∣st)=log⁡πT(v∣st)πQ(v∣st)\Delta_Q(v \mid s_t) = \log\frac{\pi_T(v \mid s_t)}{\pi_Q(v \mid s_t)}:

MethodPer-token objectiveReward signalReward tokens
G-OPDλ R−KL\lambda\, R - \mathrm{KL}R=ΔQ(yt∣st)R = \Delta_Q(y_t \mid s_t)sampled token yty_t
Direct-OPDRD−λ KLR_D - \lambda\,\mathrm{KL}RD=∑v∈Stpˉt(v) ΔPT(v∣st)R_D = \sum_{v \in S_t} \bar{p}_t(v)\,\Delta_{PT}(v \mid s_t)top-16 tokens StS_t
DQ-OPDλ M R−KL\lambda\, M\, R - \mathrm{KL}R=ΔQ(yt∣st)R = \Delta_Q(y_t \mid s_t), with M=1[ΔPT(yt∣st) ΔQ(yt∣st)>0]M = \mathbb{1}[\Delta_{PT}(y_t \mid s_t)\,\Delta_Q(y_t \mid s_t) > 0]sampled token yty_t

Here πB\pi_B is the BF16 checkpoint before post-training. The KL term is identical across the three methods; in Direct-OPD. For Direct-OPD, St=TopK⁡16(πθ(⋅∣st))S_t = \operatorname{TopK}_{16}(\pi_\theta(\cdot \mid s_t)) denotes the student's top-16 next-token support at prefix sts_t, and pˉt(v)=πθ(v∣st)∑u∈Stπθ(u∣st)\bar{p}_t(v) = \frac{\pi_\theta(v \mid s_t)}{\sum_{u \in S_t} \pi_\theta(u \mid s_t)} is the renormalized student distribution over StS_t. Thus the main difference therefore lies in what the post-training signal ΔPT\Delta_{PT} is used for. Direct-OPD uses the post-training signal as the reward itself. And DQ-OPD uses the post-training signal only to select tokens, and its reward is the quantization gap.

These two designs target different goals. Direct-OPD is designed to transfer a post-training shift to a student that has not undergone it, as in its original weak-to-strong setting, and it is also very effective there. In quantization recovery, however, the student is the quantized post-trained teacher itself and already carries most of this shift. What needs to be corrected is the quantization-induced deviation on the tokens where it undoes post-training.

We would also like to note that in settings other than performance recovery, using the post-training signal directly as the reward may well be the better choice. We will add the side-by-side formulation and this discussion to Section 4 of the revised paper. Thank you again for this suggestion, which we believe makes our contribution clearer.

[W2 & Q1] Additional model families and quantization formats

W2: The evaluation is not sufficient. Evaluation covers one model family (Qwen3) and one format (MXFP4 with MBS).

Q1: Does the method generalize beyond Qwen3 and MXFP4 with MBS? Can you evaluate additional model families and quantization formats?

Thank you for this suggestion. We extended the evaluation along both dimensions.

MethodAIME-24AIME-25GPQAMMLU-ProAvg.Recovery (%)
BF1670.6354.3749.2962.1359.10100.00
W4A4 RTN (MXFP4 + MBS128)62.0844.7943.6957.4552.0087.98
Direct-OPD62.5055.2147.3259.8756.2395.13
DQ-OPD67.7160.0047.6361.3459.17100.11

We applied DQ-OPD on SFT-only model Olmo-3-7B-Think-SFT [1] (Olmo-3-7B as base model), and it improves over RTN by +5.63 on AIME-24, +15.21 on AIME-25 and +3.94 on GPQA, and it even exceeds the BF16 teacher on AIME-25 (60.00 vs. 54.37). It also outperforms Direct-OPD by 2.94 points on average.

Besides MXFP4 with MBS, Appendix C.1 (Tables 3–4) already reports MXFP4 without MBS. There, DQ-OPD again achieves the best average on both models (61.75 vs. 59.64 for the strongest baseline on Qwen3-8B; 68.81 vs. 67.53 on Qwen3-30B-A3B). We have now also trained DQ-OPD and Direct-OPD on Qwen3-8B under NVFP4 W4A4 setting. For each method, the checkpoint is selected by the same rule, i.e., best accuracy mean in 200 steps and evaluation every 40 steps due to time limitation. Avg. and Recovery are computed over the four benchmarks shown:

MethodAIME-24AIME-25GPQAMMLU-ProAvg.Recovery (%)
BF1671.4663.1258.1373.7766.62100.00
RTN (NVFP4)65.0058.5454.8571.8562.5693.91
Direct-OPD72.0864.7957.1770.7766.2099.37
DQ-OPD73.5464.7956.7271.5166.64100.03

Compared with RTN under NVFP4, DQ-OPD improves AIME-24 by +8.54, AIME-25 by +6.25 and GPQA by +1.87 points. Its four-benchmark average matches the BF16 model (66.64 vs. 66.62), slightly better than Direct-OPD. NVFP4 here is a much milder format. RTN loses only 4.58 AIME-25 points relative to BF16 under NVFP4, versus 15.83 under MXFP4, which leaves less room for selective correction to make a difference. We believe Under the more aggressive MXFP4 format, the advantage of the directional mask is clearer, as shown in Tables 1–2 and on Olmo3.

We hope these new results and clarifications address your concerns. If so, we would be grateful if you would consider updating your score. We would be happy to provide any further details.


Reference

[1] Olmo T, Ettinger A, Bertsch A, et al. Olmo 3[J]. arXiv preprint arXiv:2512.13961, 2025.

Reviewer_3wkZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies performance recovery for language models under MXFP4 W4A4 quantization. It proposes DQ-OPD, which uses the comparison between a base model and a post-trained teacher to provide directional guidance, while teacher-derived rewards selectively supervise the quantized student. The method aims to correct quantization changes that weaken post-training preferences without requiring full teacher imitation. Experiments on Qwen3-8B and Qwen3-30B-A3B show improved average performance across seven benchmarks compared with the evaluated baselines, with a 0.71-point gap to BF16 model.

Critical strengths

1.The paper introduces a concrete way to use the base-to-teacher probability shift when deciding which quantization deviations to correct. This is an interesting extension of on-policy distillation because it makes supervision selective rather than uniformly encouraging teacher imitation.

2.The evaluation covers both dense and MoE models across seven benchmarks and includes comparisons with standard OPD, G-OPD, and Direct-OPD. The directional, regularization, and reward-scale ablations provide useful support for the design. In particular, DQ-OPD also improves over standard OPD at unit reward scale, providing evidence that the selection rule contributes beyond reward scaling alone.

Critical weaknesses
  1. The evaluation is limited to Qwen3-8B and Qwen3-30B-A3B under MXFP4 W4A4 quantization. The official Qwen3 report describes these models as primarily post-trained through distillation from stronger models. Since DQ-OPD directly relies on the base-to-teacher probability shift, its applicability to other model families and post-training procedures remains unclear. Evidence from another family or a controlled SFT-only or RL setting would strengthen the generality of the findings.

2.Although the paper reports shared training settings, it is unclear whether G-OPD and DQ-OPD received comparable hyperparameter tuning and how their final checkpoints were selected. Figure 5c shows a reward-scale sweep for DQ-OPD but not G-OPD. Clarifying these choices would help establish whether the reported advantage reflects the proposed directional guidance under comparable tuning and selection procedures.

Questions for authors
  1. Do the authors have evidence that the directional criterion remains effective beyond the two evaluated Qwen3 models,in another model family and under a different post-training procedure, such as SFT-only or direct RL with task-based rewards?

  2. Could the authors confirm whether the G-OPD baseline in Tables 1–2 uses the same reference policy, reward scale, and other training settings as DQ-OPD, differing only in the directional mask? This would clarify whether the comparison directly isolates the mask’s contribution.

3 comments

We thank Reviewer 3wkZ for the careful and constructive review, and for finding the use of the base-to-teacher shift an interesting extension of on-policy distillation. We address each comment below.

[W1 & Q1] Generality beyond Qwen3 and other post-training procedures

W1: The evaluation is limited to Qwen3-8B and Qwen3-30B-A3B under MXFP4 W4A4 quantization. The official Qwen3 report describes these models as primarily post-trained through distillation from stronger models. Since DQ-OPD directly relies on the base-to-teacher probability shift, its applicability to other model families and post-training procedures remains unclear. Evidence from another family or a controlled SFT-only or RL setting would strengthen the generality of the findings.

Q1: Do the authors have evidence that the directional criterion remains effective beyond the two evaluated Qwen3 models, in another model family and under a different post-training procedure, such as SFT-only or direct RL with task-based rewards?

We thank the reviewer for this important question. To test whether the directional criterion depends on Qwen3's distillation-based post-training, we applied DQ-OPD to the Olmo3-7B [1] family, which has a fully open post-training pipeline. Following the reviewer's suggestion, we consider the SFT-only setting. We use the pretrained Olmo3-7B base model as πB\pi_B and Olmo-3-7B-Think-SFT as the teacher πT\pi_T. The teacher is obtained from the base model by supervised fine-tuning only, without any preference optimization or RL. The base-to-teacher shift used by DQ-OPD therefore comes purely from SFT, in a different model family from Qwen3.

We select the best checkpoint per method in 200 steps training.

MethodAIME-24AIME-25GPQAMMLU-ProAvg.Recovery (%)
BF1670.6354.3749.2962.1359.10100.00
W4A4 RTN62.0844.7943.6957.4552.0087.98
Direct-OPD62.5055.2147.3259.8756.2395.13
DQ-OPD67.7160.0047.6361.3459.17100.11

DQ-OPD is consistently better than RTN and Direct-OPD, and it is ahead by +7.17 and +2.94 respectively on average. It restored the performance of BF16 SFT teacher and even slightly exceeds the BF16 SFT teacher on average (59.17 vs. 59.10).

These results suggest that the directional criterion generalizes beyond Qwen3 and its distillation-based post-training, remaining effective in a different model family whose post-training shift comes from SFT alone. Within the rebuttal time limit, we were unfortunately unable to also study a direct-RL setting; we agree that it is an interesting direction and plan to pursue this line of experiments in out future work. We thank the reviewer again for this suggestion, which we believe strengthens the paper considerably.

[W2 & Q2] Comparable tuning and checkpoint selection for G-OPD

W2: Although the paper reports shared training settings, it is unclear whether G-OPD and DQ-OPD received comparable hyperparameter tuning and how their final checkpoints were selected. Figure 5c shows a reward-scale sweep for DQ-OPD but not G-OPD. Clarifying these choices would help establish whether the reported advantage reflects the proposed directional guidance under comparable tuning and selection procedures.

Q2: Could the authors confirm whether the G-OPD baseline in Tables 1–2 uses the same reference policy, reward scale, and other training settings as DQ-OPD, differing only in the directional mask? This would clarify whether the comparison directly isolates the mask’s contribution.

Thank you for raising this important point. We agree that a controlled comparison is essential to isolate the contribution of the directional mask, and we address the three aspects in turn.

Same reference policy, reward, and settings. We confirm that in our controlled comparisons, G-OPD and DQ-OPD share the exact same formulation and training setup, differing only in the directional mask. Specifically, both methods use the identical per-token reward, reference policy, and regularizer, with the directional gate MtM_t on the reward being the sole difference:

G-OPD: A^t=λlog⁡πT(yt∣st)πQ(yt∣st)−log⁡πθ(yt∣st)πQ(yt∣st),DQ-OPD: A^t=λ Mtlog⁡πT(yt∣st)πQ(yt∣st)−log⁡πθ(yt∣st)πQ(yt∣st),\text{G-OPD: } \hat A_t = \lambda \log\frac{\pi_T(y_t|s_t)}{\pi_Q(y_t|s_t)} - \log\frac{\pi_\theta(y_t|s_t)}{\pi_Q(y_t|s_t)}, \qquad \text{DQ-OPD: } \hat A_t = \lambda\, M_t \log\frac{\pi_T(y_t|s_t)}{\pi_Q(y_t|s_t)} - \log\frac{\pi_\theta(y_t|s_t)}{\pi_Q(y_t|s_t)},

with Mt=1[ΔPT(yt∣st) ΔQ(yt∣st)>0]M_t = \mathbb{1}[\Delta_{PT}(y_t|s_t)\,\Delta_Q(y_t|s_t) > 0]. In both methods, the reference policy is πQ\pi_Q, i.e., the quantized BF16 teacher that also initializes the student, and setting Mt≡1M_t \equiv 1 recovers G-OPD exactly (Eq. (26), Appendix A.4). The training data, batch size, rollout configuration, learning rate, and number of steps are also identical (Appendix D). Regarding the reward scale λ\lambda, both G-OPD and DQ-OPD use λ=1.25\lambda = 1.25 on Qwen3-8B in the main tables. On Qwen3-30B-A3B, the main tables report G-OPD with λ=1.25\lambda = 1.25 and DQ-OPD with λ=4\lambda = 4, and Table 5 also reports DQ-OPD under the matched scale λ=1.25\lambda = 1.25. Under both λ=1.25\lambda = 1.25 and λ=4\lambda = 4, DQ-OPD achieves substantial improvements over G-OPD on almost all tasks, confirming the consistent benefit of the directional mask.

Checkpoint selection. All methods in Tables 1–2 follow the same protocol: we evaluate every 20 training steps and report the checkpoint with the highest mean accuracy. Since the same protocol is applied to every method, we believe the comparison remains fair. In addition, Figure 3 shows that DQ-OPD is ahead of OPD, G-OPD and Direct-OPD at most training steps, which suggests that the conclusion does not depend on the particular checkpoint chosen. Due to limited time, the new G-OPD runs below (λ=2,4\lambda = 2, 4) are reported based on Qwen3-8B.

Matched-λ sweep for G-OPD. We thank the reviewer for this suggestion. Following it, we trained G-OPD with the same reward scales as the DQ-OPD sweep in Table 5. Standard OPD is G-OPD with λ=1\lambda = 1, and λ=1.25\lambda = 1.25 is the main-table G-OPD. Avg. is the mean over the four benchmarks shown:

Methodλ\lambdaAIME-24AIME-25GPQAMMLU-ProAvg.
G-OPD164.5857.0854.1968.7461.15
G-OPD1.2567.0858.7554.6567.9562.11
G-OPD265.6255.0054.3465.3860.09
G-OPD460.6351.0453.7965.9757.86
DQ-OPD168.9659.3854.2470.1563.18
DQ-OPD1.2570.8360.6356.3669.8264.41
DQ-OPD267.7161.4655.5668.8263.39
DQ-OPD472.2965.8353.0367.0564.55

We found DQ-OPD is better at every matched λ, by +2.04, +2.30, +3.30 and +6.69 points at λ\lambda = 1, 1.25, 2 and 4. Even the weakest DQ-OPD setting (63.18) is above the best G-OPD setting (62.11). Among the four reward scales, λ=1.25\lambda = 1.25 turns out to be the best one for G-OPD, so Tables 1–2 already compare DQ-OPD with the strongest G-OPD configuration we tested.

On Qwen3-30B-A3B, the matched comparisons available at λ=1\lambda = 1 and 1.251.25 show the same ordering (+0.89 and +0.77 on the four-benchmark average). Due to our limited computational resources, we were unfortunately unable to train G-OPD with λ=2\lambda = 2 and 44 on the 30B model within the rebuttal period, and we apologize for this. We hope that the complete sweep on Qwen3-8B, together with the matched comparisons on the 30B model, addresses the reviewer's concern.

We hope these new results and clarifications address your concerns. If so, we would be grateful if you would consider updating your score. We would be happy to provide any further details.


Reference

[1] Olmo, Team, et al. "Olmo 3." arXiv preprint arXiv:2512.13961 (2025).

Thanks for the additional experiments. While they address some of my concerns, I still have reservations about generalization across different post-training procedures and the relatively small gains under matched settings on the larger model. I will keep my rating at 2

Reviewer_dt4cOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Direction-guided Quantization-aware On-Policy Distillation (DQ-OPD) to recover the performance degradation caused by aggressive MXFP4 W4A4 quantization. The key insight is that quantization-induced deviations are not always harmful: some weaken post-training preferences and others strengthen them. Based on this observation, DQ-OPD selectively corrects preference-weakening deviations.

The authors conduct extensive experiments on both dense and MoE models across seven benchmarks, demonstrating consistent improvements over the evaluated PTQ and distillation baselines. Although further comparisons with stronger MXFP4 PTQ methods, QAD using BF16 teacher-generated responses, and a detailed training cost analysis would strengthen the evaluation, I believe this paper presents a meaningful insight.

Critical strengths
  1. The broad scope of experiments. The author implemented post-training experiments under both dense and MOE models. Also convincing benchmarks are used to better prove the useful of DQ-OPD.
Critical weaknesses
  1. Deeper explanations are needed to explain the idea of “weakening and strengthening”.
  2. There are also other MXFP4 PTQ methods that seems to achieve good performance, such as WUSH.
Questions for authors
  1. The orientation of DQ-OPD is more likely to acceleration inference like OPD through QAT style. The training setting is 256 samples per update with 2048 prompt len and 8192 generated tokens. What about the DQ-OPD’s advantage over QAD under increased samples? What about the advantage when QAD also use response from bf16 teacher rather than from dataset?
  2. Under the same experimental setting (hardware and training configurations), how much extra GPU-hours does DQ-OPD pay compared to PTQ, OPD and QAD methods?
  3. Explain deeper about the insight behind the idea of weakening and strengthening effects rather than prove them through final accuracy results.
2 comments

We sincerely thank Reviewer dt4c for the positive assessment and for recognizing the breadth of our experiments. Your suggestions led us to several new experiments, which we summarize below. All new results were obtained during the rebuttal period with the same evaluation protocol as in the paper.

[W1 & Q3] Deeper insight into preference weakening vs. strengthening

W1: Deeper explanations are needed to explain the idea of “weakening and strengthening”.

Q3: Explain deeper about the insight behind the idea of weakening and strengthening effects rather than prove them through final accuracy results.

We thank the reviewer for this constructive suggestion. We can understand why distinguishing preference weakening and strengthening works by looking at how quantization perturbations interact with post-training preferences during on-policy optimization.

Where Weakening and Strengthening Come From

During post-training, the model evolves from the base policy πB\pi_B to the teacher πT\pi_T by reshaping token preferences: it increases the probability of tokens that support valid reasoning (ΔPT(a∣st)=log⁡πT(a∣st)πB(a∣st)>0\Delta_{\text{PT}}(a|s_t) = \log \frac{\pi_T(a|s_t)}{\pi_B(a|s_t)} > 0) and suppresses tokens that lead to errors or distractions (ΔPT(a∣st)<0\Delta_{\text{PT}}(a|s_t) < 0). What matters for downstream reasoning is preserving this learned preference structure, rather than matching the exact floating-point probabilities of πT\pi_T.

When πT\pi_T is quantized to W4A4 (πQ\pi_Q) via Round-to-Nearest (RTN) quantization, nearest-neighbor rounding acts as an unbiased, symmetric perturbation on weights and activations. Consequently, as errors propagate to token probabilities, quantization does not shift outputs in a single direction. Across on-policy rollout trajectories, we observe that for some tokens, the quantized model pushes token probabilities against the post-training shift (preference weakening, ΔPTΔQ>0\Delta_{\text{PT}}\Delta_{\text{Q}} > 0), while for others, it pushes them further along the post-training shift (preference strengthening, ΔPTΔQ<0\Delta_{\text{PT}}\Delta_{\text{Q}} < 0).

Why Full Imitation is Counterproductive and Selective Correction Works

Standard distillation (QAD and OPD) treats every discrepancy from πT\pi_T identically via the implicit reward ΔQ(a∣st)=log⁡πT(a∣st)πQ(a∣st)\Delta_{\text{Q}}(a|s_t) = \log \frac{\pi_T(a|s_t)}{\pi_Q(a|s_t)}. However, this teacher signal has opposite effects on the two types of deviations:

  • On preference-weakening tokens (ΔPTΔQ>0\Delta_{\text{PT}}\Delta_{\text{Q}} > 0), teacher correction repairs damaged preferences: Here, quantization has undone part of the post-training shift—either lowering the probability of a token promoted by post-training (πQ<πT\pi_Q < \pi_T with πT>πB\pi_T > \pi_B) or inflating a token suppressed by post-training (πQ>πT\pi_Q > \pi_T with πT<πB\pi_T < \pi_B). Applying the teacher reward ΔQ\Delta_{\text{Q}} pushes the quantized policy back in the post-training direction (Figure 2a), directly restoring the compromised preference.

  • On preference-strengthening tokens (ΔPTΔQ<0\Delta_{\text{PT}}\Delta_{\text{Q}} < 0), full imitation triggers a counterproductive "Teacher Pull-Back": Here, quantization happens to move the token probability further along the post-training direction—either assigning an even higher probability to a post-training-promoted token (πB<πT<πQ\pi_B < \pi_T < \pi_Q) or further suppressing a disfavored token (πQ<πT<πB\pi_Q < \pi_T < \pi_B). Under full imitation, the OPD reward ΔQ\Delta_{\text{Q}} opposes ΔPT\Delta_{\text{PT}}: it assigns a negative reward (ΔQ<0\Delta_{\text{Q}} < 0) that actively penalizes a desirable, post-training-promoted token just to pull its probability down to πT\pi_T, or assigns a positive reward (ΔQ>0\Delta_{\text{Q}} > 0) that boosts a token post-training tried to suppress (Figure 2b).

By masking out the teacher pull-back on strengthening tokens (Mt=0M_t = 0) while keeping the KL regularization toward πQ\pi_Q (Eq. 8–11), DQ-OPD anchors strengthening tokens stably near πQ(yt∣st)\pi_Q(y_t|s_t) (Eq. 9) and concentrates parameter updates entirely on genuine preference-weakening errors.

We will revise Section 3 to make this intuitive explanation and token-level analysis much clearer.

[W2] Comparison with other MXFP4 PTQ methods (WUSH)

W2: There are also other MXFP4 PTQ methods that seems to achieve good performance, such as WUSH.

Thank you for pointing us to WUSH, which is indeed a highly relevant baseline. We have evaluated it on Qwen3-8B under exactly the same MXFP4 W4A4 setting (with MBS) and evaluation protocol as in the paper. We used the official implementation, with either RTN or GPTQ as the quantizer after the WUSH transform. Avg. and Recovery are computed over the four benchmarks shown:

MethodMMLU-ProAIME24AIME25GPQAAvg.Recovery (%)
BF1673.7771.4663.1258.1366.62100.00
RTN69.9457.0847.2952.1256.6184.97
MR-GPTQ70.2064.1753.5454.2060.5390.85
WUSH + RTN71.4367.0852.2954.7061.3892.13
WUSH + GPTQ71.0964.7953.1254.7560.9491.47
DQ-OPD (ours)69.8270.8360.6356.3664.4196.68

We will add WUSH to the PTQ comparison and discuss it in the related work. Thank you again for the helpful pointer.


[Q1] Advantage over QAD with more samples and BF16-teacher responses

Q1: The orientation of DQ-OPD is more likely to acceleration inference like OPD through QAT style. The training setting is 256 samples per update with 2048 prompt len and 8192 generated tokens. What about the DQ-OPD’s advantage over QAD under increased samples? What about the advantage when QAD also use response from bf16 teacher rather than from dataset?

Thank you for suggesting these comparisons. We additionally evaluated QAD using responses generated by the BF16 teacher instead of the original OpenThoughts3 responses.All settings follow the paper (256 prompts per update, 2,048 / 8,192 prompt / response tokens, learning rate 1e-5, 200 steps) and the results are summarized below:

MethodAIME-24AIME-25GPQAMMLU-ProAvg.Recovery (%)
BF1671.4663.1258.1373.7766.62100.00
RTN57.0847.2952.1269.9456.6184.97
QAD (OpenThoughts3)64.3854.1754.3470.3760.8191.29
QAD (BF16-teacher)67.2957.2955.3569.7662.4293.70
OPD64.5857.0854.1968.7461.1591.79
DQ-OPD70.8360.6356.3669.8264.4196.68

Using BF16-teacher responses improves QAD’s average score from 60.81 to 62.42, providing a stronger baseline for comparison with DQ-OPD. Against this strengthened baseline, DQ-OPD achieves 64.41, outperforming QAD by 1.99 points and raising recovery from 93.70% to 96.68%, with higher accuracy across all four benchmarks. These results show that DQ-OPD’s advantage under the same training setting persists when QAD uses BF16-teacher-generated responses. We will incorporate this QAD comparison into the revised manuscript. Regarding the comparison with more training samples, due to time and computational resource constraints, we cannot complete these experiments within the rebuttal period and plan to conduct them in future work.


[Q2] Training cost (GPU-hours)

Q2: Under the same experimental setting (hardware and training configurations), how much extra GPU-hours does DQ-OPD pay compared to PTQ, OPD and QAD methods?

Thank you for raising this practical question. We measured the computational cost of Qwen3-8B using the same GPU hardware and the training configuration in Appendix D: 200 updates with 256 prompts per update. All methods use simulated quantization on the same GPU hardware, which has no native FP4 support. And OPD-based methods use synchronous rollout generation and policy updates, following the official implementations of Direct-OPD and G-OPD, without asynchronous overlap.

Method#GPUsTime per step (s)Total GPU-hours (h)
PTQ: RTN (per checkpoint)1~180~0.05
PTQ: WUSH + GPTQ (calibration)1~349~1.9
QAD8~378~168
Direct-OPD8~772~343
DQ-OPD8~777~346

On-policy methods cost about 2× QAD. In on-policy methods, generating rollouts with the student at every step dominates in the extra cost and the directional mask itself is almost free (0.65% overhead compared to Direct-OPD). PTQ incurs a much lower one-time quantization cost but also larger accuracy gap. We hope these additional results and explanations address your concerns, and we will incorporate them into the revised version. Thank you again for your constructive feedback; we would be glad to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.