- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper attempts to answer the question of how does one really perform on-policy distillation when there could be scenarios where the teacher could potentially give higher mass (in terms of probability) to inherently incorrect responses (by the student). The paper tries to filter out candidate responses where this mismatch occurs and replaces them with other candidate responses.
Critical strengths
Here are some strengths of the paper:
-
The paper attempts to solve an interesting relevant problem: how does one really perform on-policy distillation when there could be scenarios where the teacher could potentially give higher mass (in terms of probability) to inherently incorrect responses (by the student).
-
Some figures in the paper (figure 3) provides a solid intuition of what the paper is attempting to do.
Critical weaknesses
Here are some weaknesses:
-
The paper is littered with multiple writing issues, that make reading this paper extremely difficult. For example:
- In equation 1, the term is never defined. (I know what the authors want to say but it's not defined until equation 3, which again does not have the same explicit notation).
- In equation 2, is not defined. (Again I know what that authors want to say, but this is not easy to digest in one reading)
- It is not clear why predictive compatibility is defined (just after equation 3).
- There are issues with references: in line numbers 146 and 148, there are question marks.
- In table 2 and section 5.3
correct replacementanderror replacementare referenced but they dont seem to be defined anywhere, and it's hard to understand what the authors want to convey. - The in table 1, is not absolute percentage change but its relative. This makes it appear as if the gains are much higher than they actually are. I suggest that the authors go back to absolute percent gains or explicitly call out that these numbers are relative gains.
-
The base model's performance (Qwen-3-1.7B-Base) reported by the authors in table 1, does not seem to match the performance in the Qwen 3 tech report (Table 19): ([1], link: https://arxiv.org/pdf/2505.09388). The results for AIME24 and AIME25 (in the report) are higher than the method proposed by the authors (and the base model performance reported by the authors is much lower). It's not clear why and is a cause for concern. This could be because the authors evaluated it on non-thinking mode or the evaluation lengths are different. However, this should be explicitly stated in the paper, and then should be reconciled with the setting in the tech report.
-
Some critical baselines are missed:
- RGOPD [2] is a paper that seems to exactly do what the authors are suggesting. There is no comparison to this work and hence the methods merits are unclear.
- From what I understand the authors do seem to have access to the ground truth (line 255). Then clear baselines would be to compare against SFT and/or GRPO.
-
The paper does not report any standard deviations. This is extremely concerning for the results on the datasets that the paper reports as some of them are about 30 problems (for example AIME). Hence it's not clear if the gains reported are actually statistically significant.
References
[1] Yang, An, et al. "Qwen3 technical report." arXiv preprint arXiv:2505.09388 (2025)
[2] Akhondzadeh, Mohammad Sadegh, et al. "Reward-Gated On-Policy Distillation." arXiv preprint arXiv:2607.04037 (2026).
Questions for authors
Here are some questions:
-
Please refer to weakness 2. With regards to this, can the authors explain why there is a discrepancy between performance of the base model reported by the authors and the performance stated in the tech report? Moreover, can it be made sure that instead of creating a method for a specific setting, is it possible to perform distillation in a setting that matches the initial performance that is stated in the tech report?
-
Please refer to weakness 3. With regards to this, can the authors compare against the baselines suggested?
-
Please refer to weakness 4. With regards to this, can the authors provide standard deviations?
-
In table 5, why is the training setting (in terms of temperature) different from evaluation?
Note: In its current form, due to the various issues in terms of writing as well as other weaknesses raised I am recommending rejection. However, if the authors could provide evidence addressing the weaknesses raised, Im happy to revisit my decision.
4 comments
W3: Clarification of Trajectory Replacement Operations
Thank you for identifying the ambiguity in the ablation terminology. We have revised Section 5.3 and Table 2 to consistently use the terms Correct-Rollout Replacement and Unreliable-Error Replacement, aligning the ablation descriptions with the corresponding mechanisms in the Method section.
Specifically, Correct-Rollout Replacement selects the fraction of correct rollouts with the highest teacher log-likelihood for replacement. Unreliable-Error Replacement targets incorrect rollouts whose teacher log-likelihood exceeds the mean teacher log-likelihood of correct responses to the same prompt, selecting the highest-likelihood candidates among them. We also clarify the ablation settings by separately disabling these two replacement operations to examine their individual contributions.
W4: Clarification of Relative Performance Changes
We thank the reviewer for catching this ambiguity. We have revised the corresponding label in Table 1 from $ \Delta $ vs. OPD to Relative change vs. OPD (%). This explicitly indicates that the reported values represent relative percentage changes rather than absolute percentage-point differences. The underlying experimental results remain unchanged.
W5: Clarification of Baseline Selection
We appreciate the reviewer’s suggestions regarding additional baselines. Our primary objective is to improve the training efficiency of OPD, rather than to establish a comparison between different post-training paradigms. SFT and GRPO represent alternative post-training approaches with different optimization objectives and training pipelines. Our work instead investigates how to allocate training updates more effectively within the OPD framework. Therefore, we use vanilla OPD under matched experimental settings as the primary baseline, allowing us to isolate the improvements brought by RTB. We do not claim that RTB outperforms SFT or GRPO.
Regarding RG-OPD [4], although it also incorporates verifier feedback and teacher information into OPD, its intervention mechanism differs from ours. RG-OPD uses a reward–teacher likelihood gate to determine whether newly sampled trajectories should contribute to the distillation loss, applying reverse-KL supervision only when the verifier reward and teacher–student likelihood gap are directionally consistent. In contrast, RTB focuses on trajectory allocation through selective replacement of current rollouts and compatibility-based admission and replay of historical mistakes. Thus, RG-OPD primarily addresses when teacher supervision should be applied, whereas RTB addresses which trajectories should receive optimization updates and how useful historical mistakes can be retained for future training. These mechanisms address different aspects of OPD and are complementary rather than directly interchangeable.
We will clarify the relationship between RG-OPD and RTB in the Related Work section.
W6: Citation Formatting
Thank you for identifying the citation issues. We have corrected the missing references and citation formatting in the revised manuscript and recompiled the PDF to ensure that all references are properly rendered.
Q4: Different Training and Evaluation Temperatures
We thank the reviewer for raising this question. The different temperature settings serve distinct purposes. During training, we set the student rollout and teacher temperatures to 1.0 to preserve sufficient diversity in the generated trajectories. This is particularly important for RTB, as diverse student-generated trajectories provide a broader pool of candidate errors for identifying and retaining teachable mistakes. During evaluation, we use a lower temperature of 0.7 with top-p = 0.95 to obtain more focused generations while retaining sampling diversity. All methods are evaluated using the same decoding configuration to ensure fair comparisons.
This setting is also consistent with recent OPD studies. For example, Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe [1], Tail-Aware Top-k On-Policy Distillation [2], and Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation [3] all adopt a training temperature of 1.0 and an evaluation temperature of 0.7 with top-p = 0.95. We will clarify the rationale for using different temperatures in the revised manuscript.
References
[1] Yaxuan Li et al. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. arXiv:2604.13016, 2026.
[2] Huipeng Huang and Hongxin Wei. Tail-Aware Top-k On-Policy Distillation. arXiv:2608.14728, 2026.
[3] Bingxi Hou et al. Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation. arXiv:2610.21601, 2026.
[4] Mohammad Sadegh Akhondzadeh et al. Reward-Gated On-Policy Distillation. arXiv:2607.04037, 2026.
I thank you for the comments. With regards to the updated version of the paper, I think it's hard to judge it without reading it. With regards to W2, I apologize, I meant to write table 20. I think table 20 of the same Qwen report shows much higher gains for Math 500 (im not sure why). This discrepancy needs to be checked. With regards to W5, I think the baseline listed is highly relevant, the mechanism is different but the intent in my opinion is similar and hence cannot be disregarded. Q3, was not addressed and for Q4, I think this needs to be tested to see what temperature combination really works. Considering all of this, I dont think my concerns have been sufficiently addressed and hence I maintain my score.
We thank the reviewer for the follow-up comments. We respond to your newly raised concerns concisely below.
-
Qwen3 technical report Table 20 discrepancy We have rechecked Table 19 and Table 20. Both report results on Qwen3 Instruct checkpoints (thinking / non-thinking), while our experiments use Qwen3 Base, the raw pretrained checkpoint without any post-training. This fundamental difference in model checkpoints accounts for the performance gap. We will clearly highlight this distinction in the revised manuscript.
-
Temperature mismatch between training and evaluation We have conducted the requested temperature ablation, with results as follows:
| Temperature | AIME24 | AIME25 | AMC23 | MATH-500 |
|---|---|---|---|---|
| 1.0 | 0.0854 | 0.0541 | 0.3750 | 0.6620 |
| 0.7 | 0.0896 | 0.0521 | 0.3781 | 0.6680 |
| 0.5 | 0.0896 | 0.0500 | 0.3796 | 0.6560 |
| 0.3 | 0.0833 | 0.0521 | 0.3656 | 0.6660 |
The performance remains stable across this temperature range, and no single temperature consistently outperforms others on all benchmarks. Our original setting (training T=1.0 for trajectory diversity, evaluation T=0.7 for consistent comparison) follows standard OPD conventions.
We thank the reviewer for the constructive comments. We have revised the manuscript to improve the clarity and consistency of the mathematical notation, methodological descriptions, and experimental presentation. We address the relevant comments below.
W1 : Mathematical Notation and Predictive Compatibility
Thank you for pointing out the notation issues and the unclear motivation for predictive compatibility. We have revised the Preliminaries section to explicitly define the mathematical symbols before their first use and maintain consistent notation throughout the formulation.
Specifically, we now define the student and teacher distributions, trajectory prefixes, and valid response lengths before introducing the distillation objective. We also explicitly define the truncated student and teacher distributions on the shared support before presenting the token-level reverse-KL loss. Furthermore, we clarify the distinction between $ k_{\mathrm{tr}} $ and $ k_{\mathrm{diag}} $, explaining their respective roles in the distillation objective and predictive compatibility measurement.
We have also clarified the motivation for predictive compatibility. Since the truncated reverse-KL objective is computed over the student's retained top-$ k $ support, higher overlap indicates that more of the teacher's high-probability token alternatives are represented within this support, making the teacher's local preferences more accessible to the distillation objective. Thus, predictive compatibility serves as a trajectory selection signal indicating the availability of locally exploitable teacher supervision, rather than a direct measure of trajectory quality or expected learning gains.
W2: Discrepancy with the Qwen3 Technical Report
We thank the reviewer for pointing out this discrepancy. The key difference is that our experiments use Qwen3-1.7B-Base and Qwen3-4B-Base as students, whereas Table 19 of the Qwen3 Technical Report reports results for the officially post-trained Qwen3-1.7B in the thinking mode. These checkpoints have different training histories and are therefore not directly comparable.
Our choice of Base checkpoints follows common practice in recent OPD studies. For example, Fast and Effective On-Policy Distillation from Reasoning Prefixes [1], Prune-OPD [2], and Entropy-Aware On-Policy Distillation of Language Models [3] also adopt pretrained Base models as students. This setting allows us to evaluate our method from a consistent pretrained initialization. Moreover, the official Qwen3 post-training pipeline already incorporates both off-policy and on-policy distillation for lightweight models. In particular, Qwen3-32B and Qwen3-235B-A22B are explicitly used as teachers in the on-policy distillation stage (Section 4.5 of the Qwen3 Technical Report). Starting from post-trained checkpoints would introduce prior distillation effects, making it harder to isolate the contribution of our method.
We will clarify the distinction between Base and post-trained checkpoints, as well as the evaluation settings, in the revised manuscript.
References
[1] Dongxu Zhang et al. Fast and Effective On-Policy Distillation from Reasoning Prefixes. Findings of ACL, 2026.
[2] Zhicheng Yang et al. Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning. arXiv:2605.07804, 2026.
[3] Woogyeol Jin et al. Entropy-Aware On-Policy Distillation of Language Models. arXiv:2603.07079, 2026.