- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes Distilled RL, which incorporates teacher supervision into GRPO through token-level weighting rather than a separate distillation loss. Teacher-to-student probability ratios are clipped and geometrically normalized within each response. These weights modify positive-advantage updates, while nonpositive-advantage responses retain the ordinary GRPO update. The method is evaluated using mathematical post-training across three student models, with additional knowledge and coding evaluations. The reported results show consistent improvements in average mathematical performance over the evaluated baselines, and ablations examine negative-sample reset and geometric normalization.
Critical strengths
A simple and interesting coupling of teacher guidance with outcome supervision. The outcome advantage determines the direction of the update, while the teacher adjusts relative token weights on positive-advantage responses. Combining this separation with within-response log-ratio centering is a reasonable design: it allows teacher guidance without requiring unconditional distribution matching. The resulting modification is straightforward to incorporate into an existing GRPO implementation.
Consistent average improvements with useful component ablations. Distilled RL improves average mathematical performance across all three student models. The ablations establish that negative reset is important to avoid degradation and that geometric normalization provides additional gains on both evaluated students. These experiments provide useful evidence for the complete recipe, although they do not fully identify the mechanism responsible for the improvement.
Critical weaknesses
The distinct contribution relative to existing gated distillation methods is insufficiently established. Removing negative reset causes the largest ablation drop and makes the method worse than ordinary GRPO. However, preventing teacher guidance from acting against verifier feedback is a natural design choice with close precedents. SG-OPD uses verifier–teacher sign consistency to adjust distillation, while Self-Distilled Policy Gradient explicitly restricts its auxiliary distillation loss to positive-advantage responses. Neither is identical to the proposed multiplicative update, but these connections narrow the novelty of negative reset.
The analysis and experimental scope do not sufficiently support the broader claims. Figures 2 and 3 document performance plateaus or deterioration under the selected configurations, but do not establish that KL-based imitation causes a local optimum or that sampling variance is not the bottleneck. The entropy-control experiment demonstrates transfer of a distributional property, rather than previously unavailable task knowledge. Moreover, geometric normalization changes the overall scale of positive updates as well as relative token weights, leaving its mechanism unresolved.
Broader evaluation. Although the paper includes knowledge and coding evaluations, all reported post-training uses mathematical data. Direct training experiments in another domain would provide stronger support for the proposed method as a general post-training approach.
I tentatively recommend weak acceptance. While the empirical results are promising, a more thorough understanding of the interaction between negative reset and geometric normalization would be needed for a stronger recommendation.
[1] Self-Distilled Policy Gradient. Yifeng Liu, Shiyuan Zhang, Yifan Zhang, and Quanquan Gu. arXiv:2606.04036, June 2026.
[2] SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling. Haoran Xu, Hongyu Wang, Yifei Gao, Jiaze Li, Xiaofeng Zhang, and Xiaosong Yuan. arXiv:2606.09304, June 2026.
Questions for authors
How does Distilled RL compare with OPD+RL when the distillation term is applied only to positive-advantage responses and its coefficient is tuned?
3 comments
With additional experimental results, I believe both the empirical and methodological contribution of this work is limited. However, out of consistency, I maintain my score.
Dear reviewer,
Thank you for reviewing our additional results and sharing your assessment. Please allow us to once again clarify the distinctive contributions of Distilled RL.
Methodologically, we argue that KL-based imitation directs gradients toward the teacher, potentially restricting student exploration and conflicting with the RL objective. To the best of our knowledge, Distilled RL was the first on-policy distillation method without an explicit KL distillation loss. It incorporates teacher preferences directly into reward-driven updates. Negative sample reset and sequence-level geometric normalization address the resulting problems through clear, interpretable mechanisms. The controlled entropy experiment further demonstrates that teacher information can be transferred through this update.
Empirically, with the additions made during rebuttal, our evaluation covers three teacher models, four student models, two modalities, and sixteen benchmarks. The ablations show clear effects of the proposed components. We also discuss failure cases in the appendix, examining when teacher guidance becomes less effective. Together, these experiments and analyses help readers understand both the benefits and limitations of OPD and Distilled RL.
Taken together, these methodological contributions and extensive empirical analyses establish the distinct value of Distilled RL. Thank you again for your time and consideration.
Best wishes,
Authors of 10078
Response to reviewer dif1
Dear reviewer dif1,
Thank you for your careful review and constructive comments, which help clarify our method and improve the paper. We group related points below.
Thank you for these valuable references. These concurrent studies of distillation gating show the importance of this problem.
However, both works are heuristic improvements based on OPD and do not establish why their modifications are necessary. We believe a modification alone does not fully constitute novelty; explaining its underlying mechanism and necessity can provide broader insights.
We believe KL directs the gradient entirely toward the teacher, limiting the student's exploration of better solutions. We therefore introduce the teacher into RL through reverse importance sampling. Through a logical argument, we turn the difficult question of consistency between teacher guidance and the verifier into negative sample reset, and establish its necessity through ablation. Finally, because the reverse importance ratios are below one—the upper-right panels of Figures 9–11 show geometric means around 0.5–0.8—the positive and negative sample weights are mismatched, weakening positive-sample weights. We introduce sequence-level geometric normalization to correct this problem.
These components form a clear, effective framework. Our interpretable entropy study also shows teacher distributional signals passing through the policy gradient. These distinguish our method from related RL and OPD approaches.
First, we have added experiments tuning the RL–OPD mixing coefficient and applying OPD only to positive samples. We believe the difference lies in KL: it directs the gradient entirely toward the teacher, which may suppress the student's exploration and self-improvement.
The entropy-control experiment demonstrates transfer of distributional properties, rather than previously unavailable task knowledge. However, previously unavailable task knowledge also seems beyond what on-policy algorithms can acquire, which may be an inherent limitation: when sampling only from the student, correct samples may never be obtained. Using a measure beyond task performance to demonstrate this transfer of teacher knowledge is a novel aspect of our work.
Geometric normalization also corrects the weakening of positive-sample weights caused by applying teacher ratios only to positive samples, as explained in the preceding response.
We have added multimodal and VLM experiments using Qwen2.5-VL-7B-Instruct as the teacher and Qwen2.5-VL-3B-Instruct as the student, trained on GEO3K.
The HallusionBench results show improved visual hallucination performance in this setting, while LogicVista and MathVista support improved multimodal reasoning. General helpfulness and harmlessness are separate evaluation dimensions.
Best wishes,
Authors of 10078