acceptodds
Under review as a conference paper at ICLR 2027

Distilled Reinforcement Learning for LLM Post-training

Abstract

Large language model (LLM) post-training is essential for improving reasoning, adaptation, and alignment. Existing methods mainly follow two paradigms: reinforcement learning (RL) and on-policy distillation (OPD). However, RL relies on coarse-grained outcome supervision, resulting in difficult credit assignment and limited capability to acquire new knowledge. OPD, meanwhile, unconditionally matches teacher logits through KL divergence, which creates a dilemma: similar teachers provide little new knowledge, while substantially different teachers often yield ineffective guidance, largely restricting OPD to within-family distillation. We propose Distilled Reinforcement Learning (Distilled RL), which integrates teacher supervision into the RL objective to provide fine-grained guidance, selectively transfer new knowledge and avoid unconditional imitation. Distilled RL contains three components: reverse importance sampling with clipping, negative sample reset, and sequence-level geometric normalization. Through a concise and interpretable study, we show that Distilled RL can effectively transfer knowledge from a teacher model to a student model. Extensive experiments across both within-family and cross-family (significant difference in behavior between students and teachers) distillation settings show that Distilled RL substantially outperforms baselines in terms of both pass@1 and pass@k.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, On-Policy Distillation, LLM Post-Training, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_dif1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Distilled RL, which incorporates teacher supervision into GRPO through token-level weighting rather than a separate distillation loss. Teacher-to-student probability ratios are clipped and geometrically normalized within each response. These weights modify positive-advantage updates, while nonpositive-advantage responses retain the ordinary GRPO update. The method is evaluated using mathematical post-training across three student models, with additional knowledge and coding evaluations. The reported results show consistent improvements in average mathematical performance over the evaluated baselines, and ablations examine negative-sample reset and geometric normalization.

Critical strengths

A simple and interesting coupling of teacher guidance with outcome supervision. The outcome advantage determines the direction of the update, while the teacher adjusts relative token weights on positive-advantage responses. Combining this separation with within-response log-ratio centering is a reasonable design: it allows teacher guidance without requiring unconditional distribution matching. The resulting modification is straightforward to incorporate into an existing GRPO implementation.

Consistent average improvements with useful component ablations. Distilled RL improves average mathematical performance across all three student models. The ablations establish that negative reset is important to avoid degradation and that geometric normalization provides additional gains on both evaluated students. These experiments provide useful evidence for the complete recipe, although they do not fully identify the mechanism responsible for the improvement.

Critical weaknesses

The distinct contribution relative to existing gated distillation methods is insufficiently established. Removing negative reset causes the largest ablation drop and makes the method worse than ordinary GRPO. However, preventing teacher guidance from acting against verifier feedback is a natural design choice with close precedents. SG-OPD uses verifier–teacher sign consistency to adjust distillation, while Self-Distilled Policy Gradient explicitly restricts its auxiliary distillation loss to positive-advantage responses. Neither is identical to the proposed multiplicative update, but these connections narrow the novelty of negative reset.

The analysis and experimental scope do not sufficiently support the broader claims. Figures 2 and 3 document performance plateaus or deterioration under the selected configurations, but do not establish that KL-based imitation causes a local optimum or that sampling variance is not the bottleneck. The entropy-control experiment demonstrates transfer of a distributional property, rather than previously unavailable task knowledge. Moreover, geometric normalization changes the overall scale of positive updates as well as relative token weights, leaving its mechanism unresolved.

Broader evaluation. Although the paper includes knowledge and coding evaluations, all reported post-training uses mathematical data. Direct training experiments in another domain would provide stronger support for the proposed method as a general post-training approach.

I tentatively recommend weak acceptance. While the empirical results are promising, a more thorough understanding of the interaction between negative reset and geometric normalization would be needed for a stronger recommendation.

[1] Self-Distilled Policy Gradient. Yifeng Liu, Shiyuan Zhang, Yifan Zhang, and Quanquan Gu. arXiv:2606.04036, June 2026.

[2] SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling. Haoran Xu, Hongyu Wang, Yifei Gao, Jiaze Li, Xiaofeng Zhang, and Xiaosong Yuan. arXiv:2606.09304, June 2026.

Questions for authors

How does Distilled RL compare with OPD+RL when the distillation term is applied only to positive-advantage responses and its coefficient is tuned?

3 comments
AuthorsOpenReview ↗

Response to reviewer dif1

Dear reviewer dif1,

Thank you for your careful review and constructive comments, which help clarify our method and improve the paper. We group related points below.

W1: Distinction from existing gated distillation methods

Thank you for these valuable references. These concurrent studies of distillation gating show the importance of this problem.

  • [1] SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling. arXiv:2606.09304. This work introduces a small number of verifier-validated teacher rollouts early in training to reduce the trajectory-distribution mismatch between a weak student and a strong teacher. It compares the signs of the outcome advantage and the OPD token advantage, strengthening distillation when they agree and reducing its strength when they conflict.
  • [2] Self-Distilled Policy Gradient. arXiv:2606.04036. This work argues that even when the teacher has the correct answer, it may provide unreliable local supervision at intermediate prefixes of incorrect trajectories. It therefore applies distillation only to trajectories with positive outcome advantages.

However, both works are heuristic improvements based on OPD and do not establish why their modifications are necessary. We believe a modification alone does not fully constitute novelty; explaining its underlying mechanism and necessity can provide broader insights.

We believe KL directs the gradient entirely toward the teacher, limiting the student's exploration of better solutions. We therefore introduce the teacher into RL through reverse importance sampling. Through a logical argument, we turn the difficult question of consistency between teacher guidance and the verifier into negative sample reset, and establish its necessity through ablation. Finally, because the reverse importance ratios are below one—the upper-right panels of Figures 9–11 show geometric means around 0.5–0.8—the positive and negative sample weights are mismatched, weakening positive-sample weights. We introduce sequence-level geometric normalization to correct this problem.

These components form a clear, effective framework. Our interpretable entropy study also shows teacher distributional signals passing through the policy gradient. These distinguish our method from related RL and OPD approaches.

W2 and Q1: Additional baselines and analysis of information transfer

First, we have added experiments tuning the RL–OPD mixing coefficient and applying OPD only to positive samples. We believe the difference lies in KL: it directs the gradient entirely toward the teacher, which may suppress the student's exploration and self-improvement.

MethodAIME24AIME25AIME26AMC23CMIMC25GSM8KHMMT25MATH500Minerva MathOlympiadBenchAvg
OPD+RL 0.2551.6741.6747.0886.0223.8395.0725.5291.8047.0659.7056.94
OPD+RL 0.554.0639.3746.4584.0622.8994.8423.8592.0047.0559.2556.38
OPD+RL 0.7552.1939.0647.5084.1423.2894.5425.6390.4046.3259.2656.23
OPD-positive51.7741.9847.1985.3924.5395.0025.1091.0046.6958.9656.76
Distilled RL56.4542.2949.0687.8926.9594.9928.8593.2048.8961.0358.96

The entropy-control experiment demonstrates transfer of distributional properties, rather than previously unavailable task knowledge. However, previously unavailable task knowledge also seems beyond what on-policy algorithms can acquire, which may be an inherent limitation: when sampling only from the student, correct samples may never be obtained. Using a measure beyond task performance to demonstrate this transfer of teacher knowledge is a novel aspect of our work.

Geometric normalization also corrects the weakening of positive-sample weights caused by applying teacher ratios only to positive samples, as explained in the preceding response.

W3: Additional multimodal experiments

We have added multimodal and VLM experiments using Qwen2.5-VL-7B-Instruct as the teacher and Qwen2.5-VL-3B-Instruct as the student, trained on GEO3K.

MethodHallusionBench aAcc ↑HallusionBench fAcc ↑HallusionBench qAcc ↑LogicVista ↑MathVista ↑Avg ↑
Qwen2.5-VL-3B-Instruct52.4227.7927.7737.6358.9040.90
GRPO54.1528.0328.7939.7362.2042.58
OPD55.5230.0629.2339.5161.0043.06
Distilled RL56.5731.5031.4342.4162.9044.96

The HallusionBench results show improved visual hallucination performance in this setting, while LogicVista and MathVista support improved multimodal reasoning. General helpfulness and harmlessness are separate evaluation dimensions.

Best wishes,

Authors of 10078

With additional experimental results, I believe both the empirical and methodological contribution of this work is limited. However, out of consistency, I maintain my score.

Dear reviewer,

Thank you for reviewing our additional results and sharing your assessment. Please allow us to once again clarify the distinctive contributions of Distilled RL.

Methodologically, we argue that KL-based imitation directs gradients toward the teacher, potentially restricting student exploration and conflicting with the RL objective. To the best of our knowledge, Distilled RL was the first on-policy distillation method without an explicit KL distillation loss. It incorporates teacher preferences directly into reward-driven updates. Negative sample reset and sequence-level geometric normalization address the resulting problems through clear, interpretable mechanisms. The controlled entropy experiment further demonstrates that teacher information can be transferred through this update.

Empirically, with the additions made during rebuttal, our evaluation covers three teacher models, four student models, two modalities, and sixteen benchmarks. The ablations show clear effects of the proposed components. We also discuss failure cases in the appendix, examining when teacher guidance becomes less effective. Together, these experiments and analyses help readers understand both the benefits and limitations of OPD and Distilled RL.

Taken together, these methodological contributions and extensive empirical analyses establish the distinct value of Distilled RL. Thank you again for your time and consideration.

Best wishes,

Authors of 10078

Reviewer_zc1cOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies Large language model (LLM) post-training. The paper considers challenging credit assignment problem in RL and OPD. The paper proposes distilled RL and integrates teacher supervision into the RL objective. The paper tries to transfer new knowledge and avoids unconditional imitation. It designs reverse importance sampling, negative sample reset and geometric normalization. The paper provides experiments to demonstrate the performance of the proposed method.

Critical strengths

The paper has a reasonable motivation. It wants to provide fine grained rewards.

It is good that the paper compares with some related methods, such as OPD+RL, ExOPD.

The framework with three key components seems to work well. The method has good results on multiple datasets.

Critical weaknesses

Combining OPD with RL for post-training is not new. There are multiple OPD+RL methods. The novelty of the paper is limited.

The paper may provide more details and explanation about r_i,t * w_i,t. Is that possible to have other forms?

Experiments are not completed. Table 7 just uses one student model.

Questions for authors

What does unconditional imitation mean?

Why does the paper use r_i,t * w_i,t ?

4 comments
AuthorsOpenReview ↗

Response to reviewer zc1c

Dear reviewer zc1c,

Thank you for your careful review and constructive comments, which help clarify our method and improve the quality of the paper. We address related weaknesses and questions together below.

W1: Novelty beyond combining RL and distillation

Our contribution is not simply to combine RL and distillation, but to incorporate teacher guidance into RL from an importance-weighting perspective and progressively address the resulting optimization issues. This perspective is central to the method, rather than treating teacher supervision as an additional distillation loss.

The distinction from existing RL + KL regularization methods lies in the role of the teacher: KL regularization emphasizes teacher imitation, whereas RL emphasizes improvement through the student's own exploration. The vast majority of existing RL+OPD algorithms are implemented within an OPD framework and apply heuristics on top of KL, which limits the model's ability to improve independently. We believe the teacher should provide guidance rather than serve as a distribution to imitate. KL directs the gradient toward the teacher, limiting the student's exploration and self-improvement. We therefore introduce the teacher into RL from an importance-sampling perspective, together with negative sample reset and sequence-level geometric normalization. The framework comprising these three key components is clear and works well. Our interpretable entropy case study further shows that teacher distributional signals are transmitted through the policy gradient. These are the key distinctions from related RL and OPD approaches.

We have also conducted experiments tuning the RL–OPD mixing coefficient and applying OPD only to positive samples.

MethodAIME24AIME25AIME26AMC23CMIMC25GSM8KHMMT25MATH500Minerva MathOlympiadBenchAvg
OPD+RL 0.2551.6741.6747.0886.0223.8395.0725.5291.8047.0659.7056.94
OPD+RL 0.554.0639.3746.4584.0622.8994.8423.8592.0047.0559.2556.38
OPD+RL 0.7552.1939.0647.5084.1423.2894.5425.6390.4046.3259.2656.23
OPD-positive51.7741.9847.1985.3924.5395.0025.1091.0046.6958.9656.76
Distilled RL56.4542.2949.0687.8926.9594.9928.8593.2048.8961.0358.96

W2 and Q2: Why multiply the policy ratio and teacher weight?

Thank you for this insightful question. This is a deliberate design choice: the two multiplicative factors serve different roles. Specifically,

\rho_{i,t}=\frac{\pi_{\mathrm{teacher}}(o_{i,t}\mid q,o_{i,<t})}{\pi_{\mathrm{old}}(o_{i,t}\mid q,o_{i,<t})}.$$ In GRPO, the importance ratio $r_{i,t}$ addresses the mismatch between the rollout and updated policies, correcting from the sampling distribution $\pi_{\mathrm{old}}$ toward $\pi$. PPO-style clipping controls updates that move too far from that policy. In Distilled RL, the teacher weight introduces token-level guidance, setting aside teacher-ratio clipping and sequence-level geometric normalization for the moment. The objective uses $$\min\left(r_{i,t}w_{i,t}A_i, \operatorname{clip}(r_{i,t},1-\epsilon_{\mathrm{low}},1+\epsilon_{\mathrm{high}})w_{i,t}A_i\right).$$ We need $\pi$ in the numerator to provide the gradient and $\pi_{\mathrm{teacher}}$ in the numerator to introduce the teacher's target distribution. The raw product is therefore $$r_{i,t}\rho_{i,t}=\frac{\pi_\theta\,\pi_{\mathrm{teacher}}}{\pi_{\mathrm{old}}^2}.$$ Multiplication itself is not the issue. Placing the product inside PPO clipping would conflate teacher–student mismatch with policy-update magnitude. At the start of an update, the product equals the teacher ratio and need not be near one. Figures 9–11 show geometric means of $\rho_{i,t}$ around 0.5–0.8. Therefore, in the vast majority of cases, the combined ratio may activate clipping because the teacher and student differ too much, causing the vast majority of tokens to be clipped. Our design separates these functions: **$r_{i,t}$ remains inside PPO clipping for update stability, while $\rho_{i,t}$ remains outside it to provide guidance**. We believe our design is not only reasonable but also carefully considered, and we do not believe there is another more reasonable formulation. > W3: Experimental scope Table 7 is an additional teacher-generalization experiment, not an unfinished main experiment. Table 2 evaluates three students; Table 7 changes the teacher to Nemotron-7B-GRPO for DSQW-1.5B and is placed in the appendix for space reasons. We will clarify this connection. We further extend evaluation to Qwen2.5-VL-7B-Instruct → Qwen2.5-VL-3B-Instruct: | Method | HallusionBench aAcc ↑ | HallusionBench fAcc ↑ | HallusionBench qAcc ↑ | LogicVista ↑ | MathVista ↑ | Avg ↑ | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | Qwen2.5-VL-3B-Instruct | 52.42 | 27.79 | 27.77 | 37.63 | 58.90 | 40.90 | | GRPO | 54.15 | 28.03 | 28.79 | 39.73 | 62.20 | 42.58 | | OPD | 55.52 | 30.06 | 29.23 | 39.51 | 61.00 | 43.06 | | Distilled RL | 56.57 | 31.50 | 31.43 | 42.41 | 62.90 | 44.96 | This extends the experimental evidence to multimodal reasoning and visual hallucination evaluation. > Q1: Meaning of unconditional imitation Unconditional imitation means applying the teacher-distribution-matching objective regardless of the outcome reward or advantage of the sampled response. We also question the KL objective itself: broadly speaking, it directs optimization toward the teacher, whereas the best update direction need not point entirely toward the teacher and may even conflict with it. We therefore want the student to find useful update directions through RL, with the teacher mainly providing guidance. The analogy to a teacher guiding a student, rather than asking for exact imitation, helps explain this distinction. Best wishes, Authors of **10078**

Thank you for the response. However, the method still feels heuristic too. Its core novelty remains unclear.

Dear Reviewer,

Thank you for reviewing our additional results and sharing your assessment. We would like to once again clarify the distinctive contributions and fundamental novelty of Distilled RL.

To the best of our knowledge, Distilled RL is the first on-policy distillation (OPD) method that does not rely on an explicit KL divergence term. As emphasized in our previous response, we argue that KL-based approaches directly incorporate teacher preferences into reward-driven policy updates, thereby coupling the teacher's preferences with the student's reward optimization.

In this regard, Distilled RL represents a more fundamental departure from existing KL-based and gating-based OPD approaches [1, 2, 3, 4], as also discussed in the Related Work section of our revised manuscript. Unlike these approaches, which primarily focus on modifying KL-based objectives or selectively controlling teacher guidance, our method eliminates the need for explicit KL regularization. We believe that this distinction underscores the conceptual novelty and fundamental contribution of Distilled RL, rather than merely an incremental modification of existing OPD methods.

Best wishes, Authors of Submission 10078

References

[1] Liu, X., Jiao, K., Xiao, C., et al. (2026). Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence. Findings of the Association for Computational Linguistics: EMNLP 2026, 24801–24816. https://aclanthology.org/2026.findings-emnlp.1172/

[2] Han, Z., Xiao, J., Lu, Z., et al. (2026). Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance. Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, 47593–47606. https://aclanthology.org/2026.emnlp-main.2089/

[3] Xu, H., Wang, H., Gao, Y., et al. (2026). SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling. arXiv preprint arXiv:2606.09304. https://arxiv.org/abs/2606.09304

[4] Liu, Y., Zhang, S., Zhang, Y., & Gu, Q. (2026). Self-Distilled Policy Gradient. arXiv preprint arXiv:2606.04036. https://arxiv.org/abs/2606.04036

Reviewer_kdVLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies LLM post-training. The authors point out that outcome-based RL struggles with coarse supervision and credit assignment, while on-policy distillation is constrained by unconditional KL matching and mostly limited to within-family distillation. The authors thus propose Distilled RL, integrating fine-grained teacher guidance into the RL objective for selective knowledge transfer. It includes three modules: clipped reverse importance sampling, negative sample reset, and sequence-level geometric normalization. Evaluations on reasoning tasks demonstrate superior pass@1 and pass@k over baselines in both within-family and cross-family distillation.

Critical strengths
  1. The authors discussed the pain points of post-training: credit assignment limitation in outcome-based RL and the unconditional KL-matching dilemma in on-policy distillation, and frames them under a unified motivation, which makes the problem statement sharp and coherent.

  2. The proposed Distilled RL provides a way for integrating teacher logit supervision into RL objective, instead of treating RL and distillation as independent pipelines. Such design addresses the core limitation of OPD: blind imitation regardless of whether teacher signals are useful.

  3. The three sub-components are clearly defined and interpretable, rather than a black-box composite trick; the authors promise concise ablative analysis to validate each module’s contribution.

  4. Evaluation uses the widely adopted reasoning metrics pass@1 and pass@k, making results easy to compare against existing RL and distillation baselines.

Critical weaknesses
  1. The abstract does not clarify whether the gains mainly come from better knowledge transfer or better alignment preference. It is unclear whether the method brings trade-offs such as degradation on harmlessness / helpfulness, a common risk for RL-based post-training.

  2. The three proposed components (clipped reverse IS, negative reset, sequence geometric normalization) appear ad-hoc, it is unclear whether all these components are necessary or some components can be simplified.

  3. The distinction between Distilled RL and existing RL + KL-regularized methods (e.g., PPO with teacher KL penalty) is not explicitly drawn, the novelty boundary against prior hybrid RL-distillation work is ambiguous.

  4. This work reports pass@1 / pass@k on reasoning tasks, however, it does not test for overfitting, diversity collapse, or distribution shift, which are typical failure modes for sequence-level distillation and RL post-training.

Questions for authors
  1. How does Distilled RL differ from standard PPO augmented with a fixed KL penalty toward the teacher model?

  2. In the cross-family setting, is the improvement consistent across all teacher–student pairs, or does the performance gain only emerge under specific teacher-student capacity gaps?

  3. The side effects such as response diversity, hallucination rate, or safety alignment after Distilled RL training are not clearly discussed, since RL and distillation often introduce unintended trade-offs?

  4. This work uses Qwen3-8B as the teacher and Qwen3-1.7B/4B as the student, does the findings still hold when the teacher is Qwen3-32B and the student is Qwen3-8B?

  5. Among the three proposed components, which one is the dominant contributor to cross-family transfer? Could some components be removed without affecting the main empirical gains?

3 comments
AuthorsOpenReview ↗

Response to reviewer kdVL

Dear reviewer kdVL,

Thank you for your careful review and constructive feedback, which help improve the quality of our paper. We address related points together below.

W1: Knowledge transfer versus preference alignment

Our interpretable entropy case study shows that teacher distributional signals are indeed transmitted through the policy gradient. However, it is currently difficult to determine whether the gains mainly come from better knowledge transfer or better preference alignment. This is a real limitation. Thank you for raising it; we will emphasize it in the Limitations section.

W2 and Q5: Roles and necessity of the components

The three components form a progressive design, with each addressing a problem exposed by the preceding step. Reverse importance weighting introduces teacher information; removing it recovers ordinary RL. Negative sample reset prevents amplified penalties on teacher-preferred tokens in negative-advantage responses. Removing it causes the largest performance drop on both students in Table 4. Geometric normalization further improves negative sample reset by centering clipped log-ratios within each response. The upper-right panels of Figures 9–11 show raw geometric-mean ratios below one, roughly 0.5–0.8, whereas negative-sample weights are reset to one. Normalization removes the response-level log-ratio offset while preserving relative token preferences. In Distilled RL, all three components are essential in principle, with negative sample reset playing the most important role.

W3 and Q1: Novelty relative to hybrid RL–distillation methods

Our OPD+RL baseline closely corresponds to the RL formulation with a teacher KL regularizer that you describe. We have additionally conducted experiments tuning the RL–OPD mixing coefficient and applying OPD only to positive samples.

MethodAIME24AIME25AIME26AMC23CMIMC25GSM8KHMMT25MATH500Minerva MathOlympiadBenchAvg
OPD+RL 0.2551.6741.6747.0886.0223.8395.0725.5291.8047.0659.7056.94
OPD+RL 0.554.0639.3746.4584.0622.8994.8423.8592.0047.0559.2556.38
OPD+RL 0.7552.1939.0647.5084.1423.2894.5425.6390.4046.3259.2656.23
OPD-positive51.7741.9847.1985.3924.5395.0025.1091.0046.6958.9656.76
Distilled RL56.4542.2949.0687.8926.9594.9928.8593.2048.8961.0358.96

The distinction from existing RL + KL regularization methods lies in the role of the teacher: KL regularization emphasizes teacher imitation, whereas RL emphasizes improvement through the student's own exploration. The vast majority of existing RL+OPD algorithms are implemented within an OPD framework and apply heuristics on top of KL, which limits the model's ability to improve independently. We believe the teacher should provide guidance rather than serve as a distribution to imitate. KL directs the gradient toward the teacher, limiting the student's exploration and self-improvement. We therefore introduce the teacher into RL from an importance-sampling perspective, together with negative sample reset and sequence-level geometric normalization. The framework comprising these three key components is clear and works well. Our interpretable entropy case study further shows that teacher distributional signals are transmitted through the policy gradient. These are the key distinctions from related RL and OPD approaches.

Q2: Consistency across teacher–student pairs

Average gains are consistent across the evaluated within-family student sizes. Distilled RL also outperforms OPD in both cross-family settings, using Qwen3-8B-GRPO or Nemotron-7B-GRPO as the teacher for DSQW-1.5B.

Teacher competence and reasoning learnability make cross-family transfer more complex. However, OPD, our primary distillation baseline, also depends on teacher quality and teacher–student compatibility. As discussed in the Limitations section (Appendix A), performance can decline when teacher preferences have an adverse effect, as in our deliberately weakened-teacher setting. Thus, introducing a teacher is not always beneficial. This is a shared limitation of on-policy distillation; incorporating RL mitigates it to some extent but does not eliminate it. Across the reported standard within-family and cross-family configurations, the average gains over OPD nevertheless remain consistent.

W4 and Q3–Q4: Diversity, generalization, hallucination, and scale

Figure 4 shows relatively stable, nonzero policy entropy during later training in both settings. Together with Pass@16, this supports the absence of obvious collapse. Tables 3 and 8 evaluate transfer from mathematical training to knowledge and coding tasks, supporting cross-task generalization.

We have also added a multimodal experiment using Qwen2.5-VL-7B-Instruct as the teacher and Qwen2.5-VL-3B-Instruct as the student, trained on GEO3K, to evaluate visual hallucination and multimodal reasoning:

MethodHallusionBench aAcc ↑HallusionBench fAcc ↑HallusionBench qAcc ↑LogicVista ↑MathVista ↑Avg ↑
Qwen2.5-VL-3B-Instruct52.4227.7927.7737.6358.9040.90
GRPO54.1528.0328.7939.7362.2042.58
OPD55.5230.0629.2339.5161.0043.06
Distilled RL56.5731.5031.4342.4162.9044.96

The HallusionBench results show improved visual hallucination performance in this setting, while LogicVista and MathVista support improved multimodal reasoning. General helpfulness and harmlessness are separate evaluation dimensions.

Using Qwen3-32B as the teacher and Qwen3-8B as the student would be valuable, but would substantially increase resource requirements. We have only eight A100 GPUs and prioritize supporting as long a context as feasible. Under the current configuration, a single Qwen3-4B experiment already takes approximately two days. We therefore used the additional multimodal teacher–student experiment as a feasible alternative to broaden the evaluation. We believe it provides evidence of generalization across modalities, complementing the text-only experiments.

Best wishes,

Authors of 10078

Dear reviewer,

Thank you again for reviewing our paper. As the discussion period draws to a close, could you please let us know whether our responses and additional results have addressed your concerns? If any issues remain unresolved, we would appreciate your feedback so that we can provide further clarification.

Best wishes,

Authors of 10078

Reviewer_KvZLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the key limitations of existing LLM post-training paradigms: reinforcement learning (RL) suffers from coarse sequence-level credit assignment and limited exploration for new knowledge, while on-policy distillation (OPD) relies on KL matching that leads to early saturation and struggles under cross-family distribution shifts. To bridge these paradigms, the authors propose Distilled Reinforcement Learning (Distilled RL), which integrates teacher supervision directly into the token-level RL advantage weighting. The method introduces three core mechanisms: reverse importance sampling, negative sample reset, and sequence-level geometric normalization. Extensive evaluations under diverse scenarios across within-family and cross-family setups demonstrate consistent improvements over GRPO, OPD, and hybrid baselines.

Critical strengths
  1. The authors successfully identifies the fundamental failure mode of combining OPD and RL. The proposed three new designs are simple, intuitive, and effective.
  2. The proposed Distilled RL achieves decent gains across multiple model scales and benchmarks, especially under cross-family scenarios.
  3. The interpretable entropy case study provides a clean, interpretable demonstration that teacher distributional signals are indeed transmitted through the policy gradient without relying on explicit KL terms.
Critical weaknesses
  1. The proposed objective reweights token advantages heuristically via geometrically normalized likelihood ratios. It lacks a formal theoretical derivation showing whether this optimizes a principled surrogate objective, a divergence bound, or something else.
  2. The evaluations are mainly focused on mathematical reasoning trained on DAPO-17k. It remains unproven whether Distilled RL can perform well on larger student/teacher models or tasks with more complex reward models.
  3. The proposed designs implicitly assume that teacher token preferences on positive student rollouts are always high-quality, which may not always be true under real-world scenarios.
Questions for authors
  1. Can the authors provide a formal optimization perspective for the objective in Equation (3)?
  2. Because reweighting applies to all trajectories with A_i > 0, what happens when a student reaches the correct final state via an incorrect intermediate step?
  3. If average response lengths were controlled or normalized across methods, what would be the exact latency and FLOPs overhead introduced by querying the teacher forward pass on student rollouts?
2 comments
AuthorsOpenReview ↗

Response to reviewer KvZL

Dear reviewer KvZL,

Thank you for your careful review and constructive comments, which help clarify our method and improve the paper.

W1 and Q1: Design rationale for the three components

Thank you for this suggestion. The components address specific problems rather than follow from a unified theoretical objective. Reverse importance weighting introduces teacher preferences directly into RL, so the teacher provides guidance rather than a separate imitation target. However, applying it directly to negative-advantage responses amplifies penalties on teacher-preferred tokens. We therefore introduce negative sample reset, retaining the original RL update for negative samples. After reset, negative-sample weights are one, whereas Figures 9–11 show teacher ratios with geometric means below one. We further introduce sequence-level geometric normalization to remove response-level log-ratio offsets while preserving relative token preferences. The components build on one another; the ablations support their roles, and the entropy-control experiment shows that teacher distributional signals can pass through the policy gradient.

W2: Additional multimodal VLM experiments

To broaden the training and evaluation scope, we have added a multimodal experiment using Qwen2.5-VL-7B-Instruct as the teacher and Qwen2.5-VL-3B-Instruct as the student, trained on GEO3K:

MethodHallusionBench aAcc ↑HallusionBench fAcc ↑HallusionBench qAcc ↑LogicVista ↑MathVista ↑Avg ↑
Qwen2.5-VL-3B-Instruct52.4227.7927.7737.6358.9040.90
GRPO54.1528.0328.7939.7362.2042.58
OPD55.5230.0629.2339.5161.0043.06
Distilled RL56.5731.5031.4342.4162.9044.96

Distilled RL outperforms GRPO and OPD on all reported metrics, supporting its effectiveness in multimodal reasoning and visual hallucination evaluation. Avg is the unweighted mean of the five metrics.

W3: Reliability of teacher guidance

This is a valid point, discussed in our Limitations (Appendix A): the teacher is not superior to the student on every problem or throughout training, and its guidance also depends on whether its reasoning is learnable by the student. Appendix B.4 discusses when Distilled RL may fail, using a deliberately weakened teacher. In practice, choosing distillation presupposes that the teacher is stronger than the student in the relevant capability. We account for teacher competence by relaxing this requirement to high-quality preferences on positive student samples, broadening OPD applicability and improving its effectiveness. If this condition cannot be met, distillation should not be chosen; this is a prerequisite.

Q2: Correct answers with incorrect intermediate steps

For a response with a correct answer but incorrect intermediate steps, the outcome reward still gives a positive advantage, and teacher weights are not process-correctness labels. Assuming reliable teacher guidance, incorrect intermediate steps receive lower weights, but are not penalized. Subsequent correct reflection steps receive additional reinforcement, as the teacher agrees with them. This is the behavior we aim for.

Q3: Teacher-forward computation overhead

In our Qwen3-8B → Qwen3-4B Distilled RL experiment, training runs for 160 steps. The average timings are:

ComponentTime(s)Percentage
πold\pi_{\mathrm{old}}65.957.52%
πteacher\pi_{\mathrm{teacher}}85.179.72%
Full training step876.46100%

Teacher scoring takes a small part of each step: only a forward pass on student rollouts, without teacher generation or backpropagation. Appendix B.1 also analyzes overall efficiency: shorter rollouts can offset the additional teacher-forward cost. Thus, teacher guidance has limited overhead, with no unnecessary computation. Timings are averages under the actual training configuration.

Best wishes,

Authors of 10078

Dear reviewer,

Thank you again for reviewing our paper. As the discussion period draws to a close, could you please let us know whether our responses and additional results have addressed your concerns? If any issues remain unresolved, we would appreciate your feedback so that we can provide further clarification.

Best wishes,

Authors of 10078

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to AC and reviewers

Dear AC and reviewers,

Thank you for the time and feedback you have contributed to this manuscript. Your work has greatly helped improve its quality. We summarize the strengths recognized by reviewers and the main issues addressed in our first-round rebuttal below.

Strengths recognized by reviewers

  • Clear motivation (KvZL, kdVL, zc1c). Reviewers appreciated the problem formulation and motivation, recognizing that Distilled RL identifies important failure modes when combining OPD and RL.
  • Clear, interpretable, and effective components (all reviewers). Reviewers recognized the method built from three clearly defined components as intuitive, effective, and interpretable, rather than a black-box combination of tricks.
  • Broad evaluation and consistent gains (all reviewers). Reviewers recognized good performance across datasets. KvZL highlighted cross-family gains; kdVL appreciated the widely used reasoning metrics; zc1c valued comparisons with relevant baselines.
  • Interpretable entropy case study (KvZL). Reviewer KvZL recognized this experiment as a clear demonstration that teacher distributional signals can pass through the policy gradient without an explicit KL term.
  • Concise, useful component ablations (kdVL, dif1). The reviews valued the verifiable component contributions and ablation evidence: negative sample reset is important for avoiding degradation, and geometric normalization adds gains on both evaluated students, supporting the complete method.

Main concerns and our responses

  • Distinction from existing OPD+RL methods (kdVL, zc1c, dif1). We do not use KL because we believe its update direction points toward the teacher, limiting student exploration and self-improvement. Instead, importance weighting introduces teacher guidance directly into RL. We explain the three clear, important, and interpretable components as a progressive design and establish their necessity. The entropy-control experiment further shows teacher distributional signals passing through the policy gradient. These mechanistic explanations and diagnostics, missing from related heuristic approaches, can offer readers broader insights. We also added OPD+RL mixing-coefficient tuning and positive-only OPD experiments as further evidence.
  • Broader experimental scope (KvZL, kdVL, dif1). To address requests for broader tasks, we added multimodal experiments using Qwen2.5-VL-7B-Instruct as the teacher and Qwen2.5-VL-3B-Instruct as the student, trained on GEO3K and evaluated on HallusionBench, LogicVista, and MathVista. These results extend the evidence to visual hallucination and multimodal reasoning.
  • Further explanation of the objective (KvZL, zc1c). Reviewer KvZL requested a formal theoretical perspective, while zc1c asked about the design of ri,twi,tr_{i,t}w_{i,t}. We explain that on-policy distillation is largely empirically developed. We also clarify the factors' roles: ri,tr_{i,t} remains inside PPO clipping for update stability, while wi,tw_{i,t}, constructed from the teacher ratio ρi,t\rho_{i,t}, acts outside policy-ratio clipping to provide teacher guidance.
  • Other specific questions. We address these individually in the corresponding reviewer responses.

The above summarizes the main points of our first-round rebuttal.

Best wishes,

Authors of 10078

Dear AC and reviewers,

Thank you for your valuable feedback. We have uploaded a revised manuscript, with changes marked in blue. The main updates are:

  1. Method: We added two paragraphs at the end of the method section to summarize the roles of the three components and explain the two importance-weighting factors.
  2. Experiments: We updated the RL+OPD experiments and added multimodal experiments.
  3. Limitations: We clarified that this work is primarily empirical and that deeper theoretical analysis remains future work.

Best wishes,

Authors of 10078

Dear Reviewers,

The authors have posted their responses to the reviews. Please go through their rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.

The discussion period ends on November 18.

Thanks,
Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.