acceptodds
Under review as a conference paper at ICLR 2027

RL Starts Before RL: Rethinking Policy Initialization for Outcome-Based RL

Abstract

Reinforcement learning (RL) improves reasoning, but its performance depends on the policy from which training begins. We study on-policy distillation (OPD) as a preparation stage for RL and ask whether its benefits extend beyond improvements in the distilled model's initial accuracy. Under shared RL settings, students initialized with OPD reach higher final performance than those trained with direct RL or supervised fine-tuning followed by RL. This advantage can emerge even when OPD produces little immediate improvement in accuracy. Pre-RL Pass@ does not fully explain the benefit: similar or even higher values do not necessarily lead to better performance after RL. Behavioral analyses point to alignment with the teacher's distribution beyond top-1 agreement as a possible explanation. Such alignment may favor higher-quality reasoning paths while retaining alternatives that RL can further refine using outcome feedback. We further examine how trajectory sources and divergence objectives affect the value of distillation for subsequent RL. Standard reverse-KL OPD performs better before RL, but forward-KL OPD overtakes it afterward; with teacher-generated distillation trajectories, reverse KL remains ahead at both stages. These findings suggest that the preferred distillation objective depends on both the trajectory source and the training that follows. Our results support evaluating OPD as preparation for RL and selecting distillation choices by the performance achieved after subsequent training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Reinforcement Learning, RL Initialization, Optimization Support

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YcCYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies on-policy distillation (OPD) as initialization for outcome-based RL. Using Qwen3 and Qwen3-VL student–teacher pairs, it compares direct RL, SFT followed by RL, and OPD followed by RL. OPD initialization achieves the highest final performance, even when its immediate accuracy improvement is small. The paper further shows that initial Pass@k does not fully explain downstream performance, and that forward-KL OPD can outperform reverse-KL OPD after RL despite weaker standalone performance. Behavioral analyses suggest teacher distributional alignment and retained uncertainty as possible explanations.

Critical strengths

The distinction between immediate performance and suitability for subsequent RL is interesting and practically relevant. In the text experiments, OPD improves the initial average by only 0.54 points over the base student, but its advantage grows to 7.28 points after RL. The nearly overlapping initial Pass@k curves make this observation particularly compelling. This provides useful evidence that initialization should be evaluated through the learning it enables, rather than only its current benchmark accuracy.

The comparison of distillation objectives and trajectory sources provides actionable insight. With student-generated trajectories, reverse KL performs better before RL, whereas forward KL finishes 1.85 points higher afterward. The comparison with teacher-generated trajectories further helps distinguish the effects of the supervision source and objective. These findings make the paper informative beyond simply demonstrating that an additional distillation stage can help.

Critical weaknesses

The connection to large-scale post-training pipelines remains underdeveloped. The experiments use 8B students and 32B teachers within the Qwen family. It is unclear how the proposed preparation stage should fit into MOPD-style pipelines that first train domain-specific RL teachers and then consolidate their capabilities through multi-teacher distillation. The paper also explicitly excludes preparation cost from its comparison. Consequently, the results establish a benefit under the evaluated downstream RL settings, but provide limited guidance on when adding this stage is preferable to spending the same resources on further RL or distillation. This limits the practical scope of the conclusions.

The downstream training budget and checkpoint selection need clearer reporting. Appendix A.3 states that RL stops when instability is detected, but does not clearly report the actual training lengths and checkpoint-selection rule for all comparisons. Shared hyperparameters do not necessarily imply matched training budgets. Learning curves at comparable update or token counts would help distinguish differences in learning progress from differences in training duration or stability. This is particularly relevant to the smaller VL improvements.

Questions for authors

In a pipeline with domain-specific RL teachers followed by MOPD, where would you place the proposed OPD preparation stage? Would it initialize the domain experts before RL, or prepare the consolidated student for additional RL? Please clarify which conclusions from the current experiments support that use case.

What are the actual RL update counts and checkpoint-selection criteria for each compared pipeline? Do the main rankings persist when comparing checkpoints at matched update or generated-token counts?

What is the additional cost of OPD relative to downstream RL, including teacher inference? Do you have measurements or existing results that help assess whether the gains remain worthwhile under a fixed total training budget?

Reviewer_jgekOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies on-policy distillation (OPD) as preparation for reinforcement learning. Across text and vision-language experiments, OPD initialization yields better post-RL performance than direct RL or SFT followed by RL, sometimes with little immediate accuracy gain. The paper examines initial Pass@k, teacher distributional alignment, and token entropy to understand this effect.

Critical strengths
  • The question is meaningful: immediate accuracy may not be enough to identify a good starting policy for RL.

  • The experiments support the observation that policies with similar initial performance can achieve different results after RL.

  • The reversal in the ranking of forward and reverse KL before and after RL is interesting and suggests evaluating distillation choices through subsequent training.

Critical weaknesses
  • The mechanism behind the main observation remains unclear. Closer teacher alignment and retained token uncertainty correlate with better post-RL performance, but the experiments do not isolate their contributions or establish that they cause the improvement.

  • Similar aggregate accuracy or Pass@k does not sufficiently control for differences between starting policies. These policies may solve different problems or follow different reasoning paths. The analysis therefore leaves open which properties actually account for their different responses to RL.

  • The training comparison needs more detail. Actual RL steps, stopping points, checkpoint selection, and results across training seeds are not sufficiently reported. This makes it difficult to distinguish faster learning, higher attainable performance, and better compatibility with the chosen training configuration.

Overall, the empirical observation is useful, but the investigation does not yet go far enough to explain why OPD provides a better starting point for RL.

Questions for authors
  • Can controlled experiments separate the effects of teacher distributional alignment and retained uncertainty on subsequent RL performance?

  • Can the authors provide full RL learning curves, training steps, checkpoint selection criteria, and results across multiple training seeds?

Reviewer_iJwsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates on-policy distillation as a preparation stage for reinforcement learning, examining whether it improves downstream RL performance beyond simply increasing initial model accuracy. A key finding is that initial accuracy and Pass@k do not reliably predict downstream RL performance. The authors investigate teacher-distribution alignment and retained token entropy as possible explanations for OPD's benefits. the paper provides empirical evidence that distillation strategies should be evaluated based on their effects on subsequent RL

Critical strengths
  • Convincing evidence that OPD improves downstream RL beyond initial accuracy gains.
  • The comparison of forward versus reverse KL and student-generated versus teacher-generated trajectories provides valuable insights into distillation design.
Critical weaknesses
  • many short paragraphs (often a few sentences)
  • missing related work on hybrid RL + OPD (e.g. https://openreview.net/forum?id=vXVjbbcuHy)
  • The authors hypothesize that improved teacher-distribution alignment combined with retained uncertainty creates a more favorable solution space for subsequent RL. While the diagnostic results are consistent with this interpretation, they do not establish that these properties cause the observed improvements.
Questions for authors
  • https://arxiv.org/abs/2610.40564 could be an interesting discussion for the paper (this paper was published after the ICLR deadline, though)
  • Do the observed differences in teacher-distribution alignment and token entropy persist when all policies are evaluated on the same held-out prefixes?
  • What is the additional computational cost of OPD relative to SFT and direct RL, particularly given the need to obtain full teacher distributions? Does OPD retain its advantage when comparing pipelines under approximately matched total computational budgets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.