acceptodds
Under review as a conference paper at ICLR 2027

Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation

Abstract

Policy-shift distillation reuses the difference between a teacher's log probabilities before and after reinforcement learning (RL) as supervision on student-generated trajectories. However, this endpoint policy shift cannot reveal whether local action preferences evolved consistently during teacher training. The same endpoint shift can result from early and late shifts that reinforce or partially cancel each other. We investigate whether agreement between early and late teacher policy shifts can guide selective transfer. We introduce Waypoint-guided On-Policy Distillation (Waypoint-OPD), which uses an intermediate teacher checkpoint from the same RL run to compare early and late shifts in relative action preferences at student-visited prefixes. We gate the endpoint shift according to the directional agreement between the early and late shifts, suppressing it when they point in opposing directions.Compared with endpoint-only transfer, our method requires only one additional frozen-teacher scoring pass. Using RL-trained Qwen3-4B teachers, we evaluate our method with Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B as student models in mathematical and code reasoning. In all six domain–student settings, Waypoint-OPD achieves the highest domain-average score among the compared distillation methods. Ablations with the Qwen3-8B student show that removing or shuffling gates reduces mathematical reasoning performance. These results support the use of directional agreement between early and late teacher policy shifts to weight the endpoint shift during student training.

Primary area:
generative models
Keywords:
Large Language Models, On-policy Distillation, Policy-Shift Transfer

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_vVDqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Waypoint-OPD, which transfers a teacher's RL-driven policy changes via an intermediate checkpoint from the same training run. For each student-generated prefix, it measures how the teacher's preferences over the student's most likely next tokens shifted in the first versus second half of training. When the two shifts agree in direction, the full start-to-finish change is trusted; when they conflict, it is discounted or dropped. This agreement signal gates the initial-to-final policy shift, and the student is trained on dense top-k candidate scores rescaled within each batch, with a KL penalty anchoring it to its initialization.

Critical strengths

A focused and clearly specified methodological contribution. The intermediate checkpoint provides information unavailable from the endpoint pair while preserving the endpoint shift as the transferred signal. The method is straightforward to implement and requires only one additional frozen-teacher scoring pass, making it useful for reusing existing RL checkpoints.

Critical weaknesses
  • The gate helps, but why it helps remains unclear. The no-gate and shuffled-gate experiments provide useful evidence that the proposed weighting works. However, they do not show whether temporal agreement is better than simpler choices, such as weighting signals by their magnitude or the student’s confidence. They also do not separately test whether the benefit comes from removing conflicting signals or reducing the weight of partially aligned signals.

  • The reported gains need uncertainty estimates. The mathematical average improves over Direct-OPD by 1.59–2.18 percentage points, but the paper provides no confidence intervals or variation across repeated training runs. Generating 32 answers per problem does not address training variability or the limited number of distinct problems. AIME24 is also used to select checkpoints and included in the reported average. The learning curves and improvements on the remaining benchmarks are encouraging, but do not fully resolve these concerns.

Questions for authors
  • How does temporal-agreement gating compare with simpler alternatives based on signal magnitude or student confidence? Do you have evidence that the temporal information provides an additional benefit?
  • Does the gain mainly come from removing signals with nonpositive agreement, or from downweighting partially aligned signals? Can these two effects be tested separately?
  • How many independent training runs were performed? Can you report variation across runs and paired, problem-level confidence intervals for the main improvements?
  • Since AIME24 is used for checkpoint selection, do the improvements hold when it is excluded from the test average? Do they also hold with a fixed final checkpoint or a separate validation set?
Reviewer_yGrWOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Waypoint-OPD, which is an extension of on-policy distillation that uses an intermediate teacher checkpoint to selectively transfer RL-induced policy changes. At student-generated prefixes, the method compares the teacher's early and late changes in relative token prferences and use their positive cosine similarity to weight the init-to-final policy shift. This doesn't require any verifier and only requires one additional frozen-teacher scoring pass. On a wide range of math and coding benchmarks, Waypoint-OPD reaches the highest avg performance compared to other on-policy distillation methods.

Critical strengths
  • The mathematical formulation is clear and easy to understand, and the method shows good empirical performance.
  • The ablation experiments are well chosen too, such as removing or shuffling gates reducing performance and that the improvesments persist without batch norm and across nearby waypoint choices.
  • The method is cheap to employ since it doesn't require more student rollouts or an additional verifier.
Critical weaknesses
  • There should be baselines such as simpler confidence, entropy, or magitude based gating added.
  • Maybe this is typical in RL papers these days, but I still think it is bad practice to only report performance on Qwen3 models and only covering math & code. What about other models and other domains?
  • The "Codeforces-derived" dataset should be explained more clearly. What are the baseline-specific hyperparameters, tuning budgets, and PPO setting for this?
Questions for authors

see weaknesses above

Reviewer_NFGQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Waypoint-OPD to transfer RL-induced policy shifts from a smaller teacher to larger student models via on-policy distillation. Because the endpoint shift (log⁡π1−log⁡π0\log \pi_1 - \log \pi_0) alone cannot tell whether the teacher's action preferences evolved consistently or oscillated during RL, the authors introduce an intermediate teacher checkpoint from the same RL run and gate the endpoint shift by the positive weighted cosine similarity between early and late shifts over the student's top-kk candidate tokens. They evaluate the method by distilling Qwen3-4B RL teachers into Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B students on math and code benchmarks. Main results show consistent improvements over standard OPD, OPD$^2$, and Direct-OPD across all six student-domain settings, with ablations validating the roles of prefix-specific gating and batch normalization.

Critical strengths
  • The proposed method is intuitive, very easy to implement on top of existing policy-shift distillation pipelines, and adds minimal computational cost.
  • The presentation is clear and easy to follow.
  • The ablations and diagnostic analyses are thorough and well-executed, including gate removal, within-batch gate shuffling, batch-scale normalization, and waypoint step sensitivity.
Critical weaknesses
  • The methodological novelty is somewhat incremental, as it mainly combines the endpoint policy-shift transfer of Direct-OPD with a gate from an intermediate checkpoint.
  • The empirical gains over the closest baseline, Direct-OPD, are relatively modest (around 1.2–2.6 percentage points across settings). Since student models are trained for only 100 steps with a single seed and checkpoint selection is done on a small 30-problem set (AIME 2024), some of this difference could fall within training noise.
  • Using the final checkpoint as the transfer target and gating against a single midpoint implicitly assumes that the teacher's RL trajectory is fairly smooth. If the teacher's RL process is non-smooth (e.g., involving late-stage degradation), it is unclear whether the final checkpoint remains a good reference or whether suppressing prefixes with opposing early and late shifts might discard valid late-stage corrections.
  • More generally for weak-to-strong OPD methods, since this approach still requires on-policy rollouts and policy gradient updates on the large student model, it is unclear how much the small 4B teacher's capability ceiling ultimately constrains the student's performance, and how this compares to simply running RL directly on the large student under a matched compute budget.
Questions for authors
  • Since Waypoint-OPD still trains the large student with on-policy rollouts (plus the upfront 1,200-step GRPO cost of the 4B teacher), how does its performance compare to directly running GRPO on the large student under the same compute budget?
  • Could you clarify the checkpoint selection protocol in Section 4.1 (why AIME 2024 is used to select checkpoints for both math and code transfer)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.