- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper develops an analytical framework for understanding the learning dynamics of on-policy distillation (OPD). It separates the OPD feedback loop into conversion and broadcast. Conversion characterizes how teacher supervision translates into parameter updates through accessibility and coherence, explaining why supervision of similar strength can induce different degrees of parameter change. Broadcast characterizes how these updates alter predictions and subsequent training data through drift and routing, providing a mechanistic account of degeneration driven by increasingly incoherent student-generated prefixes. Motivated by this analysis, the authors show that offline distillation can prepare students for subsequent OPD by improving their ability to continue reasoning, even when preparation yields little immediate improvement in held-out accuracy.
Critical strengths
The paper proposes a systematic analytical framework that decomposes the entire OPD training process. It provides explanations for common failure modes, including training stagnation and degeneration, and presents several interesting findings.
Critical weaknesses
Limited novelty from the methodology side. The offline-preparation intervention does not clearly demonstrate the added practical value of the proposed measurements. Although the preparation stage uses distributional distillation rather than standard hard-target SFT, it serves a similar role by training the student on teacher-generated reasoning trajectories before on-policy learning. The effectiveness of SFT-based cold starts is already well established in the community. The paper does not use its proposed measurements to improve the preparation procedure—for example, by selecting preparation data, determining its duration, or deciding when to switch to OPD. Consequently, the preparation results support the value of a familiar intervention, but provide limited evidence that the new analytical framework enables better training decisions.
Questions for authors
Please establish good connection between the proposed measurements and the training methodology.
5 comments
Response to Reviewer aaDQ, Part 2
2) Preparation and continuation readiness
Now let us move on to the measurement-to-training question you raised. We add a practical continuation-readiness probe for assessing offline preparation before OPD. Our new results show that this probe distinguishes preparation checkpoints whose held-out accuracies are nearly identical but whose subsequent OPD gains differ substantially. These findings give our mechanistic analysis a practical use, while providing complementary evidence for the important role of continuation competence in OPD’s failure and recovery.
2.1) Continuation readiness recap and practical probe
Continuation readiness measurement. In its essence, continuation readiness splits the rollout into the part generated by a prefix-generating policy, and the part by a continuation policy (to finish the rest of the rollout), discussed in Section 3.3 and Appendix E.4 and E.7. When analyzing the raw learning dynamics of the student model during OPD, the prefix generating policy is simply the continually updated student policy, as the theory and mechanistic analyses needed to estimate how the “prefix quality” and “continuation readiness” co-evolved under OPD.
Practical probe. In practice, to isolate how ready the offline-prepared checkpoints are for the subsequent OPD, we just need to fix a reference prefix-generating policy. We generate a bank of reference prefixes once, and then reuse it to evaluate the checkpoints on their terminal correctness. The teacher model is one practical reference: intuitively, we may consider its prefix population a “ceiling” on where the subsequent OPD can shift a well-prepared student’s prefix distribution towards. Of course, in practice, practitioners can use reasoning prefixes that test more specific continuation skills they are interested in.
What can this practical probe reveal and NOT reveal?. Let us first consider what this probe can NOT measure, to avoid confusion about its scope of usefulness and OPD’s failure mechanisms. This probe directly supplies teacher-generated prefixes to the student to continue (which contain coherent, useful reasoning from the teacher). In contrast, in the degenerate OPD run, the unprepared student has significant trouble learning to reach such high-quality prefixes. In fact, as we see in Figure 6 and the associated analysis in Appendix E.4 and E.7, the student's prefix distribution moves in the "opposite direction": the rollouts grow increasingly high-entropy, and the chances of reaching the correct answer from the student’s prefix distribution becomes lower and lower, with this symptom showing up at increasingly shallow rollout / prefix depth as training progresses.
Therefore, this teacher-prefix continuation probe, which freezes the prefix distribution to an already-strong one, is not designed to predict whether the subsequent OPD will simply collapse since it directly ablates out prefix-visitation shift. It is, however, suited for assessing and ranking how prepared checkpoints are for the subsequent OPD: it can help us answer "is prepared checkpoint A more competent in continuing this distribution of already-strong prefixes than checkpoint B?", and distinguish checkpoints whose immediate held-out accuracies are similar but whose subsequent OPD outcomes differ substantially due to their differing continuation-readiness.
Mechanistic connection. In sum, the progression from the mechanistic analysis of the degenerate run to the practical probe is that, the failure analysis first shows how an unprepared student can follow teacher requests locally, yet become less likely to reach the right answer as OPD degrades the reasoning states (prefixes) it visits and its continuations from them. This motivates offline preparation to improve the student’s ability to sustain and complete reasoning before exposing its own rollouts to online teacher corrections. The practical teacher-prefix probe then holds the reasoning contexts fixed and measures whether preparation improves this continuation capability, helping us assess preparation’s utility for the subsequent OPD phase.
Response to Reviewer aaDQ, Part 3
2.2) New experimental results
Results and interpretations. We focus our experimental resources on the Qwen2.5-1.5B-Instruct runs as the main “model organism”, as it was also used in our degenerate-run analysis in Section 3.3, and that its offline-preparation induced the most deceptive changes in heldout accuracy. Below are the new results, where we freeze a set of teacher prefixes (1024 tokens and non-finished) on MATH, and ask the prepared checkpoint to continue the rest of the rollout. This is repeated over two random seeds for training.
| Offline-preparation passes | Held-out accuracy before OPD | Continuation success on fixed teacher prefixes | OPD accuracy change against base model |
|---|---|---|---|
| 0 | 49.0% | 52.7% | -9.8 % |
| 1 | 45.4% | 55.5% | +1.8 % |
| 2 | 45.0% | 59.4% | +8.1 % |
| 4 | 46.4% | 64.5% | +10.2 % |
Interpretations. First, let us look at the second column, immediate heldout accuracy of the offline-prepared checkpoints. We find that one and two preparation passes give nearly the same accuracy, and even four passes do not improve beyond the base model (the 0-pass model). So from immediate heldout accuracy alone, one might naively conclude that preparation has provided little benefit, or that there is little reason to continue preparation after the first pass of training.
In contrast, the continuation-readiness probe (third column) identifies increasing readiness for OPD: it clearly distinguishes the 1- and 2-pass checkpoints despite their nearly identical heldout accuracies, and ranks the four-pass checkpoint highest. The subsequent OPD gains indeed follow this ordering. Therefore, the continuation-readiness probe identifies useful preparation that checkpoint accuracy alone would miss in our experimental sweeps.
The practical lesson is that, even when your offline-prepared model’s held-out score looks flat or worse against the base model, do not panic! Measure how well it continues prefixes from a fixed reference policy, such as the teacher. Improved continuation success rate can reveal preparation whose value emerges during subsequent OPD, beyond further offline training.
Remark. Of course, we should also be aware of the broader qualifications here: we are NOT saying that offline distillation generally cannot boost a model’s heldout accuracy at all. This classic algorithm certainly has its place, which in our Qwen2.5-7B case does show its positive effects: for this model, offline preparation gives some lift to its accuracy beyond the base model as seen in Section 3.4. The important lesson is that continuation-readiness can yield a complementary metric to the immediate heldout accuracy metric for assessing how useful the offline-preparation is: even if immediate accuracy gain is minimal, the subsequent OPD gain can be significant, and you can test for it using continuation-readiness probes!
2.3) Further mechanistic insights
These results nicely complement our analysis of the degenerate OPD run in Section 3.3. We now have complementary evidence for the continuation-readiness mechanism behind OPD’s failure and recovery.
Recall that, as the degenerate run proceeded, the student’s rollouts became high-entropy and less likely to reach the right answer. In particular, prefixes visited later during training became harder to complete correctly, with this difficulty appearing at increasingly shallower prefix depths. Our new results probing the prepared checkpoints, which holds the reference prefix distribution fixed, give complementary “positive evidence”: as continuation readiness improves with preparation, the subsequent OPD gains also improve. The continuation-readiness mechanism exposed by the failure analysis therefore gives us a useful way to assess offline preparation. Together, the failure analysis and preparation results support a central role for continuation competence in OPD’s failure and recovery.
Response to Reviewer aaDQ, Part 1
Thank you for your excellent suggestion!
To ensure that we are on the same page, we will first give a summary on the overall contributions of our paper, clarifying the paper’s insights on OPD’s learning mechanisms. We will then present a new experiment that uses continuation readiness to assess the preparation checkpoints, and show how the new results not only suggest practical measurement for OPD-readiness, but also in furthering our mechanistic insights on OPD. The results show that this measurement can distinguish checkpoints whose held-out accuracies are nearly identical, but whose subsequent OPD performance differs substantially.
1) Main contributions: mechanistic insights, diagnostics, and motivated interventions
At a very high level, the paper has two main contributions, building on two main components of the dynamics of OPD: conversion (request motion) and broadcast (motion outcome correctness).
For the conversion part, we contribute early-training diagnostics beyond the usual gradient norm or training loss: given a student model, if you train against a teacher whose request coherence is poor, or tangent-accessibility stays weak, do not expect significant student policy motion along the request field (let alone benchmark improvement), even if the gradient norm or training loss is large. This part also corroborates the industry's increasingly common choice of using RL-sibling models as teachers [1-2], where we see the RL-sibling teachers emphasizing requests on reasoning-state management / control positions, which as shown in our experiments, are much friendlier to request-to-motion conversion than the larger teachers' technical-oriented requests.
For the broadcast part, the paper's main contribution is to show the dynamical mechanism behind how ill-prepared models can be harmed by OPD, and how offline preparation improves the subsequent OPD (and rescue the failing run). The intuition for "benign" OPD is that it is similar to "tutorials", where the teacher corrects the student in real time and the student responds well. However, for an ill-prepared student, it can locally follow the teacher requests but fall into chaos in its continuations. In the real OPD runs, we see the student (never trained for long thinking rollouts) pushing itself to reason for very long under the RL’d teacher, but fails to manage its reasoning states and becomes high-entropy in its rollouts, making it difficult to rescue, let alone improve on. The natural intervention is to teach the student proper reasoning continuations from the reasoning teacher model, before exposing the student's rollouts to the teacher for finer corrections online ("go to class before going to the tutorial!"). We will also show later with our new experiments that the continuation-readiness mechanism can be used as a probe to assess offline-preparation's usefulness for the subsequent OPD phase.