- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
In this paper, the authors propose Semi-OPD, a simplified alternative to on-policy distillation (OPD). Rollouts are sampled once from the initial student, scored offline by the teacher, and reused throughout training with the sampled-token reverse-KL objective; student log-probabilities are recomputed at each step. Specifically, the paper shows that the benefit of strict on-policy sampling depends on the initial teacher–student alignment. For misaligned pairs, OPD rollouts grow longer and drift off-policy with respect to the teacher, which weakens the distillation signal. Semi-OPD instead trains on shorter, full-coverage trajectories with more teacher-preferred tokens. Additionally, across 17 teacher–student pairs, Semi-OPD outperforms OPD on 14 pairs, with up to +13.6% accuracy on AIME 24/25/26 and HMMT Feb 26 and up to 11.4× lower training cost.
Critical strengths
1.Timely and practically relevant question: The paper asks when OPD’s online sampling is actually necessary, rather than treating it as a default. The proposed overlap-based criterion gives practitioners a cheap, measurable signal for choosing between online and offline student rollouts.
2.Broad and systematic empirical study: 17 teacher–student pairs spanning three model families, model sizes up to 235B, and both reasoning modes are evaluated under matched prompts, objective, and hyperparameters.
3.Insightful mechanistic analysis: The two-sided “on-policyness” view is supported by several complementary diagnostics: length and self-correction inflation, the fraction of tokens, teacher/student entropy, rollout-cap and prefix-coverage ablations, and teacher-rollout controls. The view also offers a coherent explanation for known OPD failure modes.
Critical weaknesses
1. Why use initial student weight for rollout? The paper unjustified choice of the initial student as the rollout source and missing staleness ablation. Specifically, the paper doesn't explains in a principled way why is the right anchor. Rollouts from teacher, SFT-initialized, and base students are compared, but there is no experiment using rollouts from intermediate checkpoints or periodically refreshed caches. This is the natural continuum between Semi-OPD and OPD, and without it one cannot tell whether “freezing at ” is special or merely one point on a staleness trade-off curve.
2. The most critical concern with Semi-OPD itself: its objective is not equivalent to OPD’s, and its difference from a modified SFT is unclear. The paper claims that the objective is identical to OPD’s but provides no formal statement or proof. In OPD, it is sampling from the current student that makes the sampled-token loss a reverse-KL gradient estimator. With rollouts fixed at , the expected gradient instead corresponds to a squared log-probability regression toward the teacher on ’s samples, and it matches the reverse-KL gradient only at the first update. Semi-OPD therefore performs a teacher-guided log-likelihood regression on frozen self-generated data rather than on-policy distillation. Recomputing the student’s log-probabilities at each step does not restore on-policyness, since any SFT gradient is also evaluated at the current parameters. The authors should formally characterize the objective Semi-OPD actually optimizes and its fixed point, and compare against equivalent SFT-style baselines, e.g., weighted SFT with fixed initial weights or teacher-scored rejection-sampling SFT.
3. Overlap–gain correlation is potentially confounded and weakly actionable: The 17 pairs are not independent. Low-overlap pairs are almost all Qwen3 instruct/thinking pairs and high-overlap pairs are mostly R1-Distill → RL-descendant pairs, so the correlation may largely reflect model family, training recipe, or mode. Can authors provide further exp or statements to justify their claims.
Questions for authors
Please address my concerns shown in Weaknesses, especially the problem in Point 1&2.
7 comments
W3: The overlap–gain correlation may be confounded by model family, training recipe, or reasoning mode.
We would like to clarify that the overlap regimes are not perfectly separated by model family, and that our evidence goes beyond cross-family correlation.
- Within a single family. Within the Qwen3 family alone, initial overlap ranges from about 57% to 70%, and lower-overlap pairs generally achieve larger Semi-OPD gains across the 11 Qwen3 pairs. Thus, the trend is not driven solely by comparing Qwen3 with other model families.
- Within the same teacher–student pair. More importantly, we directly change the student initialization within the same teacher–student pairs in Fig. 4. After SFT on teacher rollouts, the overlap increases slightly and the gap between Semi-OPD and OPD becomes smaller, although Semi-OPD still performs better. This shows that the relative performance changes even when the model family and teacher are fixed.
We also agree that the overlap ratio alone is not a gold metric. Our key insight is that, for teacher–student pairs with misaligned thinking patterns, on-policy rollouts can offer less benefit or even hurt accuracy. There might be better metrics to quantify this, but our general insight can still guide the choice of distillation method in different ways:
- For teacher–student pairs with very low alignment, Semi-OPD generally outperforms OPD or performs comparably, making it a good choice.
- In MOPD settings for foundation model development, such as DeepSeek 4 and GLM5, the teachers are domain specialists trained from the student model. The overlap ratio is usually high in these settings, so it is safe to proceed directly with OPD.
- The choice between Semi-OPD and OPD requires further investigation only for teacher–student pairs with moderate overlap ratios. A practical approach is to train with each method for a short run, such as 40–80 steps, compare their accuracies, and use the better-performing method for longer training.
We hope the above clarifications and experiments address your questions. Please let us know if you have further questions.
my major concern is mainly the point 1 and point 2, I think your clarification address my first question. I still don't think your clarification fully solve my second question (i.e., the weakness 2). My major issue is not the issue of who generate the initial rollout (like what you point Fig. 4 where teacher generate). My major concerns is that once use samples from the initial student, the whole process function the same as SFT. That is why I would like to know whether you could provide some math proof or further statement. I still decide to maintain my overall score.
Thank you for finding our question timely and practically relevant, our study broad and systematic, and our mechanistic analysis insightful. Below we address your concerns.
W1: Why use initial-student rollouts? Missing intermediate-checkpoint and periodic-refreshing experiments.
We would like to clarify that we make no claim that freezing initial-student rollouts is optimal for every teacher–student pair. Our goal is to challenge the necessity of strict on-policy sampling in OPD and to identify principles for deciding when to use OPD.
As such, our study first compares the two extremes of student rollout refreshing: OPD refreshes the rollouts at every step, while Semi-OPD never refreshes them. We find that Semi-OPD outperforms OPD on 14 of 17 pairs, and that teacher–student alignment is key to determining when OPD should be used.
We agree that periodic refreshing is a natural middle ground. Our submitted paper already includes a closely related experiment that switches from Semi-OPD checkpoints to OPD mid-training (Sec. 5, Figs. 9 and 18). There we found an interesting observation that OPD converges to the same level as OPD trained from scratch, regardless of its initialization. To provide a more complete picture,*we additionally ran periodic refreshing, where the rollout cache is regenerated from the current student every N steps (N = 1 is OPD; never refreshing is Semi-OPD), on a misaligned pair and a highly aligned pair:
This result aligns with our switching results in paper: once online rollouts are reintroduced, training converges to the same level as OPD. Our hypothesis is that the training rollout distribution, rather than the starting checkpoint, determines where distillation converges; characterizing this equilibrium is an interesting direction for future work. We will add these results to the paper.
W2: Semi-OPD's objective is not equivalent to OPD's, and its difference from modified SFT is unclear.
Thank you for pointing this out. We believe there was a misunderstanding about what we meant by "the same objective." Our point is that Semi-OPD and OPD use the same loss implementation for optimization, with both computing the reverse KL on sampled tokens. We fully agree that their optimization procedures differ in how they obtain the rollouts: OPD optimizes the model on fresh rollouts at every step, while Semi-OPD optimizes it on the initial-student rollouts. We intentionally keep all other factors the same, because this work aims to study whether on-policy rollouts are necessary.
In other words, our study focuses on the rollout source, and we do not claim that Semi-OPD is fully on-policy, as its name suggests. Following the same principle, we also compare against off-policy distillation on teacher-generated rollouts using the same loss (Fig. 5), and Semi-OPD outperforms it as well. We will clarify this wording in Sec. 2.2.
Thank you also for suggesting that we clarify how Semi-OPD differs from SFT variants. The key differences lie in the training objective and the training data:
Both OPD and Semi-OPD can thus be viewed as forms of SFT with specific choices of training objective and data generation procedure, and better combinations may exist. In this work, Semi-OPD uses rollouts generated by the initial student and keeps all other aspects the same as OPD, which serves the paper's goal of studying whether on-policy rollouts are necessary.
In addition, we have studied SFT on teacher rollouts as an initialization for Semi-OPD and OPD (Fig. 4). SFT slightly increases overlap but does not close the gap: Semi-OPD still outperforms OPD, and Semi-OPD without the additional SFT stage performs better.