acceptodds
Under review as a conference paper at ICLR 2027

When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better

32 views

Abstract

On-policy distillation (OPD) has become increasingly popular for transferring teacher capabilities to student models. In this work, we ask: Is on-policy sampling always beneficial for distilling arbitrary teacher–student pairs? We show that a simple alternative, Semi-OPD, which distills from offline rollouts generated by the initial student, can often outperform OPD in both accuracy and training efficiency. Across 17 teacher–student pairs ranging from 1.5B to 235B parameters, Semi-OPD outperforms OPD in 14 cases, with up to +13.6% accuracy and 11.4x training speedup. It also beats other OPD variants on low-overlap pairs. We further find that the advantage of Semi-OPD strongly depends on the alignment between the initial teacher and student, quantified by an output-token overlap ratio: OPD is beneficial only when the two are highly aligned. Our study suggests that effective distillation requires on-policyness w.r.t. both the student and the teacher. For misaligned pairs, student rollouts can become increasingly off-policy w.r.t. the teacher as context length grows, weakening the distillation signal. In contrast, Semi-OPD is often more stable, as it distills on shorter contexts while covering all reasoning stages and exposing the student to more teacher-preferred tokens. Our work motivates rethinking when to use OPD based on teacher–student overlap and adopting Semi-OPD as an efficient and strong alternative when appropriate.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, knowledge distillation, large language models, reasoning, LLM post-training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UjpAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In this paper, the authors propose Semi-OPD, a simplified alternative to on-policy distillation (OPD). Rollouts are sampled once from the initial student, scored offline by the teacher, and reused throughout training with the sampled-token reverse-KL objective; student log-probabilities are recomputed at each step. Specifically, the paper shows that the benefit of strict on-policy sampling depends on the initial teacher–student alignment. For misaligned pairs, OPD rollouts grow longer and drift off-policy with respect to the teacher, which weakens the distillation signal. Semi-OPD instead trains on shorter, full-coverage trajectories with more teacher-preferred tokens. Additionally, across 17 teacher–student pairs, Semi-OPD outperforms OPD on 14 pairs, with up to +13.6% accuracy on AIME 24/25/26 and HMMT Feb 26 and up to 11.4× lower training cost.

Critical strengths

1.Timely and practically relevant question: The paper asks when OPD’s online sampling is actually necessary, rather than treating it as a default. The proposed overlap-based criterion gives practitioners a cheap, measurable signal for choosing between online and offline student rollouts.

2.Broad and systematic empirical study: 17 teacher–student pairs spanning three model families, model sizes up to 235B, and both reasoning modes are evaluated under matched prompts, objective, and hyperparameters.

3.Insightful mechanistic analysis: The two-sided “on-policyness” view is supported by several complementary diagnostics: length and self-correction inflation, the fraction of tokens, teacher/student entropy, rollout-cap and prefix-coverage ablations, and teacher-rollout controls. The view also offers a coherent explanation for known OPD failure modes.

Critical weaknesses

1. Why use initial student weight for rollout? The paper unjustified choice of the initial student as the rollout source and missing staleness ablation. Specifically, the paper doesn't explains in a principled way why π0\pi_0 is the right anchor. Rollouts from teacher, SFT-initialized, and base students are compared, but there is no experiment using rollouts from intermediate checkpoints πk\pi_k or periodically refreshed caches. This is the natural continuum between Semi-OPD and OPD, and without it one cannot tell whether “freezing at π0\pi_0” is special or merely one point on a staleness trade-off curve.

2. The most critical concern with Semi-OPD itself: its objective is not equivalent to OPD’s, and its difference from a modified SFT is unclear. The paper claims that the objective is identical to OPD’s but provides no formal statement or proof. In OPD, it is sampling from the current student that makes the sampled-token loss a reverse-KL gradient estimator. With rollouts fixed at y∼π0y \sim \pi_0, the expected gradient instead corresponds to a squared log-probability regression toward the teacher on π0\pi_0’s samples, and it matches the reverse-KL gradient only at the first update. Semi-OPD therefore performs a teacher-guided log-likelihood regression on frozen self-generated data rather than on-policy distillation. Recomputing the student’s log-probabilities at each step does not restore on-policyness, since any SFT gradient is also evaluated at the current parameters. The authors should formally characterize the objective Semi-OPD actually optimizes and its fixed point, and compare against equivalent SFT-style baselines, e.g., weighted SFT with fixed initial weights or teacher-scored rejection-sampling SFT.

3. Overlap–gain correlation is potentially confounded and weakly actionable: The 17 pairs are not independent. Low-overlap pairs are almost all Qwen3 instruct/thinking pairs and high-overlap pairs are mostly R1-Distill → RL-descendant pairs, so the correlation may largely reflect model family, training recipe, or mode. Can authors provide further exp or statements to justify their claims.

Questions for authors

Please address my concerns shown in Weaknesses, especially the problem in Point 1&2.

7 comments

Thank you for finding our question timely and practically relevant, our study broad and systematic, and our mechanistic analysis insightful. Below we address your concerns.

W1: Why use initial-student rollouts? Missing intermediate-checkpoint and periodic-refreshing experiments.

We would like to clarify that we make no claim that freezing initial-student rollouts is optimal for every teacher–student pair. Our goal is to challenge the necessity of strict on-policy sampling in OPD and to identify principles for deciding when to use OPD.

As such, our study first compares the two extremes of student rollout refreshing: OPD refreshes the rollouts at every step, while Semi-OPD never refreshes them. We find that Semi-OPD outperforms OPD on 14 of 17 pairs, and that teacher–student alignment is key to determining when OPD should be used.

We agree that periodic refreshing is a natural middle ground. Our submitted paper already includes a closely related experiment that switches from Semi-OPD checkpoints to OPD mid-training (Sec. 5, Figs. 9 and 18). There we found an interesting observation that OPD converges to the same level as OPD trained from scratch, regardless of its initialization. To provide a more complete picture,*we additionally ran periodic refreshing, where the rollout cache is regenerated from the current student every N steps (N = 1 is OPD; never refreshing is Semi-OPD), on a misaligned pair and a highly aligned pair:

Pair (initial overlap)BenchmarkSemi-OPD (never)N = 100N = 50N = 10OPD (N = 1)
Qwen3-1.7B ← Qwen3-4B-Inst (62.5%)AIME 24/2542.432.733.434.133.8
4-benchmark avg37.628.829.029.429.1
R1-Distill-7B ← Polaris-7B (86.8%)AIME 24/2556.556.056.856.056.4
4-benchmark avg50.850.851.350.851.1
  • Misaligned pair: Semi-OPD remains the best, and every refresh interval converges to the OPD level.
  • Aligned pair: all methods are comparable, consistent with our finding that OPD is safe for aligned pairs.

This result aligns with our switching results in paper: once online rollouts are reintroduced, training converges to the same level as OPD. Our hypothesis is that the training rollout distribution, rather than the starting checkpoint, determines where distillation converges; characterizing this equilibrium is an interesting direction for future work. We will add these results to the paper.

W2: Semi-OPD's objective is not equivalent to OPD's, and its difference from modified SFT is unclear.

Thank you for pointing this out. We believe there was a misunderstanding about what we meant by "the same objective." Our point is that Semi-OPD and OPD use the same loss implementation for optimization, with both computing the reverse KL on sampled tokens. We fully agree that their optimization procedures differ in how they obtain the rollouts: OPD optimizes the model on fresh rollouts at every step, while Semi-OPD optimizes it on the initial-student rollouts. We intentionally keep all other factors the same, because this work aims to study whether on-policy rollouts are necessary.

In other words, our study focuses on the rollout source, and we do not claim that Semi-OPD is fully on-policy, as its name suggests. Following the same principle, we also compare against off-policy distillation on teacher-generated rollouts using the same loss (Fig. 5), and Semi-OPD outperforms it as well. We will clarify this wording in Sec. 2.2.

Thank you also for suggesting that we clarify how Semi-OPD differs from SFT variants. The key differences lie in the training objective and the training data:

  1. Training objective. Semi-OPD uses the reverse KL to the teacher model, while standard SFT typically uses the cross-entropy loss. Forward KL can also be used for SFT with distillation.
  2. Training data. Semi-OPD trains on rollouts generated by the initial student, which can be viewed as semi-on-policy. Standard SFT typically trains on trajectories from human experts or other models, while rejection-sampling SFT trains on student rollouts that pass verification.

Both OPD and Semi-OPD can thus be viewed as forms of SFT with specific choices of training objective and data generation procedure, and better combinations may exist. In this work, Semi-OPD uses rollouts generated by the initial student and keeps all other aspects the same as OPD, which serves the paper's goal of studying whether on-policy rollouts are necessary.

In addition, we have studied SFT on teacher rollouts as an initialization for Semi-OPD and OPD (Fig. 4). SFT slightly increases overlap but does not close the gap: Semi-OPD still outperforms OPD, and Semi-OPD without the additional SFT stage performs better.

W3: The overlap–gain correlation may be confounded by model family, training recipe, or reasoning mode.

We would like to clarify that the overlap regimes are not perfectly separated by model family, and that our evidence goes beyond cross-family correlation.

  • Within a single family. Within the Qwen3 family alone, initial overlap ranges from about 57% to 70%, and lower-overlap pairs generally achieve larger Semi-OPD gains across the 11 Qwen3 pairs. Thus, the trend is not driven solely by comparing Qwen3 with other model families.
  • Within the same teacher–student pair. More importantly, we directly change the student initialization within the same teacher–student pairs in Fig. 4. After SFT on teacher rollouts, the overlap increases slightly and the gap between Semi-OPD and OPD becomes smaller, although Semi-OPD still performs better. This shows that the relative performance changes even when the model family and teacher are fixed.

We also agree that the overlap ratio alone is not a gold metric. Our key insight is that, for teacher–student pairs with misaligned thinking patterns, on-policy rollouts can offer less benefit or even hurt accuracy. There might be better metrics to quantify this, but our general insight can still guide the choice of distillation method in different ways:

  1. For teacher–student pairs with very low alignment, Semi-OPD generally outperforms OPD or performs comparably, making it a good choice.
  2. In MOPD settings for foundation model development, such as DeepSeek 4 and GLM5, the teachers are domain specialists trained from the student model. The overlap ratio is usually high in these settings, so it is safe to proceed directly with OPD.
  3. The choice between Semi-OPD and OPD requires further investigation only for teacher–student pairs with moderate overlap ratios. A practical approach is to train with each method for a short run, such as 40–80 steps, compare their accuracies, and use the better-performing method for longer training.

We hope the above clarifications and experiments address your questions. Please let us know if you have further questions.

my major concern is mainly the point 1 and point 2, I think your clarification address my first question. I still don't think your clarification fully solve my second question (i.e., the weakness 2). My major issue is not the issue of who generate the initial rollout (like what you point Fig. 4 where teacher generate). My major concerns is that once use samples from the initial student, the whole process function the same as SFT. That is why I would like to know whether you could provide some math proof or further statement. I still decide to maintain my overall score.

Reviewer_PqhQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether student responses really need to be regenerated throughout distillation, and studies Semi-OPD: generate responses once from the initial student, score them with the teacher offline, and reuse them while updating the student’s probabilities. Semi-OPD achieves higher accuracy averaged over training in 14 of 17 model pairs, and experiments on token overlap, response length, and trajectory coverage suggest that fresh rollouts can become less useful when the student and teacher disagree.

Critical strengths

The main comparison is well controlled: both methods use the same prompts in the same order, with the same teacher and optimization settings. The results also give useful evidence that continually refreshing student responses is not always beneficial.

The compute savings include generating the offline responses and scoring them with the teacher. Semi-OPD still uses substantially fewer GPU-hours than FastOPD, with a 2.2–2.7× saving in Table 4.

I found the full-trajectory ablation in Fig. 5 particularly useful: complete responses beat prefixes even when the prefixes receive more rollouts to match the token budget. This helps explain why keeping complete initial responses can work better than simply cutting off later tokens.

Critical weaknesses

The overlap result is interesting, but it does not yet tell me how to choose a method for a new model pair. Fig. 11 shows that low overlap can still favor OPD, and the paper gives no concrete way to check whether the initial responses are good enough for Semi-OPD.

I’m less convinced by the interpretation of the pT > pS fraction in Fig. 7. This fraction can increase because student probabilities are lower, so it does not necessarily mean that the teacher finds the cached responses more likely or provides better supervision on them.

The paper compares refreshing responses at every step with never refreshing them, but does not test occasional refreshes. Without that comparison, it is hard to tell whether keeping the initial responses matters, or whether updating them less often would be enough.

Questions for authors

Is there a simple measure of initial response quality that, together with overlap, would identify cases like the base student in Fig. 11 before training?

Does the teacher assign higher likelihood to cached responses than to OPD responses when their lengths are matched?

Would refreshing the cache once per epoch retain the gains? This would also help explain where Semi-OPD fits relative to the offline and asynchronous KD methods discussed in Sec. 6.

Both objectives tested here use reverse KL, so I wonder how much the benefit of cached rollouts depends on the choice of loss and supervision. Connecting the findings to broader KD work on loss design, probability-mass allocation, and hard/soft labels [1–4] would help clarify where the conclusions might carry over; a conceptual discussion would be enough.

[1] DistiLLM: Towards Streamlined Distillation for Large Language Models. ICML, 2024.

[2] DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs. ICML, 2025.

[3] ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergence. ICML, 2025.

[4] The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Works. ICML, 2026.

3 comments

Thank you for reviewing our paper and finding our main comparison well controlled, our cost accounting complete, and the full-trajectory ablation useful. Below we address your concerns, including a new periodic-refresh experiment (W3/Q3) and new analyses of teacher–student closeness (W2/Q2) and the distillation objective (Q4).

W1/Q1: Overlap alone does not tell how to choose a method. Is there a simple measure of initial rollout quality?

Thank you for the suggestion. We agree that overlap alone is not sufficient, and that a single metric combining teacher–student overlap with initial rollout quality would make choosing between OPD and Semi-OPD easier. Building and validating such a metric is a good direction for future work. Our results already identify two factors that matter:

  • Teacher–student alignment. Initial overlap is strongly correlated with Semi-OPD’s relative gain across 17 pairs (Fig. 2a).
  • Initial rollout quality. Semi-OPD underperforms OPD when the student is a base model, but outperforms OPD when it is instruction-tuned (Fig. 11). Semi-OPD therefore needs initial trajectories that already contain useful reasoning. A simple, nearly free proxy is the accuracy of the cached initial rollouts. They are generated before training anyway and can be checked with the same verifier.

Together, these factors give practitioners concrete guidance:

  • Low overlap and a capable initial student: Semi-OPD performs better than OPD or comparably, at a much lower cost, so it is the better choice.
  • High overlap: This is common in multi-teacher OPD (MOPD) for foundation-model training, such as DeepSeek-V4 and GLM-5, where the teachers are domain specialists trained from the student. Online rollouts stay well aligned with the teacher, so using OPD directly is safe.
  • Moderate overlap, or a weak initial student: A short pilot run of each method (40–80 steps) followed by a comparison of accuracy is a reliable and cheap way to choose. It costs only a small fraction of full training.

We will revise the paper to present overlap as a diagnostic rather than a sufficient condition, and add this guidance.

W2/Q2: A higher fraction of tokens with pT>pSp_T > p_S does not necessarily indicate better supervision.

We agree that this fraction alone does not imply better supervision. We view it as relative evidence that the teacher agrees more with the token choices in initial-student rollouts than in online rollouts. To test this without relying on either method's training rollouts, we ran an additional analysis: we evaluate checkpoints on two fixed text sets generated by teachers and measure the mean per-token (log⁡pS−log⁡pT)2(\log p_S - \log p_T)^2 (lower means the student is closer to the teacher), on Qwen3-1.7B ← Qwen3-4B-Inst:

Evaluation textUntrainedOPDSemi-OPD
Initial-student rollouts (Semi-OPD's training data)2.381.591.46
Teacher-generated rollouts (held out from both methods)1.581.581.32

On the teacher's own responses, which neither method trains on, OPD does not move the student closer to the teacher, while Semi-OPD does. Thus, the higher pT>pSp_T > p_S fraction is not explained only by lower student probabilities: Semi-OPD moves the student toward the teacher's distribution, even on text it never trained on.

W3/Q3: Would occasional refreshing (e.g., once per epoch) retain the gains?

Thank you for this suggestion. Our paper already studies a closely related setting: switching from Semi-OPD to OPD mid-training (Sec. 5, Figs. 9 and 18). There, OPD converges to the same level as OPD trained from scratch, regardless of its initialization. Following your suggestion, we additionally ran periodic refreshing, where the rollout cache is regenerated from the current student every N steps (N = 1 is OPD; never refreshing is Semi-OPD; N = 100 is roughly once per epoch), on a misaligned and a highly aligned pair:

Pair (initial overlap)BenchmarkSemi-OPD (never)N = 100N = 50N = 10OPD (N = 1)
Qwen3-1.7B ← Qwen3-4B-Inst (62.5%)AIME 24/2542.432.733.434.133.8
4-benchmark avg37.628.829.029.429.1
R1-Distill-7B ← Polaris-7B (86.8%)AIME 24/2556.556.056.856.056.4
4-benchmark avg50.850.851.350.851.1
  • Misaligned pair: Semi-OPD remains the best. Even once-per-epoch refreshing converges to the OPD level.
  • Aligned pair: all methods are comparable, consistent with our finding that OPD is safe for aligned pairs.

This confirms our switching results: once online rollouts are reintroduced, training converges to the same level as OPD. Our hypothesis is that the training rollout distribution, rather than the starting checkpoint, determines where distillation converges. Characterizing this equilibrium is an interesting direction for future work.

Q4: How much does the benefit of cached rollouts depend on the distillation objective?

Yes, we also checked whether our findings depend on the objective, we compare the sampled-token reverse KL used throughout the paper with a top-kk reverse KL (Fig. 8; Qwen3-1.7B ← Qwen3-4B-Inst, AIME 24/25 avg@32):

ObjectiveMethodStep 40Step 100
Sampled-token reverse KLOPD35.534.6
Semi-OPD45.544.0
Top-kk reverse KLOPD36.136.0
Semi-OPD41.839.1

Semi-OPD outperforms OPD under both objectives, so the benefit of cached rollouts is not specific to the sampled-token loss, although the gap is larger with the sampled-token objective.

Q5: Connection to broader KD work.

Thank you for these references. [1–4] study how to supervise each token (the divergence or label type), while we study which trajectories to supervise. The two choices are largely complementary, and some of these works are consistent with our findings:

  • DistiLLM [1] replays stale student outputs and treats staleness as an efficiency trade-off with a small accuracy cost. We show that under teacher–student misalignment, staleness can instead improve accuracy.
  • Bridge-Garden [4] outperforms on-policy KD and attributes the gain to reduced exposure bias rather than closer teacher matching. Similarly, we find that mitigating exposure bias does not require strict on-policy sampling.

These loss designs can be combined directly with Semi-OPD's choice of rollouts. We will add this discussion to the related work section.

We hope the above clarifications and experiments address your questions. Please let us know if you have further questions.

Thank you for the detailed responses and revisions. They address most of my concerns, and I will maintain my positive score.

Reviewer_CL46OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Semi-OPD, a variant of on-policy distillation in which student-generated responses are collected once from the initial student model and reused throughout training rather than regenerated after each update. The central contribution is a practical argument for choosing between online and offline student rollouts based on teacher–student compatibility, however, the evidence is primarily empirical and the proposed explanation and overlap-based decision rule require more rigorous validation before they can be treated as general principles.

Critical strengths
  1. Paper is very well written, the proposed analysis and approach is simple and practical.

  2. The paper has strong empirical results supporting their alignment intuition. Their analysis includes multiple ablations and transfer analysis.

Critical weaknesses
  1. The paper's novelty and contribution is very limited. It is mainly more of proposing a metric that in many cases showed to be helpful and insightful for making decisions of when do OPT and when switch to initial rollouts. The extendability is hard to know even though the authors did a big ablations but at the end, it is all based on some intuition and no concrete result explaining the clear causation of the problem.

  2. Results in many cases are very close to each other and the tables lack any uncertainty estimation like reported variance, making it hard to understand how meaningful all differences are.

  3. Nitpicking but paper makes general claims while their analysis and the contribution of the paper is not that solid. I suggest authors to avoid general claims and take the modest approach.

Questions for authors
  1. Can you isolate whether Semi-OPD’s advantage comes from the initial rollout distribution, the quality of the generated trajectories, or avoiding repeated online rollout generation?

  2. What happens if Semi-OPD refreshes its cached rollouts periodically, for example, every 10, 50, or 100 training steps? Could this outperform both fully static Semi-OPD and fully online OPD?

  3. Do the cost savings and accuracy gains remain when OPD, Semi-OPD, and FastOPD are compared at matched compute budgets, rollout-length distributions and final accuracy targets, including the cost of generating and teacher scoring the cached data?

2 comments

Thank you for finding our paper well written, our approach simple and practical, and our empirical results and ablations strong. Below we address your concerns and questions, including a new experiment on periodically refreshing the rollout cache (Q2).

W1: Limited novelty; the overlap metric rests on intuition rather than a causal explanation.

We would like to clarify that the overlap metric is not our main contribution. It is a diagnostic that emerged from our central question: when is on-policy sampling actually necessary for distillation?

  • We are the first to systematically study when OPD should be replaced by distillation on offline student rollouts, and we find that teacher–student alignment is the key factor. This differs from prior work such as Lightning OPD, which treats fixed rollouts as an efficient approximation to OPD; we show that they can also improve accuracy.
  • Prior OPD works typically study 1–3 model pairs. We study 17 pairs across three model families, sizes up to 235B, and both reasoning modes. Semi-OPD outperforms OPD on 14/17 pairs, challenging the default practice of continually refreshing rollouts.
  • We hypothesize that effective distillation needs rollouts that are on-policy w.r.t. both the student and the teacher, and test this with controlled experiments: (1) student on-policyness matters, as initial-student rollouts outperform teacher rollouts (Fig. 5a,b); (2) longer online rollouts hurt misaligned pairs but help aligned ones (Fig. 6); (3) full reasoning-stage coverage matters, as full trajectories outperform prefixes at a matched token budget (Fig. 5c,d); (4) Semi-OPD provides more positive teacher-relative supervision (Fig. 7).

We agree that this is empirical evidence rather than a proof of causation, and we will phrase it accordingly (see W3).

W2: Tables lack uncertainty estimates. W3: Claims are too general.

Thank you for the suggestion. We will revise the wording to limit our claims to the settings we studied and present the overlap ratio as a diagnostic rather than a general rule. We would also like to clarify that we have done our best to reduce experimental variance by including more model pairs and averaging across training and evaluation runs:

  • Unlike many recent OPD works that only study 1–3 model pairs and report final accuracy tables, we evaluate 17 pairs across three model families and different model sizes, and importantly provide full evaluation curves for all 17 pairs.
  • Additionally, we run 3 seeds for 3 representative pairs, as shown in Fig. 2(b), where the shaded regions show relatively small variance across runs.
  • We also use the area between evaluation curves in Fig. 2(a) to compare Semi-OPD and OPD, rather than cherry-picking a single checkpoint.
  • All evaluation accuracy numbers are averaged over 32 runs, as mentioned in the experiment setting.

Q1: Does Semi-OPD's advantage come from the initial rollout distribution, the quality of the generated trajectories, or avoiding repeated online rollout generation?

Overall, we find that the initial rollouts are of lower quality than online rollouts in terms of correctness, but they are more friendly to distillation because they are less off-policy w.r.t. the teacher. In contrast, online rollouts contain more repetition and are more off-policy w.r.t. the teacher (Fig. 12, App. D).

To answer your question more specifically:

  1. Quality of the generated trajectories. The initial rollouts naturally have lower correctness, since they are generated by a less-trained student. As shown in Sec. 5 (Fig. 11), one limitation of Semi-OPD is that the initial rollouts need to meet a basic quality bar. However, lower quality does not mean that the rollouts are less suitable for distillation: teacher-generated rollouts have higher quality, yet distilling on them underperforms Semi-OPD (Fig. 5a,b).

  2. Repeated online rollout generation. Repetition is one artifact of online rollouts, but not the full picture. Truncating online rollouts to avoid repetition improves distillation accuracy (Fig. 6), but it still does not match Semi-OPD on misaligned teacher–student pairs: for example, FastOPD (OPD on 4K-truncated rollouts) trails Semi-OPD by 9.1 and 4.9 points on the two low-overlap pairs in Table 1.

As such, the main advantage of Semi-OPD on misaligned teacher–student pairs is likely that the initial rollout distribution is more friendly to distillation.

Q2: What if Semi-OPD refreshes its cached rollouts periodically?

Thank you for this suggestion. Our paper already studies a closely related setting: switching from Semi-OPD to OPD mid-training (Sec. 5, Figs. 9 and 18). There, OPD converges to the same level as OPD trained from scratch, regardless of its initialization. Following your suggestion, we additionally ran periodic refreshing, where the rollout cache is regenerated from the current student every N steps (N = 1 is OPD; never refreshing is Semi-OPD), on a misaligned and a highly aligned pair:

Pair (initial overlap)BenchmarkSemi-OPD (never)N = 100N = 50N = 10OPD (N = 1)
Qwen3-1.7B ← Qwen3-4B-Inst (62.5%)AIME 24/2542.432.733.434.133.8
4-benchmark avg37.628.829.029.429.1
R1-Distill-7B ← Polaris-7B (86.8%)AIME 24/2556.556.056.856.056.4
4-benchmark avg50.850.851.350.851.1
  • Misaligned pair: Semi-OPD remains the best. Every refresh interval converges to the OPD level.
  • Aligned pair: all methods are comparable, consistent with our finding that OPD is safe for aligned pairs.

This confirms our switching results: once online rollouts are reintroduced, training converges to the same level as OPD. Our hypothesis is that the training rollout distribution, rather than the starting checkpoint, determines where distillation converges. Characterizing this equilibrium is an interesting direction for future work. Therefore, for misaligned pairs, keeping the initial rollouts is the better choice. We will add these results to the paper.

Q3: Do the savings and gains hold at matched compute, rollout length, and accuracy targets, including the cost of generating and scoring the cache?

Yes. We would like to clarify that our reported training costs already include initial rollout generation, teacher scoring, and student training, as stated in Appendix C, Table 4. Therefore, the reported 2.2–2.7× speedup over FastOPD and up to 11.4× speedup over OPD already account for all three components. Specifically, on the non-thinking pairs, Semi-OPD and FastOPD use the same training rollout cap; on the thinking pairs, Semi-OPD uses a 4× longer cap than FastOPD and is still more than 2× cheaper. Regarding matched accuracy targets, we reported full evaluation accuracy curves under the same training steps for all methods as in Figure 3, and each eval point are averaged over 32 evaluation runs to ensure fairness.

We hope the above clarifications and experiments address your questions. Please let us know if you have further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.