acceptodds
Under review as a conference paper at ICLR 2027

OPDARENA: UNDERSTANDING LEARNING AND FORGETTING IN ON-POLICY DISTILLATION

Abstract

On-policy distillation (OPD) is increasingly used for language model post-training, yet how teacher supervision shapes learning and forgetting remains unclear. We present OPDArena, a framework for comparing supervised fine-tuning (SFT), reinforcement learning with verifiable rewards (RLVR), and OPD across five pretrained models and diverse teacher sources, jointly evaluating mathematical learning, general capability retention, and predictive distribution changes. We find that teacher choice strongly shapes retention: RLVR-trained teachers support task learning while preserving prior capabilities, whereas SFT-trained teachers induce greater capability loss. Out-of-domain (OOD) distribution change shows a more consistent positive association with forgetting than in-domain change. We then examine the additional information teachers provide relative to student Base. Matched teacher comparisons connect preferences learned through RLVR to lower relative OOD influence and stronger task learning with limited forgetting; calibrated local directions further support differences in cross-domain effects. Parameter and prediction trajectories show how students acquire teacher preferences through different parameter paths, with task gains and forgetting inherited at different rates.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, on-policy distillation, catastrophic forgetting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qgPSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares SFT, RLVR and on-policy distillation from several teacher sources (larger Base models, the student's own SFT or RLVR endpoint, and OPSD) for mathematical post-training of five Qwen Base models on OpenR1-50K, measuring both math accuracy and retention on seven OOD tasks. Its main finding is that RLVR-type teachers yield task learning with little forgetting, whereas SFT-type teachers and larger Base teachers induce more forgetting. The authors characterise update selectivity with the ratio ρ = D_OOD/D_in and use a matched A/B/C comparison to attribute the difference to the source of the teacher's information.

Critical strengths
  1. The paper compares many teacher sources under one OPD recipe and reports learning and retention together. The observation that a teacher with stronger OOD ability can produce a worse student (Table 2) is useful in practice.

  2. A teacher constructed by adding the larger model's RLVR increment to the student Base (group C) teaches about as well as the full RLVR teacher (Table 13), which shows that the useful signal is the post-training increment rather than the pretrained gap.

Critical weaknesses
  1. The source of the SFT data is not described, so the attribution of forgetting does not hold. The paper refers only to "reference solutions" from OpenR1-50K. According to the public dataset card, Open R1 solutions are reasoning traces generated by DeepSeek R1. SFT therefore makes small models imitate the long-form reasoning of a much stronger model, while RLVR uses only the model's own samples. The two differ not only in objective but also in how far the data lie from the model: under Base, the negative log-probability of SFT-teacher responses is 1.057 versus 0.230 for RLVR-teacher responses (Table 9, Qwen2.5-7B). The learning rates also differ tenfold (1e-5 vs 1e-6). This is common practice, but the paper's own interpolation shows that shrinking the SFT update to 60% closes more than half of its OOD gap to RLVR (Fig. 18). The paper attributes forgetting to "SFT-type information" without separating the objective from the data source and the update magnitude.

  2. The A/B/C experiment has no SFT arm and cannot support the claim of less forgetting. The paper does not explain why only the Base and RLVR endpoints of the larger model are compared; without an SFT arm, the experiment cannot distinguish RLVR from SFT. It also stops at step 120 (the main runs use 780 steps), where the OOD accuracy change of all three teacher groups lies between −0.39 and +0.94 points (Table 13): no group forgets, and B/C do not retain better than A. B/C have lower ρ because their in-domain change is much larger, not because their OOD change is smaller. The experiment therefore shows that the RLVR increment teaches more, not that it forgets less.

  3. The evaluation lacks an overlap analysis and confidence intervals. The paper does not check whether training problems overlap with the math evaluation sets, although, according to its public dataset card, the source of the training problems (NuminaMath 1.5) includes AMC/AIME problems. Four of the five math sets have only 30–40 problems yet carry 80% of the mean, so a single problem shifts the mean by 0.5–0.7 points. Several OOD conclusions rest on differences below 1 point. No confidence intervals are reported.

Questions for authors
  1. If the SFT teacher is trained on correct solutions sampled from the model itself, or is matched to the RLVR teacher in retention, does Self-OPD still forget more?

  2. Why does A/B/C contain no SFT arm, and why does it stop at step 120? Can these be added?

  3. Do the training problems overlap with the evaluation sets? Please report confidence intervals for the main results.

5 comments
AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank the reviewer for recognizing our systematic comparison of teacher sources and insights into the role of post-training information. We address your questions below.

Question: 1. If the SFT teacher is trained on correct solutions sampled from the model itself, or is matched to the RLVR teacher in retention, does Self-OPD still forget more?

Response: Training SFT teachers on their own correct solutions improves student OOD retention, but yields almost no in-domain learning gains. For retention matching, we could not find a checkpoint from the original SFT training that matches the final RLVR teacher’s retention.

The original SFT uses reference solutions generated by DeepSeek-R1 in OpenR1. To examine the role of data source, we use the Qwen2.5-7B, Qwen3-4B, and Qwen3-8B Base models to solve approximately 50,000 training problems from OpenR1-50K. For each problem, we retain only the first correct, normally completed solution within the first eight sampling attempts, yielding 24,082, 27,160, and 29,752 such solutions, respectively, with one solution per solved problem. We use these solutions as each model’s SFT training data and train for one epoch, using each example once. We then use each SFT endpoint as the teacher and train a Self-OPD student from the same Base initialization through step 780.

All accuracy entries in this response use value (change from the corresponding Base before training), with accuracy in % and changes in pp. The first two tables report in-domain / OOD accuracy. We first compare the two SFT settings: SFT on the model’s own correct solutions is evaluated at its endpoints of 377, 425, and 465 steps, and original SFT at the nearest checkpoints of 380, 420, and 460 steps, respectively.

ModelOriginal SFTSFT on own correct solutions
Qwen2.5-7B34.07 (+7.99) / 45.35 (-4.14)25.34 (-0.74) / 49.23 (-0.26)
Qwen3-4B39.71 (+12.87) / 43.26 (-2.65)29.16 (+2.32) / 47.17 (+1.26)
Qwen3-8B45.93 (+14.61) / 48.28 (-3.23)32.18 (+0.86) / 52.20 (+0.68)

We then compare Self-OPD students trained with the three teacher conditions, all evaluated at step 780. Here, the original SFT and RLVR teachers are from step 780, while the own-solution SFT teachers are from their respective endpoints above.

ModelOriginal SFT teacher → Self-OPDOwn-solution SFT teacher → Self-OPDRLVR teacher → Self-OPD
Qwen2.5-7B28.49 (+2.41) / 47.02 (-2.47)25.53 (-0.55) / 49.63 (+0.14)31.25 (+5.16) / 49.83 (+0.34)
Qwen3-4B34.71 (+7.86) / 43.45 (-2.46)27.55 (+0.70) / 46.17 (+0.26)36.39 (+9.54) / 45.80 (-0.11)
Qwen3-8B44.79 (+13.47) / 47.10 (-4.41)31.51 (+0.19) / 51.96 (+0.45)44.35 (+13.03) / 51.97 (+0.45)

These results show that Self-OPD students trained with own-solution SFT teachers exhibit little forgetting, but also very limited in-domain learning gains. We further examine these students’ distribution changes. The table reports results at step 780 relative to Base, with each cell showing in-domain JS / OOD JS. Both JS values are multiplied by 100.

ModelOriginal SFT teacher → Self-OPDOwn-solution SFT teacher → Self-OPDRLVR teacher → Self-OPD
Qwen2.5-7B7.137 / 1.9630.302 / 0.2173.801 / 0.421
Qwen3-4B6.941 / 2.1220.180 / 0.2952.622 / 0.290
Qwen3-8B6.519 / 2.6530.252 / 0.1952.848 / 0.290

With own-solution SFT teachers, students have small OOD JS, but their in-domain JS is also much smaller than under the other two teacher conditions. Together with the near-Base in-domain accuracy reported above, this shows that this setting yields little forgetting but almost no in-domain learning gains.

Regarding teacher retention matching, RLVR shows almost no OOD degradation throughout training, whereas original SFT already shows clear OOD degradation early in training. We therefore could not find an available SFT checkpoint that matches the final RLVR teacher’s retention. The table reports teacher in-domain / OOD accuracy at SFT steps 20 and 100 and RLVR step 780, with changes from Base in parentheses. Appendix C presents the complete in-domain / OOD performance curves throughout SFT and RLVR training.

ModelSFT step 20SFT step 100RLVR step 780
Qwen2.5-7B9.02 (-17.06) / 48.10 (-1.38)30.02 (+3.94) / 46.81 (-2.68)31.63 (+5.55) / 49.99 (+0.50)
Qwen3-4B26.13 (-0.72) / 45.25 (-0.66)37.48 (+10.64) / 45.50 (-0.41)35.23 (+8.39) / 46.33 (+0.42)
Qwen3-8B29.82 (-1.50) / 50.94 (-0.58)41.77 (+10.45) / 49.78 (-1.73)45.21 (+13.89) / 51.84 (+0.33)
AuthorsOpenReview ↗

Official Comment by Authors

Question: 2. Why does A/B/C contain no SFT arm, and why does it stop at step 120? Can these be added?

Response: The original A/B/C comparison separates the effects of pretrained differences and post-training increments. Due to computational resource constraints, the original manuscript reported only early results at step 120. We have now added the full SFT teacher and its increment, and extended all twenty runs across four teacher–student pairs to the full 780 steps.

Both increment teachers extract post-training changes through log-probability differences and apply them to the student Base distribution. Student initialization, training problems, and OPD settings are held fixed within each comparison. The table reports student in-domain / OOD accuracy (%) at step 780.

Larger teacher → StudentLarger BaseFull SFTSFT incrementFull RLVRRLVR increment
Qwen2.5-7B → Qwen2.5-1.5B15.50 / 34.569.50 / 34.346.57 / 35.2320.12 / 36.4719.90 / 36.36
Qwen3-4B → Qwen3-1.7B20.40 / 35.2021.37 / 33.7716.26 / 35.4225.69 / 36.5625.46 / 36.64
Qwen3-8B → Qwen3-1.7B19.90 / 34.6821.29 / 34.0215.55 / 35.7527.03 / 36.2128.52 / 36.77
Qwen3-8B → Qwen3-4B28.07 / 44.9034.77 / 42.3635.49 / 43.9140.37 / 45.6040.11 / 46.04

The following table reports distribution changes for the same students at step 780 relative to their respective Base models. Each cell gives in-domain JS / OOD JS / ρ, with both JS values multiplied by 100.

Larger teacher → StudentLarger BaseFull SFTSFT incrementFull RLVRRLVR increment
Qwen2.5-7B → Qwen2.5-1.5B0.606 / 0.833 / 1.3767.461 / 2.868 / 0.3846.969 / 1.624 / 0.2334.713 / 1.206 / 0.2564.557 / 0.940 / 0.206
Qwen3-4B → Qwen3-1.7B0.436 / 0.678 / 1.5556.451 / 2.818 / 0.4376.127 / 1.160 / 0.1894.663 / 0.663 / 0.1424.580 / 0.538 / 0.118
Qwen3-8B → Qwen3-1.7B0.485 / 0.893 / 1.8426.523 / 2.788 / 0.4276.295 / 1.177 / 0.1874.720 / 0.805 / 0.1704.597 / 0.444 / 0.097
Qwen3-8B → Qwen3-4B0.441 / 0.589 / 1.3367.014 / 2.289 / 0.3266.298 / 1.174 / 0.1863.137 / 0.766 / 0.2442.675 / 0.446 / 0.167

The full-training results support our original conclusion: post-training a larger model on the same in-domain task before using it as an OPD teacher makes student distribution updates more concentrated in-domain. Across all four pairs, students of both SFT and RLVR teachers have substantially lower ρ than students of the larger Base teacher, indicating less OOD distribution change per unit of in-domain distribution change. Isolating the post-training increment further reduces both ρ and OOD distribution change; in our results, this generally corresponds to less OOD forgetting. These findings show that task-specific preferences acquired through teacher post-training help make student updates more selective.

The RLVR increment teacher preserves in-domain learning gains close to those of the full RLVR teacher while further reducing OOD distribution change, illustrating the value of selective updates for combining learning with capability retention. We have updated Section 4.2 (Table 3) of the main text with the full-training results, including the SFT controls; endpoint results for all four pairs are provided in Appendix D.6 (Table 14).

AuthorsOpenReview ↗

Official Comment by Authors

Question: 3. Do the training problems overlap with the evaluation sets? Please report confidence intervals for the main results.

Response: We audited the 49,993 training records actually used against all 630 problems in the five in-domain evaluation sets through text normalization, approximate matching, and individual inspection. We found no identical or mathematically equivalent problems in AMC23, AIME24, AIME25, or HMMT25. In Math500, we confirmed 13 problems whose key numerical values and solution targets match corresponding training problems, accounting for 2.6% of the set.

The five evaluation sets are equally weighted, so Math500 contributes 20% of the overall score. Using saved per-problem responses, we recomputed results after excluding these 13 problems. Across the endpoint results for 38 configurations × 2 seeds, the in-domain mean changes by at most 0.14 pp, and the gain over Base by at most 0.11 pp. In the main table, in-domain gaps between key method pairs, including SFT vs RLVR and the two Self-OPD teacher types, change by no more than 0.08 pp, with their ordering unchanged. Given the remaining differences in wording, formatting, or answer-choice presentation, together with the small changes after exclusion, we retain the complete standard Math500 benchmark for comparability and have added the overlaps and exclusion sensitivity results to Appendix B.5. We thank the reviewer for raising this issue.

Regarding the influence of individual problems in the small evaluation sets, under the eight-sample evaluation setting, we average correctness over eight responses per problem, then over problems within each set, and finally over the five sets with equal weights. For a set of 30–40 problems, changing the correctness of one sampled response therefore changes the overall mean by 100/(5 × number of problems × 8), or 0.063–0.083 pp. The reviewer's 0.5–0.7 pp corresponds to a problem's accuracy changing from 0% to 100%, with all eight responses changing from incorrect to correct.

Reviewer_ymhrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares learning and forgetting under SFT, RLVR, and on-policy distillation with different teachers. Experiments cover five Qwen models, measuring mathematical performance and out-of-domain retention. RLVR teachers generally preserve students’ prior capabilities better than SFT teachers. The authors examine this difference through prediction changes and controlled teacher comparisons. Parameter trajectories and prefix analyses further explore how task gains and forgetting transfer to students.

Critical strengths

1, The teacher comparisons go beyond ranking teachers by their benchmark accuracy. The A/B/C construction separates a larger model’s Base predictions from its RLVR log-probability increment, and the local-direction analysis calibrates in-domain change. These comparisons give some support to the idea that the information a teacher adds deserves separate attention.

2, Evaluating learning and retention together helps distinguish preserving capabilities from simply learning little on the target task. The paired source-training and Self-OPD curves also show differences in when task gains and forgetting appear, which would be missed by comparing final checkpoints alone.

Critical weaknesses

1, The interpretation of the SFT–RLVR difference is still entangled with a specific feature of the supervision recipe. The paper reports that 99.54% of the SFT demonstrations begin with “Okay” or “Alright,” while the tested SFT teachers’ preferred opening tokens fall outside the student’s Top-16 candidate set. Removing feedback at the first position alone improves mathematical accuracy by 13.06 points at step 40 in the paired intervention. This suggests that demonstration style interacting with truncated feedback accounts for a substantial part of the observed difficulty. However, the intervention does not establish the long-run learning and retention outcomes after that issue is addressed, leaving the broader teacher-source explanation unresolved.

2, The retention hierarchy is based on single training runs. The author states that each configuration uses one training seed, and all five students come from the Qwen2.5 or Qwen3 families. Several OOD changes are below one percentage point, so it is hard to assess their stability without variation across training seeds and uncertainty estimates for evaluation. Correlations across successive checkpoints do not provide independent replications of the relationship between distribution change and forgetting.

Questions for authors

1, See Critical Weakness 1.

2, See Critical Weakness 2,

3, The paper does not compare with BRTS [1], which selects teacher trajectories using correctness and student alignment before adding teacher-context supervision. Could you evaluate it with the same teacher–student pair, reporting mathematical learning and OOD retention while accounting for the extra teacher-generation cost?

[1] Ke Zhang, Yunjie Tian, Dongdi Zhao, Yijiang Li, Yuanye Liu, Vishal M. Patel, and Di Fu. “On-Policy Distillation with Best-of-N Teacher Rollout Selection.” arXiv:2605.09725, 2026.

5 comments
AuthorsOpenReview ↗

Official Comment by Authors

Question: 1. How does masking first-position feedback affect long-run learning and capability retention?

Response: We agree that the evaluation should not focus only on step 40 while overlooking long-run learning and retention. The full-training results support our original conclusion: SFT teachers can deliver strong in-domain gains, but with greater OOD forgetting than RLVR teachers.

We run the SFT-teacher Self-OPD ablation through step 780, zeroing only the direct Top-16 feedback at the first valid generated position while retaining the complete generated text, subsequent context, response mask, and loss normalization. The table reports student in-domain / OOD accuracy (%).

ModelSFT teacher: ordinary Self-OPDSFT teacher: first feedback maskedRLVR teacher: ordinary Self-OPD
Qwen2.5-1.5B10.33 / 34.9314.23 / 34.5819.46 / 36.32
Qwen2.5-7B28.49 / 47.0234.57 / 46.5031.25 / 49.83
Qwen3-1.7B17.16 / 35.5920.54 / 35.7126.75 / 36.64
Qwen3-4B34.71 / 43.4538.82 / 43.3536.39 / 45.80
Qwen3-8B44.79 / 47.1044.12 / 47.6344.35 / 51.97

After masking, Qwen2.5-7B, Qwen3-4B, and Qwen3-8B reach in-domain accuracy close to or above their RLVR-teacher counterparts, but OOD accuracy remains lower for all five models. Thus, the learning and forgetting pattern associated with SFT teachers persists after masking first-position feedback.

First-position feedback does affect students' in-domain learning: masking improves in-domain accuracy for four models, with an average gain of 3.36 pp across all five, while OOD accuracy shows no appreciable change. For the strongest model, Qwen3-8B, masking brings no in-domain improvement. This suggests that stylistic preferences acquired by teachers during SFT, such as their opening style, may primarily hinder in-domain learning in weaker students.

Appendix D.5 (Figure 13) provides the complete trajectories of ordinary and first-feedback-masked Self-OPD for all five models, allowing readers to examine the intervention's effects on learning and retention at different training stages.

AuthorsOpenReview ↗

Official Comment by Authors

Question: 2. How robust are the experimental results to training randomness and evaluation uncertainty, particularly the small OOD differences?

Response: We have added a second independent seed for all 38 configurations in the main table, covering SFT, RLVR, OPD, Self-OPD, and OPSD, and updated the main results in Table 1 and the complete five-model results in Appendix C to report two-seed averages. For Self-OPD, we first train SFT/RLVR with the new seed, then use the resulting model as the teacher and train the Self-OPD student with that same new seed. Correlations across successive checkpoints characterize changes within a training run; independent training seeds assess reproducibility.

The table follows the method order in Table 1 and reports two-seed means at step 780. Each cell shows in-domain (change) / OOD (change), with accuracy in %. Parentheses give the change in the two-seed mean relative to the original single-seed result for the same configuration (pp).

ModelSFTRLVRSelf-OPD (SFT)Self-OPD (RLVR)OPSD
Qwen2.5-1.5B11.50(+0.27) / 34.25(+0.47)19.08(+0.29) / 36.23(+0.01)10.25(-0.08) / 34.99(+0.06)20.00(+0.54) / 36.60(+0.28)12.51(-0.12) / 35.06(-0.03)
Qwen2.5-7B35.87(-0.58) / 45.32(+0.40)32.56(+0.92) / 50.06(+0.07)28.29(-0.20) / 47.11(+0.08)32.25(+1.01) / 49.59(-0.24)25.27(-0.19) / 49.09(-0.27)
Qwen3-1.7B20.35(-0.55) / 35.43(-0.35)27.02(+0.69) / 36.44(-0.41)17.54(+0.37) / 36.01(+0.42)27.35(+0.60) / 36.26(-0.38)18.97(-0.71) / 35.57(-0.06)
Qwen3-4B41.77(+0.10) / 42.57(+0.01)36.87(+1.64) / 46.11(-0.22)35.72(+1.01) / 43.30(-0.16)37.68(+1.30) / 45.46(-0.33)27.10(-0.79) / 45.34(-0.13)
Qwen3-8B47.32(-0.16) / 48.54(+0.75)45.39(+0.17) / 52.12(+0.28)44.34(-0.45) / 47.23(+0.13)45.48(+1.12) / 52.12(+0.15)28.98(+0.66) / 50.87(-0.45)

The mean absolute change across the table is 0.41 pp (0.58 pp for in-domain and 0.25 pp for OOD), indicating that training seed variation has a small overall effect on the results.

The rankings of the methods on in-domain and OOD performance remain broadly stable, and the main conclusions are consistent with the original single-seed results. Appendix C presents the complete in-domain and OOD performance trajectories averaged over both seeds.

To quantify uncertainty from finite evaluation questions, we perform 20,000 paired, stratified question-bootstrap resamples on the mean over the two fixed training seeds, preserving the original benchmark weights. Each cell reports in-domain [95% CI] / OOD [95% CI], with accuracies and intervals in %; the parentheses in Self-OPD indicate the teacher source.

ModelSFTRLVRSelf-OPD (SFT)Self-OPD (RLVR)OPSD
Qwen2.5-1.5B11.50 [9.96, 13.15] / 34.25 [32.66, 35.84]19.08 [16.83, 21.42] / 36.23 [34.59, 37.86]10.25 [8.72, 11.91] / 34.99 [33.39, 36.57]20.00 [17.69, 22.41] / 36.60 [34.94, 38.21]12.51 [11.12, 13.98] / 35.06 [33.49, 36.60]
Qwen2.5-7B35.87 [31.66, 40.27] / 45.32 [43.69, 46.95]32.56 [29.16, 36.04] / 50.06 [48.42, 51.70]28.29 [25.02, 31.72] / 47.11 [45.46, 48.75]32.25 [28.99, 35.67] / 49.59 [47.97, 51.23]25.27 [22.69, 27.88] / 49.09 [47.48, 50.69]
Qwen3-1.7B20.35 [17.82, 23.09] / 35.43 [33.85, 36.99]27.02 [23.81, 30.45] / 36.44 [34.84, 38.06]17.54 [15.46, 19.74] / 36.01 [34.42, 37.60]27.35 [24.39, 30.49] / 36.26 [34.67, 37.85]18.97 [17.01, 21.06] / 35.57 [34.00, 37.11]
Qwen3-4B41.77 [37.32, 46.39] / 42.57 [40.97, 44.17]36.87 [32.95, 40.99] / 46.11 [44.45, 47.72]35.72 [31.92, 39.64] / 43.30 [41.67, 44.91]37.68 [33.64, 41.88] / 45.46 [43.80, 47.09]27.10 [24.28, 30.02] / 45.34 [43.72, 46.93]
Qwen3-8B47.32 [42.89, 51.95] / 48.54 [46.88, 50.19]45.39 [40.94, 50.02] / 52.12 [50.47, 53.77]44.34 [40.04, 48.78] / 47.23 [45.63, 48.82]45.48 [41.01, 50.14] / 52.12 [50.47, 53.78]28.98 [25.90, 32.23] / 50.87 [49.25, 52.49]

After accounting for evaluation uncertainty, the main trends in in-domain learning and OOD retention across methods remain consistent, and our original conclusions remain unchanged.

AuthorsOpenReview ↗

Official Comment by Authors

Question: 3. Could you compare BRTS with the same teacher–student pair, reporting in-domain learning and OOD retention while accounting for the extra teacher-generation cost?

Response: We thank the reviewer for suggesting BRTS. We have added the accuracy comparisons to Appendix D.11 and cited the paper. The comparison covers all five students with their SFT and RLVR teachers, for ten runs in total. Each run uses the same student initialization, teacher checkpoint, and training problems as its original Self-OPD counterpart, and trains for 780 steps. We use two teacher candidates per problem, selected by correctness and overlap with the student’s Top-16 predictions. When both candidates are incorrect, we generate one additional candidate with a hint. We retain the original Self-OPD objective and add an auxiliary distillation loss on the selected teacher response with a weight of 10.

The table reports in-domain / OOD accuracy (%) at step 780; parentheses indicate the teacher source.

ModelSelf-OPD (SFT)BRTS (SFT)Self-OPD (RLVR)BRTS (RLVR)
Qwen2.5-1.5B10.33 / 34.9311.67 / 34.8419.46 / 36.3218.53 / 35.72
Qwen2.5-7B28.49 / 47.0234.36 / 45.3331.25 / 49.8332.04 / 49.27
Qwen3-1.7B17.16 / 35.5920.06 / 35.8126.75 / 36.6426.87 / 37.67
Qwen3-4B34.71 / 43.4540.16 / 43.9336.39 / 45.8036.19 / 46.43
Qwen3-8B44.79 / 47.1045.06 / 48.3344.35 / 51.9743.47 / 52.67

BRTS improves in-domain accuracy with SFT teachers for all five models, by 3.17 pp on average, while its OOD effects vary across models. Our conclusion also holds within BRTS: RLVR teachers provide better OOD retention for all five models. Teacher-trajectory selection can therefore improve learning, while teacher-source differences in retention persist.

To address the reviewer’s question about extra teacher-generation cost, we provide the following cost comparison. We use standard Self-OPD as the baseline and include both BRTS student training and teacher-candidate generation. Beyond candidate generation, BRTS adds a teacher forward pass and a student forward and backward pass on the selected teacher response, introducing additional training computation.

All costs below are in A100 GPU hours: BRTS total = training + candidate generation, generation ratio = candidate generation / standard Self-OPD × 100%, and increase = (BRTS total / standard Self-OPD − 1) × 100%. Both methods start from existing teacher checkpoints and train the student through step 780. Teacher candidates are cached and shared across student rollouts for the same problem, so generation costs are counted once. Training costs exclude startup, checkpointing, evaluation, and repeated work after interruptions. Candidate generation uses recorded allocated GPU time, including loading, grading, unselected candidates, and recorded interrupted attempts.

ModelTeacherStandard Self-OPDBRTS total (training + generation)Generation / Self-OPDIncrease
Qwen2.5-1.5BSFT281.4587.1 (445.2 + 141.9)50.4%+108.6%
Qwen2.5-7BSFT544.4979.2 (838.3 + 140.9)25.9%+79.9%
Qwen3-1.7BSFT332.3719.2 (548.4 + 170.8)51.4%+116.4%
Qwen3-4BSFT520.0935.8 (743.0 + 192.8)37.1%+80.0%
Qwen3-8BSFT654.51111.2 (919.9 + 191.3)29.2%+69.8%
Qwen2.5-1.5BRLVR169.0458.7 (396.6 + 62.1)36.7%+171.3%
Qwen2.5-7BRLVR346.7609.9 (535.9 + 74.1)21.4%+75.9%
Qwen3-1.7BRLVR261.0632.4 (493.0 + 139.4)53.4%+142.3%
Qwen3-4BRLVR401.7744.4 (573.3 + 171.1)42.6%+85.3%
Qwen3-8BRLVR519.9981.1 (804.8 + 176.3)33.9%+88.7%

[1] Ke Zhang, Yunjie Tian, Dongdi Zhao, Yijiang Li, Yuanye Liu, Vishal M. Patel, and Di Fu. “On-Policy Distillation with Best-of-N Teacher Rollout Selection.” arXiv:2605.09725, 2026.

Reviewer_4WCwOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents OPDArena, a systematic study of learning and forgetting in on-policy distillation. The authors compare different training methods and teacher sources, and analyze their effects on mathematical learning, OOD retention, and prediction distributions.

Critical strengths

The paper investigates an interesting question about how teacher selection affects learning and forgetting. The experiments cover multiple models and teacher configurations, with useful analyses of distribution changes and teacher supervision.

Critical weaknesses
  1. Rethinking On-Policy Distillation of Large Language Models has already studied teacher–student compatibility and the importance of teacher capabilities. On-Policy Delta Distillation has explored extracting post-training information through teacher–Base probability differences. This paper combines similar ideas but does not clearly establish what new mechanism or insight is gained beyond these works.
  2. Table 1 shows that SFT often achieves larger Math improvements than RLVR while experiencing more forgetting. For example, on Qwen3-4B, SFT improves Math accuracy by 14.83 points, compared with 8.39 for RLVR. Without matching task performance, it is unclear whether the difference in forgetting comes from the training method or the amount of learning.
  3. Appendix C.5 shows that SFT teachers strongly prefer opening tokens outside the student's Top-16 candidates. Simply removing first-position feedback improves Math accuracy from 14.47% to 27.53%. This raises concerns about whether the observed differences between SFT and RLVR teachers are partly caused by the distillation setup.
  4. The proposed relative OOD influence metric does not consistently reflect actual forgetting. For Qwen3-4B, OPD has a higher ρ than SFT but substantially less OOD accuracy loss. Its usefulness for explaining forgetting remains unclear.
Questions for authors
  1. What specific new insight does OPDArena provide beyond Rethinking On-Policy Distillation and On-Policy Delta Distillation?

  2. Do the retention differences remain when SFT, RLVR, and OPD are compared at similar Math performance levels?

  3. Do the differences between SFT and RLVR teachers remain after addressing the first-position feedback issue identified in Appendix C.5?

  4. How does the proposed relative OOD influence metric help explain or predict forgetting when its ranking disagrees with actual OOD performance?

4 comments
AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank the reviewer for recognizing our systematic study of teacher selection and analysis of distribution changes. We address your questions below.

Question: 1. What specific new insight does OPDArena provide beyond Rethinking On-Policy Distillation and On-Policy Delta Distillation?

Response: First, OPDArena systematically compares SFT, RLVR, and OPD using the same initial Base models, training problems, and evaluation tasks. Within OPD, we further compare distillation from larger Base teachers, Self-OPD using SFT- or RLVR-trained checkpoints of the student's initial Base model as teachers, and OPSD self-distillation with additional reference solutions provided to the teacher. Neither Rethinking On-Policy Distillation nor On-Policy Delta Distillation compares SFT, RLVR, and OPD in a common setting, nor do they provide a systematic comparison spanning the teacher sources and self-distillation approaches described above.

Second, OPDArena jointly evaluates in-domain learning and OOD forgetting. Rethinking On-Policy Distillation studies teacher–student compatibility and its effects on mathematical performance, without evaluating how mathematical training affects general capabilities outside that domain. On-Policy Delta Distillation discusses degradation of existing capabilities, but both its training and evaluation are limited to the same three domains (math, science, and code), so the degradation it examines occurs within the training domains and differs from the OOD forgetting studied here. We train on OpenR1-50K and evaluate OOD capabilities across commonsense reasoning, knowledge-based question answering, instruction following, and diverse reasoning tasks, examining how capabilities beyond the training objective change.

A new insight from these comparisons is that on-policy training alone does not explain capability retention. On-policy training is often viewed as an important contributor to reduced forgetting, but our results show substantial differences even among on-policy methods: depending on teacher choice, OPD can exhibit retention close to on-policy RLVR or forgetting comparable to, or even greater than, off-policy SFT. Understanding OPD's forgetting therefore requires considering the teacher's supervision alongside its use of on-policy samples.

For example, on Qwen3-8B, both Self-OPD variants achieve similar in-domain accuracy, but the SFT-teacher student has lower OOD accuracy than direct SFT, while the RLVR-teacher student remains close to direct RLVR. Teacher selection for OPD therefore needs to account for both in-domain learning and how the teacher's supervision affects OOD capabilities.

Question: 2. Do the retention differences remain when SFT, RLVR, and OPD are compared at similar in-domain performance levels?

Response: At similar in-domain performance levels, RLVR still retains higher OOD accuracy than SFT; within Self-OPD, RLVR teachers still yield better student retention than SFT teachers.

The table uses RLVR's in-domain accuracy at step 780 as the reference and selects the closest checkpoint for each method. Each cell shows step number: in-domain / OOD, with accuracies in %.

ModelSFTRLVRSelf-OPD (SFT teacher)Self-OPD (RLVR teacher)
Qwen3-4Bstep80: 35.23 / 44.82step780: 35.23 / 46.33step520: 35.19 / 43.62step480: 35.23 / 45.72
Qwen3-8Bstep220: 45.05 / 48.81step780: 45.21 / 51.84step680: 45.46 / 47.47step260: 45.26 / 52.03

Appendix C provides the complete learning and forgetting trajectories for all five models across training methods and teacher configurations. These curves allow readers to examine retention at other in-domain performance levels and follow how the differences develop throughout training.

AuthorsOpenReview ↗

Official Comment by Authors

Question: 3. Do the differences between SFT and RLVR teachers remain after addressing the first-position feedback issue identified in Appendix C.5?

Response: We designed the experiment in Appendix C.5 of the original submission (Appendix D.5 in the revision) to show how response preferences in the SFT data are learned during SFT and subsequently affect Self-OPD using the resulting checkpoint as the teacher. The first-position feedback effect is therefore part of the phenomenon we aim to study and does not indicate a flaw in the experimental setup that requires correction. Comparing SFT and RLVR teachers under the same Self-OPD objective and feedback rules, without additional first-position masking, is already fair.

We nevertheless agree that the intervention should be evaluated over the full training run, rather than drawing conclusions from step 40 alone. We have therefore extended the ablation to the full 780 steps for all five models. The results continue to support our original conclusion: SFT teachers can deliver strong in-domain gains, but with greater OOD forgetting than RLVR teachers.

In SFT-teacher Self-OPD, we zero only the direct Top-16 feedback at the first valid generated position while retaining the complete generated text, subsequent context, response mask, and loss normalization. The table reports student in-domain / OOD accuracy (%).

ModelSFT teacher: ordinary Self-OPDSFT teacher: first feedback maskedRLVR teacher: ordinary Self-OPD
Qwen2.5-1.5B10.33 / 34.9314.23 / 34.5819.46 / 36.32
Qwen2.5-7B28.49 / 47.0234.57 / 46.5031.25 / 49.83
Qwen3-1.7B17.16 / 35.5920.54 / 35.7126.75 / 36.64
Qwen3-4B34.71 / 43.4538.82 / 43.3536.39 / 45.80
Qwen3-8B44.79 / 47.1044.12 / 47.6344.35 / 51.97

After masking, Qwen2.5-7B, Qwen3-4B, and Qwen3-8B reach in-domain accuracy close to or above their RLVR-teacher counterparts, but OOD accuracy remains lower for all five models. Thus, the learning and forgetting pattern associated with SFT teachers persists after masking first-position feedback.

First-position feedback does affect students' in-domain learning: masking improves in-domain accuracy for four models, with an average gain of 3.36 pp across all five, while OOD accuracy shows no appreciable change. For the strongest model, Qwen3-8B, masking brings no in-domain improvement. This suggests that stylistic preferences acquired by teachers during SFT, such as their opening style, may primarily hinder in-domain learning in weaker students.

Appendix D.5 (Figure 13) provides the complete trajectories of ordinary and first-feedback-masked Self-OPD for all five models, allowing readers to examine the intervention's effects on learning and retention at different training stages.

AuthorsOpenReview ↗

Official Comment by Authors

Question: 4. How does the proposed relative OOD influence metric help explain or predict forgetting when its ranking disagrees with actual OOD performance?

Response: Our experiments show a strong positive correlation between OOD JS and OOD forgetting (Figure 2). This is intuitive: preserving a model's predictions on OOD tasks helps retain its existing capabilities. Meanwhile, under a fixed evaluation setup, improving in-domain performance requires changing the model's predictions on in-domain tasks.

We therefore ask: how much OOD prediction change accompanies a method's in-domain prediction change? To characterize this relationship, we define

ρ=DOODDin\rho = \frac{D_{\mathrm{OOD}}}{D_{\mathrm{in}}}

where the two terms are the OOD and in-domain JS divergences from Base. The ratio measures OOD change per unit of in-domain distribution change, providing a distribution-level perspective on the relationship between learning and forgetting rather than measuring OOD forgetting alone. A lower ρ indicates prediction changes relatively more concentrated on in-domain tasks; actual learning gains and capability retention are captured by in-domain and OOD accuracy.

The reviewer's example illustrates this distinction. Both runs start from Qwen3-4B Base, using either direct SFT or OPD with Qwen3-8B Base as the teacher. The table reports step-780 results. Parentheses show accuracy changes from Base (pp). JS values are shown ×100.

MethodIn-domain accuracy (%)OOD accuracy (%)In-domain JSOOD JSρ
SFT41.68 (+14.83)42.56 (-2.90)7.3482.7330.372
OPD (Qwen3-8B Base teacher)28.07 (+1.22)44.90 (-0.55)0.4410.5891.336

A lower ρ means less OOD distribution change per unit of in-domain distribution change, suggesting that a method may achieve similar in-domain gains with less OOD forgetting.

In this example, however, SFT achieves much larger in-domain gains than OPD, accompanied by larger in-domain JS. Thus, despite its lower ρ, SFT still has higher OOD JS than OPD with the Qwen3-8B Base teacher, corresponding to greater forgetting. The metric jointly characterizes a method's trade-off between in-domain learning and OOD forgetting and should not be used to assess either in isolation.

Regarding presentation, we have carefully revised the writing to improve clarity, and hope these changes make the manuscript easier to follow.

Regarding the ethics review flag, we have reviewed our work and have not identified a specific ethical concern. If this flag was not selected inadvertently, we would appreciate clarification of the concern so that we can address it.

[1] Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, and Ning Ding. “Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe.” arXiv:2604.13016, 2026.

[2] Byeongho Heo, Jaehui Hwang, Sangdoo Yun, and Dongyoon Han. “On-Policy Delta Distillation.” arXiv:2607.15161, 2026.

[3] Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. “Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models.” Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research 306:162433–162448, 2026.

Dear Reviewer 4WCw,

Thank you again for your thoughtful review and constructive feedback. We apologize for the tight review window, as completing the additional full-training experiments took considerable time.

Since the discussion period ends tomorrow, we wanted to gently follow up on our responses and revised manuscript, covering our explanation of how jointly evaluating learning and OOD retention across SFT, RLVR, and OPD variants distinguishes OPDArena from prior studies, the matched-performance retention comparison, the full-training first-position masking ablation, and our clarification of the relative OOD influence metric. Please let us know if these clarifications and updates have addressed your concerns, or if there are any remaining points we can discuss.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.