- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper compares SFT, RLVR and on-policy distillation from several teacher sources (larger Base models, the student's own SFT or RLVR endpoint, and OPSD) for mathematical post-training of five Qwen Base models on OpenR1-50K, measuring both math accuracy and retention on seven OOD tasks. Its main finding is that RLVR-type teachers yield task learning with little forgetting, whereas SFT-type teachers and larger Base teachers induce more forgetting. The authors characterise update selectivity with the ratio ρ = D_OOD/D_in and use a matched A/B/C comparison to attribute the difference to the source of the teacher's information.
Critical strengths
-
The paper compares many teacher sources under one OPD recipe and reports learning and retention together. The observation that a teacher with stronger OOD ability can produce a worse student (Table 2) is useful in practice.
-
A teacher constructed by adding the larger model's RLVR increment to the student Base (group C) teaches about as well as the full RLVR teacher (Table 13), which shows that the useful signal is the post-training increment rather than the pretrained gap.
Critical weaknesses
-
The source of the SFT data is not described, so the attribution of forgetting does not hold. The paper refers only to "reference solutions" from OpenR1-50K. According to the public dataset card, Open R1 solutions are reasoning traces generated by DeepSeek R1. SFT therefore makes small models imitate the long-form reasoning of a much stronger model, while RLVR uses only the model's own samples. The two differ not only in objective but also in how far the data lie from the model: under Base, the negative log-probability of SFT-teacher responses is 1.057 versus 0.230 for RLVR-teacher responses (Table 9, Qwen2.5-7B). The learning rates also differ tenfold (1e-5 vs 1e-6). This is common practice, but the paper's own interpolation shows that shrinking the SFT update to 60% closes more than half of its OOD gap to RLVR (Fig. 18). The paper attributes forgetting to "SFT-type information" without separating the objective from the data source and the update magnitude.
-
The A/B/C experiment has no SFT arm and cannot support the claim of less forgetting. The paper does not explain why only the Base and RLVR endpoints of the larger model are compared; without an SFT arm, the experiment cannot distinguish RLVR from SFT. It also stops at step 120 (the main runs use 780 steps), where the OOD accuracy change of all three teacher groups lies between −0.39 and +0.94 points (Table 13): no group forgets, and B/C do not retain better than A. B/C have lower ρ because their in-domain change is much larger, not because their OOD change is smaller. The experiment therefore shows that the RLVR increment teaches more, not that it forgets less.
-
The evaluation lacks an overlap analysis and confidence intervals. The paper does not check whether training problems overlap with the math evaluation sets, although, according to its public dataset card, the source of the training problems (NuminaMath 1.5) includes AMC/AIME problems. Four of the five math sets have only 30–40 problems yet carry 80% of the mean, so a single problem shifts the mean by 0.5–0.7 points. Several OOD conclusions rest on differences below 1 point. No confidence intervals are reported.
Questions for authors
-
If the SFT teacher is trained on correct solutions sampled from the model itself, or is matched to the RLVR teacher in retention, does Self-OPD still forget more?
-
Why does A/B/C contain no SFT arm, and why does it stop at step 120? Can these be added?
-
Do the training problems overlap with the evaluation sets? Please report confidence intervals for the main results.
5 comments
Official Comment by Authors
Question: 2. Why does A/B/C contain no SFT arm, and why does it stop at step 120? Can these be added?
Response: The original A/B/C comparison separates the effects of pretrained differences and post-training increments. Due to computational resource constraints, the original manuscript reported only early results at step 120. We have now added the full SFT teacher and its increment, and extended all twenty runs across four teacher–student pairs to the full 780 steps.
Both increment teachers extract post-training changes through log-probability differences and apply them to the student Base distribution. Student initialization, training problems, and OPD settings are held fixed within each comparison. The table reports student in-domain / OOD accuracy (%) at step 780.
| Larger teacher → Student | Larger Base | Full SFT | SFT increment | Full RLVR | RLVR increment |
|---|---|---|---|---|---|
| Qwen2.5-7B → Qwen2.5-1.5B | 15.50 / 34.56 | 9.50 / 34.34 | 6.57 / 35.23 | 20.12 / 36.47 | 19.90 / 36.36 |
| Qwen3-4B → Qwen3-1.7B | 20.40 / 35.20 | 21.37 / 33.77 | 16.26 / 35.42 | 25.69 / 36.56 | 25.46 / 36.64 |
| Qwen3-8B → Qwen3-1.7B | 19.90 / 34.68 | 21.29 / 34.02 | 15.55 / 35.75 | 27.03 / 36.21 | 28.52 / 36.77 |
| Qwen3-8B → Qwen3-4B | 28.07 / 44.90 | 34.77 / 42.36 | 35.49 / 43.91 | 40.37 / 45.60 | 40.11 / 46.04 |
The following table reports distribution changes for the same students at step 780 relative to their respective Base models. Each cell gives in-domain JS / OOD JS / ρ, with both JS values multiplied by 100.
| Larger teacher → Student | Larger Base | Full SFT | SFT increment | Full RLVR | RLVR increment |
|---|---|---|---|---|---|
| Qwen2.5-7B → Qwen2.5-1.5B | 0.606 / 0.833 / 1.376 | 7.461 / 2.868 / 0.384 | 6.969 / 1.624 / 0.233 | 4.713 / 1.206 / 0.256 | 4.557 / 0.940 / 0.206 |
| Qwen3-4B → Qwen3-1.7B | 0.436 / 0.678 / 1.555 | 6.451 / 2.818 / 0.437 | 6.127 / 1.160 / 0.189 | 4.663 / 0.663 / 0.142 | 4.580 / 0.538 / 0.118 |
| Qwen3-8B → Qwen3-1.7B | 0.485 / 0.893 / 1.842 | 6.523 / 2.788 / 0.427 | 6.295 / 1.177 / 0.187 | 4.720 / 0.805 / 0.170 | 4.597 / 0.444 / 0.097 |
| Qwen3-8B → Qwen3-4B | 0.441 / 0.589 / 1.336 | 7.014 / 2.289 / 0.326 | 6.298 / 1.174 / 0.186 | 3.137 / 0.766 / 0.244 | 2.675 / 0.446 / 0.167 |
The full-training results support our original conclusion: post-training a larger model on the same in-domain task before using it as an OPD teacher makes student distribution updates more concentrated in-domain. Across all four pairs, students of both SFT and RLVR teachers have substantially lower ρ than students of the larger Base teacher, indicating less OOD distribution change per unit of in-domain distribution change. Isolating the post-training increment further reduces both ρ and OOD distribution change; in our results, this generally corresponds to less OOD forgetting. These findings show that task-specific preferences acquired through teacher post-training help make student updates more selective.
The RLVR increment teacher preserves in-domain learning gains close to those of the full RLVR teacher while further reducing OOD distribution change, illustrating the value of selective updates for combining learning with capability retention. We have updated Section 4.2 (Table 3) of the main text with the full-training results, including the SFT controls; endpoint results for all four pairs are provided in Appendix D.6 (Table 14).
Official Comment by Authors
Question: 3. Do the training problems overlap with the evaluation sets? Please report confidence intervals for the main results.
Response: We audited the 49,993 training records actually used against all 630 problems in the five in-domain evaluation sets through text normalization, approximate matching, and individual inspection. We found no identical or mathematically equivalent problems in AMC23, AIME24, AIME25, or HMMT25. In Math500, we confirmed 13 problems whose key numerical values and solution targets match corresponding training problems, accounting for 2.6% of the set.
The five evaluation sets are equally weighted, so Math500 contributes 20% of the overall score. Using saved per-problem responses, we recomputed results after excluding these 13 problems. Across the endpoint results for 38 configurations × 2 seeds, the in-domain mean changes by at most 0.14 pp, and the gain over Base by at most 0.11 pp. In the main table, in-domain gaps between key method pairs, including SFT vs RLVR and the two Self-OPD teacher types, change by no more than 0.08 pp, with their ordering unchanged. Given the remaining differences in wording, formatting, or answer-choice presentation, together with the small changes after exclusion, we retain the complete standard Math500 benchmark for comparability and have added the overlaps and exclusion sensitivity results to Appendix B.5. We thank the reviewer for raising this issue.
Regarding the influence of individual problems in the small evaluation sets, under the eight-sample evaluation setting, we average correctness over eight responses per problem, then over problems within each set, and finally over the five sets with equal weights. For a set of 30–40 problems, changing the correctness of one sampled response therefore changes the overall mean by 100/(5 × number of problems × 8), or 0.063–0.083 pp. The reviewer's 0.5–0.7 pp corresponds to a problem's accuracy changing from 0% to 100%, with all eight responses changing from incorrect to correct.
Official Comment by Authors
We sincerely thank the reviewer for recognizing our systematic comparison of teacher sources and insights into the role of post-training information. We address your questions below.
Question: 1. If the SFT teacher is trained on correct solutions sampled from the model itself, or is matched to the RLVR teacher in retention, does Self-OPD still forget more?
Response: Training SFT teachers on their own correct solutions improves student OOD retention, but yields almost no in-domain learning gains. For retention matching, we could not find a checkpoint from the original SFT training that matches the final RLVR teacher’s retention.
The original SFT uses reference solutions generated by DeepSeek-R1 in OpenR1. To examine the role of data source, we use the Qwen2.5-7B, Qwen3-4B, and Qwen3-8B Base models to solve approximately 50,000 training problems from OpenR1-50K. For each problem, we retain only the first correct, normally completed solution within the first eight sampling attempts, yielding 24,082, 27,160, and 29,752 such solutions, respectively, with one solution per solved problem. We use these solutions as each model’s SFT training data and train for one epoch, using each example once. We then use each SFT endpoint as the teacher and train a Self-OPD student from the same Base initialization through step 780.
All accuracy entries in this response use value (change from the corresponding Base before training), with accuracy in % and changes in pp. The first two tables report in-domain / OOD accuracy. We first compare the two SFT settings: SFT on the model’s own correct solutions is evaluated at its endpoints of 377, 425, and 465 steps, and original SFT at the nearest checkpoints of 380, 420, and 460 steps, respectively.
We then compare Self-OPD students trained with the three teacher conditions, all evaluated at step 780. Here, the original SFT and RLVR teachers are from step 780, while the own-solution SFT teachers are from their respective endpoints above.
These results show that Self-OPD students trained with own-solution SFT teachers exhibit little forgetting, but also very limited in-domain learning gains. We further examine these students’ distribution changes. The table reports results at step 780 relative to Base, with each cell showing in-domain JS / OOD JS. Both JS values are multiplied by 100.
With own-solution SFT teachers, students have small OOD JS, but their in-domain JS is also much smaller than under the other two teacher conditions. Together with the near-Base in-domain accuracy reported above, this shows that this setting yields little forgetting but almost no in-domain learning gains.
Regarding teacher retention matching, RLVR shows almost no OOD degradation throughout training, whereas original SFT already shows clear OOD degradation early in training. We therefore could not find an available SFT checkpoint that matches the final RLVR teacher’s retention. The table reports teacher in-domain / OOD accuracy at SFT steps 20 and 100 and RLVR step 780, with changes from Base in parentheses. Appendix C presents the complete in-domain / OOD performance curves throughout SFT and RLVR training.