- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how supervised fine-tuning (SFT) initialization affects subsequent on-policy self-distillation (OPSD). The student and frozen privileged teacher start from the same SFT checkpoint, while the teacher receives additional information such as a reference solution, final answer, or hint. The authors investigate whether standalone student accuracy, teacher accuracy, and measures of student–teacher complementarity and divergence predict downstream OPSD performance. They observe that stronger initial models do not consistently produce better final results, and that lower student–teacher divergence is associated with larger OPSD gains primarily after short SFT warm-ups. Motivated by these observations, they propose Low-Divergence SFT (LD-SFT), which increases the supervised loss weights of low-divergence tokens without changing the training targets. Experiments on Qwen3-4B provide preliminary evidence of improved downstream performance relative to a fixed 100-step Vanilla SFT baseline.
Critical strengths
- The paper addresses a practically relevant question: whether the checkpoint that performs best after SFT is also the best initialization for subsequent distillation. Evaluating initialization through downstream training outcomes is a useful perspective for designing multi-stage post-training pipelines.
- The short-warm-up observational study covers three model scales and three privileged-teacher conditions. Additional experiments with longer warm-ups on Qwen3-4B and Qwen3-8B under Reference conditioning help establish that the observed relationships depend on the training regime.
- The distinction between standalone teacher accuracy, problem-level complementarity, and token-level compatibility is useful. Evaluating teacher distributions on the same student-generated prefixes also makes the token-level diagnostics relevant to the supervision actually encountered during OPSD.
- LD-SFT is a relatively simple intervention that changes token weights while preserving the supervised targets. It uses detached divergence statistics, requires neither a separately trained teacher nor student rollout generation during SFT, and recovers the standard SFT objective when alpha is zero.
- The evaluation includes checkpoint-averaged results and a three-seed study. Both LD-SFT configurations outperform Vanilla SFT on the checkpoint-averaged AIME results in Table 27. In Table 28, alpha = 0.25 improves the reported three-seed mean best OPSD scores by 1.12 and 2.78 percentage points on AIME24 and AIME25, respectively. These results provide preliminary support beyond a single selected checkpoint from one run.
Critical weaknesses
- Important quantitative results are internally inconsistent. The Teacher Average@8 correlation rows contain 24 identical numerical entries across Tables 4 and 7, despite representing different teacher conditions. Recomputing teacher avg@8 versus best OPSD avg@8 from the displayed values in Figures 2, 10, and 11 gives Answer-condition Pearson/Spearman correlations of approximately -0.410/-0.205 for 1.7B, +0.679/+0.821 for 4B, and +0.775/+0.684 for 8B, which disagree with Table 7. Using these displayed values to recompute the three-condition averages changes the 8B Best OPSD correlations in Figure 4 from approximately -0.21/-0.18 to +0.15/+0.15, and the 4B values from +0.47/+0.09 to +0.60/+0.41. These discrepancies affect the main-text quantitative evidence, although they do not by themselves invalidate the broader observation that teacher accuracy is an imperfect predictor. The authors should release the underlying measurements, reconcile the figures and tables, and revise the affected discussion.
- The method evaluation omits a particularly important baseline: Vanilla SFT with warm-up checkpoint selection. Under Reference conditioning, Figures 1(b) and 10 report a higher downstream pooled AIME score from the Qwen3-4B Vanilla checkpoint at step 60 than at step 100, 66.46 versus 64.79. However, the LD-SFT comparisons fix all SFT warm-ups at 100 steps. The reported improvements therefore do not establish an advantage over ordinary tuning of the Vanilla warm-up duration. A comparison using matched training seeds, the same validation-based checkpoint selection procedure, and comparable tuning budgets is needed. The method evaluation is also limited to Qwen3-4B with Reference conditioning. HMMT25 lacks the checkpoint-averaged and multi-seed robustness results provided for AIME, and no multi-seed results are reported for Random-SFT, Easy-SFT, HD-SFT, Top-k OPSD, or EMA OPSD.
- The evidence for the proposed mechanism remains limited. Each condition-specific correlation uses only five checkpoints from a single SFT trajectory. Appendix B.6 usefully controls for parent performance, but the strength of the remaining evidence varies across conditions: the pooled all-teacher p-values are 0.054 for teacher advantage and 0.103 for JSD, while Reference-only JSD has stronger support with p = 0.035. These results support a qualified conditional association. Connecting that association to LD-SFT requires additional evidence. Section 4 measures full-vocabulary divergence on student-generated prefixes, whereas LD-SFT weights supervised CoT prefixes using a top-k approximation with a residual probability bucket. Moreover, changing the SFT parent changes both the student initialization and its frozen teacher, so the comparison does not isolate improved student-side trainability. Measuring the original on-policy diagnostics for LD-SFT parents and adding a shared-teacher or crossed teacher–student initialization ablation would substantially strengthen the mechanistic interpretation.
- The control experiments do not fully separate divergence-based token allocation from changes in optimization scale. Loss-preserving normalization makes the weighted forward loss equal to ordinary cross-entropy at the same parameters and minibatch, but it does not preserve gradient norms or optimizer updates. Similar loss curves from separately trained models therefore do not establish matched update scales. Furthermore, Appendix C.1 and the released implementation reassign already normalized LD-SFT weights in Random-SFT and Easy-SFT without recomputing the loss-preserving normalizer. This preserves the weight multiset but generally changes the CE-weighted loss. In particular, the rearrangement inequality implies that Easy-SFT’s assignment cannot increase that loss relative to the original LD assignment at the same model state, although equality is possible. This does not establish smaller gradients or explain Easy-SFT’s lower accuracy, but it leaves an optimization-scale difference in the comparison. Additional controls that renormalize after reassignment, together with update-magnitude or learning-rate sensitivity measurements, would make the attribution to divergence-based allocation more convincing.
- The practical cost–benefit trade-off is insufficiently quantified. LD-SFT adds a privileged-teacher forward pass and divergence computation at each SFT minibatch. Matching the number of SFT updates therefore does not match training compute, and the paper does not report measured wall-clock overhead or comparisons under matched total training budgets. The downstream OPSD experiments also use a 1,024-token completion cap, but the paper does not establish how often this truncates student trajectories or whether the conclusions persist with longer rollout budgets. Reporting training time, peak memory, generated-token counts, truncation rates, and sensitivity to the rollout cap would clarify whether the accuracy improvements justify the additional cost and remain reliable beyond the chosen training budget.
Questions for authors
Please refer to the weaknesses.
5 comments
W3-a: Each condition-specific correlation uses only five checkpoints from a single SFT trajectory…Connecting that(JSD’s) association to LD-SFT requires additional evidence.
We address this concern with three lines of evidence.
- The negative JSD–OPSD gain correlation persists across 15 checkpoints. Pooling all 15 checkpoint–condition pairs per model scale (3 privileged-information types × 5 SFT checkpoints), we observe negative correlations across all scales for both best and average gain, strongest for Qwen3-1.7B (r = −0.91) and Qwen3-8B (ρ = −0.55). See below.
Pooled Correlation Metrics for JSD Divergence Between Student and Privileged Teacher Across Different Information Settings (n=15)
| Model | Best gain r / ρ | Avg gain r / ρ |
|---|---|---|
| Qwen3-1.7B | −0.914 / −0.839 | −0.918 / −0.850 |
| Qwen3-4B | −0.371 / −0.268 | −0.359 / −0.236 |
| Qwen3-8B | −0.472 / −0.550 | −0.484 / −0.564 |
- The association persists after controlling for confounders. Regressions with model and teacher fixed effects, parent accuracy controls, and checkpoint-clustered standard errors show that lower JSD predicts better post-OPSD performance in 8/9 settings, with a significant Reference-only association (p = 0.035; See Appendix B.6).
- Intervention that reduces JSD further improves OPSD performance. LD-SFT lowers on-policy teacher–student JSD at 9/10 warm-up checkpoints (See our response to W3-c) and improves downstream OPSD, providing intervention-based evidence beyond observational correlations.
W3-b:Section 4 measures full-vocabulary divergence on student-generated prefixes, whereas LD-SFT weights supervised CoT prefixes using a top-k approximation with a residual probability bucket.
We address both differences with findings.
- Top-128 JSD closely preserves full-vocabulary token selection and weighting. For LD-SFT (α = 0.25), the two estimates show near-perfect rank correlation (ρ = 0.99954), 99.34% overlap among the lowest-divergence 25% of tokens, and highly correlated token weights (r = 0.99892).
- LD-SFT's benefits extend to student-generated prefixes. Although trained with weights computed on supervised CoT prefixes, LD-SFT also reduces full-vocabulary JSD on student-generated prefixes (See our response to W3-c), using the same metric as Section 4.
W3-c: Measuring the original on-policy diagnostics for LD-SFT parents and adding a shared-teacher or crossed teacher–student initialization ablation would substantially strengthen the mechanistic interpretation.
1. LD-SFT can indeed lower on-policy JSD within 100 steps. To link LD-SFT to our on-policy diagnostics, we measure full-vocabulary teacher–student JSD on student-generated prefixes. Relative to Vanilla-SFT at matched steps, LD-SFT lowers on-policy JSD at 9 of 10 checkpoints. This supports our proposed mechanism: low-divergence weighting alters the on-policy diagnostics associated with OPSD gains (Section 4).
Full-vocabulary on-policy JSD by warm-up step
| SFT step | Vanilla SFT | LD-SFT α = 0.25 | LD-SFT α = 1 |
|---|---|---|---|
| 20 | 0.015131 | 0.015109 ↓ | 0.015087 ↓ |
| 40 | 0.015569 | 0.015356 ↓ | 0.015142 ↓ |
| 60 | 0.020101 | 0.019979 ↓ | 0.019975 ↓ |
| 80 | 0.019767 | 0.019632 ↓ | 0.019751 ↓ |
| 100 | 0.019496 | 0.019417 ↓ | 0.019534 ↑ |
2. We clarify that crossed or shared-teacher ablation falls outside the scope of self-distillation. In OPSD, the teacher is the student’s own initialization with privileged information, so initialization defines the teacher–student pair. We therefore study how initialization affects both the teacher and the student. Using a teacher from a different initialization would violate the self-distillation setting.
W4: The control experiments do not fully separate divergence-based token allocation from changes in optimization scale… the released implementation reassigns already normalized LD-SFT weights in Random-SFT and Easy-SFT without recomputing the loss-preserving normalizer.
We clarify that LD-SFT’s gains stem from low-divergence token weighting rather than optimization-scale differences, as supported by two findings:
- Comparable optimization scales among Easy-SFT, Random-SFT, and LD-SFT. Without renormalization, gradient norms differ by at most 5.8% and checkpoint-level update magnitudes by only 1.7% across LD-SFT, Easy-SFT, and Random-SFT, making optimization scale an unlikely explanation for LD-SFT’s gains.
| Method | Mean checkpoint-level update magnitude | Gradient norm |
|---|---|---|
| LD α=0.25 | 0.005257 | 0.0385 |
| LD α=1 | 0.005304 | 0.0377 |
| Random-SFT | 0.005348 | 0.0399 |
| Easy-SFT | 0.005325 | 0.0379 |
- Persistent gains of LD-SFT without renormalization. We re-ran LD-SFT without renormalization (see below table), ensuring all three methods use the well-controlled set-up that differs only in token-weight assignment. LD-SFT still outperforms both baselines .
| Method | Mean Avg@8 across AIME 24, AIME 25, and HMMT 25 |
|---|---|
| Random-SFT + OPSD | 56.53 |
| Easy-SFT + OPSD | 56.53 |
| No-renorm LD-SFT α=0.25 + OPSD | 57.08 |
| No-renorm LD-SFT α=1 + OPSD | 58.06 |
W5: The practical cost–benefit trade-off is insufficiently quantified.
LD-SFT has comparable memory and computational costs to Vanilla SFT, with only a marginal increase in total training time. We’ve added these into Appendix C.6 of the revised pdf.
- LD-SFT incurs negligible memory overhead and only a modest runtime increase. Under identical settings on one H100 80GB GPU, peak allocated memory remains nearly unchanged (27.34 vs. 27.33 GB), as the teacher forward pass is gradient-free. The additional runtime contributes only ~5% to total training cost, since downstream OPSD accounts for over 80%.
| Method | Peak allocated memory (GB) | Peak reserved memory (GB) | Step time (s) | SFT time, 100 steps (s) | Total SFT + OPSD time (h)* |
|---|---|---|---|---|---|
| Vanilla SFT | 27.33 | 57.21 | 14.03 ± 0.98 | ≈ 1,403 | ≈ 2.54–3.40 |
| LD-SFT (α = 0.25) | 27.34 | 62.96 | 19.84 ± 1.38 | ≈ 1,984 | ≈ 2.68–3.53 |
| LD-SFT (α = 1) | 27.34 | 62.96 | 19.80 ± 1.38 | ≈ 1,980 | ≈ 2.66–3.50 |
| Initialization | Mean tokens / rollout | Total sampled tokens |
|---|---|---|
| Vanilla SFT-100 | 998.9 | 3,068,649 |
| LD α = 0.25 | 999.3 | 3,069,982 |
| LD α = 1 | 996.5 | 3,061,369 |
- LD-SFT's gains stem from reweighted supervision rather than extra compute. Allocating the additional compute to longer Vanilla SFT does not close the gap: extending Vanilla SFT to 500 steps worsens downstream OPSD performance (61.46 at 500 steps vs. 63.18 at 100 steps; See our response to Reviewer t9gG).
- Truncation affects all methods similarly and does not explain LD-SFT's gains. Following OPSD [1], we use a 1,024-token completion cap, yielding similar truncation rates (87–96%) and generated-token counts across methods. Moreover, OPSD provides dense token-level supervision even on truncated rollouts, without relying on final-answer rewards. All evaluations use a 30,000-token generation budget (Appendix A.4), confirming gains under full-length generation.
EOS-missing (Truncation) rate across OPSD training stages, initialized from 100 steps of SFT variants (%)
| SFT variant | Step 5 | Step 50 | Step 95 |
|---|---|---|---|
| Vanilla | 91.67 | 89.38 | 96.25 |
| Random | 91.15 | 87.50 | 93.75 |
| Easy | 91.15 | 88.75 | 93.75 |
| HD | 89.06 | 91.88 | 95.00 |
| LD α = 0.25 | 90.62 | 94.38 | 90.00 |
| LD α = 1 | 92.71 | 92.50 | 88.75 |
[1] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. ICML 2026.
W1: The Teacher Average@8 correlation rows contain 24 identical numerical entries across Tables 4 and 7, despite representing different teacher conditions.We thank the reviewer for identifying this issue and apologize for the confusion caused by the Appendix tables. The duplicated values in Tables 4 and 7 resulted from an incorrect CSV path in our correlation script, affecting the Appendix Teacher Average@8 correlations for the answer conditioning. All underlying evaluations and other reported results remain correct and our overall conclusions remain completely unchanged.
The corrected Pearson/Spearman correlations for answer condition in Table 7 are −0.402/−0.205 (1.7B), +0.681/+0.821 (4B), and +0.775/+0.684 (8B), consistent with Figures 2, 10, and 11. We also corrected Figure 4 and Table 13, which average across three conditions, with updated correlations of −0.56/−0.48 (1.7B), +0.60/+0.41 (4B), and +0.15/+0.15 (8B), matching the reviewer's recomputation.
All affected results have been updated in the manuscript. Our conclusion remains unchanged: teacher accuracy alone does not reliably predict downstream OPSD performance across model scales and teacher conditions.
W2-a: The method evaluation omits a particularly important baseline: Vanilla SFT with warm-up checkpoint selection… A comparison using matched training seeds, the same validation-based checkpoint selection procedure, and comparable tuning budgets is needed.LD-SFT outperforms Vanilla SFT using same validation-based warm-up checkpoint selection, comparable tuning budgets and matched training seeds.
Under Reference conditioning, all methods use training seed 0 and the same grid: SFT steps {20, 40, 60, 80, 100} × OPSD steps {25, 50, 75, 100}. For each method, we jointly select both checkpoints by avg@8 on a shared, independent 200-sample hold-out set, disjoint from training and AIME test data. Each fixed α setting receives the same 20-configuration search budget as Vanilla SFT and is reported separately.
Moreover, LD-SFT (α = 1) outperforms Vanilla SFT + OPSD at all 8 fixed OPSD-step comparisons across AIME24 and AIME25, confirming that its gains do not depend on checkpoint selection (see our response to Reviewer 2tKc’s Q1).
W2-c: The method evaluation is also limited to Qwen3-4B with Reference conditioning.LD-SFT also works on Qwen3-1.7B and 8B. In addition to Qwen3-4B, we also add experiments on Qwen3-1.7B and 8B. At both additional scales, OPSD initialized from LD-SFT reaches higher final accuracy than OPSD initialized from Vanilla SFT on all three benchmarks.
W2-d: HMMT25 lacks the checkpoint-averaged and multi-seed robustness results provided for AIME, and no multi-seed results are reported for Random-SFT, Easy-SFT, HD-SFT, Top-k OPSD, or EMA OPSD.LD-SFT outperforms Vanilla SFT on HMMT25 in both mean (38.75% vs. 38.12%) and best accuracy (43.33% vs. 39.58%). Below reports OPSD avg@8 across training checkpoints.
LD-SFT works across seeds. We extend the three-seed results in Table 30 (Appendix C.3) to all baselines and to HMMT25. LD-SFT achieves the highest mean on all three benchmarks: AIME24 (70.6, α = 0.25), AIME25 (62.9, α = 1), and HMMT25 (41.0, α = 1). On AIME25, both LD-SFT variants exceed every baseline. These results have been updated into the revised pdf.