acceptodds
Under review as a conference paper at ICLR 2027

Stronger, Yet Not Better: Demystifying Self-Distillation Initialization

Abstract

On-policy Self-Distillation (OPSD) efficiently trains LLMs by providing dense, token-level teacher supervision on the student's own rollouts. However, this supervision degrades when student errors push the student's policy too far from the teacher's, making OPSD highly sensitive to initialization: the student checkpoint and teacher configuration that keep supervision reliable and learnable. Prior OPSD methods often assume that stronger students or privileged teachers lead to better distillation. We challenge this assumption with hundreds of experiments spanning model scales, SFT checkpoints, and privileged teacher configurations. At the problem level, higher teacher accuracy on student-failed problems predicts larger post-initialization OPSD gains. At the token level, lower teacher–student divergence predicts larger gains when the student is undertrained, while this relationship weakens or reverses with overfitting. These results show that effective initialization depends not only on model strength, but also on teacher complementarity and learnable teacher–student divergence. Guided by these findings, we introduce a divergence-aware SFT warm-up that upweights low-divergence tokens during OPSD initialization, prioritizing easier-to-learn targets. It outperforms standard SFT+OPSD, yielding larger gains over initialization on AIME24 (+11.67 vs. +5.00) and AIME25 (+11.25 vs. +6.25). Overall, our results suggest that future methods should optimize OPSD initialization for both model capability and teacher–student learnability, rather than simply choosing stronger students or teachers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy self-distillation, supervised fine-tuning, large language models, post-training

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_6cCyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how supervised fine-tuning (SFT) initialization affects subsequent on-policy self-distillation (OPSD). The student and frozen privileged teacher start from the same SFT checkpoint, while the teacher receives additional information such as a reference solution, final answer, or hint. The authors investigate whether standalone student accuracy, teacher accuracy, and measures of student–teacher complementarity and divergence predict downstream OPSD performance. They observe that stronger initial models do not consistently produce better final results, and that lower student–teacher divergence is associated with larger OPSD gains primarily after short SFT warm-ups. Motivated by these observations, they propose Low-Divergence SFT (LD-SFT), which increases the supervised loss weights of low-divergence tokens without changing the training targets. Experiments on Qwen3-4B provide preliminary evidence of improved downstream performance relative to a fixed 100-step Vanilla SFT baseline.

Critical strengths
  • The paper addresses a practically relevant question: whether the checkpoint that performs best after SFT is also the best initialization for subsequent distillation. Evaluating initialization through downstream training outcomes is a useful perspective for designing multi-stage post-training pipelines.
  • The short-warm-up observational study covers three model scales and three privileged-teacher conditions. Additional experiments with longer warm-ups on Qwen3-4B and Qwen3-8B under Reference conditioning help establish that the observed relationships depend on the training regime.
  • The distinction between standalone teacher accuracy, problem-level complementarity, and token-level compatibility is useful. Evaluating teacher distributions on the same student-generated prefixes also makes the token-level diagnostics relevant to the supervision actually encountered during OPSD.
  • LD-SFT is a relatively simple intervention that changes token weights while preserving the supervised targets. It uses detached divergence statistics, requires neither a separately trained teacher nor student rollout generation during SFT, and recovers the standard SFT objective when alpha is zero.
  • The evaluation includes checkpoint-averaged results and a three-seed study. Both LD-SFT configurations outperform Vanilla SFT on the checkpoint-averaged AIME results in Table 27. In Table 28, alpha = 0.25 improves the reported three-seed mean best OPSD scores by 1.12 and 2.78 percentage points on AIME24 and AIME25, respectively. These results provide preliminary support beyond a single selected checkpoint from one run.
Critical weaknesses
  • Important quantitative results are internally inconsistent. The Teacher Average@8 correlation rows contain 24 identical numerical entries across Tables 4 and 7, despite representing different teacher conditions. Recomputing teacher avg@8 versus best OPSD avg@8 from the displayed values in Figures 2, 10, and 11 gives Answer-condition Pearson/Spearman correlations of approximately -0.410/-0.205 for 1.7B, +0.679/+0.821 for 4B, and +0.775/+0.684 for 8B, which disagree with Table 7. Using these displayed values to recompute the three-condition averages changes the 8B Best OPSD correlations in Figure 4 from approximately -0.21/-0.18 to +0.15/+0.15, and the 4B values from +0.47/+0.09 to +0.60/+0.41. These discrepancies affect the main-text quantitative evidence, although they do not by themselves invalidate the broader observation that teacher accuracy is an imperfect predictor. The authors should release the underlying measurements, reconcile the figures and tables, and revise the affected discussion.
  • The method evaluation omits a particularly important baseline: Vanilla SFT with warm-up checkpoint selection. Under Reference conditioning, Figures 1(b) and 10 report a higher downstream pooled AIME score from the Qwen3-4B Vanilla checkpoint at step 60 than at step 100, 66.46 versus 64.79. However, the LD-SFT comparisons fix all SFT warm-ups at 100 steps. The reported improvements therefore do not establish an advantage over ordinary tuning of the Vanilla warm-up duration. A comparison using matched training seeds, the same validation-based checkpoint selection procedure, and comparable tuning budgets is needed. The method evaluation is also limited to Qwen3-4B with Reference conditioning. HMMT25 lacks the checkpoint-averaged and multi-seed robustness results provided for AIME, and no multi-seed results are reported for Random-SFT, Easy-SFT, HD-SFT, Top-k OPSD, or EMA OPSD.
  • The evidence for the proposed mechanism remains limited. Each condition-specific correlation uses only five checkpoints from a single SFT trajectory. Appendix B.6 usefully controls for parent performance, but the strength of the remaining evidence varies across conditions: the pooled all-teacher p-values are 0.054 for teacher advantage and 0.103 for JSD, while Reference-only JSD has stronger support with p = 0.035. These results support a qualified conditional association. Connecting that association to LD-SFT requires additional evidence. Section 4 measures full-vocabulary divergence on student-generated prefixes, whereas LD-SFT weights supervised CoT prefixes using a top-k approximation with a residual probability bucket. Moreover, changing the SFT parent changes both the student initialization and its frozen teacher, so the comparison does not isolate improved student-side trainability. Measuring the original on-policy diagnostics for LD-SFT parents and adding a shared-teacher or crossed teacher–student initialization ablation would substantially strengthen the mechanistic interpretation.
  • The control experiments do not fully separate divergence-based token allocation from changes in optimization scale. Loss-preserving normalization makes the weighted forward loss equal to ordinary cross-entropy at the same parameters and minibatch, but it does not preserve gradient norms or optimizer updates. Similar loss curves from separately trained models therefore do not establish matched update scales. Furthermore, Appendix C.1 and the released implementation reassign already normalized LD-SFT weights in Random-SFT and Easy-SFT without recomputing the loss-preserving normalizer. This preserves the weight multiset but generally changes the CE-weighted loss. In particular, the rearrangement inequality implies that Easy-SFT’s assignment cannot increase that loss relative to the original LD assignment at the same model state, although equality is possible. This does not establish smaller gradients or explain Easy-SFT’s lower accuracy, but it leaves an optimization-scale difference in the comparison. Additional controls that renormalize after reassignment, together with update-magnitude or learning-rate sensitivity measurements, would make the attribution to divergence-based allocation more convincing.
  • The practical cost–benefit trade-off is insufficiently quantified. LD-SFT adds a privileged-teacher forward pass and divergence computation at each SFT minibatch. Matching the number of SFT updates therefore does not match training compute, and the paper does not report measured wall-clock overhead or comparisons under matched total training budgets. The downstream OPSD experiments also use a 1,024-token completion cap, but the paper does not establish how often this truncates student trajectories or whether the conclusions persist with longer rollout budgets. Reporting training time, peak memory, generated-token counts, truncation rates, and sensitivity to the rollout cap would clarify whether the accuracy improvements justify the additional cost and remain reliable beyond the chosen training budget.
Questions for authors

Please refer to the weaknesses.

5 comments

W1: The Teacher Average@8 correlation rows contain 24 identical numerical entries across Tables 4 and 7, despite representing different teacher conditions.

We thank the reviewer for identifying this issue and apologize for the confusion caused by the Appendix tables. The duplicated values in Tables 4 and 7 resulted from an incorrect CSV path in our correlation script, affecting the Appendix Teacher Average@8 correlations for the answer conditioning. All underlying evaluations and other reported results remain correct and our overall conclusions remain completely unchanged.

The corrected Pearson/Spearman correlations for answer condition in Table 7 are −0.402/−0.205 (1.7B), +0.681/+0.821 (4B), and +0.775/+0.684 (8B), consistent with Figures 2, 10, and 11. We also corrected Figure 4 and Table 13, which average across three conditions, with updated correlations of −0.56/−0.48 (1.7B), +0.60/+0.41 (4B), and +0.15/+0.15 (8B), matching the reviewer's recomputation.

All affected results have been updated in the manuscript. Our conclusion remains unchanged: teacher accuracy alone does not reliably predict downstream OPSD performance across model scales and teacher conditions.


W2-a: The method evaluation omits a particularly important baseline: Vanilla SFT with warm-up checkpoint selection… A comparison using matched training seeds, the same validation-based checkpoint selection procedure, and comparable tuning budgets is needed.

LD-SFT outperforms Vanilla SFT using same validation-based warm-up checkpoint selection, comparable tuning budgets and matched training seeds.

Under Reference conditioning, all methods use training seed 0 and the same grid: SFT steps {20, 40, 60, 80, 100} × OPSD steps {25, 50, 75, 100}. For each method, we jointly select both checkpoints by avg@8 on a shared, independent 200-sample hold-out set, disjoint from training and AIME test data. Each fixed α setting receives the same 20-configuration search budget as Vanilla SFT and is reported separately.

MethodSFT stepOPSD stepAIME24AIME25Pooled
Vanilla SFT605070.0062.9266.46
LD-SFT α = 0.251005072.9262.0867.50
LD-SFT α = 1.0607567.9266.2567.09

Moreover, LD-SFT (α = 1) outperforms Vanilla SFT + OPSD at all 8 fixed OPSD-step comparisons across AIME24 and AIME25, confirming that its gains do not depend on checkpoint selection (see our response to Reviewer 2tKc’s Q1).


W2-c: The method evaluation is also limited to Qwen3-4B with Reference conditioning.

LD-SFT also works on Qwen3-1.7B and 8B. In addition to Qwen3-4B, we also add experiments on Qwen3-1.7B and 8B. At both additional scales, OPSD initialized from LD-SFT reaches higher final accuracy than OPSD initialized from Vanilla SFT on all three benchmarks.

ModelMethod / stageAIME24 (Δ)AIME25 (Δ)HMMT25 (Δ)
1.7BVanilla SFT35.8326.2515.83
↳ OPSD42.08 (+6.25)32.50 (+6.25)20.42 (+4.59)
1.7BLD-SFT35.0031.2517.50
↳ OPSD44.58 (+9.58)34.58 (+3.33)22.50 (+5.00)
8BVanilla SFT65.8355.0033.33
↳ OPSD73.75 (+7.92)62.92 (+7.92)41.67 (+8.34)
8BLD-SFT65.4249.5833.33
↳ OPSD75.00 (+9.58)63.33 (+13.75)42.92 (+9.59)

W2-d: HMMT25 lacks the checkpoint-averaged and multi-seed robustness results provided for AIME, and no multi-seed results are reported for Random-SFT, Easy-SFT, HD-SFT, Top-k OPSD, or EMA OPSD.

LD-SFT outperforms Vanilla SFT on HMMT25 in both mean (38.75% vs. 38.12%) and best accuracy (43.33% vs. 39.58%). Below reports OPSD avg@8 across training checkpoints.

MethodS25S50S75S100MeanBest
Vanilla SFT37.5035.8339.5839.5838.1239.58
LD α = 1.039.5843.3335.0037.0838.7543.33

LD-SFT works across seeds. We extend the three-seed results in Table 30 (Appendix C.3) to all baselines and to HMMT25. LD-SFT achieves the highest mean on all three benchmarks: AIME24 (70.6, α = 0.25), AIME25 (62.9, α = 1), and HMMT25 (41.0, α = 1). On AIME25, both LD-SFT variants exceed every baseline. These results have been updated into the revised pdf.

Method / stageAIME24AIME25HMMT25
Vanilla SFT63.9 ± 1.354.0 ± 0.631.7 ± 2.3
↳ OPSD69.4 ± 0.560.0 ± 1.140.1 ± 1.3
↳ EMA OPSD70.3 ± 0.961.4 ± 2.540.1 ± 2.1
↳ Top-k OPSD70.3 ± 1.361.5 ± 0.539.4 ± 1.9
Random-SFT62.6 ± 0.254.4 ± 2.932.9 ± 3.3
↳ OPSD69.9 ± 0.960.3 ± 2.340.0 ± 1.1
Easy-SFT59.7 ± 3.352.6 ± 2.434.4 ± 1.9
↳ OPSD70.1 ± 3.261.7 ± 0.440.6 ± 0.2
HD-SFT60.8 ± 0.852.1 ± 1.432.9 ± 1.8
↳ OPSD70.3 ± 0.659.6 ± 1.540.0 ± 1.5
LD-SFT (α = 0.25)61.4 ± 2.753.5 ± 2.432.1 ± 1.9
↳ OPSD70.6 ± 2.362.4 ± 1.639.3 ± 1.3
LD-SFT (α = 1)60.8 ± 2.551.8 ± 4.131.5 ± 2.3
↳ OPSD69.6 ± 0.462.9 ± 0.441.0 ± 2.1

W3-a: Each condition-specific correlation uses only five checkpoints from a single SFT trajectory…Connecting that(JSD’s) association to LD-SFT requires additional evidence.

We address this concern with three lines of evidence.

  • The negative JSD–OPSD gain correlation persists across 15 checkpoints. Pooling all 15 checkpoint–condition pairs per model scale (3 privileged-information types × 5 SFT checkpoints), we observe negative correlations across all scales for both best and average gain, strongest for Qwen3-1.7B (r = −0.91) and Qwen3-8B (ρ = −0.55). See below.

Pooled Correlation Metrics for JSD Divergence Between Student and Privileged Teacher Across Different Information Settings (n=15)

ModelBest gain r / ρAvg gain r / ρ
Qwen3-1.7B−0.914 / −0.839−0.918 / −0.850
Qwen3-4B−0.371 / −0.268−0.359 / −0.236
Qwen3-8B−0.472 / −0.550−0.484 / −0.564
  • The association persists after controlling for confounders. Regressions with model and teacher fixed effects, parent accuracy controls, and checkpoint-clustered standard errors show that lower JSD predicts better post-OPSD performance in 8/9 settings, with a significant Reference-only association (p = 0.035; See Appendix B.6).
  • Intervention that reduces JSD further improves OPSD performance. LD-SFT lowers on-policy teacher–student JSD at 9/10 warm-up checkpoints (See our response to W3-c) and improves downstream OPSD, providing intervention-based evidence beyond observational correlations.

W3-b:Section 4 measures full-vocabulary divergence on student-generated prefixes, whereas LD-SFT weights supervised CoT prefixes using a top-k approximation with a residual probability bucket.

We address both differences with findings.

  • Top-128 JSD closely preserves full-vocabulary token selection and weighting. For LD-SFT (α = 0.25), the two estimates show near-perfect rank correlation (ρ = 0.99954), 99.34% overlap among the lowest-divergence 25% of tokens, and highly correlated token weights (r = 0.99892).
  • LD-SFT's benefits extend to student-generated prefixes. Although trained with weights computed on supervised CoT prefixes, LD-SFT also reduces full-vocabulary JSD on student-generated prefixes (See our response to W3-c), using the same metric as Section 4.

W3-c: Measuring the original on-policy diagnostics for LD-SFT parents and adding a shared-teacher or crossed teacher–student initialization ablation would substantially strengthen the mechanistic interpretation.

1. LD-SFT can indeed lower on-policy JSD within 100 steps. To link LD-SFT to our on-policy diagnostics, we measure full-vocabulary teacher–student JSD on student-generated prefixes. Relative to Vanilla-SFT at matched steps, LD-SFT lowers on-policy JSD at 9 of 10 checkpoints. This supports our proposed mechanism: low-divergence weighting alters the on-policy diagnostics associated with OPSD gains (Section 4).

Full-vocabulary on-policy JSD by warm-up step

SFT stepVanilla SFTLD-SFT α = 0.25LD-SFT α = 1
200.0151310.015109 ↓0.015087 ↓
400.0155690.015356 ↓0.015142 ↓
600.0201010.019979 ↓0.019975 ↓
800.0197670.019632 ↓0.019751 ↓
1000.0194960.019417 ↓0.019534 ↑

2. We clarify that crossed or shared-teacher ablation falls outside the scope of self-distillation. In OPSD, the teacher is the student’s own initialization with privileged information, so initialization defines the teacher–student pair. We therefore study how initialization affects both the teacher and the student. Using a teacher from a different initialization would violate the self-distillation setting.

W4: The control experiments do not fully separate divergence-based token allocation from changes in optimization scale… the released implementation reassigns already normalized LD-SFT weights in Random-SFT and Easy-SFT without recomputing the loss-preserving normalizer.

We clarify that LD-SFT’s gains stem from low-divergence token weighting rather than optimization-scale differences, as supported by two findings:

  1. Comparable optimization scales among Easy-SFT, Random-SFT, and LD-SFT. Without renormalization, gradient norms differ by at most 5.8% and checkpoint-level update magnitudes by only 1.7% across LD-SFT, Easy-SFT, and Random-SFT, making optimization scale an unlikely explanation for LD-SFT’s gains.
MethodMean checkpoint-level update magnitudeGradient norm
LD α=0.250.0052570.0385
LD α=10.0053040.0377
Random-SFT0.0053480.0399
Easy-SFT0.0053250.0379
  1. Persistent gains of LD-SFT without renormalization. We re-ran LD-SFT without renormalization (see below table), ensuring all three methods use the well-controlled set-up that differs only in token-weight assignment. LD-SFT still outperforms both baselines .
MethodMean Avg@8 across AIME 24, AIME 25, and HMMT 25
Random-SFT + OPSD56.53
Easy-SFT + OPSD56.53
No-renorm LD-SFT α=0.25 + OPSD57.08
No-renorm LD-SFT α=1 + OPSD58.06


W5: The practical cost–benefit trade-off is insufficiently quantified.

LD-SFT has comparable memory and computational costs to Vanilla SFT, with only a marginal increase in total training time. We’ve added these into Appendix C.6 of the revised pdf.

  1. LD-SFT incurs negligible memory overhead and only a modest runtime increase. Under identical settings on one H100 80GB GPU, peak allocated memory remains nearly unchanged (27.34 vs. 27.33 GB), as the teacher forward pass is gradient-free. The additional runtime contributes only ~5% to total training cost, since downstream OPSD accounts for over 80%.
MethodPeak allocated memory (GB)Peak reserved memory (GB)Step time (s)SFT time, 100 steps (s)Total SFT + OPSD time (h)*
Vanilla SFT27.3357.2114.03 ± 0.98≈ 1,403≈ 2.54–3.40
LD-SFT (α = 0.25)27.3462.9619.84 ± 1.38≈ 1,984≈ 2.68–3.53
LD-SFT (α = 1)27.3462.9619.80 ± 1.38≈ 1,980≈ 2.66–3.50
InitializationMean tokens / rolloutTotal sampled tokens
Vanilla SFT-100998.93,068,649
LD α = 0.25999.33,069,982
LD α = 1996.53,061,369
  1. LD-SFT's gains stem from reweighted supervision rather than extra compute. Allocating the additional compute to longer Vanilla SFT does not close the gap: extending Vanilla SFT to 500 steps worsens downstream OPSD performance (61.46 at 500 steps vs. 63.18 at 100 steps; See our response to Reviewer t9gG).
  2. Truncation affects all methods similarly and does not explain LD-SFT's gains. Following OPSD [1], we use a 1,024-token completion cap, yielding similar truncation rates (87–96%) and generated-token counts across methods. Moreover, OPSD provides dense token-level supervision even on truncated rollouts, without relying on final-answer rewards. All evaluations use a 30,000-token generation budget (Appendix A.4), confirming gains under full-length generation.

EOS-missing (Truncation) rate across OPSD training stages, initialized from 100 steps of SFT variants (%)

SFT variantStep 5Step 50Step 95
Vanilla91.6789.3896.25
Random91.1587.5093.75
Easy91.1588.7593.75
HD89.0691.8895.00
LD α = 0.2590.6294.3890.00
LD α = 192.7192.5088.75

[1] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. ICML 2026.

Reviewer_2tKcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to initialize on-policy self-distillation (OPSD) for reasoning-capable large language models (LLMs), in which the same model acts as a “teacher” given a privileged context (reference solution, answer, or prompt).

Research findings (Qwen3-1.7B/4B/8B, varying SFT lengths and teacher types, OpenThoughts math tasks):

  • While longer SFTs improve SFT accuracy, they cause downstream OPSD gains to plateau or even decline; brief warm-up training is sufficient.
  • Teacher accuracy is a poor predictor of the quality of the distilled student model (prompt-based teachers have lower accuracy but produce higher-quality student models).
  • OPSD gain is correlated with teacher complementarity at the problem level; under short SFT conditions, it is also correlated with low teacher-student divergence at the token level (this relationship weakens or reverses as SFT lengthens).

Method: LD-SFT reweights the SFT token loss to prioritize tokens with low divergence (percentile-ranked weights, loss-normalized).

Results: With the same OPSD, LD-SFT achieved scores of 72.9, 63.8, and 43.3 on AIME24, AIME25, and HMMT25, respectively, while standard SFT achieved only 69.2, 60.4, and 39.6 (for example, an 11.67-point improvement on AIME24, compared to only 5.00 points for standard SFT).

Critical strengths
  1. A timely question whose conclusion is both actionable and surprising (a brief warm-up is sufficient; a stronger SFT or a more precise teacher does not necessarily lead to a better OPSD), and this conclusion is supported by a systematic analysis of model size, SFT length, and the type of privileged information.

  2. A simple and well-controlled method: LD-SFT keeps the data and objectives unchanged, maintains the loss scale, and compares itself with reasonable alternatives (random/simple/highly divergent SFTs, Top-k, and EMA OPSD), complete with bootstrap confidence intervals and random number seeds provided in the appendix.

Critical weaknesses

The primary improvements are relative to a low starting point (LD-SFT’s starting point is lower than that of the standard SFT; Easy-SFT also achieved similar improvements of +11.25 and +11.67, respectively), so part of the effect stems from the room for improvement. The scope of the study is narrow (Qwen3, mathematics, non-thinking modes), and the explanation of “ease of use and novelty” constitutes a post hoc attribution.

Questions for authors
  1. Do the main results still hold when a fixed (or validation-selected) OPSD step size and multiple seeds are used in the main table?

  2. Does the phenomenon that “a stronger SFT weakens the OPSD boost” stem in part from the “headroom effect”? Can the final OPSD performance at the matching initial accuracy be used as a benchmark in the analysis?

  3. Can LD-SFT be transferred to other model families, non-mathematical tasks, or other self-distillation variants (such as SDFT and SDPO)?

2 comments

Q1: Do the main results still hold when a fixed (or validation-selected) OPSD step size and multiple seeds are used in the main table?

1. Our conclusion holds across fixed OPSD steps. Figure 8 already compares all SFT variants at identical OPSD steps (25, 50, 75, 100). We now provide exact values and add EMA OPSD and Top-k OPSD baselines to Table 28, Appendix C of the revised PDF. LD-SFT (α = 1) outperforms Vanilla SFT + OPSD in all 8 step–benchmark combinations on AIME24/25. An LD-SFT variant also achieves the highest or tied-highest score in 7 of 8 cases, except AIME24 at step 75, where Top-k OPSD leads. Thus, the conclusion does not depend on which OPSD step is chosen.

AIME 24AIME 25
VariantStep 25Step 50Step 75Step 100Step 25Step 50Step 75Step 100
Vanilla-SFT + OPSD68.7567.9267.0869.1756.2559.1756.6760.42
EASY-SFT + OPSD63.7565.4267.5067.5061.6759.5857.0859.17
Random-SFT + OPSD68.7568.7567.5069.5858.3357.5056.2558.75
HD-SFT + OPSD69.1769.5868.3370.4259.1757.9254.1760.00
Vanilla-SFT + EMA OPSD66.6770.0070.0063.3358.7558.7555.8358.75
Vanilla-SFT + Top-k OPSD66.6766.2571.2565.4260.0062.0860.0060.42
LD-SFT (α=0.25) + OPSD68.3372.9270.0070.4261.6762.0860.0057.50
LD-SFT (α=1) + OPSD69.5870.0067.5070.0063.7562.9257.0862.50

2. LD-SFT works across seeds. We extend the three-seed results in Table 30 (Appendix C.3) to all baselines and to HMMT25. LD-SFT achieves the highest mean on all three benchmarks: AIME24 (70.6, α = 0.25), AIME25 (62.9, α = 1), and HMMT25 (41.0, α = 1). On AIME25, both LD-SFT variants exceed every baseline. These results have been updated into the revised pdf.

Method / stageAIME24AIME25HMMT25
Vanilla SFT63.9 ± 1.354.0 ± 0.631.7 ± 2.3
↳ OPSD69.4 ± 0.560.0 ± 1.140.1 ± 1.3
↳ EMA OPSD70.3 ± 0.961.4 ± 2.540.1 ± 2.1
↳ Top-k OPSD70.3 ± 1.361.5 ± 0.539.4 ± 1.9
Random-SFT62.6 ± 0.254.4 ± 2.932.9 ± 3.3
↳ OPSD69.9 ± 0.960.3 ± 2.340.0 ± 1.1
Easy-SFT59.7 ± 3.352.6 ± 2.434.4 ± 1.9
↳ OPSD70.1 ± 3.261.7 ± 0.440.6 ± 0.2
HD-SFT60.8 ± 0.852.1 ± 1.432.9 ± 1.8
↳ OPSD70.3 ± 0.659.6 ± 1.540.0 ± 1.5
LD-SFT (α = 0.25)61.4 ± 2.753.5 ± 2.432.1 ± 1.9
↳ OPSD70.6 ± 2.362.4 ± 1.639.3 ± 1.3
LD-SFT (α = 1)60.8 ± 2.551.8 ± 4.131.5 ± 2.3
↳ OPSD69.6 ± 0.462.9 ± 0.441.0 ± 2.1

Q2: Does the phenomenon that “a stronger SFT weakens the OPSD boost” stem in part from the “headroom effect”? Can the final OPSD performance at the matching initial accuracy be used as a benchmark in the analysis?

We agree that headroom might partly contribute, but it does not explain our main findings.

  1. Diagnostics remain predictive at matched accuracy. Appendix B.6 shows that after controlling for starting accuracy, teacher advantage and JSD still predict final OPSD performance in 8 of 9 settings (e.g., Reference-only JSD: β = −22.096, p = 0.035).
  2. LD-SFT outperforms Vanilla SFT at matched accuracy. We compare LD-SFT and Vanilla SFT in five settings where their pre-OPSD accuracies differ by at most 0.83 points. Across three model scales and three benchmarks, LD-SFT reaches 1.25–3.75 points higher final accuracy. Easy-SFT gets a similar gain (+11.25 on AIME24) but lower final accuracy (67.50 vs. 72.92; Table 27). Headroom alone therefore does not explain LD-SFT’s advantage.

Near-matched starting accuracy · across model scales and benchmarks

Model / benchmarkInitializationPre-OPSDBest OPSDGain
Qwen3-4B / AIME24Vanilla SFT (step 60)61.6770.00+8.33
Easy-SFT (step 100)56.2567.50+11.25
LD-SFT (α = 0.25) (step 100)61.2572.92+11.67
Qwen3-4B / AIME25Vanilla SFT (step 100)54.1760.42+6.25
LD-SFT (α = 1) (step 100)54.5863.75+9.17
Qwen3-4B / HMMT25Vanilla SFT (step 100)29.5839.58+10.00
LD-SFT (α = 1) (step 100)30.0043.33+13.33
Qwen3-1.7B / AIME24Vanilla SFT (step 100)35.8342.08+6.25
LD-SFT (step 100)35.0044.58+9.58
Qwen3-8B / HMMT25Vanilla SFT (step 100)33.3341.67+8.34
LD-SFT (step 100)33.3342.92+9.59

Q3: Can LD-SFT be transferred to other model families, non-mathematical tasks, or other self-distillation variants (such as SDFT and SDPO)?

  1. LD-SFT can be transferred to models other than Qwen3-4B. Following OPSD [1] and its subsequent variants like U-OPSD [2] and β-OPSD [3], we use the Qwen3 family because its thinking and non-thinking modes enable the thinking-teacher / non-thinking-student setup used in the official OPSD repository. To evaluate generalization across model scales, we additionally test Qwen3-1.7B and Qwen3-8B. As shown below, LD-SFT initialization consistently outperforms Vanilla SFT initialization in final OPSD accuracy across all three benchmarks at both scales.
ModelMethod / stageAIME24 (Δ)AIME25 (Δ)HMMT25 (Δ)
1.7BVanilla SFT35.8326.2515.83
↳ OPSD42.08 (+6.25)32.50 (+6.25)20.42 (+4.59)
1.7BLD-SFT35.0031.2517.50
↳ OPSD44.58 (+9.58)34.58 (+3.33)22.50 (+5.00)
8BVanilla SFT65.8355.0033.33
↳ OPSD73.75 (+7.92)62.92 (+7.92)41.67 (+8.34)
8BLD-SFT65.4249.5833.33
↳ OPSD75.00 (+9.58)63.33 (+13.75)42.92 (+9.59)

[1] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. ICML 2026.

[2] On-Policy Self-Distillation without Any Supervision. https://arxiv.org/abs/2608.06296

[3] β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation. https://arxiv.org/pdf/2607.28582

  1. LD-SFT can be transferred to coding tasks. To test whether LD-SFT could generalize to other domains, we compare the OPSD training initialized from LD-SFT and Vanilla SFT on coding tasks. Both arms used the same 3,240 OpenCodeReasoning examples for 100 SFT steps, followed by 100 steps of reference-conditioned OPSD. We evaluated the base model, both SFT checkpoints, and OPSD checkpoints at steps 25, 50, 75, and 100 on the same 200 LiveCodeBench AtCoder problems.

LiveCodeBench AtCoder · Qwen3-4B

InitializationBase (Pre-SFT)Pre-OPSDStep 25Step 50Step 75Step 100Best
Vanilla SFT28.1936.1948.3848.3847.7548.3148.38
LD-SFT α = 0.2528.1936.9449.1347.0649.0648.5049.13
  1. LD-SFT can be transferred to other distillation methods like SDFT. To test the generalizability of LD-SFT on other self-distillation methods, we also compare SDFT [4] initialized from LD-SFT and SFT. As shown below, LD-SFT (α=0.25) improves SDFT on AIME25 and HMMT25, reflecting that our conclusion does not change while transferring to other self-distillation methods.
MethodAIME24AIME25HMMT25
Vanilla SFT + SDFT65.8355.0037.50
LD α=0.25 SFT + SDFT65.8358.7539.58
LD α=1.0 SFT + SDFT66.6755.8335.00

[4] Self-Distillation Enables Continual Learning. ICML 26 (Spotlight).


W: The explanation of “ease of use and novelty” constitutes a post hoc attribution.

We clarify that our ease–novelty hypothesis is supported by subsequent experiments beyond the initial correlations. We agree that the ease–novelty explanation was proposed after observing the correlations, and we will present it explicitly as a hypothesis (Section 4.3 already calls it “one possible explanation”).

However, this hypothesis makes testable predictions, and our later experiments check them:

  • Low-divergence weighting should help after a short SFT. LD-SFT, designed from this hypothesis, improves downstream OPSD over Vanilla SFT (Table 1).
  • High-divergence weighting should not help as much. HD-SFT, which reverses the weighting, performs worse than LD-SFT on all three benchmarks (best OPSD: 70.42 vs. 72.92 on AIME24, 60.00 vs. 63.75 on AIME25, 38.75 vs. 43.33 on HMMT25).

While these results do not prove the mechanism, they provide evidence beyond the original correlations.

Reviewer_t9gGOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper considers the problem of initialization for OPSD. The authors discover that the initialization is sensitive to the amount of SFT training. A shorter-trained SFT can perform better in some situations than a longer trained SFT checkpoint. From the viewpoint of divergence, the longer-trained SFT checkpoint is closer to the teacher, but the authors discover that long SFT can destroy the novelty between the student and the teacher and hurt downstream OPSD performance. In contrast, for short SFT, lower divergence between teacher and student indicates better initialization for OPSD. The authors then devise a weighting scheme based on this intuition -- LD-SFT. The method weights tokens based on teacher-student agreement via Jensen-Shannon divergence; LD-SFT specifically upweights low-divergence (high-agreement) tokens. The authors show that this method leads to better initialization via an extensive set of experiments.

Critical strengths

This paper:

  • Extensively studies the effect of short-SFT and long-SFT on OPSD initialization
  • The paper clearly highlights that SFT accuracy is not a good predictor of OPSD improvements and overall SFT+OPSD gains
  • Shows that teacher advantage correlates highly with best OPSD gains (teacher succeeds and student is weak)
  • JSD correlates highly-negatively with best OPSD gains for short SFT
  • These properties are used in a logical way to create a weighted cross-entropy criterion for SFT training
  • Results show this weighted SFT approach is reasonable and produces good results Overall, the paper is clear, proceeds logically from intuition to algorithm, and shows good gains for the approach in the experiments.
Critical weaknesses

There's no major weaknesses in the paper. Here are some minor points:

  • The teacher advantage is defined in the appendix; this makes interpreting Figure 5 a bit cumbersome
  • The authors focus on non-thinking mode in the main paper; thinking mode is delegated to the appendix. Thinking mode doesn't appear to work well with the new method.
  • Benchmarks are limited in domain to Math
  • It is unclear if the method works in the long-train SFT case
Questions for authors

Does the method work in the long-train SFT case?

3 comments

W1: The teacher advantage is defined in the appendix; this makes interpreting Figure 5 a bit cumbersome

Thank you for the feedback. We have moved the definition of teacher advantage into the main text to make Figure 5 easier to understand.


W2: The authors focus on non-thinking mode in the main paper; thinking mode is delegated to the appendix. Thinking mode doesn't appear to work well with the new method.

We would like to clarify that the thinking-mode experiment in Appendix A.5 (Table 2) does not use LD-SFT. It applies standard OPSD [1] to a Vanilla-SFT parent with a thinking-mode student. The drop in performance is therefore a property of OPSD with a thinking student, not of LD-SFT.

As discussed in Appendix A.5, we use a non-thinking student for two reasons:

  • It is the recommended OPSD setup. The original OPSD paper [1] found that a non-thinking student distilled from a thinking teacher outperforms a thinking student, and our Table 2 confirms this. We follow the same setup.
  • It is where initialization matters most. In thinking mode, Qwen3-4B already reaches 71.25 avg@8 on AIME24 without any training, versus 22.92 in non-thinking mode. That leaves little room to measure how different SFT warm-ups affect downstream OPSD, which is the question our paper studies.

For clarity, Section 3.1 of the revised PDF now briefly states why we use a non-thinking student and refers to Appendix A.5 for details. We have also clarified in Appendix A.5 (Table 2) that the thinking-mode experiment uses a Vanilla-SFT parent rather than LD-SFT.

[1] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. ICML 2026.


W3: Benchmarks are limited in domain to Math.

LD-SFT’s benefit generalizes to coding tasks. To assess generalization, we compare OPSD initialized from LD-SFT versus Vanilla SFT on coding tasks. Both use 3,240 OpenCodeReasoning examples for 100 SFT steps, followed by 100 reference-conditioned OPSD steps. Evaluated on 200 LiveCodeBench AtCoder problems, LD-SFT initialization achieves a higher best score and outperforms Vanilla SFT at three of four OPSD checkpoints.

LiveCodeBench AtCoder · Qwen3-4B

InitializationBase (Pre-SFT)Pre-OPSDStep 25Step 50Step 75Step 100Best
Vanilla SFT28.1936.1948.3848.3847.7548.3148.38
LD-SFT α = 0.2528.1936.9449.1347.0649.0648.5049.13

W4 & Q1: It is unclear if the method works in the long-train SFT case.

LD-SFT remains effective with longer SFT training. After 500 SFT steps, LD-SFT improves checkpoint-averaged OPSD on pooled AIME24+AIME25 by +1.15 points over Vanilla SFT (62.60% vs. 61.46%), compared with +2.19 points after 100 steps.

Short- vs. long-train SFT (100 vs. 500 steps) · Pooled AIME24+AIME25 avg@8 (%)

SFT warm-upMethodMean OPSDBest OPSD
100 stepsVanilla63.1864.80
100 stepsLD-SFT (α=0.25)65.37 (+2.19)67.50 (+2.70)
500 stepsVanilla61.4662.30
500 stepsLD-SFT62.60 (+1.15)63.54 (+1.25)

Although LD-SFT achieves lower JSD during the first 100 steps, its JSD slightly exceeds Vanilla SFT at steps 200–500 (long-train SFT). This aligns with our Figure 7, where the negative correlation between JSD and OPSD gain weakens or reverses in long-train SFT. A possible explanation is that LD-SFT prioritizes low-divergence tokens, accelerating early learning. However, as both methods eventually learn these tokens, LD-SFT's continued underweighting of high-divergence tokens may hinder further JSD reduction.

Full-vocabulary on-policy JSD across SFT checkpoints · Qwen3-4B

SFT stepVanilla mean JSDLD-SFT α = 0.25 mean JSD
200.0151310.015109 ↓
400.0155690.015356 ↓
600.0201010.019979 ↓
800.0197670.019632 ↓
1000.0194960.019417 ↓
2000.0160200.016176 ↑
3000.0156110.015745 ↑
4000.0153240.015443 ↑
5000.0153970.015399 ↑

Raised my score to 4 based on the authors comments

Great thanks for your positive feedback and for taking the time to reconsider our work! We sincerely appreciate your thoughtful comments and constructive suggestions throughout the review process.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Strengths and Revisions

Dear Reviewers and Area Chair,

We thank the reviewers for their thoughtful feedback and the AC for overseeing the review process.

Recognized strengths. We appreciate that the reviewers recognized the practical importance and surprising findings of studying OPSD initialization (2tKc, 6cCy), the systematic analysis across training stages, model scales, and teacher configurations (t9gG, 2tKc, 6cCy), and the simple, well-motivated LD-SFT method with promising empirical gains (t9gG, 2tKc, 6cCy).

To address the remaining concerns, we have posted detailed responses and substantially revised the manuscript:

  • Generalizability: LD-SFT generalizes across model scales, coding tasks, SDFT, and longer SFT training. (t9gG W3–W4; 2tKc Q3; 6cCy W2-c; Appendix C.5, Tables 33–36)
  • Robustness: LD-SFT’s gains hold across multiple seeds, fixed OPSD checkpoints, and stronger baselines. (2tKc Q1; 6cCy W2-a/W2-d; Appendix C.2–C.3, Tables 28–30)
  • Fair comparisons: LD-SFT maintains its advantage under matched initial accuracy and validation-based checkpoint selection. (2tKc Q2; 6cCy W2-a; Appendix B.6, Tables 23–27)
  • Mechanistic evidence and efficiency: Further analyses support divergence-based initialization and demonstrate minimal computational overhead. (2tKc W; 6cCy W3–W5; Appendices B.6, C.4–C.6, Tables 25–26, 31–32, 37–40)

We have also clarified points that may have been unclear in the previous version.

As the discussion period draws to a close, we would be very grateful if you could review our responses and let us know whether they adequately address your concerns or if further clarification would be helpful.

Thank you again for your valuable time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.