- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes S³-OPSD, a framework for improving the stability and efficiency of on-policy self-distillation. It combines sparse supervision at selected rollout positions, SGD optimization, and teacher-target smoothing. The position-selection criterion is motivated by a theoretical connection between distillation and reward-guided policy updates. Experiments cover multiple model sizes from two model families, mathematical reasoning, and multitask learning, with component ablations and resource-efficiency comparisons. The main benefit is better performance retention during extended training, together with lower memory requirements and faster training under a fixed memory budget.
Critical strengths
-
The core idea is clear and practically meaningful. Selecting a small number of informative positions while controlling parameter updates and teacher drift offers a sensible approach to stabilizing self-distillation. The theoretical analysis provides a motivation for the selection criterion, and the ablations support the usefulness of the proposed components.
-
The experimental evaluation is broad and supports the stability claim. The paper evaluates multiple model sizes and two model families, includes mathematical reasoning and multitask experiments, and examines component contributions and resource costs. The sustained performance during extended training is a meaningful advantage over baselines that deteriorate. The measured memory savings also support the practical efficiency of the approach.
Critical weaknesses
-
The accuracy advantage is limited and depends on how checkpoints are compared. Figure 1 shows relatively small differences in peak accuracy. Comparisons at later training steps highlight the deterioration of the baselines, but do not necessarily establish an advantage over their best checkpoints or early stopping. I find the evidence for improved stability more convincing than the evidence for substantially higher accuracy. The “faster” claim would also be clearer with a comparison of wall-clock time to reach a common accuracy threshold.
-
Several closely related OPSD methods are missing from the comparisons. These include SR-OPSD, which constructs a geometric mixture of the self-teacher and a frozen reference policy; GeoSD, which controls distillation-induced drift through an overlap-aware loss and predictive-distance regularization; and DASH, which adaptively weights token-level supervision using the teacher–student divergence sequence. Notably, DASH reports higher average accuracy on the same three mathematical benchmarks for Qwen3-1.7B (45.07 versus 40.09) and Qwen3-8B (66.40 versus 65.83), while performing slightly worse at 4B. Although differences in training and evaluation prevent a direct ranking, these results raise questions about the accuracy advantage of S³-OPSD, particularly at 1.7B. Comparisons under matched training budgets, evaluation settings, and checkpoint selection would help establish its practical advantage. Clearer discussion of the methodological differences would also strengthen the novelty claim.
Questions for authors
-
How would you characterize the practical advantage over SDFT and OPSD when each method uses its best checkpoint or early stopping? Would it be more accurate to frame the main contribution as improved stability and resource efficiency, with modest gains in peak accuracy?
-
In Figure 1, how much faster does S³-OPSD reach a common accuracy threshold, such as 54%? This would help clarify how the per-step speedup translates into the time required to obtain a useful model.
-
The theoretical analysis assumes that the teacher behaves like a reward-guided Doob-h transform of the student. Could you explain how well this assumption describes the actual teacher conditioned on a reference solution? In particular, when might the proposed score identify differences in wording or formatting rather than decisions that affect reasoning correctness?
-
Prior work has explored selective token supervision (TIP), sparse parameter updates with SGD (Do We Need Adam?), and geometric target reformulation for stable distillation (Veto). Could you clarify the main methodological novelty beyond combining these ideas in OPSD? In particular, how do the proposed importance score and fixed-reference teacher smoothing differ from these approaches?
Overall, I find the paper clearly presented and supported by a thorough experimental evaluation. I am less convinced that it substantially improves peak accuracy over strong self-distillation baselines, particularly in Figure 1. Nevertheless, the improved stability during extended OPSD training, the resource-efficiency benefits, and the theoretical motivation make this a useful contribution. I therefore lean toward weak acceptance. My confidence is relatively low, mainly because I am less familiar with the theoretical framework and cannot fully assess its assumptions and derivations.
5 comments
Rebuttal for Reviewer faKs [2/3]
[W2] Missing comparisons with closely related OPSD methods. Several closely related OPSD methods are missing from the comparisons, including SR-OPSD, GeoSD, and DASH. Notably, DASH reports higher average accuracy on the same three mathematical benchmarks for Qwen3-1.7B (45.07 vs. 40.09) and Qwen3-8B (66.40 vs. 65.83), which raises questions about the accuracy advantage of S³-OPSD, particularly at 1.7B. Comparisons under matched training budgets, evaluation settings, and checkpoint selection would help establish its practical advantage, and a clearer discussion of the methodological differences would strengthen the novelty claim.
We thank the reviewer for pointing us to these methods. SR-OPSD (Aug., 2026), DASH (Aug., 2026), and GeoSD (Jul., 2026) appeared within about two months of the submission deadline, so they are contemporaneous under the ICLR reviewer guidelines. Nevertheless, we agree that a comparison and a clearer discussion strengthen the paper and since DASH provides public code, we ran experiments using its official implementation and compared them with -OPSD. We trained DASH with its official implementation and hyperparameters. We used the same budget of Table 1 for Qwen3-1.7B.
From Table R2, DASH achieves a strong initial average of 40.00 at step 100, better than OPSD’s 38.70, but its performance declines to 28.70 by step 500. -OPSD, by contrast, remains nearly stable. At step 500, it outperforms DASH on all three benchmarks, highlighting its stronger stability. Moreover, DASH reuses the dense full-vocabulary loss of OPSD, so it inherits OPSD's cost. Measured on the same GPU under the same 80 GB budget, DASH takes 20.25s and 69.6GB while -OPSD takes 7.35s and 47.7GB, which is ~2.8× faster with lower memory use than DASH. We added these results in the revision as Appendix Sec. D.7 and Table A14. We also discuss SR-OPSD, GeoSD, and DASH in Sec. 2.2 of the revision (L123–127).
Table R2. Comparison with DASH under the training and evaluation protocol of Table 1 (Qwen3-1.7B; Avg@12 at training steps 100 / 300 / 500).
| Method | AIME24 (100 / 300 / 500) | AIME25 (100 / 300 / 500) | HMMT25 (100 / 300 / 500) | Average (100 / 300 / 500) |
|---|---|---|---|---|
| Base(Instruct) | 46.94 | 34.17 | 23.33 | 34.81 |
| +OPSD | 54.44/44.72/39.72 | 36.39/32.22/28.06 | 25.28/23.33/20.28 | 38.70/33.42/29.35 |
| +DASH | 53.61/45.00/39.17 | 38.06/31.94/28.06 | 28.33/21.67/18.89 | 40.00/32.87/28.70 |
| +-OPSD(ours) | 54.72/55.00/54.17 | 38.89/38.61/39.17 | 26.67/26.39/25.56 | 40.09/40.00/39.63 |
[Q2] In Figure 1, how much faster does S³-OPSD reach a common accuracy threshold, such as 54%? This would help clarify how the per-step speedup translates into the time required to obtain a useful model.
Table R3 reports the first evaluated checkpoint to reach at least 54% AIME24 accuracy and the corresponding cumulative training time for the runs in Fig. 1. Both -OPSD and OPSD reach this threshold at step 50, but -OPSD requires only 6.1 min, compared with 17.0 min for OPSD, owing to its 2.8$\times$ per-step speedup (7.35 s vs. 20.39 s). SDFT reaches the threshold at step 75 after 24.4 min. Thus, the per-step speedup translates directly into less training time to reach the same target accuracy.
Table R3. Step and time to reach 54% AIME24 Avg@8 for the Qwen3-1.7B runs of Fig. 1. Checkpoints are evaluated every 25 steps.
| Method | Timeperstep(s) | Stepto54% | Timeto54%(min) |
|---|---|---|---|
| SDFT | 19.48 | 75 | 24.4 |
| OPSD | 20.39 | 50 | 17.0 |
| -OPSD | 7.35 | 50 | 6.1 |
Rebuttal for Reviewer faKs [3/3]
[Q3] The theoretical analysis assumes that the teacher behaves like a reward-guided Doob-h transform of the student. Could you explain how well this assumption describes the actual teacher conditioned on a reference solution? In particular, when might the proposed score identify differences in wording or formatting rather than decisions that affect reasoning correctness?
(i) Does track importance?
Following the reviewer's suggestion, we estimated a top-8 approximation to using nested Monte-Carlo rollouts on Qwen3-1.7B and included the analysis as Sec. 4.2 and Fig. 6 in the revision. Specifically, for each of 128 problems from Math-500 [R1], we sampled one student response and uniformly selected 32 positions from its first 1,024 tokens (4,096 states in total). At each state, we selected the most likely next tokens, sampled continuations per candidate, and graded their final answers, using the success rate as the value estimate for each candidate. We then computed with the candidate probabilities renormalized over the eight candidates. Note that is a top-8 approximation of the full-vocabulary rather than itself; the eight candidates cover a median of 0.99999 of the next-token probability mass, so the omitted mass is small at almost all states. We compared with the proposed importance score after discarding states with or as unreliable (2,208 states). The results show that increases monotonically with , with a Spearman correlation of . Because the 32 states of a problem come from the same response and are not independent, we conducted a cluster bootstrap that resamples problems (10,000 resamples) and it resulted in a 95% CI of for with exact sign test .
(ii) Can the score pick up wording or formatting differences?
In Sec. 4.2 (Fig. 5), tokens selected by are enriched for reasoning transitions and step openings, while numerical and formatting tokens are underrepresented. In addition, in the Monte-Carlo test above is computed from final-answer correctness only, so wording variants that do not change correctness contribute little to ; the strong correlation (). These results support an association between and correctness-related importance in the evaluated setting. However, can also respond to stylistic differences, and we do not claim that every high-scoring state corresponds to a correctness-critical decision.
Reference
[R1] Lightman, Hunter, et al. Let's verify step by step. ICLR, 2024.
[Q4] Methodological novelty over prior work. Prior work has explored selective token supervision (TIP), sparse parameter updates with SGD (Do We Need Adam?), and geometric target reformulation for stable distillation (Veto). Could the authors clarify the main methodological novelty beyond combining these ideas in OPSD, in particular how the proposed importance score and fixed-reference teacher smoothing differ from these approaches?
We appreciate the reviewer pointing us to these closely related works; the comparison helps us articulate our contributions more clearly. Below, we clarify how -OPSD differs from each of them.
TIP. TIP selects tokens by KL divergence and entropy. Its theoretical justification relies on the strong assumption of negligible gradient correlation across tokens, which is hard to justify for language models. In contrast, we derive token importance from the widely adopted reward-tilting interpretation (Sec. 3.1), which yields a principled importance measure with explicit bounds relating to the advantage variance (Eq. 10).
Do We Need Adam?. This work shows that SGD can be a strong optimizer choice in RL. We go further in three ways. (i) Theoretically, we show that the OPSD objective is locally a policy-gradient objective (Eq. 6). (ii) These findings are consistent with this perspective by showing that the work's finding transfers directly to OPSD and yields strong performance. (iii) Most importantly, we show that SGD improves OPSD stability and analyze this benefit through weight sparsity (Sec. 3.2).
Veto. Veto forms a geometric target with the online student distribution, whereas we form it with the fixed initial reference distribution (Eq. 11). This anchors the target and mitigates excessive distributional changes during training. We added the citation of Veto and this difference to the revision in Sec. 3.3 (L292–293).
Rebuttal for Reviewer faKs [1/3]
We thank the reviewer for carefully reading our paper and for constructive feedback. We are glad that the reviewer found the core idea clear and practically meaningful, and that the theoretical motivation and ablations support the proposed components. We appreciate the recognition that our method sustains performance during extended training while baselines deteriorate, as stability is a central goal of our work. We appreciate the opportunity to discuss these points further and respond to each of them below.
We agree that peak accuracies in Fig. 1 are close. However, the best checkpoint is only reachable in practice if evaluation is frequent, and stability is what makes that frequency unnecessary.
For the wall-clock time to reach a common accuracy threshold, please see our answer to [Q2].