acceptodds
Under review as a conference paper at ICLR 2027

-OPSD: Sparse, Stable and Speedy On-Policy Self-Distillation

Abstract

Improving LLM reasoning often requires costly demonstrations, rollouts, or strong external teachers. On-policy self-distillation (OPSD) uses the same model conditioned on privileged context as its teacher, but dense supervision is costly and OPSD sometimes exhibit unstable training. We propose -OPSD, a sparse, stable, and speedy framework combining state selection, stable optimization, and target smoothing. Viewing the teacher as an implicit reward-guided Doob- transform, we define state importance as action-wise advantage variance and derive a low-cost proxy bounded above and below by this variance. We show that SGD enables sparse weight updates, reducing parameter displacement and improving stability over momentum-based optimizers. At selected states, we smooth targets by mixing the teacher distribution with the initial student distribution as a fixed reference policy. Across model families and sizes, mathematical reasoning benchmarks, and multi-task learning, -OPSD improves performance at equal training steps and remains stable over extended training. It also enables memory-efficient training and, at matched effective batch size, reduces training time by up to 13 relative to GRPO and 3 relative to OPSD baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Self-Distillation, principled sparse supervision, bounded distribution matching

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_faKsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes S³-OPSD, a framework for improving the stability and efficiency of on-policy self-distillation. It combines sparse supervision at selected rollout positions, SGD optimization, and teacher-target smoothing. The position-selection criterion is motivated by a theoretical connection between distillation and reward-guided policy updates. Experiments cover multiple model sizes from two model families, mathematical reasoning, and multitask learning, with component ablations and resource-efficiency comparisons. The main benefit is better performance retention during extended training, together with lower memory requirements and faster training under a fixed memory budget.

Critical strengths
  1. The core idea is clear and practically meaningful. Selecting a small number of informative positions while controlling parameter updates and teacher drift offers a sensible approach to stabilizing self-distillation. The theoretical analysis provides a motivation for the selection criterion, and the ablations support the usefulness of the proposed components.

  2. The experimental evaluation is broad and supports the stability claim. The paper evaluates multiple model sizes and two model families, includes mathematical reasoning and multitask experiments, and examines component contributions and resource costs. The sustained performance during extended training is a meaningful advantage over baselines that deteriorate. The measured memory savings also support the practical efficiency of the approach.

Critical weaknesses
  1. The accuracy advantage is limited and depends on how checkpoints are compared. Figure 1 shows relatively small differences in peak accuracy. Comparisons at later training steps highlight the deterioration of the baselines, but do not necessarily establish an advantage over their best checkpoints or early stopping. I find the evidence for improved stability more convincing than the evidence for substantially higher accuracy. The “faster” claim would also be clearer with a comparison of wall-clock time to reach a common accuracy threshold.

  2. Several closely related OPSD methods are missing from the comparisons. These include SR-OPSD, which constructs a geometric mixture of the self-teacher and a frozen reference policy; GeoSD, which controls distillation-induced drift through an overlap-aware loss and predictive-distance regularization; and DASH, which adaptively weights token-level supervision using the teacher–student divergence sequence. Notably, DASH reports higher average accuracy on the same three mathematical benchmarks for Qwen3-1.7B (45.07 versus 40.09) and Qwen3-8B (66.40 versus 65.83), while performing slightly worse at 4B. Although differences in training and evaluation prevent a direct ranking, these results raise questions about the accuracy advantage of S³-OPSD, particularly at 1.7B. Comparisons under matched training budgets, evaluation settings, and checkpoint selection would help establish its practical advantage. Clearer discussion of the methodological differences would also strengthen the novelty claim.

Questions for authors
  1. How would you characterize the practical advantage over SDFT and OPSD when each method uses its best checkpoint or early stopping? Would it be more accurate to frame the main contribution as improved stability and resource efficiency, with modest gains in peak accuracy?

  2. In Figure 1, how much faster does S³-OPSD reach a common accuracy threshold, such as 54%? This would help clarify how the per-step speedup translates into the time required to obtain a useful model.

  3. The theoretical analysis assumes that the teacher behaves like a reward-guided Doob-h transform of the student. Could you explain how well this assumption describes the actual teacher conditioned on a reference solution? In particular, when might the proposed score identify differences in wording or formatting rather than decisions that affect reasoning correctness?

  4. Prior work has explored selective token supervision (TIP), sparse parameter updates with SGD (Do We Need Adam?), and geometric target reformulation for stable distillation (Veto). Could you clarify the main methodological novelty beyond combining these ideas in OPSD? In particular, how do the proposed importance score and fixed-reference teacher smoothing differ from these approaches?

Overall, I find the paper clearly presented and supported by a thorough experimental evaluation. I am less convinced that it substantially improves peak accuracy over strong self-distillation baselines, particularly in Figure 1. Nevertheless, the improved stability during extended OPSD training, the resource-efficiency benefits, and the theoretical motivation make this a useful contribution. I therefore lean toward weak acceptance. My confidence is relatively low, mainly because I am less familiar with the theoretical framework and cannot fully assess its assumptions and derivations.

5 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer faKs [1/3]

We thank the reviewer for carefully reading our paper and for constructive feedback. We are glad that the reviewer found the core idea clear and practically meaningful, and that the theoretical motivation and ablations support the proposed components. We appreciate the recognition that our method sustains performance during extended training while baselines deteriorate, as stability is a central goal of our work. We appreciate the opportunity to discuss these points further and respond to each of them below.

[W1, Q1] Practical advantage under best-checkpoint comparison. Figure 1 shows relatively small differences in peak accuracy, and comparisons at later training steps highlight the deterioration of the baselines but do not necessarily establish an advantage over their best checkpoints or early stopping. How would the authors characterize the practical advantage over SDFT and OPSD when each method uses its best checkpoint or early stopping? Would it be more accurate to frame the main contribution as improved stability and resource efficiency, with modest gains in peak accuracy? The "faster" claim would also be clearer with a comparison of wall-clock time to reach a common accuracy threshold.

We agree that peak accuracies in Fig. 1 are close. However, the best checkpoint is only reachable in practice if evaluation is frequent, and stability is what makes that frequency unnecessary.

  • Best-checkpoint selection is expensive in practice. Fig. 1 evaluates every 25 steps, and we know in hindsight where to stop. In our math setting, evaluation costs as much as training or more, so the evaluation interval itself becomes a hyperparameter. SDFT and OPSD deteriorate after their peak, so a coarse interval can miss the peak entirely. S3S^3-OPSD stays near its peak, so sparse evaluation still yields a near-best checkpoint. In this sense, stability directly improves practicality.
  • The gap widens as training gets longer. In multi-task learning (Fig. 4), more data means more steps. Because S3S^3-OPSD remains stable over extended training, it keeps improving. It reaches the highest seen accuracy and is the only method that improves over the base model on all seven seen tasks. It also trains 1.3–3.7× faster than the other on-policy methods and surpasses the best seen accuracy of every on-policy baseline within 0.3 h (Fig. 4, middle).

For the wall-clock time to reach a common accuracy threshold, please see our answer to [Q2].

AuthorsOpenReview ↗

Rebuttal for Reviewer faKs [2/3]

[W2] Missing comparisons with closely related OPSD methods. Several closely related OPSD methods are missing from the comparisons, including SR-OPSD, GeoSD, and DASH. Notably, DASH reports higher average accuracy on the same three mathematical benchmarks for Qwen3-1.7B (45.07 vs. 40.09) and Qwen3-8B (66.40 vs. 65.83), which raises questions about the accuracy advantage of S³-OPSD, particularly at 1.7B. Comparisons under matched training budgets, evaluation settings, and checkpoint selection would help establish its practical advantage, and a clearer discussion of the methodological differences would strengthen the novelty claim.

We thank the reviewer for pointing us to these methods. SR-OPSD (Aug., 2026), DASH (Aug., 2026), and GeoSD (Jul., 2026) appeared within about two months of the submission deadline, so they are contemporaneous under the ICLR reviewer guidelines. Nevertheless, we agree that a comparison and a clearer discussion strengthen the paper and since DASH provides public code, we ran experiments using its official implementation and compared them with S3S^3-OPSD. We trained DASH with its official implementation and hyperparameters. We used the same budget of Table 1 for Qwen3-1.7B.

From Table R2, DASH achieves a strong initial average of 40.00 at step 100, better than OPSD’s 38.70, but its performance declines to 28.70 by step 500. S3S^3-OPSD, by contrast, remains nearly stable. At step 500, it outperforms DASH on all three benchmarks, highlighting its stronger stability. Moreover, DASH reuses the dense full-vocabulary loss of OPSD, so it inherits OPSD's cost. Measured on the same GPU under the same 80 GB budget, DASH takes 20.25s and 69.6GB while S3S^3-OPSD takes 7.35s and 47.7GB, which is ~2.8× faster with lower memory use than DASH. We added these results in the revision as Appendix Sec. D.7 and Table A14. We also discuss SR-OPSD, GeoSD, and DASH in Sec. 2.2 of the revision (L123–127).

Table R2. Comparison with DASH under the training and evaluation protocol of Table 1 (Qwen3-1.7B; Avg@12 at training steps 100 / 300 / 500).

MethodAIME24 (100 / 300 / 500)AIME25 (100 / 300 / 500)HMMT25 (100 / 300 / 500)Average (100 / 300 / 500)
Base(Instruct)46.9434.1723.3334.81
+OPSD54.44/44.72/39.7236.39/32.22/28.0625.28/23.33/20.2838.70/33.42/29.35
+DASH53.61/45.00/39.1738.06/31.94/28.0628.33/21.67/18.8940.00/32.87/28.70
+S3S^3-OPSD(ours)54.72/55.00/54.1738.89/38.61/39.1726.67/26.39/25.5640.09/40.00/39.63

[Q2] In Figure 1, how much faster does S³-OPSD reach a common accuracy threshold, such as 54%? This would help clarify how the per-step speedup translates into the time required to obtain a useful model.

Table R3 reports the first evaluated checkpoint to reach at least 54% AIME24 accuracy and the corresponding cumulative training time for the runs in Fig. 1. Both S3S^3-OPSD and OPSD reach this threshold at step 50, but S3S^3-OPSD requires only 6.1 min, compared with 17.0 min for OPSD, owing to its 2.8$\times$ per-step speedup (7.35 s vs. 20.39 s). SDFT reaches the threshold at step 75 after 24.4 min. Thus, the per-step speedup translates directly into less training time to reach the same target accuracy.

Table R3. Step and time to reach 54% AIME24 Avg@8 for the Qwen3-1.7B runs of Fig. 1. Checkpoints are evaluated every 25 steps.

MethodTimeperstep(s)Stepto54%Timeto54%(min)
SDFT19.487524.4
OPSD20.395017.0
S3S^3-OPSD7.35506.1
AuthorsOpenReview ↗

Rebuttal for Reviewer faKs [3/3]

[Q3] The theoretical analysis assumes that the teacher behaves like a reward-guided Doob-h transform of the student. Could you explain how well this assumption describes the actual teacher conditioned on a reference solution? In particular, when might the proposed score identify differences in wording or formatting rather than decisions that affect reasoning correctness?

(i) Does StS_t track importance?

Following the reviewer's suggestion, we estimated a top-8 approximation C^t\hat C_t to CtC_t using nested Monte-Carlo rollouts on Qwen3-1.7B and included the analysis as Sec. 4.2 and Fig. 6 in the revision. Specifically, for each of 128 problems from Math-500 [R1], we sampled one student response and uniformly selected 32 positions from its first 1,024 tokens (4,096 states in total). At each state, we selected the K=8K{=}8 most likely next tokens, sampled N=32N{=}32 continuations per candidate, and graded their final answers, using the success rate as the value estimate V^j\hat V_j for each candidate. We then computed C^t=∑jw~j(V^j−Vˉ)2\hat C_t=\sum_j \tilde w_j(\hat V_j-\bar V)^2 with w~j\tilde w_j the candidate probabilities renormalized over the eight candidates. Note that C^t\hat C_t is a top-8 approximation of the full-vocabulary CtC_t rather than CtC_t itself; the eight candidates cover a median of 0.99999 of the next-token probability mass, so the omitted mass is small at almost all states. We compared C^t\hat C_t with the proposed importance score StS_t after discarding states with C^t<10−16\hat C_t<10^{-16} or St<10−7S_t<10^{-7} as unreliable (2,208 states). The results show that CtC_t increases monotonically with StS_t, with a Spearman correlation of ρ=0.813\rho=0.813. Because the 32 states of a problem come from the same response and are not independent, we conducted a cluster bootstrap that resamples problems (10,000 resamples) and it resulted in a 95% CI of [0.79,0.83][0.79, 0.83] for ρ\rho with exact sign test p<10−36p<10^{-36}.

(ii) Can the score pick up wording or formatting differences?

In Sec. 4.2 (Fig. 5), tokens selected by StS_t are enriched for reasoning transitions and step openings, while numerical and formatting tokens are underrepresented. In addition, CtC_t in the Monte-Carlo test above is computed from final-answer correctness only, so wording variants that do not change correctness contribute little to CtC_t; the strong correlation (ρ=0.813\rho=0.813). These results support an association between StS_t and correctness-related importance in the evaluated setting. However, StS_t can also respond to stylistic differences, and we do not claim that every high-scoring state corresponds to a correctness-critical decision.

Reference

[R1] Lightman, Hunter, et al. Let's verify step by step. ICLR, 2024.

[Q4] Methodological novelty over prior work. Prior work has explored selective token supervision (TIP), sparse parameter updates with SGD (Do We Need Adam?), and geometric target reformulation for stable distillation (Veto). Could the authors clarify the main methodological novelty beyond combining these ideas in OPSD, in particular how the proposed importance score and fixed-reference teacher smoothing differ from these approaches?

We appreciate the reviewer pointing us to these closely related works; the comparison helps us articulate our contributions more clearly. Below, we clarify how S3S^3-OPSD differs from each of them.

TIP. TIP selects tokens by KL divergence and entropy. Its theoretical justification relies on the strong assumption of negligible gradient correlation across tokens, which is hard to justify for language models. In contrast, we derive token importance from the widely adopted reward-tilting interpretation (Sec. 3.1), which yields a principled importance measure with explicit bounds relating StS_t to the advantage variance CtC_t (Eq. 10).

Do We Need Adam?. This work shows that SGD can be a strong optimizer choice in RL. We go further in three ways. (i) Theoretically, we show that the OPSD objective is locally a policy-gradient objective (Eq. 6). (ii) These findings are consistent with this perspective by showing that the work's finding transfers directly to OPSD and yields strong performance. (iii) Most importantly, we show that SGD improves OPSD stability and analyze this benefit through weight sparsity (Sec. 3.2).

Veto. Veto forms a geometric target with the online student distribution, whereas we form it with the fixed initial reference distribution (Eq. 11). This anchors the target and mitigates excessive distributional changes during training. We added the citation of Veto and this difference to the revision in Sec. 3.3 (L292–293).

Reviewer_xU5mOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes S^3-OPSD a sparse and stable way of post-training models with On-Policy Self Distillation. The paper presents good insights about how sparse updates, by viewing OPSD teacher as the Doob transform of the student can make training better. The paper is well written and is an interesting read but has clear limitations that the authors should address.

Critical strengths
  1. The paper presents a principled method to approach On-Policy Self Distillation, it is easy to understand and intuitive.
  2. The results mostly are good.
  3. The paper presentation and writing is clear and well presented.
  4. The authors discuss the effect of optimizers in the case of self-distillation which is relatively novel.
Critical weaknesses
  1. To the best of my knowledge, I feel the method has many moving parts that were plugged together to make work. This is not a knock against the paper, but such papers benefit from a clearly articulated algorithm so that it is clear from the first impression what problem have the authors tried to solve. Currently, the paper lacks a formal algorithm which would make the reader understand the work better, along with any ablations currently missing.

  2. The authors in Figure 3, show that OPSD can be made much more stable with the SGD optimizer, but in the main table does not seem to compare against baselines with the SGD optimizer. Moreover, this is a contribution of the paper showing that making sparser updates recovers the performance of other OPSD methods as training proceeds. So why not compare their method with this novel insight as well, and since their method works quite well regardless of optimizer, that is already a strong point of theirs.

  3. OPSD, has been shown to not work for coding tasks and more recently, multi-turn conversations where the mismatch between teacher and student becomes too large [1]. I feel with the flux of OPSD papers, the community should make an honest effort addressing the applicability of this algorithm on a broader and wider scale. The paper should atleast have a discussion about this limitation, if not address the problems of OPSD.

  4. There are some typos in the Appendix please fix.

[1] Denser ≠ Better: Limits of On-Policy Self-Distillation for Continual Post-Training, arxiv 2026

Questions for authors
  1. Can the authors ablate if doing sparser OPSD for coding, atleast makes the model training more stable for coding tasks, where we know OPSD suffers? If it does not help pleae include a paragraph about the limitations about the applicability of OPSD as an algorithm.
  2. Check other weaknesses above.
5 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer xU5m [1/3]

We sincerely thank the reviewer for taking the time to review our paper and for the thoughtful and constructive comments. We are glad that the reviewer found our approach to on-policy self-distillation principled and intuitive, and the paper clearly written. We especially appreciate that the reviewer highlighted our analysis of the optimizer’s role in self-distillation as a relatively novel contribution. We have given careful thought to each of the reviewer’s comments and respond to them in detail below.

[W1] Lack of a formal algorithm. The method has many moving parts that were plugged together, and such papers benefit from a clearly articulated algorithm so that it is clear from the first impression what problem the authors have tried to solve. The paper currently lacks a formal algorithm, which would help the reader understand the work better, along with any ablations currently missing.

Formal algorithm. We thank the reviewer for the constructive suggestion. We agree and have added Algorithm 1 to Sec. 3.4 of the revision.

Ablations. Each component is already ablated individually in Table 2: token selection (ID 1–3), blockwise selection (ID 4–7), optimizer (ID 8), and teacher smoothing (ID 9–13). To follow the reviewer’s suggestions, we further add:

  • SGD applied to OPSD and SDFT (Appendix Sec. D.6)
  • Concurrent baseline comparison (Appendix Sec. D.7)
  • Memory and time efficiency breakdown (Appendix Sec. D.8)
  • Applicability to coding task (Appendix Sec. D.9)
  • Learning rate study (Appendix Sec. D.10)
  • Validation of StS_t via Monte-Carlo estimation (Main Sec. 4.2)

We are happy to discuss any further questions.

AuthorsOpenReview ↗

Rebuttal for Reviewer xU5m [2/3]

[W2] Missing comparison with SGD-based baselines. Figure 3 shows that OPSD can be made much more stable with the SGD optimizer, but the main table does not compare against baselines with the SGD optimizer. Since showing that sparser updates recover the performance of other OPSD methods is itself a contribution of the paper, why not compare the method against baselines using this insight as well? The method working well regardless of optimizer would already be a strong point.

We train OPSD and SDFT with SGD under exactly the setting of Table 1 with Qwen3-1.7B at steps 100, 300, and 500 by only replacing AdamW (5×10−65\times10^{-6}) with SGD without momentum at 10−210^{-2}. From Table R1, SGD substantially mitigates late-stage degradation in both baselines. At step 500, SDFT improves from 0.00 with AdamW to 26.48 with SGD, while OPSD improves from 29.35 to 36.95. However, SDFT and OPSD with SGD still drop 12.41 and 2.12 points respectively from step 100 to 500, whereas S3^3-OPSD nearly maintains its performance (40.09 to 39.63). This suggests that SGD alone does not prevent degradation of SDFT and OPSD; sparse state selection and fixed-reference smoothing provide additional stability. We added these results in the revision Appendix Sec. D.6 and Table A13.

Table R1. Comparison of SDFT, OPSD, and S3S^3-OPSD with AdamW and SGD on Qwen3-1.7B (Avg@12 at training steps 100 / 300 / 500).

MethodOptimizerAIME24 (100 / 300 / 500)AIME25 (100 / 300 / 500)HMMT25 (100 / 300 / 500)Average (100 / 300 / 500)
Base (Instruct)–46.9434.1723.3334.81
+ SDFTAdamW54.44 / 28.06 / 0.0038.89 / 20.56 / 0.0026.94 / 15.00 / 0.0040.09 / 21.21 / 0.00
+ SDFTSGD52.50 / 50.83 / 35.2838.33 / 36.11 / 26.9425.83 / 23.89 / 17.2238.89 / 36.94 / 26.48
+ OPSDAdamW54.44 / 44.72 / 39.7236.39 / 32.22 / 28.0625.28 / 23.33 / 20.2838.70 / 33.42 / 29.35
+ OPSDSGD53.06 / 53.61 / 50.2838.33 / 38.06 / 36.6725.83 / 24.72 / 23.8939.07 / 38.80 / 36.95
+ S3S^3-OPSDAdamW55.00 / 51.39 / 49.4438.61 / 38.33 / 36.1127.22 / 24.17 / 21.6740.28 / 37.96 / 35.74
+ S3S^3-OPSDSGD (ours)54.72 / 55.00 / 54.1738.89 / 38.61 / 39.1726.67 / 26.39 / 25.5640.09 / 40.00 / 39.63
AuthorsOpenReview ↗

Rebuttal for Reviewer xU5m [3/3]

[W3, Q1] Applicability of OPSD to coding tasks. OPSD has been shown not to work well for coding tasks, where the mismatch between teacher and student becomes too large [1]. Can the authors ablate whether sparser OPSD at least makes model training more stable for coding tasks? If it does not help, the paper should at least include a discussion of this limitation on the applicability of OPSD.

Sparse OPSD makes coding training more stable in our experiments. The paper's multi-task setting already includes coding, with SQL query generation. To isolate coding as a standalone task, we additionally train and evaluate on LiveCodeBench [R1].

Existing results (multi-task). Table R2 below is excerpted from Table A10 (Qwen3-1.7B, step 500). On the trained coding task (SQL query generation), dense OPSD falls below the base model, while S3S^3-OPSD improves over it and achieves the best score among all methods.

Table R2. Excerpt from Table A10; Δ\Delta is the change from the base model. Learning curves are shown in Fig. A1.

MethodSQL (seen)Δ\Delta vs. base
Base (Instruct)33.33
+ OPSD29.33−4.00
+ SDFT22.67−10.67
+ S3S^3-OPSD36.00+2.67

New results (single-task, LiveCodeBench). We additionally evaluate coding on Qwen3-1.7B over 200 training steps, using held-out LiveCodeBench problems (Table R3). OPSD declines from 32.44% at step 100 to 28.63% at step 200, below the base model (30.15%). In contrast, S3S^3-OPSD improves from 31.87% to 33.59%, supporting more stable coding training with our combined sparse-supervision and SGD configuration. We added these results to Appendix Sec. D.9 and Table A16, citing the suggested paper.

Table R3. Single-task coding on Qwen3-1.7B. LiveCodeBench (%, Avg@4, Easy / Medium / Hard) at training steps 100 / 150 / 200.

MethodEasy (100 / 150 / 200)Medium (100 / 150 / 200)Hard (100 / 150 / 200)Overall (100 / 150 / 200)
Base (Instruct)80.6531.413.6930.15
+ SDFT84.68 / 87.10 / 75.8129.49 / 29.49 / 28.216.56 / 6.97 / 5.7431.87 / 32.64 / 29.01
+ OPSD83.87 / 86.29 / 81.4533.97 / 28.21 / 25.005.33 / 6.56 / 4.1032.44 / 31.87 / 28.63
+ S3S^3-OPSD84.68 / 87.10 / 87.1031.41 / 31.41 / 32.695.33 / 6.97 / 6.9731.87 / 33.21 / 33.59

Reference
[R1] Jain, Naman, et al. Livecodebench: Holistic and contamination free evaluation of large language models for code. ICLR, 2025.

[W4] There are some typos in the Appendix please fix.

Thank you for the careful reading. We have corrected the typos in the Appendix, including “matchmatical” → “mathematical.” Additional corrections are listed below.

LocationCorrection
L946 (App. B.3)ηγ\eta\gamma → ηγ\eta_\gamma
L1013–1014Added the missing punctuation in the HumanEval+ sentence.
L1020“given below” → “given above”
L1265“for almost models” → “for almost all models”
Reviewer_DuugOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes S³-OPSD, a sparse, stable, and efficient framework for on-policy self-distillation. The method introduces an advantage-variance-inspired token importance score for sparse supervision, blockwise selection to maintain coverage across reasoning trajectories, SGD to reduce parameter drift, and teacher distribution smoothing to improve training stability. Experiments across multiple model families, mathematical reasoning benchmarks, and multi-task learning demonstrate improved performance and stability over existing baselines, along with reduced memory usage and faster training.

Critical strengths
  1. Well-motivated and coherent method design. The proposed components address clear limitations of existing on-policy self-distillation methods. The advantage-variance-inspired token selection provides a principled motivation for sparse supervision, while blockwise selection, SGD, and teacher distribution smoothing jointly improve training stability and efficiency.

  2. Strong empirical results and comprehensive evaluation. The paper demonstrates consistent improvements across multiple model families, model sizes, and reasoning benchmarks, with additional validation in multi-task learning. The ablation studies support the effectiveness of the individual components, while the substantial improvements in training stability, memory efficiency, and wall-clock time highlight the practical value of the approach.

Critical weaknesses
  1. Limited justification of the theoretical assumptions. The connection between the proposed token importance score and advantage variance relies on approximating the privileged-context-conditioned teacher as a reward-guided Doob-h transform of the student. However, the paper does not directly validate this approximation in practical OPSD settings. This weakens the theoretical justification for the proposed importance measure, although its empirical effectiveness is supported by the ablation studies.

  2. Insufficient analysis of the efficiency gains. While the paper reports substantial memory savings and training speedups, the contributions of sparse token supervision and optimizer choice to these improvements are not clearly disentangled. In particular, the implementation-level mechanism behind the significant memory reduction deserves further clarification, as it is central to the paper's efficiency claims.

Questions for authors
  1. Validity of the Doob-h transform assumption. The theoretical connection between the proposed importance score and advantage variance relies on approximating the privileged-context-conditioned teacher as a reward-guided Doob-h transform of the student. Could the authors provide empirical evidence supporting this approximation in practical OPSD settings, or directly examine the correlation between the proposed importance score and estimated advantage variance?

  2. Sources of memory efficiency. The paper reports substantial memory savings despite computing the importance score over the full vocabulary at every rollout position. Could the authors clarify how sparse token supervision reduces activation memory in the actual implementation? It would also be helpful to disentangle the memory savings attributable to sparse supervision from those resulting from replacing AdamW with SGD.

  3. Optimizer choice and training stability. The paper attributes improved stability partly to the sparse parameter updates induced by SGD. How sensitive are these findings to optimizer hyperparameters? Could the authors provide additional comparisons with appropriately tuned AdamW configurations to establish whether SGD's advantage is intrinsic to the optimizer rather than the specific training settings?

5 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer Duug [1/3]

We sincerely thank the reviewer for the time and care taken in reviewing our paper, and for the thoughtful and constructive feedback. We are especially glad that the reviewer found the method design well-motivated and coherent, in particular the advantage-variance perspective on sparse supervision and the way blockwise selection, SGD, and teacher distribution smoothing together improve stability and efficiency. We also appreciate the recognition of the consistent gains across model families, sizes, and benchmarks, and of the practical benefits in training stability, memory, and wall-clock time. The reviewer’s questions have helped us strengthen the paper, and we address each of them carefully below.

[W1, Q1] Validity of the Doob-h transform assumption. The connection between the proposed token importance score and advantage variance relies on approximating the privileged-context-conditioned teacher as a reward-guided Doob-h transform of the student, but this approximation is not directly validated in practical OPSD settings. Could the authors provide empirical evidence supporting this approximation, or directly examine the correlation between the proposed importance score and estimated advantage variance?

Following the reviewer's suggestion, we estimated a top-8 approximation C^t\hat C_t to CtC_t using nested Monte-Carlo rollouts on Qwen3-1.7B and included the analysis as Sec. 4.2 and Fig. 6 in the revision. Specifically, for each of 128 problems from Math-500 [R1], we sampled one student response and uniformly selected 32 positions from its first 1,024 tokens (4,096 states in total). At each state, we selected the K=8K{=}8 most likely next tokens, sampled N=32N{=}32 continuations per candidate, and graded their final answers, using the success rate as the value estimate V^j\hat V_j for each candidate. We then computed C^t=∑jw~j(V^j−Vˉ)2\hat C_t=\sum_j \tilde w_j(\hat V_j-\bar V)^2 with w~j\tilde w_j the candidate probabilities renormalized over the eight candidates. Note that C^t\hat C_t is a top-8 approximation of the full-vocabulary CtC_t rather than CtC_t itself; the eight candidates cover a median of 0.99999 of the next-token probability mass, so the omitted mass is small at almost all states. We compared C^t\hat C_t with the proposed importance score StS_t after discarding states with C^t<10−16\hat C_t<10^{-16} or St<10−7S_t<10^{-7} as unreliable (2,208 states). The results show that CtC_t increases monotonically with StS_t, with a Spearman correlation of ρ=0.813\rho=0.813. Because the 32 states of a problem come from the same response and are not independent, we conducted a cluster bootstrap that resamples problems (10,000 resamples) and it resulted in a 95% CI of [0.79,0.83][0.79, 0.83] for ρ\rho with exact sign test p<10−36p<10^{-36}.

Reference
[R1] Lightman, Hunter, et al. Let's verify step by step. ICLR, 2024.

AuthorsOpenReview ↗

Rebuttal for Reviewer Duug [2/3]

[W2, Q2] Sources of memory efficiency. The paper reports substantial memory savings and training speedups despite computing the importance score over the full vocabulary at every rollout position, but the contributions of sparse token supervision and optimizer choice are not clearly disentangled. Could the authors clarify how sparse token supervision reduces activation memory in the actual implementation, and disentangle the memory savings attributable to sparse supervision from those resulting from replacing AdamW with SGD?

In summary, replacing AdamW with SGD provides negligible memory and computation savings; the efficiency gains come primarily from sparse supervision, which enables all 32 rollouts to fit in one micro-batch and accelerates generation under 80GB memory budget.

Specifically, Table R2 isolates the optimizer choice for OPSD on Qwen3-1.7B. Replacing AdamW with SGD saves 0.52 GiB, matching the two fp32 moment buffers for the 69.7M trainable LoRA parameters. This reduces peak memory by only 0.8%. Although the optimizer update takes 1.21 ms with SGD versus 2.93 ms with AdamW, both contribute negligibly to the roughly 20s training step.

In contrast, sparse supervision reduces memory by computing training logits only at selected positions, while scoring all positions from stored hidden states. This enables S3S^3-OPSD to process all rollouts per step in one micro-batch and generate them in a single call. Table R3 shows similar step times at a matched micro-batch size, but substantially faster generation with the larger batch, reducing total step time to 7.10s. Thus, the speedup primarily comes from larger generation batches enabled by memory savings. We added both tables to the revision in Appendix Sec. D.8 and Table A15.

Table R2. Memory and training time of OPSD on Qwen3-1.7B with AdamW or SGD.

MethodOptimizerOptimizer state (GiB)Peak allocated (GiB)Optimizer update (ms)Time per step (s)
OPSDAdamW0.5263.652.9319.69
OPSDSGD0.0063.131.2119.82

Table R3. Time per step on Qwen3-1.7B under 80GB memory budget. Each step uses 32 rollouts, split according to the micro-batch size.

MethodRollouts per micro-batchGeneration callsGeneration (s)Teacher and student passes, loss (s)Scoring: StS_t over vocabulary (s)Backward and update (s)Total (s)
OPSD4815.892.38–2.1220.39
SDFT4814.732.70–2.0519.48
S3S^3-OPSD4815.022.760.221.8419.84
S3S^3-OPSD3213.122.380.191.667.35
AuthorsOpenReview ↗

Rebuttal for Reviewer Duug [3/3]

[Q3] Optimizer choice and training stability. The paper attributes improved stability partly to the sparse parameter updates induced by SGD. How sensitive are these findings to optimizer hyperparameters? Could the authors provide additional comparisons with appropriately tuned AdamW configurations to establish whether SGD's advantage is intrinsic to the optimizer rather than the specific training settings?

SGD’s stability advantage is robust to the learning rate, whereas AdamW faces a trade-off between early performance and stability. At the default learning rate (5×10−65\times10^{-6}), S3S^3-OPSD with AdamW (Table 2, ID 8; excerpted in Table R4) peaks at step 100 but loses 5.55 points by step 500, while SGD stays nearly flat (−0.83). Fig. 3(d) shows the same pattern for plain OPSD and SDFT.

Table R4. Excerpt from Table 2 (S3S^3-OPSD on Qwen3-1.7B; Avg@12 at training steps 100 / 300 / 500). Δ\DeltaAvg is the change in average accuracy from step 100 to step 500.

IDOptimizerAIME24 (100 / 300 / 500)HMMT25 (100 / 300 / 500)Average (100 / 300 / 500)Δ\DeltaAvg (500 − 100)
8AdamW55.00 / 51.39 / 49.4427.22 / 24.17 / 21.6741.11 / 37.78 / 35.56−5.55
0SGD (ours)54.72 / 55.00 / 54.1726.67 / 26.39 / 25.5640.70 / 40.70 / 39.87−0.83

We further evaluated S3S^3-OPSD across a range of learning rates for AdamW and SGD without momentum (Table R5). AdamW exhibits a trade-off between early performance and training stability: an LR of 1×10−61\times10^{-6} maintains stable accuracy, whereas 5×10−65\times10^{-6} achieves higher accuracy at step 100 but degrades with further training, and 1×10−51\times10^{-5} leads to collapse. In contrast, SGD maintains strong and stable accuracy across a wider range of learning rates (10−310^{-3} to 3×10−23\times10^{-2}; 51.94–54.44 at step 500), indicating that its advantage does not depend on a narrowly tuned setting. We added these results to Appendix Sec. D.10 and Table A17.

Table R5. AIME24 accuracy of S3S^3-OPSD across learning rates of SGD without momentum at training steps 100, 300, and 500.

MethodOptimizerLRstep 100step 300step 500
S3S^3-OPSDAdamW1×10−61\times10^{-6}52.2853.3353.61
S3S^3-OPSDAdamW5×10−65\times10^{-6}55.0051.3949.44
S3S^3-OPSDAdamW1×10−51\times10^{-5}51.674.170.56
S3S^3-OPSDSGD1×10−31\times10^{-3}48.8953.6154.17
S3S^3-OPSDSGD3×10−33\times10^{-3}52.5054.7254.44
S3S^3-OPSDSGD1×10−21\times10^{-2} (Selected)54.7255.0054.17
S3S^3-OPSDSGD3×10−23\times10^{-2}54.1753.6151.94
S3S^3-OPSDSGD1×10−11\times10^{-1}51.3945.2841.94

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

We thank the AC and all reviewers for their constructive feedback. Reviewers found the method well-motivated and clearly presented (all), the stability and efficiency gains practically meaningful (Duug, faKs), and the optimizer analysis novel (xU5m). We summarize the main concerns and our responses below.

1. Validity of the Doob-h assumption (Duug, faKs)

  • Concern: The link between the importance score and advantage variance is not validated empirically.
  • Response: We estimated advantage variance with nested Monte-Carlo rollouts. Our score increases monotonically with it, and the correlation is significant under a cluster bootstrap (Sec. 4.2, Fig. 6).

2. Source of efficiency gains (Duug)

  • Concern: It is unclear how much of the efficiency comes from sparse supervision and how much from SGD.
  • Response: SGD saves only 0.52 GiB (0.8% of peak memory). Sparse supervision lets all rollouts fit in one batch, which gives the 2.8× speedup (App. D.8).

3. Role of SGD (Duug, xU5m)

  • Concern: SGD's advantage may depend on hyperparameters, and the baselines were not run with SGD.
  • Response: SGD stays stable across a wide range of learning rates (App. D.10). OPSD and SDFT with SGD still degrade, while S³-OPSD stays stable (App. D.6).

4. Peak accuracy and related methods (faKs)

  • Concern: Peak-accuracy gains are modest, and recent OPSD methods (DASH, SR-OPSD, GeoSD) are missing.
  • Response: We agree that the main gains are stability and efficiency. S³-OPSD reaches 54% AIME24 in 6.1 min, vs. 17.0 min for OPSD (updated Fig. 1). Against DASH, it is more stable, 2.8× faster and uses less memory (App. D.7). We also clarified our differences from DASH, SR-OPSD, GeoSD, TIP, Do We Need Adam? and Veto (Sec. 2.2, 3.3).

5. Applicability to coding (xU5m)

  • Concern: OPSD is known to struggle on coding tasks.
  • Response: On LiveCodeBench, OPSD drops below the base model, while S³-OPSD keeps improving (App. D.9).

6. Clarity (xU5m)

  • Concern: There is no formal algorithm, and the appendix has typos.
  • Response: We added Algorithm 1 (Sec. 3.4) and fixed the typos.

We thank the AC and reviewers again for helping us improve the paper.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.