- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes an on-policy self-distillation method TISD that intervenes at high teacher–student disagreement points, forces a teacher-preferred token, and regenerates the student’s subsequent trajectory for distillation. The approach aims to expose the student to alternative trajectories that standard on-policy distillation may miss. Experiments on coding and science reasoning tasks show consistent but relatively modest improvements over existing methods.
Critical strengths
Clear motivation and intuitive design. The paper identifies a limitation of standard on-policy self-distillation and proposes a simple, targeted trajectory intervention strategy to address it.
Useful diagnostic analysis. The authors investigate how teacher-guided branching affects subsequent generation and evaluate the method on both coding and science reasoning tasks, providing empirical support for the proposed approach.
Critical weaknesses
Limited performance gains. Improvements over the baseline are relatively small, particularly under equal training-time budgets. More evidence of statistical significance and robustness across random seeds is needed to establish the practical value of the method.
Insufficient isolation of the key mechanism. The experiments do not fully disentangle the benefit of teacher-guided token intervention from that of additional student rollouts and training data. Stronger compute-matched ablations would better establish the source of the improvements.
Questions for authors
Are the reported improvements consistent across multiple random seeds, and are they statistically significant, especially under equal training-time budgets?
Can the authors provide compute-matched ablations that isolate the effect of teacher-guided intervention from the benefit of additional rollouts? How sensitive is TISD to the branch-point selection strategy and intervention frequency?
3 comments
W2 and Q2. Isolating the intervention, branch-point selection, and intervention frequency
W2. Insufficient isolation of the key mechanism. The experiments do not fully disentangle the benefit of teacher-guided token intervention from that of additional student rollouts and training data. Stronger compute-matched ablations would better establish the source of the improvements.
Q2. Can the authors provide compute-matched ablations that isolate the effect of teacher-guided intervention from the benefit of additional rollouts? How sensitive is TISD to the branch-point selection strategy and intervention frequency?
Response.
The submitted paper includes two controls that directly address this concern; we clarify their compute matching and implications below.
(i) TISD does not increase the number of retained training trajectories per update. Since the regenerated trajectory replaces the failed rollout, TISD trains on the same number of trajectories per update as SDPO. Moreover, the "student own token" control (Figure 7) uses the identical pipeline, with the same branch position, student suffix regeneration, number of trajectories, and compute, and differs only in keeping the student's original token. Table R4 summarizes this comparison.
Table R4. Compute-matched ablation on LCB-Small (Qwen3-8B, best test-case pass rate, %), summarizing Figure 7. Mean ± std over three runs for the control and TISD.
| Method | Branch token | Suffix regeneration | Pass rate |
|---|---|---|---|
| SDPO | none | none | 65.5 ± 1.0 |
| Student own token (compute-matched control) | student | student | 65.0 ± 0.2 |
| TISD | teacher | student | 66.5 ± 0.2 |
The control falls below SDPO, so regeneration alone does not help; instead, the gain comes from the teacher-selected action. The 1.5 pp gap between TISD and the control is consistent across seeds: every TISD run (66.3 to 66.7) exceeds every control run (64.9 to 65.2). This aligns with our diagnostic analysis (Section 3): at the same position, the student solves 9.9% of failed rollouts after the teacher token but only 1.7% after its own token (Figure 3).
(ii) Peak-divergence branch selection outperforms the alternatives. Peak divergence reaches 66.5 ± 0.2% on LCB-Small, compared with 65.2% for a random position and 64.6% for the first student-teacher disagreement, and only peak divergence improves over SDPO (65.5%) (Figure 7). Consistently, peak divergence also yields the highest success rate in our diagnostic: 9.9%, versus 6.1% at the first disagreement and 1.7% at a random position (Appendix Table 4).
(iii) A single intervention per trajectory is sufficient. Branching the regenerated trajectory again gives 65.3 ± 0.8% with two rounds and 66.3 ± 1.2% with three, compared with 66.5 ± 0.2% with one (Figure 8). Extra rounds add compute by regenerating multiple times (4.1~4.9% overhead) without improving the performance, so TISD uses one branch per failed rollout.
We hope these responses address the reviewer's concerns, and we would be happy to provide further clarification.
Dear Reviewer 8ihX,
Thank you again for your time and for the insightful review. We would like to kindly follow up on our response and share a brief update.
Regarding your question on robustness across runs (W1 and Q1), the remaining runs have now finished, and we have updated the response and the revised manuscript accordingly. Table 1 now reports the mean and standard deviation for every entry. TISD achieves the highest mean on all four models under both the equal-step and equal-time budgets, with average gains over SDPO of 1.5 pp and 1.2 pp, respectively. The updated results maintain the same significant Friedman test result (p = 0.013 at equal steps, p = 0.032 at equal time). On math reasoning, TISD reaches 65.8 ± 0.5 mean Avg@16 over three runs, compared with at most 64.6 for the baselines.
We hope our response, together with the compute-matched control and the branch-selection and intervention-frequency ablations (W2 and Q2), addresses your concerns. If any point remains unclear, we would be glad to provide further clarification or additional analysis during the discussion period. If our response has resolved your concerns, we would greatly appreciate it if you could consider updating your assessment.
Thank you very much for your consideration.
Best regards,
The Authors
We thank the reviewer for the insightful feedback, and we appreciate the recognition of TISD's clear motivation and intuitive design, and of our diagnostic analysis of how teacher-guided branching affects subsequent generation. We have addressed all of the reviewer’s points below, and all changes are marked in red in the revised manuscript.
W1 and Q1. Robustness across multiple runs and statistical significance
Response.
We provide additional experiments under both equal-step and equal-time budgets, in which TISD shows consistent and statistically significant improvements over the baselines. We further analyze where TISD brings the largest benefit.
(i) TISD outperforms SDPO under both equal-step and equal-time budgets. Since 80 TISD updates take about as long as 88 SDPO updates (Appendix D.3, Table 13), we conservatively give SDPO 90 updates, with the additional rebuttal experiments of Qwen3-4B (Table R1).
Table R1. Best Avg@4 (%) on LCBv6, mean ± std over three runs. SDPO ≤90 updates matches the training time of TISD ≤80 updates. Bold: best per row.
The 10 extra updates improve SDPO by only 0.3 pp on average, so TISD stays ahead on all four models.
(ii) The gain extends to math and is largest on hard science problems.
Math. In this rebuttal, we also evaluated TISD on math reasoning in the OPSD setting (Zhao et al., 2026). With Qwen3-8B, TISD achieves the highest mean Avg@16, 1.2 pp above the best baseline (Table R2).
Table R2. Avg@16 (%) on math reasoning for Qwen3-8B trained with ground-truth solution feedback (OPSD setting). Bold: best.
Hard science problems. The smaller gain on SciKnowEval comes from the full Materials domain, where GRPO outperforms both SDPO and TISD (Table 2). Materials has among the highest base model accuracies in SciKnowEval L3 (59.0%), which leaves few failed rollouts for TISD to intervene on. We therefore built a hard subset of Materials L3 problems on which the base model has ≤50% rollout accuracy. On this subset, TISD outperforms SDPO by 4.21 pp at equal steps and 4.65 pp at equal time (Table R3), reversing the gap on full Materials.
Table R3. Accuracy (%) of Qwen3-8B on the SciKnowEval Materials hard subset (L3 problems with ≤50% base-model rollout accuracy).
(iii) The improvements are consistent across runs and statistically significant. The main Qwen3-8B experiments (coding and science) in the original manuscript already report averages over three runs for GRPO, SDPO, and TISD. For this rebuttal, we added additional seeds, and the revised Tables 1 and 2 report the mean and standard deviation. On LCBv6, TISD reaches 49.2 ± 0.7 versus 48.1 ± 1.7 for SDPO with Qwen3-8B, and 50.8 ± 0.4 versus 49.8 ± 1.2 with Qwen3-14B. To assess significance, we compare TISD, SDPO, and GRPO across nine settings (four models on LCBv6, LCB-Large, and the four SciKnowEval domains). TISD achieves the best average performance under both budgets, and a Friedman test confirms that the differences among methods are significant (p = 0.013 at equal steps, p = 0.032 at equal time).