acceptodds
Under review as a conference paper at ICLR 2027

TISD: On-Policy Self-Distillation with Trajectory Intervention

Abstract

On-policy self-distillation (OPSD) provides dense teacher targets, but evaluates them only along student-sampled rollouts. When the privileged teacher favors an alternative action at a visited prefix, OPSD can provide a target for the branch decision but cannot supervise the successor contexts induced by that action unless the student samples it. This creates a training-time data-collection bottleneck and suggests a different role for teacher–student disagreement: proposing a trajectory branch rather than identifying a sufficient local repair. Our diagnostic framework using controlled token interventions reveals that a teacher-preferred token at peak disagreement can improve student continuation success, while its local corrective value is limited. Motivated by this finding, we introduce a simple branch–regenerate–distill algorithm, Trajectory-Intervention Self-Distillation (TISD). TISD forces a teacher-selected branch action, returns suffix generation to the student, and distills the full trajectory under the privileged-context-conditioned teacher. Across the coding models, TISD improves average Avg@4 over SDPO by 1.2 percentage points. Across the science domains, it improves average Avg@128 by 0.8 points under an equal-step budget and by 0.3 points under an equal-time budget. These results support teacher-guided branching as a way to expose useful successor contexts for self-distillation. The source code is available at https://anonymous.4open.science/r/TISD-B87E/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy self-distillation, large language model, reasoninig, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8ihXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an on-policy self-distillation method TISD that intervenes at high teacher–student disagreement points, forces a teacher-preferred token, and regenerates the student’s subsequent trajectory for distillation. The approach aims to expose the student to alternative trajectories that standard on-policy distillation may miss. Experiments on coding and science reasoning tasks show consistent but relatively modest improvements over existing methods.

Critical strengths

Clear motivation and intuitive design. The paper identifies a limitation of standard on-policy self-distillation and proposes a simple, targeted trajectory intervention strategy to address it.

Useful diagnostic analysis. The authors investigate how teacher-guided branching affects subsequent generation and evaluate the method on both coding and science reasoning tasks, providing empirical support for the proposed approach.

Critical weaknesses

Limited performance gains. Improvements over the baseline are relatively small, particularly under equal training-time budgets. More evidence of statistical significance and robustness across random seeds is needed to establish the practical value of the method.

Insufficient isolation of the key mechanism. The experiments do not fully disentangle the benefit of teacher-guided token intervention from that of additional student rollouts and training data. Stronger compute-matched ablations would better establish the source of the improvements.

Questions for authors

Are the reported improvements consistent across multiple random seeds, and are they statistically significant, especially under equal training-time budgets?

Can the authors provide compute-matched ablations that isolate the effect of teacher-guided intervention from the benefit of additional rollouts? How sensitive is TISD to the branch-point selection strategy and intervention frequency?

3 comments

We thank the reviewer for the insightful feedback, and we appreciate the recognition of TISD's clear motivation and intuitive design, and of our diagnostic analysis of how teacher-guided branching affects subsequent generation. We have addressed all of the reviewer’s points below, and all changes are marked in red in the revised manuscript.

W1 and Q1. Robustness across multiple runs and statistical significance

W1. Limited performance gains. Improvements over the baseline are relatively small, particularly under equal training-time budgets. More evidence of statistical significance and robustness across random seeds is needed to establish the practical value of the method.

Q1. Are the reported improvements consistent across multiple random seeds, and are they statistically significant, especially under equal training-time budgets?

Response.

We provide additional experiments under both equal-step and equal-time budgets, in which TISD shows consistent and statistically significant improvements over the baselines. We further analyze where TISD brings the largest benefit.

(i) TISD outperforms SDPO under both equal-step and equal-time budgets. Since 80 TISD updates take about as long as 88 SDPO updates (Appendix D.3, Table 13), we conservatively give SDPO 90 updates, with the additional rebuttal experiments of Qwen3-4B (Table R1).

Table R1. Best Avg@4 (%) on LCBv6, mean ± std over three runs. SDPO ≤90 updates matches the training time of TISD ≤80 updates. Bold: best per row.

ModelSDPO ≤80 upd.SDPO ≤90 upd. (equal time)TISD ≤80 upd.
Qwen3-4B44.2 ± 0.144.3 ± 0.346.9 ± 1.0
Qwen3-8B48.1 ± 1.748.3 ± 1.749.2 ± 0.7
Qwen3-14B49.8 ± 1.250.1 ± 0.950.8 ± 0.4
Olmo3-7B50.3 ± 0.850.9 ± 0.251.3 ± 0.1
Average48.148.449.6

The 10 extra updates improve SDPO by only 0.3 pp on average, so TISD stays ahead on all four models.

(ii) The gain extends to math and is largest on hard science problems.

Math. In this rebuttal, we also evaluated TISD on math reasoning in the OPSD setting (Zhao et al., 2026). With Qwen3-8B, TISD achieves the highest mean Avg@16, 1.2 pp above the best baseline (Table R2).

Table R2. Avg@16 (%) on math reasoning for Qwen3-8B trained with ground-truth solution feedback (OPSD setting). Bold: best.

MethodAIME24AIME25HMMT25Mean
Base75.865.643.961.8
SFT76.970.846.364.7
GRPO77.770.845.464.6
OPSD77.3 ± 0.270.6 ± 0.345.9 ± 0.464.6 ± 0.1
TRD75.669.045.663.4
TISD77.9 ± 1.171.4 ± 0.947.9 ± 0.965.8 ± 0.5

Hard science problems. The smaller gain on SciKnowEval comes from the full Materials domain, where GRPO outperforms both SDPO and TISD (Table 2). Materials has among the highest base model accuracies in SciKnowEval L3 (59.0%), which leaves few failed rollouts for TISD to intervene on. We therefore built a hard subset of Materials L3 problems on which the base model has ≤50% rollout accuracy. On this subset, TISD outperforms SDPO by 4.21 pp at equal steps and 4.65 pp at equal time (Table R3), reversing the gap on full Materials.

Table R3. Accuracy (%) of Qwen3-8B on the SciKnowEval Materials hard subset (L3 problems with ≤50% base-model rollout accuracy).

TISD ≤200 upd.SDPO ≤200 upd.TISD ≤10 hSDPO ≤10 h
84.3580.1484.7980.14

(iii) The improvements are consistent across runs and statistically significant. The main Qwen3-8B experiments (coding and science) in the original manuscript already report averages over three runs for GRPO, SDPO, and TISD. For this rebuttal, we added additional seeds, and the revised Tables 1 and 2 report the mean and standard deviation. On LCBv6, TISD reaches 49.2 ± 0.7 versus 48.1 ± 1.7 for SDPO with Qwen3-8B, and 50.8 ± 0.4 versus 49.8 ± 1.2 with Qwen3-14B. To assess significance, we compare TISD, SDPO, and GRPO across nine settings (four models on LCBv6, LCB-Large, and the four SciKnowEval domains). TISD achieves the best average performance under both budgets, and a Friedman test confirms that the differences among methods are significant (p = 0.013 at equal steps, p = 0.032 at equal time).

W2 and Q2. Isolating the intervention, branch-point selection, and intervention frequency

W2. Insufficient isolation of the key mechanism. The experiments do not fully disentangle the benefit of teacher-guided token intervention from that of additional student rollouts and training data. Stronger compute-matched ablations would better establish the source of the improvements.

Q2. Can the authors provide compute-matched ablations that isolate the effect of teacher-guided intervention from the benefit of additional rollouts? How sensitive is TISD to the branch-point selection strategy and intervention frequency?

Response.

The submitted paper includes two controls that directly address this concern; we clarify their compute matching and implications below.

(i) TISD does not increase the number of retained training trajectories per update. Since the regenerated trajectory replaces the failed rollout, TISD trains on the same number of trajectories per update as SDPO. Moreover, the "student own token" control (Figure 7) uses the identical pipeline, with the same branch position, student suffix regeneration, number of trajectories, and compute, and differs only in keeping the student's original token. Table R4 summarizes this comparison.

Table R4. Compute-matched ablation on LCB-Small (Qwen3-8B, best test-case pass rate, %), summarizing Figure 7. Mean ± std over three runs for the control and TISD.

MethodBranch tokenSuffix regenerationPass rate
SDPOnonenone65.5 ± 1.0
Student own token (compute-matched control)studentstudent65.0 ± 0.2
TISDteacherstudent66.5 ± 0.2

The control falls below SDPO, so regeneration alone does not help; instead, the gain comes from the teacher-selected action. The 1.5 pp gap between TISD and the control is consistent across seeds: every TISD run (66.3 to 66.7) exceeds every control run (64.9 to 65.2). This aligns with our diagnostic analysis (Section 3): at the same position, the student solves 9.9% of failed rollouts after the teacher token but only 1.7% after its own token (Figure 3).

(ii) Peak-divergence branch selection outperforms the alternatives. Peak divergence reaches 66.5 ± 0.2% on LCB-Small, compared with 65.2% for a random position and 64.6% for the first student-teacher disagreement, and only peak divergence improves over SDPO (65.5%) (Figure 7). Consistently, peak divergence also yields the highest success rate in our diagnostic: 9.9%, versus 6.1% at the first disagreement and 1.7% at a random position (Appendix Table 4).

(iii) A single intervention per trajectory is sufficient. Branching the regenerated trajectory again gives 65.3 ± 0.8% with two rounds and 66.3 ± 1.2% with three, compared with 66.5 ± 0.2% with one (Figure 8). Extra rounds add compute by regenerating multiple times (4.1~4.9% overhead) without improving the performance, so TISD uses one branch per failed rollout.

We hope these responses address the reviewer's concerns, and we would be happy to provide further clarification.

Dear Reviewer 8ihX,

Thank you again for your time and for the insightful review. We would like to kindly follow up on our response and share a brief update.

Regarding your question on robustness across runs (W1 and Q1), the remaining runs have now finished, and we have updated the response and the revised manuscript accordingly. Table 1 now reports the mean and standard deviation for every entry. TISD achieves the highest mean on all four models under both the equal-step and equal-time budgets, with average gains over SDPO of 1.5 pp and 1.2 pp, respectively. The updated results maintain the same significant Friedman test result (p = 0.013 at equal steps, p = 0.032 at equal time). On math reasoning, TISD reaches 65.8 ± 0.5 mean Avg@16 over three runs, compared with at most 64.6 for the baselines.

We hope our response, together with the compute-matched control and the branch-selection and intervention-frequency ablations (W2 and Q2), addresses your concerns. If any point remains unclear, we would be glad to provide further clarification or additional analysis during the discussion period. If our response has resolved your concerns, we would greatly appreciate it if you could consider updating your assessment.

Thank you very much for your consideration.

Best regards,

The Authors

Reviewer_JjnfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a new version of on-policy self-distillation. Here, a student generation is regenerated with a single token hint from the teacher. First, the point where the teacher and student disagree the most is selected, and then the next token is selected using the teacher, and the student is left to regenerate the rest of the complete solution. This is shown to result in better solutions on average. Then, using these solutions in self-distillation leads to improved performance.

Critical strengths

The method is sound and simple and is shown to produce improved results.

It’s interesting to note that using a single token from the teacher and then regenerating using the student will lead to improved solutions on average.

Critical weaknesses

There should be a longer discussion on the computational overhead of the method. Since some solutions are regenerated, this could come at a high computational cost. It’s good that Table 2 gives results where the training time is matched. In this setting, we see that the improvements of TISD over SDPO are minimal (0.3 vs 0.8 in the equal steps setting).

All LiveCodeBench results are given in the equal-step setting, which is less relevant than equal time, given the time overhead of the proposed method. Training for the same amount of time should also be done in this setting.

Are the results reported using multiple seeds or a single run? How is the checkpoint selection made? The final checkpoint or the best across training?

Questions for authors

See Weaknesses.

3 comments

We thank the reviewer for the constructive feedback. We appreciate the assessment that TISD is sound and simple, that a single teacher token followed by student regeneration yields improved solutions, and that the time-matched comparison in Table 2 is valuable. We have addressed all of the reviewer’s points below, and all changes are marked in red in the revised manuscript.

W1 and W2. Computational overhead and equal-time comparison

W1. There should be a longer discussion on the computational overhead of the method. Since some solutions are regenerated, this could come at a high computational cost. It's good that Table 2 gives results where the training time is matched. In this setting, we see that the improvements of TISD over SDPO are minimal (0.3 vs 0.8 in the equal steps setting).

W2. All LiveCodeBench results are given in the equal-step setting, which is less relevant than equal time, given the time overhead of the proposed method. Training for the same amount of time should also be done in this setting.

Response.

TISD's gain over SDPO is largely preserved at equal training time, and it holds across coding, math, and hard science problems.

(i) The gain is largely preserved at equal training time. TISD adds one operation to SDPO, regenerating the suffix with the student, and applies it only to failed rollouts. One update therefore takes 16.60 min for TISD and 15.05 min for SDPO (+10.3%). GRPO takes 16.69 min and TRD 16.36 min, so TISD costs about the same as the other baselines. This cost analysis is in the submitted manuscript (Appendix D.3, Table 13).

In this rebuttal, we include new equal-time experiments on LiveCodeBench. 80 TISD updates take about as long as 88 SDPO updates, so we conservatively extended the SDPO to 90 updates (Table R1). We also added Qwen3-4B in the rebuttal.

Table R1. Best Avg@4 (%) on LCBv6, mean ± std over three runs. SDPO ≤90 updates matches the training time of TISD ≤80 updates. Bold: best per row.

ModelSDPO ≤80 upd.SDPO ≤90 upd. (equal time)TISD ≤80 upd.
Qwen3-4B44.2 ± 0.144.3 ± 0.346.9 ± 1.0
Qwen3-8B48.1 ± 1.748.3 ± 1.749.2 ± 0.7
Qwen3-14B49.8 ± 1.250.1 ± 0.950.8 ± 0.4
Olmo3-7B50.3 ± 0.850.9 ± 0.251.3 ± 0.1
Average48.148.449.6

At equal time, TISD leads SDPO by 1.2 pp on average over the four models, close to its 1.5 pp gain at equal steps. The 10 extra updates improve SDPO by only 0.3 pp on average (at most 0.6 pp on any model), so SDPO stays behind TISD on all four models. TISD also learns faster: on LCB-Small, it matches SDPO's best test-case pass rate (65.5%) with 56% fewer updates, or about 51% less training time (Figure 6).

(ii) The gain holds across coding, math, and hard science problems.

Coding. The newly added Qwen3-4B shows the same trend (Table R1). On LCB-Large, TISD reaches 57.3%, which is 4.0 pp above SDPO, and it surpasses GRPO's best (54.3%) with 33% fewer updates (Figure 6).

Math. In this rebuttal, we also evaluated TISD on math reasoning in the OPSD setting (Zhao et al., 2026). With Qwen3-8B, TISD achieves the highest mean Avg@16, 1.2 pp above the best baseline (Table R2).

Table R2. Avg@16 (%) on math reasoning for Qwen3-8B trained with ground-truth solution feedback (OPSD setting). Bold: best.

MethodAIME24AIME25HMMT25Mean
Base75.865.643.961.8
SFT77.4 ± 0.569.3 ± 1.546.3 ± 0.264.3 ± 0.5
GRPO77.4 ± 0.570.2 ± 0.646.3 ± 0.864.6 ± 0.3
OPSD77.3 ± 0.270.5 ± 0.345.9 ± 0.464.6 ± 0.1
TRD75.6 ± 0.069.3 ± 1.046.7 ± 2.163.9 ± 1.0
TISD77.9 ± 1.171.4 ± 0.847.9 ± 0.965.8 ± 0.5

Hard science problems. The 0.3 pp equal-time gain in Table 2 is an average over four domains, and Materials lowers it: Materials has among the highest base accuracies in SciKnowEval L3 (59.0%) and is the only domain where GRPO outperforms both distillation methods. Since TISD intervenes on failed rollouts, we further examine whether its benefits are more pronounced on problems that the base model struggles with. We therefore built a hard subset of Materials L3 problems on which the base model has ≤50% rollout accuracy. On this subset, TISD outperforms SDPO by 4.21 pp at equal steps and 4.65 pp at equal time (Table R3), reversing the gap on full Materials. The gain therefore does not shrink under the time budget when problems are difficult.

Table R3. Accuracy (%) of Qwen3-8B on the SciKnowEval Materials hard subset (L3 problems with ≤50% base-model rollout accuracy).

TISD ≤200 upd.SDPO ≤200 upd.TISD ≤10 hSDPO ≤10 h
84.3580.1484.7980.14

Following the reviewer’s suggestion, we moved the computational cost analysis from Appendix D.3 to the main text as a new Section 5.3, expanded it with the per-update cost breakdown, and added the equal-time LiveCodeBench comparison.

References

  • (Zhao et al., 2026) Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models, ICML '26

W3. Training seeds and checkpoint selection

Are the results reported using multiple seeds or a single run? How is the checkpoint selection made? The final checkpoint or the best across training?

Response.

For every method, we evaluate a checkpoint every 5 updates and report the best one within the training budget, following the SDPO evaluation protocol (Hübotter et al., 2026).

The main Qwen3-8B experiments (coding and science) in the original manuscript already report averages over three runs for GRPO, SDPO, and TISD. For this rebuttal, we added additional seeds and are continuing to run other baselines, which we will keep updated. On LCBv6, TISD reaches 49.2 ± 0.7 versus 48.1 ± 1.7 for SDPO with Qwen3-8B, and 50.8 ± 0.4 versus 49.8 ± 1.2 with Qwen3-14B.

We hope these responses address the reviewer’s concerns. We would be happy to provide further clarification.

References

  • (Hübotter et al., 2026) Reinforcement Learning via Self-Distillation, ICML '26

Dear Reviewer Jjnf,

Thank you again for your time and for the constructive review. We would like to kindly follow up on our response and share a brief update.

In our earlier response to W3, we noted that additional runs were still in progress. These runs have now finished, and we have updated the response and the revised manuscript accordingly. Table 1 now reports the mean and standard deviation for every entry, and the equal-time LiveCodeBench comparison you requested (W2) also reports standard deviations. At equal training time, TISD stays ahead of SDPO on all four models, with an average gain of 1.2 pp (1.5 pp at equal steps). The ten additional SDPO updates improve SDPO by only 0.3 pp on average.

We hope our response addresses your concerns about the computational overhead, the equal-time comparison, and the reporting of multiple runs and checkpoint selection. If any point remains unclear, we would be glad to provide further clarification or additional analysis during the discussion period. If our response has resolved your concerns, we would greatly appreciate it if you could consider updating your assessment.

Thank you very much for your consideration.

Best regards,

The Authors

Reviewer_CagQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets on-policy self-distillation (OPSD), where the teacher is the student conditioned on privileged context. It observes that SDPO-style OPSD supervises only student-sampled prefixes, so successors of teacher-preferred tokens go unlearned. Their method, TISD, is a branch–regenerate–distill loop that lets the student sample the suffix and then distills the resulting trajectory with the SDPO objective. Across several benchmarks, TISD yields modest improvements over SDPO.

Critical strengths
  1. The paper is well written and easy to follow.

  2. The paper gives an extensive analysis of teacher–student divergence in OPSD and, more interestingly, an interventional argument rather than a correlation plot: in-place substitution, teacher continuation, and teacher-token-plus-regeneration. These probes offer genuine insight into OPSD's mechanism.

  3. The evaluation spans model sizes and families (Qwen3-8B/14B, Olmo3-7B-Instruct), and includes comparisons against other state-of-the-art methods.

Critical weaknesses
  1. The method adds real cost for very small gains. Every failed rollout needs an extra branch-and-regenerate pass, which the authors measure at 10.3% longer per training step than SDPO. The return is +1.2 pp Avg@4 on code, an average over three models that is driven almost entirely by Qwen3-14B (+2.2, against +0.2 for Olmo3-7B), and +0.8 pp on science at equal steps, falling to +0.3 pp at equal time.

  2. The core premise is not well justified. TISD assumes that OPSD's bottleneck is the absence of teacher supervision on successors of teacher-preferred actions, and that handing suffix generation back to the student keeps training on-policy. But the branch token is forced precisely where student and teacher disagree most, i.e. where the student would rarely sample that token; conditioning the suffix on such a prefix moves the trajectory off the student's own state distribution by construction. The difference from TRD is therefore one of degree, not kind: a single teacher-chosen token rather than a whole teacher-written suffix.

Questions for authors
  1. I am somewhat confused about the training setting. The paper states, "We train Qwen3-8B, Qwen3-14B, and Olmo3-7B-Instruct on LiveCodeBench v6" — does this mean training on the test set?
4 comments

We sincerely thank Reviewer CagQ for the thoughtful feedback and for recognizing the clarity of the paper, our interventional analysis of OPSD, and the evaluation across model sizes and families. We have addressed all of the reviewer’s points below, and all changes are marked in red in the revised manuscript.

W1. Computational cost and size of the gains

The method adds real cost for very small gains. Every failed rollout needs an extra branch-and-regenerate pass, which the authors measure at 10.3% longer per training step than SDPO. The return is +1.2 pp Avg@4 on code, an average over three models that is driven almost entirely by Qwen3-14B (+2.2, against +0.2 for Olmo3-7B), and +0.8 pp on science at equal steps, falling to +0.3 pp at equal time.

Response.

We would like to highlight that TISD achieves faster learning and higher performance than SDPO under equal training-time cost, with meaningful gains across coding, math, and challenging science problems. We provide additional analyses and experiments to support these findings, as detailed below.

(i) TISD's advantage holds on all (submitted) models at equal training time. TISD takes 16.60 min per update, compared with 15.05 for SDPO, 16.69 for GRPO, and 16.36 for TRD (Appendix D.3, Table 13). Since 80 TISD updates take about as long as 88 SDPO updates, we conservatively extended the SDPO runs 90 updates to compare the two methods at equal training time. We also added Qwen3-4B in this rebuttal (Table R1).

Table R1. Best Avg@4 (%) on LCBv6, mean ± std over three runs. SDPO ≤90 updates matches the training time of TISD ≤80 updates. Bold: best per row.

ModelSDPO ≤80 upd.SDPO ≤90 upd. (equal time)TISD ≤80 upd.
Qwen3-4B44.2 ± 0.144.3 ± 0.346.9 ± 1.0
Qwen3-8B48.1 ± 1.748.3 ± 1.749.2 ± 0.7
Qwen3-14B49.8 ± 1.250.1 ± 0.950.8 ± 0.4
Olmo3-7B50.3 ± 0.850.9 ± 0.251.3 ± 0.1
Average48.148.449.6

At equal time, TISD leads SDPO by 1.2 pp on average over the four models, close to its 1.5 pp gain at equal steps. TISD also learns faster: on LCB-Small, it matches SDPO’s best test-case pass rate (65.5%) with 56% fewer updates, or about 51% less training time (Figure 6).

(ii) TISD shows meaningful gains across coding, math, and hard science problems.

Coding. The newly added Qwen3-4B shows the same trend (Table R1). On LCB-Large, TISD reaches 57.3%, which is 4.0 pp above SDPO, and it surpasses GRPO's best (54.3%) with 33% fewer updates (Figure 6).

Math. We provide additional experiments on math reasoning in the OPSD setting (Zhao et al., 2026). With Qwen3-8B, TISD achieves the highest mean Avg@16, 1.2 pp above the best baseline (Table R2).

Table R2. Avg@16 (%) on math reasoning for Qwen3-8B trained with ground-truth solution feedback (OPSD setting). Bold: best.

MethodAIME24AIME25HMMT25Mean
Base75.865.643.961.8
SFT77.4 ± 0.569.3 ± 1.546.3 ± 0.264.3 ± 0.5
GRPO77.4 ± 0.570.2 ± 0.646.3 ± 0.864.6 ± 0.3
OPSD77.3 ± 0.270.5 ± 0.345.9 ± 0.464.6 ± 0.1
TRD75.6 ± 0.069.3 ± 1.046.7 ± 2.163.9 ± 1.0
TISD77.9 ± 1.171.4 ± 0.847.9 ± 0.965.8 ± 0.5

Hard science problems. The 0.3 pp equal-time gain on science is an average over four domains, and Materials lowers it: Materials has among the highest base accuracies in SciKnowEval L3 (59.0%) and is the only domain where GRPO outperforms both distillation methods (Table 2). Since TISD intervenes on failed rollouts, we further examine whether its benefits are more pronounced on problems that the base model struggles with. We therefore built a hard subset of Materials L3 problems on which the base model has ≤50% rollout accuracy. On this subset, TISD outperforms SDPO by 4.21 pp at equal steps and 4.65 pp at equal time (Table R3), reversing the gap on full Materials. The gain does not shrink under the equal time budget.

Table R3. Accuracy (%) of Qwen3-8B on the SciKnowEval Materials hard subset (L3 problems with ≤50% base-model rollout accuracy).

TISD ≤200 upd.SDPO ≤200 upd.TISD ≤10 hSDPO ≤10 h
84.3580.1484.7980.14

W2. The rollout distribution and the distinction from TRD

The core premise is not well justified. TISD assumes that OPSD's bottleneck is the absence of teacher supervision on successors of teacher-preferred actions, and that handing suffix generation back to the student keeps training on-policy. But the branch token is forced precisely where student and teacher disagree most, i.e. where the student would rarely sample that token; conditioning the suffix on such a prefix moves the trajectory off the student's own state distribution by construction. The difference from TRD is therefore one of degree, not kind: a single teacher-chosen token rather than a whole teacher-written suffix.

Response.

We agree that forcing a teacher-preferred token changes the distribution of visited prefixes, and we did not claim that the resulting full trajectory is strictly on-policy. We revised the manuscript to make this clear. What we claim is that, conditional on the intervened prefix, the entire suffix is sampled from the student policy, whereas in TRD it is generated by a privileged teacher.

OPSD explicitly trains the student to increase the probability of the teacher-preferred action. TISD therefore provides supervision on successor states that become more relevant as the student learns that action, even if they are rarely visited under the current policy. For example, on failed Qwen3-8B rollouts, the student solves 9.9% of failures after the teacher token intervention, but only 1.7% after its own token (Figure 3). Moreover, although these continuations succeed more often, their student-teacher KL divergence is 2.37x larger than that of the original suffixes (Figure 5b), which leaves more for the student to learn from the teacher on these successor states.

Importantly, TISD limits the intervention to a single action while preserving student-generated continuations. Unlike TRD, it does not train the student to imitate an entire suffix generated under privileged context. This difference is not only one of degree in practice: TRD collapses on SciKnowEval biology, where Avg@128 drops from 64.8% to 0.0% (Figure 4), and on math reasoning it falls 0.4 to 0.7 pp below SFT, GRPO, and OPSD (63.9% vs. 64.3 to 64.6% mean Avg@16), whereas TISD is the best method at 65.8% (Table R2).

As our ablations in the submitted manuscript show (Figures 7 and 8), the benefit also depends on where and how often the teacher intervenes. On LCB-Small, a single intervention at the peak-divergence position reaches a 66.5 ± 0.2% test-case pass rate, compared with 65.0 ± 0.2% when the student’s original token is kept and 65.5% for SDPO (Figure 7). The gain disappears at a random position (65.2%) or at the first disagreement (64.6%), and additional intervention rounds do not improve it further (65.3 ± 0.8% with two and 66.3 ± 1.2% with three; Figure 8).

Q1. LiveCodeBench training and evaluation splits

I am somewhat confused about the training setting. The paper states, "We train Qwen3-8B, Qwen3-14B, and Olmo3-7B-Instruct on LiveCodeBench v6", does this mean training on the test set?

Response.

We strictly follow the SDPO protocol (Hübotter et al., 2026): during training, where the model is trained on the 50% of the testcases, while other half is held-out and only evaluated during testing.

We hope these responses address the reviewer's concerns, and we would be happy to provide further clarification.

References

  • (Hübotter et al., 2026) Reinforcement Learning via Self-Distillation, ICML '26
  • (Zhao et al., 2026) Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models, ICML '26

Dear Reviewer CagQ,

Thank you again for your time and for the thoughtful review. We would like to kindly follow up on our response and share a brief update.

Since our initial response, the remaining runs have finished, and we have updated the response and the revised manuscript accordingly. Table 1 now reports the mean and standard deviation for every entry. Regarding your concern on cost versus gain (W1), TISD stays ahead of SDPO on all four models at equal training time, with an average gain of 1.2 pp (1.5 pp at equal steps). On math reasoning, TISD reaches 65.8 ± 0.5 mean Avg@16, compared with at most 64.6 for the baselines.

We hope our response addresses your concerns about the computational cost, the role of the single-token intervention relative to TRD (W2), and the LiveCodeBench training and evaluation split (Q1). If any point remains unclear, we would be glad to provide further clarification or additional analysis during the discussion period. If our response has resolved your concerns, we would greatly appreciate it if you could consider updating your assessment.

Thank you very much for your consideration.

Best regards,

The Authors

Thanks for the detailed response. I believe my main concerns have been addressed. Overall, I consider the paper borderline for ICLR. I lean slightly toward accepting it and have raised my rating to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.