acceptodds
Under review as a conference paper at ICLR 2027

On-Policy Distillation of Language Models by Repairing the Future

Abstract

On-policy distillation (OPD) supervises student-generated trajectories, but local token corrections do not guarantee better final outcomes because they can redirect the entire autoregressive continuation. We propose REPAIR-OPD, which identifies candidate repair points from teacher confidence and teacher–student disagreement, applies only a local teacher correction, and lets the student regenerate the remaining trajectory. Only Judge-verified wrong-to-correct repairs are used for learning. Across five mathematical-reasoning and two out-of-domain benchmarks, REPAIR-OPD achieves an overall mean of 54.55%, outperforming the strongest OPD baseline by 4.72%, and remains effective across smaller Qwen and cross-family Llama settings. In a matched one-round efficiency test on 1,000 source questions, REPAIR-OPD requires 1.36× OPD's end-to-end training time, while its seven-benchmark mean response length reaches 1.69× that of OPD, indicating a controlled cost–performance trade-off. Our results suggest that effective OPD should optimize not only local corrections, but the future trajectories they induce. Code is available at [the anonymous repository](https://anonymous.4open.science/r/repair-opd-code-2027-021D/README.md).

Primary area:
reinforcement learning
Keywords:
large language models, on-policy distillation, post-training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nhM3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

REPAIR-OPD learns from successful repairs of student-generated responses. A teacher proposes a local token replacement, the student regenerates the suffix, and a bounded search retains responses accepted by a Judge. These repairs are paired with the original failures for BCE training, with an additional unlikelihood penalty against repetition. Experiments show improvements mainly on mathematical reasoning, with further tests on smaller Qwen and Llama models.

Critical strengths
  1. The method changes which trajectories are collected for training, rather than only selecting supervision positions on an existing rollout. Combining local teacher guidance, student continuation, and outcome verification gives a concrete and potentially useful training procedure. (§3, pp. 3–5)

  2. The main results include a meaningful matched-seed comparison. REPAIR-OPD improves the overall score over TA-OPD in all three seeds, with respective means of 54.55% and 49.02%. The smaller-Qwen and Llama results provide some additional support for the method’s applicability. (Tables 1–4, pp. 6–7)

Critical weaknesses
  1. The experiments do not isolate the value of the teacher-selected intervention. Repair also introduces additional sampling, Judge filtering, and a different training objective. Without matched resampling controls, it is unclear how much the local replacement contributes beyond these other changes. The large UL ablation effect also makes comparisons with baseline-plus-UL important. The results support the complete procedure, but provide weaker evidence for the proposed explanation of its gains. (§3.3, p. 4; Figures 3–4, pp. 7–9)

  2. Some evidence central to the repair process needs clarification. The Judge’s prompt, acceptance criteria, and calibration are not sufficiently documented. Separately, Appendix D.1 contains a confirmed presentation inconsistency: the displayed response changes before the marked intervention, although Equation (4) requires that prefix to remain unchanged. This could reflect editing of the example; it does not by itself establish an implementation error. Raw traces are needed to resolve the discrepancy. (Equation (4), p. 4; Appendix D.1, pp. 16–17)

Questions for authors
  1. Can you compare teacher-token replacement with retaining the original token and resampling the suffix at the same position, using the same generation budget, Judge, BCE objective, and UL penalty? A budget-matched full-response resampling comparison would also help establish the value of localized repair.

  2. Please provide the Judge’s prompt, acceptance rule, and any calibration results. How are the “answer correct, reasoning invalid” cases in Figure 7 handled? Please also specify the GPQA-D evaluation denominator and aggregation procedure, and β in Equation (6). (pp. 4, 12–14)

  3. Is Appendix D.1 an unedited generation trace? Please provide the original token sequence and intervention index, or identify any presentation edits that explain the changed prefix. (pp. 16–17)

3 comments
AuthorsOpenReview ↗

Thank you & Looking forward to further discussion

Dear Reviewer nhM3,

Thank you again for reviewing our paper and for asking about the resampling controls, the Judge, and the repair trace.

  1. Resampling controls and baselines with UL (Weakness 1 and Q1). REPAIR-OPD scores 54.76% overall, compared with 47.87% for resampling the suffix with the original token retained and 47.26% for full-response resampling. We also report OPD and TA-OPD with UL.
  2. Judge verification (Weakness 2 and Q2). We provided the Judge’s prompt and acceptance rule, including how it handles correct answers with invalid reasoning. The Judge agreed with human assessments on 42 of 50 reviewed responses (84%). We also specified the GPQA-D denominator and β in Equation (6).
  3. Appendix D.1 (Weakness 2 and Q3). We have corrected this error in the revised PDF. We have also included the unedited trace for Case 1 in the supplementary materials.

The full results and clarifications are in our rebuttal. We would appreciate hearing whether they address your concerns and would be happy to discuss any remaining questions.

Warm regards,
Authors

Reviewer_nhM3OpenReview ↗

Thanks for your responses

Thank you for the additional experiments and clarifications. The additional resampling controls and OPD/TA-OPD + UL comparisons substantially strengthen the empirical evidence. The clarifications on the Judge protocol and evaluation also address my earlier concerns. I therefore raise my final rating from 2 to 3.

We really appreciate your thoughtful comments, which helped us identify issues in our paper and improve our work. Thank you again for your time and valuable suggestions!

Reviewer_1jGpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a new method Repair-OPD, a distillation method that converts local teacher corrections into outcome-verified preference pairs. Starting from an incorrect student rollout, it selects candidate repair positions using teacher–student disagreement, inserts a teacher-proposed replacement, and lets the student regenerate the remaining suffix. This study demonstrates the high performance of the proposed method through various experimental results.

Critical strengths
  1. Rather than simply substituting the token proposed by the teacher, the proposed method verifies whether the student’s regenerated trajectory after the correction actually reaches the correct answer. Grounding distillation in the final outcome, rather than in myopic token-level corrections, is intuitively appealing for performance gains. Moreover, the proposed method is also simple and easy to adopt, which gives a practical advantage.

  2. The method shows large performance improvements across a range of mathematical reasoning settings. Also, this improvement is verified through experiments spanning multiple model architectures, results over various seeds, and an extensive ablation study, all of which make the behavior of the proposed method easy to understand and assess.

Critical weaknesses
  1. The novelty of the proposed method over the previous work is not clearly established. TRD [1] and its follow-up works already advocates correcting trajectories before distillation, while PAD [2] and CDSL [3] explore related approaches that consider downstream utility or lookahead rewards. REPAIR-OPD differs through local teacher interventions, student suffix regeneration, and learning from verified wrong-to-correct pairs. However, its additional value over teacher-generated trajectory refinement is not sufficiently verified. Additional performance comparison under the same conditions (information access, computation cost, training objectives etc.) would help distinguish and clarify the contribution of the proposed method.

  2. There has been insufficient validation of the selection criterion of the repair point. This manuscript’s core claim is that a token should be evaluated by the future it induces. However, the position at which a token is corrected is not selected on how the correction changes the downstream success of student’s autoregressive future. Instead, it is chosen via a heuristic score of teacher–student disagreement at that token. I think this point is slightly different with the paper’s core motivation. Moreover, since the generation process, which is stochastic, may recover a correct trajectory even without changing the selected token, the experiments do not demonstrate effective intervention from additional sampling. I could not find either a theoretical guarantee or an empirical analysis establishing that the positions selected by the proposed disagreement metric are indeed those where intervention yields a substantial gain in future. As a result, in my opinion, the link between the selection criterion and the claimed effect to be insufficiently verified.

  3. The current experimental results do not sufficiently exhibit that the performance improvement stems from the core part of proposed method. The ablation studies show the substantial improvements from teacher answer access (+6.9%p) and UL penalty (+4.8%p). In particular, the variant without answer access performs on par with the TA-OPD baseline. Therefore, these results imply that a large portion of the improvement may be attributable to these additional components rather than to the proposed method itself. Furthermore, it remains unclear whether the advantage persists when OPD and TA-OPD utilize the same additional components. Therefore, I think it has limited attribution of the overall improvement specifically to the proposed repair procedure.

<Minor>
  • Lines 138–139 It looks like * should be replaced with a subscript.

[1] Trajectory-Refined Distillation

[2] Reject Only Critical Tokens: Pivot-Aware Speculative Decoding

[3] Constrained Decoding with Speculative Lookaheads

Questions for authors
  1. This manuscript adopts pass@8 as its primary metric. Since the proposed method also increases generation length (Figure 5b), I would like to see the corresponding pass@1 performance comparisons.

  2. The current ablation study is subtractive: removing one component from the full method. I am also curious about an additive ablation that starts from the vanilla method and adds each component.

  3. Furthermore, for the additional components that contributes the performance improvement, it would be necessary to report the performance of the baselines when equipped with the same components.

5 comments
AuthorsOpenReview ↗

Thank you & Looking forward to further discussion

Dear Reviewer 1jGp,

Thank you again for reviewing our paper and for raising questions about the repair procedure and its contribution.

  1. Teacher-generated trajectory refinement (Weakness 1). We added a comparison with TRD-style refinement using BCE and UL. REPAIR-OPD (seed 7) achieves an overall mean of 54.76%, compared with 53.26% for the TRD-style baseline.
  2. Selection of the repair point (Weakness 2). We compared teacher-token replacement with suffix resampling without changing the selected token and with full-response resampling. We also report repair success for disagreement-based and random position selection.
  3. Additional components and evaluations (Weakness 3 and Q1–Q3). Teacher answer access is shared in the main comparison, and REPAIR-OPD retains a higher overall mean than OPD and TA-OPD with UL. We also added pass@1 comparisons and an additive ablation starting from vanilla OPD.

The detailed results are in our rebuttal. We would appreciate hearing whether these comparisons address your questions and would be happy to discuss any points that need further clarification.

Warm regards,
Authors

Thank you again for your valuable feedback. We have further strengthened our comparisons and polished the writing and several details in the paper. We would greatly appreciate hearing your thoughts if you have a chance!

Dear Reviewer 1jGp,

With only two days remaining in the ICLR 2027 discussion period, we would greatly appreciate it if you could share your thoughts on our rebuttal. We would be happy to address any remaining concerns.

Thank you again for your time and consideration!

Best regards, Authors

Reviewer_dtRdOpenReview ↗
Rating
2(−1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies On-Policy Distillation (OPD) for language models, focusing on a limitation of existing methods: local token-level corrections from a teacher do not necessarily lead to better final outcomes. The authors propose REPAIR-OPD, which uses teacher-guided token corrections, lets the student regenerate the remaining trajectory, and retains only outcome-verified successful repairs for preference learning. Experiments across mathematical reasoning and out-of-domain benchmarks show improvements over several OPD baselines.

Critical strengths

The paper identifies an important limitation of token-level OPD: local teacher preferences may not align with final task success. The proposed trajectory-repair framework is intuitive and provides a practical way to incorporate outcome feedback into distillation.

The experiments cover several relevant OPD baselines, different model sizes and families, and multiple benchmarks. The authors also report training and inference costs, which helps understand the performance-efficiency trade-off.

Critical weaknesses
  1. Insufficient ablations and fair comparisons to support the core contribution.
    The main claim is that repairing and regenerating future trajectories improves OPD. However, REPAIR-OPD introduces several changes simultaneously, including teacher-guided token selection, additional student sampling, outcome-based filtering, preference optimization, and unlikelihood regularization. It is unclear which components are responsible for the gains.

    More importantly, the paper lacks compute-matched comparisons with simpler outcome-based training approaches, such as Best-of-N rejection sampling followed by SFT, or outcome-based DPO/BCE without repair. These baselines should use the same Judge, training objective, and computational budget. Without such comparisons, it is difficult to determine whether the gains come from the proposed repair mechanism or simply from additional sampling and outcome-based supervision.

  2. Limited evidence of out-of-domain generalization.
    Although the paper emphasizes transfer beyond mathematical reasoning, the OOD results are not particularly convincing. The OOD mean slightly decreases from 49.51% (TA-OPD) to 49.34% (REPAIR-OPD), while GPQA Diamond drops from 40.78% to 38.09%. Most improvements appear to come from mathematical reasoning benchmarks. This raises concerns about how well the proposed method generalizes beyond its training domain.

Questions for authors

Please see weakness

12 comments
AuthorsOpenReview ↗

Thank you & Looking forward to further discussion

Dear Reviewer dtRd,

Thank you again for reviewing our paper and for your questions about the ablations and out-of-domain results.

  1. Ablations and fair comparisons (Weakness 1). Removing teacher-token replacement or outcome-based filtering causes the largest overall drops among the component ablations. REPAIR-OPD scores 7.50 percentage points above outcome-based BCE without repair and 4.29 points above Best-of-N followed by SFT.
  2. Out-of-domain generalization (Weakness 2). We added MBPP+ to evaluate code generation. REPAIR-OPD still trails TA-OPD on GPQA-D in the main setting. Its OOD mean exceeds TA-OPD’s by 1.90 and 5.20 percentage points in the smaller Qwen and Llama settings, respectively.

The full comparisons are in our rebuttal. We would appreciate hearing whether these results address your concerns and would be happy to discuss any remaining questions.

Warm regards,
Authors

We appreciate your thoughtful feedback, which has helped us improve our paper. We have since refined the writing and strengthened the experimental comparisons. We would be glad to hear any further comments you may have and hope our updates have addressed your concerns!

Dear Reviewer dtRd,

With only two days remaining in the ICLR 2027 discussion period, we would greatly appreciate it if you could share your thoughts on our rebuttal. We would be happy to address any remaining concerns.

Thank you again for your time and consideration!

Best regards, Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

ICLR 2027 — Urgent Concern: Reviewer Non-Response to Author Rebuttal

Dear Area Chair and Reviewers,

As explicitly stated in the ICLR 2027 Area Chair Guidelines, "Responding to rebuttal comments is a core part of the reviewer’s responsibilities."

We have carefully addressed the reviewers' concerns in our rebuttal and provided detailed clarifications and additional evidence. However, we have not yet received follow-up responses to our rebuttal.

As the discussion period is approaching its deadline, we would sincerely appreciate it if the reviewers could revisit our responses and let us know whether their concerns have been adequately addressed or if any further clarification is needed.

We would also be grateful if the Area Chair could help facilitate this discussion, as encouraged by the official ICLR guidelines.

We believe that constructive discussion is essential for a fair and thorough evaluation of our work, and we sincerely hope that our rebuttal and additional evidence can be fully considered in the final assessment.

Thank you very much for your time and consideration.

Warm regards, The Authors

Dear reviewer dtRd and 1jGp,

This is a kind reminder that the time you can discuss with authors is closing. Please check the rebuttals and discuss as needed. It would be really helpful if you acknowledge them.

Thanks for your effort, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.