acceptodds
Under review as a conference paper at ICLR 2027

Teaching with Hindsight: Reflection-Induced Correction for On-Policy Distillation

Abstract

On-policy distillation (OPD) improves language-model reasoning by transferring teacher knowledge on student-generated trajectories, aligning supervision with the reasoning patterns and errors encountered under the student’s own policy. However, standard OPD relies on prefix-conditioned teacher predictions without explicitly incorporating the teacher’s retrospective diagnosis of why a completed reasoning attempt failed. To address this gap, we propose Reflection-Corrected On-Policy Distillation (RC-OPD), which translates the teacher’s reasoning about student failures into corrective supervision on the student’s original trajectories. For verifier-identified failures, a frozen teacher generates reflective feedback and evaluates the same response under standard and reflection-conditioned contexts. Token-level log-probability differences quantify the changes in teacher assessment induced by reflection, making its additional guidance explicit relative to the teacher’s existing predictions. RC-OPD selectively incorporates these adjustments into base OPD over diagnosed reasoning suffixes and bounds the combined distillation signal before integrating it with outcome-based reinforcement learning. This allows retrospective diagnosis to refine predictive supervision while retaining standard guidance elsewhere. The student is optimized solely on the original problem and its sampled response, requiring neither diagnostic feedback nor additional reflection at inference. Experiments on Qwen3 models across mathematical reasoning and code generation show that RC-OPD consistently outperforms strong reinforcement-learning and distillation baselines across all tested scales, demonstrating the value of retrospective diagnosis as a complement to predictive teacher supervision. Code will be released upon acceptance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
On-Policy Distillation, Knowledge Transfer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_d1aTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reflection-Corrected On-Policy Distillation (RC-OPD), which improves online distillation by leveraging teachers' post-hoc diagnosis of students' failed trajectories. It freezes teachers' generated error explanations and correction suggestions, and scores the original student trajectories with and without reflective context. The token-level logarithmic probability difference between the two is used to adjust the distillation signal after adjusting the diagnostic boundary, and then fused with GRPO advantages through bounded transformation. Students do not require reflective feedback or additional revisions during reasoning.

Experiments cover Qwen3-1.7B, 4B, 8B, and five mathematical and three code benchmarks. The paper reports the highest average student accuracy across all scales, and analyzes each component through reflective, localization, randomized region, and basic distillation ablation analyses.

Critical strengths
  1. Performance improvements across model size and task type are evident and have practical value. In Table 1, the average mathematical accuracy for the three scales improved by 2.74, 1.99, and 3.29 percentage points relative to their respective strongest baselines; the average code score improved by 1.26, 0.86, and 2.57 percentage points, respectively, according to the values ​​in Table 2. These results provide broad support for the effectiveness of the method, and the benefits do not require additional teacher calls or modifications to student input during deployment.

  2. Ablation designs, related to the core mechanism, provide preliminary attribution evidence. In Table 3, removing reflection reduced the average accuracy from 56.26% to 54.81%; using randomized regions with the same number of affected tokens yielded 54.89%. The latter control is particularly valuable because it examines whether diagnostic region selection is superior to simply restricting the correction range. Combined with the token-level case, these results support the ability of ex-post feedback to alter the supervision of erroneous reasoning in general OPD.

Critical weaknesses
  1. The empirical value of the new design compared to existing feedback conditional distillation has not yet been fully established. With alpha=1 used in all main experiments, the advantage on the selected suffix simplifies.

  2. Appendix C.1 also clearly illustrates this. Therefore, the key contribution lies in freezing the external teacher, feedback screening, local application, and combination with RL. SDPO and OPCD have previously investigated the prediction of feedback conditional teachers distilled along the student's original trajectory, but these most relevant methods were not included in the main experiments. The authors do provide a valuable w/o Localization control; however, the complete method only improves upon this control by 0.45 percentage points, and the analysis is limited to a 4B mathematical setting, without reporting independent training replication or uncertainty. Therefore, current evidence is insufficient to determine whether the core distinction of local error correction brings stable and sufficiently meaningful benefits.

  3. Significant gaps remain in the core process and evaluation protocol affecting verification. Section 3.3 relies on the diagnostic boundary b_i and feedback admission m_i, but Appendix B's prompts only require verdict, diagnosis, and correction, without specifying the location field; Table 6's "1,200 characters" also fails to explain how it maps to token boundaries. Specific rules for determining diagnostic effectiveness and alignment reliability, as well as a complete prompt format for reflective conditional scoring, are all inadequate. Furthermore, Section 4.1 describes the LiveCodeBench metric as pass@1 of four-sample estimation, while Appendix D defines it as the proportion of "at least one of four programs passing," i.e., pass@4. This contradiction directly impacts the interpretation of code results; these implementation gaps prevent readers from reliably reproducing the paper's most crucial selective error correction mechanism.

Questions for authors
  1. Does the LiveCodeBench value in Table 2 correspond to pass@1 or pass@4? Please provide the actual aggregation formula and confirm that all methods use the same implementation. Which description in the main text and appendix needs correction?

  2. Please provide the complete output format for diagnosis and localization, the mapping from character blocks to token boundaries, the feedback admission rules, and the actual prompt template for reflection conditional scoring. Is the diagnosis "justified" based on the teacher's own statement, or has it undergone additional verification? What is the actual admission rate?

  3. Are the complete methods in Table 3, with/o Reflection, and with/o Localization independently trained and replicated? What is the uncertainty of their differences? This will determine whether I consider the benefits of reflection and localization to be stable.

  4. Are there comparable results with SDPO/OPCD-like feedback distillation methods? How much does the training cost increase after adding diagnosis generation and additional teacher scoring, and does the advantage still exist under the same computational budget? This information will affect the judgment of the actual value of the new design.

Reviewer_FVSNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

They propose reflection-induced correction for on-policy distillation, which provides reflection supervision over responses that are rejected by the verifier. RC-OPD lets the teacher model to diagnose student responses and then scores the student’s original response with and without the diagnosis. Such reward is applied from an estimated error boundary and combined with ordinary on-policy distillation and GRPO rewards. They report highest performance on math and code benchmarks at different model scales of qwen3 model series.

Critical strengths

RC-OPD turns a teacher’s diagnosis into token-level guidance on the student’s own response, offering dense supervision over verifier rewards. They report highest performance on math and code benchmarks at different model scales of qwen3 model series.

Critical weaknesses

I’m not fully convinced by the paper of their novelty over existing methods and I believe the technical contribution over feedback-conditioned on-policy distillation needs stronger empircal support. At alpha=1, RC-OPD’s residual simply replaces the standard teacher score with the reflection-conditioned score on the selected suffix. Table 3 isolates reflection and suffix selection on 4B math, but the paper does not compare directly with prior methods or show whether the component gains hold across its other model sizes and code tasks. The paper does not measure whether diagnoses identify real unresolved errors or whether the selected boundaries mark those errors. It is therefore unclear how much of the gain comes from accurate correction rather than simply conditioning the teacher on additional text.

Questions for authors
  1. The main text describes LiveCodeBench as empirical pass@1, while the appendix counts a problem correct if any of four sampled programs passes, which is pass@4. These discrepancies need clarification.
  2. Do Tables 1–2 report final checkpoints or selected checkpoints? How is the evaluated checkpoint gets selected?
  3. Table 3 isolates reflection and suffix selection on 4B math. Do these gains persist across code generation, the other student sizes?
  4. The “w/o Localization” ablation is still localization-based. How does a boundary-free variant would perform accordingly? This will help clarify whether the boundary pipeline adds value beyond the suffix mask.
Reviewer_auedOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reflection-Corrected On-Policy Distillation (RC-OPD), which improves on-policy distillation by exploiting a frozen teacher's retrospective diagnosis of student failure trajectories.

A frozen teacher diagnoses failed student responses and uses reflection-conditioned scoring to refine OPD supervision on the diagnosed suffix. The corrected signal is tanh-bounded and combined with the GRPO advantage. Inference requires neither the teacher nor reflection.

Across all three student sizes, RC-OPD achieves the highest average scores among the reported baselines on both mathematical reasoning and code generation.

Critical strengths
  1. The method has a clear and practical training design. Reflection modifies only teacher-side supervision, while the student learns from the original problems and trajectories without additional diagnostic steps at inference. Retaining base OPD guidance and bounding the teacher’s contribution also makes the method easy to combine with outcome-based RL, supporting its practical applicability.

  2. The evaluation has reasonable breadth and targeted ablations. Tables 1–2 cover two task types and three model sizes, with improvements in average performance throughout. Table 3 includes a random-span control with a matched token count, which better isolates the role of region selection than simply removing localization. These results support further investigation of the method.

Critical weaknesses
  1. The incremental benefit over closely related feedback-conditioned distillation methods remains insufficiently established. At the default alpha = 1, the corrected advantage on the selected suffix reduces to the reflection-conditioned teacher log-ratio, as acknowledged in Eq. 13. The contribution therefore rests on how diagnostic feedback, selective application, and bounded fusion are combined. Closely related methods such as SDPO and OPCD also distill feedback-conditioned predictions, but are not evaluated directly. Although Table 3 provides useful evidence for reflection and localization, it does not establish an advantage over these related approaches under comparable conditions. A controlled comparison would clarify the value of the proposed combination.

  2. The robustness of the component gains and key implementation details are insufficiently documented. Table 3 reports gains of 1.45 percentage points from reflection and 0.45 points from localization, without uncertainty estimates or reported variation across training seeds.

Questions for authors

How does RC-OPD compare with a feedback-conditioned distillation baseline using the same teacher, feedback, and training budget? Such a comparison would help clarify its incremental benefit over the feedback-conditioned distillation approaches represented by SDPO and OPCD.

How many independent training runs underlie Table 3? Can you report variation across training seeds or uncertainty estimates for the reflection and localization gains to establish whether these improvements are consistent?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.