- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Reflection-Corrected On-Policy Distillation (RC-OPD), which improves online distillation by leveraging teachers' post-hoc diagnosis of students' failed trajectories. It freezes teachers' generated error explanations and correction suggestions, and scores the original student trajectories with and without reflective context. The token-level logarithmic probability difference between the two is used to adjust the distillation signal after adjusting the diagnostic boundary, and then fused with GRPO advantages through bounded transformation. Students do not require reflective feedback or additional revisions during reasoning.
Experiments cover Qwen3-1.7B, 4B, 8B, and five mathematical and three code benchmarks. The paper reports the highest average student accuracy across all scales, and analyzes each component through reflective, localization, randomized region, and basic distillation ablation analyses.
Critical strengths
-
Performance improvements across model size and task type are evident and have practical value. In Table 1, the average mathematical accuracy for the three scales improved by 2.74, 1.99, and 3.29 percentage points relative to their respective strongest baselines; the average code score improved by 1.26, 0.86, and 2.57 percentage points, respectively, according to the values in Table 2. These results provide broad support for the effectiveness of the method, and the benefits do not require additional teacher calls or modifications to student input during deployment.
-
Ablation designs, related to the core mechanism, provide preliminary attribution evidence. In Table 3, removing reflection reduced the average accuracy from 56.26% to 54.81%; using randomized regions with the same number of affected tokens yielded 54.89%. The latter control is particularly valuable because it examines whether diagnostic region selection is superior to simply restricting the correction range. Combined with the token-level case, these results support the ability of ex-post feedback to alter the supervision of erroneous reasoning in general OPD.
Critical weaknesses
-
The empirical value of the new design compared to existing feedback conditional distillation has not yet been fully established. With alpha=1 used in all main experiments, the advantage on the selected suffix simplifies.
-
Appendix C.1 also clearly illustrates this. Therefore, the key contribution lies in freezing the external teacher, feedback screening, local application, and combination with RL. SDPO and OPCD have previously investigated the prediction of feedback conditional teachers distilled along the student's original trajectory, but these most relevant methods were not included in the main experiments. The authors do provide a valuable w/o Localization control; however, the complete method only improves upon this control by 0.45 percentage points, and the analysis is limited to a 4B mathematical setting, without reporting independent training replication or uncertainty. Therefore, current evidence is insufficient to determine whether the core distinction of local error correction brings stable and sufficiently meaningful benefits.
-
Significant gaps remain in the core process and evaluation protocol affecting verification. Section 3.3 relies on the diagnostic boundary b_i and feedback admission m_i, but Appendix B's prompts only require verdict, diagnosis, and correction, without specifying the location field; Table 6's "1,200 characters" also fails to explain how it maps to token boundaries. Specific rules for determining diagnostic effectiveness and alignment reliability, as well as a complete prompt format for reflective conditional scoring, are all inadequate. Furthermore, Section 4.1 describes the LiveCodeBench metric as pass@1 of four-sample estimation, while Appendix D defines it as the proportion of "at least one of four programs passing," i.e., pass@4. This contradiction directly impacts the interpretation of code results; these implementation gaps prevent readers from reliably reproducing the paper's most crucial selective error correction mechanism.
Questions for authors
-
Does the LiveCodeBench value in Table 2 correspond to pass@1 or pass@4? Please provide the actual aggregation formula and confirm that all methods use the same implementation. Which description in the main text and appendix needs correction?
-
Please provide the complete output format for diagnosis and localization, the mapping from character blocks to token boundaries, the feedback admission rules, and the actual prompt template for reflection conditional scoring. Is the diagnosis "justified" based on the teacher's own statement, or has it undergone additional verification? What is the actual admission rate?
-
Are the complete methods in Table 3, with/o Reflection, and with/o Localization independently trained and replicated? What is the uncertainty of their differences? This will determine whether I consider the benefits of reflection and localization to be stable.
-
Are there comparable results with SDPO/OPCD-like feedback distillation methods? How much does the training cost increase after adding diagnosis generation and additional teacher scoring, and does the advantage still exist under the same computational budget? This information will affect the judgment of the actual value of the new design.