- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper looks at on-policy self-distillation (OPSD) for mathematical reasoning. The authors argue that a teacher conditioned on the reference solution causes two problems: "reasoning mismatch," where the student borrows conclusions from the reference that it cannot derive itself, and a "distillation trap," where steps the student already got right are needlessly constrained. They propose RC-OPD. An external LLM (DeepSeek) finds the first erroneous stage in a failed student rollout and repairs it. The student then continues from the repaired prefix, for up to rounds, and the repair is accepted only if the student reaches the correct answer. For accepted repairs, the teacher is conditioned on a failure explanation for the erroneous segment and on a short derivation for the valid prefix (weighted by ). Failed repairs fall back to plain OPSD. Experiments on Qwen3-1.7B/4B/8B, evaluated on AIME24/25 and HMMT25, show higher average accuracy than OPSD and other recent distillation baselines.
Critical strengths
- The motivation is backed by measurement. Before proposing the method, the authors quantify how often reference-conditioned answers are correct only because they borrow the reference's conclusions. This turns "reasoning mismatch" into a testable problem rather than an intuition, and it is the main reason I take the paper seriously.
- Accepting a repair only when the student can continue from it is simple and falsifiable. The ablation shows that this validation step contributes the most. This suggests the gain is not just from the extra hints of an external model, but also from checking whether the student can follow the repair.
- The method is precisely specified and broadly compared. Equations define the segment-level supervision clearly, and the experiments compare against several recent distillation baselines at multiple model sizes, which makes the method easy to reimplement and position.
Critical weaknesses
- Novelty is limited, and overlap with recent work is not discussed. The core loop is: use the reference to locate and repair the first error, accept the repair if the student's continuation succeeds, and then train on it. This is very close to InT (arXiv 2601.14209), which is not cited and differs mainly in using SFT/RL instead of OPSD. Within OPSD, ROSD already localizes the first erroneous span and leaves the valid prefix alone, and the concurrent Woodpecker Distillation (arXiv 2608.05168) also patches locally, verifies the patch by continuation, and then distills. What is new is mainly the iterative multi-round repair and the separate teacher conditioning for error and prefix segments. That reads as an engineering extension of OPSD/ROSD, and it largely determines how much this paper contributes.
- The paper says little about what errors the students actually make. The whole method depends on diagnosing the student's first error, yet there is no taxonomy or distribution of the errors found, and no human check of the DeepSeek diagnoses is reported. Without this, it is hard to tell whether the gains come from fixing root causes or from something else.
- The statistical evidence for the main results is weak, and hyperparameters were tuned on the test sets. Each benchmark has only 30 problems, the margins over the best baseline are small, and Table 1 reports no variance. In addition, (Figure 6) and (Figure 7) were selected by comparing results on the test benchmarks, with no held-out split.
- The comparison is not entirely fair. Only RC-OPD gets diagnoses and repairs from an external LLM. No OPSD baseline receives the same hints, and the number of external calls and their cost are not reported. Part of the improvement may simply come from having a stronger external model in the loop.
Questions for authors
- How does RC-OPD differ from InT, Woodpecker Distillation and LEMMA? Could you add an InT-style repair-then-train baseline that uses the same diagnostic model and budget?
- What kinds of first errors does the diagnosis find? Could you report their distribution by benchmark and model size, the fraction of diagnoses checked by humans (with agreement)?
- Could you report per-seed results (or mean ± std) for RC-OPD, OPSD and PW-OPSD in Table 1, with a paired per-problem bootstrap test? Do the conclusions hold if and are fixed in advance or chosen on a held-out split?
- Could you add an OPSD baseline whose teacher also receives the external model's diagnosis or repair text, but without continuation-based validation or segment-specific conditioning? How many external LLM calls does RC-OPD need per training problem?
5 comments
Response to Reviewer rFbm (Part 1)
Response to Q1 & W1: Comparison with concurrent and related work
We sincerely thank the reviewer for drawing our attention to these concurrent works. We appreciate your concerns and would also like to clarify a few potential misunderstandings. InT, Woodpecker Distillation, and LEMMA all use reflection on errors to construct training signals. We agree that this idea is widely accepted and was not introduced by RC-OPD. Nevertheless, RC-OPD differs from these methods in several significant ways:
- RC-OPD converts local repairs validated through student continuation into distributional supervision on the student’s own trajectories. Compared with SFT in InT and SFT on corrected trajectories in LEMMA, RC-OPD conveys corrective information through the full teacher distribution.
- RC-OPD constructs separate teacher conditioning for erroneous segments and valid prefixes. This effective mechanism also distinguishes it from Woodpecker Distillation.
In summary,our distinctive contribution is an effective on-policy supervision approach that combines targeted error correction with the preservation of valid reasoning, while using multiple rounds of diagnosis, repair, and continuation to address subsequently exposed errors.
Following your suggestion, we have also added an InT-style repair-then-train experiment. The results are shown below and have been included in Appendix A.3 (Table 6) of the revised manuscript.
| Method | AIME24 | AIME25 | HMMT25 | Mean |
|---|---|---|---|---|
| Initial Student | 71.67 | 67.50 | 41.67 | 60.28 |
| InT-style repair-then-train | 71.67 | 66.67 | 46.67 | 61.67 |
| RC-OPD | 78.33 | 69.17 | 50.83 | 66.11 |
Response to Q2 & W2: Error categories and agreement with manual verification
Thank you for your valuable suggestion. We manually reviewed the diagnosis records. Specifically, we took the first 500 training examples, collected the diagnosis records generated for these examples, and checked earliest-error localization and anchor repair in the approximately 300 resulting records. As shown below, both earliest-error localization and anchor repair show strong agreement with manual verification. We have added the table to Appendix B.1 (Table 7) of the revised manuscript.
| Model Size | Diagnoses | Correct Earliest-Error Localization | Correct Anchor Repair |
|---|---|---|---|
| 1.7B | 329 | 263 / 329, 79.94% | 307 / 329, 93.31% |
| 4B | 290 | 251 / 290, 86.55% | 266 / 290, 91.72% |
| 8B | 261 | 229 / 261, 87.74% | 246 / 261, 94.25% |
We also provide a more detailed breakdown of the models’ error types below and in Appendix B.1 (Table 8).
| Model Size | Arithmetic or algebraic errors | Problem misinterpretation, overlooked conditions, or incorrect modeling | Misapplication of theorems, formulas, or concepts | Double counting, omitted cases, or incomplete case analysis | Unsupported conclusions | Final answer formulation or reporting errors | Total |
|---|---|---|---|---|---|---|---|
| 1.7B | 81 (26.21%) | 136 (44.01%) | 52 (16.83%) | 21 (6.80%) | 17 (5.50%) | 2 (0.65%) | 309 (100%) |
| 4B | 87 (33.33%) | 87 (33.33%) | 53 (20.31%) | 15 (5.75%) | 16 (6.13%) | 3 (1.15%) | 261 (100%) |
| 8B | 77 (33.48%) | 78 (33.91%) | 38 (16.52%) | 21 (9.13%) | 16 (6.96%) | 0 (0.00%) | 230 (100%) |
| Total | 245 (30.63%) | 301 (37.63%) | 143 (17.88%) | 57 (7.13%) | 49 (6.13%) | 5 (0.63%) | 800 (100%) |
Response to Reviewer rFbm (Part 2)
Response to Q3 & W3: Multi-seed results and bootstrap tests
Thank you for your suggestion regarding the completeness of the experimental reporting. Table 1 in original manuscript already reports averages over multiple seeds. For completeness, we collected the results for three seeds from the experimental records and report them as mean ± standard deviation below and in Appendix A.2 (Table 3).
| Method | Qwen3-1.7B | Qwen3-4B | Qwen3-8B | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| AIME2024 | AIME2025 | HMMT2025 | Mean | AIME2024 | AIME2025 | HMMT2025 | Mean | AIME2024 | AIME2025 | HMMT2025 | Mean | |
| OPSD | 54.17 ± 2.20 | 39.17 ± 1.67 | 27.50 ± 0.83 | 40.28 ± 0.73 | 75.83 ± 2.20 | 65.00 ± 0.83 | 46.67 ± 3.00 | 62.50 ± 1.27 | 78.33 ± 1.67 | 64.17 ± 3.00 | 47.50 ± 1.44 | 63.33 ± 1.21 |
| PW-OPSD | 55.00 ± 2.50 | 41.67 ± 1.44 | 30.00 ± 2.20 | 42.22 ± 1.55 | 76.67 ± 0.83 | 68.33 ± 3.00 | 49.17 ± 1.67 | 64.72 ± 0.56 | 80.83 ± 2.20 | 70.83 ± 2.89 | 44.17 ± 3.63 | 65.28 ± 1.82 |
| RC-OPD | 54.17 ± 3.00 | 47.50 ± 1.44 | 30.83 ± 2.20 | 44.17 ± 1.00 | 78.33 ± 1.67 | 69.17 ± 3.63 | 50.83 ± 0.83 | 66.11 ± 0.83 | 78.33 ± 2.50 | 76.67 ± 2.89 | 45.83 ± 3.33 | 66.94 ± 2.17 |
We further conducted bootstrap tests using the pooled data from multiple seeds. The results below show that RC-OPD significantly outperforms the baselines. We have included these results in Appendix A.2 (Table 4) of the revised manuscript and highlighted them in blue.
| Comparison | Qwen3-1.7B | │ | Qwen3-4B | │ | Qwen3-8B | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Δ (pp) | 95% CI | p | │ | Δ (pp) | 95% CI | p | │ | Δ (pp) | 95% CI | p | |
| RC-OPD vs OPSD | +3.89 | [0.33, 7.48] | 0.0304 | │ | +3.61 | [0.17, 7.05] | 0.0400 | │ | +3.61 | [0.04, 7.16] | 0.0472 |
| RC-OPD vs PW-OPSD | +1.94 | [0.65, 3.24] | 0.0062 | │ | +1.39 | [0.09, 2.78] | 0.0466 | │ | +1.67 | [0.19, 3.15] | 0.0324 |
Δ (pp): RC-OPD − baseline in percentage points; CI: 95% percentile confidence interval; p: two-sided equal-tailed bootstrap p-value.
Response to Q4 & W4: Additional baseline and external-model call statistics
Thank you for your suggestion. We understand your concern about the contribution of external-model diagnosis. Following your suggestions, we added an OPSD + Hindsight baseline, in which the diagnosis and repaired text are supplied as hindsight (Appendix A.3, Table 6). As shown below, directly adding this information to OPSD does not yield a significant gain. Under the OPSD setting, the diagnosis and repaired text for the erroneous segment also affect tokens in the other, correct portions of the trajectory. This finding is consistent with the RC-OPD design of using separate teacher conditioning for correct and erroneous segments.
| Method | AIME24 | AIME25 | HMMT25 | Mean Avg@4 |
|---|---|---|---|---|
| OPSD | 75.83 | 65.00 | 46.67 | 62.50 |
| OPSD + Hindsight | 71.67 | 71.67 | 45.00 | 62.78 |
We also appreciate your concern about the number of external-model calls. We summarized the experimental records as follows and included these statistics in Appendix B.8 (Table 10):
| Model | Calls (Diagnosis/Repair) per Sample |
|---|---|
| 1.7B | 0.883 |
| 4B | 0.701 |
| 8B | 0.431 |
Response to Reviewer rFbm (Part 0)
We sincerely thank the reviewer for their careful reading, valuable time, and positive evaluation of our quantitative motivation, continuation-based repair validation, precise method formulation, and broad experimental comparisons. We have carefully incorporated the reviewer’s helpful suggestions by adding an InT-style repair-then-train comparison, an OPSD + Hindsight baseline, diagnostic verification and error-type statistics, fuller multi-seed and bootstrap reporting, and external-model call statistics. We also clarify how RC-OPD differs from the related approaches and how the existing experimental results were obtained. We therefore provide targeted clarifications and additional explanations below.