acceptodds
Under review as a conference paper at ICLR 2027

Learning to Revise Reasoning with Segment-wise On-Policy Distillation

Abstract

On-policy distillation (OPD) improves large language model reasoning by training students on their own rollouts with dense token-wise supervision from the teacher. However, token-wise OPD does not explicitly provide a coherent alternative reasoning step showing how the student’s step could be revised to improve subsequent reasoning. Furthermore, this paradigm can become less effective when the student produces a degenerate reasoning prefix, as subsequent teacher supervision remains conditioned on that prefix and may reinforce poor reasoning patterns. In this work, we focus on learning reasoning revision with segment-wise OPD to rework intermediate reasoning steps and better support subsequent reasoning. Through controlled reasoning interventions, we find that replacing student segments with teacher redrafts improves subsequent reasoning accuracy. Therefore, we address the problem of turning teacher redrafts into explicit supervision for learning to revise reasoning. We propose Segment-wise On-Policy Distillation (Seg-OPD), which selects student segments based on an uncertainty metric and obtains corresponding teacher redrafts. Seg-OPD trains the student to prefer teacher redrafts over their paired student segments while retaining dense token-wise OPD supervision. Extensive experiments on mathematical reasoning and competitive programming tasks show that Seg-OPD-trained students achieve higher revision success rates than baselines. Seg-OPD consistently outperforms the compared state-of-the-art baselines in reasoning accuracy with an average relative improvement of 5.22% across diverse models and tasks. Code is available at https://anonymous.4open.science/r/Seg-OPD.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, On-Policy Distillation, Post-Training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7fvjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies why on-policy self-distillation (OPSD), in which a model is distilled from a copy of itself that sees a privileged context (answer, worked solution, or its own rollout), sometimes causes “thinking collapse”: reasoning becomes much shorter and accuracy drops. The authors frame OPSD as pushing the model from controlled toward automatic processing, and use V-usable information (PVI) as a common measure. Experiments on Qwen3-8B and Gemma-4-E4B show that only long, self-generated contexts (RC, RU) cause collapse, which worsens with task difficulty (e.g., ZebraLogic XL, Qwen: 39% → ~5%) and carries over out of domain. A synthetic shortest-path task separates information density from context length, and PVI predicts both accuracy and reasoning shortening. The paper also finds that students trained with richer contexts follow malicious in-context content more readily.

Critical strengths
  1. Controlled design. Data, recipe, number of steps and decoding are shared, and only c varies, across two model families and in-/out-of-domain tasks. ZebraLogic’s difficulty tiers provide a useful difficulty axis.

  2. The paper moves from “collapse happens” to when and why. It shows that shortening alone is not harmful (MBPP+, ZebraLogic Small, and the length-matched AO+R/R+AO contexts that improve accuracy).

  3. The synthetic shortest-path task varies information density (I_BFS) and context length (random-token padding) separately, with a verified unique shortest path. PVI then unifies the two axes as a predictor of accuracy and length reduction, and the </think>-probability analysis offers a plausible account of why padding shortens reasoning.

Critical weaknesses
  1. The claim that only self-generated contexts cause collapse is confounded with length and style: RC/RU average ~5,500 tokens versus ~680 for FS, and there is no short self-generated context (e.g., truncated rollouts) or long human-written context. The AO+R/R+AO controls help but contain only the answer plus random tokens. RC and RU also behave almost identically, so correctness filtering appears not to matter, which should be stated. The paper does not separate premature convergence of the reasoning chain from imitation of the teacher’s style.

  2. The ZebraLogic tiers show a monotone trend in both models, but no criterion predicts when collapse will be harmful (e.g., a regression on base accuracy or base reasoning length across tasks). The PVI relationship is established only for the teacher on the synthetic task (Qwen3-8B, 25 graphs) and is not shown to predict student collapse on natural tasks. The r=0.99 between teacher and student length rests on only a few context conditions.

  3. Each configuration appears to be a single training run (seed 42, one epoch), and the CIs come from bootstrapping over problems or prompts, so they do not capture training variance; AIME has only 30 problems per year. In the seen/unseen comparison, the unseen set is selected from problems BASE failed on, whereas the seen set spans difficulty bins, so longer reasoning on unseen problems may reflect difficulty rather than adaptation.

Questions for authors
  1. What is the final teacher–student JSD for each context type, and how does it evolve during training? Is the stronger imprint of RC/RU associated with a smaller gap because the context is on-policy?

  2. The reasoning-PVI curves are aggregated over live rollouts only. Do the RC/RU curves differ between correct and incorrect rollouts, and are the late parts of the curves driven by a subset of long rollouts?

  3. Would a short self-generated context, or a long non-self-generated natural-language context, still trigger (or avoid) collapse?

Reviewer_563VOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Seg-OPD selects uncertain sentences from failed student rollouts. A fixed teacher rewrites a selected step and continues to a complete solution. If that solution passes a correctness check, the first sentence becomes the preferred replacement for the original step. Training combines this preference with token-wise distillation outside the selected segments. The experiments cover four student sizes on math and programming tasks. The student runs alone at evaluation.

Critical strengths

Alternatives start from the same prefix, here is the step the student generated, and here is a replacement for it. The intervention comparison with random anchors support choosing uncertain steps. The judge looks at retained sentence without teacher continuation, since sentence is all student gets as a preference target.

Critical weaknesses

It reaches 73.29 on MATH500, well below plain OPD at 90.71. Replacing the preference loss with SFT changes gradient scale too: preference loss uses beta = 0.01, while SFT has a different scale under same outer weight.

There are exceptions to the gains, including Qwen3-4B on MATH500 at 89.10 versus 90.71 for OPD. For behavioral claims, both revision success and local quality audit use LLM judgments.

Questions for authors
  1. After matching gradient scales and tuning SFT weight with same validation protocol, does preference training still do better?
  2. How much do main gains vary across training seeds? Could a small human-labeled sample check revision-success judgments?
7 comments
AuthorsOpenReview ↗

Response Part 1

We truly appreciate the reviewer's valuable feedback and the recognition of our work. We hope the following discussion can address the reviewer's remaining concerns.


"It reaches 73.29 on MATH500, well below plain OPD at 90.71. Replacing the preference loss with SFT changes gradient scale too: preference loss uses beta = 0.01, while SFT has a different scale under same outer weight."(W1)


"After matching gradient scales and tuning SFT weight with same validation protocol, does preference training still do better?"(Q1)

We conducted the suggested experiments and found that Seg OPD retains higher reasoning accuracy on all eight datasets, as shown below. We also clarify that redraft SFT and segment-wise preference learning optimize different objectives and therefore need not produce matching gradient scales, as discussed in Appendix B.6. We attribute the superiority of our method to the following differences between the two objectives:

  • Our method explicitly trains the student to prefer teacher redrafts over their paired student segments. In contrast, redraft SFT increases redraft likelihood without explicitly discouraging the original segments, which may therefore remain likely choices at inference [1].

  • Our method emphasizes insufficiently learned revision preferences through the coefficient βσ(−βmπθ(p))\beta\sigma(-\beta m_{\pi_\theta}(p)), which decreases as the revision margin grows. Rescaling SFT loss is not equal to this margin-dependent weighting, as we analyzed in Appendix B.6.

For this rebuttal, we evaluate three variants of Seg-OPD\N. Specifically:

  • Seg-OPD\G rescales the SFT loss weight in Seg-OPD\N using the ratio of the mean preference and SFT gradient norms recorded in Fig. 8.
  • Seg-OPD\W independently tunes the SFT loss weight using the same held-out validation split and configuration-selection criterion as Seg-OPD.
  • Seg-OPD-SFT combines the gradient calibration in Seg-OPD\G and tunes the SFT loss weight using the same held-out validation split and other settings as Seg-OPD\W.

We kindly invite the reviewer to check Appendix B.6 for a more detailed analysis of these variants. For this rebuttal, we use Qwen3-4B as the student and train each compared method with different training seeds. We report the mean reasoning accuracy (%) in the following table:

DatasetsSeg-OPDOPDSeg-OPD\NSeg-OPD\GSeg-OPD\WSeg-OPD-SFT
AIME2464.5063.1644.1045.5848.3963.24
AIME2563.1361.2858.0958.8659.0062.07
AIME2652.5149.5450.1649.0251.2150.17
MATH50089.4689.0974.7684.4182.3589.23
MINERVA48.8945.8441.6443.5144.1746.19
Codeforces45.8642.3938.6143.6144.5442.97
TACO43.6943.3231.7538.0933.9943.00
HumanEval+85.9682.4972.4378.2377.5483.27

From the table, we observe that:

  • Both gradient calibration and SFT loss-weight tuning can improve performance. This indicates that balancing redraft imitation with token-wise OPD still enables the student to benefit from teacher redrafts.

  • Our method outperforms the additional SFT variants. This indicates the effectiveness and practical value of our method, which explicitly teaches the student to prefer teacher redrafts over student segments, beyond adjusting the strength of redraft imitation.

[1] Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data. ICML 2024

AuthorsOpenReview ↗

Response Part 2


"There are exceptions to the gains, including Qwen3-4B on MATH500 at 89.10 versus 90.71 for OPD." (W2-1) "How much do main gains vary across training seeds? "(Q2-1)

We appreciate the reviewer raising this question. To assess how the gains for the main comparisons vary across training seeds, we conduct additional experiments for all main comparisons. For each of the four students, we train Seg OPD and all four baselines using the same five training seeds, including the original seed 42 and four additional seeds.

We evaluate each final checkpoint using the same five evaluation seeds. We report Seg OPD’s mean reasoning accuracy and its mean paired gain over each baseline, both with 95% confidence intervals across the five training seeds, in the table below:

StudentBaselineCodeforcesTACOHumanEval+AIME24AIME25AIME26MATH500MINERVAAvg.
DeepSeek-1.5BOPD+3.23 ± 0.31+3.01 ± 0.11+8.03 ± 0.19+8.18 ± 0.36+3.64 ± 0.42+1.98 ± 0.59+2.98 ± 0.13+2.22 ± 0.17+4.16
E-OPD+1.48 ± 0.18+2.05 ± 0.10+2.35 ± 0.27+8.27 ± 0.45+4.34 ± 0.41+4.62 ± 0.60+3.37 ± 0.06+1.74 ± 0.23+3.53
G-OPD+2.79 ± 0.34+1.23 ± 0.32+2.63 ± 0.23+1.33 ± 0.59+1.27 ± 0.61+3.25 ± 0.72+2.57 ± 0.08+0.78 ± 0.22+1.98
OmniOPD+2.50 ± 0.31+1.68 ± 0.18+1.83 ± 0.11+9.10 ± 0.40+7.54 ± 0.37+7.12 ± 0.70+3.38 ± 0.10+2.35 ± 0.30+4.44
Seg OPD36.33 ± 0.1031.24 ± 0.0881.17 ± 0.2351.33 ± 0.5939.90 ± 0.9235.32 ± 0.5989.34 ± 0.0839.39 ± 0.1650.50
Qwen3-4BOPD+3.47 ± 0.23+0.37 ± 0.14+3.47 ± 0.24+1.34 ± 0.59+1.85 ± 0.36+2.97 ± 0.72+0.37 ± 1.37+3.05 ± 0.24+2.11
E-OPD+2.06 ± 0.27+2.75 ± 0.13+1.17 ± 0.33+2.13 ± 0.37+1.21 ± 0.91+1.45 ± 0.38+0.39 ± 0.91+2.36 ± 0.19+1.69
G-OPD+2.43 ± 0.25+3.14 ± 0.29+2.15 ± 0.17+2.44 ± 0.40+1.95 ± 0.13+0.61 ± 0.17+0.34 ± 0.90+1.29 ± 0.15+1.79
OmniOPD+6.08 ± 0.27+4.98 ± 0.18+3.36 ± 0.21+0.87 ± 0.37+3.62 ± 0.43+2.01 ± 0.59+0.51 ± 0.18+4.12 ± 0.22+3.19
Seg OPD45.86 ± 0.1443.69 ± 0.1685.96 ± 0.1264.50 ± 0.3663.13 ± 0.3752.51 ± 0.6989.46 ± 0.2648.89 ± 0.1561.75
Qwen3-8BOPD+2.47 ± 0.26+0.29 ± 0.30+5.91 ± 0.19+4.31 ± 0.42+4.01 ± 0.59+2.71 ± 0.11+0.53 ± 0.23+3.05 ± 0.12+2.91
E-OPD+3.03 ± 0.12+1.11 ± 0.28+3.86 ± 0.20+0.34 ± 0.58+1.67 ± 0.41+4.64 ± 0.59+0.28 ± 1.55+1.17 ± 0.11+2.01
G-OPD+2.91 ± 0.21+3.37 ± 0.15+2.05 ± 0.20+2.64 ± 0.59+7.30 ± 0.59+0.35 ± 0.42+0.39 ± 1.69+1.26 ± 0.24+2.53
OmniOPD+1.93 ± 0.23+0.39 ± 0.36+3.99 ± 0.21+7.86 ± 0.69+2.38 ± 0.44+0.78 ± 0.38+0.41 ± 0.25+2.10 ± 0.18+2.48
Seg OPD48.80 ± 0.2545.66 ± 0.2682.38 ± 0.3863.61 ± 0.4559.67 ± 0.4147.17 ± 0.3688.70 ± 0.2545.99 ± 0.0960.25
Qwen3-14BOPD+4.23 ± 0.14+2.49 ± 0.15+2.41 ± 0.20+2.67 ± 0.59+3.65 ± 0.42+2.50 ± 0.36+0.95 ± 0.24+0.37 ± 0.36+2.41
E-OPD+5.57 ± 0.32+1.92 ± 0.16+1.92 ± 0.40+2.30 ± 0.42+3.86 ± 0.37+0.37 ± 0.52+0.70 ± 0.11+1.91 ± 0.25+2.32
G-OPD+4.97 ± 0.18+1.95 ± 0.15+1.41 ± 0.40+2.50 ± 0.69+3.18 ± 0.36+0.31 ± 0.42+0.98 ± 0.16+0.39 ± 1.12+1.96
OmniOPD+1.81 ± 0.23+0.48 ± 0.64+0.65 ± 0.17+3.61 ± 0.45+2.19 ± 0.69+3.22 ± 0.70+0.33 ± 0.63+0.80 ± 0.11+1.64
Seg OPD41.65 ± 0.1748.02 ± 0.2783.50 ± 0.2163.33 ± 0.5955.68 ± 0.4249.57 ± 0.5088.73 ± 0.2346.29 ± 0.2259.60
Overall gain---------+2.57

From the table, we have the following key insights:

  • Across all students and datasets, Seg OPD achieves an overall mean gain of 2.57% over five training seeds, with positive mean gains in all comparisons.

  • Training seed variability helps explain the exceptions in the original comparison. For instance, Qwen3-4B on MATH500 achieves a mean gain of 0.37 percentage points over OPD across five training seeds.

We also kindly invite the reviewer to check the additional multi-seed results in Appendix B.8.

AuthorsOpenReview ↗

Response Part 3


"For behavioral claims, both revision success and local quality audit use LLM judgments."(W2-2)

"Could a small human-labeled sample check revision-success judgments?"(Q2-2)

We appreciate the reviewer raising the questions. We clarify the rationale for using LLM judgments as follows:

  • To alleviate the potential influence introduced by the LLM judge, we aim to leverage a strong model that is independent of both the teacher and student models. Specifically, we use GPT-5.6 Sol as a strong LLM judge to assess revision success and revision-unit quality.

  • By using carefully designed prompts with explicit criteria and a fixed LLM judge, we establish a unified standard for assessing and comparing the revision behavior across models and methods.

For this rebuttal, we further check the reliability of the LLM judgments through human evaluations as detailed below. We conduct four evaluation runs per setting, each assessed by a different human evaluator, including two authors and two external evaluators. Following Appendix A.1, the human evaluators and the LLM judge identify revision attempts and assess their success on the same rollouts. The table below reports mean attempt and success counts per run, average overlap rates (i.e., the human evaluator and LLM both recognize one attempt or revision success), and RSR.

StudentDatasetMethodHuman/LLM attemptsAttempt overlap(%)Human/LLM successesSuccess overlap(%)Human/LLM RSR(%)
Qwen3-4BAIME25OPD175.00/174.0095.877.75/77.2598.744.4/44.4
Seg-OPD174.25/174.0095.9126.50/126.5098.872.6/72.7
AIME26OPD130.25/130.7595.957.00/57.0098.343.8/43.6
Seg-OPD145.50/145.0096.3102.75/102.2598.170.6/70.5
HumanEval+OPD538.50/538.7596.7231.00/231.7598.842.9/43.0
Seg-OPD516.25/518.0095.0351.00/352.2598.268.0/68.0
TACOOPD1,200.50/1,208.2595.3546.75/545.0098.445.5/45.1
Seg-OPD1,234.50/1,234.5095.4840.00/834.5098.368.0/67.6
CodeforcesOPD414.50/416.0096.1184.50/184.2598.344.5/44.3
Seg-OPD394.00/393.7595.7277.00/277.2598.870.3/70.4
Qwen3-8BAIME25OPD172.25/172.0095.973.25/73.0098.342.5/42.4
Seg-OPD164.75/164.7596.1117.25/117.2598.771.2/71.2
AIME26OPD170.00/169.5095.771.00/71.0098.641.8/41.9
Seg-OPD168.25/166.5095.8120.50/119.2598.171.6/71.6
HumanEval+OPD522.50/522.2596.1221.25/221.0098.842.3/42.3
Seg-OPD516.75/515.7595.4362.25/361.0098.470.1/70.0
TACOOPD1,201.75/1,201.0095.4533.25/533.2598.244.4/44.4
Seg-OPD1,203.25/1,197.7596.1839.00/838.5098.369.7/70.0
CodeforcesOPD396.50/395.2596.0168.75/168.0098.442.6/42.5
Seg-OPD444.00/443.0095.5319.50/317.7598.572.0/71.7

From the table, we have the following key findings:

  • With human evaluation, Seg-OPD achieves higher RSR than OPD by at least 22.5% across the two students and five datasets. This supports the effectiveness of our method in helping students revise their own reasoning.

  • Revision attempts and successful revisions show overlap rates of at least 95.0% and 98.1%, respectively. This provides human evidence supporting the consistency of our evaluation criteria and the reliability of our reasoning-behavior analysis.

Furthermore, we also ask these evaluators to assess local revision-unit quality using the same 1,024 revision units from Tab.4, with 256 pairs assigned to each. Given the shared prefix and two anonymous steps in random order, evaluators label each step for reasoning errors and indicate which step better advances the solution or reports no clear preference. We report the results in the following table:

EvaluatorRedraft error (%)Segment error (%)Redraft preferred (%)Segment preferred (%)No clear preference (%)
Author 11.1748.4485.9410.163.91
Author 21.9545.7083.5911.724.69
External evaluator 12.3446.8887.1112.500.39
External evaluator 21.5644.5386.7210.942.34

From the table, we have the following key findings:

  • Evaluators generally judged the retained redrafts to be correct and preferred them to the corresponding segments. As evaluators receive only the shared prefix and the two candidate steps, their preference shows that the improvement is visible in the retained sentence. This supports our approach of using retained redrafts as local preference objectives for reasoning revision.
  • Even when both steps were judged correct, evaluators more often preferred the redraft for the progress it made toward a solution. This supports our choice to select uncertain steps for revision, as a locally correct student step can be worth revisiting when another step makes better use of the same prefix.

We also kindly invite the reviewer to check these additional human evaluations in Appendices A.1 and A.2.

Reviewer_ZK36OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Segment-wise On-Policy Distillation to improve reasoning revision in language models by selecting high-uncertainty segments from failed student rollouts, generating teacher redrafts, and training students to prefer these redrafts while retaining token-wise distillation on other tokens. The method is evaluated on four student models across mathematical reasoning and programming benchmarks. The authors report an average relative improvement of 5.22% over existing OPD baselines, along with improved revision success rates and reduced reasoning loops. Ablation studies investigate the contributions of segment selection, targeted teacher redrafting, and preference learning.

Critical strengths
  1. The paper evaluates Seg-OPD across four student models of different sizes and eight mathematical reasoning and programming benchmarks, comparing it against four OPD methods. The reported improvements in overall performance provide encouraging evidence that the proposed approach is practically effective across diverse tasks and model scales.

  2. Seg-OPD combines uncertainty-based segment selection, targeted teacher redrafting, segment-wise preference learning, and token-wise distillation. The component ablations and revision-unit quality analysis provide useful evidence for the design choices and make the proposed approach relatively easy to analyze and reproduce.

Critical weaknesses
  1. The paper argues that token-wise OPD provides insufficient supervision for reasoning revision and that degenerate student prefixes can lead to unreliable teacher guidance (Introduction, p. 2; Section 3.2, p. 4). However, Seg-OPD masks selected segments from the token-wise loss without modifying the sampled rollout (Section 4.2, Eq. 8). Consequently, subsequent distillation still conditions on the original prefixes containing these segments. Moreover, the segment-wise preference objective encourages the student to prefer teacher redrafts but does not directly establish that the student learns to identify and correct its own reasoning errors. This gap between the stated motivation, the actual training mechanism, and the claimed revision capability weakens the paper's central contribution.

  2. Although Seg-OPD improves benchmark performance, the experiments do not fully isolate the contribution of segment-wise preference learning from additional teacher supervision and optimization effects. In particular, Seg-OPD incurs approximately 1.73–1.93 times the training time of standard OPD (Appendix D, Table 6), yet compute-matched comparisons are not reported. Furthermore, the SFT ablation exhibits substantially different gradient magnitudes from the preference-based method (Appendix B.3, Figure 8), making it difficult to attribute the performance gap solely to preference learning. These limitations leave the method's incremental benefit and the proposed mechanism insufficiently established.

Questions for authors
  1. The paper claims that token-wise OPD provides only local corrections and does not explicitly teach coherent multi-token reasoning revision. What evidence shows that this is a practical limitation rather than merely the absence of an explicit segment-level preference objective? How does this limitation differ from those addressed by existing chunk-level OPD methods?
  2. Figure 2(a) shows that replacing student-generated segments with teacher redrafts improves subsequent reasoning accuracy. Why does this observation specifically motivate segment-wise preference learning, rather than SFT on teacher redrafts or distillation from corrected trajectories?
  3. The paper argues that degenerate prefixes degrade teacher supervision, but Eq. (8) explicitly states that segment masking does not modify the sampled rollout. Consequently, subsequent token-wise OPD still conditions on the original potentially problematic segments. How does the proposed method directly address this limitation?
  4. High entropy does not necessarily indicate incorrect reasoning, and the proposed score, mean entropy minus standard deviation, appears heuristic. Can the authors justify this scoring function and demonstrate that selected segments are more likely to contain actual reasoning errors than segments selected using alternative uncertainty measures?
  5. The method retains the first sentence of a teacher draft only when the complete draft passes a correctness check. How do the authors ensure that the retained sentence itself is correct and constitutes a genuine improvement over the student segment? Could they report segment-level error rates and provide independent human evaluations?
  6. Could the authors compare Seg-OPD with an approach that replaces the selected student segment with its teacher redraft and then distills the revised continuation? An oracle-based segment-selection baseline would also help determine whether the gains arise from preference learning or from improved target selection.
  7. The ablation without targeted teacher redrafting removes both the student segment from the teacher's input and the explicit revision instruction. Could these factors be isolated in separate experiments? Could the authors also evaluate the individual contributions of segment masking and token-wise OPD?
  8. Could the authors report multi-seed results with confidence intervals for the main comparisons, clarify the aggregation of Pass@1 scores on small benchmarks such as AIME, and specify how the reported 5.22% average relative improvement is calculated? Reporting the number of accepted revision pairs across models and training stages would further improve reproducibility.
11 comments
AuthorsOpenReview ↗

Response Part 1

We sincerely appreciate the reviewer's constructive feedback, hoping the following discussions can address the reviewer's remaining concerns.


"Seg-OPD masks selected segments from the token-wise loss without modifying the sampled rollout. Consequently, subsequent distillation still conditions on the original prefixes containing these segments. How does the proposed method directly address this limitation?" (W1-1, Q3)

Our method directly addresses low-quality teacher guidance induced by degenerate prefixes through segment-wise preference learning. Seg-OPD changes how the selected, potentially problematic reasoning segment is supervised while retaining dense token-wise supervision elsewhere. Specifically:

  • To avoid relying on low-quality teacher guidance induced by degenerate prefixes, we identify a reasoning pivot and expand it into a segment containing the subsequent tokens primarily affected by it. We mask token-wise supervision within this segment to prevent such potentially low-quality teacher guidance from affecting student learning. Instead, we train the student to prefer the teacher redraft over the segment.

  • We retain supervision beyond the selected segment because prefix effects can diminish during generation, which is consistent with prior work [1,2]. This choice also supports training efficiency as retaining the original rollouts preserves dense feedback without introducing extra rollout cost. By avoiding student regeneration after teacher redrafting and the associated context-switching overhead, our method achieves an effective performance–cost trade-off.

[1] Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation? EMNLP 2021.

[2] Are Reasoning LLMs Robust to Interventions on Their Chain-of-Thought? ICLR 2026.


"Moreover, the segment-wise preference objective encourages the student to prefer teacher redrafts but does not directly establish that the student learns to identify and correct its own reasoning errors. "(W1-2)

Our method enables students to identify and correct their reasoning that needs revision by learning to prefer teacher redrafts over their own segments. We evaluate this capability through revision success rate (RSR) on student rollouts generated without teacher intervention. Both the revision attempts and the alternative reasoning are generated by the student itself, and RSR measures the proportion of revision attempts that correct an earlier error with a substantively different alternative, as we stated in lines 198-200.

For this rebuttal, we extend the RSR comparison between our method and OPD across four student models and eight datasets, reporting their RSR (%) in the following table:

StudentMethodCodeforcesTACOHumanEval+AIME24AIME25AIME26MATH500MINERVA
DeepSeek-1.5BOPD34.633.835.437.336.936.137.636.6
Seg-OPD57.456.258.962.061.360.460.859.7
Qwen3-4BOPD42.345.143.042.841.640.944.243.3
Seg-OPD68.267.668.065.166.467.565.866.9
Qwen3-8BOPD42.544.442.343.142.441.943.743.2
Seg-OPD71.770.070.069.171.271.669.870.5
Qwen3-14BOPD42.142.941.843.642.741.444.043.5
Seg-OPD65.864.966.368.367.967.567.268.1

The results demonstrate that our method achieves RSRs of 56.2–71.7% across four student models and consistently outperforms OPD in all available comparisons, with absolute gains of 21.6–29.7%, highlighting the effectiveness of our method in teaching students to identify and correct their own reasoning errors. We further confirm this RSR advantage through human evaluation across two student models and five datasets in Appendix A.1. We also kindly invite the reviewer to check the additional RSR results in Appendix B.4.

AuthorsOpenReview ↗

Response Part 2


"What evidence shows that this is a practical limitation rather than merely the absence of an explicit segment-level preference objective?" (Q1-1)

The practical limitation is that students trained with token-wise OPD are less successful at correcting their own reasoning errors than those trained with explicit revision supervision in Seg-OPD. This is supported by Seg-OPD's higher RSR in Fig.4 and 7.

For this rebuttal, we introduce OPD-A to test whether multi-token feedback aggregation alone can achieve the same gains in reasoning accuracy. OPD-A averages the token-wise OPD feedback within each segment and uses this average to weight the updates of all tokens in it. It retains token-wise OPD elsewhere and does not use teacher redrafts.

We train a Qwen3-4B student with OPD, OPD-A, and Seg OPD and evaluate across all eight datasets. For a fair comparison, we adopt Seg-OPD's segment selection for OPD-A. We report their mean reasoning accuracy (%) in the following table:

MethodCodeforcesTACOHumanEval+AIME24AIME25AIME26MATH500MINERVA
OPD42.3943.3282.4963.1661.2849.5489.0945.84
OPD-A43.6443.1983.5362.3762.1650.7289.1346.87
Seg-OPD45.8643.6985.9664.5063.1352.5189.4648.89

From the table, we have the following key insights:

  • OPD-A improves reasoning accuracy over OPD on 6 of 8 datasets, indicating that aggregating token-wise feedback at the segment level can improve reasoning performance.

  • Our method outperforms OPD-A across all datasets, supporting the practical value of explicitly learning to prefer teacher redrafts over student segments beyond simply aggregating token-wise feedback.

We kindly invite the reviewer to check Appendix B.7.1 for a more detailed analysis on these variants.


"How does this limitation differ from those addressed by existing chunk-level OPD methods?"(Q1-2)

We compared our method with the representative chunk-level method OmniOPD [1]. For this rebuttal, we further compare it with two of the latest chunk-level methods, RelayOPD [2] and CHOP [3], and clarify the inherent differences as follows:

  • For the objective, our method aims to teach students to revise their own reasoning steps to better support subsequent reasoning. In contrast, OmniOPD aims to obtain chunk-level supervision without teacher logits; RelayOPD redirects student rollouts through teacher intervention; and CHOP aims to alleviate unreliable token-level supervision through adaptive supervision routing.

  • For supervision, our method pairs student segments with targeted teacher redrafts and trains the student to prefer the redrafts, while retaining token-wise OPD elsewhere. In contrast, OmniOPD reweights original student chunks by their similarity to teacher continuations; RelayOPD applies token-wise distillation to student trajectories that include teacher chunks; and CHOP matches teacher and student chunk distributions in representation space.

  • For segment selection, our method expands high-entropy anchors to sentence boundaries in failed rollouts and ranks the segments by their aggregated uncertainty. In contrast, OmniOPD selects fixed-length chunks using token entropy; RelayOPD triggers teacher takeover using disagreement over tokens; and CHOP uses teacher reliability to choose where to apply chunk supervision. We also kindly invite the reviewer to check these additional comparisons in Appendix B.5.

For this rebuttal, we further compare our method with these methods using Qwen3-4B and Qwen3-8B as students, and report reasoning accuracy (%) in the following table:

StudentMethodCodeforcesTACOHumanEval+AIME24AIME25AIME26MATH500MINERVA
Qwen3-4BOmniOPD39.7838.7182.6063.6359.5150.5088.9544.77
RelayOPD42.1838.5484.7663.8358.8951.7288.9446.38
CHOP39.4241.2683.1561.1760.4450.3188.4247.62
Seg-OPD45.8643.6985.9664.5063.1352.5189.4648.89
Qwen3-8BOmniOPD46.8745.2778.3955.7557.2946.3988.2943.89
RelayOPD47.9245.1278.1958.6256.4546.9488.1445.17
CHOP46.1544.2880.6448.3044.7045.1888.3343.72
Seg-OPD48.8045.6682.3863.6159.6747.1788.7045.99

The results show that our method generally outperforms the closest chunk-level OPD baselines, highlighting the effectiveness of reasoning revision by preference learning with teacher redraft and retaining dense token-wise supervision.

[1] OmniOPD: Logit-free on-policy distillation via speculative verification. arXiv:2606.01476

[2] Pass the baton: Trajectory-relayed on-policy distillation. arXiv:2607.26057

[3] When Token Supervision Fails: Segment-Level On-Policy Distillation for Long-Horizon Agents.

AuthorsOpenReview ↗

Response Part 3


"Seg-OPD incurs approximately 1.73–1.93 times the training time of standard OPD, yet compute-matched comparisons are not reported." (W2-1)

We thank the reviewer for asking about compute-matched comparisons. While Tab.20 reports the relative training time of a fixed 48 optimization steps to make a clear demonstration of our results, it does not measure convergence time.

For this rebuttal, we compare Seg-OPD and OPD under matched wall-clock training budgets. Specifically, we normalize the training time of a 48-step OPD run to 1.0, and we evaluate the latest checkpoint completed within each budget.

Following the training protocol stated in Appendix A.6, we report the average reasoning accuracy (%) of OPD/Seg-OPD across diverse datasets with Qwen3-4B for math and Qwen3-8B for code tasks in the following table. We also kindly invite the reviewer to check the compute-matched comparisons in Appendix D.

TaskDatasets(↓)&Time budgets(→)0.40.60.81.01.21.41.6
MathAIME2443.08/36.9551.94/52.1453.96/53.9162.84/63.8962.25/62.1263.51/62.2863.08/64.23
AIME2553.92/53.9158.88/57.4657.15/58.4960.73/61.1261.42/63.0460.11/63.0861.14/62.86
AIME2636.34/31.9442.16/40.3648.14/47.8349.28/49.6349.83/51.5448.54/51.6749.52/52.51
MATH50085.46/83.7287.70/86.5088.88/88.7489.08/89.3889.16/89.4289.05/89.2589.18/89.56
MINERVA40.04/37.6742.29/42.8844.67/46.4245.56/48.2944.49/47.7345.63/47.8445.44/48.62
CodeCodeforces27.39/32.0331.51/38.3936.66/41.1646.12/47.6544.74/47.1946.17/48.3046.53/48.83
TACO23.20/27.2830.67/29.7839.69/37.4544.78/45.8941.39/45.0844.83/44.4144.85/45.74
HumanEval+76.58/66.4474.64/74.2772.14/76.4676.28/77.6275.96/78.2476.28/82.5176.54/82.19

From the tables, we have the following key insights:

  • Our method generally achieves higher reasoning accuracy than OPD under matched wall-clock training budgets. These results demonstrate that our method achieves an effective performance-cost trade-off across the evaluated datasets and student models.

  • As we stated in Appendix D, the additional training cost of Seg-OPD comes from teacher redrafting and segment-wise preference learning, which provide explicit revision supervision beyond token-wise OPD. The results show that the performance gains are rooted in our design instead of simply longer training.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response from Authors

We sincerely thank you again for your valuable time and effort in assessing our work. We have responded to the reviewers' questions through additional experiments and discussions. Specifically, we have added 16 new tables and figures with corresponding analyses to our manuscript, and the updated content is highlighted in blue. Specifically, we made the following major updates:

  • We further assess revision success and revision-unit quality through human evaluations to show the effectiveness of reasoning revision and the reliability of our reasoning behavior analysis.

  • We conduct compute-matched comparisons and evaluate additional SFT variants with gradient calibration and loss weight tuning to show that the superiority of our preference learning design is beyond additional training time and redraft imitation.

  • We conduct additional experiments with four more training seeds for all main comparisons to further validate the performance gains across diverse student models and datasets.

  • We further compare our method with two of the latest chunk-level OPD methods to show the effectiveness of learning reasoning revision through segment-wise preference learning and further strengthen our technical contributions.

  • We conduct additional ablation studies on teacher redrafting, segment masking, and token-wise supervision to show their contributions to learning reasoning revision and the performance of our method.

The additional analyses and discussions listed above aim to address the reviewers' questions, including how our method tackles the identified challenges, how our preliminary analysis informs our design, our method’s design choices and distinctions from existing methods, and the fairness and reliability of our evaluations.

We summarize the key contributions of our work as follows:

  • We investigate an important and novel topic of learning reasoning revision with segment-wise OPD, focusing on how to teach the student models to revise their own reasoning steps.

  • Building upon a preliminary analysis showing the effectiveness of teacher redrafted segments, we propose a novel framework, Seg-OPD, which uses segment-wise preference learning to train students to prefer teacher redrafts over the paired student segments, while retaining dense token-wise supervision outside the selected segments. An uncertainty-based segment selection process designed for reasoning revision learning is further adopted.

  • We conduct extensive experiments with four student models across eight math and code benchmarks. Seg-OPD achieves an overall mean accuracy gain of 2.57 percentage points over four OPD baselines across five training seeds in our newly added experiments. Our reasoning behavior analysis further demonstrates higher revision success rates and fewer reasoning loops after training.

We appreciate the thoughtful feedback from the reviewers again. We hope our efforts can adequately address the remaining concerns.

Sincerely,

Authors of Submission 22600

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.