acceptodds
Under review as a conference paper at ICLR 2027

Hindsight Compacts but Does Not Repair: Rethinking On-Policy Self-Distillation in Reasoning Models

Abstract

On-Policy Self-Distillation (OPSD) has emerged as a promising post-training method: requiring only privileged context, it enables token-level credit assignment from a self-teacher with hindsight, yielding higher accuracy with shorter responses. However, in thinking-enabled mathematical reasoning, where solving a problem takes long reasoning traces that explore, reflect, and backtrack, its length reductions persist but its accuracy gains largely do not. Does OPSD's hindsight signal, then, still repair failed trajectories, or does it only compact viable ones? We hypothesize compaction: in long traces, hindsight reveals which steps were unnecessary more readily than which steps would have repaired the solution. To test this, we apply OPSD separately to correct and incorrect rollouts. Training only on correct rollouts shortens responses by 18-29% while largely preserving accuracy, whereas training only on incorrect rollouts degrades it. The accuracy gap holds across three models, six benchmarks, and three seeds, and under a same-prompt control for difficulty. Neither branch raises the pass@k ceiling, and both suppress exploration and reflection markers, which viable traces can spare but failed ones need. Changing the divergence, enriching or reinjecting the privileged context, and training longer only move OPSD along the same accuracy-length tradeoff. Hindsight compacts reasoning the model can already produce but does not repair reasoning it cannot.

Primary area:
generative models
Keywords:
large language models, reasoning models, on-policy self-distillation, reasoning trace compression

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_if28OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether OPSD’s hindsight-guided signal repairs failed trajectories or compacts viable reasoning. Across multiple models and mathematical benchmarks, Correct-only OPSD shortens responses while largely preserving accuracy, whereas Incorrect-only OPSD degrades accuracy.

Critical strengths

The Correct-only versus Incorrect-only comparison provides a focused diagnostic of OPSD’s hindsight-guided signal. Prompt-matched controls and consistent results across models, benchmarks, and training seeds support the finding that rollout correctness matters for safe compaction, offering useful empirical guidance for understanding and applying OPSD.

Critical weaknesses
  1. The paper does not directly establish whether privileged context enables effective corrective guidance on incorrect rollouts. Their degradation could reflect inadequate teacher guidance rather than an inability of OPSD to repair failed reasoning, leaving the central interpretation insufficiently supported.

  2. Reduced epistemic-marker density does not establish that useful exploration and reflection are suppressed or that this causes the accuracy loss. Behavioral validation or controlled interventions are needed to substantiate the proposed mechanism.

Questions for authors
  1. Does privileged context enable the self-teacher to provide valid corrective guidance on incorrect rollouts? What evidence distinguishes inadequate teacher guidance from a failure of OPSD to transfer effective guidance?

  2. Beyond reduced epistemic-marker density, what evidence links the accuracy loss to suppressed exploration or reflection, rather than a change in expression style?

Reviewer_PgkSOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether on-policy self-distillation improves long-form mathematical reasoning by repairing failed reasoning or primarily by shortening viable reasoning. Its central intervention applies the distillation objective separately to correct and incorrect student rollouts. The proposed insights are supported by a rich set of controlled experiments spanning prompt-matched comparisons, three models, six benchmarks, pass@k evaluation and per-problem analysis.

The empirical contrast is interesting and potentially useful. However, several analyses conflate aggregate performance with trajectory repair, and the causal explanation is stronger than the evidence supports. A more precise operational definition of repair, together with targeted controls, would materially strengthen the contribution.

Critical strengths

This paper proposes a clear and consequential research question. Understanding what dense hindsight supervision actually contributes is valuable beyond another accuracy–length comparison.

Critical weaknesses
  1. Although the authors present observational results and perform careful ablations in Section 4 to rule out alternative accounts for why OPSD cannot repair incorrect rollouts, the work does not supply direct positive theoretical analysis to validate its proposed causal mechanism.
Questions for authors
  1. The discrepancy between the 8192 training length and 30k evaluation budget creates confounding truncation artifacts; given Qwen3-8B’s 14–17k average output length on AIME, many rollouts are truncated, and treating truncated samples as incorrect risks making Incorrect-only training target truncated prefixes instead of true reasoning failures, which requires clarifying that how truncated samples are assigned labels.
  2. The results of Section 3 show that training on Incorrtect-only rollouts yields can not repair the model, but the abalation studies in Section 4 did not run in Incorrect-only method, which is not convincing. For example, in the case of a incorrect rollout, the reflection a student writes after seeing the ground truth is likely just a post-hoc rationalization—essentially, the signal is nothing more than noise. why not test whether the failure can be rectified by providing the correct reasoning for that failed trajectory.
  3. what if we apply the RLVR method on incorrect rollouts and OPSD on correct rollouts?
Reviewer_BjXoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether on-policy self-distillation really improves reasoning by repairing failed trajectories or just compressing existing successful trajectories. It compares results based on different training set: correct, incorrect, and all rollouts, and found that correct-only training substantially reduces response length while preserving accuracy, whereas incorrect-only training degrades accuracy. It further examines pass@k, exploration related token patterns, and the effects of different KL objectives, privileged contexts, context reinjection, and longer training. Experiments across three reasoning models, six mathematical benchmarks, and multiple random seeds support the observed trend. The authors conclude that OPSD primarily serves as an efficient reasoning compression method rather than a mechanism for repairing failed reasoning traces.

Critical strengths

An important and timely research question. The paper studied whether OPSD really improves reasoning or primarily compresses reasoning trajectories. Understanding the mechanisms and the effectiveness of OPSD, especially from reasoning capability perspective is a hot topic.

Some interesting empirical findings and thoughtful follow-up investigations. The authors examine several possible explanations for their observations, including prompt difficulty, divergence choice, privileged context, and training duration.

Critical weaknesses
  1. Insufficient controls to establish an OPSD-specific limitation. The central Correct-only vs. Incorrect-only experiment (Table 2) does not establish whether the performance gap is specific to OPSD or just a general consequence of training on different data distributions. The control (Table 7) addresses problem difficulty but does not account for differences in reasoning states, trajectory lengths, or failure types. For example, one key missing baseline is RL (e.g. GRPO) under comparable training conditions using matched trajectory distributions. These experiments would help determine whether the degradation on incorrect rollouts is specific to OPSD. The GRPO comparison in Table 6 focuses on compression efficiency, and is irrelevant to this central question.

  2. The proposed compaction vs repair mechanism is not directly validated. The paper concludes that OPSD compacts successful reasoning traj but fails to repair incorrect reasoning, yet the experiments mainly measure proxies like final accuracy or response length rather than what actually changes during the reasoning process. To substantiate this claim, I suggest authors to:

    • Examine the privileged teacher's reasoning behavior. Does the teacher actually generate better reasoning paths, or does it simply become more concise and confident? In particular, can the teacher recover from the reasoning states where the student is about to make mistakes?
    • Test whether OPSD transfers corrective reasoning. If the teacher can generate correct continuations from the critical states, does the OPSD trained student become more likely to produce them? This would distinguish the problem from a lack of useful teacher supervision from a failure of the distillation mechanism.
    • Directly verify compaction and exploration suppression. Which reasoning steps are removed from successful trajectories, and does training actually reduce meaningful exploration? Table 3 only counts words such as "wait" and "maybe", which does not equal to changes in reasoning behavior.

These analyses are important because the current results do not directly explain the reasoning mechanism behind it, which is the key claim of this paoer. The additional ablations on divergence, privileged context, reinjection, and training duration do not close this gap.

Questions for authors
  1. What other evidence supports the interpretation that OPSD suppresses meaningful exploration and self-correction, rather than simply changing the wording or structure of reasoning traces? In particular, how do the lexical markers in Table 3 establish a connection between reduced exploration and accuracy degradation?

  2. How are rollouts that reach 8192 token training limit handled in the Correct-only and Incorrect-only settings? What proportion of the training trajectories are truncated?

Reviewer_atsSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines what OPSD teaches thinking-enabled reasoning models. It applies the same distillation objective separately to correct and incorrect student rollouts. Training on correct rollouts substantially shortens responses with smaller accuracy losses. Training on incorrect rollouts generally hurts accuracy. The authors test several privileged contexts, divergence choices, context reinjection, and longer training. They interpret the results as evidence that hindsight supervision mainly compresses reasoning the model can already produce. Experiments cover three related 7–8B models and several mathematical reasoning benchmarks.

Critical strengths

The separation between accuracy and response length is useful. A shorter response can look like more efficient reasoning even when the model has lost accuracy. The outcome-filtered experiments make this problem visible. On Qwen3-8B, correct-only training reduces length by about 29% with a 1-point accuracy loss. Incorrect-only training reduces length less and loses 6.6 points. This is a meaningful practical warning about how OPSD results should be evaluated.

The prompt-matched experiment in Appendix A addresses an obvious confound. Correct and incorrect rollouts tend to come from different problems, so simply comparing the two training sets would mix trajectory quality with prompt difficulty. Matching the prompts preserves a substantial gap. I also appreciate the attempts to change the privileged context and its position in the trace. These controls make the observed failure pattern more convincing.

Critical weaknesses

The conclusion that hindsight “does not repair” is stronger than the measurements establish. Incorrect-only training can repair some failures while damaging more previously successful cases. Table 9 does not separate these mechanisms. Its repaired mass is (1 − b)a and its damaged mass is b(1 − a), so their difference is exactly a − b. The reported Net therefore restates the accuracy change. Even an unchanged success probability would produce positive mass in both transition columns. Likewise, pass@8 measures success within eight attempts, not the boundary of what a model can solve. These results support net degradation under the tested recipe. A stronger claim about repair needs a direct test, such as continuing failed prefixes with and without the privileged teacher signal.

The handling of truncated training rollouts is unclear and matters for this comparison. Training stops generation at 8,192 tokens, while evaluation allows 30,000. The base model's mean responses on AIME24 and AIME25 are already much longer than the training cap. This does not establish how often DAPO training responses are truncated, but it makes that rate important to report. Figure 2 distinguishes truncated rollouts from completed incorrect ones. Section 3 does not clearly say whether truncated rollouts enter Incorrect-only training. If they do, the branch may include unfinished solutions that could succeed with more tokens. The prompt-matched control does not resolve this issue. Please separate completed failures from truncated attempts and check whether the result persists with a longer training budget.

The explanation involving suppressed reconsideration remains indirect. Marker counts show a change in wording, but they do not show that a useful correction step was removed. Correct-only training also reduces these markers more than incorrect-only training while preserving accuracy better. The paper should discuss Dense Supervision but Sparse Guidance: A Token-Level Analysis of On-Policy Self-Distillation. That work tests whether teacher-preferred tokens can redirect subsequent generation on coding tasks. Its setting differs from this paper, but its intervention is directly relevant to distinguishing local correction from changes in the later trajectory.

The paper is readable, though the categorical “does not repair” statement is repeated more often than the evidence warrants. Figure 2 should also define its shaded bands. It is currently unclear what variation they represent.

Questions for authors
  1. Can you directly measure recovery from failed prefixes under the privileged teacher signal? How would you distinguish an absence of repair from repairs outweighed by damage elsewhere?
  2. Are truncated rollouts included in Incorrect-only training? What fraction of that branch do they comprise, and does the accuracy gap remain when training uses only completed incorrect responses?
  3. Do trace-level examples or interventions show useful reconsideration being removed? Marker frequency alone seems insufficient to identify that mechanism.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.