acceptodds
Under review as a conference paper at ICLR 2027

From Rollouts to Hints: Selective Trajectory Reuse for On-Policy Self-Distillation

Abstract

On-policy self-distillation (OPSD) turns a reasoning model’s own rollouts into dense training paths: a privileged teacher conditions on a reference solution and guides the student along prefixes the student actually visits. Yet the generated trajectory itself remains underused. The reference always occupies the teacher’s privileged context, even when the student produces a verified solution that is shorter and better matched to its decoding distribution, and every generated token receives distillation despite large differences in decision relevance. We introduce SEL-OPSD, which treats generated trajectories as reusable supervision assets at two scales. HintSelect verifies a group of current-policy rollouts and lets a concise correct candidate serve as both the student trajectory and, when it clears a length margin, the teacher hint. KeyStep preserves the original OPSD divergence but restricts it to positions selected by student uncertainty, teacher–student disagreement, mathematical-token priors, and final-answer risk. Across three Qwen3 scales and three competition-level mathematics benchmarks, SEL-OPSD improves Avg@12 over dense OPSD in all nine model–benchmark settings, with a mean gain of 2.69 percentage points and a 95% bootstrap interval of [2.41, 3.03]. It also reduces deployment response length by approximately 30%. Factorial and matched-sparsity ablations isolate the contributions of hint reuse, verified-trajectory training, and token selection, showing that the gains cannot be explained by best-of-group filtering or sparse loss alone.

Primary area:
interpretability and explainable AI
Keywords:
on-policy self-distillation, mathematical reasoning, selective supervision, trajectory reuse, token-level distillation, inference efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_HRRMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SEL-OPSD, an extension of on-policy self-distillation (OPSD) for reasoning LLMs that selectively reuses generated trajectories at two levels. At the trajectory level, HINTSELECT samples multiple current-policy rollouts, selects the shortest verifier-correct trajectory for student training, and uses it as the teacher’s privileged hint when it is sufficiently shorter than the reference solution. At the token level, KEYSTEP selects distillation positions based on student uncertainty, teacher–student disagreement, mathematical-token priors, and an answer-region bonus. Experiments on Qwen3-1.7B/4B/8B across mathematical reasoning benchmarks report improvements over dense OPSD in Avg@12 and reduced deployment response length.

Critical strengths
  1. Decomposition of OPSD supervision. The paper separates student-trajectory selection, teacher privileged-context selection, and token-level supervision, which is a meaningful way to analyze and extend OPSD. The paper separately ablates student-trajectory selection and teacher-hint replacement, which helps distinguish the contribution of each mechanism rather than attributing all gains to the combined method.

  2. Reasonable token-level selection mechanism. KEYSTEP is well motivated as a form of hard-example mining: uncertainty and teacher–student disagreement identify positions where privileged supervision may be most useful, while the fixed token budget keeps the additional computation small.

Critical weaknesses
  1. Clarity is insufficient. The paper is unevenly structured: for example, Section 3 is very short, while the main-results section mostly restates the numbers without sufficiently explaining the observed trends or why the proposed components should lead to them. More importantly, some reported results are internally inconsistent. The Qwen3-8B/AIME24 values in Figure 4 do not match those in Tables 2/3, and some reported seed-level statistics are difficult to reconcile with the stated evaluation protocol. Figure 2 also contains a clear arithmetic error: for 2n+4=6n−82n+4=6n-8, trajectories ending in n=6n=6 are marked as correct, although the correct solution is n=3n=3. These issues make the presentation harder to trust and the empirical conclusions harder to verify.

  2. The logic and attribution of HINTSELECT are not fully convincing. HINTSELECT always chooses the shortest verifier-correct rollout as the student trajectory and may also use it as the teacher hint. However, being shorter and having the correct final answer does not necessarily mean that the trajectory contains better reasoning or is a better privileged hint, since the verifier mainly checks answer correctness rather than the validity of intermediate reasoning. This also introduces an explicit preference for short trajectories, which is particularly relevant since reduced response length is one of the paper’s main reported benefits. The current ablation does not isolate this effect: the “trajectory-only” variant still selects the shortest correct rollout, so it mixes correctness filtering with length-based selection. A random or uniformly sampled correct-trajectory baseline would be needed to determine whether the gain actually comes from choosing shorter trajectories rather than simply training on verified-correct ones.

Questions for authors
  1. Can the authors add a randomly selected verified-correct trajectory baseline at the same GG and compute? This would separate correctness filtering from the additional shortest-trajectory bias and clarify whether the response-length reduction is caused by improved reasoning or explicit length selection.
  2. Can the authors clarify the HINTSELECT algorithm used in the experiments? The main text defines the candidate pool as including both sampled rollouts and the reference solution, while Algorithm 2 selects the shortest trajectory only among verified sampled rollouts. Please clarify which version was implemented, when the student trajectory is replaced, and under what conditions the reference remains the teacher hint.
  3. Can the authors clarify the methodological contribution of KEYSTEP beyond generic hard-token selection? In particular, a comparison against a matched top-KK baseline using the actual per-token OPSD loss would help determine whether the proposed multi-signal score provides benefits beyond standard hard-example mining.
Reviewer_ZgvTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Sel-OPSD, which improves on-policy self-distillation through trajectory and token selection. HintSelect chooses the shortest answer-verified response from current-policy rollouts as the student training trajectory and also uses it as the privileged teacher hint when it satisfies a length margin relative to the reference solution. KeyStep combines student uncertainty, teacher–student disagreement, a mathematical-token prior, and answer-region information to apply the original distillation loss only at selected positions. Experiments cover three Qwen3 scales and six mathematics benchmarks. The paper reports a mean Avg@12 improvement of 2.69 percentage points across nine primary model–benchmark configurations, alongside approximately 30% shorter deployment responses for the 8B model.

Critical strengths
  1. The method addresses a practically useful question: how to extract more supervision from responses already generated during training. Reusing verified responses for both student training and teacher conditioning, with the reference retained as a fallback, provides a modification that can be integrated into existing OPSD systems.

  2. The reported improvements are reasonably consistent. All three model scales outperform dense OPSD on the three primary competition benchmarks, and additional results on GSM8K, MATH-500, and OlympiadBench also show positive changes. Improved accuracy accompanied by shorter responses suggests potential deployment value.

  3. The paper explicitly considers contribution attribution. Table 2 separately enables hint replacement and trajectory replacement, providing evidence for their individual contributions. Random-mask and single-signal comparisons also reflect a useful attempt to distinguish informed token selection from sparsification alone. The authors additionally disclose the increased training cost.

Critical weaknesses
  1. The incremental contribution over the closest methods is insufficiently established. KeyStep’s combination of uncertainty and teacher–student disagreement is closely related to selective distillation methods such as TIP, while reasoning compression has been studied through self-distillation methods such as CRISP. Current comparisons mainly cover conventional distillation and self-training baselines. More direct comparisons are needed to establish the additional value of the mathematical prior, answer gating, and hint-selection combination.

  2. Comparisons at equal training steps do not separate method benefits from additional sampling computation. Table 8 reports 45.1 seconds per step for the complete method versus 15.5 seconds for dense OPSD, a roughly 2.9-fold difference. The authors explicitly target deployment efficiency and provide an amortization analysis, but accuracy comparisons at matched training time or generated-token budgets remain missing. This limits assessment of the overall cost–benefit trade-off.

  3. The mechanism behind HintSelect requires further controls. The factorial experiment separates hint replacement from trajectory replacement, but does not distinguish benefits from shorter hints, better alignment with the student distribution, or conditioning the teacher on the exact response being trained on. Final-answer verification also does not establish the validity or completeness of the reasoning. Length-matched reference hints and hints from a different verified response would better test the proposed explanation for student-generated hints.

  4. Some reported results and ablation settings are insufficiently consistent to verify the central attribution claims. For 8B/AIME24, Figure 4 reports dense OPSD Avg@12 as 76.4, whereas Tables 2/3 report 77.78; the full method’s Pass@12 and Maj@12 also differ. Furthermore, the lower panel of Table 3 reports 80.28 for Hybrid + answer gate, while KeyStep-only in the upper panel achieves 78.61, without specifying whether HintSelect is held fixed across the lower-panel rows. These ambiguities prevent a clear assessment of the individual gains from token selection and answer gating.

  5. The headline confidence interval and significance analysis do not fully capture evaluation uncertainty. Appendix D bootstraps nine model–benchmark differences and applies a sign test across them, although configurations share benchmark questions and may be correlated across model scales. These analyses describe consistency across the reported configurations, but inference about performance on new questions requires more care, particularly because each primary benchmark contains only 30 problems.

Questions for authors
  1. With HintSelect and the retained-token budget held fixed, how does KeyStep compare with a TIP-style selector? Does Sel-OPSD retain an accuracy–length advantage over CRISP or self-distillation with a simple conciseness instruction?

  2. At matched training time or total generated-token budgets, how do dense OPSD and Sel-OPSD compare in accuracy and deployment response length? Does the training-cost amortization conclusion hold against these matched baselines?

  3. With the student training trajectory fixed, what happens when the teacher receives a length-matched reference, a different verified response, or the training trajectory itself? Can these controls separate hint length, student-distribution alignment, and reuse of the identical response?

  4. Could the authors reconcile Figure 4 with Tables 2/3 and specify the hint selection, trajectory selection, and retained-position settings for every row in the lower panel of Table 3? What is the independent gain from answer gating under otherwise identical conditions?

  5. Using the existing per-question predictions, could the authors report a paired uncertainty analysis that preserves dependence across questions and models, and clarify the scope of the configuration-level confidence interval?

Reviewer_39AaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes two small modifications to on-policy self-distillation (OPSD): selecting the teacher’s hint from student-generated responses and restricting supervision to selected token positions. HINTSELECT samples several student responses, selects the shortest verified-correct response as the training trajectory, and also uses it as the teacher’s hint when it is sufficiently shorter than the reference. KEYSTEP then restricts the existing distillation loss to token positions selected using student uncertainty, teacher–student disagreement, mathematical notation, and final-answer information. Overall the paper seems like a convex combination of several ideas from recent work; successful-rollout teacher conditioning is related to MOPD, uncertainty/disagreement-based token selection to TIP, and concise self-teaching to CRISP. The contribution lies in the particular selection rules and their combination. Experiments report consistent accuracy gains over dense OPSD and shorter deployment responses, but training costs approximately three times more per step, and direct comparisons with these closely related methods are missing.

Critical strengths

The paper combines verified student-generated hints with selective token supervision in a coherent extension of OPSD.

It reports accuracy improvements over dense OPSD across three model sizes and several mathematics benchmarks, alongside shorter deployment responses and three-seed variability estimates.

Its extensive ablations usefully separate teacher-hint replacement from verified-trajectory selection, while matched random masking tests whether token selection helps beyond sparsification alone. The explicit training-cost breakdown also makes the added rollout expense clear.

Critical weaknesses

My main concern for this paper is that it seems like a putting together of existing / simplistic ideas on top of OPSD, and thus has rather low innovation - something that will be hard to correct for even if the performance somewhat improves over (currently missing, see below) baselines.

HINTSELECT overlaps with MOPD’s use of verified student rollouts to construct a privileged self-teacher. Its more specific choices are selecting the shortest correct response, applying a length margin, and potentially using the same response as both hint and training trajectory. Without a direct comparison, the paper does not establish that these choices improve upon existing rollout-conditioned self-teaching.

Reference: Yu et al. (2026), “Multi-Rollout On-Policy Distillation via Peer Successes and Failures,” arXiv:2605.12652.

KEYSTEP’s main continuous signals—student uncertainty and teacher–student disagreement—are central to TIP’s analysis and token-selection methods. Adding mathematical-token and answer-region heuristics appears incremental, and comparisons with random masking or individual signals do not substitute for a matched TIP baseline.

Reference: Xu et al. (2026), “TIP: Token Importance in On-Policy Distillation,” arXiv:2604.14084.

The inference-efficiency claim also needs comparison with CRISP, which already uses a concise self-teacher to shorten reasoning. Although its mechanism differs from selecting short verified hints, it directly addresses the same deployment objective. The paper does not show that its substantially more expensive grouped-rollout procedure provides a better accuracy–length tradeoff.

Reference: Sang et al. (2026), “CRISP: Compressed Reasoning via Iterative Self-Policy Distillation,” originally released as “On-Policy Self-Distillation for Reasoning Compression,” arXiv:2603.05433.

Correctness-based allocation of supervision is also established by SCOPE. The paper’s verified-trajectory ablation is useful, but does not establish an advantage over existing methods that distinguish successful and unsuccessful rollouts rather than primarily selecting a successful trajectory for distillation.

Reference: Zheng et al. (2026), “SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting,” arXiv:2604.10688.

The internal ablations leave several choices insufficiently isolated. These include selecting the shortest rather than a random correct response, using the exact training response rather than a distinct successful peer as the hint, and adding the mathematical-token prior and mandatory tail window. In particular, same-response hinting may provide a reconstruction signal; the current controls do not separate this from the claimed benefit of a better-matched teacher.

Reference: Current submission, Sections 4.2–4.3, Table 3, and Appendix B.

Overall, the method combines established self-teaching, filtering, and token-selection ideas while increasing training time per step by approximately threefold. The reported gains over dense OPSD are encouraging, but without the closest baselines and equal-compute controls, they do not establish a substantial methodological advance. Small improvements over those baselines would still need to justify the additional computation and heuristic complexity.

Questions for authors

Mainly listed in Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.