- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SEL-OPSD, an extension of on-policy self-distillation (OPSD) for reasoning LLMs that selectively reuses generated trajectories at two levels. At the trajectory level, HINTSELECT samples multiple current-policy rollouts, selects the shortest verifier-correct trajectory for student training, and uses it as the teacher’s privileged hint when it is sufficiently shorter than the reference solution. At the token level, KEYSTEP selects distillation positions based on student uncertainty, teacher–student disagreement, mathematical-token priors, and an answer-region bonus. Experiments on Qwen3-1.7B/4B/8B across mathematical reasoning benchmarks report improvements over dense OPSD in Avg@12 and reduced deployment response length.
Critical strengths
-
Decomposition of OPSD supervision. The paper separates student-trajectory selection, teacher privileged-context selection, and token-level supervision, which is a meaningful way to analyze and extend OPSD. The paper separately ablates student-trajectory selection and teacher-hint replacement, which helps distinguish the contribution of each mechanism rather than attributing all gains to the combined method.
-
Reasonable token-level selection mechanism. KEYSTEP is well motivated as a form of hard-example mining: uncertainty and teacher–student disagreement identify positions where privileged supervision may be most useful, while the fixed token budget keeps the additional computation small.
Critical weaknesses
-
Clarity is insufficient. The paper is unevenly structured: for example, Section 3 is very short, while the main-results section mostly restates the numbers without sufficiently explaining the observed trends or why the proposed components should lead to them. More importantly, some reported results are internally inconsistent. The Qwen3-8B/AIME24 values in Figure 4 do not match those in Tables 2/3, and some reported seed-level statistics are difficult to reconcile with the stated evaluation protocol. Figure 2 also contains a clear arithmetic error: for , trajectories ending in are marked as correct, although the correct solution is . These issues make the presentation harder to trust and the empirical conclusions harder to verify.
-
The logic and attribution of HINTSELECT are not fully convincing. HINTSELECT always chooses the shortest verifier-correct rollout as the student trajectory and may also use it as the teacher hint. However, being shorter and having the correct final answer does not necessarily mean that the trajectory contains better reasoning or is a better privileged hint, since the verifier mainly checks answer correctness rather than the validity of intermediate reasoning. This also introduces an explicit preference for short trajectories, which is particularly relevant since reduced response length is one of the paper’s main reported benefits. The current ablation does not isolate this effect: the “trajectory-only” variant still selects the shortest correct rollout, so it mixes correctness filtering with length-based selection. A random or uniformly sampled correct-trajectory baseline would be needed to determine whether the gain actually comes from choosing shorter trajectories rather than simply training on verified-correct ones.
Questions for authors
- Can the authors add a randomly selected verified-correct trajectory baseline at the same and compute? This would separate correctness filtering from the additional shortest-trajectory bias and clarify whether the response-length reduction is caused by improved reasoning or explicit length selection.
- Can the authors clarify the HINTSELECT algorithm used in the experiments? The main text defines the candidate pool as including both sampled rollouts and the reference solution, while Algorithm 2 selects the shortest trajectory only among verified sampled rollouts. Please clarify which version was implemented, when the student trajectory is replaced, and under what conditions the reference remains the teacher hint.
- Can the authors clarify the methodological contribution of KEYSTEP beyond generic hard-token selection? In particular, a comparison against a matched top- baseline using the actual per-token OPSD loss would help determine whether the proposed multi-signal score provides benefits beyond standard hard-example mining.