- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces temporal self-distillation for RLVR, where a model learns from its own later checkpoints. It argues that the most useful teacher is not necessarily the strongest one, but a near future checkpoint that balances newly acquired capabilities with learner compatibility. The paper implements this idea through NPO, which rolls back the learner and incorporates verified future self trajectories into RL training, and NPD, which distills token-level teacher distributions on student-generated trajectories. It further proposes AutoNPO to adaptively select intervention times and rollback distances, allowing the guided learner to become a teacher for subsequent interventions. Experiments show improvements over GRPO across image-text, text-only, and video reasoning benchmarks.
Critical strengths
The paper explores a worthwhile direction for guiding RLVR through the model’s own later checkpoints, without requiring an expensive external teacher. Reusing capabilities acquired during training offers an alternative source of supervision, while considering teacher and learner compatibility rather than teacher strength alone provides a reasonable motivation for near future guidance. Also, the evaluation includes a sufficient set of relevant baselines, covering on-policy RL, external-teacher guidance, experience replay, and self-distillation.
Critical weaknesses
-
The paper does not convincingly establish whether future-self guidance changes the learner’s optimization trajectory and raises its attainable performance, or merely accelerates the acquisition of capabilities it would eventually develop through standard RLVR. This distinction is central to assessing the method’s contribution and requires more direct evidence. A particular concern arises when all learner-generated rollouts for a difficult prompt are incorrect. The injected teacher trajectory becomes one of the few trajectories receiving a positive advantage. Repeatedly reinforcing a teacher’s single cached solution may concentrate learning on that particular trajectory rather than encourage the discovery of alternative reasoning paths. The potential influence of teacher tokens that have low probability under the learner also warrants analysis. This raises the possibility that the learner develops a distribution even narrower than the teacher’s, reproducing a limited subset of its behavior rather than moving beyond it. In experimental aspect, matched-budget comparison against simply continuing RL from the teacher checkpoint is needed to establish why rolling back and retraining an earlier version of the same model is preferable.
-
The experiments do not establish whether the proposed method remains effective across different model families. This is an important limitation in assessing its generalizability, particularly because the approach depends on the model’s ability to acquire useful guidance through its own continued training. In contrast, LUFFY demonstrates the effectiveness of external teacher guidance on weaker foundation models such as LLaMA-3.1-8B. Experiments on non-Qwen backbones are therefore necessary to determine whether NPO offers a broadly applicable improvement or depends on favorable properties of the evaluated model family.
-
Repeated cycles of selecting a later checkpoint as the teacher, rolling back the learner, and replaying training introduce additional optimization steps. Consequently, the actual training budget may substantially exceed what is implied by the learner’s final checkpoint index. The paper should explicitly report the cumulative number of training updates, counting both the updates performed before each rollback and those performed during replay, and clarify whether the baselines receive the same update budget.
There is a comparison under shared GPU hour budget that accounts for AutoNPO’s rollback and guidance costs. However, matching GPU hours does not establish that the methods perform the same number of optimization updates or generate the same number of rollouts. To separate algorithmic gains from simply spending more rollout compute, performance should also be plotted against cumulative rollout counts and cumulative training steps, rather than GPU hours alone. Rollout accounting should include teacher’s generation and confirmation probes, while update accounting should include every replayed segment. Additional budget comparisons along the number of rollouts or training steps is needed to determine whether the reported advantage persists under these controls.
Questions for authors
- Why do the text-only experiments start from a LUFFY checkpoint? Table 3 evaluates the combined effect of LUFFY followed by NPO, rather than NPO applied independently to the base model. To demonstrate that future self guidance is more effective than external teacher guidance in the text-only setting, it would be helpful to apply NPO directly to Qwen2.5-Math-7B and compared with LUFFY starting from the same base model. Could the authors provide this comparison?
- Why is there no AutoNPD counterpart to AutoNPO? Is there a particular reason why adaptive temporal teacher selection is not extended to NPD?
- Are teacher checkpoints in NPD selected using the same score as AutoNPO? If not, what criterion determines the training step from which the teacher is selected?
- Could the authors discuss the scalability of the proposed approach? In particular, is the method expected to remain both effective and computationally efficient as training is scaled up?