- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the use of complete student reasoning trajectories to improve on-policy self-distillation and identifies the "The hindsight paradox." To address this, the authors propose PAST, which treats the complete student trajectory as additional privileged information for the teacher and adapts the teacher based on the correctness of the final answer: for correct trajectories, the teacher is aligned with the student's next-token distribution to preserve successful behaviors; for failed trajectories, the teacher is optimized using verifiable rewards while employing KL regularization to constrain the divergence between the teacher and the student. The adapted teacher then provides distillation supervision based on the student's original generation prefix, thereby maintaining the interface where the student relies solely on the current prefix during inference. Experiments demonstrate that PAST achieves an improvement of approximately 5.6 percentage points in the Avg@12 macro-average across three mathematical reasoning benchmarks compared to Vanilla OPSD, with ablation studies further supporting the synergistic effect of combining complete trajectories with teacher adaptation.
Critical strengths
-
The paper proposes the "The hindsight paradox," focusing on the question that once a teacher has obtained the complete trajectory, what information can be transferred to students who only see the current prefix?
-
The method proposed in this paper achieves an improvement of approximately 5.6 percentage points over Vanilla OPSD across three mathematical reasoning benchmarks.
-
The paper is well-written with nice figures.
Critical weaknesses
-
The paper uses counterexamples to illustrate that teacher supervision relying on future information may be cancelled out after conditional averaging. However, the experiments primarily showcase final performance, teacher success rates, and the impact of trajectory removal and shuffling. These results do not directly clarify the extent of supervision cancellation in vanilla OPSD or the degree to which PAST mitigates it; performance gains could also stem from additional teacher optimization and improved search for successful responses.
-
The primary experiments focus on Qwen3-1.7B and three competitive mathematics benchmarks, all of which rely on the verification of relatively clear final answers. It is recommended to conduct experiments with large models across different model families and to test various tasks with verifiable outcomes, such as code generation.
-
The paper has not yet sufficiently validated the necessity of the additional teacher-adaptation stage under a matched computational budget; consequently, it is difficult to distinguish whether the performance gains stem from this mechanism or simply from the additional training and sampling effort. It is recommended to include a control experiment with a matched computational budget—such as allocating the same resources to additional student GRPO or Vanilla OPSD updates—to clearly demonstrate the actual advantages of the two-stage training approach.
Questions for authors
See Weakness