acceptodds
Under review as a conference paper at ICLR 2027

PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation

Abstract

On-policy self-distillation (OPSD) uses a privileged teacher to supervise a reasoning model on prefixes sampled from its own rollouts. Yet each rollout also reveals how the student’s response unfolds and whether it succeeds, hindsight that standard OPSD does not use to form the teacher. We introduce Privileged Adaptation from Student Trajectories (PAST), which treats each completed student trajectory as additional privileged information for the OPSD teacher while leaving the student’s distillation prefixes unchanged. PAST preserves the student’s next-token distribution on correct trajectories and adapts the teacher on failed trajectories toward verified success while keeping it close to the student. We characterize what a teacher informed by the trajectory can transfer to a prefix-only student. Forward-KL distillation projects the teacher distributions to their conditional arithmetic mean given the prefix. This projection separates variation across trajectories that remains privileged from the mean policy shift available to the student. For correct trajectories, the unclipped population objective also has the frozen student as an ideal distributional fixed point. Across three mathematical reasoning benchmarks, PAST improves the Avg@12 macro average over Vanilla OPSD by 5.6 percentage points. A 2×2 factorial study shows gains from complete trajectories and teacher adaptation, while trajectory removal and shuffling confirm that the adapted teacher uses the matching hindsight context.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy self-distillation, privileged information, language model reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FVRCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the use of complete student reasoning trajectories to improve on-policy self-distillation and identifies the "The hindsight paradox." To address this, the authors propose PAST, which treats the complete student trajectory as additional privileged information for the teacher and adapts the teacher based on the correctness of the final answer: for correct trajectories, the teacher is aligned with the student's next-token distribution to preserve successful behaviors; for failed trajectories, the teacher is optimized using verifiable rewards while employing KL regularization to constrain the divergence between the teacher and the student. The adapted teacher then provides distillation supervision based on the student's original generation prefix, thereby maintaining the interface where the student relies solely on the current prefix during inference. Experiments demonstrate that PAST achieves an improvement of approximately 5.6 percentage points in the Avg@12 macro-average across three mathematical reasoning benchmarks compared to Vanilla OPSD, with ablation studies further supporting the synergistic effect of combining complete trajectories with teacher adaptation.

Critical strengths
  1. The paper proposes the "The hindsight paradox," focusing on the question that once a teacher has obtained the complete trajectory, what information can be transferred to students who only see the current prefix?

  2. The method proposed in this paper achieves an improvement of approximately 5.6 percentage points over Vanilla OPSD across three mathematical reasoning benchmarks.

  3. The paper is well-written with nice figures.

Critical weaknesses
  1. The paper uses counterexamples to illustrate that teacher supervision relying on future information may be cancelled out after conditional averaging. However, the experiments primarily showcase final performance, teacher success rates, and the impact of trajectory removal and shuffling. These results do not directly clarify the extent of supervision cancellation in vanilla OPSD or the degree to which PAST mitigates it; performance gains could also stem from additional teacher optimization and improved search for successful responses.

  2. The primary experiments focus on Qwen3-1.7B and three competitive mathematics benchmarks, all of which rely on the verification of relatively clear final answers. It is recommended to conduct experiments with large models across different model families and to test various tasks with verifiable outcomes, such as code generation.

  3. The paper has not yet sufficiently validated the necessity of the additional teacher-adaptation stage under a matched computational budget; consequently, it is difficult to distinguish whether the performance gains stem from this mechanism or simply from the additional training and sampling effort. It is recommended to include a control experiment with a matched computational budget—such as allocating the same resources to additional student GRPO or Vanilla OPSD updates—to clearly demonstrate the actual advantages of the two-stage training approach.

Questions for authors

See Weakness

Reviewer_FicBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the limitations of OPSD, and analyzes what a privileged teacher can teach students within a theoretical frame. Then, this paper suggests PAST (a variant of OPSD); update the teacher to teach the students well and then, update the students with OPD. The authors found that PAST improves upon OPSD across three mathematical benchmarks.

Critical strengths
  1. This paper addresses a timely problem raised in many papers that OPSD doesn't work under certain conditions.
  2. This paper presents an in-depth analysis of why simply giving hindsight to the teacher fails and proposes PAST as a solution.
Critical weaknesses
  1. The approach to the problem looks reasonable but it incurs additional cost from training the teacher with GRPO. Especially, baselines compared in this paper are H2SD, SD-Zero which are also computationally expensive. But there are some critical problems that these baselines have not been well verified. Thus, although PAST appears to outperform them I wonder how it would perform if the computational budget is matched to that of OPSD and GRPO. So I think budget matched comparison should be included.
  2. PAST algorithm incorporates procedure of training teachers. Therefore I think the authors also need to compare it with OPD, soft distillation, hard distillation method using other different teacher model.
Questions for authors
  1. Can you show the results of other model in addition to Qwen-3-1.7B? Authors only present the results with a single model.
  2. Can you present other results with other method suggested in weakness section?
Reviewer_YAgGOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PAST, an on-policy self-distillation method that adapts a privileged teacher using completed student trajectories. It preserves the student’s distributions on correct trajectories and optimizes verified success with student-proximity regularization on failed ones. The adapted teacher is distilled on the original student prefixes. The analysis characterizes which trajectory-conditioned changes can transfer to a prefix-only student.

Critical strengths
  • The motivation is clear, the method exploits information already available in student rollouts while preserving the deployed student’s prefix-only interface.

  • The design is coherent, distribution preservation on correct trajectories complements success-seeking adaptation on failed trajectories.

  • The projection analysis and counterexample explain why privileged teacher success need not translate into student improvement. This is useful theoretial perspective.

Critical weaknesses
  • The distinction from existing hindsight-based methods and trainable-teacher distillation is not fully established; the basic KL projection identity is standard. So the novelty is limited.

  • The analysis assumes unclipped population KL, whereas training uses clipping and shared parameters. Teacher continuation success also does not establish improvement under frozen-student continuations.

Questions for authors
  • Does PAST retain its advantage under matched GPU-time or total-compute budgets? Could more evidence be provided?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.