- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
-
The paper identifies Prefix-Induced Supervision Attenuation (PISA) in on-policy distillation and provides a clear analysis of how confidence and agreement saturation weaken token-level learning signals.
-
The proposed Trajectory Dropout is simple and computationally efficient, demonstrating consistent performance improvements across multiple reasoning benchmarks, model scales, and OPD variants.
Critical strengths
-
The paper provides detailed analyses and ablation studies, including gradient-norm-matched comparisons, to better understand the effects of TD beyond simply increasing gradient magnitude.
-
TD can be easily integrated into existing OPD methods without changing the rollout process or teacher supervision. This makes the approach practical, with only a small additional computational overhead.
Critical weaknesses
-
Confidence saturation is defined by high student confidence and low teacher probability. However, teacher–student disagreement does not necessarily indicate that the student's prediction needs correction. Is there evidence that TD still provides useful corrective signals when the teacher is unreliable or performs worse than the student on certain tasks?
-
In agreement saturation, the teacher–student advantage is close to zero under full-context conditioning. This may indicate that the student already makes predictions consistent with the teacher by appropriately using earlier reasoning information. In such cases, why should the disagreement artificially introduced by removing context be considered useful supervision? Further analysis is needed to distinguish whether this actually strengthens reasoning dependencies or merely reflects the student's sensitivity to masking.
-
The paper shows that TD increases training signals in PISA regions and improves downstream performance. However, it remains unclear whether the increased signals actually cause the performance gains. Can the authors directly examine the relationship between the additional gradients generated in PISA regions and improvements in downstream task performance?
Questions for authors
-
The diagnostic experiments show that many tokens receive weak learning signals. However, there is insufficient evidence that random masking selectively recovers useful learning opportunities. The authors should justify the use of random masking by comparing it with PISA-targeted masking or other selective masking strategies.
-
It is unclear whether TD's performance gains arise from genuinely useful prefix-dependent supervision or simply from additional gradient perturbations induced by masking. Could the authors conduct a controlled experiment using artificial perturbations matched to the magnitude and distribution of TD's induced advantages to distinguish between these explanations?
4 comments
Follow-up on the PISA-Only Ablation
Thank you for the detailed responses and additional experiments.
The PISA-only experiment is informative, but it does not yet establish a PISA-specific benefit. Appendix D shows that PISA regions contain approximately 73–75% of response tokens, while PISA-only TD retains approximately 78% of TD's total performance gain (3.51/4.51 percentage points). This may reflect the proportion of tokens receiving TD rather than their PISA-specific contribution.
Could the authors compare this with applying TD to an equally sized random subset of target tokens, using the same masking procedure and training budget?
Response to "Follow-up on the PISA-Only Ablation"
We thank the reviewer for this valuable feedback. Following the suggestion, we have added the two controlled comparisons below:
-
TD only in a size-matched random subset. We randomly select a subset of response target tokens equal in size to the PISA regions and apply the signals induced by TD’s random masking only at these positions, retaining full-context OPD signals at all other positions. We use the same masking procedure, training budget, and loss normalization as PISA-only to test whether selecting PISA regions provides additional benefits after controlling for the number of tokens receiving TD signals.
-
Non-PISA-region control. We apply TD signals only to target tokens within **
**, retaining full-context OPD signals at all other positions, to further assess the gains attributable to updates outside PISA regions.
| AIME24 | AIME25 | AIME26 | Olymp. | HMMT Feb26 | HMMT Nov25 | Avg | |
|---|---|---|---|---|---|---|---|
| OPD | 34.58 | 22.92 | 23.33 | 55.41 | 20.08 | 13.33 | 28.27 |
| OPD+TD | 40.42 | 30.42 | 25.83 | 58.07 | 22.35 | 19.58 | 32.78 |
| TD only in [Designated PISA region] | 39.17 | 28.75 | 25.00 | 57.93 | 22.35 | 17.50 | 31.78 |
| **TD only in |
**|35.42|22.50|24.58|55.56|19.32|13.75|28.52| |TD only in a size-matched random subset|37.92|27.92|22.08|56.93|18.94|15.83|29.94|
The results indicate that the gains from our method arise primarily from interventions in PISA regions. If these additional experiments resolve your main concerns, we hope you will consider updating your score. We welcome any further discussion!
Gentle Reminder
Dear Reviewer,
As we near the end of the rebuttal period, I have addressed all of your concerns point-by-point. If our responses and revisions have adequately addressed your concerns, we sincerely hope you will consider adjusting the score.
We look forward to your feedback and welcome any further discussion!
Best regards,
The Authors of Submission 6429
Response to Reviewer rYZC
We are grateful to the reviewer for acknowledging our ablation experiments and the practical usefulness of TD! Our point-by-point responses to your comments follow.
W1: Reliability of teacher supervision
We agree that teacher–student disagreement does not necessarily imply that the student is wrong, and that unreliable teacher supervision may be ineffective. Our original experimental design already accounts for this concern through data filtering: we retain only training examples for which the teacher's responses are correct, thereby mitigating the influence of unreliable supervision. This setting is explicitly stated in Section 4.1 of the original manuscript:
W2: Usefulness of additional supervision under agreement saturation
We agree that teacher–student agreement under full context may reflect the correct use of earlier reasoning information and does not itself indicate a prediction error. Our concern is that direct distillation signals at these positions are near zero, which alone does not establish whether useful learning opportunities remain.
TD re-exposes weak signals through prefix interventions and evaluates the usefulness of the resulting updates under full context. Appendix D demonstrates signal re-exposure. Appendix E further shows that TD yields a greater reduction in teacher-alignment KL on held-out data even after matching the norms of the actual parameter updates.
Our evidence therefore extends beyond disagreement induced by masking to the empirical usefulness of the overall updates under full context.
Q1:Effectiveness of random masking
TD uses random masking as a lightweight mechanism for exposing attenuated learning signals. PISA identifies weak-signal target positions but does not directly specify which prefix spans to mask. Selecting spans according to their effects on PISA signals would require additional intervention evaluations during training. Our design therefore favors a simple random intervention, whose utility we examine through controlled experiments.
Specifically, we use the signals produced by TD’s random masking only for target tokens in a designated PISA region, while retaining full-context OPD signals for all other targets. All targets remain included in training, with unchanged loss normalization. This comparison directly tests whether the PISA signals exposed by random masking contribute to downstream improvements
These controlled comparisons directly assess the contribution of PISA-region interventions to downstream performance, providing intervention-based evidence for the relationship between stronger signals and improved accuracy.
W3 & Q2: Controlling update magnitude and artificial perturbations
1. Effect of update magnitude
Appendix E already provides a comparison with matched actual parameter-update norms for OPD and TD. TD still achieves a greater reduction in teacher-alignment KL on held-out data, with **an additional reduction of 0.027 (95% CI
00˙22,00˙34)**. This shows that the improvement cannot be explained solely by larger update magnitudes.
2. Artificial advantage-perturbation control
To further test whether the gains arise merely from additional perturbations, we construct artificial perturbations matched to the magnitude and empirical distribution of the token-level advantage increments induced by TD. We randomly add these perturbations to the advantages used in standard OPD training.
The comparison is as follows:
The results show that this artificial-perturbation control does not reproduce TD's gains.
Taken together, neither larger update magnitudes nor the artificial perturbations tested here suffice to explain TD's gains. These findings support the interpretation that TD benefits from structured, context-dependent supervision.
These constitute our full response. If we have addressed your main concerns, we hope you will consider revising the score. We remain happy to engage in further discussion.