acceptodds
Under review as a conference paper at ICLR 2027

What Shared Prefixes Hide: Trajectory Dropout for On-Policy Distillation

Abstract

On-policy distillation (OPD) trains a student model on its own trajectories using dense token-level feedback from a stronger teacher model. Since each update is conditioned on the reasoning prefix already generated by the student, the prefix also shapes how effectively teacher feedback is converted into learning. We find that shared prefixes can sometimes attenuate useful supervision, a phenomenon we call Prefix-Induced Supervision Attenuation (PISA). This attenuation arises in two common cases. (i) High student confidence can weaken corrective gradients even when the teacher disagrees. (ii) Tokens that rely on earlier reasoning can receive learning signals as weak as those for simple local continuations. To solve this problem, we propose Trajectory Dropout, a simple training-time intervention that exposes these weakened signals. The student first performs a standard full-context rollout to generate a complete trajectory. During training, we randomly drop a certain proportion of the student's reasoning trajectory, while the teacher continues to observe the complete trajectory for token-level supervision. This intervention strengthens corrections for overconfident predictions and restores additional supervision for prefix-dependent tokens. Trajectory dropout consistently improves performance across teacher–student model pairs of different scales and six mathematical reasoning benchmarks. It can also be directly integrated into existing OPD variants, including TRD and FastOPD, further improving their performance. These results demonstrate that trajectory dropout provides a simple and general mechanism for strengthening token-level supervision across model scales and OPD objectives.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, On-Policy Distillation, Trajectory Dropout

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rYZCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. The paper identifies Prefix-Induced Supervision Attenuation (PISA) in on-policy distillation and provides a clear analysis of how confidence and agreement saturation weaken token-level learning signals.

  2. The proposed Trajectory Dropout is simple and computationally efficient, demonstrating consistent performance improvements across multiple reasoning benchmarks, model scales, and OPD variants.

Critical strengths
  1. The paper provides detailed analyses and ablation studies, including gradient-norm-matched comparisons, to better understand the effects of TD beyond simply increasing gradient magnitude.

  2. TD can be easily integrated into existing OPD methods without changing the rollout process or teacher supervision. This makes the approach practical, with only a small additional computational overhead.

Critical weaknesses
  1. Confidence saturation is defined by high student confidence and low teacher probability. However, teacher–student disagreement does not necessarily indicate that the student's prediction needs correction. Is there evidence that TD still provides useful corrective signals when the teacher is unreliable or performs worse than the student on certain tasks?

  2. In agreement saturation, the teacher–student advantage is close to zero under full-context conditioning. This may indicate that the student already makes predictions consistent with the teacher by appropriately using earlier reasoning information. In such cases, why should the disagreement artificially introduced by removing context be considered useful supervision? Further analysis is needed to distinguish whether this actually strengthens reasoning dependencies or merely reflects the student's sensitivity to masking.

  3. The paper shows that TD increases training signals in PISA regions and improves downstream performance. However, it remains unclear whether the increased signals actually cause the performance gains. Can the authors directly examine the relationship between the additional gradients generated in PISA regions and improvements in downstream task performance?

Questions for authors
  1. The diagnostic experiments show that many tokens receive weak learning signals. However, there is insufficient evidence that random masking selectively recovers useful learning opportunities. The authors should justify the use of random masking by comparing it with PISA-targeted masking or other selective masking strategies.

  2. It is unclear whether TD's performance gains arise from genuinely useful prefix-dependent supervision or simply from additional gradient perturbations induced by masking. Could the authors conduct a controlled experiment using artificial perturbations matched to the magnitude and distribution of TD's induced advantages to distinguish between these explanations?

4 comments
AuthorsOpenReview ↗

Response to Reviewer rYZC

We are grateful to the reviewer for acknowledging our ablation experiments and the practical usefulness of TD! Our point-by-point responses to your comments follow.

W1: Reliability of teacher supervision

We agree that teacher–student disagreement does not necessarily imply that the student is wrong, and that unreliable teacher supervision may be ineffective. Our original experimental design already accounts for this concern through data filtering: we retain only training examples for which the teacher's responses are correct, thereby mitigating the influence of unreliable supervision. This setting is explicitly stated in Section 4.1 of the original manuscript:

We train on the teacher-correct English subset of DAPO-Math-17K.

W2: Usefulness of additional supervision under agreement saturation

We agree that teacher–student agreement under full context may reflect the correct use of earlier reasoning information and does not itself indicate a prediction error. Our concern is that direct distillation signals at these positions are near zero, which alone does not establish whether useful learning opportunities remain.

TD re-exposes weak signals through prefix interventions and evaluates the usefulness of the resulting updates under full context. Appendix D demonstrates signal re-exposure. Appendix E further shows that TD yields a greater reduction in teacher-alignment KL on held-out data even after matching the norms of the actual parameter updates.

Our evidence therefore extends beyond disagreement induced by masking to the empirical usefulness of the overall updates under full context.

Q1:Effectiveness of random masking

TD uses random masking as a lightweight mechanism for exposing attenuated learning signals. PISA identifies weak-signal target positions but does not directly specify which prefix spans to mask. Selecting spans according to their effects on PISA signals would require additional intervention evaluations during training. Our design therefore favors a simple random intervention, whose utility we examine through controlled experiments.

Specifically, we use the signals produced by TD’s random masking only for target tokens in a designated PISA region, while retaining full-context OPD signals for all other targets. All targets remain included in training, with unchanged loss normalization. This comparison directly tests whether the PISA signals exposed by random masking contribute to downstream improvements

AIME24AIME25AIME26Olymp.HMMT Feb26HMMT Nov25Avg
OPD34.5822.9223.3355.4120.0813.3328.27
OPD+TD40.4230.4225.8358.0722.3519.5832.78
TD only in [Designated PISA region]39.1728.7525.0057.9322.3517.5031.78

These controlled comparisons directly assess the contribution of PISA-region interventions to downstream performance, providing intervention-based evidence for the relationship between stronger signals and improved accuracy.

W3 & Q2: Controlling update magnitude and artificial perturbations

1. Effect of update magnitude

Appendix E already provides a comparison with matched actual parameter-update norms for OPD and TD. TD still achieves a greater reduction in teacher-alignment KL on held-out data, with **an additional reduction of 0.027 (95% CI

00˙22,00˙340\.022, 0\.034

)**. This shows that the improvement cannot be explained solely by larger update magnitudes.

2. Artificial advantage-perturbation control

To further test whether the gains arise merely from additional perturbations, we construct artificial perturbations matched to the magnitude and empirical distribution of the token-level advantage increments induced by TD. We randomly add these perturbations to the advantages used in standard OPD training.

The comparison is as follows:

AIME24AIME25AIME26Olymp.HMMT Feb26HMMT Nov25Avg
vanilla OPD34.5822.9223.3355.4120.0813.3328.27
OPD + artificial advantage perturbations31.2521.6726.2555.7021.9715.8328.78

The results show that this artificial-perturbation control does not reproduce TD's gains.

Taken together, neither larger update magnitudes nor the artificial perturbations tested here suffice to explain TD's gains. These findings support the interpretation that TD benefits from structured, context-dependent supervision.

These constitute our full response. If we have addressed your main concerns, we hope you will consider revising the score. We remain happy to engage in further discussion.

Follow-up on the PISA-Only Ablation

Thank you for the detailed responses and additional experiments.

The PISA-only experiment is informative, but it does not yet establish a PISA-specific benefit. Appendix D shows that PISA regions contain approximately 73–75% of response tokens, while PISA-only TD retains approximately 78% of TD's total performance gain (3.51/4.51 percentage points). This may reflect the proportion of tokens receiving TD rather than their PISA-specific contribution.

Could the authors compare this with applying TD to an equally sized random subset of target tokens, using the same masking procedure and training budget?

AuthorsOpenReview ↗

Response to "Follow-up on the PISA-Only Ablation"

We thank the reviewer for this valuable feedback. Following the suggestion, we have added the two controlled comparisons below:

  1. TD only in a size-matched random subset. We randomly select a subset of response target tokens equal in size to the PISA regions and apply the signals induced by TD’s random masking only at these positions, retaining full-context OPD signals at all other positions. We use the same masking procedure, training budget, and loss normalization as PISA-only to test whether selecting PISA regions provides additional benefits after controlling for the number of tokens receiving TD signals.

  2. Non-PISA-region control. We apply TD signals only to target tokens within **

Designatednon−PISAregionDesignated non-PISA region

**, retaining full-context OPD signals at all other positions, to further assess the gains attributable to updates outside PISA regions.

AIME24AIME25AIME26Olymp.HMMT Feb26HMMT Nov25Avg
OPD34.5822.9223.3355.4120.0813.3328.27
OPD+TD40.4230.4225.8358.0722.3519.5832.78
TD only in [Designated PISA region]39.1728.7525.0057.9322.3517.5031.78
**TD only in
Designatednon−PISAregionDesignated non-PISA region

**|35.42|22.50|24.58|55.56|19.32|13.75|28.52| |TD only in a size-matched random subset|37.92|27.92|22.08|56.93|18.94|15.83|29.94|

The results indicate that the gains from our method arise primarily from interventions in PISA regions. If these additional experiments resolve your main concerns, we hope you will consider updating your score. We welcome any further discussion!

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer,

As we near the end of the rebuttal period, I have addressed all of your concerns point-by-point. If our responses and revisions have adequately addressed​ your concerns, we sincerely hope you will consider adjusting the score.

We look forward to your feedback and welcome any further discussion!

Best regards,

The Authors of Submission 6429

Reviewer_Mu4yOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates token-level gradient dynamics in On-Policy Distillation (OPD) and identifies Prefix-Induced Supervision Attenuation (PISA), a phenomenon where conditioning on shared student-generated prefixes severely weakens learning signals across ~74% of response tokens. The authors decompose PISA into two mechanisms: confidence saturation, where high student confidence suppresses downward corrective gradients despite teacher disagreement, and agreement saturation, where near-zero teacher–student advantage conflates nontrivial long-range reasoning steps with routine local continuations. To mitigate PISA, they propose Trajectory Dropout (TD), a training-time intervention that decouples standard full-context on-policy rollouts and full-prefix teacher scoring from masked-view student optimization. During parameter updates, TD randomly masks contiguous spans of the student's reasoning history while optionally preserving a local protection window, thereby amplifying corrective gradients for overconfident tokens and inducing disambiguating advantage signals for prefix-sensitive tokens. Norm-controlled one-step ablations and channel decompositions demonstrate that TD's benefits stem from superior gradient alignment with the teacher rather than increased update magnitudes or generic noise regularization.

Critical strengths
  • The paper identifies and formalizes Prefix-Induced Supervision Attenuation (PISA), decomposing gradient vanishing in OPD into confidence saturation and agreement saturation with rigorous empirical diagnostics. This is a very interesting direction to improve OPD.

  • Trajectory Dropout decouples full-context on-policy rollouts from masked-view optimization, restoring corrective gradients and exposing long-range prefix dependencies with negligible wall-clock overhead.

  • Norm-controlled one-step update experiments and two-channel decompositions confirm that improvements stem from superior gradient alignment rather than step-size scaling or generic noise regularization.

  • Demonstrates robust accuracy improvements across six mathematical reasoning benchmarks, out-of-domain tasks (MBPP+, GPQA), and multiple OPD objectives (standard OPD, TRD, Fast-OPD).

Critical weaknesses
  • Empirical validation is limited to the Qwen3 family at small scales (0.6B and 1.7B students with a 4B teacher), leaving scalability to frontier models (>7B) and cross-family distillation unverified. It would be interesting and more convicing to see performance after scaling up the model size or spanning on model families.

  • Random contiguous span dropout is content-agnosticm, and it does not explicitly target the diagnosed PISA regions (R_I, R_(II)) and risks masking critical reasoning premises alongside redundant context. How could we explain or solve this?

  • Aggressive prefix masking occasionally reduces student probability below teacher probability (r_t < p_t), inverting the downward corrective gradient direction (~8% of confidence-saturated tokens); the proposed sign-consistent fallback in Appendix F.6 is not evaluated in main training runs.

  • The paper may have limit training-time thinking evaluation. Models are trained exclusively in non-thinking mode on a teacher-correct math subset (DAPO-Math-17K), leaving TD's training dynamics on native long-horizon chain-of-thought rollouts unexplored.

Questions for authors

Please refer to the weakness. Happy to adjust my assessment if the authors can fully resolve my questions.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Mu4y(1/2)

We are grateful to the reviewer for this thoughtful and encouraging review. In particular, we appreciate your recognition of our work: the identification and formalization of PISA, the simplicity and low overhead of TD, and its broad effectiveness—across mathematical reasoning, out-of-domain tasks, and multiple OPD objectives. We also thank you for the positive assessment of our controlled-update-norm experiments and our analysis of the underlying mechanism. We now respond to each of your points in turn, with additional clarifications and new results.

W1: Performance with larger models

We add a training comparison using a larger teacher–student pair, with the following setup:

  • Teacher: DeepSeek-R1-Qwen3-8B.
  • Student: Qwen3-4B-Instruct.
  • Training data: DAPO-17K. The evaluation results are as follows: ||AIME24|AIME25|AIME26|HMMT Feb26|HMMT Nov25|Olymp.| |---|---|---|---|---|---|---| |No-train baseline|60.42|45.83|52.08|31.44|42.08|70.62| |OPD|65.42|57.08|64.17|34.85|42.92|75.52| |OPD+TD|67.92|59.17|67.50|37.50|48.75|77.00|

TD also yields performance improvements with this larger teacher–student pair, providing further support for its applicability at larger model scales.

W2: Effectiveness of random masking and disruption of useful context

We agree that random contiguous-span masking does not explicitly target PISA regions and may affect useful intermediate information. TD adopts a lightweight random intervention, whose effects are examined directly. With trajectories and region assignments held fixed, Appendix D shows that the signal-mass share of the same PISA tokens increases from approximately 0.5% under full context to 16.39%–22.88% under maskingThis demonstrates that random masking can re-expose weak signals in these regions without explicitly targeting them.

To limit context disruption, TD leaves the original problem prompt intact and preserves causal access within each contiguous span. We do not assume that all masked intermediate information can be recovered.

  • Figure 6aa: The local-protection ablation shows that preserving nearby access substantially mitigates the performance loss.

  • Figure 6bb: The model's performance is robust to the dropout rate within a certain range; however, it deteriorates when the dropout rate becomes too high.

These results support the overall benefits of the intervention within the evaluated range, while also showing that its strength and the structure of local access need to be controlled.

W3: Full-training comparison with the sign-consistent fallback

We have added a full-training comparison with the sign-consistent fallback.

Teacher: Qwen3-4B-Instruct

Student: Qwen3-1.7B no-thinking

AIME24AIME25AIME26Olymp.HMMT Feb26HMMT Nov25Avg
OPD+TD40.4230.4225.8358.0722.3519.5832.78
OPD+TD+sign-consistent41.6728.7528.7558.2621.5920.0033.17

Their overall performance remains similar, indicating that enabling this safeguard has a limited effect on final performance in the evaluated setting.

This result is consistent with the diagnostic observations in Appendix F.6: approximately 8% of confidence-saturated tokens exhibit an advantage sign reversal, and these tokens contribute less than 5% of the total signal mass across all tokens.

AuthorsOpenReview ↗

Response to Reviewer Mu4y(2/2)

W4: Training evaluation in native thinking mode

Many prior OPD-related studies conduct both training and evaluation exclusively in non-thinking mode, including EOPD(ICML 2026), SDPO(ICML 2026), and Vision-OPD (NeurIPS 2026).

In Table 1, although training uses non-thinking mode, TD's gains carry over to thinking-on inference.

We add a training comparison using long reasoning trajectories generated by the student in thinking mode.

Teacher: qwen3-4b thinking mode

Student: qwen3-1.7b thinking mode

AIME24AIME25AIME26Olymp.HMMT Feb26HMMT Nov25Avg
No-train baseline53.3334.1738.7566.8128.4127.9241.57
vanilla OPD55.0033.3341.2567.0027.2728.7542.10
OPD+TD53.7535.8340.4267.1131.8231.2543.36

We observe that both OPD and OPD+TD yield limited performance gains when trained in native thinking mode. One possible explanation concerns exploration within long reasoning trajectories. Thinking mode gives the student more scope for intermediate attempts, verification, and backtracking. The value of these behaviors depends on their contribution to subsequent reasoning, which may not align with the teacher’s probability preferences for the current token. Consequently, while correcting errors, token-level distillation may also restrict exploration paths that are useful to the student, limiting the net gains. TD improves the conversion of teacher supervision into learning signals but retains the underlying distribution-matching objective, and may therefore share this limitation.

The above constitutes our full response. If you have any remaining questions or concerns, we would be glad to elaborate further.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer,

As we near the end of the rebuttal period, I have addressed all of your concerns point-by-point. If our responses and revisions have adequately addressed​ your concerns, we sincerely hope you will consider adjusting the assessment.

We look forward to your feedback and welcome any further discussion!

Best regards,

The Authors of Submission 6429

Reviewer_Mu4yOpenReview ↗

Official comments from Reviewer Mu4y

Thanks for the authors for their detailed responses. Most of my concerns have been resolved. I will maintain my positive score.

Reviewer_zTUWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper identifies two supposed problems in on-policy distilation (OPD), namely confidence saturation and agreement saturation. The former arises for highly confident student predictions that the teacher disfavors, while the later arrises when the two models assign similar log probabilities to a token. The proposed main idea to fix these problems is simple: the authors propose to drop a certain proportion of the student’s reasoning trajectory.

Critical strengths
  • The simplicity of the proposed approach is an advantage, especially since it can easily compose with other proposed improvements to OPD.
Critical weaknesses
  • I find the paper difficult to read. The introduction is quite repetitive, restating the same ideas and obeservations several times without necessarily providing a different perspective or complementary insight. The introduction also makes claims without introducing the necessary background and without providing direct evidence or a forward reference to the experiments. To give one example, in "The latter are qualitatively different from routine local continuations because their predictability depends on earlier reasoning" it's not clear how the authors measured or concluded qualitative differences, what counts as routine vs non-routine local continuations, or what predictability means. As another example the authors say "Figure 2A shows its uneven distribution in a full-context audit." The details on how this was computed (which datasets, which model, etc.) are not provided, either in line or in the caption, nor is "full-context audit" defined. There many are other such sentences that make the paper hard to read. Relatedly, the authors refer to the student's reasoning trajectory and "alter the student’s access to earlier reasoning" but it's not clear if this refers only to reasoning tokens (e.g. <think> ... </think>) and if yes, why are they given special treatment. Especially since in section 4.1 the authors state "Both students use non-thinking mode".
  • The signal analysis (Appendix D) that motivates the approach is only done on one model Qwen3-1.7B student at only a few steps (10, 40, and 70) and is not clear how well it generalises beyond this.
  • The biggest gains seemed to be in the "Non-thinking inference" setting, while for "thinking inference" the gain is much more modest. Moroever, the results are all on a relatively small subset of student-teacher pairs.
  • While confidence saturation and agreement saturation are indeed phenomena that can be observed in this setting, I find the authors arugments for why we need to correct for them not entirely convincing. The authors do point out that a lot of tokens (~75% according to the two regions the authors define) are problematic since they contribute small percentage of own-logit signal mass, but how this is related to e.g. generalisation is not clear.
  • The authors start from sampled OPD (Eq. 1), which while valid and popular, is not the "canonical" Full-vocabulary KL OPD loss, and they do not discuss whether PISA also affects to Full-vocabulary KL OPD, or whether TD provides a similar boost in that setting.
Questions for authors
  1. Can you clarify your wording around "reasoning trajectory" and which subsets of tokens are eligible for dropout?
  2. Can you include additional student-teacher pairs, including at least one larger model where the base performance is stronger to see if the benefits extend to this case?
  3. Can you test whether Full-vocabulary KL OPD also benefits from TD?
  4. Similarly, given the popularity of OPD, there are many extensions, and since TD is an orthogonal idea it can be applied on top, can you add it on top of any of these OPD extensions (e.g. any of the papers that you already cite)?
  5. Can you discuss the connection to Noisy OPD (NOPD) by Wang et al. in more detail? Beyond the superficial difference that Wang et al. study Vision-Language models, the proposed dropout can be seen as one particular form of corruption.
3 comments
AuthorsOpenReview ↗

Response to Reviewer zTUW (1/2)

We are grateful to the reviewer for the thoughtful comments, and we provide our clarifications and additional results in response to each question below.

W1 & Q1: Clarifying terminology, Figure 2A, and masking scope

To begin, we reiterate our central argument: OPD provides dense token-level supervision, but dense supervision does not necessarily translate into dense learning signals. We identify two forms of attenuation: confidence saturation weakens teacher-directed corrective updates, while agreement saturation can leave potential opportunities for further learning unexploited. TD partially restores these weakened signals through prefix intervention.

In response to the reviewer's comment, we have revised the Introduction to remove redundant information and improve clarity. Below, we clarify the terminology, the analysis underlying Figure 2A, and the scope of masking.

1. Routine local continuations and prefix dependence

By “routine local continuations,” we mean tokens that can be accurately predicted using only the local context near the target position. In contrast, predictions that depend on earlier reasoning require intermediate results outside that local window. Predictability refers to the conditional probability that the model assigns to the target token given the available context.

The key distinction is that similar teacher–student advantages under full context do not imply the same prefix dependence. Appendix D quantifies changes in predictions and training signals through paired prefix interventions while keeping the target tokens fixed; it does not preclassify all tokens into these two categories. We have revised the corresponding explanation in the Introduction.

2. Data and computation underlying Figure 2A

Figure 2A uses the same data as Appendix D and shows the distribution of training signals under full context, without TD. We will explicitly reference Appendix D in the caption and add the model, data, and computation details.

3. Meaning of “reasoning trajectory” and token eligibility for masking

In this paper, “reasoning trajectory” refers to the student's entire generated response, rather than specifically to content within <think>...</think> tags. Non-thinking denotes the model's generation mode; the response itself may still contain step-by-step reasoning. All response positions except the final one are eligible to be selected as masked source positions.

The prompt is never masked, and all response tokens are retained and remain supervised.

W2 & W3 & Q2: Signal analysis across model scales and training with larger models

1. Signal phenomena across model scales

Our observations are not limited to Qwen3-1.7B. We also observe similar signal phenomena in Qwen3-0.6B and Qwen3-4B-Instruct, providing further evidence that they occur across model scales. The corresponding observations will be supplemented in Appendix D.

2. Training results for a larger teacher–student pair

We additionally compare OPD and OPD + TD using a larger teacher–student pair:

  • Teacher: DeepSeek-R1-Qwen3-8B.

  • Student: Qwen3-4B-Instruct.

  • Training data: DAPO-17K.

The evaluation results are as follows:

AIME24AIME25AIME26HMMT Feb26HMMT Nov25Olymp.
No-train baseline60.4245.8352.0831.4442.0870.62
OPD65.4257.0864.1734.8542.9275.52
OPD+TD67.9259.1767.5037.5048.7577.00

TD also yields performance improvements with this larger teacher–student pair.

W3: Transfer to thinking-mode inference

Many prior OPD-related studies conduct both training and evaluation exclusively in non-thinking mode, including EOPD (ICML 2026), SDPO (ICML 2026), and Vision-OPD(NeurIPS 2026).

We train both OPD and OPD + TD in non-thinking mode and evaluate them not only in non-thinking mode but also with thinking enabled. Enabling thinking at inference changes the generation conditions and the distribution of reasoning trajectories. Under thinking-mode inference, TD improves the average math score from 45.40 to 47.29 +18˙9percentagepoints+1\.89 percentage points. This improvement shows that its gains carry over to an inference mode not used during distillation, providing evidence of generalization across inference modes.

AuthorsOpenReview ↗

Response to Reviewer zTUW (2/2)

W4: Potentially missed supervision opportunities under saturation

Our concern is that saturation may leave useful supervision opportunities underutilized. Under confidence saturation, highly confident predictions that the teacher disfavors still receive weak corrective signals, limiting the use of the teacher's corrective feedback. Under agreement saturation, both predictions that depend on earlier history and routine local continuations can receive near-zero advantages, potentially obscuring remaining learning opportunities at the former positions.

The approximately 75% figure indicates the prevalence of weak signals; it does not imply that all these tokens require stronger updates. We assess the value of intervention empirically:

  • Appendix D: TD re-exposes signals at a subset of these positions.

  • Appendix E: TD achieves a greater reduction in teacher-alignment KL on held-out data even after matching the norms of the actual parameter updates.

  • Main experiments: TD improves performance on mathematical reasoning and out-of-domain tasks under standard full-context inference.

Together, these results provide empirical support for the training utility and generalization benefits of prefix interventions.

W5 & Q3: Applicability to full-vocabulary KL OPD

To directly test whether TD also benefits full-vocabulary KL OPD, we add a training comparison with the following setup:

  • Teacher: Qwen3-4B-Instruct.

  • Student: Qwen3-1.7B.

  • Distillation objective: Full-vocabulary KL.

The evaluation results are as follows:

AIME24AIME25AIME26Olymp.HMMT Feb26HMMT Nov25Avg
Full-vocabulary KL OPD32.9221.6721.2554.6718.1812.0826.79
Full-vocabulary KL OPD + TD36.2525.0025.0055.6321.9714.5829.74

Adding TD also improves performance in this experiment, supporting its applicability to full-vocabulary KL OPD.

Q4: Experiments with existing OPD extensions

We would like to clarify that experiments combining TD with existing OPD extensions are already included in the original manuscript. The abstract summarizes this finding, and the main results table1 provides the corresponding comparisons.

Specifically, we apply TD to two cited OPD extensions, TRD and Fast-OPD, and compare each with its counterpart without TD. Tables 1 and 3 report the following average math scores:

OPD extension1.7B student: original method → with TD0.6B student: original method → with TD
TRD18.76 → 22.688.24 → 9.29
Fast-OPD29.48 → 33.7616.79 → 18.56

Both extensions improve in average score at both student scales, supporting TD's compatibility with existing OPD extensions.

Q5: Connection to and differences from NOPD

We agree that both methods use perturbations to strengthen supervision, but the perturbed information plays different roles. NOPD perturbs the original evidence provided in the problem—the image. TD preserves the complete problem statement and masks parts of the history in the student's self-generated trajectory. TD therefore addresses how self-generated history can weaken the effectiveness of teacher supervision even when the task evidence remains intact, and how to alleviate this attenuation (PISA).

These are all of our responses. We look forward to continuing the conversation, and if we have addressed your primary concerns, we would appreciate your reconsideration of the score.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer,

As we near the end of the rebuttal period, I have addressed all of your concerns point-by-point. If our responses and revisions have adequately addressed​ your concerns, we sincerely hope you will consider adjusting the score.

We look forward to your feedback and welcome any further discussion!

Best regards,

The Authors of Submission 6429

Comments on the paper

2 comments
AuthorsOpenReview ↗

Kind request regarding reviewer discussion

Dear Reviewers:

As the discussion period draws to a close, we wanted to reach out to see whether our responses have fully addressed your questions, and whether any points still require further clarification.

We are grateful for the time and care you have invested in reviewing our submission. We have carefully considered your feedback and responded to each point you raised.

If our responses have resolved your main concerns, we respectfully hope you might consider updating your score to reflect these revisions. Should anything remain unclear, we would be glad to provide further clarification before the discussion period ends.

Thank you once again for your valuable feedback and consideration.

Best regards,

The Authors of Submission 6429

AuthorsOpenReview ↗

General Response

We thank all reviewers for their thoughtful feedback and for recognizing the identification and analysis of PISA (Mu4y, rYZC), TD's simplicity and ease of integration (zTUW, rYZC), its computational efficiency (Mu4y, rYZC), the value of our gradient analyses and controlled ablations (Mu4y, rYZC), and its consistent performance improvements across benchmarks and OPD variants (Mu4y, rYZC).

Following their suggestions, we provide the following revisions, clarifications, and additional experiments:

  • Presentation and terminology (zTUW W1/Q1): We streamline the Introduction and clarify the terminology, masking scope, and experimental setup underlying Figure 2A.

  • Model scales (zTUW W2–W3/Q2; Mu4y W1): We extend the signal analysis to 0.6B and 4B models and add an 8B-teacher/4B-student experiment, where TD improves performance on all six mathematical benchmarks.

  • OPD objectives and compatibility (zTUW W5/Q3–Q4): We add full-vocabulary KL training experiments showing an approximately improvement and clarify that experiments combining TD with TRD and Fast-OPD were already reported in the original submission.

  • Native thinking-mode training (zTUW W3; Mu4y W4): We additionally train on student-generated thinking-mode trajectories.

  • Contribution of PISA-region updates (zTUW W4; Mu4y W2; rYZC W2–W3/Q1 and follow-up): With the same masking procedure and training budget, PISA-only TD achieves 31.78, compared with 29.94 for an equally sized random target subset and 28.52 for non-PISA-only TD, confirming that TD's performance gains primarily come from PISA-region.

  • Artificial perturbation control (rYZC W3/Q2): Complementing our existing norm-matched analysis, we add artificial advantage perturbations matched to TD’s increment magnitude and empirical distribution, ruling out matched random advantage perturbations as the explanation for TD’s performance gains.

  • Sign consistency (Mu4y W3): We evaluate the sign-consistent fallback through full training, obtaining similar average performance to standard TD (33.17 versus 32.78).

  • Teacher reliability (rYZC W1): We clarify the teacher-correct data filtering used in training and acknowledge that teacher–student disagreement does not necessarily indicate student error.

  • Relationship to NOPD (zTUW Q5): We clarify that TD preserves the original problem input while restricting access to the student’s generated history, whereas NOPD corrupts the input image.

Review Status

As of this writing, reviewer rYZC has provided a follow-up on our PISA-only ablation. In response, we conducted two additional controlled experiments, providing further evidence that TD's performance gains primarily arise from interventions in PISA regions. We are currently awaiting rYZC's feedback on these new results. Reviewers Mu4y and zTUW have not yet responded to our rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.