acceptodds
Under review as a conference paper at ICLR 2027

Learning Dynamics of On-policy Distillation

Abstract

On-policy distillation (OPD) is emerging as an important paradigm for language-model post-training, combining dense teacher supervision with training on student-generated trajectories. This creates a feedback loop where each update changes the student and thereby the training distribution that generates subsequent updates. Yet, how this feedback shapes learning and generalization remains poorly understood. We conduct a mechanistic analysis of OPD's learning dynamics, distinguishing the two feedback loop stages: conversion of the distillation signal into parameter updates, and broadcast through which these updates change predictions across prefixes and reshape the rollout distribution. This framework helps explain both limited student responsiveness despite substantial teacher-student disagreement, and degeneration when updates that improve local teacher agreement also increase the probability of unsuccessful continuations, exposing the student to unreliable supervision in subsequent rollouts. Finally, we show that offline distillation can prepare a student for OPD without significantly improving immediate held-out accuracy. While OPD alone may degrade generation, it can yield substantial accuracy gains after sufficient offline preparation. Together, our results connect teacher supervision, student geometry, and the evolving rollout distribution in determining the learning dynamics of OPD and its effects on downstream performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Learning Dynamics, On-policy Distillation, Reasoning, Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aaDQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper develops an analytical framework for understanding the learning dynamics of on-policy distillation (OPD). It separates the OPD feedback loop into conversion and broadcast. Conversion characterizes how teacher supervision translates into parameter updates through accessibility and coherence, explaining why supervision of similar strength can induce different degrees of parameter change. Broadcast characterizes how these updates alter predictions and subsequent training data through drift and routing, providing a mechanistic account of degeneration driven by increasingly incoherent student-generated prefixes. Motivated by this analysis, the authors show that offline distillation can prepare students for subsequent OPD by improving their ability to continue reasoning, even when preparation yields little immediate improvement in held-out accuracy.

Critical strengths

The paper proposes a systematic analytical framework that decomposes the entire OPD training process. It provides explanations for common failure modes, including training stagnation and degeneration, and presents several interesting findings.

Critical weaknesses

Limited novelty from the methodology side. The offline-preparation intervention does not clearly demonstrate the added practical value of the proposed measurements. Although the preparation stage uses distributional distillation rather than standard hard-target SFT, it serves a similar role by training the student on teacher-generated reasoning trajectories before on-policy learning. The effectiveness of SFT-based cold starts is already well established in the community. The paper does not use its proposed measurements to improve the preparation procedure—for example, by selecting preparation data, determining its duration, or deciding when to switch to OPD. Consequently, the preparation results support the value of a familiar intervention, but provide limited evidence that the new analytical framework enables better training decisions.

Questions for authors

Please establish good connection between the proposed measurements and the training methodology.

5 comments
AuthorsOpenReview ↗

Response to Reviewer aaDQ, Part 1

Thank you for your excellent suggestion!

To ensure that we are on the same page, we will first give a summary on the overall contributions of our paper, clarifying the paper’s insights on OPD’s learning mechanisms. We will then present a new experiment that uses continuation readiness to assess the preparation checkpoints, and show how the new results not only suggest practical measurement for OPD-readiness, but also in furthering our mechanistic insights on OPD. The results show that this measurement can distinguish checkpoints whose held-out accuracies are nearly identical, but whose subsequent OPD performance differs substantially.

1) Main contributions: mechanistic insights, diagnostics, and motivated interventions

At a very high level, the paper has two main contributions, building on two main components of the dynamics of OPD: conversion (request motion) and broadcast (motion outcome correctness).

For the conversion part, we contribute early-training diagnostics beyond the usual gradient norm or training loss: given a student model, if you train against a teacher whose request coherence is poor, or tangent-accessibility stays weak, do not expect significant student policy motion along the request field (let alone benchmark improvement), even if the gradient norm or training loss is large. This part also corroborates the industry's increasingly common choice of using RL-sibling models as teachers [1-2], where we see the RL-sibling teachers emphasizing requests on reasoning-state management / control positions, which as shown in our experiments, are much friendlier to request-to-motion conversion than the larger teachers' technical-oriented requests.

For the broadcast part, the paper's main contribution is to show the dynamical mechanism behind how ill-prepared models can be harmed by OPD, and how offline preparation improves the subsequent OPD (and rescue the failing run). The intuition for "benign" OPD is that it is similar to "tutorials", where the teacher corrects the student in real time and the student responds well. However, for an ill-prepared student, it can locally follow the teacher requests but fall into chaos in its continuations. In the real OPD runs, we see the student (never trained for long thinking rollouts) pushing itself to reason for very long under the RL’d teacher, but fails to manage its reasoning states and becomes high-entropy in its rollouts, making it difficult to rescue, let alone improve on. The natural intervention is to teach the student proper reasoning continuations from the reasoning teacher model, before exposing the student's rollouts to the teacher for finer corrections online ("go to class before going to the tutorial!"). We will also show later with our new experiments that the continuation-readiness mechanism can be used as a probe to assess offline-preparation's usefulness for the subsequent OPD phase.

AuthorsOpenReview ↗

Response to Reviewer aaDQ, Part 2

2) Preparation and continuation readiness

Now let us move on to the measurement-to-training question you raised. We add a practical continuation-readiness probe for assessing offline preparation before OPD. Our new results show that this probe distinguishes preparation checkpoints whose held-out accuracies are nearly identical but whose subsequent OPD gains differ substantially. These findings give our mechanistic analysis a practical use, while providing complementary evidence for the important role of continuation competence in OPD’s failure and recovery.

2.1) Continuation readiness recap and practical probe

Continuation readiness measurement. In its essence, continuation readiness splits the rollout into the part generated by a prefix-generating policy, and the part by a continuation policy (to finish the rest of the rollout), discussed in Section 3.3 and Appendix E.4 and E.7. When analyzing the raw learning dynamics of the student model during OPD, the prefix generating policy is simply the continually updated student policy, as the theory and mechanistic analyses needed to estimate how the “prefix quality” and “continuation readiness” co-evolved under OPD.

Practical probe. In practice, to isolate how ready the offline-prepared checkpoints are for the subsequent OPD, we just need to fix a reference prefix-generating policy. We generate a bank of reference prefixes once, and then reuse it to evaluate the checkpoints on their terminal correctness. The teacher model is one practical reference: intuitively, we may consider its prefix population a “ceiling” on where the subsequent OPD can shift a well-prepared student’s prefix distribution towards. Of course, in practice, practitioners can use reasoning prefixes that test more specific continuation skills they are interested in.

What can this practical probe reveal and NOT reveal?. Let us first consider what this probe can NOT measure, to avoid confusion about its scope of usefulness and OPD’s failure mechanisms. This probe directly supplies teacher-generated prefixes to the student to continue (which contain coherent, useful reasoning from the teacher). In contrast, in the degenerate OPD run, the unprepared student has significant trouble learning to reach such high-quality prefixes. In fact, as we see in Figure 6 and the associated analysis in Appendix E.4 and E.7, the student's prefix distribution moves in the "opposite direction": the rollouts grow increasingly high-entropy, and the chances of reaching the correct answer from the student’s prefix distribution becomes lower and lower, with this symptom showing up at increasingly shallow rollout / prefix depth as training progresses.

Therefore, this teacher-prefix continuation probe, which freezes the prefix distribution to an already-strong one, is not designed to predict whether the subsequent OPD will simply collapse since it directly ablates out prefix-visitation shift. It is, however, suited for assessing and ranking how prepared checkpoints are for the subsequent OPD: it can help us answer "is prepared checkpoint A more competent in continuing this distribution of already-strong prefixes than checkpoint B?", and distinguish checkpoints whose immediate held-out accuracies are similar but whose subsequent OPD outcomes differ substantially due to their differing continuation-readiness.

Mechanistic connection. In sum, the progression from the mechanistic analysis of the degenerate run to the practical probe is that, the failure analysis first shows how an unprepared student can follow teacher requests locally, yet become less likely to reach the right answer as OPD degrades the reasoning states (prefixes) it visits and its continuations from them. This motivates offline preparation to improve the student’s ability to sustain and complete reasoning before exposing its own rollouts to online teacher corrections. The practical teacher-prefix probe then holds the reasoning contexts fixed and measures whether preparation improves this continuation capability, helping us assess preparation’s utility for the subsequent OPD phase.

AuthorsOpenReview ↗

Response to Reviewer aaDQ, Part 3

2.2) New experimental results

Results and interpretations. We focus our experimental resources on the Qwen2.5-1.5B-Instruct runs as the main “model organism”, as it was also used in our degenerate-run analysis in Section 3.3, and that its offline-preparation induced the most deceptive changes in heldout accuracy. Below are the new results, where we freeze a set of teacher prefixes (1024 tokens and non-finished) on MATH, and ask the prepared checkpoint to continue the rest of the rollout. This is repeated over two random seeds for training.

Offline-preparation passesHeld-out accuracy before OPDContinuation success on fixed teacher prefixesOPD accuracy change against base model
049.0%52.7%-9.8 %
145.4%55.5%+1.8 %
245.0%59.4%+8.1 %
446.4%64.5%+10.2 %

Interpretations. First, let us look at the second column, immediate heldout accuracy of the offline-prepared checkpoints. We find that one and two preparation passes give nearly the same accuracy, and even four passes do not improve beyond the base model (the 0-pass model). So from immediate heldout accuracy alone, one might naively conclude that preparation has provided little benefit, or that there is little reason to continue preparation after the first pass of training.

In contrast, the continuation-readiness probe (third column) identifies increasing readiness for OPD: it clearly distinguishes the 1- and 2-pass checkpoints despite their nearly identical heldout accuracies, and ranks the four-pass checkpoint highest. The subsequent OPD gains indeed follow this ordering. Therefore, the continuation-readiness probe identifies useful preparation that checkpoint accuracy alone would miss in our experimental sweeps.

The practical lesson is that, even when your offline-prepared model’s held-out score looks flat or worse against the base model, do not panic! Measure how well it continues prefixes from a fixed reference policy, such as the teacher. Improved continuation success rate can reveal preparation whose value emerges during subsequent OPD, beyond further offline training.

Remark. Of course, we should also be aware of the broader qualifications here: we are NOT saying that offline distillation generally cannot boost a model’s heldout accuracy at all. This classic algorithm certainly has its place, which in our Qwen2.5-7B case does show its positive effects: for this model, offline preparation gives some lift to its accuracy beyond the base model as seen in Section 3.4. The important lesson is that continuation-readiness can yield a complementary metric to the immediate heldout accuracy metric for assessing how useful the offline-preparation is: even if immediate accuracy gain is minimal, the subsequent OPD gain can be significant, and you can test for it using continuation-readiness probes!

2.3) Further mechanistic insights

These results nicely complement our analysis of the degenerate OPD run in Section 3.3. We now have complementary evidence for the continuation-readiness mechanism behind OPD’s failure and recovery.

Recall that, as the degenerate run proceeded, the student’s rollouts became high-entropy and less likely to reach the right answer. In particular, prefixes visited later during training became harder to complete correctly, with this difficulty appearing at increasingly shallower prefix depths. Our new results probing the prepared checkpoints, which holds the reference prefix distribution fixed, give complementary “positive evidence”: as continuation readiness improves with preparation, the subsequent OPD gains also improve. The continuation-readiness mechanism exposed by the failure analysis therefore gives us a useful way to assess offline preparation. Together, the failure analysis and preparation results support a central role for continuation competence in OPD’s failure and recovery.

Reviewer_RQT9OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

The paper studies the learning dynamics of OPD. It asks the question of why learning from a stronger teacher can lead to worse results. The explanation offered is that the student can learn to reason for longer without knowing how to finish, and then trains on its own confused answers. The authors find that practicing on teacher-written off-policy solutions first can help the student perform better.

Critical strengths

The authors make an attempt to explain the learning dynamics of OPD.

Critical weaknesses

The paper is difficult to follow. The central ideas is hard to identify. It uses unusually wording and is potentially heavily AI generated. Here are some phrases that give away that the paper is heavily AI generated:

  • digit-bearing positions
  • greater stalled pressure / greater healthy pressue
  • model organism
  • epistemic-control requests / epistemic behaviors
  • arm / stratum

The paper looks incredibly dense in content, yet feels very empty.

Questions for authors

Could you summarize the main ideas and takeaways of this paper in a clear manner?

Reviewer_eqc1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes on-policy distillation as a feedback loop between teacher supervision, student updates, and changing rollout distributions. The author(s) decompose learning into the conversion of supervision into parameter motion, and the broadcast of updates across prefixes, and they distinguish fixed-prefix prediction changes from changes in prefix visitation. Experiments investigate stalled transfer, generation collapse, and the deferred benefits of offline preparation using mathematical reasoning tasks and several student–teacher pairs.

Critical strengths
  1. Useful diagnostics beyond training loss. Frozen-request kernel measurements and crossed checkpoint–rollout evaluations distinguish changes in student responsiveness from changes in the histories that are supervised. These provide informative evidence about why improving local teacher agreement can coexist with worsening generation.
  2. Evidence for preparation’s deferred value. The preparation-dose study shows that similar initial accuracy can precede different OPD outcomes. Comparisons against continued offline distillation, matched in prompt blocks, and other methods strengthen the practical relevance.
Critical weaknesses
  1. The connection between the geometric diagnostics and actual optimization is insufficiently validated. The conversion audit uses a restricted, noncentered diagnostic field, while the response laws assume raw gradient dynamics. The appendix explicitly excludes the complete optimizer step from these diagnostics (Sections C.2, D.3, E.2). In addition, the main kernel measurements emphasize high-request sites from only eleven initial rollouts. Therefore, the reported accessibility/coherence differences do not establish that these quantities govern the actual updates responsible for successful or stalled transfer.
  2. The evidence does not isolate the proposed failure-and-recovery mechanism. Crossed-bank accounting describes changes in predictions and sampled histories, but bank replacement can also reflect prompt-composition differences (Section E.1). Offline preparation changes several aspects of the student simultaneously, so its benefits do not uniquely establish improved continuation competence as the mediator. All training runs use a single seed and evaluation bootstraps do not establish robustness of the training dynamics. These limitations matter because the main contribution is a mechanistic explanation rather than a new training method.
Questions for authors
  1. Have you checked whether the conversion and broadcast diagnostics predict the direction and magnitude of observed policy changes under the complete implemented objective and actual optimizer updates?
  2. What evidence distinguishes improved continuation competence from other effects of offline preparation? For example, do matched-prefix continuation-value measurements across preparation doses predict subsequent OPD gains beyond starting accuracy?
6 comments
AuthorsOpenReview ↗

Response to Reviewer eqc1, Part 1

Thank you for the careful reading and these thoughtful questions! We will first share the new experimental results and discuss what they tell us about OPD’s failure and recovery (primarily on your Weakness 2 and Question 2). Briefly speaking:

  • Prompt composition control. The routing-drift pattern in Section 3.3 persists when evaluated on fixed prompts. The student still shifts toward high-entropy reasoning prefixes while fitting the teacher locally.
  • Continuation competence probes. Continuation readiness distinguishes preparation checkpoints whose held-out accuracies are nearly identical but whose subsequent OPD gains differ substantially. This supports its practical use for assessing preparation and strengthens the mechanistic connection between continuation competence and preparation’s benefits.
  • Training seeds. We repeated the preparation study with a second training seed and reproduced the key observations in Sections 3.3 and 3.4.

We will then clarify the measurements and design choices (primarily on your Weakness 1 and Question 1):

  • Request field. The conversion analysis in Section 3.2 uses the exact request fields logged during training.
  • Optimizer generality. We will justify how conversion diagnostics remain meaningful under AdamW, with further empirical support from our preconditioned coherence measurements. The broadcast analysis already includes the effects of the actual optimizer updates.
  • Rollouts and request sampling. In Section 3.2, our rollout pool already yields over 2,000 carefully annotated sentences per student scale, requiring extensive annotation resources.This pool gives us enough evidence to clearly distinguish the teachers’ main differences in semantic emphasis, even after accounting for statistical uncertainty. Furthermore, these differences become more pronounced as request norm increases, motivating our focus on how the student converts the largest-norm requests.

1) W2 (i) - Does the degenerate run’s failure dynamics persist when prompt composition is controlled?

This is a very nice question, as it points to a competing explanation for the dynamics we observe! We therefore ran a fixed-prompt control, and find that the dynamics of routing-dominated entropy climb followed by drift-dominated loss reduction on unreliable prefixes persist under this prompt-composition control.

Let us first make precise the control and its purpose. The original cross-bank routing-drift analysis follows what actually happened during training: it uses the prompts and rollouts sampled at the corresponding training steps. To understand the actual learning dynamics of the degenerate run, this is the natural analysis to conduct, and reported in our Figure 5. However, this leaves an interesting possibility to examine: perhaps later batches contain harder prompts, which themselves lead the student into longer and less reliable reasoning! Now, if the same pattern persists when every checkpoint receives the same prompts, then changes in prompt composition cannot explain it. This gives us a concrete control.

We therefore hold the prompts to those sampled at step 0, generate rollouts from checkpoints at steps 0, 20, 40, 80, and 100, and apply the same cross-bank estimators. Below are the new results.

Readout functionCheckpointRouting contributionDrift contributionReadout value
Entropy0––0.693
Entropy20+0.947-0.0401.599
Entropy40+3.080+0.0054.685
Entropy80-1.445-0.3902.850
Entropy100-0.130-0.0542.666
Loss0––0.714
Loss20+0.200-0.1670.747
Loss40-0.113-0.2750.358
Loss80+0.222-0.2900.290
Loss100+0.085-0.0570.319

Here, routing and drift contributions refer to changes in consecutive checkpoint intervals; the last column gives the readout function’s value at the end of each interval. Values are in reported in nats.

The interpretation is quite clear:

  • The early entropy climb is routing-dominated. Entropy then stays large, in the band of around 2 - 5 nats.
  • Loss climbs in the initial phase of training, just as in Figure 5 – the model is pushed into unfamiliar reasoning regions, and local fitting has not taken over the learning process yet.
  • Drift then increasingly dominates the larger loss-reducing contribution during steps 20–80. Particularly, during 40–80, it reduces loss despite routing pushing loss upward.

Therefore, the core dynamics of the local readouts do reproduce under this prompt-composition control. Basically, under prompt-under control, the student still moves into high-entropy reasoning regions on the same prompts, and subsequently reduce its local loss by fitting the requests on those unreliable visited prefixes. We will incorporate this control into the revised appendix to complement Figure 5.

AuthorsOpenReview ↗

Response to Reviewer eqc1, Part 2

2) W2 (ii) and Q2 - What evidence connects preparation to continuation competence?

We also appreciate this suggestion: comparing checkpoints on the same prefixes gives us a direct way to examine the continuation capability highlighted by the failure analysis. We now compare preparation doses on a fixed bank of teacher-generated prefixes (more specifically, comparing their continuation success rate on this bank of teacher prefixes). As we will see, this probe helps distinguish checkpoints whose held-out accuracies are nearly identical but whose subsequent OPD performance differs substantially.

2.1) From failure analysis to controlled continuation probe

Let us first recall our analysis in Section 3.3. At any rollout depth, successful completion depends on two things: which prefixes the student reaches, and how successfully it continues from those prefixes. If we only let each checkpoint complete its own rollouts, these two effects change together. Our failure analyses of the degenerate OPD runs separate them by fixing the prefix-generating policy while varying the continuation policy, and the other way around (detailed in Section 3.3 and Appendix E.4, E.7).

Recall that the degenerate run showed both problems: as training progressed, the student visited prefixes that were harder to complete at increasingly shallow rollout depth, and its continuation success also declined when we held the prefixes fixed. Essentially, while the request field from the teacher does broadcast into significant motion of the student along future rollouts, they struggle to reach strong reasoning prefixes, along with degrading continuation competence in reaching the correct answer. This suggests the natural intervention of offline preparation: directly expose the student to the teacher’s high-quality prefixes while teaching it the right predictions, before exposing its own rollouts to the teacher's online corrections.

The same separation also gives us a practical way to assess this preparation. We generate a bank of reference prefixes once, and then reuse it to evaluate how often each preparation checkpoint reaches the correct answer. Here, the teacher supplies the reference prefixes. Because every checkpoint receives the same prefixes, differences in the probe measure how successfully the checkpoints continue the same reasoning contexts.

2.2) New experimental results

We focus our experimental resources on Qwen2.5-1.5B-Instruct as the main “model organism”, since it was also used in our degenerate-run analysis and its preparation produces particularly deceptive changes in held-out accuracy. We freeze a set of teacher prefixes on MATH (1024 rollout tokens and not terminated), and ask each prepared checkpoint to continue the rest of the rollout. The results below are averaged over two training seeds.

Offline-preparation # of passesHeldout accuracy before OPDContinuation success on fixed teacher prefixesOPD accuracy change against base model
145.4%55.5%+1.8%
245.0%59.4%+8.1%
446.4%64.5%+10.2%

First, let us look at the second column, the immediate held-out accuracy of the offline-prepared checkpoints. 1 and 2 preparation passes give nearly the same accuracy, and even 4 passes barely improves beyond the 1-pass checkpoint (just 1% higher). So from immediate heldout accuracy alone, one might conclude that extended preparation has provided little benefit, or that there is little reason to continue preparation after the first pass.

In contrast, the continuation-readiness probe identifies increasing readiness for OPD: it clearly distinguishes the 1- and 2-pass checkpoints despite their nearly identical held-out accuracies, and ranks the four-pass checkpoint highest. The subsequent OPD gains clearly follow this trend. Therefore, the probe identifies useful preparation that checkpoint accuracy alone would miss in this sweep.

Mechanistic connection. These results nicely complement our analysis of the degenerate OPD run. The failure analysis showed us how both the visited prefixes and the student's continuations from them can degrade. The new experiment holds the reference prefix distribution fixed and gives complementary “positive evidence”: as continuation readiness improves with preparation, subsequent OPD gains also improve. Together, the failure analysis and preparation results support a central role for continuation competence in OPD's failure and recovery. Of course, preparation can also change other properties, including the eNTK, but our argument does not require continuation competence to be the sole mediator – we just need to show that it is indeed an important contributor to the success and failure dynamics of OPD, and that it is practically useful to in the (offline and online) distillation pipeline.

AuthorsOpenReview ↗

Response to Reviewer eqc1, Part 3

4) W1 (i) - What request vectors are actually used for our diagnostics?

We apologize for the unclear writing in the original manuscript. Here is how exactly we construct the diagnostic request field for the request-conversion analysis in Section 3.2. At every training step, we store the student rollouts, the student and teacher predictions, and the actual request vector B(s)B(s) at every prefix ss used during training. This vector is the actual negative logit gradient of the training loss. Therefore, at every token position sampled for the conversion analysis, the coherence, accessibility, and eigendecomposition measurements use the exact same centered request vector logged during training. We will make this clearer in the appendix.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall response

We have uploaded a revised manuscript incorporating the reviewers’ feedback. Our key updates are highlighted in blue. We thank the reviewers for the thoughtful questions and suggestions! Below, we summarize the main updates and where to find them in the revised manuscript.

1. Clearer explanations of the paper’s mechanistic insights

Following reviewer RQT9’s feedback, we have clarified the flow of our analysis: from conversion of teacher requests into student policy motion, to broadcast of that motion into future rollouts and reasoning outcomes. The new Figure 1 illustrates these insights at a high level, and we now give more intuitive explanations before introducing the formalism. We have also further clarified the mechanistic contribution and interpretation in Section 1 and Appendix A.

2. New experiments strengthening the mechanistic analysis and its practical usefulness

Following Reviewers eqc1 and aaDQ’s suggestions, we have added the following results:

  • Prompt composition control. The routing-drift pattern in Section 3.3 persists when evaluated on fixed prompts: the unprepared student still shifts toward high-entropy reasoning prefixes while fitting the teacher locally. This further supports the analysis in Figure 5. The new control is reported in Appendix E.8.

  • Continuation-readiness probe. By testing continuation success on the same teacher-generated prefixes, we distinguish preparation checkpoints whose immediate held-out accuracies are nearly identical but whose subsequent OPD gains differ substantially. This gives a practical way to assess preparation, building on the failure mechanism of the degenerate OPD run in Section 3.3. The results are reported in Appendix F.3.

  • A second training seed. We repeated the Qwen2.5-1.5B preparation-dose study with a second training seed. Both seeds reproduce the key trend in Section 3.4: more preparation improves subsequent OPD performance, even when immediate held-out accuracy shows little benefit. The individual seed results are reported in Appendix F.3, Table 15, and discussed in Appendix A, Q2.

3. Clearer explanations of the measurements and design choices

We have also clarified the following points raised by Reviewer eqc1:

  • Request field clarification? The conversion analysis uses the exact request vectors logged at every position during training. The kernel-adaptation probe holds the initial request fields fixed to isolate how the student’s tangent features change during training. These measurement choices are clarified and further explained in Appendices D.3, D.4 and D.5.

  • Optimizer generality? We explain how the conversion diagnostics remain meaningful under AdamW and why the kernel-adaptation probe excludes the preconditioner matrix. The new preconditioned coherence measurements preserve the healthy-vs-stalled contrast: at 7B, the healthy run retains about twice the coherence of the stalled run on the large-normed requests. The broadcast measurements already use the actual AdamW-trained checkpoints and therefore include the effects of the complete optimizer updates. These points are addressed in Appendices C.2 and D.6.

  • Are we using enough samples in Section 3.2, and why study the large-normed requests? Our rollout pool contains over 2,000 carefully annotated sentences per student scale, with manual checking and correction. The teachers’ main differences in semantic emphasis remain clear after accounting for uncertainty across rollouts. These differences also become more pronounced as request norm increases, motivating our focus on how the student converts the strongest requests. The evidence and annotation procedure are given in Section 3.2, Figure 3(a), and Appendix D.2.

We thank the reviewers again for helping us improve both the clarity of the paper, mechanistic insights, and the connection between our measurements and analysis decisions!

Dear Reviewers,

As the discussion period is ending soon, please let us know if there is any further concern you would like us to address, or parts that need further clarifications. Thanks!

Best, Authors of Submission 30860

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.