acceptodds
Under review as a conference paper at ICLR 2027

Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals

Abstract

On-policy distillation (OPD) enables effective capability transfer between language models, yet the mechanisms underlying its failures are not fully understood. Across code generation and mathematical reasoning, OPD with larger-scale teachers exhibits early loss plateaus, with an average final loss reduction of 25.1% after 200 updates, compared with 96.2% for self-RL teachers, obtained by further reinforcement learning (RL) training of the initial student. To understand this difference, we analyze OPD as an idealized continuous-time dynamical system in the small-learning-rate limit. Our diagnostics link these plateaus to premature learning-signal collapse, where the gradient becomes weak relative to the remaining loss. We further prove a local recovery guarantee for teachers sufficiently close to the initial student in a shared parameterization under regularity conditions, offering a conditional explanation for the success of self-RL teachers in our experiments. Across runs with and without loss plateaus, we observe small relative parameter changes (0.025–0.098%) and high similarity between the student's representations before and after OPD (linear CKA across layers). These observations suggest that limited representation adaptation may contribute to learning-signal collapse, a hypothesis that remains to be tested.

Primary area:
reinforcement learning
Keywords:
On-policy distillation, Knowledge distillation, Large language models, Optimization dynamics, Learning-signal collapse, Representation analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TnvdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the failure modes of on-policy distillation (OPD), particularly in the large-to-small distillation setting. The authors first observe that when the teacher is an RL-optimized variant of the student model, the distillation loss can decrease to nearly zero, allowing the student to almost fully recover the teacher's performance. However, when the teacher is a larger model than the student, the distillation loss quickly plateaus, hindering further learning and preventing the student from effectively closing the performance gap. The authors then provide detailed mathematical derivations to explain these observations, and further support their analysis with empirical evidence from weight updates and Centered Kernel Alignment (CKA) experiments.

Critical strengths
  1. The studied problem is important, large-to-small distillation is a very practical and promising technique in building small foundation models, while the pipeline does not work well currently.

  2. The theoretical and empirical analysis are well-conducted and self-contained.

Critical weaknesses
  1. My first concern is about the contribution and insights. The conclusions and the empirical analysis (e.g., weight update analysis) in this paper are well-known. The theoretical analysis provides some new insights (I did not check the math derivations carefully).

  2. Although the theoretical analysis is novel, I would have expected the authors to propose a method to mitigate the identified problem. Understanding the underlying mechanisms, while valuable, seems insufficient on its own to constitute a complete contribution. A more compelling study would build on these insights to develop and evaluate a targeted solution to address the identified limitations of large-to-small OPD.

Questions for authors

See weakness above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Tnvd

Dear Reviewer Tnvd,

We greatly appreciate your thoughtful feedback and constructive suggestions. Below, we provide detailed responses and clarifications. If our responses address your concerns, we would be grateful if you would consider updating your score.

W1. The conclusions and the empirical analysis, such as the weight-update analysis, are well known

Earlier work reported that OPD from a larger teacher can stall and that post-training changes weights little. The paper characterizes when OPD stalls and how, and explains why it works with self-RL teachers. The new results strengthen each part.

PartResultEvidence
When it stallsWith the student fixed, OPD converges with teachers RL-trained from the student and stalls to different degrees with larger ones. No optimizer setting removes the plateau: a larger learning rate, longer training, relaxed clipping, a full-vocabulary objective and SFT initialization all leave it in place.Response to Reviewer C7ft, W1.1; general response (1/2); Section 4.3
How it stallsThe gradients of different prompts remain but stop agreeing, so the learning signal collapses while most of the loss remains. This is measured with an independent estimator instead of the logged gradient norm.General response (2/2); response to Reviewer C7ft, W2.1
Why self-RL teachers workA local recovery guarantee for teachers close to the student. Its low-curvature residual is 2.2–2.6% of the initial quadratic loss for JustRL-1.5B and 0.7% for DeepScaleR-1.5B, so the guarantee is informative for these teachers.Response to Reviewer hvW7, Q2

DeepScaleR-1.5B is a same-size public model RL-trained from our initial student on other data (response to Reviewer C7ft, W1.1). On the weight-update analysis, see "Small parameter and representation changes (Section 6)" in general response (2/2): the size of the change does not decide the outcome; its direction does.

W2. The paper should propose and evaluate a method that mitigates the problem

We thank the reviewer for raising this. The paper does not offer a mitigation. Its contribution is a diagnosis of when OPD stalls, how it stalls, and why it works with self-RL teachers. We did try the natural remedies, and none removes the plateau: SFT on the teacher's responses; SFT followed by OPD (Section 4.3); a full-vocabulary objective and more rollouts per prompt (Appendix H); and, in this rebuttal, a larger learning rate, longer training and relaxed clipping (general response 1/2). We therefore think that simple changes to the training recipe are unlikely to solve it: how far OPD gets is set mainly by the teacher and the student themselves. Consistently, changing the teacher gains 4.3 to 6.9 accuracy points, against about one at most for any optimizer setting.

Reviewer_C7ftOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Previous work has found that the loss during on-policy distillation (OPD) of LLMs for reasoning post-training stagnates in certain conditions. The paper shows that this occurs when distilling from a larger student, and that using from a teacher that is the same model as the student, but RLed on the data, removes this issue. The authors provide some theoretical arguments for why this is rooted in optimization issues, and show additional empirical results on how parameters and representations barely change during OPD.

Critical strengths

Despite the many works on on-policy distillation, its theoretical properties are still poorly understood, and efforts towards understanding it are welcome and greatly benefit the community. The reporting of the results is careful and honest.

The self-RL baseline in the experiments is an interesting baseline, it shows that OPD from the same student (but already more trained) is much better behaved than OPD from a larger teacher.

Critical weaknesses

While the self-RL teacher baseline in the empirical section is interesting, it confounds, as far as I understand, same training data and same model size. And, this is the only new contribution (along some minor ablations) of that section: the idea that larger teacher are worse teachers, and that OPD loss stagnates precedes this work.

The empirical results show that distilling from larger teachers yields to loss stagnating, but the theory fails short of explaining it. Section 5.1 (and Figure 2) is, as far as I can tell, exactly what we would expect from loss stagnating or not (assuming α\alpha is large enough). Section 5.2 is making some progress in that direction, but only, as far as I understand, for teachers that are of the same size as the student.

The section on parameter changes and changes in representations is not really informative: it is exactly what we would expect from on-policy learning and relatively small number of steps, in particular with a small learning rate value of 1e-6.

As a result, the paper does not provide convincing evidence that the loss stagnation is due to optimization issues rather than teacher-student capacity gap.

Questions for authors

What would happen when increasing the learning rate? Making optimization longer? Would we expect leaving that local optima or being stucked there indefinitely?

4 comments
AuthorsOpenReview ↗

Response to Reviewer C7ft (1/2): W1, W2

Dear Reviewer C7ft,

We greatly appreciate your thoughtful feedback and constructive suggestions. Below, we provide detailed responses and clarifications. If our responses address your concerns, we would be grateful if you would consider updating your score.

W1.1 The self-RL baseline confounds same training data with same model size

Same data is not required. DeepScaleR-1.5B (the public checkpoint agentica-org/DeepScaleR-1.5B-Preview) is obtained, like JustRL-1.5B, by RL fine-tuning of our initial student R1-Distill-1.5B, but on other math data (AIME 1984–2023, AMC, Omni-MATH and STILL; 13% of our training prompts appear in it). OPD converges with it as with JustRL-1.5B, so the converging teachers share the student's size, not its data.

StudentTeacherSizeLoss reductionAccuracy gain
R1-Distill-1.5BJustRL-1.5Bsame96.3%+11.9
DeepScaleR-1.5B (new)same97.8%+7.4
R1-Distill-7B7B35.4%+0.5
Skywork-7B7B37.7%+5.9
JustRL-R1-7B (new)7B52.7%+4.8
Qwen3-1.7BQwen3-4B4B30.4%+2.5
Qwen3-4B-RL-Math4B65.1%+23.6
Qwen3-4BQwen3-30B-A3B (new)30B55.3%+30.7

Loss reduction as in Table 2; accuracy gain over the initial student, last five validation points. The R1-Distill-7B row is the two-seed baseline of general response (1/2); (new) marks teachers not in the submission.

W1.2 That larger teachers are worse and that the OPD loss stagnates precede this work

See the response to Reviewer Tnvd, W1.

W2.1 Section 5.1 and Figure 2 are what one expects from a stagnating loss

See "Is the weak learning signal more than a restatement of the plateau?" in general response (2/2). The per-response gradients of the estimator described there show what a stagnating loss does not imply (teacher R1-Distill-7B; LR 1e-6 unless stated):

CheckpointSquared norm of per-response gradientsTheir average pairwise cosineμ\mu [95% CI]Sampling-noise share of the logged squared gradient norm
Initial student1.200.421.81 [1.65, 1.97]2%
Step 500.480.030.059 [0.047, 0.072]37%
Step 2000.630.020.044 [0.037, 0.052]48%
LR raised to 3e-6 at step 50; step 1500.650.0020.005 [−0.005, 0.015]93%

The loss stops falling because the gradients of different prompts stop agreeing, not because each of them vanishes, and the logged gradient norm of Figure 2 understates the collapse. On α\alpha: with DeepScaleR-1.5B the point estimates of 1+α1 + \alpha at 8 checkpoints are 0.13 to 1.69, and 4 of the 8 intervals exclude zero. With R1-Distill-7B the 95% interval at step 50 is [−55, +40], too wide to determine the sign, because ∥g∥2\Vert g \Vert^2 is small and there is one response per prompt.

W2.2 The recovery result covers only teachers of the same size as the student

Correct. The theorem needs a shared parameterization, so it explains why nearby teachers are reached and says nothing about larger ones. For the plateau we have measurements, not theory. Its premise holds for a second same-size teacher: the low-curvature residual is 0.7% for DeepScaleR-1.5B (response to Reviewer hvW7, Q2).

AuthorsOpenReview ↗

Response to Reviewer C7ft (2/2): W3, W4, Q1

W3. The section on parameter and representation changes is not informative

See "Small parameter and representation changes (Section 6)" in general response (2/2).

W4. No convincing evidence that the stagnation is an optimization issue rather than a capacity gap

The paper does not exclude a capacity gap, and we agree that it is likely part of the explanation. Two results are consistent with it. No optimizer setting moves the plateau (general response 1/2). And when we train the student from step 50 on a fixed cache of its own responses (64 prompts, 2,048 positions, full-vocabulary KL, LR 1e-5, 150 updates), the KL on those positions falls by 97%, but the fit does not carry over to other responses to the same prompts (fixed-context KL / initial 0.94, against 0.85 for the same run with freshly sampled responses).

It is not all of the explanation, because some larger teachers transfer far more than others of the same size (the larger teachers of the table under W1.1):

StudentLarger teacherTeacher sizeLoss reductionAccuracy gain
R1-Distill-1.5BR1-Distill-7B7B35.4%+0.5
Skywork-7B7B37.7%+5.9
JustRL-R1-7B7B52.7%+4.8
Qwen3-1.7BQwen3-4B4B30.4%+2.5
Qwen3-4B-RL-Math4B65.1%+23.6
Qwen3-4BQwen3-30B-A3B30B55.3%+30.7

Qwen3-30B-A3B is over seven times the size of its student, yet the loss falls by 55.3% and accuracy rises by 30.7 points. Qwen3-4B-RL-Math and Qwen3-4B have the same size, yet the gains are 23.6 and 2.5 points. What the student takes from a larger teacher depends on the teacher, not only on its size. We will state in the paper that the plateau is consistent with a capacity gap and clarify wording that reads as an optimization-only account.

Q1. With a larger learning rate or longer optimization, does the student leave the local optimum or stay stuck?

It stays at the same distance from the teacher. See "Larger learning rate, longer training, and clipping" in general response (1/2). The student keeps moving, but not toward the teacher (fixed-context KL as defined in general response 1/2):

Run (teacher R1-Distill-7B)Relative parameter change, as in Table 4Fixed-context KL / initial
LR 1e-6, step 2000.050%0.85
LR 1e-6, step 4000.082%0.82
LR 3e-6, step 2000.12%0.86
LR 3e-6, step 4000.19%0.86
LR 1e-5 from step 50, step 2000.27%0.95

I acknowledge the author's rebuttal. While its new results go in the right direction, I remain skeptical that it is indeed a global minimum and not just a temporary plateau (training for longer / with larger learning is not enough to prove that it's not a plateau, same for the ||g||^2/L metric). In addition, I do believe that the new insights the paper provides are too limited for the paper to be accepted.

Dear Reviewer C7ft,

Thank you for the follow-up.

1. Temporary plateau. Within our compute budget, we ran 11 additional OPD training runs in math and code: learning rates 3× and 10× larger, twice as many updates, and relaxed or no clipping (general response 1/2). Due to limited compute, we could not train longer than this. In none of these runs does the loss resume falling: the fixed-context KL is 0.821 at step 300 and 0.822 at step 400, and a 10× larger learning rate moves the student five times farther while the KL rises from 0.85 to 0.95. These results are strong evidence that the plateau is not removed by the training budgets and optimizer settings used in practice.

2. This is not the main claim of the paper. The paper nowhere states that the plateau is a minimum. Its contributions are:

  • The phenomenon: OPD plateaus early with larger teachers, and keeps converging with teachers RL-trained from the student (self-RL teachers).
  • The mechanism: the plateau is explained by the collapse of the learning signal.
  • To our knowledge, the first theoretical guarantee that OPD with a self-RL teacher succeeds.

3. New insights. We respectfully disagree. The review raised two specific concerns, and both have been addressed.

  • The self-RL comparison mixed training data and model size. DeepScaleR-1.5B is trained on other data and gives the same result (response to W1.1).
  • Section 5.1 looked like a restatement of the plateau. The new measurements show more: per-response gradients keep their size but cancel across prompts (response to W2.1).

The review also notes that the theory of OPD is poorly understood. Work on it benefits the community.

Reviewer_hvW7OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies why on-policy distillation (OPD) sometimes fails despite using stronger teachers. It provides theoretical analysis and empirical evidence across code generation and mathematical reasoning tasks.

Critical strengths

This paper is clearly written overall, so it is easy to follow. It establishes a well-motivated problem and a logical progression from empirical observations to theoretical analysis. The experiments are reasonably complete and clearly presented.

Critical weaknesses

The proposed explanation remains somewhat correlational. The paper does not demonstrate whether weak learning signals cause OPD failures or simply accompany them. Plus, there is a gap: while the theoretical guarantees use idealized gradient dynamics and restrictive assumptions, the actual experiments use adamw and a truncated distillation objective.

Questions for authors
  1. Figure 2 measures the learning signal using pre-clipping gradient norms. I think examining whether the actual adamw update magnitudes also diminish during the observed plateaus, considering gradient clipping and adaptive preconditioning, will be a more interesting and convincing.
  2. Theorem 5.2 relates OPD recovery to teacher–student distance and the initial Fisher spectrum. Can the authors evaluate the low-curvature residual term for their self-RL teachers to assess whether the theoretical bound explains the observed convergence?
  3. I wonder for the larger-scale teachers exhibiting plateaus, does increasing the learning rate or modifying gradient clipping restore loss reduction? Such a controlled comparison could help distinguish insufficient effective updates from an intrinsic limitation of the distillation objective.
3 comments
AuthorsOpenReview ↗

Response to Reviewer hvW7 (1/2): W1, W2, Q1

Dear Reviewer hvW7,

We greatly appreciate your thoughtful feedback and constructive suggestions. Below, we provide detailed responses and clarifications. If our responses address your concerns, we would be grateful if you would consider updating your score.

W1. Do weak learning signals cause the failure, or only accompany it?

See "Is the weak learning signal more than a restatement of the plateau?" in general response (2/2). An independent estimator shows the same collapse, and the weak signal is the immediate reason the loss stops falling: to first order, one update removes 0.16% of the loss in the plateau, against 4.2% at the start (Q1 below).

W2. The guarantees use idealized gradient dynamics, while the experiments use AdamW and a truncated objective

We measured both differences (teacher R1-Distill-7B, LR 1e-6).

Training (analysis)MeasurementInitial studentStep 50Step 200
Reverse KL on the student's top-16 tokens (full KL)Cosine between the gradients of the two objectives, per response (16 responses of the initial student)0.9940.9920.980
Same, for gradients averaged over the 64 prompts of a training batch0.9990.60—
AdamW with clipping at 1 (gradient flow)Update norm and loss removed per updatesee Q1

Per response the top-16 gradient stays aligned with the full-KL gradient, also at larger learning rates (0.963 at LR 3e-6 and 0.895 at LR 1e-5, step 200). Averaged over prompts the two separate by step 50, because the top-16 average is itself close to zero by then. The theorem is a statement about gradient flow with same-size teachers, not a numerical prediction for these runs.

Q1. Do the actual AdamW updates also shrink during the plateau?

Yes. We logged the parameter change Δθm\Delta\theta_m of every optimizer step mm (student R1-Distill-1.5B, LR 1e-6, 200 updates). Each cell is the mean over the first 5 updates → the mean over the last 50. ηN\eta\sqrt{N} (η\eta the learning rate, NN the number of parameters) is the largest norm an Adam step can have, and g^\hat{g} is the minibatch gradient. DeepScaleR-1.5B is a same-size public model RL-trained from our initial student on other data, and JustRL-R1-7B is a second 7B teacher (response to Reviewer C7ft, W1.1).

TeacherPre-clip gradient normUpdate norm ∥Δθm∥/(ηN)\Vert\Delta\theta_m\Vert/(\eta\sqrt{N})First-order loss decrease per update, −⟨g^,Δθm⟩/L-\langle \hat{g}, \Delta\theta_m \rangle/\mathcal{L}Clipped updates
R1-Distill-7B, seed 10.665 → 0.105 (÷6.3)0.731 → 0.204 (÷3.6)4.16% → 0.16% (÷26)0 / 200
R1-Distill-7B, seed 20.664 → 0.103 (÷6.5)0.734 → 0.203 (÷3.6)4.11% → 0.16% (÷27)0 / 200
JustRL-R1-7B1.168 → 0.116 (÷10.1)0.758 → 0.225 (÷3.4)4.30% → 0.18% (÷24)16 / 200
DeepScaleR-1.5B (same size)1.625 → 0.029 (÷56)0.781 → 0.096 (÷8.1)13.97% → 1.42% (÷10)24 / 200

With R1-Distill-7B the update falls from 73% to 20% of Adam's maximum step, so adaptive preconditioning does not restore it, and the loss removed per update falls 26-fold while more than 70% of the loss remains. Clipping is never active with this teacher. With DeepScaleR-1.5B the loss removed per update stays above 1% while the loss falls by 97%. The first-order decrease uses the gradient of the same minibatch, so it is an upper bound.

AuthorsOpenReview ↗

Response to Reviewer hvW7 (2/2): Q2, Q3

Q2. Can you evaluate the low-curvature residual of Theorem 5.2 for the self-RL teachers?

The residual EΔ(δ)E_\Delta(\delta) is small: at δ=0.3\delta = 0.3 it is 2.2–2.6% of the initial quadratic loss 12Δ⊤H0Δ\frac{1}{2}\Delta^\top H_0 \Delta for JustRL-1.5B and 0.7% for DeepScaleR-1.5B. We ran Lanczos on the initial Fisher matrix H0H_0 started from Δ=θT−θ0\Delta = \theta_T - \theta_0 (responses of the initial student, 32 token positions each, full vocabulary, full reorthogonalization) and obtained the share of 12Δ⊤H0Δ\frac{1}{2}\Delta^\top H_0 \Delta below each curvature δ\delta by Gauss quadrature. The relative distance ∥Δ∥/∥θ0∥\Vert\Delta\Vert/\Vert\theta_0\Vert is 0.497% for JustRL-1.5B and 0.073% for DeepScaleR-1.5B.

Cutoff δ\deltaJustRL-1.5B: 256 responses, 19 Lanczos stepsJustRL-1.5B: 64 responses, 36 stepsDeepScaleR-1.5B: 64 responses, 24 steps
0.32.2%2.6%0.7%
115%6%5%
1055%50%39%
10098%93%95%

These shares are upper bounds: the near-zero share decreases with Lanczos depth (27.6%, 8.9%, 3.8%, 2.2% at 5, 10, 15, 19 steps for JustRL-1.5B). A forward-only estimate gives 12Δ⊤H0Δ=0.202\frac{1}{2}\Delta^\top H_0 \Delta = 0.202, against 0.197 from Lanczos.

Whether the bound explains the convergence also depends on how far the quadratic model holds (Fisher cosine as defined in general response 2/2):

JustRL-1.5BDeepScaleR-1.5B
Measured KL / quadratic prediction along θ0+cΔ\theta_0 + c\Delta, at c=1/8,1/4,1/2,1c = 1/8, 1/4, 1/2, 11.06, 1.39, 1.86, 1.871.01, 1.04, 1.09, 1.20
Cosine between the student's displacement and Δ\Delta: Fisher metric / parameter space0.73 / 0.0780.976 / 0.27
Student's remaining KL to the teacher on the initial student's responses29%1.5%

For DeepScaleR-1.5B the quadratic model holds along the whole segment and the student moves straight toward the teacher in the Fisher metric, so the bound describes the convergence. For JustRL-1.5B it holds only for c≤1/8c \le 1/8, so the theorem is a local description there.

Q3. Does a higher learning rate or a different clipping threshold restore loss reduction?

No. See "Larger learning rate, longer training, and clipping" in general response (1/2). On your distinction between insufficient updates and a limitation of the objective, both with teacher R1-Distill-7B (fixed-context KL as defined in general response 1/2):

ReadingTestResult
Insufficient effective updatesMedian update norm ∥Δθm∥\Vert\Delta\theta_m\Vert (as in Q1) at LR 1e-6, 3e-6, 1e-5 (steps 151–175)0.0084, 0.0235, 0.0801; fixed-context KL / initial 0.85, 0.85, 0.95
Limitation of the top-16 objectiveTrain on the full-vocabulary KL instead (LR 1e-5 from step 50, 150 updates)Training loss 0.106 → 0.108; fixed-context KL / initial 0.85

Thank the authors for the clarifications. I decide to keep my score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (2/2): Learning signal, and parameter and representation changes

*Is the weak learning signal more than a restatement of the plateau?

Question. Does the weak learning signal cause the plateau or only accompany it, and is Figure 2 more than what a stagnating loss implies? (hvW7, C7ft)

Response. We re-measured the learning signal μ=∥g∥2/(2L)\mu = \Vert g \Vert^2/(2\mathcal{L}) (gg the expected gradient of the OPD update, L\mathcal{L} the loss; Section 5.1) with an independent method, which does not use the logged gradient norm of Figure 2, and it shows the same collapse. At each checkpoint we take the 64 on-policy responses of the next training batch (one per prompt), compute the gradient of the training objective on each response, and estimate ∥g∥2\Vert g \Vert^2 from inner products between gradients of different responses, which removes the sampling noise. With R1-Distill-7B, μ\mu falls 31-fold in 50 updates while the loss falls by 20%. For the same student with DeepScaleR-1.5B, a same-size public model RL-trained from our initial student on other data (response to Reviewer C7ft, W1.1), μ\mu is still 4.2 when 12% of the loss remains. The weak signal is therefore a directly measured property of training with this teacher, not a restatement of the stagnating loss. The weak signal is the immediate reason the loss stops falling: to first order, one update now removes 0.16% of the loss, against 4.2% at the start (reply to hvW7). What remains open is why the gradients of different prompts stop agreeing.

CheckpointR1-Distill-7B: μ\muLossDeepScaleR-1.5B: μ\muLoss
Initial student1.810.13228.10.045
Step 101.230.11925.10.037
Step 300.290.11714.80.013
Step 500.0590.1054.20.0053

Small parameter and representation changes (Section 6)

Question. Are the small parameter and representation changes simply what one expects from a few on-policy updates at learning rate 1e-6? (C7ft, Tnvd)

Response. The size of these changes is indeed set by the learning rate and the number of updates. The insight of Section 6 is a different one: the size of the change does not decide the outcome; its direction does. The paper states that the displacement magnitude alone does not separate successful from stalled runs, and leaves the role of direction as a hypothesis. The new runs test both (Min. CKA and relative parameter change as in Table 4).

Student of R1-Distill-1.5B after 200 updatesMin. CKARelative parameter changeFixed-context KL / initial
Teacher DeepScaleR-1.5B0.99930.046%0.015
Teacher R1-Distill-7B, LR 1e-60.99860.050%0.85
Teacher R1-Distill-7B, LR 3e-60.9940.12%0.86
Teacher R1-Distill-7B, LR 1e-5 from step 500.99050.27%0.95
  • Moving further does not help. A 3× or 10× learning rate moves the student 2.4 and 5.4 times as far, and the KL to the teacher stays at 0.86 or rises to 0.95.
  • Different teachers pull the same student in different directions. The displacements under DeepScaleR-1.5B and R1-Distill-7B have cosine 0.046, while two seeds with the same teacher have cosine 0.94.
  • A small move in the right direction is enough. The student that converges changes the least, and it moves toward its teacher.

We measure direction in the Fisher metric of the initial student. With H0H_0 its Fisher matrix, a small parameter change vv changes the output distribution by a KL of 12v⊤H0v\frac{1}{2} v^\top H_0 v, so a direction counts by how much it changes the outputs, and not at all if it leaves them unchanged. For the student's displacement dd and the teacher–student difference Δ\Delta, the Fisher cosine is d⊤H0Δ/(d⊤H0d)(Δ⊤H0Δ)d^\top H_0 \Delta / \sqrt{(d^\top H_0 d)(\Delta^\top H_0 \Delta)}.

Same-size teacherFisher cosineParameter-space cosineProgress along Δ\DeltaKL remaining
DeepScaleR-1.5B0.9760.271.0051.5%
JustRL-1.5B0.730.0780.7029%

In the directions that change the outputs, the DeepScaleR-1.5B student moves straight to its teacher; raw parameter space hides this. Progress along Δ\Delta is d⊤H0Δ/(Δ⊤H0Δ)d^\top H_0 \Delta / (\Delta^\top H_0 \Delta), equal to 1 when the student has covered the whole distance. The cosine needs a teacher with the student's parameterization, so it is not defined for R1-Distill-7B.

We will rewrite Section 6 around this result and drop the sentence that links the collapse to limited representation adaptation, which these runs do not support.

AuthorsOpenReview ↗

General Response (1/2): Learning rate, longer training, and clipping

Dear Reviewers and Area Chair,

We sincerely thank all reviewers for their thoughtful feedback. We are encouraged that reviewers found the problem important and well motivated (hvW7, C7ft, Tnvd), the theoretical and empirical analysis well conducted and carefully reported (hvW7, C7ft, Tnvd), and the self-RL teacher an interesting baseline (C7ft). Below, we answer the questions raised by more than one reviewer, with the new experiments we ran for them. Individual responses follow. New runs use the paper's setup unless stated.

Measurements used below

  • Accuracy: validation accuracy as in the paper (avg@8 for math, avg@4 for code, benchmarks equally weighted), averaged over the last five validation points of a run (Table 2 of the paper reports the final point of a single run). Differences are paired over problems, with 95% bootstrap intervals (10,000 resamples of problems).
  • Fixed-context KL: full-vocabulary KL(student∥teacher)\mathrm{KL}(\mathrm{student} \parallel \mathrm{teacher}) on one set of contexts shared by all checkpoints: 512 responses sampled from the initial student (128 training prompts), 32 token positions each. We report its ratio to the initial student's value. Unlike the training loss, it does not depend on each run's own rollouts or on the top-16 truncation.

Larger learning rate, longer training, and clipping

Question. Does a larger learning rate, longer training, or a different clipping threshold restore progress with a larger teacher, or does the student stay stuck? (hvW7, C7ft)

Response. None of them does.

Math, student R1-Distill-1.5B, teacher R1-Distill-7B. The initial student scores 39.85. Table 2 of the paper reports a gain of +2.80 for this pair, computed from the final validation point of the paper's run (42.66). Averaged over the last five validation points, the paper's run scores 40.60 (+0.74) and the two baseline seeds below score 40.33 (+0.48):

RunAccuracy vs. baseline [95% CI]Fixed-context KL / initial
Baseline: LR 1e-6, 200 updates (2 seeds)40.330.85
LR 3e-6 from the initial student (3 seeds)+0.06 [−0.82, +0.94]0.86
LR 3e-6 from step 50+0.47 [−0.57, +1.49]0.85
LR 1e-5 from step 50+0.79 [−0.46, +2.08]0.95
LR 3e-6, 400 updates+1.04 [+0.02, +2.08]0.86
LR 1e-6, 400 updates+0.78 [−0.25, +1.80]0.82

Code, student Qwen3-4B, teacher Qwen3-14B. The baseline is the paper's run (4 GPUs). The four other runs fork at step 50 from a re-run of the first 50 updates on 2 GPUs; over those updates the re-run matches the paper's run (training loss within about 1%, validation accuracy −0.10 [−0.76, +0.53]):

RunAccuracy vs. baseline [95% CI]Training loss vs. baseline, steps 151–200
Baseline: LR 1e-6, clipping at 153.17—
Clipping at 5+0.13 [−0.52, +0.80]+1.0%
No clipping−0.13 [−0.76, +0.47]+1.4%
LR 3e-6+0.04 [−0.78, +0.89]−4.6%
LR 1e-5−1.91 [−3.08, −0.77]−5.4%
  • Learning rate. In math, a 3× or 10× larger learning rate changes accuracy by less than one point, and the fixed-context KL stays at 85–86% of its initial value and rises to 95% at LR 1e-5. In code, 3× leaves accuracy unchanged (+0.04) and 10× lowers it by 1.9 points.
  • Longer training. At LR 1e-6, 200 more updates lower the fixed-context KL from 0.85 to 0.82, all of it before step 300 (0.821 at step 300, 0.822 at step 400). The two 400-update runs gain about one accuracy point; one interval excludes zero, without correction for multiple comparisons. For comparison, changing the teacher gains 4.3 to 6.9 points (response to Reviewer C7ft, W1.1).
  • Clipping. In the math run no update is clipped (0 of 200). In the code run 30 of 150 are, and relaxing or removing clipping changes neither the loss nor the accuracy.

The plateau is therefore not due to updates that are too small or clipped.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.