acceptodds
Under review as a conference paper at ICLR 2027

When Students Fail to Learn from Teachers: Learning Dynamics of On-Policy Distillation

Abstract

On-policy distillation (OPD) has become a core technique for post-training large language models, enabling students to learn from dense token-level supervision on their own generations. However, even with a capable teacher, OPD training can be unstable, leading to excessive response length, repetitive generation, and severe performance degradation. One example is spurious agreement, where the student maintains high overlap with the teacher at the states visited but produces repetitive generations. Although such failures have been observed, the mechanisms behind these failures remain poorly understood. To address this gap, we investigate OPD through the lens of learning dynamics, tracing three mechanisms from local analysis to subsequent on-policy sampling. We first identify states where sampled-token OPD updates fail to correct teacher–student mismatch and characterize them using the student's mass on the teacher's top- set. We then show that learning from the teacher is difficult at mismatched states and that their gradients can also interfere with learning at other tokens. Finally, we study how continued training on mismatched states influences on-policy sampling, suppressing the student's current modes and leading the student toward states where the teacher is unreliable. These findings help explain the benefits of supervised fine-tuning (SFT) initialization and motivate targeted correction of states where student and teacher are mismatched. We introduce state-dependent KL routing, which applies forward KL when student coverage of teacher-preferred tokens is low or a substantial fraction of teacher probability mass is underestimated. Across three Qwen3 model scales and both Base and SFT initialization, our method improves mean Avg@8 by 1.38–2.50 percentage points and mean Best@8 by 1.01–6.24 points over sampled-token OPD on mathematical reasoning benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, On-policy distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hfMyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates why on-policy distillation (OPD) can fail to effectively transfer knowledge from a strong teacher to a student, sometimes resulting in repetitive generation and degraded performance. The authors analyze OPD learning dynamics at three levels: local token distributions, cross-position gradient interactions, and subsequent on-policy sampling trajectories. Their analysis suggests that sampled-token updates may provide insufficient correction when the student assigns low probability to teacher-preferred tokens, while gradients from mismatched positions can interfere with learning elsewhere and alter future sampling behavior. Based on these findings, the authors explain the benefits of supervised fine-tuning (SFT) initialization and propose state-dependent KL routing, which selectively applies forward KL supervision at states exhibiting substantial teacher–student mismatch. Experiments on three Qwen3 student model scales, with both Base and SFT initialization, demonstrate consistent improvements over standard sampled-token OPD on mathematical reasoning benchmarks.

Critical strengths
  1. This paper provides some new insights to the OPD community. The paper goes beyond identifying empirical failure modes and provides a structured investigation of OPD from local token-distribution updates to cross-position gradient interactions and subsequent on-policy sampling. I appreciate the deep analysis made in the paper to help the community better understand the learning dynamics of OPD.

  2. Beyond pure analysis, the authors also propose a new method named state-dependent KL routing, which makes the whole story complete. I also appreciate the detailed ablation analysis and empirical results provided in the paper.

Critical weaknesses
  1. My major concern is about the limited generalizability of the empirical validation. The experiments are mainly conducted in small-sized Qwen models. I wonder whether the conclusions and the insights can be generalized to larger-scaler and non-Qwen families. Also, the authors only conduct experiments on math tasks, which are insufficient.

  2. The proposed method state-dependent KL routing seems to only bring limited improvement over the baselines. Also, the authors mainly discuss in the sampled-token OPD setting throughout the entire paper. However, full vocabulary OPD (e.g., DeepSeek-V4) is a more advanced setting. Do the conclusions remain useful in the full vocabulary OPD setting?

Questions for authors

See above.

1 comment

We thank the reviewer for the encouraging comments and the valuable questions.

S1&S2: This paper provides some new insights to the OPD community...Beyond pure analysis..

Response

Thank you for the encouraging comments on both the analysis and the routing method that follows from it. We are glad to bring some new insight to the community.

W1. My major concern is about the limited generalizability of the empirical validation.

Response

Thank you for raising this concern. We address the three points in turn.

Tasks beyond math. We added a coding task. We train on CodeR1-12k and evaluate on HumanEval+, MBPP+ and LiveCodeBench v6, with Qwen3-8B as the teacher and three Base students. The table shows mean Avg@8 over the three benchmarks at step 200; the full table is in our response to Reviewer Laj6.

StudentOPDEOPDAOPDOurs
1.7B43.0843.5744.0946.00
4B52.5551.4551.6955.74
8B55.5655.2457.1959.00

Our method has the best mean Avg@8 at all three scales, improving over sampled-token OPD by 2.92 / 3.19 / 3.44 points.

Model scale. Our experiments cover students from 1.7B to 8B. Within this range, the gains over sampled-token OPD are consistent in both math (Table 1) and coding. Due to computational resource limit, we do not have results for larger models.

Model families. The main results use Qwen3 models. The diagnostic experiments in Section 4 also use DeepSeek-R1-Distill-Qwen-1.5B and JustRL-1.5B. Our analysis itself does not depend on the model family: Proposition 1 only assumes a softmax output, and the cross-position decomposition in Section 4.2 holds for any model with shared parameters.

In the revision, we added the coding results and discussed these limitations.

W2. The proposed method state-dependent KL routing seems to only bring limited improvement over the baselines.

Response

Thank you for these two points. The gains are moderate but consistent, and our analysis still apply to full-vocabulary OPD.

On the improvement. We agree that the absolute gains are moderate. However, our contribution is not only the algorithm. It also includes the learning-dynamics analysis of OPD at three levels: local distributions, cross-position effects, and on-policy sampling. Moreover, the improvement is consistent across three student sizes, on math reasoning with both Base and SFT initialization, and on a newly added coding task.

On full-vocabulary OPD. The full-vocabulary update is the expectation of Eq. (4) over the sampled token, which gives

Δθb=η πθ(b∣s)(A(s,b)−Aˉ(s)),Aˉ(s)=Eu∼πθ[A(s,u)].\Delta\theta_b=\eta\,\pi_ \theta(b\mid s)\big(A(s,b)-\bar A(s)\big),\qquad \bar A(s)=\mathbb{E}_ {u\sim\pi_ \theta}[A(s,u)].

The update of each token is still scaled by its student probability, so tokens that the teacher prefers but the student rarely produces still receive little correction. This comes from reverse KL itself, not from sampling. The advantage of full-vocabulary OPD is that it reduces the noise from sampling, which is the benefit reported in DeepSeek-V4, at the cost of a large GPU memory overhead for storing the full-vocabulary logits of student and teacher models over the whole sequence.

To test this, we ran OPD with a top-KK approximation of the full-vocabulary reverse KL on Qwen3-1.7B-Base with the Qwen3-8B teacher. The same failure appears. On math, the mean response length grows from 1.2k to 4.1k tokens over 200 steps, and the fraction of rollouts hitting the 4,096-token cap rises from 14% to 100%. On code, the length grows from 0.8k to 4.0k tokens, and the cap-hit rate rises from 5% to 95%. The outputs also become repetitive: the validation loop rate peaks at 81% on math and 52% on code, and ends at 42% and 30%. Increasing KK to 32 does not help: the loop rate peaks at 88% on math and ends at 42% and 32%. So removing the sampling noise does not remove the failure.

We added this discussion and the top-KK experiment above to the revision (Appendix D).

Reviewer_Laj6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper theoretically explains spurious agreement, which occurs when the sampled-token advantage is small despite substantial differences between teacher and student distributions, using first-order approximation of OPD updates. Based on this fact and empirical analysis, they introduce state-dependent KL routing with SFT initialization. Compared to other OPD baselines, their method improves both pass@1 and pass@8 with a small doom-looping rate.

Critical strengths
  • This paper provides theoretical analysis of OPD gradient updates following [1], which seems solid.
  • This paper supports their theoretical results by multiple empirical analysis.

[1] Ren and Sutherland. Learning Dynamics of LLM Finetuning. ICLR2025.

Critical weaknesses
  • While this paper provides a solid explanation of spurious agreement, the proposed method remains heuristic and fails to sufficiently explain why routing to Forward KL works. Authors should discuss or conduct additional experiments to explain why forward KL mitigates spurious agreement.
  • Experiments are restricted to the math domain using only the same Qwen3 models, and evaluations rely on small, closely related datasets (AIME, AMC). It would strengthen the paper to include additional evaluations on broader benchmarks (eg, BeyondAIME, MATH500, Minerva, OlympiadBench), or experiments on other domains (eg, code, science).
Questions for authors
  • Is there any computational overhead compared to standard reverse KL OPD? If so, could the authors report wall-clock times or discuss computational complexity?
  • I understand that SFT initialization helps OPD, but could the authors discuss the difference or the relation to SFT initialization before RL [2][3][4][5]?

[2] Li et al. Getting Your LLMs Ready for Reinforcement Learning with Lightweight SFT. ICLR2026.

[3] Matsutani et al. RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs. ICLR2026.

[4] Kang et al. Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead. ICLR2026

[5] Limozin et al. SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning. 2026.

4 comments

We thank the reviewer for the positive assessment and the helpful suggestions.

S1 & S2 This paper provides theoretical analysis of OPD gradient updates following [1], which seems solid. This paper supports their theoretical results by multiple empirical analysis.

Response

Thank you for the positive review. We are glad that the theoretical analysis of the OPD updates is solid and the empirical analyses are supportive of it.

W1. While this paper provides a solid explanation of spurious agreement, the proposed method remains heuristic and fails to sufficiently explain why routing to Forward KL works.

Response

Thank you for this comment. In short, forward KL works because its update is driven by the teacher's probability, while the sampled-token update of a token depends on the student's probability of that token. Under the single-state softmax setting of Proposition 1, the logit updates are

sampled-token OPD (Eq. 4):Δθb=η A(s,u)(1{b=u}−πθ(b∣s)),u∼πθ(⋅∣s),\text{sampled-token OPD (Eq. 4):}\quad \Delta\theta_b=\eta\,A(s,u)\big(\mathbb{1}\{b=u\}-\pi_\theta(b\mid s)\big),\quad u\sim\pi_\theta(\cdot\mid s),

forward KL:Δθb=η[πT(b∣s) 1{b∈SK}−πθ(b∣s)∑c∈SKπT(c∣s)].\text{forward KL:}\quad \Delta\theta_b=\eta\Big[\pi_T(b\mid s)\,\mathbb{1}\{b\in S_K\}-\pi_\theta(b\mid s)\sum_{c\in S_K}\pi_T(c\mid s)\Big].

With sampled-token OPD, a token is updated directly only when it is sampled, and otherwise its update is scaled by πθ(b∣s)\pi_\theta(b\mid s). A teacher-preferred token with near-zero student probability therefore receives almost no update, so when the student's mode is suppressed, the probability mass goes to other continuation tokens. With forward KL, the teacher-preferred token is raised directly.

The routing rule follows from this. Low top-pp coverage and high underestimated mass VV identify the states where the sampled-token update is ineffective. At other states we keep reverse KL, and applying forward KL everywhere does not perform better.

We added this analysis to Section 5.2.

W2. W2. Experiments are restricted to the math domain using only the same Qwen3 models,...

Response

Thank you for the careful reading and the advice. To extend the evaluation beyond math, we added a coding task.

We train on CodeR1-12k and evaluate on three benchmarks: HumanEval+ (164 problems), MBPP+ (378 problems) and LiveCodeBench v6 (175 problems). We use Qwen3-8B in non-thinking mode as the teacher and three Base students (Qwen3-1.7B, Qwen3-4B and Qwen3-8B). For all methods, we report the checkpoint at step 200.

Table 0: Coding results with Qwen3-8B as the teacher. Each cell reports Avg@8 / Best@8. Bold marks the best result for each student size.

StudentMethodHumanEval+ (Avg / Best)MBPP+ (Avg / Best)LiveCodeBench v6 (Avg / Best)Mean (Avg / Best)
1.7BOPD64.71 / 77.2655.59 / 66.108.93 / 17.4943.08 / 53.62
1.7BEOPD65.17 / 77.3955.19 / 66.4210.36 / 18.4943.57 / 54.10
1.7BAOPD64.64 / 78.2755.56 / 66.6012.07 / 19.5844.09 / 54.81
1.7BOurs67.07 / 76.7655.99 / 68.6014.93 / 21.7546.00 / 55.70
4BOPD79.27 / 85.9768.39 / 74.1110.00 / 24.7252.55 / 61.60
4BEOPD78.43 / 88.4466.57 / 73.579.36 / 21.5451.45 / 61.18
4BAOPD73.09 / 87.8262.90 / 77.7019.07 / 27.2551.69 / 64.26
4BOurs79.19 / 86.6467.10 / 73.8320.93 / 28.1455.74 / 62.87
8BOPD78.73 / 82.2172.45 / 77.7715.50 / 29.6855.56 / 63.22
8BEOPD79.27 / 84.8572.39 / 77.9814.07 / 27.9855.24 / 63.60
8BAOPD79.65 /86.3270.27 / 78.9721.64 / 31.4057.19 / 65.56
8BOurs80.64 / 87.2872.72 / 79.2223.64 / 31.8159.00 / 66.10

Our method has the best mean Avg@8 at all three scales, improving over sampled-token OPD by 2.92 / 3.19 / 3.44 points.

We added these results to the revision.

Q1. Is there any computational overhead compared to standard reverse KL OPD?

Response

Thank you for this suggestion. Our method adds a modest computational overhead compared to standard OPD. With our default K=16K=16, the end-to-end time per training step increases by 2.5–6.8% across the three student sizes. The peak memory of a training step is unchanged, with 2.5–5.5 GB of additional memory in the actor-update stage.

To quantify this overhead, we conducted controlled experiments on 4×H100 GPUs to measure times and addition memory with a Qwen3-8B teacher and student of three sizes averaged over 10 steps. In our implementation, teacher scoring is performed with vLLM, with a fixed GPU memory budget for the teacher. Standard OPD only needs the log-probability of the sampled token, while other method using forward KL requires the teacher's top-(K) log-probabilities at each position.

Time (Table B). The additional cost comes mostly from teacher scoring and grows roughly linearly with KK. At K=16K=16, teacher scoring increases from 2.6–3.1 s to 4.5–5.9 s per step, but it still accounts for only about 6–10% of the total step time, which is dominated by rollout generation.

Memory (Table C). Memory does not grow with KK as the teacher-scoring stage runs under a fixed vLLM memory budget. In the actor update, the top-KK teacher targets are small (KK values per position), and the additional 2.5–5.5 GB over the baseline comes from the routing and forward-KL computation, which does not depend on KK.

Table 1: Time per training step (seconds) for each stage. "OPD" is sampled-token OPD; the other rows are our method with different K.

StudentMethodRollout genTeacher scoringStudent log-probActor updateWeight syncWhole stepvs k1
1.7BOPD40.83.02.26.11.353.5-
K=840.13.82.07.01.254.3+1.5%
K=1640.75.92.17.11.257.1+6.8%
K=3240.09.82.17.31.460.6+13.4%
4BOPD52.13.13.511.51.671.8-
K=851.53.53.110.91.770.7-1.5%
K=1650.15.63.110.74.0*73.6+2.5%
K=3251.19.13.210.91.676.0+5.8%
8BOPD50.72.63.712.42.471.9-
K=851.13.03.612.82.272.7+1.0%
K=1651.24.53.512.92.174.3+3.3%
K=3250.37.83.613.22.277.2+7.3%

Table 2: Peak Memory (GB) for each state.

StudentMethodRollout genTeacher scoringStudent log-probActor updateWhole step
1.7BOPD50.368.67.417.968.6
K=850.468.87.423.468.8
K=1650.267.57.423.367.5
K=3250.468.37.423.168.3
4BOPD50.467.510.027.667.5
K=850.467.610.033.167.6
K=1650.470.010.133.170.0
K=3250.466.710.033.166.7
8BOPD50.768.116.454.268.1
K=850.768.916.556.868.9
K=1650.870.916.956.770.9
K=3250.766.316.956.866.3

We added this computational complexity and time analysis and comparison in Appendix G.

Q2. I understand that SFT initialization helps OPD, but could the authors discuss the difference or the relation to SFT initialization before RL [2][3][4][5]?

Response

Thank you for pointing out these works. Our finding is related to SFT initialization before RL, but the role of SFT is different in OPD.

From [2] and [4], the message is that a good SFT initialization should preserve enough useful and diverse behaviors so that subsequent on-policy training can still improve the policy, rather than achieving the highest SFT accuracy. [2] emphasizes the diversity, and [4] finds that Pass@ k is predictive of post-RL performance. [3] further explains the reason that SFT expands correct reasoning trajectories, while RL mainly compresses incorrect ones. [5] instead studies a bug in the previous SFT loss calculation.

In OPD, we do not have the bug in the SFT stage. SFT initialization helps not only by improving the policy support, but also by making the student more compatible with the specific teacher, reducing the teacher–student mismatch at the states the student visits.

We added this discussion and cited [2–5] in the revision.

Dear Reviewer Laj6,

Thank you again for your thoughtful feedback. We have provided a detailed response to the points raised in your review, including additional clarification and results.

Thank you for your time and consideration.

Thank you very much for clarification on Forward KL and additional experiments on code. My concerns are largely addressed and I have raised my score to 3.

Reviewer_sjXtOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the learning dynamics and failure modes of sampled-token on-policy distillation (OPD). The authors identify three mechanisms that can prevent effective knowledge transfer from a teacher to a student: (1) insufficient probability updates for teacher-preferred tokens when the student's initial probability mass is low, (2) cross-position gradient interactions that can alter or reverse individual token update directions, and (3) changes in on-policy sampling that can lead to repetitive generation and spurious teacher–student agreement.

The paper provides a first-order analysis of token probability changes under a softmax policy and empirical diagnostics of these mechanisms in LLM training. Motivated by these findings, the authors propose state-dependent KL routing, which selectively replaces sampled-token reverse KL with approximate forward KL when the student has insufficient coverage of teacher-preferred tokens or substantially underestimates the teacher's probability mass.

Critical strengths
  1. Detailed and insightful analysis of OPD learning dynamics.

The paper provides a systematic investigation connecting local token probability changes, cross-position gradient interactions, and subsequent on-policy sampling behavior. I particularly appreciate the cross-position analysis in Section 4.2 and its validation through first-order predictions and leave-one-out interventions. These results offer useful insights into why teacher feedback may not translate into the intended student updates.

  1. A practically motivated method with consistent improvements over sampled-token OPD.

The proposed state-dependent KL routing is directly motivated by the diagnostic findings. The method achieves improvements over standard OPD across all six model-size and initialization configurations in Table 1. It also applies forward KL selectively rather than uniformly, making it an interesting approach to addressing teacher–student mismatch without requiring forward-KL supervision at every position.

Critical weaknesses
  1. The relationship between the theoretical analysis and the actual OPD optimization objective requires clarification, and the causal explanation of training failures remains incomplete.

Equation (1) defines an expectation over student-generated trajectories, whose state distribution depends on the student parameters. However, Equation (2) appears to omit the gradient contributions arising from this parameter-dependent state distribution. It therefore seems to describe a semi-gradient update with detached on-policy states rather than the exact gradient of Equation (1). The authors should clarify this distinction.

Furthermore, while the cross-position analysis convincingly demonstrates interactions between token gradients, the evidence that these interactions are a primary cause of OPD failure is less conclusive. The analysis establishes the presence of interactions, but stronger causal interventions would help distinguish harmful mismatch-induced interference from interactions that naturally arise in shared-parameter optimization.

  1. The experimental evidence is not yet sufficient to establish the robustness and generality of the proposed method.

The improvements over sampled-token OPD are relatively modest in mean Avg@8, ranging from 1.38 to 2.50 percentage points across the reported configurations. The paper does not report variability across independent training seeds, making it difficult to assess statistical reliability. Moreover, in the SFT-initialized setting, the proposed method does not improve over the original SFT checkpoint for the 4B and 8B models.

An additional concern is that the generation-collapse example in Section 4.3 involves different termination tokens between the student and teacher. Without controlling for this mismatch, it is difficult to determine how much of the observed instability reflects a general limitation of sampled-token OPD rather than a specific formatting or termination-token incompatibility. The experiments are also restricted to mathematical reasoning tasks.

Questions for authors
  1. Clarification of the optimization objective.

Equation (1) defines an objective over student-generated trajectories, while Equation (2) appears to omit the gradient through the student-dependent trajectory distribution. Is Equation (2) intended to describe a semi-gradient update with detached sampled states? If so, could the authors clarify the distinction between the sequence-level objective and the actual optimization procedure?

  1. Is the observed generation collapse primarily caused by termination-token mismatch?

In Section 4.3, the student and teacher use different termination tokens. Could the authors provide a controlled experiment in which the termination tokens are aligned, or in which termination-token gradients are excluded? Does spurious agreement still occur under these conditions? This would help establish whether the proposed failure mechanism extends beyond this particular incompatibility.

  1. How robust are the improvements across training seeds and checkpoint-selection procedures?

Could the authors report results across multiple independent training seeds, including standard deviations or confidence intervals? Also, were the reported evaluation benchmarks used to select the best checkpoints? Finally, could the authors clarify why OPD variants, including the proposed method, underperform the SFT checkpoint at the 4B and 8B scales in Table 1(b)? These results would help assess the practical benefits of the method beyond improvements relative to sampled-token OPD.

3 comments

We thank the reviewer for the careful reading and the detailed, constructive comments.

S1&S2: Detailed and insightful analysis of OPD learning dynamics. A practically motivated method with consistent improvements over sampled-token OPD.

Response

Thank you for the positive and thoughtful assessment. We are glad that the cross-position learning-dynamics analysis in Section 4.2 and its validation useful, and that the connection between these findings and our state-dependent KL routing was clear. We also appreciate the recognition of the method’s consistent gains over sampled-token OPD and its selective use of forward KL.

W1: The relationship between the theoretical analysis and the actual OPD optimization objective requires clarification, and the causal explanation of training failures remains incomplete.

Response

Thanks for your constructive advice and suggestions.

Theoretical analysis. Please see Q1 for the relationship between the theoretical analysis and the actual OPD optimization.

Cross-position analysis. We would like to clarify that the paper does not claim that cross-position interactions are a primary cause of OPD failure. Our claim in Section 4.2 is that a mismatch at one position can affect the updates at other positions through shared parameters, and that mismatched positions carry a large weight in the update: since Mij=ηAj⟨hi,hj⟩M_{ij}=\eta A_j\langle h_i,h_j\rangle, a single termination position with A=−22.53A=-22.53 accounts for 31.81% of the total influence among 200 positions (Table 3). We stated this scope more explicitly in the revision.

Moreover, we ran two interventions on mismatched positions in the Sec. 4.3 setting (Qwen3-1.7B-Base student with Qwen3-8B teacher). First, masking the termination-token gradient or aligning the termination tokens removes the collapse (details in our response to Q2). Second, with the termination tokens aligned, we injected a large negative advantage on five ordinary tokens (###, ---\n\n, \n\n, The, So). The student quickly stops producing them and responses stop normally, with only a small drop in accuracy (16.9–18.3 vs. 18.5 mean Avg@8). So sampled-token OPD can suppress a token the student produces, but cannot teach a token the student almost never produces. The suppression causes failure when the teacher places almost all of its probability on a token that the student cannot produce and that is essential to the response, as with the termination token.

W2. The experimental evidence is not yet sufficient to establish the robustness and generality of the proposed method.

Response

Thanks for careful reading. We address the four points in turn.

Modest gains and training seeds. We agree that the gains in mean Avg@8 are moderate and that due to computational resource limit, we cannot run with multiple seeds. However, the improvement is consistent across different model scale: our method is better than sampled-token OPD in all 6 configurations (with SFT and without SFT on 3 student sizes).

Comparison with the SFT checkpoint. The statement that "the proposed method does not improve over the original SFT checkpoint " does not match Table 1 in the paper. Our method improves mean Avg@8 over the SFT checkpoint at all three scales: 20.08 to 22.94 (1.7B), 47.01 to 52.19 (4B) and 52.94 to 59.66 (8B). The cases where the SFT checkpoint is higher are all in Best@8: the 4B student (mean 71.46 vs. 69.49), and AMC23 for the 8B student (95.00 vs. 94.06). This is consistent with RL: on-policy training concentrates probability on fewer solutions, which raises Pass@1 but can lower Pass@k [1,2]. OPD is also trained on the student's own samples, so we expect the same effect. It appears for all OPD variants on the 4B student, not only for our method.

Termination-token mismatch. We ran more experiments, including aligning the termination tokens, masking the graidents. See more details for Q2.

Other domains. We added a coding task, training on CodeR1-12k and evaluating on HumanEval+, MBPP+ and LiveCodeBench v6.

[1] Yue, Yang, et al. "Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?." Advances in Neural Information Processing Systems 38 (2026): 57654-57689.

[2] Matsutani, Kohsei, et al. "RL squeezes, sft expands: A comparative study of reasoning llms." International Conference on Learning Representations. Vol. 2026. 2026.

Q1. Clarification of the optimization objective...

Response

Thank you for highlighting this distinction. Eq. (2) is the policy-gradient estimator of Eq. (1) with the reward-to-go truncated to the immediate term AtA_t, which is equivalent in expectation to treating the distribution of student-visited prefixes as fixed.

The full gradient of sequence-level objective Eq. (1) is

∇θLOPD=−Ey∼πθ[∑t=1T(∑t′≥tAt′)∇θlog⁡πθ(yt∣st)],\nabla_\theta \mathcal{L}_ {\mathrm{OPD}} = - \mathbb{E}_ {y \sim \pi_\theta}\Big[\sum_ {t=1}^{T}\Big(\sum_ {t'\ge t} A_{t'}\Big)\nabla_ \theta \log \pi_ \theta(y_t\mid s_t)\Big],

and Eq. (2) omits the term ∑t′>tAt′\sum_{t'> t} A_{t'} and keeps only the t′=tt'=t term of the inner sum. In expectation, this resulting term is the gradient of the per-state reverse KL with the fixed prefix distribution dtd_t. Eq. (2) is its single-sample estimator. Therefore, Eq. (2) still takes the gradient through the sampling of yty_t at sts_t but only the effect of yty_t on later states is dropped.

Our optimization procedure follows Eq. (2), and all of our analysis is based on this update rather than on the exact gradient of Eq. (1).

In the Appendix B.1, we discuss the distinction between Eq.(1) and Eq.(2).

Q2. Is the observed generation collapse primarily caused by termination-token mismatch?

Response

Thank you for this insightful suggestion. We ran both controls. In short, the collapse in Sec. 4.3 is caused by the termination-token mismatch.

On the Sec. 4.3 setting (Qwen3-1.7B-Base student, Qwen3-8B teacher, 200 steps), we compare (A) the original setting, (B) the student's EOS mapped to the teacher's end-of-turn token, and (C ) the termination-token gradient masked.

A:originalB:EOS alignedC: EOS gradient masked
Rollouts hitting the max length0.950.250.25
Validation response length (tokens)7,2343,9393,761
teacher mass on Top-16 overlap0.9880.989≈0.99

Both controls remove the collapse, so spurious agreement does not occur under them.

On whether the failure extends beyond this mismatch: a mismatch on ordinary tokens does not produce the collapse (see the second intervention in our response to W1), so we do not claim that every mismatch leads to it.

We added these controls and stated in the revision that the collapse in Sec. 4.3 is caused by the termination-token mismatch.

Q3. How robust are the improvements across training seeds and checkpoint-selection procedures?

Response

Thank you for raising these important points. In short: (1) due to the limited time, we could not complete multiple independent training seeds; (2) we evaluate every 40 steps over 200 training steps and report the best checkpoint for math reasoning task; and (3) the apparent degradation relative to SFT is limited to Best@8 on several datasets, while our method improves Avg@8 over the SFT checkpoint. This is consistent that RL on policy training concentrates probability on fewer solutions, which raises Pass@1 but can lower Pass@k [1,2].

[1] Yue, Yang, et al. "Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?." Advances in Neural Information Processing Systems 38 (2026): 57654-57689.

[2] Matsutani, Kohsei, et al. "Rl squeezes, sft expands: A comparative study of reasoning llms." International Conference on Learning Representations. Vol. 2026. 2026.

Dear Reviewer sjXt,

Thank you again for your thoughtful feedback. We have provided a detailed response to the points raised in your review, including additional clarification and results.

We would appreciate any further feedback if you have any.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.