acceptodds
Under review as a conference paper at ICLR 2027

OPD: Overlap-Gated On-Policy Distillation on Reliable Prefix

21 views

Abstract

On-policy distillation (OPD) supplies dense token-level supervision by aligning a student to the teacher's next-token distribution on its own generated rollouts. Existing works mostly implement top- OPD, applying reverse Kullback-Leibler () divergence between the teacher's and the student's renormalized distributions over teacher top- support. However, the teacher scores student-generated rollouts that are inherently off-policy for itself, leaving the reliability of its next-token guidance unclear. Restricting the divergence to the teacher top- support also discards the remaining mass outside that support. These gaps leave two questions implicit: 1) which tokens to align, and 2) how to align those tokens. To answer these questions, we propose Overlap-gated On-Policy Distillation (OPD). As for 1), we quantitatively analyze the reliability of teacher supervision and find that alignment on the student's trajectory prefix is more reliable than alignment on the full context. Therefore, our OPD is applied only to the 20% prefix tokens of each student rollout. As for 2), we theoretically characterize two problems of top- OPD: the Teacher-support Uncovering and the Dominant-Mode Mismatch. Based on these analyses, we propose two support strategies: the Reset Support}strategy attaches a reset atom to every token, pooling mass outside the teacher top-, and the Support Convergence strategy converges the support to the already aligned intersection. Our OPD employs the designed overlap-gated -divergence to interpolate between mode-covering and mode-seeking for per-token alignment, where the overlap is used as a gate to switch the support and adaptively modulate . Extensive experiments on seven common benchmarks demonstrate the superiority of OPD, improving average performance by up to 5.29 points compared to OPD.

Primary area:
reinforcement learning
Keywords:
On-Policy Distillitation, Aclpha-divergence, Dominant-Mode Dilution problem, Teacher-Support Uncoverage

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8bfQOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Overlap-gated On-Policy Distillation (O²PD) to address two issues in on-policy knowledge distillation: (1)where should distillation be applied, and (2) how should teacher and student distributions be aligned. For (1), the authors identify a cross-prefix asymmetry, which means teacher performance decreases when conditioned on longer and longer student-generated prefixes. This suggests that teacher supervision is more dependable early in student rollouts. So O²PD constrains distillation to the first 20% of the generated tokens. For (2), the authors find two limitations of conventional top-k reverse KL distillation: Teacher-support Uncovering and Dominant-Mode Mismatch. O²PD introduces reset support and support convergence with an an overlap-gated α\alpha-divergence to address. The experiments on math reasoning benchmarks and teacher–student pairs show O²PD gains improvements over OPD and ablation supports the design.

Critical strengths
  1. The paper provides a useful analysis of the limitations of existing top-k OPD methods. The authors identify two issues, Teacher-support Uncovering and Dominant-Mode Mismatch, and provide theoretical analysis to explain them. This helps motivate the proposed support strategies and gives a clearer understanding of why standard reverse KL may be insufficient.

  2. The method shows consistent improvements across different teacher–student pairs. O²PD outperforms the compared OPD baselines across three teacher–student pairs and seven math reasoning benchmarks, with gains of up to 5.29 points over standard OPD. The ablation studies also provide some evidence for the effectiveness of the proposed components.

Critical weaknesses
  1. The evidence showed in this paper for prefix-only alignment is not fully convincing. The drop in teacher accuracy on longer student prefixes may simply be due to mistakes already made by the student. This does not necessarily mean the teacher provides worse token-level supervision at later positions. The analysis is also limited to one model pair on MATH500. It would be useful to see whether the same pattern holds in other settings.

  2. Some key design choices are not well justified. The theory explains the motivation for using different support strategies, but it is less clear why the authors choose median-based switching and min–max normalization for α. These choices seem somewhat arbitrary. Also, removing support convergence only drops the score from 50.6 to 49.9, so I am not sure how important this component is in practice.

Questions for authors

please see the weakness.

5 comments

We thank the reviewer for the careful comments. The three items below are the weaknesses you stated.

1. Is the evidence for prefix alignment sufficient?

The original cross-prefix protocol used only Qwen3-1.7B ←\leftarrow Qwen3-4B on MATH500. Figure 11, newly added in the submitted revision, extends it to three teacher--student pairs. The metric is mean pass@1 on Minerva, Olympiad, and MATH500. All three pairs show the same pattern: the score rises when the student continues a teacher prefix and falls when the teacher continues a student prefix. At ℓ=0.2\ell=0.2 the three students are already 44.7, 45.4, and 44.2, above the pass@1 of 43.71, 43.59, and 43.71 when each student generates the full response. Using only part of the teacher prefix already exceeds the student alone, so teacher supervision on the prefix is effective.

Fig. 4b measured only 0.2,0.4,0.6,0.8,1.0\\{0.2, 0.4, 0.6, 0.8, 1.0\\} with the Qwen3-4B-NTRL teacher. Figure 12, newly added in the submitted revision, adds 0.05, 0.10, and 0.15, and adds DSR1-1.5B ←\leftarrow JustRL-1.5B. The table gives the values in Figure 12, newly added in the submitted revision. The metric is again the mean of Avg@8 on AIME24, AIME25, AIME26, and HMMT25.

ℓ\ellNTRL OPDNTRL O$^{2}$PDDSR1 OPDDSR1 O^{2}\PD
0.0533.837.826.429.0
0.1035.840.228.431.0
0.1537.242.230.932.5
0.2037.442.631.232.6
0.4036.842.031.532.4
0.6037.041.831.332.2
0.8036.341.331.032.0
1.036.040.530.831.8

Both pairs are lowest at ℓ=0.05\ell=0.05, because the supervised prefix is too short to carry the signal, and then rise through the short prefixes. The experiments find that 20% of the tokens are already enough to bring a gain and often reach a strong result, so we adopt this simple and effective choice. The 80/20 rule may still hold in the present setting. Further details are in Appendix I (highlighted in blue\color{blue}{\textbf{highlighted in blue}} in the revised PDF).

2. Why switch support at ρ0.5\rho_{0.5}, and why map α\alpha from the prefix overlap min and max?

The purpose of on-policy distillation is to align the student's next-token distribution to the teacher's on the student's own rollouts. The support switch decides which prefix tokens stay on Reset Support T∪rT\cup\\{r\\} and which move to Support Convergence on I∪rI\cup\\{r\\}. The overlap level moves with the teacher--student pair and with the training stage, so the switch is ρ0.5\rho_{0.5} of the current prefix rather than a fixed ρ\rho. An absolute threshold selects a share that changes with the distribution; a percentile selects a share that does not. The table gives the mean overlap of the prefix tokens, the first 20% of each rollout, at k=16k=16, 3232, and 6464.

PairStagek=16k=16k=32k=32k=64k=64
Qwen3-1.7B ← Qwen3-4B-NTRLBefore0.6450.6360.633
Qwen3-1.7B ← Qwen3-4B-NTRLAfter0.7080.6980.695
Qwen3-1.7B ← Qwen3-4BBefore0.7480.7200.702
Qwen3-1.7B ← Qwen3-4BAfter0.8610.8340.809

The mean is not the same for the two pairs, and it rises after training at every kk. Figures 9 and 10, newly added in the submitted revision, give the token-level overlap distributions: Figure 9 is before training and Figure 10 is after training. The tokens are not piled on that mean, so the overlap distribution itself moves with the teacher--student pair and with the training stage. A fixed ρ\rho therefore does not select a stable share of the prefix. Switching at ρ0.5\rho_{0.5} keeps the lower half of the current prefix on T∪rT\cup\\{r\\}.

A fixed threshold does not replace the percentile. On the same Qwen3-1.7B ←\leftarrow Qwen3-4B-NTRL pair, sweeping τ\tau, both Avg@8 and pass@8 mostly peak at τ=0.75\tau=0.75, where both peak at τ=1.0\tau=1.0. The summary Avg@8 of 40.83 is still below 41.94 at ρ0.5\rho_{0.5}. τ=0.5\tau=0.5 is only 39.31 and does not reach ρ0.5\rho_{0.5}. The best absolute cutoff is still below switching at ρ0.5\rho_{0.5}. Here ρ0.5\rho_{0.5} (the switch point used in the paper) is the median percentile of the overlap values on the current prefix, not the absolute cutoff τ=0.5\tau=0.5: tokens with ρ<ρ0.5\rho\lt \rho_{0.5} stay on T∪rT\cup\\{r\\}, and tokens with ρ≥ρ0.5\rho\ge\rho_{0.5} move to I∪rI\cup\\{r\\}.

τ\tauAIME24 Avg@8pass@8AIME25 Avg@8pass@8HMMT25 Avg@8pass@8Avg@8pass@8
0.2550.4266.6740.0056.6722.0830.0037.5051.11
0.551.2566.6742.9263.3323.7536.6739.3155.56
0.7552.9270.0044.5866.6725.0040.0040.8358.89
1.053.3373.3342.0863.3323.3336.6739.5857.78
————————
ρ0.5\rho_{0.5}54.1773.3345.8370.0025.8343.3341.9462.22

α\alpha is mapped from the same prefix-overlap min and max: the lowest overlap in the current prefix is mapped toward Forward KL, and the highest toward reverse KL. The switch therefore remains ρ0.5\rho_{0.5} of the current prefix: the lower half stays on T∪rT\cup\\{r\\}, and that share does not change as overlap rises across pairs or during training. Further details are in Appendix H.

3. What is the role of Support Convergence in practice?

w/o II keeps the prefix, the reset atom, and α\alpha varying with ρ\rho, and always uses the support T∪rT\cup\\{r\\}. It no longer switches to I∪rI\cup\\{r\\} at high overlap. On Qwen3-1.7B ←\leftarrow Qwen3-4B-NTRL, OPD is 45.29, w/o II is 49.9, and full O$^{2}$PD is 50.58. The prefix, the reset atom, and α\alpha already raise the score from 45.29 to 49.9; switching to II adds about 0.7.

The role of Support Convergence is, once overlap has risen, to move the support to I∪rI\cup\\{r\\}, fold the leftover teacher mass on T∖IT\setminus I into the reset atom, and pin the teacher's mass on the shared atoms, rather than leftover-renormalizing them on T∪rT\cup\\{r\\}. The entropy curves in Figure 6 of the paper correspond to this role: entropy falls during training at every percentile, and the closer the percentile is to ρ1.0\rho_{1.0}, the more tokens remain on T∪rT\cup\\{r\\} and the larger the drop; the entropy at ρ0.5\rho_{0.5} is not the lowest, so the student can still keep exploration. The table reads out the four curves in Figure 6. The last column is the drop at step 150 relative to ρ0.25\rho_{0.25}.

Percentile1255075100125150Drop vs. ρ0.25\rho_{0.25}
ρ0.25\rho_{0.25}0.2780.2580.2180.2290.2360.2080.2070%
ρ0.5\rho_{0.5}0.2550.180.1990.2040.1880.1860.18610.3%
ρ0.75\rho_{0.75}0.2600.2620.1990.1750.1660.1770.16918.6%
ρ1.0\rho_{1.0}0.2450.2720.1670.1420.1440.1590.13634.6%

ρ1.0\rho_{1.0} leaves almost the whole prefix on T∪rT\cup\\{r\\}, and at step 150 its entropy is 34.6% below ρ0.25\rho_{0.25}. ρ0.5\rho_{0.5} is 10.3% below, and avoids the largest drop. ρ0.25\rho_{0.25} has the highest entropy, but its three-benchmark Avg is only 39.45, below 41.94 at ρ0.5\rho_{0.5}. Moving the support to I∪rI\cup\\{r\\} under Support Convergence avoids an excessive drop in actor entropy.

Reviewer_849gOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies where and how to apply top-k on-policy distillation. It first uses cross-prefix completion and position-wise alignment analyses to motivate supervising only the first 20% of each student-generated response. It then characterizes two limitations of reverse KL between distributions renormalized on the teacher's top-k set: weak covering pressure for teacher-preferred tokens with little student mass, and failure to constrain absolute probability mass on retained tokens. O2PD adds an aggregate reset bucket, uses teacher–student top-k overlap to interpolate an alpha divergence between forward and reverse KL, and switches the retained support from the teacher top-k set to its intersection with the student's top-k set at an overlap percentile. Experiments on seven mathematical reasoning benchmarks and three student–teacher pairs report average improvements of 1.89, 4.81, and 5.29 points over OPD, and 1.06, 2.65, and 2.51 points over the strongest reported competing method.

Critical strengths
  1. The observation on Cross-prefix Asymmetry is novel and interesting. It serves as strong motivation for the proposed approach.
  2. Good theoretical support is provided.
  3. Good presentation and writing clarity.
  4. The propose method is empirically effective, achieving significant improvements over OPD.
Critical weaknesses

The experiments are performed with small models. It is unclear whether this proposed method can scale to larger models.

Furthermore, the teacher and student pairs are all from the same model family, which reduces the learning difficulty.

Questions for authors
  1. Is the proposed method applicable to cases where teacher and student are not from the model family?
  2. Would O2PD be effective for cases where there is a larger size mismatch between teacher and student?
3 comments

We thank the reviewer for the comments and for the positive assessment. The two items below are the questions you asked to be answered.

1. Does the method apply when the teacher and the student are not from the same model family?

Overlap ρ=∣T∩S∣/k\rho=|T\cap S|/k compares token ids in the teacher and student top-kk. Those ids must refer to the same word for TT and SS to have an intersection, and only then can covering and pinning be switched. Across families the vocabularies usually differ, so the same id is not necessarily the same word, the intersection is undefined, and the present method cannot be applied directly. Adapting the method across vocabularies is difficult, because T∩ST\cap S stays undefined until the token ids are aligned. We would still be glad to explore that adaptation.

2. Does O$^{2}$PD remain effective at a larger size gap?

The largest teacher in the paper is 4B. We add a pair with a larger size gap: Qwen3.5-4B ←\leftarrow Qwen3.5-35B-A3B, with the same training and evaluation setup as in the paper. The table uses the same protocol as the main table: Avg@8 on AIME24/25/26 and HMMT25, pass@1 on Minerva, Olympiad, and MATH500, and Avg is the arithmetic mean of the seven.

MethodAIME24AIME25AIME26HMMT25MinervaOlympiadMATH500Avg
Student61.6739.1748.3335.8352.9460.4487.6055.14
Teacher83.7573.7585.4372.5058.569.591.876.46
————————
OPD70.4247.9259.5839.1755.5163.2687.6060.49
ESR71.2549.1764.5845.0054.4164.0088.2062.37
EOPD73.7550.4263.3343.3357.7265.4889.2063.32
TIP72.0855.0065.4244.1755.8864.8990.8064.03
O$^{2}$PD74.5859.5868.3350.0058.0966.0790.6066.75
Δ vs. OPD+4.16+11.66+8.75+10.83+2.58+2.81+3.00+6.26

The table shows that our O$^{2}$PD reaches 66.75 Avg performance, +6.26 above OPD, and stays above other strong baselines ESR, EOPD, and TIP. This verifies that our method still holds when a much larger teacher distills a smaller student.

Thank you to the authors for the response. I maintain the original assessment of weak accept.

We thank you for reading the response and for the continued support. We are glad that the additional larger-gap pair was useful, and we remain happy to answer any further questions.

Reviewer_mrTdOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies two questions in on-policy distillation (OPD): which tokens to align and how to align them. For the first, it introduces a cross-prefix completion protocol showing that teacher supervision is reliable on the prefix of student rollouts but degrades on the suffix — the student's pass@1 rises when continuing from teacher-generated prefixes while the teacher's pass@1 falls when continuing from student-generated prefixes — and it therefore restricts the distillation loss to the first 20% of each student rollout. For the second, it formally characterizes two failure modes of top-k RKL OPD: Teacher-support Uncovering (the covering gradient on teacher-only tokens is proportional to student mass, which vanishes there) and Dominant-Mode Mismatch (renormalized RKL matches only the distribution shape on the teacher top-k, not the total mass). It then proposes O2PD, an overlap-gated α-divergence: a reset atom pools all mass outside the support; FKL on the teacher top-k plus the reset atom covers teacher-only tokens at low overlap, while RKL on the intersection plus the reset atom pins teacher mass on shared tokens at high overlap; the per-token top-k overlap ρ continuously interpolates α between the two KL directions and switches the support at the batch median. Experiments on three student–teacher pairs and seven benchmarks compare against OPD, ESR, EOPD, and TIP: O2PD improves over OPD by +1.89/+4.81/+5.29 average points and exceeds all baselines, with five-component ablations, prefix/suffix sweeps, and a support-switch-percentile study; anonymous code is provided.

Critical strengths

1. A rigorous characterization of top-k OPD's failure modes, with formal statements and empirical confirmation. Lemma 1 and Props. 1–4 pin down exactly why the default recipe fails: the covering gradient on teacher-only tokens is proportional to student mass (so uncovered tokens stay uncovered, Fig. 3a/b), and renormalized RKL matches only the shape on the teacher top-k, not the total mass (so even high-overlap tokens remain under-aligned, Fig. 3c). This is consequential because it converts folklore about top-k OPD into a precise mechanism, and each design choice in O2PD follows directly from one of these statements rather than from intuition.

2. A clean, well-validated method evaluated against real OPD baselines. The method is evaluated against ESR, EOPD, and TIP (not only vanilla OPD) across three heterogeneous teacher–student pairs and seven benchmarks, improving over OPD by +1.89/+4.81/+5.29 with the best average in every pair. The five-component ablations (w/o How, w/o Where, FKL/RKL, w/o I, w/o r) place every variant between OPD and full O2PD, the prefix/suffix sweep and percentile study examine the main hyperparameters, and anonymous code is provided. This is consequential because the gains are attributable to the designed mechanisms rather than to pipeline effects, which is rare in this line of work.

Critical weaknesses

1. The method's two central design choices rest on implicit assumptions that are not examined.

  • First, the theory determines when FKL is needed in terms of probability mass: the covering channel is weak when the student's probability on teacher-only tokens is small (Prop. 1). The gate, however, triggers FKL from the set-overlap ρ, and the paper does not establish the connecting guarantee — that low ρ implies small student probability on the teacher's preferred tokens. A token outside the student's top-k can still carry nontrivial student mass, and conversely high ρ does not imply aligned mass (cf. Fig. 3c); the link between ρ and the mass-based failure mode is supported only empirically (Fig. 3a), with no discussion of when ρ fails as a proxy (e.g., sensitivity to k or to the mass distribution within the top-k). Moreover, the batch-relative normalization (low/high) implies that absolute overlap levels never matter — only rankings within a batch — which is not justified.
  • Second, the FKL branch implicitly assumes the teacher's high-probability tokens are worth covering: FKL forces student mass onto every teacher-preferred token, yet it is applied precisely to the low-ρ tokens where teacher and student disagree most — with no analysis of teacher errors, no role for teacher confidence in the gate (in contrast to EOPD's entropy-based mixing), and no robustness study under a noisy or partially incorrect teacher. This is consequential because both assumptions concern where the method's supervision comes from and when it can be trusted; if either fails, the method actively teaches the student to match unreliable or wrong teacher mass.

2. The prefix fraction ℓ=0.2 — the "Where" decision — is weakly justified for a parameter this central. The prefix sweep (Fig. 4b) tests only {0.2, 0.4, 0.6, 0.8, 1.0} and only in the NTRL setting: the curve decreases monotonically from 0.2 to 1.0, so the sweep cannot distinguish "0.2 is optimal" from "smaller fractions are better", and the other two teacher–student pairs are not swept. Moreover, the ratio is a fixed global constant: the paper criticizes ESR's fixed token cutoff for not scaling with response length, but scaling by length still does not make the cutoff trajectory-adaptive — the paper's own Fig. 2a shows teacher reliability declining continuously with prefix length, so the reliable-prefix boundary varies across rollouts, and trajectory-adaptive weighting is the natural alternative. This is consequential because the largest gains (+5.29) occur in exactly the setting where the sweep was run, leaving unclear how ℓ transfers to the other settings and how it interacts with the How components.

Questions for authors

(All questions are related to the weaknesses.)

  1. Please extend the prefix sweep to smaller fractions (e.g., 0.05, 0.1, 0.15) in at least two teacher–student pairs. The current curve decreases monotonically from 0.2 to 1.0, so it cannot show whether 0.2 is optimal or merely the smallest value tested; also discuss the trade-off as the number of supervised tokens shrinks. (Clarification request — material to the "Where" decision.)
  2. How do you guarantee that the teacher's top-k tokens are worth covering? Please analyze FKL's behavior when the teacher is overconfident or wrong (e.g., a teacher-noise or corruption study), and justify why teacher confidence or entropy does not enter the gate, given that EOPD mixes FKL based on teacher entropy. (Clarification request — material to the "How" decision.)
  3. The theory says FKL is needed when the student's probability on teacher-only tokens is small (Prop. 1), but the gate triggers FKL from the set-overlap ρ. Can you establish the guarantee that low ρ implies small student probability on the teacher's preferred tokens (and high ρ implies aligned mass), or state formally the conditions under which this holds? Please also discuss sensitivity to k (the reset atom's mass 1−p(T) grows as k shrinks; k=16 is used) and to the mass distribution within the top-k.

Note: The paper provides a rigorous diagnosis of top-k OPD's failure modes and a well-ablated method that outperforms real OPD baselines across three teacher–student pairs; the theory is a genuine contribution. However, the two central design decisions — gating on the set-overlap ρ, whose link to the mass-based theory is only empirical, and trusting the teacher's top-k under FKL without any error analysis — rest on unexamined assumptions, and the fixed 20% prefix ratio is validated only on a coarse grid in a single setting. I would raise my score if the authors provide reasonable responses to the questions above, in particular: an extended prefix sweep in multiple settings, an analysis of FKL's behavior under teacher errors, and a formal or empirical justification of the ρ-based gate.

3 comments

We thank the reviewer for the careful comments. The four items below are the questions you indicated would be considered for a higher score.

1. Does low ρ\rho mean that the student assigns little probability to the teacher's preferred tokens?

To show that, at low overlap, the student assigns little probability to teacher-only top-kk tokens, we measure the student's own top-kk mass q(S)q(S). Teacher-only tokens lie outside SS, so q(T∖S)≤1−q(S)q(T\setminus S)\le 1-q(S). In the table, mean is the average of per-reply means, and min and max are the two ends of those reply means.

ModelkkBefore meanminmaxAfter meanminmax
DSR1-1.5B160.99730.97830.99990.99800.98361.0000
DSR1-1.5B320.99880.98971.00000.99900.99211.0000
DSR1-1.5B640.99940.99291.00000.99960.99481.0000
Qwen3-1.7B160.99860.98931.00000.99900.99201.0000
Qwen3-1.7B320.99950.99411.00000.99960.99581.0000
Qwen3-1.7B640.99980.99671.00000.99990.99761.0000

At k=16k=16, the average 1−q(S)1-q(S) is only 0.0027 and 0.0014; even at the minimum it is only 0.0217 and 0.0107. After training the minima rise to 0.9836 and 0.9920, so the upper bound is smaller. The tail continues to shrink as kk grows. Low overlap only places more teacher tokens in this already small tail, and the student probability on those tokens remains small. We have resubmitted the revised manuscript. Further details are in Appendix G (highlighted in blue\color{blue}{\textbf{highlighted in blue}} in the revised PDF).

2. Why use relative overlap within a trajectory rather than absolute overlap?

The purpose of on-policy distillation is to align the student's next-token distribution to the teacher's on the student's own rollouts. The support switch decides which prefix tokens stay on Reset Support T∪rT\cup\\{r\\} and which move to Support Convergence on I∪rI\cup\\{r\\}. The overlap level moves with the teacher--student pair and with the training stage, so the switch is ρ0.5\rho_{0.5} of the current prefix rather than a fixed ρ\rho. An absolute threshold selects a share that changes with the distribution; a percentile selects a share that does not. The table gives the mean overlap of the prefix tokens, the first 20% of each rollout, at k=16k=16, 3232, and 6464.

PairStagek=16k=16k=32k=32k=64k=64
Qwen3-1.7B ← Qwen3-4B-NTRLBefore0.6450.6360.633
Qwen3-1.7B ← Qwen3-4B-NTRLAfter0.7080.6980.695
Qwen3-1.7B ← Qwen3-4BBefore0.7480.7200.702
Qwen3-1.7B ← Qwen3-4BAfter0.8610.8340.809

The mean is not the same for the two pairs, and it rises after training at every kk. Figures 9 and 10, newly added in the submitted revision, give the token-level overlap distributions: Figure 9 is before training and Figure 10 is after training. The tokens are not piled on that mean, so the overlap distribution itself moves with the teacher--student pair and with the training stage. A fixed ρ\rho therefore does not select a stable share of the prefix. Switching at ρ0.5\rho_{0.5} keeps the lower half of the current prefix on T∪rT\cup\\{r\\}.

A fixed threshold does not replace the percentile. On the same Qwen3-1.7B ←\leftarrow Qwen3-4B-NTRL pair, sweeping τ\tau, both Avg@8 and pass@8 mostly peak at τ=0.75\tau=0.75, where both peak at τ=1.0\tau=1.0. The summary Avg@8 of 40.83 is still below 41.94 at ρ0.5\rho_{0.5}. τ=0.5\tau=0.5 is only 39.31, comparable to ρ0.25\rho_{0.25} at 39.45, and does not reach ρ0.5\rho_{0.5}. The best absolute cutoff is still below switching at ρ0.5\rho_{0.5}. Here ρ0.5\rho_{0.5} (the switch point used in the paper) is the median percentile of the overlap values on the current prefix, not the absolute cutoff τ=0.5\tau=0.5: tokens with ρ<ρ0.5\rho\lt \rho_{0.5} stay on T∪rT\cup\\{r\\}, and tokens with ρ≥ρ0.5\rho\ge\rho_{0.5} move to I∪rI\cup\\{r\\}.

τ\tauAIME24 Avg@8pass@8AIME25 Avg@8pass@8HMMT25 Avg@8pass@8Avg@8pass@8
0.2550.4266.6740.0056.6722.0830.0037.5051.11
0.551.2566.6742.9263.3323.7536.6739.3155.56
0.7552.9270.0044.5866.6725.0040.0040.8358.89
1.053.3373.3342.0863.3323.3336.6739.5857.78
————————
ρ0.5\rho_{0.5}54.1773.3345.8370.0025.8343.3341.9462.22

The switch therefore remains ρ0.5\rho_{0.5} of the current prefix: the lower half stays on T∪rT\cup\\{r\\}, and that share does not change as overlap rises across pairs or during training. Further details are in Appendix H.

3. Are the teacher's top-kk tokens worth covering with Forward KL?

On-policy distillation aligns the student to the teacher's next-token distribution on the student's own rollouts [1, 6, 20, 28]. Forward KL (FKL\mathrm{FKL}) at low overlap copies that teacher distribution, not a token verified as correct. When the teacher concentrates its mass on one token, that is the mass placed on the student.

Entropy answers a different question, so it does not enter the gate. EOPD [13] mixes FKL\mathrm{FKL} into reverse KL (RKL\mathrm{RKL}) on high teacher-entropy tokens, to avoid diversity collapse. High entropy does not mean that the token is correct, and low entropy does not mean that it is wrong. Overconfidence is low entropy: the mass sits on one token. EOPD is then closer to RKL\mathrm{RKL}, and at low overlap the covering gradient of RKL\mathrm{RKL} is still multiplied by student mass, so that channel is closed. Entropy therefore does not mark an incorrect teacher token. TIP [26] uses student entropy and marks an overconfident student, which is likewise not a label of teacher-token correctness. In the main table O$^{2}$PD remains above EOPD (26.80 / 39.48 / 50.58 versus 25.74 / 35.97 / 47.62). On these three stronger teachers, switching Reset Support and Support Convergence by overlap scores higher than mixing FKL\mathrm{FKL} by teacher entropy. That comparison does not show that FKL\mathrm{FKL} is safe when the teacher is wrong. What it does support is a relative claim: the learning target is still the teacher rather than the student. In Figure 11, even if a teacher prefix can contain incorrect guidance, continuing from that prefix already raises mean pass@1 above the student generating the full response. Teacher prefixes are therefore more reliable than the student’s own prefixes in this setting, which is why we keep covering the teacher distribution at low overlap rather than adding a correctness filter. The following table is the seven-benchmark Avg from the main table for the three pairs in the paper. On the seven-benchmark average, each teacher scores higher than its student.

PairStudentTeacher
Qwen3-1.7B ←\leftarrow Qwen3-4B23.3736.42
DSR1-1.5B ←\leftarrow JustRL-1.5B30.9444.43
Qwen3-1.7B ←\leftarrow Qwen3-4B-NTRL23.3762.24

The teacher's stronger overall performance means that directly choosing the teacher's distribution as the learning target is simple and effective. There is no token-level correctness label, so the gate does not add another heuristic filter on teacher entropy, confidence, etc. The gap does not imply that every top-kk token at low ρ\rho is correct. In Figure 11, newly added in the submitted revision, when the student continues a teacher prefix, mean pass@1 on Minerva, Olympiad, and MATH500 rises with the prefix ratio. At ℓ=0.2\ell=0.2, using only part of the teacher prefix already exceeds the student generating the full response. Student alone in the table is the mean of that student's Minerva, Olympiad, and MATH500 scores in the main table.

PairStudent alone (three benchmarks)ℓ=0.2\ell=0.2
Qwen3-1.7B ←\leftarrow Qwen3-4B43.7144.7
DSR1-1.5B ←\leftarrow JustRL-1.5B43.5945.4
Qwen3-1.7B ←\leftarrow Qwen3-4B-NTRL43.7144.2

Thus, even if the teacher can give incorrect guidance, the teacher remains stronger than the student overall. Figure 12, newly added in the submitted revision, shows that distilling only the first 5% of tokens already exceeds the undistilled student on both pairs, while 20% is where O$^{2}$PD reaches its best score. We therefore keep the loss on the prefix rather than adding a further heuristic filter on teacher tokens.

4. Is the prefix ratio ℓ=0.2\ell=0.2 sufficient?

The original sweep ran from 0.2 to 1.0 and only with the Qwen3-4B-NTRL teacher, so it could not tell whether 0.2 was a peak or the smallest prefix fraction tested. Figure 12, newly added in the submitted revision, adds 0.05, 0.10, and 0.15, and adds DSR1-1.5B ←\leftarrow JustRL-1.5B. The table is the mean of Avg@8 on AIME24, AIME25, AIME26, and HMMT25.

ℓ\ellNTRL OPDNTRL O$^{2}$PDDSR1 OPDDSR1 O^{2}\PD
0.0533.837.826.429.0
0.1035.840.228.431.0
0.1537.242.230.932.5
0.2037.442.631.232.6
0.4036.842.031.532.4
0.6037.041.831.332.2
0.8036.341.331.032.0
1.036.040.530.831.8

Both pairs are lowest at 0.05, and the short prefixes rise. That segment is where having fewer supervised tokens costs performance: O$^{2}$PD with the Qwen3-4B-NTRL teacher rises from 37.8 to 42.6 at ℓ=0.20\ell=0.20, and with JustRL-1.5B it rises from 29.0 to 32.6. Lengthening the prefix further does not raise O$^{2}PD;thefulltrajectoryis40.5and31.8.OnDSR1,32.5at0.15isalreadycloseto32.6at0.20.OPD; the full trajectory is 40.5 and 31.8. On DSR1, 32.5 at 0.15 is already close to 32.6 at 0.20. O^{2}$PD is above OPD at every measured ratio. The OPD peak is not the same across pairs: it is at ℓ=0.20\ell=0.20 with the Qwen3-4B-NTRL teacher and at ℓ=0.40\ell=0.40 (31.5) with JustRL-1.5B.

Figure 11, newly added in the submitted revision, is why the loss is placed on the prefix: on all three pairs, pass@1 rises when the student continues a teacher prefix and falls when the teacher continues a student prefix. The ratio at which the drop occurs is not the same, so 0.20 is chosen from the downstream scores in Figure 12, newly added in the submitted revision. ℓ\ell remains a global fraction that scales with response length. Thus 0.20 is already enough: the first 20% of tokens carry the gain, and lengthening the prefix further may not help much. The 80/20 rule may still hold in the present setting. Further details are in Appendix I (highlighted in blue\color{blue}{\textbf{highlighted in blue}}) in the revised PDF).

Comments on the paper

1 comment

We thank all three reviewers for the careful comments, and we thank the Area Chair and Senior Area Chairs for their time.

We have resubmitted a revised manuscript. New figures, tables, and appendix sections are highlighted in blue\color{blue}{\textbf{highlighted in blue}} in the revised PDF. The replies below give the measurements that answer the questions you raised. Appendix G measures student mass on teacher-only tokens, for the link between low overlap and Proposition 1 (Figure 8). Appendix H compares the prefix-overlap percentile ρ0.5\rho_{0.5} with a fixed threshold in Table 7, and reports how the overlap distribution moves with the teacher--student pair and with the training stage (Figures 9 and 10). Appendix I extends the cross-prefix protocol to three pairs (Figure 11) and extends the prefix-ratio sweep to 0.05, 0.10, and 0.15 on two pairs (Figure 12). Beyond the revised manuscript, the replies also add a larger size-gap pair, Qwen3.5-4B ←\leftarrow Qwen3.5-35B-A3B. We hope these additions clarify the motivation, the two design choices, and the experimental findings. The reviewer-by-reviewer replies follow.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.