- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Overlap-gated On-Policy Distillation (O²PD) to address two issues in on-policy knowledge distillation: (1)where should distillation be applied, and (2) how should teacher and student distributions be aligned. For (1), the authors identify a cross-prefix asymmetry, which means teacher performance decreases when conditioned on longer and longer student-generated prefixes. This suggests that teacher supervision is more dependable early in student rollouts. So O²PD constrains distillation to the first 20% of the generated tokens. For (2), the authors find two limitations of conventional top-k reverse KL distillation: Teacher-support Uncovering and Dominant-Mode Mismatch. O²PD introduces reset support and support convergence with an an overlap-gated -divergence to address. The experiments on math reasoning benchmarks and teacher–student pairs show O²PD gains improvements over OPD and ablation supports the design.
Critical strengths
-
The paper provides a useful analysis of the limitations of existing top-k OPD methods. The authors identify two issues, Teacher-support Uncovering and Dominant-Mode Mismatch, and provide theoretical analysis to explain them. This helps motivate the proposed support strategies and gives a clearer understanding of why standard reverse KL may be insufficient.
-
The method shows consistent improvements across different teacher–student pairs. O²PD outperforms the compared OPD baselines across three teacher–student pairs and seven math reasoning benchmarks, with gains of up to 5.29 points over standard OPD. The ablation studies also provide some evidence for the effectiveness of the proposed components.
Critical weaknesses
-
The evidence showed in this paper for prefix-only alignment is not fully convincing. The drop in teacher accuracy on longer student prefixes may simply be due to mistakes already made by the student. This does not necessarily mean the teacher provides worse token-level supervision at later positions. The analysis is also limited to one model pair on MATH500. It would be useful to see whether the same pattern holds in other settings.
-
Some key design choices are not well justified. The theory explains the motivation for using different support strategies, but it is less clear why the authors choose median-based switching and min–max normalization for α. These choices seem somewhat arbitrary. Also, removing support convergence only drops the score from 50.6 to 49.9, so I am not sure how important this component is in practice.
Questions for authors
please see the weakness.
5 comments
2. Why switch support at , and why map from the prefix overlap min and max?
The purpose of on-policy distillation is to align the student's next-token distribution to the teacher's on the student's own rollouts. The support switch decides which prefix tokens stay on Reset Support and which move to Support Convergence on . The overlap level moves with the teacher--student pair and with the training stage, so the switch is of the current prefix rather than a fixed . An absolute threshold selects a share that changes with the distribution; a percentile selects a share that does not. The table gives the mean overlap of the prefix tokens, the first 20% of each rollout, at , , and .
| Pair | Stage | |||
|---|---|---|---|---|
| Qwen3-1.7B ← Qwen3-4B-NTRL | Before | 0.645 | 0.636 | 0.633 |
| Qwen3-1.7B ← Qwen3-4B-NTRL | After | 0.708 | 0.698 | 0.695 |
| Qwen3-1.7B ← Qwen3-4B | Before | 0.748 | 0.720 | 0.702 |
| Qwen3-1.7B ← Qwen3-4B | After | 0.861 | 0.834 | 0.809 |
The mean is not the same for the two pairs, and it rises after training at every . Figures 9 and 10, newly added in the submitted revision, give the token-level overlap distributions: Figure 9 is before training and Figure 10 is after training. The tokens are not piled on that mean, so the overlap distribution itself moves with the teacher--student pair and with the training stage. A fixed therefore does not select a stable share of the prefix. Switching at keeps the lower half of the current prefix on .
A fixed threshold does not replace the percentile. On the same Qwen3-1.7B Qwen3-4B-NTRL pair, sweeping , both Avg@8 and pass@8 mostly peak at , where both peak at . The summary Avg@8 of 40.83 is still below 41.94 at . is only 39.31 and does not reach . The best absolute cutoff is still below switching at . Here (the switch point used in the paper) is the median percentile of the overlap values on the current prefix, not the absolute cutoff : tokens with stay on , and tokens with move to .
| AIME24 Avg@8 | pass@8 | AIME25 Avg@8 | pass@8 | HMMT25 Avg@8 | pass@8 | Avg@8 | pass@8 | |
|---|---|---|---|---|---|---|---|---|
| 0.25 | 50.42 | 66.67 | 40.00 | 56.67 | 22.08 | 30.00 | 37.50 | 51.11 |
| 0.5 | 51.25 | 66.67 | 42.92 | 63.33 | 23.75 | 36.67 | 39.31 | 55.56 |
| 0.75 | 52.92 | 70.00 | 44.58 | 66.67 | 25.00 | 40.00 | 40.83 | 58.89 |
| 1.0 | 53.33 | 73.33 | 42.08 | 63.33 | 23.33 | 36.67 | 39.58 | 57.78 |
| — | — | — | — | — | — | — | — | |
| 54.17 | 73.33 | 45.83 | 70.00 | 25.83 | 43.33 | 41.94 | 62.22 |
is mapped from the same prefix-overlap min and max: the lowest overlap in the current prefix is mapped toward Forward KL, and the highest toward reverse KL. The switch therefore remains of the current prefix: the lower half stays on , and that share does not change as overlap rises across pairs or during training. Further details are in Appendix H.
3. What is the role of Support Convergence in practice?
w/o keeps the prefix, the reset atom, and varying with , and always uses the support . It no longer switches to at high overlap. On Qwen3-1.7B Qwen3-4B-NTRL, OPD is 45.29, w/o is 49.9, and full O$^{2}$PD is 50.58. The prefix, the reset atom, and already raise the score from 45.29 to 49.9; switching to adds about 0.7.
The role of Support Convergence is, once overlap has risen, to move the support to , fold the leftover teacher mass on into the reset atom, and pin the teacher's mass on the shared atoms, rather than leftover-renormalizing them on . The entropy curves in Figure 6 of the paper correspond to this role: entropy falls during training at every percentile, and the closer the percentile is to , the more tokens remain on and the larger the drop; the entropy at is not the lowest, so the student can still keep exploration. The table reads out the four curves in Figure 6. The last column is the drop at step 150 relative to .
| Percentile | 1 | 25 | 50 | 75 | 100 | 125 | 150 | Drop vs. |
|---|---|---|---|---|---|---|---|---|
| 0.278 | 0.258 | 0.218 | 0.229 | 0.236 | 0.208 | 0.207 | 0% | |
| 0.255 | 0.18 | 0.199 | 0.204 | 0.188 | 0.186 | 0.186 | 10.3% | |
| 0.260 | 0.262 | 0.199 | 0.175 | 0.166 | 0.177 | 0.169 | 18.6% | |
| 0.245 | 0.272 | 0.167 | 0.142 | 0.144 | 0.159 | 0.136 | 34.6% |
leaves almost the whole prefix on , and at step 150 its entropy is 34.6% below . is 10.3% below, and avoids the largest drop. has the highest entropy, but its three-benchmark Avg is only 39.45, below 41.94 at . Moving the support to under Support Convergence avoids an excessive drop in actor entropy.
We thank the reviewer for the careful comments. The three items below are the weaknesses you stated.
1. Is the evidence for prefix alignment sufficient?
The original cross-prefix protocol used only Qwen3-1.7B ← Qwen3-4B on MATH500. Figure 11, newly added in the submitted revision, extends it to three teacher--student pairs. The metric is mean pass@1 on Minerva, Olympiad, and MATH500. All three pairs show the same pattern: the score rises when the student continues a teacher prefix and falls when the teacher continues a student prefix. At ℓ=0.2 the three students are already 44.7, 45.4, and 44.2, above the pass@1 of 43.71, 43.59, and 43.71 when each student generates the full response. Using only part of the teacher prefix already exceeds the student alone, so teacher supervision on the prefix is effective.
Fig. 4b measured only 0.2,0.4,0.6,0.8,1.0 with the Qwen3-4B-NTRL teacher. Figure 12, newly added in the submitted revision, adds 0.05, 0.10, and 0.15, and adds DSR1-1.5B ← JustRL-1.5B. The table gives the values in Figure 12, newly added in the submitted revision. The metric is again the mean of Avg@8 on AIME24, AIME25, AIME26, and HMMT25.
Both pairs are lowest at ℓ=0.05, because the supervised prefix is too short to carry the signal, and then rise through the short prefixes. The experiments find that 20% of the tokens are already enough to bring a gain and often reach a strong result, so we adopt this simple and effective choice. The 80/20 rule may still hold in the present setting. Further details are in Appendix I (highlighted in blue in the revised PDF).