- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies which teacher signals are useful in on-policy distillation. It distinguishes disagreement aligned with the student’s local support from disagreement outside that support, and proposes TA-OPD to select teachable token positions. Experiments show that supervising a small subset of token positions can match or outperform full-token OPD.
Critical strengths
The distinction between disagreement magnitude and support alignment is interesting. The fixed-context diagnostics and controlled ablations provide useful evidence, and the method is simple and empirically evaluated across multiple teacher–student settings.
Critical weaknesses
The paper prioritizes support-aligned corrections, but its explanation of why support-misaligned corrections are less teachable relies mainly on empirical observations and lacks a clear mechanistic explanation.
Questions for authors
Why are support-misaligned corrections less teachable? Could they also help OPD, and under what conditions would they be beneficial?
4 comments
Dear Reviewer,
Thank you so much for your thoughtful initial reviews!
With only two days left before the discussion window closes, I wanted to send a friendly reminder to check out the authors' rebuttal. They have put a lot of effort into answering the specific questions you raised.
Whenever you have a spare moment today, please drop a brief reply in the thread to let the authors know if their responses have addressed your concerns, or if any questions remain. Your timely feedback makes a huge difference in ensuring a fair and constructive review process!
Warm regards
Thank you for the clarification. The reverse-KL gradient analysis and finite-sampling explanation substantially address my concern about the mechanism. I will maintain my score.
Thank you again for your careful follow-up and constructive discussion. We are glad that our responses and analysis have addressed your main concerns. The revised manuscript now includes the reverse-KL gradient analysis and finite-sampling explanation.
Thank you for your insightful comment and for highlighting the need for a clearer mechanistic explanation. We have revised the PDF to strengthen this discussion and added a figure (Appendix D, Figure 7) illustrating the state-dependent nature of token teachability.
W&Q: Why are support-misaligned corrections less teachable?
The key mechanism arises from the optimization geometry of reverse-KL OPD and its finite-sampling implementation. For the reverse-KL objective in Eq. (6), the gradient with respect to a student logit zv is
$ \frac{\partial D_{\mathrm{KL}}(p_\theta|p_T)}{\partial z_v} =p_\theta(v)\left[\log\frac{p_\theta(v)}{p_T(v)}-D_{\mathrm{KL}}(p_\theta|p_T)\right]. $
When a teacher-preferred token v⋆ has very low student probability, its direct logit-gradient contribution is suppressed by $ p_\theta (v^\star) $, despite potentially large disagreement. In sampled-token OPD, this alternative is also rarely sampled, further limiting how frequently its teacher–student discrepancy directly contributes to stochastic updates. In contrast, support-aligned corrections involve alternatives already assigned appreciable student probability, making their feedback more readily accessible under finite sampling.
Can support-misaligned corrections still help? Yes. Updates on other sampled tokens can indirectly redistribute probability toward v⋆. Moreover, teachability is state-dependent: as v⋆ gains probability and enters the student's top-(K), both teacher support coverage and its direct sampling frequency can increase. Our new schematic illustrates this possible transition under a fixed teacher distribution. Greater exploration, continued training, or direct supervision, such as full-distribution distillation may therefore make initially off-support corrections more useful.
Experiment results: Appendix D, Table 11 provides a matched-ratio Q3 intervention. High-DI positions achieved a positive bootstrap gain (+0.006±0.003), but a slightly negative matched gain (−0.003). High-DL positions yielded larger gains (+0.015±0.010 and +0.010, respectively). These observations are consistent with weaker, rather than absent, learning from support-misaligned corrections. We emphasize that the gradient analysis provides a local optimization mechanism, while the schematic is illustrative rather than an empirical training trajectory. Together, they clarify why support alignment can matter without implying that off-support corrections are intrinsically unlearnable.