- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the "teacher unanchoring" problem in on-policy distillation (OPD) for LLMs, where a student policy drifts outside the teacher’s acceptance region, causing the teacher to evaluate states it rarely visits and deliver noisy or uninformative feedback. The authors propose Interactive-Policy Distillation (IPD), an interactive rollout and distillation framework
Critical strengths
A Principled Bridge Between On-Policy and Off-Policy Distillation:Unlike prior work that relies on fixed teacher prefixes (TrOPD) or single-token teacher corrections (SKD), IPD introduces a dynamic bidirectional state machine governed by top- acceptance regions. Pairing this interactive trajectory generation with a source-split loss (reverse KL for on-policy tokens, forward KL for teacher-generated demonstrations) provides a mathematically natural and empirically superior formulation.
Essential Systems Contribution (Fused Inference Engine):Interleaved multi-model rollouts typically suffer from prohibitive communication latency and continuous KV cache invalidation in distributed training frameworks. By developing a fused engine co-hosting both models with synchronized cache alignment, parallel chunk verification, and role-grouped scheduling, the authors eliminate the main deployment barrier for interactive distillation, achieving up to a 71.4$\times$ throughput improvement.
Critical weaknesses
End-to-End Wall-Clock Overhead Relative to Standard OPD:
While Figure 3 convincingly shows that the fused engine is 21$\times$–71$\times$ faster than a naive two-engine implementation, the paper does not report the end-to-end wall-clock training time of IPD versus standard OPD. Standard OPD only executes student autoregressive decoding followed by a single teacher prefill. In IPD, teacher takeovers require additional teacher autoregressive decoding and student prefill passes. Demonstrating that IPD's data efficiency (Figure 5) translates into net wall-clock training gains over OPD would significantly strengthen the practical claim.
Questions for authors
NA
Responses to Reviewer 8RsB
First, we would like to thank the reviewer for recognizing our contributions on a principled bridge between on-policy and off-policy distillation, which differs from prior work and provides a mathematically natural and empirically superior formulation, and essential inference infrastructures for the non-trivial Interleaved multi-model rollouts, as well as your valuable feedback during the reviewing stage. Regarding your concerns, we provide our responses as follows.
W1: Wall-clock Time Analysis
For 4B to 1.7B-Base group, we calculate the average wall-clock time per step during OPD/SKD/IPD training:
Given that IPD needs frequent switches between the large model and the small model, its wall-clock time is naturally longer than OPD. Meanwhile, this per-step overhead can be hedged by the higher data efficiency and faster convergence of IPD. We also updated our manuscript with a new subsection (Appendix C.3, marked in blue), where we illustrate the time-performance curve of OPD and IPD in Figure 10. Specifically, under the same time budget, IPD yields a better student. For the same-level student performance, IPD needs less total training time.
Again, we thank the reviewer's effort and valuable feedback, which helps us further improve the quality of our paper. We hope our responses can address your concerns.