- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces Interpolated Policy Distillation (IPD), which generates training rollouts from a token-level interpolation of student and teacher distributions. A speculative sampler preserves this target distribution exactly, while its acceptance decisions determine the supervision: accepted student proposals receive an OPD-style objective, and residual corrections receive cross-entropy supervision. Experiments on text and multimodal reasoning show improvements in average accuracy over the evaluated baselines, particularly for weak and pruned students. Ablations investigate the rollout policy, interpolation strength, and sensitivity to SFT initialization.
Critical strengths
- Explicit and controllable rollout policy. IPD specifies the student-teacher combination directly at the next-token distribution level, and derives a sampler that preserves this distribution. This provides a clear probabilistic account of teacher intervention and distinguishes the approach from heuristic switching rules.
- Substantial gains in challenging student settings. IPD improves average accuracy across the evaluated text and multimodal configurations. Its gains are especially pronounced for the 0.6B base student and the pruned vision-language student, supporting its relevance when student-generated trajectories are unreliable.
Critical weaknesses
- The practical efficiency claim lacks quantitative validation. Although speculative sampling reduces sequential teacher queries relative to the direct-sampling prototype, the paper reports no end-to-end training time, rollout throughput, or memory measurements, and does not specify the draft-window length. Consequently, the computational cost of the accuracy improvements, especially relative to vanilla OPD, remains unclear.
- The evidence for the proposed quality–learnability explanation is weakened by the trajectory-quality metric. The judge prompt in Appendix A.3 explicitly excludes reasoning correctness and answer accuracy, despite the surrounding discussion interpreting higher scores as indicating more valid reasoning. This limits what Figure 2(b) establishes about the claimed trade-off, although it does not invalidate the downstream accuracy results.
Questions for authors
- What draft-window length was used in the reported experiments? Are measurements available for end-to-end training time and rollout throughput relative to vanilla OPD and the direct-sampling prototype, under the same hardware and training workload?
- Does the judge prompt shown in Appendix A.3 match the prompt actually used for Figure 2(b)? If so, how should the stated interpretation of this score as indicating more valid reasoning be reconciled with its explicit exclusion of reasoning correctness?
5 comments
Author Response
We sincerely thank the reviewer again for your time and constructive feedback during the rebuttal process. We hope our detailed responses and additional experiments have adequately addressed your concerns and clarified our contributions.
We would greatly appreciate any further feedback and would be happy to clarify any remaining concerns. Thanks again for supporting this work and we would also greatly appreciate your reconsideration of our work in light of the additional evidence provided.
Author Response
We sincerely apologize for a labeling error in our previous rebuttal: the "Matched" and "Unmatched" columns for Relay-OPD were inadvertently reversed. We have corrected the labels in our updated response. This error does not affect the experimental results or our conclusions. We apologize again for the confusion and thank the reviewers for their understanding.
Author Response
We sincerely thank the reviewer for recognizing the strengths of our work and providing constructive suggestions. We also appreciate your careful identification of the errors in our manuscript. We appreciate the opportunity to address your concerns and clarify our findings. We sincerely apologize for these mistakes.
Thanks for your question. We used a draft-window length of 5 in our experiments. The computational cost comparison is presented below. We conducted a controlled efficiency comparison on 8 GPUs (4 for rollout and 4 for teacher scoring), using Qwen3-0.6B as the student and Qwen3-4B as the teacher. We used a batch size of 8, a response length of 512, and 4,096 generated tokens per step. For IPD, we set γ=0.1 and the draft-window length to k=5.
Times and throughput are measured at the second training step. Teacher verifier calls are accumulated over two steps and exclude the common post-rollout teacher scoring.
Compared with direct sampling, IPD achieves a 3.68× rollout speedup and a 3.12× per-step speedup, while reducing sequential teacher verification calls by 4.78×. The proposal acceptance rate reaches 98.38%, explaining the effectiveness of block-wise verification. Compared with vanilla OPD, IPD incurs approximately 1.68× per-step wall-clock overhead, reflecting the additional cost of teacher-guided rollout generation.
Q2: Does the judge prompt shown in Appendix A.3 match the prompt actually used for Figure 2(b)? If so, how should the stated interpretation of this score as indicating more valid reasoning be reconciled with its explicit exclusion of reasoning correctness?
We sincerely thank the reviewer for identifying this important issue. We acknowledge that the manuscript contains two significant mistakes in the descriptions of trajectory quality and learnability. We apologize for these mistakes and would like to clarify them individually.
- Trajectory Quality. The LLM judge prompt presented in Appendix A.3 is indeed the one used in our experiments. The statement in the Introduction, "whether it constitutes a correct and valid reasoning process," was therefore inaccurate, and we will correct throughout the manuscript. We intentionally did not require the generated trajectories to produce correct final answers when evaluating trajectory quality. Our intention was to measure the readability, coherence, and structural well-formedness of the generated reasoning trajectories. This choice is motivated by the characteristics of practical OPD training. In our setting, most training rollouts are truncated because the reasoning problems typically require substantially more tokens to solve than the maximum rollout length permits. Using final-answer correctness as the primary proxy for trajectory quality would therefore be inappropriate for these truncated trajectories.
We also conduct an additional experiment with a sufficiently large rollout length, using final-answer correctness as an alternative quality metric:
| γ | 0 | 0.05 | 0.1 | 0.2 | 0.5 | 1.0 |
|:---|---:|---:|---:|---:|---:|---:|
| Correctness (%) ↑ | 8.1 | 23.2 | 33.6 | 36.6 | 59.5 | 79.8 |
The results show that trajectory correctness also increases with stronger teacher intervention, supporting the quality trend observed in Figure 2(b).
- Learnability. We would also like to clarify an error in the mathematical definition of our learnability proxy. The metric actually used in our experiments computes the mean absolute sampled-token log-ratio, rather than taking the absolute value after averaging:
-\frac{1}{T}\sum_{t=1}^{T} \left| \log\frac{\pi_\theta(y_t\mid s_t)} {\pi_T(y_t\mid s_t)} \right| \right).$$ The manuscript mistakenly places the absolute value outside the summation. This is a transcription error, which we will correct. We sincerely appreciate the reviewer's time and effort during the rebuttal process, as well as your positive feedback and recognition of our work. We hope our responses have adequately addressed your concerns. Thank you again for your support and constructive engagement throughout the discussion.