acceptodds
Under review as a conference paper at ICLR 2027

Interpolated Policy Distillation: A Controllable Continuum Between Off-Policy and On-Policy Distillation

Abstract

Off-policy and on-policy distillation have traditionally been formulated as separate paradigms, each favoring a different property of distillation trajectories. Teacher-generated (off-policy) traces are typically high-quality but lie far from the student's distribution, whereas student-generated (on-policy) rollouts are more learnable but often contain erroneous reasoning. We view these paradigms as the endpoints of a policy continuum and posit that a more effective rollout policy may lie in between. We introduce **Interpolated Policy Distillation (IPD)**, which defines the next-token distribution at every decoding step as an explicit linear interpolation between the student and teacher distributions. The interpolation operates at the distribution level, token by token, and its coefficient provides direct control over the balance between trajectory quality and student learnability. Naively sampling from this policy would require sequentially querying the teacher at every token and is thus expensive. To make IPD practical, we accelerate it with a new speculative-decoding rule while exactly preserving the interpolated next-token distribution.At the trajectory level, the resulting rollouts naturally interleave student- and teacher-generated segments. Unlike recent heuristic segment-interleaving methods, however, this interleaving is induced by an exactly realized token-level interpolated policy rather than by hand-designed switching rules.Across text-only and multimodal reasoning benchmarks, IPD consistently outperforms both endpoint policies (SFT and OPD), their conventional two-stage combination (SFT-then-OPD), and recent heuristic segment-interleaving methods, demonstrating that token-level policy interpolation better balances trajectory quality and student learnability.

Primary area:
learning theory
Keywords:
LLM, On-policy distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vFfXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Interpolated Policy Distillation (IPD), which generates training rollouts from a token-level interpolation of student and teacher distributions. A speculative sampler preserves this target distribution exactly, while its acceptance decisions determine the supervision: accepted student proposals receive an OPD-style objective, and residual corrections receive cross-entropy supervision. Experiments on text and multimodal reasoning show improvements in average accuracy over the evaluated baselines, particularly for weak and pruned students. Ablations investigate the rollout policy, interpolation strength, and sensitivity to SFT initialization.

Critical strengths
  1. Explicit and controllable rollout policy. IPD specifies the student-teacher combination directly at the next-token distribution level, and derives a sampler that preserves this distribution. This provides a clear probabilistic account of teacher intervention and distinguishes the approach from heuristic switching rules.
  2. Substantial gains in challenging student settings. IPD improves average accuracy across the evaluated text and multimodal configurations. Its gains are especially pronounced for the 0.6B base student and the pruned vision-language student, supporting its relevance when student-generated trajectories are unreliable.
Critical weaknesses
  1. The practical efficiency claim lacks quantitative validation. Although speculative sampling reduces sequential teacher queries relative to the direct-sampling prototype, the paper reports no end-to-end training time, rollout throughput, or memory measurements, and does not specify the draft-window length. Consequently, the computational cost of the accuracy improvements, especially relative to vanilla OPD, remains unclear.
  2. The evidence for the proposed quality–learnability explanation is weakened by the trajectory-quality metric. The judge prompt in Appendix A.3 explicitly excludes reasoning correctness and answer accuracy, despite the surrounding discussion interpreting higher scores as indicating more valid reasoning. This limits what Figure 2(b) establishes about the claimed trade-off, although it does not invalidate the downstream accuracy results.
Questions for authors
  1. What draft-window length was used in the reported experiments? Are measurements available for end-to-end training time and rollout throughput relative to vanilla OPD and the direct-sampling prototype, under the same hardware and training workload?
  2. Does the judge prompt shown in Appendix A.3 match the prompt actually used for Figure 2(b)? If so, how should the stated interpretation of this score as indicating more valid reasoning be reconciled with its explicit exclusion of reasoning correctness?
5 comments
AuthorsOpenReview ↗

Author Response

We sincerely thank the reviewer for recognizing the strengths of our work and providing constructive suggestions. We also appreciate your careful identification of the errors in our manuscript. We appreciate the opportunity to address your concerns and clarify our findings. We sincerely apologize for these mistakes.

  • Q1: What draft-window length was used in the reported experiments? Are measurements available for end-to-end training time and rollout throughput relative to vanilla OPD and the direct-sampling prototype, under the same hardware and training workload?

Thanks for your question. We used a draft-window length of 5 in our experiments. The computational cost comparison is presented below. We conducted a controlled efficiency comparison on 8 GPUs (4 for rollout and 4 for teacher scoring), using Qwen3-0.6B as the student and Qwen3-4B as the teacher. We used a batch size of 8, a response length of 512, and 4,096 generated tokens per step. For IPD, we set γ=0.1\gamma=0.1 and the draft-window length to k=5k=5.

MethodStep Time (s)Rollout Time (s)E2E Throughput (tok/s)Teacher Verifier Calls
Vanilla OPD7.684.67533.10
Direct Sampling40.3537.17101.54,108
IPD (k=5k=5)12.9310.10348.4860

Times and throughput are measured at the second training step. Teacher verifier calls are accumulated over two steps and exclude the common post-rollout teacher scoring.

Compared with direct sampling, IPD achieves a 3.68× rollout speedup and a 3.12× per-step speedup, while reducing sequential teacher verification calls by 4.78×. The proposal acceptance rate reaches 98.38%, explaining the effectiveness of block-wise verification. Compared with vanilla OPD, IPD incurs approximately 1.68× per-step wall-clock overhead, reflecting the additional cost of teacher-guided rollout generation.

  • Q2: Does the judge prompt shown in Appendix A.3 match the prompt actually used for Figure 2(b)? If so, how should the stated interpretation of this score as indicating more valid reasoning be reconciled with its explicit exclusion of reasoning correctness?

    We sincerely thank the reviewer for identifying this important issue. We acknowledge that the manuscript contains two significant mistakes in the descriptions of trajectory quality and learnability. We apologize for these mistakes and would like to clarify them individually.

  1. Trajectory Quality. The LLM judge prompt presented in Appendix A.3 is indeed the one used in our experiments. The statement in the Introduction, "whether it constitutes a correct and valid reasoning process," was therefore inaccurate, and we will correct throughout the manuscript. We intentionally did not require the generated trajectories to produce correct final answers when evaluating trajectory quality. Our intention was to measure the readability, coherence, and structural well-formedness of the generated reasoning trajectories. This choice is motivated by the characteristics of practical OPD training. In our setting, most training rollouts are truncated because the reasoning problems typically require substantially more tokens to solve than the maximum rollout length permits. Using final-answer correctness as the primary proxy for trajectory quality would therefore be inappropriate for these truncated trajectories. We also conduct an additional experiment with a sufficiently large rollout length, using final-answer correctness as an alternative quality metric: | γ\gamma | 0 | 0.05 | 0.1 | 0.2 | 0.5 | 1.0 | |:---|---:|---:|---:|---:|---:|---:| | Correctness (%) ↑\uparrow | 8.1 | 23.2 | 33.6 | 36.6 | 59.5 | 79.8 | The results show that trajectory correctness also increases with stronger teacher intervention, supporting the quality trend observed in Figure 2(b).
  2. Learnability. We would also like to clarify an error in the mathematical definition of our learnability proxy. The metric actually used in our experiments computes the mean absolute sampled-token log-ratio, rather than taking the absolute value after averaging:
-\frac{1}{T}\sum_{t=1}^{T} \left| \log\frac{\pi_\theta(y_t\mid s_t)} {\pi_T(y_t\mid s_t)} \right| \right).$$ The manuscript mistakenly places the absolute value outside the summation. This is a transcription error, which we will correct. We sincerely appreciate the reviewer's time and effort during the rebuttal process, as well as your positive feedback and recognition of our work. We hope our responses have adequately addressed your concerns. Thank you again for your support and constructive engagement throughout the discussion.
AuthorsOpenReview ↗

Author Response

We sincerely thank the reviewer again for your time and constructive feedback during the rebuttal process. We hope our detailed responses and additional experiments have adequately addressed your concerns and clarified our contributions.

We would greatly appreciate any further feedback and would be happy to clarify any remaining concerns. Thanks again for supporting this work and we would also greatly appreciate your reconsideration of our work in light of the additional evidence provided.

AuthorsOpenReview ↗

Author Response

We sincerely apologize for a labeling error in our previous rebuttal: the "Matched" and "Unmatched" columns for Relay-OPD were inadvertently reversed. We have corrected the labels in our updated response. This error does not affect the experimental results or our conclusions. We apologize again for the confusion and thank the reviewers for their understanding.

Reviewer_F2HrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes IPD that defines the next-token distribution as the interpolation of the teacher token distribution and the student token distribution. It uses speculative sampling to generate rollouts from this distribution and applies OPD loss to accepted student proposals and SFT loss to teacher-guided correction tokens. The paper reports improvements on text-only and multimodal reasoning benchmarks, especially for weak or pruned students.

Critical strengths
  • The paper proposes a simple interpolation-based approach to OPD.
  • Overall, the motivation is clearly presented for each component of the method.
  • There are some performance improvements across different student settings and benchmarks.
Critical weaknesses
  • The related work is so weak. First of all, only covering hybrid student-teacher rollouts in Section 2.3 is not right. The authors should also cover on-policy and off-policy distillation in general. GKD and the DistillLLM line of work should also be discussed more clearly in relation to IPD. Also, the discussion misses the very similar and related work of MiniLLM [1]. The authors should be more careful in covering related work.
  • This also connects to the main concern about novelty. MiniLLM already interpolates the teacher distribution and the student distribution at each decoding step through teacher-mixed sampling. So basically, the naive interpolation in Eq. (1) is the same token-level linear mixture used in MiniLLM. The authors should clarify what is new beyond this existing rollout policy, particularly in speculative sampling and verification-aware supervision.
  • Also, I think this paper should cover OPD works that combine forward KL and reverse KL, like EOPD [2], as fundamentally they are doing similar jobs by supplementing reverse-KL supervision with selective mass-covering supervision. I know that the mechanisms are different as EOPD uses entropy-gated forward KL, whereas IPD uses CE on residual correction tokens. But the connection should be discussed, and EOPD should also be considered as a baseline. So missing baselines are another weakness. There are not many PPO-based OPD baselines in the main table for this paper.
  • The justification for the loss design is also not fully convincing. The authors acknowledge that the PPO ratio does not correct for sampling from the mixed policy, but a small total variation distance alone does not establish that the log-ratio-based update is close to an on-policy update. Also, acceptance does not necessarily imply a small log-ratio, as proposals with higher teacher probability than student probability are always accepted under Eq. (3). I believe more evidence is needed for the claimed stability benefit of acceptance-based loss routing.
  • The paper applies the speculative decoding idea for efficiency too, but there is no empirical efficiency analysis showing how much it actually gains. The theoretical discussion of acceptance rates is not enough. The authors should report efficiency metrics like rollout throughput, teacher forward passes, and end-to-end training time against the direct-sampling prototype and vanilla OPD.
  • The number of independent training seeds is unclear. The authors should report mean and standard deviation across training seeds, rather than only Avg@32 evaluation results.
  • Improvement over Relay-OPD also looks marginal: 0.97 and 1.51 average accuracy points in Table 1. Without results across training seeds, it is hard to tell how reliable these gains are.
  • Also, Figure 4 makes the OPD setup look quite bad, as performance degrades with longer training. The authors should clarify whether the OPD baseline was sufficiently tuned and how checkpoints were selected. Otherwise, it is difficult to separate the benefit of IPD from instability in the baseline setup.
  • Overall, the presentation could be better. The figures are not good in terms of color and readability. For example, Figure 6(a) is hard to interpret because the curves and line styles are difficult to distinguish.

References:

[1] Gu et al. “MiniLLM: Knowledge Distillation of Large Language Models.” ICLR 2024.

[2] Jin et al. “Entropy-Aware On-Policy Distillation of Language Models.” ICML 2026.

Questions for authors

See the weaknesses above for the details:

  • How does IPD differ from MiniLLM’s teacher-mixed sampling?
  • How does IPD compare with EOPD and other PPO-based baselines?
  • What are the actual end-to-end efficiency gains?
  • Are improvements consistent across multiple independent training seeds?
  • What supports the stability claims for verification-aware supervision?
6 comments
AuthorsOpenReview ↗

Author Response (1/2)

We sincerely thank the reviewer for the careful and thoughtful comments and suggestions on our work, as well as the time and effort devoted to the rebuttal process. We greatly appreciate your constructive feedback and the opportunity to address these concerns, correct our mistakes, and clarify our contributions and findings.

  • Q1: How does IPD differ from MiniLLM's teacher-mixed sampling? The related work is weak.

    We thank the reviewer for pointing out this. First of all, we have expanded Section 2.3 of the revised manuscript to include discussions of off-policy and on-policy distillation, providing a more comprehensive review of related work and clarifying IPD's relationship to prior methods.

    We thank the reviewer for highlighting the connection with MiniLLM. We acknowledge that MiniLLM's teacher-mixed sampling is mathematically equivalent to our Eq. (1). However, IPD differs in its research focus, algorithmic contributions, and empirical findings.

    1. A broader research perspective. MiniLLM uses teacher-mixed sampling primarily as an auxiliary technique to mitigate reward hacking only in instruction-following tasks. In contrast, IPD systematically investigates the on/off-policy rollout continuum, revealing that neither endpoint is necessarily optimal. We unify the two distillation paradigms through the trade-off between trajectory quality and learnability perspective, motivating intermediate rollout policies that better balance these properties, particularly for challenging long-horizon reasoning tasks.

    2. Novelty beyond interpolation. MiniLLM's direct teacher-mixed sampling incurs substantial sequential teacher-decoding overhead, making it inefficient for long-horizon reasoning tasks. IPD addresses this with exact speculative sampling, preserving the interpolated distribution while accelerating rollouts. Moreover, we leverage verification outcomes for differentiated supervision: OPD for accepted proposals and CE for residual corrections, jointly improving sampling efficiency and supervision flexibility.

    In summary, while the interpolation formula itself is not new, IPD goes beyond MiniLLM by systematically investigating the on/off-policy continuum and introducing an efficient, verification-aware distillation framework, with demonstrated effectiveness in challenging long-horizon tasks. We will clarify these distinctions and properly acknowledge MiniLLM in the revised manuscript.

  • Q2: How does IPD compare with EOPD and other PPO-based baselines?

    We thank the reviewer for this suggestion. We have added a discussion of EOPD to the revised manuscript. Below, we compare IPD, EOPD, and PowerOPD[1] (α=0.1\alpha=0.1) on GSM8K and MATH-500 under matched experimental settings:

    BenchmarkIPD (Ours)EOPDPowerOPD (α=0.1\alpha=0.1)
    GSM8K70.0266.2768.40
    MATH-50044.6837.4843.82
    Avg.57.3551.8856.11
  • Q3: What are the actual end-to-end efficiency gains?

    We sincerely thank the reviewer for raising this important question. The computational cost comparison is presented below. We conducted a controlled efficiency comparison on 8 GPUs (4 for rollout and 4 for teacher scoring), using Qwen3-0.6B as the student and Qwen3-4B as the teacher. We used a batch size of 8, a response length of 512, and 4,096 generated tokens per step. For IPD, we set γ=0.1\gamma=0.1 and the draft-window length to k=5k=5.

    MethodStep Time (s)Rollout Time (s)E2E Throughput (tok/s)Teacher Verifier Calls
    Vanilla OPD7.684.67533.10
    Direct Sampling40.3537.17101.54,108
    IPD12.9310.10348.4860

    Times and throughput are measured at the second training step. Teacher verifier calls are accumulated over two steps and exclude the common post-rollout teacher scoring.

    Compared with direct sampling, IPD achieves a 3.68× rollout speedup and a 3.12× per-step speedup, while reducing sequential teacher verification calls by 4.78×. The proposal acceptance rate reaches 98.38%, explaining the effectiveness of block-wise verification. Compared with vanilla OPD, IPD incurs approximately 1.68× per-step wall-clock overhead, reflecting the additional cost of teacher-guided rollout generation.

AuthorsOpenReview ↗

Author Response (2/2)

  • Q4: Are improvements consistent across multiple independent training seeds?

    We report the average accuracy on GSM8K and AMC23 results of IPD, Relay-OPD, and SFT-then-OPD using Qwen3-0.6B-Base as the student and Qwen3-4B as the teacher, across three independent training seeds under matched experimental settings. We additionally include Relay-OPD with its officially recommended configuration (unmatched).

    SeedIPD (Ours)Relay-OPD (Unmatched)Relay-OPD (Matched)SFT-then-OPD
    Seed 147.1645.1336.7734.82
    Seed 246.2345.8939.2535.18
    Seed 347.3946.3340.5937.23
    Mean ± Std46.93 ± 0.6145.78 ± 0.6138.87 ± 1.9435.74 ± 1.30

    Notably, Relay-OPD performs poorly under our matched training configuration, and achieves competitive performance only when using its officially recommended settings. As analyzed in Appendix A.4, this is primarily due to the weak student initialization: degenerate student generations fail to reliably trigger Relay-OPD's teacher handoff mechanism, resulting in insufficient teacher intervention. We therefore report both matched and unmatched results to ensure a transparent and comprehensive comparison.

  • Q5: What supports the stability claims for verification-aware supervision?

    We thank the reviewer for raising these concerns and would like to clarify two points.

    1. PPO importance ratio. We considered correcting the PPO ratio for mixed-policy sampling during experiment. However, our experiments showed negligible differences in downstream performance. We therefore retained the original PPO ratio to maintain consistency with vanilla OPD and ensure a controlled comparison. Results on Qwen3-0.6B-Base distilled from Qwen3-4B are shown below:

    BenchmarkOriginal PPO RatioMixed-Policy-Corrected Ratio
    GSM8K70.0269.88
    MATH-50044.6845.01
    Avg.57.3557.45

    We acknowledge that the original ratio does not fully correct the sampling mismatch; our claim is limited to its negligible empirical impact in these settings. We will discuss about it in the revised manuscripts.

    2. Accepted-token log-ratios. We agree that acceptance does not guarantee a small log-ratio for every token. However, Appendix A.2 (Eq. 21) proves that the expected absolute k1 signal on the accepted branch is no greater than that of vanilla OPD.

    Furthermore, let qq and pp denote the student and teacher distributions, respectively. For any τ>0\tau>0, student sampling satisfies

    \leq e^{-\tau}.$$ Thus, extremely large teacher-favored log-ratios are exponentially unlikely under student sampling, despite being always accepted. Meanwhile, speculative rejection suppresses student-favored tokens with large positive $\log(q/p)$ signals. We further report the empirical frequency of extreme log-ratios: | Metric | Vanilla OPD | IPD (Accepted) | |---|---|---| | max{k_1} | 25.85 | 11.27 | | \|k_1\|(> 10ratio %) | 11.7% | 0.56% | Together, these theoretical and empirical results support the stability motivation of acceptance-based supervision.
  • Q6: Bad performance of OPD baseline in Figure 4.

    We thank the reviewer for raising this concern. As specified in Appendix B (Table 7), vanilla OPD and IPD use identical training configurations, including learning rate, batch size, and training budget. Importantly, all reported results are based on the best-performing checkpoints, rather than the final checkpoints, so the degradation of OPD during later training does not artificially disadvantage the baseline.

    We adopted a longer training schedule in the multimodal setting to examine training dynamics under sustained optimization. However, IPD consistently outperforms OPD even at earlier checkpoints with a training budget comparable to our text-only experiments, as shown in Figure 4. Thus, the advantage of IPD is not merely a consequence of OPD's later-stage degradation.

    We will clarify the checkpoint selection protocol and the motivation for the longer training schedule in the revised manuscript.

  • Q7: presentation could be better.

    We thank the reviewer for this helpful suggestion. We have improved the color scheme, line styles, and legends to make the curves clearer and more distinguishable in the revised manuscript. We sincerely apologize for the lack of clarity.

[1] Zhao et al. "PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation." EMNLP 2026.

We sincerely thank the reviewer again for the thoughtful feedback and valuable suggestions on our IPD work. We hope our responses have fully addressed your concerns. We apologize again for the errors in the original manuscript and would be grateful if you could reconsider your rating in light of these responses.

AuthorsOpenReview ↗

Author Response

We sincerely thank you again for your time and constructive feedback during the rebuttal and we hope our responses and additional experiments have adequately addressed your concerns. We would greatly appreciate any further feedback and would be glad to clarify any remaining concerns. Looking forward to hearing your reply and would be grateful if you could reconsider your rating in light of these responses

Reviewer_2Q5DOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes Interpolated Policy Distillation (IPD). At each step, the student's rollout tokens come from a mix of the student and teacher distributions, weighted by γ. Sampling is made efficient with a speculative-decoding rule that keeps this mixed distribution exact. Accepted student tokens are trained with the OPD objective, and teacher corrections are trained with cross-entropy. The method beats OPD and segment-interleaving baselines on text and multimodal reasoning, with the largest gains for weak or pruned students.

Critical strengths

Defining the rollout policy at the distribution level and then sampling it exactly is more principled than hand-designed switching rules. The prototype experiment helps separate the effect of the rollout policy from the effect of the supervision. The gains for weak and pruned students are large.

Critical weaknesses

The paper's main motivation is a trade-off between trajectory quality and student learnability, but neither side is measured convincingly. The main text defines quality as correct and valid reasoning. However, the LLM-judge rubric in Appendix A.3 tells the judge to ignore correctness and score only fluency and coherence, so the quality curve doesn't measure what the paper claims.

The learnability proxy has its own inconsistency. The text says it averages the absolute log-ratios, but the formula takes the absolute value after averaging, so positive and negative terms can cancel. The proxy is also close to circular. It measures how closely teacher and student agree on the sampled tokens, so its decline as teacher influence grows is largely built into how it is defined.

Questions for authors
  1. Can you measure trajectory quality by correctness, for example the final-answer accuracy of the rollouts, and fix the inconsistency in the learnability metric?

  2. What are the actual speedups, in wall-clock time and number of teacher forward passes, compared with direct sampling and with OPD?

  3. Can you report results over several seeds, and compare against Relay-OPD and SFT-then-OPD under matched settings?

  4. Was the default interpolation coefficient of 0.1 tuned on the evaluation benchmarks?

  5. In the pruned multimodal setting, how does IPD compare with an SFT or SFT-then-OPD baseline?

5 comments
AuthorsOpenReview ↗

Author Response(1/2)

We sincerely thank the reviewer for carefully identifying the mistakes in our manuscript. We greatly appreciate your constructive feedback and the opportunity to address these issues, correct our mistakes, and clarify our contributions and findings.

  • Q1: Can you measure trajectory quality by correctness, for example the final-answer accuracy of the rollouts, and fix the inconsistency in the leaenability metric?

    We sincerely thank the reviewer for identifying this important issue. We acknowledge that the manuscript contains two significant mistakes in the descriptions of trajectory quality and learnability. We apologize for these mistakes and would like to clarify them individually.

  1. Trajectory Quality. The LLM judge prompt presented in Appendix A.3 is indeed the one used in our experiments. The statement in the Introduction, "whether it constitutes a correct and valid reasoning process," was therefore inaccurate, and we will correct throughout the manuscript. We intentionally did not require the generated trajectories to produce correct final answers when evaluating trajectory quality. Our intention was to measure the readability, coherence, and structural well-formedness of the generated reasoning trajectories. This choice is motivated by the characteristics of practical OPD training. In our setting, most training rollouts are truncated because the reasoning problems typically require substantially more tokens to solve than the maximum rollout length permits. Using final-answer correctness as the primary proxy for trajectory quality would therefore be inappropriate for these truncated trajectories. Nevertheless, we still conduct an additional experiment with a sufficiently large rollout length, using final-answer correctness as an alternative quality metric: | γ\gamma | 0 | 0.05 | 0.1 | 0.2 | 0.5 | 1.0 | |:---|---:|---:|---:|---:|---:|---:| | Correctness (%) ↑\uparrow | 8.1 | 23.2 | 33.6 | 36.6 | 59.5 | 79.8 | The results show that trajectory correctness also increases with stronger teacher intervention, supporting the quality trend observed in Figure 2(b).
  2. Learnability. We would also like to clarify an error in the mathematical definition of our learnability proxy. The metric actually used in our experiments computes the mean absolute sampled-token log-ratio, rather than taking the absolute value after averaging:
-\frac{1}{T}\sum_{t=1}^{T} \left| \log\frac{\pi_\theta(y_t\mid s_t)} {\pi_T(y_t\mid s_t)} \right| \right).$$ The manuscript mistakenly places the absolute value outside the summation. This is a transcription error, which we will correct. Regarding the circularity concern, we agree that the observed trend is closely related to the metric's definition. However, **our intention is not to present this trend as a novel empirical finding**, but to characterize student-policy compatibility from a trust-region perspective. Specifically, Eq. (2) establishes that $$D_{\mathrm{TV}}(m_\gamma,\pi_\theta) =\gamma D_{\mathrm{TV}}(\pi_T,\pi_\theta),$$ showing that $\gamma$ explicitly controls the rollout policy's proximity to the student at each prefix. While this does not guarantee monotonicity of our sampled-token proxy, it aligns with the trust-region theory, which suggests that excessive teacher–student discrepancies can impair optimization. Existing approaches address this by constraining supervision targets (e.g., TOP-D[1]), filtering extreme-disagreement tokens, or sampling within a student-centered trust region(e.g. TRB[2]). Thus, **our proxy measures student–teacher compatibility rather than empirical learning gains**. We will clarify this distinction in Figure 2(b), while the practical benefits of balancing teacher guidance and policy proximity are supported by our downstream results and $\gamma$ ablation. - **Q2: What are the actual speedups, in wall-clock time and number of teacher forward passes, compared with direct sampling and with OPD?** We conducted a controlled efficiency comparison on 8 GPUs (4 for rollout and 4 for teacher scoring), using Qwen3-0.6B as the student and Qwen3-4B as the teacher. We used a batch size of 8, a response length of 512, and 4,096 generated tokens per step. For IPD, we set $\gamma=0.1$ and the draft-window length to $k=5$. | Method | Step Time (s) | Rollout Time (s) | E2E Throughput (tok/s) | Teacher Verifier Calls | |---|---:|---:|---:|---:| | Vanilla OPD | **7.68** | **4.67** | **533.1** | 0 | | Direct Sampling | 40.35 | 37.17 | 101.5 | 4,108 | | IPD | 12.93 | 10.10 | 348.4 | **860** | *Times and throughput are measured at the second training step. Teacher verifier calls are accumulated over two steps and exclude the common post-rollout teacher scoring.*
AuthorsOpenReview ↗

Author Response(2/2)

(...continued from the previous comment) Compared with direct sampling, IPD achieves a 3.68× rollout speedup and a 3.12× per-step speedup, while reducing sequential teacher verification calls by 4.78×. The proposal acceptance rate reaches 98.38%, explaining the effectiveness of block-wise verification. Compared with vanilla OPD, IPD incurs approximately 1.68× per-step wall-clock overhead, reflecting the additional cost of teacher-guided rollout generation.

  • Q3: Can you report results over several seeds, and compare against Relay-OPD and SFT-then-OPD under matched settings?

We report the average accuracy on GSM8K and AMC23 results of IPD, Relay-OPD, and SFT-then-OPD using Qwen3-0.6B-Base as the student and Qwen3-4B as the teacher, across three independent training seeds under matched experimental settings. We additionally include Relay-OPD with its officially recommended configuration (unmatched).

SeedIPD (Ours)Relay-OPD (Unmatched)Relay-OPD (Matched)SFT-then-OPD
Seed 147.1645.1336.7734.82
Seed 246.2345.8939.2535.18
Seed 347.3946.3340.5937.23
Mean ± Std46.93 ± 0.6145.78 ± 0.6138.87 ± 1.9435.74 ± 1.30

Notably, Relay-OPD performs poorly under our matched training configuration, and achieves competitive performance only when using its officially recommended settings. As analyzed in Appendix A.4, this is primarily due to the weak student initialization: degenerate student generations fail to reliably trigger Relay-OPD's teacher handoff mechanism, resulting in insufficient teacher intervention. We therefore report both matched and unmatched results to ensure a transparent and comprehensive comparison.

  • Q4: Was the default interpolation coefficient of 0.1 tuned on the evaluation benchmarks?

Thank you for raising this important question. We initially selected γ=0.1\gamma=0.1 based on preliminary experiments on MATH-500 using the lightweight Qwen3-0.6B student. We then fixed γ=0.1\gamma=0.1 for all subsequent experiments without further tuning.

  • Q5: In the pruned multimodal setting, how does IPD compare with an SFT or SFT-then-OPD baseline?

We report below a comparison of IPD, SFT, and SFT-then-OPD in the pruned multimodal setting, using Qwen3-VL-2B as the student and Qwen3-VL-8B as the teacher. The reported scores are the average accuracy across MathVision and MathVista. For SFT-then-OPD, we first performed SFT on 20K teacher-generated trajectories generated by Qwen3-VL-8B from randomly sampled prompts in Innovator-VL-RL-172K.

MetricIPD (Ours)SFTSFT-then-OPD
Accuracy (%)37.1828.9136.29

We sincerely appreciate the reviewer's time and effort in evaluating our work and providing constructive feedback. We will include further experimental details and clarifications in the revised manuscript. We hope our responses have adequately addressed the reviewer's concerns and clarified our findings. We apologize again for the errors in the original manuscript and would be grateful if the reviewer could reconsider their rating in light of these responses.

[1] Zhengpeng Xie, Li Lyna Zhang, Zeke Xie, and Mao Yang. “Trust Region Policy Distillation.” Advances in Neural Information Processing Systems (NeurIPS), 2026.

[2] Daniil Plyusov et.al Trust-Region Behavior Blending for On-Policy Distillation (https://arxiv.org/abs/2605.31159)

My major concerns have been addressed; I decided to raise my score to weak accept.

Comments on the paper

2 comments
AuthorsOpenReview ↗

To all reviewers

We sincerely thank all reviewers for their time and effort in carefully evaluating our work. We greatly appreciate their feedback and constructive suggestions, which have been invaluable in helping us improve our work.

  • Quality–Learnability Metrics.

    We apologize for the inaccuracies in the metric descriptions, which have now been corrected in the revised manuscript. Specifically, trajectory quality is intended to assess properties such as coherence and well-formedness, rather than relying solely on final-answer correctness, since many rollouts are truncated before reaching a final answer. Additionally, the learnability metric should compute the absolute value before averaging, rather than taking the absolute value after averaging. This was another typographical error that has also been corrected.

    We once again sincerely apologize for any confusion and inconvenience these two errors may have caused.

  • Computational Efficiency.

    We thank all reviewers for raising this important question of practical computational costs, which we agree is a crucial aspect of empirical validation. In our responses, we provide detailed comparisons of wall-clock time and teacher verification calls. Compared with direct sampling, IPD achieves a 3.12× per-step wall-clock speedup, a 3.68× rollout speedup, and a 4.78× reduction in teacher verification calls. These results demonstrate that IPD achieves strong performance without incurring prohibitive computational overhead.

In our responses, we also provide detailed multi-seed results and comparisons with additional baselines, where IPD consistently achieves state-of-the-art performance. We have further expanded the discussion of related work in the revised manuscript, clarifying both our connections to prior approaches and the novelty of IPD.

Overall, IPD aims to establish a unified framework that bridges on-policy and off-policy distillation, enabling more challenging distillation tasks. Our experimental results demonstrate the effectiveness of this framework. In particular, we have added evaluations on long-horizon interactive agent tasks in Appendix A.8 of the revised manuscript, with preliminary results further demonstrating the effectiveness of IPD in such settings.

All changes in the revised manuscript are highlighted in blue for clarity. Once again, we sincerely thank all reviewers for their time, effort, and valuable contributions to improving our work. We hope that our revisions and additional evidence adequately address their concerns, and we respectfully ask the reviewers to reconsider their assessments and consider raising their scores.

AuthorsOpenReview ↗

A summary of discussion for meta-review

Dear ACs and all reviewers:

We sincerely thank you and all three reviewers for the careful evaluation of our paper and the constructive discussions throughout the rebuttal period.

We have taken every concern and weakness raised by the reviewers seriously, providing detailed responses, conducting substantial additional experiments, and revising the manuscript accordingly. We would like to summarize the major improvements and key findings to facilitate the meta-review process.

Summary of Major Concerns (Have been addressed)

  1. Trajectory Quality, Learnability metric. (Reviewer 2Q5D, vFfX)

We sincerely apologize for the confusion caused by a typographical error in the learnability formulation in our original manuscript, which we promptly corrected during the rebuttal. We also clarified that our trajectory-quality proxy measures coherence and well-formedness rather than final-answer correctness, since most training rollouts are truncated before reaching a final answer. Therefore, relying on final-answer accuracy would not faithfully reflect the trajectory quality encountered during actual training.

  1. Computational Efficiency and Performance–Cost Trade-off. (Reviewer 2Q5D, F2Hr, vFfX)

In response to the reviewers' concerns about computational overhead, we conducted additional wall-clock profiling and teacher-intervention analysis. Our results show that IPD achieves better performance than Relay-OPD with comparable or slightly lower per-step cost (12.93 s vs. 13.22 s), while requiring fewer teacher-intervened tokens (1.62% vs. 2.76%). Furthermore, by annealing the interpolation coefficient toward zero, IPD reduces its training overhead to only 15% above vanilla OPD, while maintaining nearly identical downstream accuracy (69.96% vs. 70.02%). These findings demonstrate that IPD offers a favorable performance–cost trade-off.

  1. Training Stability, Multiple Seeds. (Reviewer 2Q5D, F2Hr)

In response to the reviewers' concerns about training stability and reproducibility, we conducted additional three-seed experiments across multiple baselines. IPD consistently outperforms Relay-OPD (Official) across all three seeds, achieving 46.93 ± 0.61% average accuracy compared with 45.78 ± 0.61% for Relay-OPD, 38.87 ± 1.94% for Relay-OPD (Matched), and 35.74 ± 1.30% for SFT-then-OPD. These results provide additional evidence of IPD's consistent performance across random seeds.

Summary of Feedback from Reviewers

  1. Reviewer 2Q5D (3 score): Explicitly acknowledged that major concerns had been addressed and increased rating from 1 to 3 during the rebuttal period.

  2. Reviewer vFfX (3 score): Explicitly acknowledged that our responses had addressed the concerns, while maintaining the original rating. We sincerely appreciate this positive assessment of our clarifications and additional evidence.

  3. Reviewer F2Hr (2 score): Acknowledged that a portion of the original concerns had been addressed, but raised two remaining questions regarding methodological novelty and the performance–cost trade-off against Relay-OPD. We subsequently provided a detailed clarification of the methodological contributions and additional quantitative comparisons demonstrating improved performance at comparable computational cost. Unfortunately, we have not received a further response confirming whether these final clarifications resolved the reviewer's remaining concerns. We respectfully hope that the AC will take these additional responses and supporting evidence into consideration.

Concluding Remarks

In summary, the rebuttal process has substantially strengthened our IPD through corrected and clarified theoretical formulations, more reliable evaluation metrics, stronger baseline comparisons, multi-seed experiments, comprehensive computational profiling, and additional agent-task validation. The corresponding revisions have been incorporated into the updated manuscript, with changes highlighted blue for ease of review.

We sincerely appreciate the time and effort devoted by the AC and all reviewers, and respectfully hope that these substantial improvements and the positive rebuttal feedback will be fully considered in the final assessment of our work.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.