acceptodds
Under review as a conference paper at ICLR 2027

Interactive-Policy Distillation with Bidirectional Propose-and-Verify

12 views

Abstract

On-policy distillation (OPD) trains a compact student model on its self-generated trajectories with dense token-level teacher feedback. However, naive OPD may suffer from teacher unanchoring, where the student's reasoning trajectory drifts outside the teacher's acceptance region, causing the teacher to be queried on states it would hardly visit and thus provide unreliable supervision. We propose **Interactive-Policy Distillation (IPD)**, which dynamically corrects training trajectories within adaptive teacher intervention. Under a bidirectional propose-and-verify state machine, the student and teacher alternately exchange their roles as proposer and verifier, and collaboratively generate mixed-source trajectories. Then different supervisions are applied according to the source of each token. This bidirectional propose-and-verify mechanism and the source-split loss make IPD not only a more performant distillation method, but also a unified bridge between on-policy and off-policy paradigms. To make the interleaved dual-model rollouts more efficient, we also design a dedicated fused inference engine that co-hosts both models in one serving instance with separate KV caches and instantiates the state machine model to distribute, collect, and process requests. On math reasoning tasks and across multiple teacher-student model pairs, student models trained with IPD not only outperform those trained with OPD, but also demonstrate higher data efficiency. Specifically, when distilling Qwen3-30B-A3B (non-thinking) into Qwen3-1.7B-Base, IPD brings a mean@8 and a best@8 benchmark-averaged accuracy improvement compared with OPD. Besides, IPD only consumes about of the training examples and steps to outperform OPD trained on the whole training dataset for one epoch. We also investigate the impact of different loss variants and takeover / handback configurations, and demonstrate the robustness of IPD on different training data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Post-Training, Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8RsBOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the "teacher unanchoring" problem in on-policy distillation (OPD) for LLMs, where a student policy drifts outside the teacher’s acceptance region, causing the teacher to evaluate states it rarely visits and deliver noisy or uninformative feedback. The authors propose Interactive-Policy Distillation (IPD), an interactive rollout and distillation framework

Critical strengths

A Principled Bridge Between On-Policy and Off-Policy Distillation:Unlike prior work that relies on fixed teacher prefixes (TrOPD) or single-token teacher corrections (SKD), IPD introduces a dynamic bidirectional state machine governed by top-KK acceptance regions. Pairing this interactive trajectory generation with a source-split loss (reverse KL for on-policy tokens, forward KL for teacher-generated demonstrations) provides a mathematically natural and empirically superior formulation.

Essential Systems Contribution (Fused Inference Engine):Interleaved multi-model rollouts typically suffer from prohibitive communication latency and continuous KV cache invalidation in distributed training frameworks. By developing a fused engine co-hosting both models with synchronized cache alignment, parallel chunk verification, and role-grouped scheduling, the authors eliminate the main deployment barrier for interactive distillation, achieving up to a 71.4$\times$ throughput improvement.

Critical weaknesses

End-to-End Wall-Clock Overhead Relative to Standard OPD:

While Figure 3 convincingly shows that the fused engine is 21$\times$–71$\times$ faster than a naive two-engine implementation, the paper does not report the end-to-end wall-clock training time of IPD versus standard OPD. Standard OPD only executes student autoregressive decoding followed by a single teacher prefill. In IPD, teacher takeovers require additional teacher autoregressive decoding and student prefill passes. Demonstrating that IPD's data efficiency (Figure 5) translates into net wall-clock training gains over OPD would significantly strengthen the practical claim.

Questions for authors

NA

1 comment
AuthorsOpenReview ↗

Responses to Reviewer 8RsB

First, we would like to thank the reviewer for recognizing our contributions on a principled bridge between on-policy and off-policy distillation, which differs from prior work and provides a mathematically natural and empirically superior formulation, and essential inference infrastructures for the non-trivial Interleaved multi-model rollouts, as well as your valuable feedback during the reviewing stage. Regarding your concerns, we provide our responses as follows.

  • W1: Wall-clock Time Analysis

    For 4B to 1.7B-Base group, we calculate the average wall-clock time per step during OPD/SKD/IPD training:

    • OPD: 124.72s
    • IPD: 158.00s

    Given that IPD needs frequent switches between the large model and the small model, its wall-clock time is naturally longer than OPD. Meanwhile, this per-step overhead can be hedged by the higher data efficiency and faster convergence of IPD. We also updated our manuscript with a new subsection (Appendix C.3, marked in blue), where we illustrate the time-performance curve of OPD and IPD in Figure 10. Specifically, under the same time budget, IPD yields a better student. For the same-level student performance, IPD needs less total training time.

Again, we thank the reviewer's effort and valuable feedback, which helps us further improve the quality of our paper. We hope our responses can address your concerns.

Reviewer_WR2TOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary

This paper addresses the mismatch between teacher- and student-generated prefix distributions through Interactive-Policy Distillation (IPD). It combines a bidirectional propose-and-verify state machine with source-dependent supervision, allowing teacher intervention followed by a return to student generation. The paper also introduces a fused inference engine to support these interleaved rollouts. Its contributions includes a trajectory-generation strategy with systems support for implementation. Experiments demonstrate improvements in mathematical reasoning and sample efficiency across several teacher–student configurations.

Critical strengths
  • Addresses an important practical challenge in distilling large models into smaller students. The interactive generation strategy is intuitive and well motivated.

  • Evaluates multiple mathematical reasoning benchmarks and teacher–student model pairs.

Critical weaknesses
  • Novelty over SKD: The conceptual advance of this work over SKD seems incremental.

  • Limited baselines & related work: Limited baselines and related work: besides OPD and SKD, there are many related work that deserves experimental comparison. For instance, methods that considers bidirectional distillation (https://arxiv.org/pdf/2505.16297), or mix onpolicy with offpolicy data (GKD). DistiLLM-2 is also relevant given its use of different objectives for teacher- and student-generated responses. Regarding related work, using weight correcting schemes on teacher provided data samples such as DFT https://arxiv.org/abs/2508.05629 or PEAR https://arxiv.org/abs/2602.01058 also deserves discussion.

  • Sample efficiency versus implementation complexity: IPD relies on a fused inference engine to efficiently support frequent teacher–student switching and coordinated KV-cache management. Although this engineering contribution enables the proposed method, it also increases implementation complexity and limits adoption within standard training and inference frameworks.

  • Marginal improvements in some configurations: The improvements over SKD are particularly small for the 4B students.

Questions for authors
  • What conclusion does Figure 2 indicate? The paper is partially motivated by a phenomenon denoted as 'teacher unachoring' when the teacher feedback is conditioned on student provided prefixes - but in my opinion, Figure 2 does not clear show what are potential limitations of it. The figure also does not show why interleaved student–teacher generation should beat teacher-only or student-only rollouts.
  • It would be interesting to see if the shift between student and teacher generation has some informative patterns on math or related reasoning tasks, such as, how often do the shift happen and where, whether it is sensitive to choice of K, or size gap between the teacher-student model pairs. This may provide more empirical insights of your interactive sampling strategy.
5 comments
AuthorsOpenReview ↗

Responses to Reviewer WR2T (1/3)

First, we would like to thank the reviewer for recognizing our contribution in addressing an important practical challenge in LLM distillation and for finding our algorithm intuitive and well motivated, as well as for your valuable feedback during the reviewing stage. Regarding your concerns, we provide our responses as follows.

  • W1: Clarify Novelty

    SKD is indeed the prior work closest to ours, and we also discussed it as an important related work in our paper. IPD shares the idea of interleaving dual models during sampling, but still has important differences, which we would like to summarize as follows.

    • Bidirectional propose-and-verify. In SKD, the student always proposes and the teacher always verifies. The two models play asymmetric roles, and a rejected token is simply replaced by one teacher token before the student resumes. However, correcting an individual token does not necessarily resolve the underlying reasoning drift, as the student may immediately deviate again after correction. IPD addresses this limitation through a bidirectional propose-and-verify mechanism, where the teacher can adaptively take over generation when the student goes off track and retain control until the student’s handback criterion is met. Specifically, the two models dynamically switch their roles as proposer and verifier, enabling not only adaptive takeover but also adaptive handback based on the student’s acceptance of teacher-generated tokens. This shifts teacher intervention from isolated token-level correction to adaptive control over the reasoning trajectory.
    • Source-split KL. SKD trains every token of the mixed-source sequence with one forward KL. IPD applies a reverse KL to the student-generated tokens and a forward KL to the teacher-generated ones, so that each divergence is estimated on samples from the distribution it takes its expectation over, and the on-policy and off-policy distillations are smoothly bridged with a mathematically natural formulation.
    • A dedicated inference engine. As Reviewer 8RsB mentioned, interleaved multi-model rollouts typically suffer from prohibitive communication latency and continuous KV cache invalidation in distributed training frameworks. We built a fused engine to not only better serve our own proposed algorithm, but also other algorithms like SKD that require interleaved multi-model rollouts.
AuthorsOpenReview ↗

Responses to Reviewer WR2T (2/3)

  • W2: Additional Baselines

    We run additional baselines, including EOPD, TrOPD, and DistiLLM-2, with Qwen3-4B (non-thinking) as the teacher and Qwen3-1.7B-Base as the student. We also update our manuscript and include the discussion of ToDi (https://arxiv.org/pdf/2505.16297), DFT (https://arxiv.org/abs/2508.05629), and PEAR (https://arxiv.org/abs/2602.01058) in Appendix B.4 (marked in blue).

    All the additional baselines share the standard experimental configurations (e.g., training data, learning rate, batch size, response length, sampling hyperparameters) unless specified. Overall, the results are shown in the following table.

    MethodMATH-500AIME24AIME25AMC23MinervaOlympiadBenchAvg.
    OPD66.68 / 84.067.92 / 21.772.92 / 8.7035.69 / 57.2926.42 / 44.0431.00 / 48.4528.44 / 44.05
    SKD66.70 / 84.508.75 / 17.414.58 / 10.7335.09 / 52.9027.94 / 43.6929.75 / 49.5228.80 / 43.13
    EOPD64.60 / 83.196.25 / 16.345.83 / 14.9230.87 / 54.5926.52 / 43.0129.79 / 48.7027.31 / 43.46
    TrOPD66.20 / 83.877.08 / 19.113.75 / 13.7131.93 / 55.6626.84 / 44.4429.48 / 48.1127.55 / 44.15
    DistiLLM-263.88 / 82.467.92 / 20.403.75 / 13.6129.67 / 48.7025.55 / 42.4127.80 / 45.7526.43 / 42.22
    IPD69.28 / 84.599.17 / 21.696.67 / 15.0636.14 / 57.2529.60 / 46.2631.80 / 49.6330.44 / 45.75

    Specifically:

    • EOPD: we follow their paper and implement EOPD in our codebase on top of the OPD arm, with an additional forward KL term calculated on the teacher's renormalized top-16 and applied where the full-vocabulary teacher entropy exceeds τ = 0.8, with a weight α = 1.0.
    • TrOPD: we use their official repository (github.com/Xingrun-Xing2/TrOPD) to run on our data and models. For each prompt, one response generated by the teacher is used as off-policy data and is trained together with the on-policy samples in a single update. We keep the official hyperparameters: an off-policy weight of β = 0.001, a forward KL on the teacher's top-64 tokens for rejected tokens, and a cosine schedule for α, whose length is set to our 279 training steps.
    • DistiLLM-2: We implement the distillm_v2 loss in our codebase, following their official repository (github.com/jongwooko/distillm-2). For each prompt, the loss combines a skew KL on one teacher-generated response and a skew reverse KL on one student-generated response, with the official defaults of α = 0.1 and equal weights for the two terms. Following Algorithm 1 of their paper, the responses are collected before each training epoch, and since all methods here are trained for one epoch, they are collected once, from the teacher and from the initial student. The divergences are computed on the union of the student's and the teacher's top-16 tokens.

    As observed, IPD still achieves the best average score and leads most benchmarks individually.

  • Q1: Further Explanation of Figure 2

    In Figure 2, our motivation is to demonstrate that the teacher can provide unreliable learning signals in vanilla OPD training, which uses reverse KL as the objective. Specifically, on incorrect student reasoning traces, the reverse KL divergence shows no significant difference from that on correct student reasoning traces. We use entropy as an indicator, and as shown in Figure 2, prefixes from incorrect traces will confuse the teacher and result in a flat and uncertain teacher distribution with high entropy, and as a result induce low reverse KL that is not a reliable signal.

AuthorsOpenReview ↗

Responses to Reviewer WR2T (3/3)

  • Q2: Student-teacher shift analysis

    Thank you for this suggestion. We agree that such an analysis will be interesting and give more insight into the interactive sampling. The main observations are as follows.

    • How often. In the Qwen3-4B to Qwen3-1.7B-Base setting with K = 16, the teacher writes 4.9% of the tokens at the first training step. This share falls below 1% within about 20 steps and stays at about 0.6% for the rest of training.

    • Where. We did not keep intermediate checkpoints, so we ran the IPD rollout on 512 training prompts (4 samples each) with the student before training (step 0) and with the final checkpoint (step 279), and recorded where the shifts happened. Specifically, both takeovers and handbacks are triggered mostly by natural-language words and rarely by numbers or symbols, which in some sense indicates that the shifts happen mainly where the reasoning is being worded, rather than inside a calculation.

      TokensWordsNumbersSymbolsOther
      Step 0, all tokens26.9%20.1%35.6%17.4%
      Step 0, takeover trigger70.9%1.5%11.8%15.8%
      Step 0, handback trigger65.3%2.4%15.0%17.3%
      Step 279, all tokens18.4%28.3%20.2%33.2%
      Step 279, takeover trigger86.6%2.4%5.9%5.1%
      Step 279, handback trigger69.7%1.4%14.6%14.3%
    • Sensitivity to K. A smaller K makes the takeover and handback rules stricter and the shifts more frequent. Averaged over training steps, the teacher writes 5.8%, 1.6%, 0.7%, and 0.4% of the tokens for K = 4, 8, 16, and 32, as shown in Figure 7. The final accuracy does not show a significant sensitivity to the choice of K.

    • Size gap. The same pattern holds for all four teacher–student pairs: in the beginning, the teacher has more interventions, writing 4.0% to 6.3% of the tokens at the first step, and then the intervention ratio decays as training proceeds. A larger gap leads to slightly more interventions. Specifically, for the 1.7B student, with the 30B teacher the initial intervention ratio is about 6.3%, while with the smaller 4B teacher it is about 4.9%.

Again, we thank the reviewer's effort and valuable feedback, which helps us further improve the quality of our paper. We hope our responses can address your concerns.

Reviewer_cyJdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The issue that this paper is trying to address is that in vanilla OPD for some student trajectories the teacher is forced to provide supervision even though they may be unlikely under its distribution. The key idea is to dynamically switch between the student and teacher in terms of who generates the next token, and the rules for switching are based on whether the token generated by one is in the top-K most likely tokens by the other (or not). An important asymmetry (which should be imho highlighted more in the paper) is that when a student-generated token is rejected, it is replaced by a teacher-generated token, while teacher-generated tokens are retained until we switch back to the student. Forward or reverse KL is used depending on who generated the token, with the mixed-source trajectory. In a way this interpolates between on- and off-policy distillation. For efficiency the authors also propose a fused inference engine that avoid hosting two models on separate vLLMs.

Critical strengths
  • The fused inference engine is appreciated even though it is relatively straightforward and did not require technical innovation. The reported speed-up looks impressive, but the comparison is with a naive two-engine implementation.
  • The data efficiency aspect of IPD is interesting and worth investigating further.
Critical weaknesses
  • The empirical evaluation provides interesting evidence in support of the proposed method, however, I would characterise this as preliminary. Several natural baselines come to mind, such as for example randomly choosing to train with reverse or forward KL for entire trajectories with some predefined proportions. Another natural baseline would be to randomly choose whether the teacher or student generates tokens, while matching the proportion of teacher-generated tokens in IPD. This would help us see if the improvements are from adaptive switching or simply from mixing student- and teacher-generated tokens.
  • Relatedly, justifying why a mixed-source prefix makes sense to begin is also a good idea since it could happen that the final mixed trajectory is unlikely under both student and teacher, potentially reintroducing a distribution mismatch similar to the one that motivates the method.
  • Moreover, the identified issue, which the authors termed "teacher unanchoring" has been observed in previous papers which propose alternative solutions. For example, EOPD (Jin et al., 2026), TrOPD (Xing et al., 2026), and DistiLLM-2 (Ko et al., 2025) should be also included as baselines, especially since they are already cited. Comparing with SKD is a good idea, since as the authors point out it can be seen as a special case, but it is far from enough in my opinion.
  • The ablations on the KL direction are useful, but they don't disentangle adaptive switching vs. mixed-source trajectories and source-dependent KL supervision. Interestingly, the pure reverse KL variant performs relatively close to source-split KL.
  • When distilling from Qwen3-4B to Qwen3-1.7B-Base the gain in performance could be because of improved reasoning and/or mathematical ability or it could be simply because the Base model got better at instruction and format following (which is not good at out of the box), or a mix of both. The authors do not disentangle these effects. Evaluating on instruction-following benchmarks or considering instruction-tuned students would help clarify this.
  • The evidence for "unanchoring" is indirect. Higher entropy does not necessarily imply that the teacher supervision is unreliable. It could for example reflect more difficult or incoherent contexts.
  • The experiments are not as comprehensive as they could be in terms of student-teacher configuration and scales of models. There are 4 configurations, but all from the same family, with relatively small base models and only on math. There are also no OOD generalisation experiments.
Questions for authors
  1. Can you include the additional baselines I mentioned in the weaknesses?
  2. Can you run some experiments to disentangle the gain in performance?
  3. Relatedly do the gains survive if you distill from a larger to a smaller non-Base model?
  4. Can you also report pass@1 results?
  5. Can you better justify mixed-source prefixes?
  6. Do you also see gains if you evaluate e.g. on coding tasks while training on math to test OOD?
4 comments
AuthorsOpenReview ↗

Responses to Reviewer cyJd (1/3)

First, we would like to thank the reviewer for valuable feedback during the reviewing stage. Regarding your concerns, we provide our responses as follows.

  • W1&W3&Q1: Additional Baselines

    We include additional baselines with Qwen3-4B (non-thinking) as the teacher and Qwen3-1.7B-Base as the student. All the additional baselines share the standard experimental configurations (e.g., training data, learning rate, batch size, response length, sampling hyperparameters) unless specified. Overall, the results are shown in the following table.

    MethodMATH-500AIME24AIME25AMC23MinervaOlympiadBenchAvg.
    OPD66.68 / 84.067.92 / 21.772.92 / 8.7035.69 / 57.2926.42 / 44.0431.00 / 48.4528.44 / 44.05
    OPD-RandKL (p=0.5)68.15 / 84.608.75 / 23.304.58 / 11.5433.89 / 56.0227.99 / 43.8330.94 / 48.7729.05 / 44.68
    SKD66.70 / 84.508.75 / 17.414.58 / 10.7335.09 / 52.9027.94 / 43.6929.75 / 49.5228.80 / 43.13
    EOPD64.60 / 83.196.25 / 16.345.83 / 14.9230.87 / 54.5926.52 / 43.0129.79 / 48.7027.31 / 43.46
    TrOPD66.20 / 83.877.08 / 19.113.75 / 13.7131.93 / 55.6626.84 / 44.4429.48 / 48.1127.55 / 44.15
    DistiLLM-263.88 / 82.467.92 / 20.403.75 / 13.6129.67 / 48.7025.55 / 42.4127.80 / 45.7526.43 / 42.22
    IPD-RandIntervention68.20 / 83.998.33 / 19.417.08 / 13.0933.89 / 52.3027.57 / 43.3431.85 / 49.1929.49 / 43.55
    IPD69.28 / 84.599.17 / 21.696.67 / 15.0636.14 / 57.2529.60 / 46.2631.80 / 49.6330.44 / 45.75
    Specifically:
    • EOPD: we follow their paper and implement EOPD in our codebase on top of OPD, with an additional forward KL term calculated on the teacher's renormalized top-16 and applied where the full-vocabulary teacher entropy exceeds τ = 0.8, with a weight α = 1.0.
    • TrOPD: we use their official repository (github.com/Xingrun-Xing2/TrOPD) to run on our data and models. For each prompt, one response generated by the teacher is used as off-policy data and is trained together with the on-policy samples in a single update. We keep the official hyperparameters: an off-policy weight of β = 0.001, a forward KL on the teacher's top-64 tokens for rejected tokens, and a cosine schedule for α, whose length is set to our 279 training steps.
    • DistiLLM-2: We implement the distillm_v2 loss in our codebase, following their official repository (github.com/jongwooko/distillm-2). For each prompt, the loss combines a skew KL on one teacher-generated response and a skew reverse KL on one student-generated response, with the official defaults of α = 0.1 and equal weights for the two terms. Following Algorithm 1 of their paper, the responses are collected before each training epoch, and since all methods here are trained for one epoch, they are collected once, from the teacher and from the initial student. The divergences are computed on the union of the student's and the teacher's top-16 tokens.
    • OPD-RandKL: on top of OPD, we set the ratio to p=0.5, which means we use forward KL for randomly selected 50% of sequences in each batch, and reverse KL for the remaining.
    • IPD-RandIntervention: on top of IPD, we replace the takeover and handback rules with a random intervention. Each token proposed by the student is replaced by a token sampled from the teacher with a fixed probability of 0.74%, which matches the average proportion of teacher-generated tokens in IPD over training, and the student continues right after it. The source-split KL and all other settings are the same as IPD.

    As observed, IPD still achieves the best average score and leads most benchmarks individually.

AuthorsOpenReview ↗

Responses to Reviewer cyJd (2/3)

  • W5&Q2: Disentangling the Performance Gain

    We separately measure the format compliance and correctness in Qwen3-4B (non-thinking) to Qwen3-1.7B-Base group. In the saved experimental results, the grader records two things per response: whether a final answer can be parsed and whether it is correct. We therefore can decompose every score as mean@8 = parse rate × conditional accuracy, where parse rate is the fraction of responses with a parseable final answer and conditional accuracy is the correctness among those responses.

    Methodmean@8parse rateconditional accuracy
    Qwen3-1.7B-Base9.940.323.3
    OPD28.484.832.0
    SKD28.887.431.5
    IPD30.484.034.4

    Per-benchmark conditional accuracy (correctness among parseable responses):

    MethodMATH-500AIME24AIME25AMC23MinervaOlympiadBench
    Qwen3-1.7B-Base55.42.76.029.021.925.0
    OPD71.212.03.641.027.136.7
    SKD69.612.45.539.128.334.2
    IPD72.813.88.843.230.137.6

    From the disentangling analysis:

    • The gain over the Base model is a mix of format compliance and reasoning correctness, and we can quantify them. The parse rate rises from 40% to 84%, but conditional accuracy also rises from 23.3 to 34.4 (+11.1). Distillation not only improves format compliance, but more importantly, also improves real reasoning capability.
    • IPD's advantage over the distillation baselines is not a format effect. IPD's parse rate (84.0%) is equal to or lower than OPD (84.8%) / SKD (87.4%), while its conditional accuracy is higher and leads on all six benchmarks. Therefore, IPD's higher score is not from format overfitting, and the improvement over other baselines is from better reasoning capability.
  • W5&W7&Q3: Non-Base Model from Another Family

    First, we would like to clarify that most distillation studies primarily focus on Qwen3 family, like Rethinking-OPD (https://arxiv.org/pdf/2604.13016) and the Thinking Machines’ OPD Blog (https://thinkingmachines.ai/blog/on-policy-distillation/). Among the Qwen3 family, we have conducted comprehensive experiments on different student-teacher configurations and model scales.

    Besides, distilling a non-base model to another non-base model, especially from the same family, is not a common setting, and the additional baselines (e.g., EOPD, DistiLLM-2) mentioned by the reviewer also use a base model as the student. The reason has also been discussed in prior work (e.g., Rethinking-OPD), indicating that a significant distillation headroom exists only when the teacher has learned more knowledge than the student. Given that the non-base models from the same family usually share the same training data and recipe and only differ in size, a same-stage non-base to non-base distillation is unlikely to show significant improvement.

    Here, to further address the reviewer’s concern about both the non-base model and another model family, we additionally run OPD/SKD/IPD with OLMo-2-1124-7B-Instruct as the teacher and OLMo-2-0425-1B-SFT, which has better format & instruction following and also a relatively high distillation headroom, as the student.

    Given their lower capability than Qwen3, we use MATH as training data and evaluate the final checkpoint after one epoch on GSM8K and MATH-500. The maximum response length is 3072, as OLMo2 has a 4096-token context window. All other training and evaluation settings are the same as Qwen3. We report the mean@8/best@8 as follows. As observed, IPD also works on non-base model distillation from another family and outperforms OPD.

    MethodGSM8KMATH-500Avg.
    OLMo-2-0425-1B-SFT44.04 / 78.1410.88 / 31.1427.46 / 54.64
    OPD57.67 / 79.5316.00 / 34.0836.83 / 56.80
    SKD59.46 / 83.2615.43 / 35.5537.44 / 59.41
    IPD61.69 / 83.6118.50 / 38.0640.10 / 60.83

    In summary, the performance disentangling experiment above can prove that IPD brings real reasoning capability improvement beyond format alignment, and the experiment here can prove the effectiveness of IPD on both non-base distillation and another model family.

AuthorsOpenReview ↗

Responses to Reviewer cyJd (3/3)

  • Q4: Pass@1 under Greedy Decoding

    In addition to mean@8 and best@8, we provide the pass@1 score under greedy decoding here (Qwen3-4B to Qwen3-1.7B-Base).

    MethodAvg.MATH-500AIME24AIME25AMC23MinervaOlympiad
    OPD27.064.06.73.328.926.132.7
    SKD29.069.410.06.731.327.928.4
    IPD31.969.413.36.741.028.732.1
  • W2&Q5: Justifying why a mixed-source prefix makes sense

    Our design of IPD starts from the intuition that the teacher should intervene more when the student has a large distribution gap and might generate prefixes that confuse the teacher and consequently get unreliable supervision, and gradually less when the student is more aligned with the teacher during training.

    As Figure 7 shows, at the beginning, the intervention ratio is relatively large (4.9% at step 0), where the student has the largest gap from the teacher. As training goes on, the student is gradually aligned with the teacher and narrowing the gap, and the intervention ratio decreases, leaning more and more toward an on-policy paradigm. In the later training stage, the intervention ratio is at a very low level (0.4% at the final step), and the trajectory is almost generated by the student.

    The teacher’s intervention has a higher impact in the early stages and brings the student back from a deviated region, and that in some sense explains the faster convergence and higher data efficiency of IPD.

  • W7&Q6: OOD Performance

    We run additional OOD evaluations in STEM, Coding, and General Instruction Following domains. The results are shown as follows.

    • STEM (including MMLU-Pro Physics, Chemistry, Biology, and Computer Science subsets). Results are reported as mean@8 / best@8.

      MethodAvg.PhysicsChemistryBiologyComputer Science
      Qwen3-1.7B-Base5.4 / 30.44.5 / 27.24.0 / 23.78.9 / 43.96.3 / 35.5
      OPD30.1 / 71.425.2 / 68.223.2 / 68.249.6 / 83.130.9 / 69.7
      SKD30.5 / 74.626.9 / 71.824.7 / 71.646.6 / 84.829.8 / 73.6
      IPD31.7 / 76.028.6 / 74.727.7 / 74.643.8 / 83.231.2 / 71.7
      Qwen3-4B (non-thinking)70.9 / 89.769.4 / 91.069.0 / 89.179.9 / 91.264.9 / 84.5
    • Coding (HumanEval/HumanEval+)

      MethodHumanEval pass@1HumanEval+ pass@1
      Qwen3-1.7B-Base64.659.8
      OPD69.564.0
      SKD69.564.0
      IPD73.268.3
      Qwen3-4B (non-thinking)84.177.4
    • General Instruction Following (IFEval)

      MethodPrompt-level StrictPrompt-level LooseInstruction-level Strict
      Qwen3-1.7B-Base15.716.228.4
      OPD26.730.139.8
      SKD31.636.545.5
      IPD32.838.046.1
      Qwen3-4B (non-thinking)80.884.186.6

    As observed, even though only distilled on math reasoning data, the teacher also transfers part of its capability on other domains to the student, as discussed in https://arxiv.org/abs/2608.16647. Meanwhile, among OOD benchmarks, IPD still shows better performance than baselines, indicating its good OOD generalizability.

Given the limited time and accessible compute, we are unable to run the additional experiments for every model configuration, but we believe we have shown comprehensive and convincing results for the selected configuration. Again, we thank the reviewer for your effort and valuable feedback, which helps us further improve the quality of our paper. We hope our responses can address your concerns, and we sincerely appreciate it if you could consider raising your rating.

AuthorsOpenReview ↗

Follow-up to Reviewer cyJd

Dear Reviewer,

Thank you again for your valuable feedback. Regarding your concerns, we have provided detailed responses. As the discussion period draws to a close, we would greatly appreciate hearing whether you have any remaining concerns. We would be happy to get involved in further discussions while the discussion period is still open. If our responses have addressed your concerns, we would be really grateful if you would consider updating your rating accordingly.

Thank you again for your time and consideration.

Submission22469 Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Looking Forward to Your Further Feedback

Dear Reviewers,

Thank you once again for your valuable feedback. We are encouraged that the reviews agree on the main strengths of our paper, including

  • Reviewers WR2T and 8RsB both endorse the design of IPD. Reviewer WR2T finds that it addresses an important practical challenge with an intuitive and well-motivated strategy, and Reviewer 8RsB describes it as a principled bridge between on-policy and off-policy distillation with a mathematically natural and empirically superior formulation.
  • The fused inference engine for the non-trivial multi-model interleaved rollout is regarded as an essential systems contribution by Reviewer 8RsB and is also appreciated by Reviewer cyJd, who further finds the data efficiency of IPD interesting.
  • All three reviewers find the paper clear and easy to follow.

As the discussion period draws to a close, we would like to respectfully send a kind reminder that our detailed responses are available in the comments. For ease of reading, we’ve summarized them below.

  • We added five baselines: EOPD, TrOPD, DistiLLM-2, and two random baselines that randomize the KL direction or teacher intervention. IPD remains the best in average scores and still leads most benchmarks individually. We also added a discussion of ToDi, DFT, and PEAR in Appendix B.4.
  • We decomposed the accuracy into the parse rate and the conditional accuracy among parseable responses. IPD has a higher conditional accuracy on all six benchmarks (34.4 on average, compared with 32.0 for OPD and 31.5 for SKD) without a higher parse rate, indicating that the gain comes from better reasoning rather than format compliance.
  • We reported pass@1 under greedy decoding, where IPD achieves 31.9, outperforming 29.0 for SKD and 27.0 for OPD.
  • We evaluated the students trained on math on OOD benchmarks, and IPD achieves the best results on STEM, coding, and general instruction following.
  • We added a non-base distillation experiment from another model family (OLMo-2-7B-Instruct to OLMo-2-1B-SFT), where IPD achieves the best average scores and leads every benchmark.
  • We carried out a further intervention analysis on how often / where interventions happen and the sensitivity to K / model size gap, providing further insights. The eventual low intervention ratio and almost-on-policy rollout also indicate that the concern about a new distribution mismatch does not arise in practice.
  • We clarified the differences between IPD and SKD and further explained Figure 2.
  • We provide the analysis of training wall-clock time versus student performance. An IPD step is slower than an OPD step, but IPD yields a better student within the same training time, and costs less time for comparable students.

If you have any remaining questions or concerns, we would appreciate hearing from you soon so we have time to respond before the discussion closes on November 18. If our responses have addressed your concerns, we would be really grateful if you would consider updating your overall assessment. Thank you again for your time and feedback.

Best regards,

Submission22469 Authors

AuthorsOpenReview ↗

Summary of Reviews and Author Responses

Dear Area Chairs and Reviewers,

We thank all reviewers for their feedback. As the discussion period is about to end and we have not received any follow-up comments, we summarize the reviews and our responses below.

Strengths recognized by the reviewers

  • Our paper addresses an important practical challenge in distilling large models into smaller students, and the interactive generation strategy is intuitive and well motivated (WR2T).
  • Our method is a principled bridge between on-policy and off-policy distillation, with a mathematically natural and empirically superior formulation (8RsB).
  • The fused inference engine is an essential systems contribution that removes the main deployment barrier for interactive distillation (8RsB). It is also appreciated by Reviewer cyJd, who further finds the data efficiency of IPD interesting.
  • All the reviewers find our paper clear and easy to follow.

Responses to the reviewers

  • Reviewer cyJd. The reviewer's questions primarily suggest additional experiments. We have carried out all of them and provided analysis, and the conclusion of the paper remains unchanged in every case.
    1. Additional baselines. We added two random baselines proposed by the reviewer and three mentioned methods (EOPD, TrOPD, DistiLLM-2). IPD still achieves the best average scores and leads most benchmarks individually. Comparison with random baselines also proves that the gain comes from our unique designs of the bidirectional propose-and-verify mechanism and source-split KL, instead of randomly mixing teacher tokens or KL directions.
    2. Disentangling performance gain. We decomposed mean@8 accuracy into the parse rate and the conditional accuracy among parseable responses. IPD does not have a higher parse rate than OPD / SKD, and meanwhile it has a higher conditional accuracy among parseable responses on all six benchmarks, reflecting IPD's better reasoning gain rather than format overfitting.
    3. Non-base distillation. We distilled OLMo-2-7B-Instruct into OLMo-2-1B-SFT, which addresses both the non-base distillation and another family distillation concerns. IPD still achieves the best average scores and leads every benchmark.
    4. Pass@1. IPD achieves the best pass@1 under greedy decoding, outperforming SKD and OPD.
    5. Mixed-source prefixes. We clarified that the intervention ratio at the beginning is relatively large, where the student has the largest gap from the teacher, and as training goes on, the student is gradually aligned with the teacher and the intervention ratio decreases, leaning more and more toward an on-policy paradigm. The intervention ratio is eventually very low, and the trajectory is almost generated by only the student. The concern about a new distribution mismatch does not arise in practice.
    6. OOD. Trained on math only, all the distillation methods can generalize to other domains. IPD achieves the best OOD performance, including STEM, coding, and general instruction following.
  • Reviewer WR2T. The reviewer's questions are primarily on further explanations and analysis. We accordingly clarified our novelty, reported additional baselines, and provided further explanations and analysis.
    1. Novelty over SKD. We further clarified our novelty over SKD, including the unique designs of bidirectional propose-and-verify and source-split KL, and a dedicated inference engine that benefits not only IPD but also other algorithms like SKD that require interleaved multi-model rollouts.
    2. Baselines and related work. We added DistiLLM-2, EOPD, and TrOPD to baselines. IPD still achieves the best average scores and leads most benchmarks individually. ToDi, DFT, and PEAR are discussed in the new Appendix B.4.
    3. Figure 2 explanation. We explained that the teacher's distribution becomes flat on incorrect student trajectories while the reverse KL stays as low as on correct ones. The reverse KL is therefore not a reliable learning signal when the teacher is confused by the student prefix, which motivates us to apply adaptive teacher intervention to address it.
    4. Student-teacher shift. We carried out a further intervention analysis on how often / where interventions happen and the sensitivity to K / model size gap, providing further insights.
  • Reviewer 8RsB. The reviewer's question is primarily about the wall-clock time. For Qwen3-4B to Qwen3-1.7B-Base, one IPD step takes 158.00s, and one OPD step takes 124.72s, but IPD converges faster, so it actually gives a better student under the same training time, or costs less time for the same-level student.

Manuscript updates

  • Appendix B.4: discussion of ToDi, DFT, and PEAR.
  • Appendix C.3: wall-clock time and student performance analysis.

We would be grateful if the reviewers could let us know whether all the concerns have been addressed, and we are happy to clarify any remaining concerns and get involved in further discussions.

Best regards,

Submission22469 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.