acceptodds
Under review as a conference paper at ICLR 2027

Mixed-Policy Distillation: Bringing Teacher and Student onto Common Ground

Abstract

Distillation involves both the student and teacher policies, but trajectories aligned with one are not necessarily suitable for the other. In language model post-training, teacher-generated trajectories can depart from the student's inference distribution, while on-policy distillation (OPD) can expose the teacher to unfamiliar student-generated states and weaken its guidance. We introduce Mixed-Policy Distillation (MPD), a post-training framework that constructs trajectories for compatibility with both teacher and student. MPD combines a student-generated prefix with a teacher-generated continuation through an uncertainty-guided handoff and distills the teacher's token distributions across the full mixed trajectory. In this design, the teacher helps shape subsequent states while its guidance remains anchored in the student's own reasoning. Empirical analysis shows that, after the handoff, mixed trajectories achieve lower joint perplexity and stronger next-token agreement than trajectories generated entirely by either policy alone. Experiments across mathematics, science, and tool use on various Qwen and Gemma models demonstrate improved performance over both off-policy and on-policy counterparts with both externally trained teachers and privileged-information-conditioned self-teachers. These results identify mixed-policy trajectory construction as a promising design choice for distillation-based post-training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Knowledge Distillation, Language Model Post-Training, Trajectory Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_URdnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on on policy distillation. The author proposes Mixed-Policy Distillation (MPD), which constructs a student-generated prefix with a teacher-generated continuation, to solve the problem that the teacher is not familiar to the student-generated states.

Critical strengths

1 The paper is logically structured, the method is easy to follow, and the diagrams are clear and intuitive.

2 Experimental results demonstrate that the proposed method outperforms offKD and the original OPD, while also showing strong performance on OOD data.

Critical weaknesses

1 In the experiments section, the main results table compares the method only against two basic baselines—offKD and OPD—while omitting comparisons with other methods that also employ mixed OPD strategies, such as Speculative KD and TRD , which the paper mentions.

2 As shown in Figure 2, while a mixed policy might use a single concatenation to steer the trajectory away from regions of overlap, it is possible that the trajectory could subsequently drift into areas requiring "teacher support." Relying on a single correction step seems somewhat crude.

Questions for authors

1 Why is the location of the student's highest entropy necessarily the optimal point for the teacher to take over? High entropy indicates student uncertainty, but it does not guarantee that the teacher can provide the most effective guidance at that specific point.

2 Does MPD retain its performance advantage over OPD when the computational budget is held constant? Since MPD requires generating the full student response to identify the point of highest entropy, it incurs additional computational costs.

3 comments
AuthorsOpenReview ↗

Clarifications on MPD’s Teacher Intervention Strategy, Handoff Design, and Training Efficiency

Q1. Speculative KD and TRD

Our experimental budget prioritizes standard reference baselines and broad evaluation of MPD. The completed four-model, three-domain, two-teacher study has already consumed substantial compute and funding. Each additional method requires implementation and adaptation, hyperparameter tuning, validation, and full training and evaluation. The remaining budget cannot support these additional comparisons.

Speculative KD is an ICLR 2025 publication that replaces student token proposals using teacher feedback. TRD's cited arXiv version is listed as under review and refines student trajectories under teacher guidance. For recent preprints such as TRD, adding a benchmark entails implementing and validating another training pipeline in our setting, alongside the cost of the training itself.

MPD instead uses a student prefix followed by a continuous teacher continuation. We discuss these distinct intervention mechanisms in related work and use recognised methods (OffKD and OPD) as the experimentally evaluated reference points.

Q2. Does one handoff provide enough teacher support?

One handoff initiates a complete continuation in which the teacher keeps generating subsequent tokens. It is not a one-token correction followed by an immediate return to independent student generation. The student prefix selects the learner-visited state, and the teacher then demonstrates an extended path forward. This is why a single handoff can still provide sustained guidance.

The paper also varies the number of cut points M selected from the same original student rollout:

MAccuracy reported in the manuscript (%)Teacher tokens per step reported in the manuscript
168.4451,122
269.62102,123
468.04192,617
868.04379,510

In this reported comparison, increasing M substantially increases teacher generation without a monotonic accuracy improvement. It motivates the simple default as a practical cost–performance choice. The multi-cut study also does not directly establish the benefit of repeatedly alternating policies within one finalized sequence.

Q3. Why should the teacher take over at high student entropy?

High entropy identifies student uncertainty, which makes it an interpretable place to offer a demonstration. Maximum KL instead identifies disagreement between student and teacher. The paper compares these two signals with random handoff in the stated science setting.

The learning intuition is that a teacher continuation can show how to resolve an uncertain branch while remaining anchored in the student's reasoning. This is consistent with findings about high-entropy reasoning forks in Beyond the 80/20 Rule, while TIP also demonstrates the relevance of teacher–student disagreement.

Neither result guarantees an optimal handoff. We use entropy as a motivated practical rule and PPL/overlap as diagnostics of the resulting trajectory, rather than a proof that teacher benefit is maximal at the selected token.

Q4. Time to accuracy and generation overhead

MPD reaches high accuracy earlier in the existing Qwen3.5-4B science runs. The first evaluated checkpoints meeting each threshold are:

Accuracy thresholdOPD: step / logged step-hoursMPD: step / logged step-hours
At least 60%60 / 3.77 h10 / 1.12 h
At least 63%200 / 12.54 h20 / 2.22 h
At least 65%Not reached within 334 steps20 / 2.22 h

Times are cumulative logged training-step durations at the first qualifying evaluation.

At step 20, MPD achieves 65.29%, close to its late-training mean of 65.46%. The recorded training-step time includes the complete student rollout, including the discarded suffix, and MPD's teacher-continuation stage. The continuous teacher demonstration is designed to help the student exploit difficult examples sooner, motivating the additional generation work per update.

My question still concerns the link between student uncertainty and the benefits of teacher intervention; I hope the author can provide a clearer explanation of why teacher guidance is more effective in this context.

AuthorsOpenReview ↗

Explanation for Entropy-Guided Teacher Intervention at Reasoning Forks

Thank you for the follow-up. We use entropy to place teacher intervention before an uncertain reasoning choice determines the rest of the rollout. At a reasoning fork, the student may assign substantial probability to several continuations. If it chooses an unproductive branch, later tokens build on that choice, and teacher supervision must operate on the resulting states. Taking over before this choice allows the teacher to steer the continuation toward a more productive path.

Compared with intervening during a routine stretch of generation, intervention at such a fork has a greater opportunity to change the direction of the entire suffix. For MPD, the learning benefit we seek is a useful continuation precisely where the student struggles to continue independently. The student-generated prefix connects this continuation to its own partial reasoning; the teacher suffix demonstrates how to resolve the uncertain choice and develop the solution afterward. Distillation on this trajectory therefore teaches the student how to proceed through a difficulty encountered in its own generation.

Intervention studies support this motivation. In CURE, regenerating trajectories from high-entropy rather than random cut points improves the trained model's average accuracy from 51.4% to 53.7% under the same reported training budget and hyperparameters (Section 4.5). Shen et al. further show that entropy-triggered delegation to a stronger model improves completed responses through sparse token interventions (Appendix C.3). These results support using high-entropy positions to intervene on the continuation and construct more useful trajectories for learning.

Reviewer_AAH5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies on-policy distillation (OPD), which is a very hot topic in current LLM post-training. The authors claim that using either student-only rollouts or teacher-only rollouts is not the most effective way for distillation. The authors then propose Mixed Policy Distillation (MPD), which mixes the student-generated prefix and teacher-continued suffix for distillation. The authors find that MPD improves the metrics of sequence ppl and top-K overlap between student and teacher predictions. The experimental results on Qwen and Gemma families shows that MPD demonstrated better performance than off-policy and on-policy distillation baselines.

Critical strengths
  1. The paper is generally well-written.

  2. The studied problem is important, which may bring some new information to the community.

  3. MPD also improves OOD performance, which is interesting.

Critical weaknesses
  1. First, the motivation and methodology do not seem sound to me. In Section 3, the authors do not clearly present the motivation for and logic behind proposing the MPD. OPD has been shown to be a highly effective distillation method, yet the authors do not justify to the reader why “student-generated trajectories are not necessarily suitable for teacher supervision,” as they provide no empirical results or evidence. Regarding the method itself, why not use a teacher-generated prefix followed by a student model-generated continuation for distillation? This seems more natural, as a good beginning is half the battle: the teacher can guide the student model in the right direction from the outset. Finally, the selection of the handoff position is very ad hoc, lacking any rigorous mathematical or empirical justification.

  2. Second, the experiments and results are not comprehensive and convincing. The experimental models are quite small. For the OPD baseline under GRPO-trained teachers, one would expect the student model to faithfully reproduce the teacher’s behavior. However, the results in the figure reveal a substantial gap between OPD and the teacher, raising the concern that OPD has not converged or that the results are highly variable and therefore not sufficiently reliable. Finally, the improvement of MPD over baselines are very marginal (only 1~2 points with high variance).

Questions for authors

Refer to Weakness above.

Typo: Line 323, "Base model" -> "base model"

2 comments
AuthorsOpenReview ↗

Clarifications on MPD Design Rationale, Teacher Guidance, and Empirical Scope

Q1. Why can student-generated trajectories weaken teacher guidance?

Student trajectories are on-policy for the learner but may present atypical conditioning states to the teacher, weakening its effective guidance.

The Many Faces of OPD (Sec. 5.1, App. A.22) reports that Qwen3-14B accuracy on 198 GPQA-Diamond questions drops from 62.12% to 45.96% when continuing truncated Qwen3-1.7B prefixes. Supervision Fidelity Decay similarly reports weaker teacher discrimination with longer student prefixes under reverse KL.

Our hypothesis is that useful trajectories should remain compatible with both policies: student roll-in ensures reachable states, while teacher continuation provides coherent reasoning. Student/teacher/joint PPL and Top-K overlap probe this compatibility.

Q2. Why student prefix followed by teacher suffix?

Our goal is to let the teacher demonstrate how to continue from a state the student actually reaches. The student first develops a partial solution, followed by a sustained teacher continuation. This follows the learner-roll-in/expert-guidance intuition of imitation learning (e.g., AggreVaTe uses learner-visited states and expert cost-to-go information).

A teacher-generated prefix may improve the correctness of an assisted completion. However, this may benefit assisted inference without necessarily producing a better training trajectory for the student's later independent inference. Such prefixes may place the student at reasoning states it would not reach on its own. Student-first construction directly addresses this concern.

Moreover, our entropy-based handoff identifies uncertain positions along the student's own rollout. For reverse-order construction, we lack an equally intuitive criterion for selecting the handoff point. We do not claim reverse ordering is ineffective; rather, our rationale explains the chosen direction. A direct comparison would require additional training beyond our remaining budget.

Q3. Why choose the highest-entropy position?

Maximum entropy identifies uncertain next-step decisions where teacher continuation may be particularly informative.

Beyond the 80/20 Rule associates high-entropy tokens with reasoning forks, while TIP examines student entropy and teacher–student disagreement, including informative low-entropy/high-disagreement positions.

Our Qwen3.5-4B science ablation yields 68.44% (entropy), 66.86% (KL), and 64.29% (random). These results support entropy as an interpretable heuristic in this setting, without establishing theoretical optimality or guaranteeing maximal teacher benefit.

Q4. OPD training and the teacher–student gap

OPD completed its planned training schedule. The existing records provide the following evidence:

EvidenceResult
OPD training, Qwen3.5-4B science2 epochs, 334 updates
OPD best and final accuracy63.31% (step 200); 59.96% (step 334)
Training after best checkpoint134 further updates without improvement
GRPO teacher accuracy66.86%
Qwen3.5-4B science, OPD → MPD63.31% → 68.44% (+5.13 pp)
Gemma3-4B science, OPSD → MPSD56.01% → 67.85% (+11.84 pp)
Gemma3-12B tool use, OPSD → MPSD56.70% → 69.07% (+12.37 pp)

The remaining teacher–student gap may reflect their different abilities on this training set: some problems are solvable by the teacher but remain beyond the student's current reasoning ability. Even after OPD reaches a plateau, these difficult examples can remain hard to learn from through teacher feedback on student-generated trajectories. MPD is designed to make these same examples more useful: the student first reaches its own reasoning state, and the teacher then demonstrates a continuous path forward. This gives the student a coherent way through a difficult problem and helps it exploit the teacher's capability more effectively.

Q5. Model scale and experimental budget

Our experiments cover four Qwen/Gemma models (4B–12B) across three domains and two teacher settings, providing diversity in model families, tasks, and supervision.

A 27B comparison or comprehensive scaling study would require substantial additional training beyond our remaining budget.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer,

Thank you again for your thoughtful comments and feedback. We would like to kindly follow up and ask whether our previous response has adequately addressed your concerns.

We would greatly appreciate hearing your thoughts. If any questions or concerns remain, we would be happy to provide further clarification and continue the discussion.

Thank you for your time and consideration.

Best regards,
The Authors

Reviewer_WAXROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Mixed-Policy Distillation (MPD), which retains a student-generated prefix, switches to teacher generation at the student’s highest-entropy position, and applies token-level distillation across the resulting trajectory. Experiments cover four Qwen/Gemma models across mathematics, science, and tool use, using GRPO-trained teachers and privileged-information self-teachers. The authors claimed that MPD generally improves over on-policy and off-policy distillation, with additional analyses examining trajectory compatibility and handoff selection.

Critical strengths
  1. The author(s) proposed a simple, actionable approach. MPD directly addresses the complementary limitations of student-only and teacher-only trajectories with a clearly specified procedure that also extends naturally to self-distillation.
  2. Broad empirical coverage. Evaluation spans four models, three domains, and two teacher settings. The substantial self-distillation gains on Gemma models in Table 2 provide particularly encouraging evidence of practical utility.
Critical weaknesses
  1. The comparisons do not establish the value of the specific mixing strategy. The main baselines are pure on-policy and off-policy distillation. There is a lack of sequence-level mixture baseline or empirical comparison with the related teacher-guided methods discussed in Section 5. In addition, MPD generates a complete student rollout before adding a teacher continuation, but total compute and wall-clock comparisons are absent. It remains unclear whether entropy-guided handoff offers an advantage over simpler alternatives at comparable cost.
  2. The supporting evidence is insufficiently robust. The compatibility analysis relies on a handful of science examples, while handoff ablations cover only one model–domain setting. Lower perplexity and higher token overlap do not directly establish better corrective supervision. No multi-seed variability or confidence intervals are reported, making small gains, such as Gemma3-4B’s 16.97 versus 16.69 Math Avg., and are difficult to assess.
Questions for authors
  1. Does MPD retain its advantage over a sequence-level mixture of teacher/student trajectories and a fixed-position handoff under matched total compute? Please include the discarded student suffix when accounting for generation cost.
  2. How stable are the main gains across training seeds, and do the compatibility improvements hold over the full evaluation set and correspond to improved answer correctness?
2 comments
AuthorsOpenReview ↗

Clarifications on MPD Design Choices, Empirical Evidence, and Limitations

Q1. Why within-trajectory mixing, and what do baselines establish?

Unlike sequence-level mixing, which alternates between complete student and teacher trajectories, MPD conditions the teacher's continuation on a student-generated prefix. This provides demonstrations grounded in states the student actually visits, potentially improving learning from difficult trajectories.

OffKD and OPD represent the teacher-only and student-only endpoints, but do not establish superiority over untested sequence-level mixtures. Related methods such as Speculative KD and TRD also improve learning trajectories through teacher intervention, whereas MPD uses an uncertainty-based handoff followed by a continuous teacher suffix.

Given computational constraints, we limit empirical claims to evaluated baselines and clarify these distinctions in the related-work discussion.

Q2. Why entropy, and how generalizable is this choice?

Student entropy identifies uncertain next-step decisions where teacher continuation may be most useful, whereas teacher–student KL measures distributional disagreement.

Our Qwen3.5-4B science ablation shows:

Handoff ruleAccuracy (%)
Maximum student entropy68.44
Maximum forward KL66.86
Random position64.29

These results support entropy as a practical heuristic in this setting, rather than establishing universal optimality. A comprehensive cross-model and cross-domain ablation is beyond our remaining compute budget.

Q3. Early convergence and complete generation cost

MPD achieves strong performance early despite higher per-update generation costs. On Qwen3.5-4B science, it reaches 62.72% at step 10 (1.12 h) and 65.29% at step 20 (2.22 h). Step 20 represents only ~6% of the 334 updates, already approaching the later-checkpoint mean of 65.46% and exceeding OPD's best recorded 63.31%.

The first observed attainment of accuracy thresholds further illustrates this:

ThresholdOPDMPD
≥60%Step 60 (3.77 h)Step 10 (1.12 h)
≥63%Step 200 (12.54 h)Step 20 (2.22 h)
≥65%Not reachedStep 20 (2.22 h)

Importantly, the discarded student suffix is included in generation costs. Although MPD incurs greater per-update overhead, it reaches useful performance levels earlier in the recorded runs.

Q4. Multi-seed variability and small gains

We acknowledge that single-run results cannot establish statistical significance. Multi-seed retraining is infeasible given the substantial cost of the completed experimental matrix.

Nevertheless, the observed improvements vary meaningfully across settings:

SettingBaselineMPD variantGain (pp)
Qwen3.5-4B, science, GRPO63.3168.44+5.13
Gemma3-4B, science, self-teacher56.0167.85+11.84
Gemma3-12B, tool use, self-teacher56.7069.07+12.37
Gemma3-4B, math, GRPO16.6916.97+0.28

We therefore present these as point-estimate improvements, explicitly distinguishing substantial gains from marginal ones.

Q5. What do PPL, overlap, and illustrated cases establish?

These probes characterize trajectory compatibility rather than directly proving corrective supervision or answer correctness.

Student and teacher PPL measure how surprising a trajectory is to each policy; joint PPL is their geometric mean. Top-K overlap measures next-token distribution agreement at a shared prefix. Together, they help assess whether trajectories remain accessible to the student while supporting coherent teacher continuations.

The illustrated cases, selected from top-k samples of evaluation sets, demonstrate possible behaviors but are not comprehensive evidence. We therefore present these analyses as descriptive support alongside downstream task performance.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer,

Thank you again for your thoughtful comments and feedback. We would like to kindly follow up and ask whether our previous response has adequately addressed your concerns.

We would greatly appreciate hearing your thoughts. If any questions or concerns remain, we would be happy to provide further clarification and continue the discussion.

Thank you for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.