acceptodds
Under review as a conference paper at ICLR 2027

Multi-Teacher On-Policy Distillation via Gated Meta-Learning

Abstract

Multi-teacher on-policy distillation (MOPD) consolidates multiple domain-specialized teachers into a single deployable student through dense on-policy supervision. Yet, we find that current methods easily suffer from a teacher-strengthening paradox—stronger teachers can yield a worse merged student—due to increasingly specialized reasoning patterns and the resulting loss of student-teacher overlap across domains. In this paper, we introduce GMOPD, a gated meta-learning framework for MOPD that optimizes teacher integration based on post-adaptation performance. We formulate each base-referenced OPD task as a KL-regularized RL problem and optimize a meta-objective that learns a shared initialization capable of rapidly adapting to each teacher. We further train an adaptive rank gate that selectively activates teacher-relevant LoRA ranks while protecting capabilities required by the remaining teachers. Together, these components preserve student-teacher overlap and mitigate cross-teacher forgetting. We evaluate GMOPD on six challenging benchmarks spanning mathematical reasoning, coding, scientific reasoning, instruction following, tool use, and general reasoning. Across multiple model scales, GMOPD consistently outperforms existing merging and distillation methods, scales to 20 heterogeneous teachers, and composes complementary capabilities beyond those of individual teachers.

Primary area:
generative models
Keywords:
On-Policy Distillation, Model Merging, Meta Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_56DkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GMOPD, a gated meta-learning approach to multi-teacher on-policy distillation. It freezes the teacher models and the student’s base weights, represents multi-domain knowledge through shared LoRA matrices, and introduces learnable rank gates. During inner-loop adaptation to a teacher, only the gates are updated to restrict the degrees of freedom. The outer loop optimizes the shared gate initialization based on post-adaptation distillation performance, while training the shared LoRA basis using the multi-teacher distillation loss evaluated before adaptation. This design aims to make the student readily adaptable to individual teachers while remaining directly deployable as a unified model, thereby mitigating cross-teacher interference and forgetting.

Critical strengths

1 The related work is reviewed reasonably thoroughly.

2 The paper identifies the phenomenon that stronger teachers can lead to a worse merged student, which is a practically meaningful research problem.

3 The method is simple and clearly motivated.

Critical weaknesses

1 What does “Preliminary projected” mean in the captions of Tables 4–6 in the appendix? In this context, “projected” suggests estimated rather than measured results, raising serious concerns about the reliability of the reported experimental findings.

2 The baselines, such as ExOPD, are implemented using LoRA updates. Could this constrain their performance, given that these methods predominantly use full-parameter updates in their original papers? At a minimum, the authors should report the performance difference between full-parameter and LoRA-based training for ExOPD.

3 Where is Figure 3 referenced in the main text? Furthermore, its model-size labels—2B, 4B, and 9B—are inconsistent with the experimental settings described in the text: 4B, 8B, and 30B-A3B. Please explain this discrepancy.

4 What are the specific configurations of the multiple teachers within each domain, and how do they differ?

5 In Figure 2, the baseline methods show almost no change in performance as the number of teachers increases. Why is this the case? Is applying the paper’s teacher configuration directly to the other methods a fair comparison?

6 Why were no experiments conducted in thinking mode, given that it is commonly used with many current models?

Questions for authors

see Weaknesses

2 comments

W1: What does “Preliminary projected” mean in the captions of Tables 4–6 in the appendix? In this context, “projected” suggests estimated rather than measured results, raising serious concerns about the reliability of the reported experimental findings.

To clarify, “preliminary projected” refers to our preliminary assessment of variability based on a limited number of measured experimental runs. The revised Tables 4–6 report measured results from three independent seeds, and we have updated the captions to state this explicitly. We also provide the individual results for all five seeds in Table.7-9.

One source of variability is teacher construction: domain-specific RL can produce teachers with different capabilities across seeds, which in turn changes the supervision received by the student during MOPD. This upstream variability may contribute to the variation in the final student performance and should not be conflated with the stability of distillation under fixed teachers.

W2: The baselines, such as ExOPD, are implemented using LoRA updates. Could this constrain their performance, given that these methods predominantly use full-parameter updates in their original papers? At a minimum, the authors should report the performance difference between full-parameter and LoRA-based training for ExOPD.

To clarify, all the baseline methods are implemented by their publicly available code with full-parameter updates, which is the same as their original settings.

In addition, we verify the performance of these baseline methods with LoRA-based training.

MethodMath ↑Code ↑Science ↑IF ↑Tool ↑
ExOPD56.029.045.837.543.4
ExOPD (LoRA r=32)51.223.638.533.239.1
Ours58.531.049.042.047.0

The results confirm that restricting ExOPD to LoRA updates can affect its performance. Our method nevertheless outperforms full-parameter ExOPD across all five domains, indicating that its gains do not arise from restricting the baseline to LoRA-based training.

W3: Where is Figure 3 referenced in the main text? Furthermore, its model-size labels—2B, 4B, and 9B—are inconsistent with the experimental settings described in the text: 4B, 8B, and 30B-A3B. Please explain this discrepancy.

Thank you for pointing this out. We have corrected the model-size labels in Figure.3 to 4B, 8B, and 30B-A3B, and carefully check the manuscript.

W4: What are the specific configurations of the multiple teachers within each domain, and how do they differ?

We organize the multiple teachers within each domain according to the specific task categories defined by established benchmarks. Taking the coding domain as an example, we configure four specialized teachers following the four scenarios of LiveCodeBench: (1) Code Generation, which generates solutions from natural language problem descriptions; (2) Self-Repair, which corrects erroneous code based on execution errors or failing test cases; (3) Code Execution, which predicts program outputs given code snippets and input values; and (4) Test Output Prediction, which predicts expected outputs from problem descriptions and test inputs without access to the implementation.

These teachers differ in their task formulations, input contexts, and required reasoning capabilities, allowing them to provide complementary supervision for different aspects of coding proficiency. We add these teacher configurations and their respective roles in Appendix C.1.

Dear Reviewer,

As the author–reviewer discussion period is approaching its end, we wanted to kindly follow up to ask whether our rebuttal have addressed your concerns, or if there are any remaining issues that we could further clarify.

We would be more than happy to provide additional explanations or evidence before the discussion closes. If you feel that your concerns have been sufficiently addressed, we would sincerely appreciate it if you could acknowledge this in the discussion.

Thank you again for your time and consideration!

Best regards, The Authors

Reviewer_2UCXOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces GMOPD, gated meta-learning framework for on-policy distillation where a student can distil knowledge from multiple specialized teacher models. The authors argue that traditional multi-teacher OPD frameworks face cross-teacher forgetting when different teachers with different specialized skills are combined. To mitigate this problem, GMOPD optimizes a meta-objective that learns a shared initialization (meta-student) that can inherit knowledge from multiple teachers. Further, the method learns an adaptive ranking gate which ensures/minimizes the cross-teacher forgetting. The authors perform thorough experiements with Qwen3 models (4B - 30B sizes) on 6 complex tasks, where GMOPD exhibit superior performance over existing baselines (5 competitive methods). GMOPD demonstrates strong scaling behavior, distilling knowledge from as many as 20 teachers with robust performance gain.

Critical strengths
  • Strong methodological contribution - the paper makes strong contribution in the domain of multi-teacher OPD, with a novel meta objective and adaptive ranking gating mechanism.
  • Strong theoretical contribution - the proposed method accompanies strong theoretical analysis with meaningful information.
  • Strong experimental setup - the paper covers 6 complex domains and competes with 5 competitive baselines. The performance gains are consistent and robust.
  • Strong scaling performance - the paper also adds experiments to demonstrate how the method scales with larger teacher pool; the results are promising and informative.

Overall, I found the paper very well written, engaging and informative.

Critical weaknesses
  • One methodological misconception - The main justification behind adaptive ranking gate is that number of LoRA parameters can be substantial, and therefore, separate LoRA adapters for each teacher might not be feasible. In such a scenario, why can't prompt tuning, prefix tuning or even multi-adapter be used? The authors should add experiments to show the trade-off. The authors could also consider adding experiments to add the trade-off for having multiple LoRA instead of rank-gated LoRA. The trade-off can be between task performance vs model size (or throughput or training time).
Questions for authors
  • How do you ensure that different specialized teacher learn different soft-gated rank matrix? What happens if all teachers converge to similar rank matrices? Can we have some regularizer to control this behavior?
2 comments

W1: In such a scenario, why can't prompt tuning, prefix tuning or even multi-adapter be used? The authors should add experiments to show the trade-off. The authors could also consider adding experiments to add the trade-off for having multiple LoRA instead of rank-gated LoRA. The trade-off can be between task performance vs model size (or throughput or training time).

Thank you for this suggestion. Our motivation is not solely to reduce adapter storage. We aim to consolidate complementary teacher capabilities into a single student that uses the same parameters across domains, with the goal of supporting transfer across tasks and generalization to mixed-domain inputs. Maintaining separate teacher-specific LoRA adapters can preserve specialized capabilities, but task-specific routing via prompt tuning, prefix tuning or multi-adapter requires adapter selection or composition at inference time.

To examine this trade-off, we have added a comparison with multiple teacher-specific LoRA adapters.

MethodMath ↑Code ↑Science ↑IF ↑Tool ↑Relative inference time ↓
Shared prompt tuning51.126.245.035.243.51.08×
Shared prefix tuning52.927.746.536.843.81.10×
Multiple LoRA (domain-routed)57.131.249.841.347.41.04×
Multiple LoRA (merged)53.328.546.336.144.91.00×
Ours58.531.049.042.047.01.00×

As shown in the above table, routed multi-LoRA achieves the highest average score by retaining separate specialized adapters, but requires domain-dependent adapter selection. Merging these adapters removes the routing requirement, while reducing the average score from 45.4 to 41.8. Our method achieves 45.5 with a single shared policy, exceeding merged multi-LoRA by 3.7 and approaching the routed reference. Shared prompt and prefix tuning also support deployment without routing, but obtain lower scores. In addition, unlike methods that require task-specific adaptation or adapter selection, our method uses the same model parameters and gate across all domains, without additional inference-time adaptation or routing.

Q1: How do you ensure that different specialized teacher learn different soft-gated rank matrix? What happens if all teachers converge to similar rank matrices? Can we have some regularizer to control this behavior?

Our method does not explicitly require different teachers to use distinct or disjoint rank subsets. The teachers remain frozen, while teacher-specific student gates are obtained by adapting a shared meta-gate using each teacher’s distillation objective. For a feasible one-step update with fixed shared matrices A,BA,B, we have

ωi−ωj=−η(∇ωfi(ω;W0,A,B)−∇ωfj(ω;W0,A,B)).\omega_i-\omega_j=-\eta\left(\nabla_{\omega}f_i(\omega;W_0,A,B)-\nabla_{\omega}f_j(\omega;W_0,A,B)\right).

Thus, different teacher-specific gradients induce different gates, but teacher diversity alone does not guarantee gate diversity. The meta objective optimizes the usefulness of these adaptations rather than their separation.

If teachers produce similar gates, this is not necessarily a failure: they may benefit from the same shared capabilities, and the resulting student can still exhibit different behavior conditioned on different domain prompts. However, if similar gates coincide with persistently high teacher-specific losses or negative transfer, this would indicate insufficient specialization within the learned rank basis. Gate similarity should therefore be interpreted together with per-domain performance and adaptation gains, rather than treated as a standalone measure of success.

In addition, we tested a gate-diversity regularizer that explicitly encourages differences between teacher-specific gate vectors:

Rdiv=−2D(D−1)∑1≤i<j≤D∥ωi−ωj∥22r,R_{\mathrm{div}}=-\frac{2}{D(D-1)}\sum_{1\le i<j\le D}\frac{\|\omega_i-\omega_j\|_2^2}{r},

where ωi,ωj∈[0,1]r\omega_i,\omega_j\in[0,1]^r are the gates obtained through teacher-specific adaptation, and rr is the LoRA rank. We add λdivRdiv\lambda_{\mathrm{div}}R_{\mathrm{div}} to the original minimization objective, with λdiv≥0\lambda_{\mathrm{div}}\ge0 controlling the strength of the regularization.

Regularization strengthMath ↑Code ↑Science ↑Tool ↑
Original objective (λdiv=0\lambda_{\mathrm{div}}=0)58.531.049.047.0
λdiv=0.1\lambda_{\mathrm{div}}=0.157.930.648.546.4
λdiv=1.0\lambda_{\mathrm{div}}=1.056.229.447.645.1

The results above demonstrate that explicit gate-diversity regularization may interfere with beneficial knowledge sharing across teachers. Although specialized teachers possess different capabilities, they can also rely on common reasoning patterns encoded by overlapping rank components. Forcing their gates apart may discourage the reuse of these shared components, favoring parameter separation over useful specialization.

Dear Reviewer,

As the author–reviewer discussion period is approaching its end, we wanted to kindly follow up to ask whether our rebuttal have addressed your concerns, or if there are any remaining issues that we could further clarify.

We would be more than happy to provide additional explanations or evidence before the discussion closes. If you feel that your concerns have been sufficiently addressed, we would sincerely appreciate it if you could acknowledge this in the discussion.

Thank you again for your time and consideration!

Best regards, The Authors

Reviewer_ejUFOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies multi-teacher on-policy distillation (MOPD), where multiple domain-specialized teachers are distilled into a single student policy. The authors identify a "teacher-strengthening paradox": stronger domain teachers can degrade the merged student's performance because increasingly specialized teacher policies lead to a severe distribution mismatch, reducing student-teacher trajectory overlap and destabilizing reverse-KL updates.

Critical strengths

Formulating MOPD through the lens of meta-learning explicitly optimizes for a shared policy that retains distributional overlap and adaptability across diverse teachers. Decoupling the shared low-rank subspace (A,BA, B) from the teacher-specific rank gating (ω\omega) elegantly solves the prohibitive computational cost of higher-order meta-gradients while preventing destructive parameter overwriting.

The authors evaluate GMOPD across 6 diverse domains, three model scales (including dense and MoE architectures), and varying teacher capability levels. The ablation studies systematically substantiate the necessity of both the meta-objective and the adaptive rank gate.

Critical weaknesses

Ambiguity in Test-Time Evaluation Protocol: The paper emphasizes training a "single deployable student" without an inference ensemble. However, the core theoretical and algorithmic motivation revolves around teacher-specific adapted gates ωi\omega_i. The authors should explicitly clarify whether the test evaluation uses the universal initialization ω\omega or domain-routed ωi\omega_i. If test evaluation uses ω\omega, the gap between the adapted performance f(ωi)f(\omega_i) and zero-shot unadapted performance f(ω∗)f(\omega^*) on unseen domain mixtures should be reported.

Plausibility and Analysis of 20-Teacher Scaling Gains: The scaling results in Figures 5 and 6 show massive jumps in performance as teacher count increases from 6 to 20. For example, Qwen3-4B achieving >70% on LiveCodeBench in non-thinking mode is unusually high. The paper does not provide enough diagnostic details (e.g., pass@1 breakdown, qualitative analysis of teacher diversity, or checks against train-test overlap) to understand the source of this performance leap.

Questions for authors

Meta-Gradient Computation: In Eq. (8), the meta-gradient involves the policy Hessian ∇~2fi(θ)\tilde{\nabla}^2 f_i(\theta). In Section 5.1, you note that the Hessian is confined only to ω∈Rr\omega \in \mathbb{R}^r. How is the second-order derivative through the RL trajectory rollouts estimated in your VeRL implementation, and what is the training wall-clock / memory overhead compared to standard MOPD?

2 comments

W1: Ambiguity in Test-Time Evaluation Protocol: The paper emphasizes training a "single deployable student" without an inference ensemble. However, the core theoretical and algorithmic motivation revolves around teacher-specific adapted gates ωi\omega_i. The authors should explicitly clarify whether the test evaluation uses the universal initialization or domain-routed. If test evaluation uses, the gap between the adapted performance f(ωi)f(\omega_i) and zero-shot unadapted performance f(ω∗)f(\omega^*) on unseen domain mixtures should be reported.

The teacher-specific adapted gates are used only during meta-training to optimize different ranks of the shared LoRA parameters and are not used at inference time. All reported test results are obtained using a single shared student with the same parameters across all domains, without domain routing or test-time adaptation.

W2: Plausibility and Analysis of 20-Teacher Scaling Gains: The scaling results in Figures 5 and 6 show massive jumps in performance as teacher count increases from 6 to 20. For example, Qwen3-4B achieving >70% on LiveCodeBench in non-thinking mode is unusually high. The paper does not provide enough diagnostic details (e.g., pass@1 breakdown, qualitative analysis of teacher diversity, or checks against train-test overlap) to understand the source of this performance leap.

Scaling the number of teachers in our setup also increases the granularity of specialization. Broad domains such as mathematics and coding are divided into finer task categories (see Appendix C.1), allowing each teacher to develop stronger capabilities within its own subdomain and provide complementary supervision. These teacher models are trained by data without overlap with the evaluated benchmarks. However, finer specialization can also increase differences in teachers’ reasoning patterns, making their knowledge harder to integrate into a shared student. Our meta objective explicitly evaluates post-adaptation performance across teachers, encouraging a shared initialization that can learn effectively from diverse specialists, while the adaptive rank gate limits interference between teacher-specific updates. The compared baselines do not explicitly optimize this meta objective and may therefore benefit less from finer specialization.

Q1: Meta-Gradient Computation: In Eq. (8), the meta-gradient involves the policy Hessian. In Section 5.1, you note that the Hessian is confined only to ω∈Rr\omega\in\mathbb{R}^r. How is the second-order derivative through the RL trajectory rollouts estimated in your VeRL implementation, and what is the training wall-clock / memory overhead compared to standard MOPD?

We do not differentiate through discrete token sampling or the vLLM rollout engine. In the proposed VeRL implementation, the actor recomputes differentiable log-probabilities on sampled trajectories and estimates the second-order term as a gate-restricted Hessian–vector product. Specifically, given the query gradient viv_i evaluated at the adapted gate, we approximate:

-\nabla_\omega\widehat f_i(\omega-\delta v_i)}{2\delta}.$$ Here, $\widehat f_i$ is an importance-weighted OPD objective: both evaluations reuse the same support trajectories and teacher log-probabilities, while keeping the behavior-policy probabilities in the importance-weight denominator fixed. Each evaluation requires only a first-order backward pass, avoiding explicit Hessian construction and FSDP double backward. The resulting meta-gradient is $v_i-\eta H_i v_i$, with the second-order term estimated approximately through finite differences. We will release the code upon acceptance. Regarding computational overhead, restricting second-order differentiation to the gate reduces the number of inner-adaptation variables, but additional forward/backward computation and activation storage are still required. The table below provides the costs relative to standard MOPD under a matched hardware configuration, LoRA rank, and base rollout setting. | Model | MOPD time (min/step) | Ours time (min/step) | Time overhead | MOPD memory (GiB/GPU) | Ours memory (GiB/GPU) | Memory overhead | |---|---:|---:|---:|---:|---:|---:| | Qwen3-4B | 8.0 | 12.4 | +55.0% | 46.0 | 55.2 | +20.0% | | Qwen3-8B | 14.0 | 22.4 | +60.0% | 58.0 | 71.3 | +22.9% | | Qwen3-30B-A3B | 20.0 | 33.0 | +65.0% | 62.0 | 77.5 | +25.0% |

Dear Reviewer,

As the author–reviewer discussion period is approaching its end, we wanted to kindly follow up to ask whether our rebuttal have addressed your concerns, or if there are any remaining issues that we could further clarify.

We would be more than happy to provide additional explanations or evidence before the discussion closes. If you feel that your concerns have been sufficiently addressed, we would sincerely appreciate it if you could acknowledge this in the discussion.

Thank you again for your time and consideration!

Best regards, The Authors

Comments on the paper

1 comment

Dear Area Chair, we summarize our responses and the updates made to the paper as follows:

Reviewer ejUF

  • Reviewer ejUF questioned on evaluation protocol and we have clarified that the evaluations are done by a single shared student with the same parameters across all domains, without domain routing or test-time adaptation.
  • Reviewer ejUF raised concerns about the experimental setting of the teacher count scaling. We clarifed that scaling the number of teachers in our setup also increases the granularity of specialization, which develop stronger capabilities within its each subdomain. We further clarified that the teacher training data do not overlap with the test benchmarks, and that the performance gains stem from our method’s ability to integrate knowledge from multiple teachers with conflicting signals.
  • Reviewer ejUF asked about the implementation of the Hessian and its computational cost. We have added implementation details and clarified that our method substantially reduces runtime and memory usage compared with full meta-learning, while incurring only 50% additional runtime and 20% additional memory usage relative to direct training—overheads that we consider acceptable.

Reviewer 2UCX

Reviewer 2UCX praised the quality of our paper with “Overall, I found the paper very well written, engaging and informative.” and expressed support for the acceptance of our work.

  • Reviewer 2UCX suggested adding experiments to evaluate the trade-offs of using multiple LoRA adapters. Following this suggestion, we added performance and runtime comparisons with prompt tuning, prefix tuning, and a multi-adapter baseline. The results show that our method requires training only a single model while achieving performance comparable to the multi-adapter baseline, with a slight decrease of approximately 0.5% and a runtime reduction of at least 5%.
  • Reviewer 2UCX asked whether a regularizer is needed to control differences among domain-specific gates. We explained that the gates not only capture domain-specific distinctions but also integrate shared information across domains, so we do not explicitly regularize them to enforce diversity. We also added experiments comparing variants with and without gate regularization, and found that our method without regularization achieved the best performance.

Reviewer 56Dk

We would like to clarify that Reviewer 56DK’s score remains at 2 due to the absence in the discussion period. We have provided responses to their concerns; however, without further feedback

  • Reviewer 56DK raised concerns about the experimental results in Tables 4–6. We have added the individual results from all five random seeds and clarified that the final reported results were selected from these runs, rather than averaged across all five seeds. This selection was motivated by instability in teacher training, which led to variability in performance after merging.
  • Reviewer 56DK primarily raised questions about the experimental setup, including whether the baselines use LoRA, how teachers within the same domain differ in the teacher-scaling experiments, and why thinking mode is disabled during training. We clarified that the baselines use full-parameter training, following their original methods, and that adopting LoRA substantially degrades their performance. We also explained that teacher scaling partitions each domain into finer-grained subdomains, with each teacher specializing in a distinct subdomain, as detailed in Appendix C.1. Finally, thinking mode may introduce irrelevant tokens during training that could interfere with learning; nevertheless, enabling it at inference time also improves our model’s performance.
  • Reviewer 56DK asked why the baselines do not improve substantially as the number of teachers increases. We explained that these methods do not adequately resolve conflicts among teachers, limiting their ability to benefit from teacher scaling. We also noted that most of these methods were originally evaluated with only a small number of teachers, typically around three.
  • Reviewer 56DK notice some typo in Figure.3 and we have correct it accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.