acceptodds
Under review as a conference paper at ICLR 2027

Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation

14 views

Abstract

Multi-teacher on-policy distillation allows a student to learn from complementary specialists on its own trajectories. Domain-routed approaches, however, select one teacher per example and keep it fixed throughout the response. This design both depends on domain labels that mixed training corpora often lack and cannot adapt teacher selection when the expertise required changes within a trajectory. We observe that each specialist deviates more from a shared reference on in-domain prompts than on out-of-domain prompts, on average. Based on this observation, we propose TrustMOPD, which replaces example-level teacher selection with label-free, token-level supervision allocation. At each student-generated prefix, TrustMOPD measures this displacement in next-token preferences, calibrates its magnitude across teachers, and uses the resulting scores as proxies for local reliability to weight teacher-specific distillation losses. Evaluated across mathematics, code, and instruction following, TrustMOPD closes 91.5% and 98.0% of the overall-score gap between the initial student and oracle-routed teachers when trained on SingleCap and MultiCap, respectively, compared with 54.4% and 54.5% for the strongest label-free baseline in each setting. On SingleCap, it approaches label-based MOPD without using domain labels. Our code is available at https://anonymous.4open.science/r/TrustMOPD-F624.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Teacher On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_k2SyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TrustMOPD, a label-free method for allocating multi-teacher on-policy distillation supervision at individual student-generated prefixes. Each specialist's centered log-probability displacement from its shared pre-RL reference is measured, divided by a frozen offline calibration mean, and converted into teacher weights through power normalization. Training minimizes a weighted reverse-KL objective, implemented through a geometric mixture of teacher distributions.

Critical strengths
  1. TrustMOPD allocates specialist supervision without domain labels, outcome verifiers, or a separately trained router. On SINGLECAP it nearly matches label-routed MOPD, with overall scores of 30.04 versus 30.18 (Table 1), making it a useful option when routing labels are unavailable.

  2. Measuring each specialist relative to its pre-RL reference is a useful way to expose specialization rather than relying only on teacher confidence. Offline calibration makes these signals comparable, and improvements persist with another teacher family and independently sourced prompts (Table 11).

Critical weaknesses
  1. A teacher that changes more from the reference is not necessarily giving better advice. The score measures the size of that change, not whether the teacher is correct at the current prefix. The theory does not establish this connection, and the domain-level results mainly show that the method recognizes specialization (Section 3.4; Figure 3). Appendix E gives two useful examples, but a larger correctness check is needed to show when the score selects the better teacher.

  2. The tested teacher pools are relatively structured, specialists share a reference and are trained for separate domains. Teacher scores are also normalized once using the initial student's rollouts (Sections 3.2 and 4.1). It is unclear how stable the weights remain when the calibration sample is unbalanced, teachers have overlapping skills, or the student's outputs change during training. The paper acknowledges the shared-reference requirement; the concern is how broadly the demonstrated routing gains apply.

Questions for authors
  1. When teachers disagree, how often does the higher-weight teacher give the correct guidance? Please check more than the two Appendix E examples.

  2. Do the weights and final scores remain stable with an unbalanced calibration sample or overlapping specialists? Do the fixed normalization constants still work on later student rollouts?

Reviewer_s2iiOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to distill complementary capabilities from multiple specialized LLMs into a single student model. Existing Multi-Teacher On-Policy Distillation (MOPD) methods typically rely on domain labels to select teachers. The authors propose TrustMOPD, which uses three domain-specific RL teachers derived from a shared reference model. The method computes teacher-reference displacement, calibrates the scores across teachers, and uses them for token-level weighted distillation.

Critical strengths
  • The paper identifies two important limitations of domain-routed MOPD: its dependence on domain labels and its inability to adapt teacher selection within a response.
  • The proposed method is simple and does not require training an additional router.
  • TrustMOPD outperforms the strongest label-free baselines by approximately 2–2.5 overall points on both training datasets.
  • The experiments are conducted with five random seeds, and standard deviations are reported.
Critical weaknesses
  1. The relationship between displacement and local reliability is not sufficiently justified. This is my main concern. The core assumption is that a teacher's displacement from the reference model reflects its local reliability. However, a larger policy change does not necessarily indicate more correct or useful supervision. I suggest the authors provide more direct evidence, including:

    • Quantitative correlation between displacement scores and local teacher correctness on verifiable tasks, rather than only domain alignment.
    • Whether displacement-based teacher rankings agree with rankings based on actual supervision quality at the same states.
    • Failure cases where large displacement corresponds to incorrect predictions, irrelevant behavior, or reward hacking.
    • Controlled experiments with weaker or deliberately perturbed teachers to test whether the method can avoid unreliable supervision.
  2. The evaluation does not directly demonstrate mixed-capability integration. The experiments show that training with mixed-capability data improves the student's average performance on separate math, code, and instruction-following benchmarks. However, this does not necessarily mean the student can effectively combine these capabilities within the same task. I suggest evaluating on mixed-capability benchmarks and reporting joint success rates.

  3. The benefit of token-level allocation is not fully convincing. On SingleCap, token-level and response-level allocation achieve nearly identical performance. On MultiCap, token-level allocation improves the overall score by 0.92 points, but the error bars in Figure 4 extend close to or across zero. The authors should provide statistical significance tests and stronger evidence that token-level allocation is necessary compared with simpler response-level alternatives.

  4. The theoretical contribution is relatively limited. While the theoretical results help explain the method, several conclusions follow directly from standard KL-regularized RL and algebraic properties of KL divergence. The analysis mainly provides an interpretation of the proposed displacement metric rather than new guarantees about teacher reliability or student performance.

Questions for authors

please see weakness

Reviewer_zcxEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the challenge of supervision allocation in multi-teacher distillation with a shared strategy, proposing a method called TrustMOPD. The core idea is to measure the shift in each teacher's predictive distribution relative to a shared reference model; this shift is then calibrated and converted into token-level weights, allowing the student model to dynamically adjust its reliance on different teachers during the generation process. A key contribution is the development of a teacher-weighting mechanism that requires neither domain labels nor an auxiliary trained router, supported by a theoretical rationale under ideal conditions. The authors evaluate the approach across mathematics, coding, and instruction-following tasks, using both single-capability and mixed-capability training data. The results demonstrate that the method outperforms unlabeled baselines and, in single-capability settings, approaches the performance of routing methods that rely on domain labels. These findings are further corroborated by ablation studies and experiments involving various models and data sources. However, while the current results confirm overall performance gains, they do not yet fully establish that the shift signal reliably reflects a teacher's local trustworthiness.

Critical strengths

The supervision allocation scheme is practical and does not rely on additional domain annotations. TrustMOPD dynamically adjusts teacher weights on a token-by-token basis during the student generation process; it requires no domain labels, separately trained routers, or external verifiers to assign supervision. This approach offers a streamlined method for integrating teachers from multiple domains, allowing the influence of each teacher to shift according to the content being generated—a capability particularly well-suited for scenarios where a single response demands a diverse range of skills.

The reported performance gains are robustly supported by experimental evidence. In both single-skill and mixed-skill training settings, this method outperforms the best-performing unlabeled baseline by an average of 2.09 and 2.45 percentage points, respectively. These findings are further reinforced by results across five different random seeds, as well as additional experiments involving variations in model backbones and training data sources.

Critical weaknesses

Interpreting distribution shifts as a proxy for local reliability lacks critical evidence. While the proposed method assigns weights based on the shifts—relative to a teacher’s reference model—a large shift could just as easily stem from a change in preferences or a confident, yet incorrect, response. Although the theory establishes a link between these shifts and the differences in the teachers' own utility functions under ideal conditions, it fails to demonstrate that such shifts can reliably identify which teacher is more accurate or helpful for a given generation step. Furthermore, overall performance gains and a handful of anecdotal examples are insufficient to validate this core hypothesis.

The experiments also fail to directly verify the collaboration of multiple capabilities within a single response. A key motivation of the paper is that the specific teacher capabilities required may evolve throughout the generation process; however, while the training set includes tasks that demand a mix of skills, the primary evaluations assess mathematics, coding, and instruction-following in isolation. There is no assessment of whether a single response can effectively satisfy multiple, concurrent requirements. While the ablation studies on token-level and response-level weighting offer some support, they do not directly prove that dynamic allocation improves this type of joint performance. This gap leaves the central application claim unverified and limits my assessment of both the necessity of the proposed method and the significance of its contribution.

Questions for authors

Is there direct evidence that calibrated displacement scores can predict a teacher model’s local correctness—given the current prefix of student-generated tokens—and that they outperform simpler signals such as entropy or likelihood?

In evaluation settings where a single response is required to be both logically correct and compliant with specific instructional constraints, does token-level weighting—as opposed to response-level weighting—consistently improve the success rate of satisfying both requirements simultaneously?

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their responses to the reviews. Please go through their rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.

The discussion period ends on November 18.

Thanks,
Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.