acceptodds
Under review as a conference paper at ICLR 2027

Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation

Abstract

Multi-teacher on-policy distillation (MOPD) extends on-policy distillation (OPD) by routing domain-specific data to specialized teachers, integrating their capabilities through dense token-level supervision on student-generated trajectories. However, domain routing only distinguishes the supervision sources for different domains; updates from all domains still act on shared parameters. When teachers differ substantially in response style and domain expertise, supervision that is effective in isolation may still lead to persistent gradient conflicts and capability degradation during joint optimization. To address this challenge, we propose Cross-Teacher Guarding for MOPD (ROAD MOPD), which introduces non-routed teachers as auxiliary cross-domain critics on the same student trajectories and coordinates multi-teacher supervision through token-level credit assignment and gradient updates. Specifically, cross-teacher-guided credit assignment uses teacher agreement to strengthen token-level corrections jointly supported by the teachers and attenuate conflicting signals; preventive multi-teacher gradient projection uses auxiliary gradients from the same trajectories to constrain potentially conflicting updates and adaptively controls projection strength to retain target-domain learning signals. In experiments, ROAD MOPD mitigates degradation in the disadvantaged domain, gains over 5 percentage points under high conflict, and improves overall performance in low-conflict and three-domain settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy distillation, Multi-teacher distillation, Gradient conflict mitigation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pGyfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies gradient interference in multi-teacher on-policy distillation (MOPD), showing that domain-routed supervision can still induce conflicting updates on shared student parameters and degrade capability in high-conflict domain combinations. It proposes ROAD MOPD, which uses non-routed teachers as same-trajectory critics for agreement-aware token reweighting and norm-constrained gradient projection, improving joint performance across math, code, and medical domains.

Critical strengths
  1. The paper identifies a concrete and practically relevant failure mode of domain-routed MOPD: routing supervision by domain does not prevent conflicting updates to shared student parameters.
  2. The proposed use of non-routed teachers as same-trajectory cross-domain critics is conceptually distinctive from both standard MOPD and conventional multi-task gradient surgery. And the two components of ROAD MOPD—agreement-aware token credit assignment and norm-constrained preventive gradient projection—are well aligned with the diagnosed token- and parameter-level interference.
  3. The empirical evaluation is broad and reasonably well controlled, covering high-conflict, low-conflict, multiple student sizes, and three-domain settings, together with relevant MOPD and PCGrad baselines. It also includes useful ablations and mechanistic probes, and provides detailed implementation and reproducibility information.
Critical weaknesses

W1. The central compatibility proxy is not sufficiently validated. The main conceptual concern is that ROAD MOPD relies on a strong but insufficiently validated assumption: a non-routed teacher’s preference or gradient on a target-domain trajectory is treated as a proxy for whether the corresponding update is compatible with the capability represented by that teacher.

W2. The evidence for update incompatibility as a general MOPD failure mode is somewhat narrow and potentially confounded. The paper motivates ROAD MOPD through a strong contrast between Math–Code and the two Medical-containing domain pairs. However, both high-conflict settings involve the Medical domain, while Math–Code exhibits very low conflict. This makes it difficult to determine whether the observed degradation is primarily caused by a general limitation of domain-routed MOPD or by properties specific to the Medical configuration.

W3. Several important design choices remain heuristic and are not sufficiently stress-tested The token-level mechanism uses

\tanh((\ell_t^r-\ell_t^{ref})/\tau) \tanh((\ell_t^a-\ell_t^{ref})/\tau)$$ followed by a linear mapping to $\lambda_t$, while the gradient-level method introduces a norm-retention floor $m$. These choices are intuitive, but the manuscript provides limited evidence that these particular formulations are important for capturing cross-domain compatibility rather than simply being reasonable design choices.
Questions for authors
  1. What is the justification for treating a non-routed teacher’s preference on a target-domain trajectory as a proxy for preserving that teacher’s own-domain capability?
  2. Why is the same-trajectory auxiliary gradient a more appropriate compatibility signal than the auxiliary teacher’s gradient on its own-domain trajectories?
  3. Could the authors clarify the rationale behind the specific token-level agreement function in Eq. (4) and its mapping to the reward coefficient in Eq. (5)?
  4. How sensitive are the results to the particular definition of sts_t, the mapping from sts_t to λt\lambda_t, and the retention threshold mm? Have the authors considered simpler alternatives such as sign agreement, raw reference-relative log-probability products, teacher-distribution divergence, or several values of mm?
Reviewer_tNCGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates gradient conflicts between supervision signals from different domains in Multi-Teacher On-Policy Distillation, which may lead to capability degradation. The authors propose ROAD MOPD, which adjusts token-level supervision strength through Cross-Teacher-Guided Credit Assignment and mitigates conflicts in parameter updates via Preventive Multi-Teacher Gradient Projection. Experiments across mathematics, code, and medical domains using two student models demonstrate that the proposed method effectively alleviates performance degradation in high-conflict settings and improves overall multi-domain distillation performance.

Critical strengths
  1. By analyzing gradient compatibility and performance changes across different domain combinations, the authors reveal the limitations of Domain-Routed Supervision in joint multi-domain optimization. This provides a valuable perspective on understanding negative transfer in MOPD.

  2. ROAD MOPD introduces Non-Routed Teachers as Cross-Domain Critics to coordinate supervision signals from different teachers at both the token and gradient levels. The method does not require additional Student Rollouts, and the experiments cover two student models, three domain combinations, and joint three-domain training. The results demonstrate that the proposed method significantly mitigates performance degradation in the Medical Domain under high-conflict settings, while the ablation studies further validate the effectiveness of its two core components.

Critical weaknesses
  1. The proposed method uses gradients produced by an Auxiliary Teacher on Target-Domain Trajectories as constraints to mitigate negative transfer across domains. However, such Same-Trajectory Gradient Conflict is not necessarily equivalent to gradient conflicts between different domains' training data. This relationship requires further validation.

  2. Although ROAD MOPD does not require additional Student Trajectories, it introduces additional Auxiliary Teacher Scoring and Gradient Computation. The paper lacks a quantitative analysis of training time, computational overhead, and cost-effectiveness.

Questions for authors
  1. Compared with Vanilla MOPD and other baselines, how much additional training time and computational overhead does ROAD MOPD incur?

  2. Could the authors further investigate the relationship between Same-Trajectory Auxiliary Gradient Conflict, actual Cross-Domain Gradient Conflict, and model performance degradation?

Reviewer_yZmEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies multi-teacher on-policy distillation (MOPD), where domain-specific data is routed to specialized teachers that provide dense token-level supervision on student-generated trajectories. The authors identify a failure mode of standard domain-routed MOPD: although routing assigns supervision sources per domain, updates from all domains still act on shared parameters, and when teachers differ substantially (e.g., Medical vs. Math/Code), joint optimization produces persistent gradient conflicts and asymmetric capability degradation (e.g., Vanilla MOPD lowers Medical accuracy below the untrained student's level). The proposed method, ROAD MOPD, introduces non-routed teachers as auxiliary cross-domain critics on the same student trajectories. It coordinates supervision through two mechanisms: (1) cross-teacher-guided credit assignment, which compares the routed and auxiliary teachers' token-level preferences relative to a frozen reference and modulates the per-token reward scale by their agreement; and (2) preventive multi-teacher gradient projection, which removes components of the target-domain gradient that conflict with the auxiliary teacher's same-trajectory gradient, subject to a norm-retention floor that preserves target-domain learning signals. Experiments cover 11 benchmarks across Math, Code, and Medical domains with Qwen3-4B and Qwen3-1.7B students. Under high conflict, ROAD MOPD recovers Medical accuracy by 5.6–8.8 points over Vanilla MOPD and achieves the best two-domain average in all four paired settings; it also improves low-conflict (Math×Code) and three-domain settings, with ablations and a PCGrad comparison supporting each component.

Critical strengths

1. A well-documented failure mode of domain-routed MOPD. Fig. 1 shows that Math×Code training averages only 2.2% conflicting layers while Medical×Math and Medical×Code average 36.6% and 45.2%, with Medical performance consistently degraded. This evidence reframes MOPD research from routing design toward update compatibility, and provides a clear, reproducible benchmark scenario (the Medical pairings) for future work. This is consequential because it explains why prior joint-learning results (e.g., Math+Code) may not generalize to arbitrary domain combinations, a point that matters for practitioners mixing domains.

2. A fresh use of non-routed teachers as same-trajectory critics, with empirical support for the mechanism. Using auxiliary teachers purely as compatibility signals (never as supervisors) is conceptually clean, and the paper provides mechanism-level evidence: teacher disagreement correlates with measured gradient conflict (Fig. 3), and the method recovers the disadvantaged Medical domain by 5.6–8.8 points over Vanilla MOPD, reaching or exceeding the single-domain OPD reference in all four paired settings. This is consequential because it shows the auxiliary-teacher signal is not merely heuristic but tracks a measurable property of the optimization.

Critical weaknesses

1. Unquantified computational overhead that scales with the number of teachers — the paper's central practical concern. Per optimizer step, ROAD MOPD requires K teacher forward passes and K student backward passes per domain (K² in total), i.e., roughly K× the cost of Vanilla MOPD, before accounting for the 8B Medical teacher scoring 4B/1.7B students and the per-tensor projection iterations. The appendix (Limitations) admits the overhead only qualitatively; no wall-clock time, GPU hours, throughput, or peak-memory comparisons are reported, and no cost-reduction study (e.g., projecting only high-conflict layers) is attempted. This is consequential because distillation methods are adopted primarily for training efficiency, and the overhead grows exactly in the multi-domain regime the method is designed for — the paper's headline gains (5.6–8.8 points on Medical) must be weighed against an unquantified multiple of training cost.

2. Gains outside the high-conflict Medical pairings are modest. In the low-conflict Math×Code setting, ROAD MOPD's overall average exceeds ExOPD by only 0.35 points; with the 1.7B student, its Math and Code averages trail ExOPD by 1.23 and 0.94 points, respectively, and its advantage comes from weighting domains equally. The method's benefit is thus concentrated in the specific failure scenario it was designed for. This is consequential because it limits the generality of the claimed improvement: for compatible domain combinations, much simpler baselines are competitive.

Questions for authors
  1. Please report the training cost of ROAD MOPD relative to Vanilla MOPD under matched hardware and batch settings: wall-clock time per step, total GPU hours, and peak memory, ideally broken down into auxiliary teacher scoring, auxiliary backward passes, and projection. How does the relative cost grow with the number of domains K?
  2. Is full per-step, all-tensor projection necessary? Since Fig. 1 shows conflicts concentrate in a subset of layers, did you try restricting projection to high-conflict layers or running compatibility checks periodically? What fraction of the final gains is retained under such cheaper variants? (Clarification request — the answer directly affects my assessment of the cost concern.)
  3. How sensitive are the results to the norm-retention floor m and the agreement modulation strength ρg? Please provide a sensitivity study or the range over which conclusions hold.
  4. In all pairings, Medical is the domain that degrades. Is this due to teacher quality, response-style differences, or weaker knowledge overlap? Any analysis disentangling these factors would clarify when the proposed guarding is needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.