acceptodds
Under review as a conference paper at ICLR 2027

MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation

Abstract

Multi-teacher on-policy distillation (MOPD) integrates specialized capabilities into a single student, but existing practice typically hard-routes each prompt to a domain-matched teacher for the entire rollout. This dependence on prompt-level domain labels restricts using unlabeled training mixtures and leaves complementary signals from other teachers unused. We introduce **MOPD-Router**, a framework that routes supervision over the full teacher pool at each student-generated token, without domain labels or training a separate routing model. Its plug-in interface supports different metrics for selecting and weighting teacher-specific OPD signals. Within this interface, we propose **ExpertAlign**, which scores each teacher by whether its correction to the student at the current token expresses the specialization that teacher acquired during post-training, and compare it against two reference metrics built on teacher confidence (Entropy) and teacher–student discrepancy (Novelty). Experiments on unlabeled and domain-labeled training mixtures under strong-to-weak and same-size distillation scenarios show that ExpertAlign achieves the strongest overall performance in all four settings. On unlabeled data, it improves the overall score by 5.88 (+12.3%) points over Mean aggregation; on domain-labeled data, it outperforms standard MOPD by 3.95 (+7.8%) points without using available domain labels. These results demonstrate token-level routing can exploit cross-domain complementary supervision, and reduce exclusive reliance on prompt-level domain assignment.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Multi-Teacher Distillation, Capacity Integration, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JpQdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the problem of Multi-teacher on policy distillation (MOPD) by proposing MOPD-Router, a dense token-level multi-teacher distillation method where student token level supervision is routed by a weighting metric to all available teacher models based on each teacher's alignment between specialization direction and the teacher's correction direction. Rather than the hard prompt-level domain teacher routing of existing MOPD methods, this paper's proposed method is supervised by a mixture of teacher experts at the token level, with teacher weighting for each student token calculated based on the teacher expert direction alignment to the correction direction. The paper's claimed contributions are its MOPD-Router plug-and-play method for multi-teacher expert selection during policy distillation, it's proposed ExpertAlign teacher specialization aware routing metric, and experimental results showing that the proposed method's cross-domain teacher signals provide the superior distillation performance compared to competing hard domain label MOPD methods.

Critical strengths

For strengths, the paper itself is well written, with the problem statement and proposed method clear and easy to understand. The authors highlight an important issue with existing hard-domain expert routing, where many tasks can require expertise in multiple domain categories that makes a single expert insufficient. Additionally, from the experimental results shown, the proposed MOPD-Router outperforms existing hard MOPD methods like standard MOPD and open-MOPD when averaged across all the selected evaluation datasets over the three different task categories of Math, Coding, and Instruction Following.

Critical weaknesses

For weaknesses, the paper suffers from two main weaknesses in my opinion.
The first weakness is the actual paper contributions. While authors claim the proposed MOPD-Router and ExpertAlign metric as main contributions of the paper, most of the method components and ideas are taken directly from existing literature with little innovations on top. For example, as mentioned in the paper introduction, dense token level student supervision is a standard operation in OPD. ExpertAlign itself uses the log probability delta between teacher and student/base model introduced in prior work. The main difference comes from using the delta as a routing signal instead of a distillation signal. This seems like an incremental work rather than novel method.
The second, more important, weakness is that the evaluation protocol and comparison fairness seems questionable to me. This paper's training and evaluation set differ from those in MOPD and open-MOPD, meaning all comparisons to these two methods are based on the authors' own implementation of these methods on their chosen training/evaluation set. A fair comparison would also involve comparing MOPD-Router on the training and evaluation set used the MOPD or open-MOPD papers (choose the one that's the strongest competitor). If this is not practical, then at least explain in the paper what steps you took to make sure your training and implementation of these methods are fair representation of these methods' capabilities.

Additionally, the paper emphasizes that MOPD-Router is a label-free method, however it relies on a shared base model for expert selection. That same base model could just be prompted to assign hard domain labels, so in practice the requirement is not that different from label-based routing. This also raises a question about using a separate llm for hard domain labels in the unlabeled setting when in theory the domain routing should actually align with the teacher's training distribution. This difference is reflected in the experimental results, where the gap between MOPD-Router and open-MOPD is much smaller when training domain labels are available. I'm also curious how much of the performance gain can be attributed to ExpertAlign being a direct estimate of the task alignment to each expert domain, vs gain from the soft multi-expert routing. An additional experiment training both open-MOPD and MOPD using hard prompt-level domain routing calculated from the ExpertAlign metric would answer this question.

Questions for authors

See the weaknesses section.

6 comments
AuthorsOpenReview ↗

Response to Reviewer JpQd (W1, W2)

Thank you for the constructive comments. The experiment you propose at the end is already in the paper (Table 4); we explain below.

W1. Contributions

First of all, we respectfully clarify that MOPD-Router is not a standard operation. Dense token-level supervision is indeed standard within OPD, but how that supervision is allocated among multiple teachers is not: standard MOPD pipelines assign one teacher per prompt from a domain label and keep it for the whole rollout.

To our knowledge, MOPD-Router is the first systematic study of token-level teacher routing in MOPD. It replaces prompt-level assignment with label-free, token-level routing over the full teacher pool, without training a separate router, and compares three metrics from three information sources under one interface, objective and budget, across labeled and unlabeled data and two student scales. Beyond the method, the study also shows that off-domain teachers supply complementary supervision, which lets a same-size student surpass its RL teachers on math and code.

ExpertAlign builds on, rather than reuses, the teacher–base delta. We refer to the idea from prior work [1, 2] that the teacher–base log-policy difference$e_i$ captures the specialization a teacher acquired in post-training. However, prior work uses it only as a distillation signal for a single teacher. We propose a student-dependent teaching vector did_i and use its alignment with the expertise vector as a token-level routing criterion across teachers. To show that the teaching vector is necessary, we have added an new ablation that routes by the expertise vector eie_i alone. On the labeled set, ‖e_i‖-only routing reaches 37.73 / 52.67 Overall for the 1.7B / 4B students, against 40.19 / 54.58 for ExpertAlign, and is lower in all three domains (math/code/if) at both scales.

We have revised the contribution statement in Section 1 and related work in Section 2.2 to draw these lines explicitly.


W2. Evaluation protocol and fairness

  • Benchmarks. Our evaluation suite contains the benchmarks used by Open-MOPD (AIME 2024/2025, LiveCodeBench, IFEval, IFBench) and overlaps largely with those of MOPD (AIME 2025, IFEval, IFBench). Our evaluation protocol therefore does not differ substantially from those of MOPD and Open-MOPD.
  • Training data and teachers. We use public, widely adopted resources as training data: DeepMath-103K, Eurus-RL-Code and HiR-16K, with the released ExOPD math and code teachers. HiR-16K [3] contains about 16K instruction-following prompts collected from MulDimIF, VerIF, IFTrain and Chatbot Arena. All of these datasets are well known and widely used in prior RL and OPD work.

In addition, we take the following measures to ensure a fair comparison across methods:

  1. All methods share the objective, teacher pool, rollout configuration, optimization budget and hyperparameters, and differ only in routing (Appendix B, Table 5).

  2. For Open-MOPD, we use its officially released training framework with our training data.

  3. To give the label-based baselines their strongest form, we evaluate them in two training-data settings: a domain-labeled set, where they use the true domain labels, and an unlabeled set, where every example is labeled by a strong LLM, DeepSeek-V4-Flash, with labels validated against human annotation.

AuthorsOpenReview ↗

Response to Reviewer JpQd (W3)

W3. Prompted labels vs. label-free routing, and the source of the gain

Why not prompt a model for domain labels? Thank the reviewer for such a valuable suggestion, and we do evaluate this option, with a much stronger labeler than the 4B base. On the unlabeled set, DeepSeek-V4-Flash labels each example from its prompt and reference answer, agreeing with human labels on 87.3% of a validation sample in Appendix C. Standard MOPD and Open-MOPD trained with these labels still trail MOPD-Router with ExpertAlign by 2.25–5.00 Overall, according to Table 8. Moreover, any prompted label assigns one teacher to the entire rollout, which is exactly the limitation MOPD-Router removes.

Domain estimate vs. token-level multi-teacher routing. We understand reviewer's question as whether ExpertAlign helps mainly by being an accurate estimate of each prompt's domain, or by softly mixing several teachers. We have conducted the corresponding ablation experiments on the labeled set, and the results are shown in Table 4:

VariantWhat it keepsOverall-1.7BOverall-4B
MOPD-Router w/ ExpertAlignToken-level alignment over the full pool40.1954.58
w/o Token-Level RoutingExpertAlign as a per-response domain estimate36.5752.61
w/o Cross-Domain TeachersToken-level alignment, labeled-domain teacher only37.7951.83
Mean AggregationAll teachers by average35.7848.16
Standard MOPDTrue hard domain labels37.5250.63
Open-MOPDTrue hard domain labels39.0852.37

According to these results, neither factor alone explains the gain. Used only as a domain estimate, ExpertAlign performs at the level of label-based routing. Mixing all teachers softly without the metric (Mean) is worse still. Removing either token-level allocation or the off-domain teachers costs 1.97–3.62 Overall. The gain therefore comes from applying the alignment estimate at every token over the full teacher pool. The experiment of hard prompt-level labels derived from ExpertAlign, is an offline version of the second row; at best it recovers the true domain labels used by Standard MOPD and Open-MOPD, which still perform worse than the full MOPD-Router with ExpertAlign.

In summary, these controlled comparisons show that the improvement of MOPD-Router does not come from better domain labels or from soft mixing alone. It comes from routing every token over the full teacher pool with metric ExpertAlign. If we have misunderstood your question, please let us know; we would be glad to discuss it further and address any remaining concerns.


We sincerely appreciate the reviewer's constructive suggestions and believe that the additional explanations significantly improve the quality of our submission. We hope that this provides sufficient reasons to raise the score.

References

[1] Heo, Byeongho, Jaehui Hwang, Sangdoo Yun, Dongyoon Han. On-Policy Delta Distillation. arXiv:2607.15161, arXiv, 2026.

[2] Feng, Shiyuan, Huan-ang Gao, Haohan Chi, Hanlin Wu, Zhilong Zhang, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou. Weak-to-Strong Generalization via Direct On-Policy Distillation. arXiv:2607.05394, arXiv, 2026.

[3] Zhang, Kongcheng, Qi Yao, Shunyu Liu, Wenjian Zhang, Min Cen, Yang Zhou, Wenkai Fang, Yiru Zhao, Baisheng Lai, Mingli Song. Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Following. arXiv:2512.23457, arXiv, 2025.

Thank you for the detailed response to my review. Your response and additional experiments, has partially addressed my questions regarding the proposed method. However, the main two weaknesses identified in my review are still mostly unaddressed or deflected. Regarding novelty, the paper "H-OPD: Confidence Aware Heterogeneous Multi-Teacher Multimodal On-policy Distillation" already proposes token level arbitration for MOPD. While I agree that the ExpertAlign metric itself builds on the teacher-base delta idea, this contribution alone is incremental rather than novel. The most important weakness I identified, comparison fairness, has not been sufficiently addressed in the response. My concern is not whether the evaluation benchmarks are reasonable for the paper's task, but rather whether the existing competitor methods have been adequately represented in the evaluation. A method’s published training configuration is tuned for its own setup, including the student, teacher pool, data and objective. Changing any of these requires retuning. The paper changes all of them for the competing methods and then applies a single shared configuration, while the only reported method-specific tuning (top-k for ExpertAlign, selected on the evaluation benchmarks; Appendix F.4) favors the proposed method. A fair comparison requires either evaluating on a baseline’s published setup, where its configuration and reported numbers apply, or tuning every method with an equal budget on a held-out validation set in the new setup. Where this is impractical, a more accurate framing would present the experiments as controlled comparisons against routing strategies inspired by prior work, e.g., “prompt-level hard routing (MOPD-style)” and “domain budget balancing (Open-MOPD-style, re-implemented)”, rather than attributing the results to the original methods. Note that such reframing would only make the claims accurate; it would not address the underlying weakness. Without a comparison that fairly represents the prior methods, either on their published setups or with equal tuning effort, the paper cannot establish improvement over existing work, and my assessment would remain the same.

Reviewer_32tPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MOPD-Router, a token-level teacher routing framework for MOPD. Unlike conventional MOPD approaches that assign a single teacher to each prompt based on domain labels, MOPD-Router dynamically selects and weights supervision signals from multiple teachers at each token position without requiring domain labels or an additional routing model. The authors investigate three routing metrics, including Entropy, Novelty, and ExpertAlign. In particular, ExpertAlign determines teacher relevance by measuring the alignment between the teacher's specialization relative to its pre-training base model and its supervision direction relative to the student.

Critical strengths
  1. Existing MOPD methods commonly rely on prompt-level domain assignments, whereas the proposed framework investigates token-level teacher routing without explicit domain labels.
  2. The experiments consider both strong-to-weak and same-size distillation, along with labeled and unlabeled training mixtures.
Critical weaknesses
  1. MOPD-Router primarily replaces prompt-level teacher assignment with token-level adaptive weighting, while ExpertAlign relies on cosine similarity between teacher specialization and supervision directions. These techniques are closely related to existing adaptive distillation and expert routing approaches. The paper should better clarify its methodological novelty over prior work.
  2. ExpertAlign assumes that teacher–base model differences represent acquired expertise and that positive alignment with teacher–student corrections indicates useful supervision. However, these assumptions remain largely heuristic. A more rigorous analysis is needed to establish why the proposed metric reliably identifies beneficial teacher signals.
  3. ExpertAlign assumes that teacher–base model differences represent acquired expertise and that positive alignment with teacher–student corrections indicates useful supervision. However, these assumptions remain largely heuristic. A more rigorous analysis is needed to establish why the proposed metric reliably identifies beneficial teacher signals.
  4. The paper compares against Standard MOPD, Open-MOPD, Mean Aggregation, and two alternative routing metrics, but lacks comparisons with stronger adaptive routing approaches, such as learned routers and similarity-based teacher selection. These comparisons are necessary to establish whether ExpertAlign provides advantages beyond general adaptive teacher weighting.
  5. The paper does not adequately investigate how ExpertAlign behaves when teachers provide conflicting or unreliable supervision. Positive alignment does not necessarily guarantee accurate or beneficial corrections. Experiments involving weaker teachers, conflicting specializations, and imbalanced teacher pools would help identify the limitations of the routing strategy.
Questions for authors

Please see the weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 32tP

Thank you for the constructive comments. We address these points in order.

W1. Novelty over adaptive distillation and expert routing

  • Framework. To our knowledge, MOPD-Router is the first systematic study of token-level teacher routing in Multi-teacher On-Policy Distillation. It puts metrics from three information sources (teacher only, teacher–student, teacher–student–base) behind one plug-in interface, under the same objective and budget, on labeled and unlabeled data and two student scales. Adaptive OPD methods reweight supervision from a single teacher; they do not allocate it among specialists.
  • Metric. We refer to the idea from prior work [1, 2] that the teacher–base log-policy difference eie_i captures the specialization a teacher acquired in post-training. However, prior work uses it only as a distillation signal for a single teacher. We propose a student-dependent teaching vector did_i and use its alignment with the expertise vector as a token-level routing criterion across teachers. To show that the teaching vector is necessary, we have added a new ablation in the revised version that routes by the expertise vector eie_i alone. On the labeled set, ‖e_i‖-only routing reaches 37.73 / 52.67 Overall for the 1.7B / 4B students, against 40.19 / 54.58 for ExpertAlign, and is lower in all three domains (math/code/if) at both scales.

We have revised the contribution statement in Section 1 and related work in Section 2.2 to draw these lines explicitly.

W2, 3. Why does the metric identify beneficial signals?

The new version of Appendix F.6 provides a theoretical analysis of what ExpertAlign measures. Every teacher's correction decomposes as di=ei+g,g=log⁡pbase−log⁡pSd_i = e_i + g, g = \log p^{base} - \log p^S , where gg is the same for all teachers, so teachers differ only through their specializations. Three properties follow:

  • Training signal. With normalized weights, the routed update moves the student toward a log-linear pool pbase∏i(pi/pbase)wip^{base}\prod_i (p^i/p^{base})^{w_i}, so each routing weight is the fraction of a teacher's specialization that the target retains. Mean keeps only 1/M of a specialist's shift whenever the other teachers coincide with the base, which explains why it falls below Standard MOPD in all four settings.
  • Gate. ⟨ei,di⟩=αi∥ei∥2\langle e_i, d_i \rangle = \alpha_i \|e_i\|^2, where αi\alpha_i is the remaining gap between student and teacher along eie_i. The gate thus keeps teacher i only while its specialization has not yet been absorbed by the student.
  • Cosine. The part of did_i orthogonal to eie_i comes from the shared gg and carries nothing specific to teacher i. The cosine is therefore the share of the correction that expresses teacher i's own specialization, and weighting by it favors teachers with specific corrections over base-like ones.

ExpertAlign therefore places the routing weight at each token on the teachers whose specialization is relevant there and not yet absorbed by the student. We refer the reviewer to Appendix F.6 for the full analysis.

W4. Learned routers and similarity-based selection

  • Learned routers. A learned router needs supervision data. However, token-level routing labels do not exist, and obtaining them by human or LLM annotation at the token level is prohibitively expensive. A prompt-level router can at best recover the true domain label, which is covered by our experiments. Standard MOPD and Open-MOPD route with LLM-produced labels on the unlabeled set, and MOPD-Router with ExpertAlign exceeds them by 2.25–5.00 in Table 1.
  • Similarity-based selection. In our experiments, metric Novelty selects teachers by teacher–student distributional discrepancy and is already compared. We further added sampled-token log-prob routing (37.85 / 49.68 Overall for 1.7B / 4B on the labeled set), which, like the ‖e_i‖ routing above, stays below ExpertAlign. Together with Mean and Entropy, these results show that the gain comes from the choice of routing metric, rather than from adaptive weighting alone.

W5. Conflicting or unreliable teachers

Teacher reliability is an open problem for OPD and MOPD: it arises whenever a student learns from imperfect teachers, regardless of how their supervision is routed, and it is not introduced by token-level routing. Detecting conflicting or unreliable supervision is therefore a separate research question, complementary to our focus on how to route supervision among a given pool of specialized teachers. We have added this direction to the future work section.


We sincerely appreciate the reviewer's constructive suggestions and believe that the additional explanations significantly improve the quality of our submission. We hope that this provides sufficient reasons to raise the score.

References

[1] Heo, Byeongho, et al. On-Policy Delta Distillation. arXiv:2607.15161, 2026.

[2] Feng, Shiyuan, et al. Weak-to-Strong Generalization via Direct On-Policy Distillation. arXiv:2607.05394, 2026.

I appreciate the authors’ response and their answers to my questions. However, I still believe my initial overall rating reflects my assessment of the paper, and therefore I would like to keep it.

Reviewer_Bn3XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the problem of teacher routing in Multi-Teacher One-Policy Distillation (MOPD). While standard MOPD employs "hard routing"—assigning each prompt to a single domain-specific teacher based on prompt-level labels—MOPD-Router eliminates the need for such labels or learnable routing modules. Instead, for every token generated by the student, it dynamically reweights the contributions of all teachers in the pool based on the OPD advantage value At(i)A^{(i)}_t associated with each teacher’s sampled token.

Critical strengths
  1. There is empirical evidence suggesting that label-free, "token-level" selective routing outperforms "prompt-level" hard routing. The ablation study section is particularly valuable. A simple "full-pool averaging" approach underperforms compared to standard MOPD (see Table 2: 35.78 vs. 37.52, and 48.16 vs. 50.63), indicating that the mere inclusion of "off-domain teacher models" can actually have a negative impact. Table 3 examines "positive-alignment gating" (i.e., the Uniform strategy) and "cosine weighting" separately, showing that both contribute to performance gains. Table 4 demonstrates that, with a fixed gating mechanism, incorporating off-domain teacher models yields an improvement of 2.40/2.75 points; furthermore, the token-level allocation strategy outperforms a "response-level fixed mixture" strategy by 3.62/1.97 points, respectively. This finding is clear and highly relevant for practitioners: signals from off-domain teacher models are only truly effective when selection occurs at the token level. This is the primary reason I am inclined to accept this paper.

  2. This is a practical method backed by solid empirical evidence. It requires neither labels nor the training of a dedicated routing module. It only necessitates one additional forward pass through the shared base model (with an end-to-end overhead of approximately 1.18 times the original cost per token; see Table 14) and is applicable to unlabeled, mixed-data scenarios. In contrast, hard-routing strategies typically rely on large language models (LLMs) for annotation—a process that shows only 76% agreement with human annotation (see Table 7). Across all four experimental settings, and after applying the Holm correction, this method demonstrates statistically significant performance improvements over both standard MOPD and a simple averaging (Mean) strategy. In experiments using multiple random seeds, the performance gains proved highly robust (an improvement of +3.76 ± 0.32 over standard MOPD, with a standard deviation of 0.12 for ExpertAlign). The authors have also provided the accompanying code. Given that MOPD is already widely employed in the post-training pipelines of state-of-the-art models, this research holds significant practical value and is readily applicable in real-world settings.

Critical weaknesses
  1. The mechanism claimed by ExpertAlign—routing based on "specialized perception"—remains unsubstantiated; in fact, the paper’s own geometric derivations point toward a different explanation. According to Equation (14) in Appendix F.5, ⟨ei,t,di,t⟩=∣ei,t∣2(1−ri,t)\langle e_{i,t}, d_{i,t}\rangle = |e_{i,t}|^2(1-r_{i,t}), where ri,tr_{i,t} represents the projection of the student model onto the direction of teacher model ii’s "offset relative to the baseline model." Consequently, as long as the student model has not yet "overshot" teacher model ii along this offset direction, the gating mechanism will retain that teacher model. This is, in essence, a test of "residual progress" rather than an assessment of whether a given offset is actually useful for the current token. Two implications follow: In a "same-scale" setting, where the student model is initialized as the shared baseline model—meaning di,t=ei,td_{i,t} = e_{i,t} and the cosine similarity for all teacher models is 1—ExpertAlign effectively functions as "Mean Aggregation" at step 0. Any subsequent effects stem from the evolution (drift) of the student model itself, rather than from any initial "identification of specialization." In a "strong-to-weak" setting, the term qt=log⁡pS−log⁡pbaseq_t = \log p^{S} - \log p^{base} essentially compares a 1.7-billion-parameter (1.7B) model against a 4-billion-parameter (4B) baseline model. Since this metric conflates differences in model capacity with the "specialized knowledge acquired," the notion of "progress" it implies is ambiguous. An alternative, plausible interpretation is that ExpertAlign primarily serves to suppress "interference between teacher models." Given that di=ei−qd_i = e_i - q, where qq encapsulates the bias the student model has absorbed from other teacher models, the component of did_i orthogonal to eie_i largely represents an attempt by teacher model ii to counteract the contributions of other teacher models. The cosine-weighting mechanism effectively reduces the weight assigned to this component. This explains why "simple averaging" across the entire pool of teacher models can actually yield negative results. Furthermore, Figure 2 (left) does not support the "specialization" hypothesis: across various prompt domains—including mathematics and coding—the "Instruction-Following" (IF) teacher model receives the highest routing weights (40.9% for math prompts, compared to 37.4% for the math-specific teacher model; and 36.6% for coding prompts, compared to 34.4% for the coding-specific teacher model). This appears to be precisely the kind of "teacher-specific bias" that the paper critiques in its section on "Entropy." The "off-domain" proportions highlighted in the paper (exceeding 50% for every row) are, in fact, lower than the 2/3 proportion one would expect under uniform routing across three teacher models; thus, the chart fails to demonstrate an effective utilization of "off-domain" expertise. While the human-annotated rollout process shown in Figure 2 (right) and Figure 3 is illustrative, it represents only an isolated case. This is a critical point, as the concept of "specialization" serves as the core methodological claim of the paper. While empirical performance gains are indeed observed—regardless of the underlying mechanism—the value of the insights offered to the reader hinges on which mechanism is, in fact, the correct one.

  2. Due to the specific choices regarding baseline models and hyperparameter tuning, the extent to which ExpertAlign offers a truly unique advantage remains unclear. In the label-free, token-level comparison, the authors employ a "Mean" approach alongside two custom-designed metrics—one of which is explicitly characterized as a "naive" method. The paper overlooks several straightforward yet robust label-free alternatives, such as token selection or weighting based on the log-probabilities or advantage values ​​of sampled tokens under the teacher model, weighting by the magnitude of the baseline deviation ∣ei,t∣|e_{i,t}|, or the H-OPD confidence arbitration mechanism proposed in existing literature. Furthermore, in the "labeled 4B" setting (see Appendix F.4), the parameter k=16k=16 was selected based on performance on test benchmarks; notably, this is the only experimental configuration in the paper that utilizes multiple random seeds. Across the range of k=1k=1 to 1616, the overall performance metrics fall within overlapping error bars. Moreover, since the calculation method shifts to "unnormalized product weights" when k=1k=1 (see footnote 1), this parameter sweep does not constitute a pure ablation study of the kk value. The temperature parameters for Entropy and Novelty are fixed at 0.1, with no mention of any tuning process. In Table 8, the bootstrap confidence intervals (CIs) for Open-MOPD are approximately half the width of those for other methods, suggesting a potential inconsistency in the evaluation methodology. Finally, while the paper states that ExpertAlign "outperforms" Open-MOPD on labeled data, the difference is not statistically significant (pHolm=.208p_{Holm}=.208). Although Appendix E describes the two as "statistically comparable," a lack of statistical significance is not equivalent to equivalence between the methods. While these issues may be inconsequential when considered in isolation, taken together, they suggest that the advantage demonstrated by ExpertAlign over the strongest baseline model—which also utilizes labeled data (an improvement of 1.1 to 2.2)—is less certain than the core results in the paper might imply. Furthermore, it should be noted that all teacher models belong to the same family and share a common base model—a prerequisite for ExpertAlign. The authors have acknowledged this, so I do not view it as a critical issue.

Questions for authors

see weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer Bn3X (W1)

Thank you for the careful and constructive comments. We address each point below.

W1. What mechanism does ExpertAlign implement?

W1.1 "This is, in essence, a test of 'residual progress' rather than an assessment of whether a given offset is actually useful for the current token."

Residual progress and usefulness. We believe that in MOPD, residual progress and usefulness are, to some extent, the same thing, because the goal of MOPD is precisely to transfer each teacher's acquired specialization into the student. A teacher's supervision is therefore useful at a token exactly when its specialization there has not yet been absorbed by the student, and this is what ExpertAlign tests. The test is made per token: the expertise and teaching vectors are recomputed at every student-visited context, so a teacher is retained only at the positions where it still has specialization to transfer. Empirically, this criterion does select useful supervision: the gate alone improves Overall by 1.22–4.75 over Mean in all four settings according to Table 3.

W1.2 "An alternative, plausible interpretation is that ExpertAlign primarily serves to suppress 'interference between teacher models.'"

Interference suppression. This reading is consistent with ours, and Appendix F.6 in the revised version now makes it precise. With normalized weights, the routed update moves the student toward a log-linear pool of the teachers whose exponents are the routing weights. Mean gives every teacher exponent 1/M, so at a token where only one teacher is specialized, the others coincide with the base and pull the target back toward it, keeping only 1/M of the specialist's shift. ExpertAlign removes such teachers through the gate and down-weights base-like corrections through the cosine. This explains why Mean falls below Standard MOPD in all four settings. We thank the reviewer for this constructive comment.

W1.3 "In a 'same-scale' setting... ExpertAlign effectively functions as 'Mean Aggregation' at step 0."

Same-size at step 0. Reviewer's understanding is correct: q=0q = 0, every cosine is 1, ExpertAlign equals Mean at step 0, and this is an intended design. A student-conditioned metric has nothing to differentiate before the student moves, and differentiation starts with the first update. The equivalence with Mean does not persist: over training, a strict subset of teachers is selected on 48.8–63.0% of tokens (Appendix F.5), while Mean would keep all three.

W1.4 "Since this metric conflates differences in model capacity with the 'specialized knowledge acquired,' the notion of 'progress' it implies is ambiguous."

Strong-to-weak. We agree on the scope. The "progress" reading of rir_i in Appendix F.5 assumes that the student starts from the shared base, which holds only in the same-size setting. To address this, Appendix F.6 in the revised version gives a new theoretical analysis of ExpertAlign that holds for any student, including the strong-to-weak setting.

However, the routing itself remains well-founded in strong-to-weak distillation. The capacity gap is part of what the student should learn from a teacher, and the standard OPD signal (teacher minus student) already contains it under any routing rule. It also does not blur which teacher is selected: eie_i compares a 4B teacher with the 4B base and carries no capacity term. In short, the teaching vector carries both the capacity difference and the specialization difference. Both are valuable supervision, and absorbing either one is real progress for the student, so having them together in the teaching vector is appropriate.

W1.5 "Figure 2 (left) does not support the 'specialization' hypothesis: [...] the 'Instruction-Following' (IF) teacher model receives the highest routing weights [...] the chart fails to demonstrate an effective utilization of 'off-domain' expertise. [...]"

Figure 2 (left) is a descriptive view of how routing mass is distributed. It is not our evidence that off-domain supervision is used effectively: the share of mass a teacher receives does not by itself say whether that supervision helps, so neither "above 50%" nor "below the uniform 2/3" settles the question. What the matrix does show is that each teacher receives its largest share on its own domain, without labels and without collapsing to hard routing. The actual evidence for cross-domain value is the controlled ablation in Table 4. Removing off-domain teachers costs 2.40 / 2.75 Overall, and freezing the same scores at response level costs 3.62 / 1.97.

We have revised the description of Figure 2 in Section 4.3.4 accordingly. The IF teacher's large share reflects that format and constraint handling is needed in every response. Its share still varies with domain and position, unlike teacher-specific bias, where the math teacher is top-1 on about 70% of tokens for both students throughout training (Table 10).

AuthorsOpenReview ↗

Response to Reviewer Bn3X (W2)

W2. Baselines, tuning and statistics

W2.1 "The paper overlooks several straightforward yet robust label-free alternatives..."

Other baselines. We added the suggested token-level baselines on the labeled set under the identical pipeline and budget: routing by the teacher's log-probability of the sampled token, and routing by the specialization magnitude. Results on the unlabeled set will be also added in the revised version.

StudentRouting metricMathCodeIFOverall
1.7BStandard MOPD25.0050.0243.8037.52
1.7BExpertAlign26.9853.5646.5640.19
1.7BSampled-token log-prob (added)21.7752.8847.4637.85
1.7BSpecialization vector ‖e_i‖ (added)24.5950.2345.2837.73
4BStandard MOPD42.9260.4651.3150.63
4BExpertAlign46.9864.2255.3054.58
4BSampled-token log-prob (added)39.0661.4853.2449.68
4BSpecialization vector ‖e_i‖ (added)45.3863.0451.7252.67

Both baselines remain below ExpertAlign at both scales. Log-prob routing matches Standard MOPD at 1.7B and falls below it at 4B, consistent with our finding that teacher confidence is not a reliable routing signal in Section 4.3.1. ‖e_i‖ routing is lower than ExpertAlign in all three domains at both scales, showing that knowing where a teacher is specialized is not sufficient without the student-dependent teaching vector. Confidence-based arbitration is also covered by our Entropy metric (Section 3.3) and its calibrated variant (Appendix F.1).

W2.2 *"The parameter

k=16k = 16

was selected based on performance on test benchmarks [...] this parameter sweep does not constitute a pure ablation study of the value."*

Choice of k. k=16k = 16 was chosen once, on the labeled 4B setting, and reused unchanged in the other three settings, where ExpertAlign is also best. The conclusion does not depend on it: every k≤16k \le 16 gives an Overall of about 53.3–54.6, above Standard MOPD (50.63) and Open-MOPD (52.37). The k=1k = 1 variant (Appendix F.4, footnote 1) is necessary, because a one-dimensional cosine is ±1 and would discard the alignment magnitude.

W2.3 "The temperature parameters for Entropy and Novelty are fixed at 0.1, with no mention of any tuning process."

Temperatures. Both routing temperatures were fixed at 0.1 a priori for all settings and were not tuned on any benchmark. ExpertAlign has no temperature.

W2.4 "In Table 8, the bootstrap confidence intervals (CIs) for Open-MOPD are approximately half the width of those for other methods, suggesting a potential inconsistency in the evaluation methodology."

Table 8 interval widths. We re-checked the evaluation code: all four methods use the same evaluation pipeline, decoding parameters, benchmark items and bootstrap procedure. The narrower intervals of Open-MOPD are the normal outcome of this aligned evaluation, and the intervals in Table 8 are correct as reported.

W2.5 "A lack of statistical significance is not equivalent to equivalence between the methods."

Significance and equivalence. We agree that a lack of statistical significance does not establish equivalence, and we do not intend to claim equivalence between these two methods. Our claim is that label-free ExpertAlign achieves higher Overall scores than label-using Open-MOPD in all four settings, with significant gains in three of them. This claim is supported by multi-seed experiments: across three training seeds on labeled 4B, the gain is 1.88 ± 0.31, and holds for every seed. We have revised the corresponding statements in Section 4.2 and Appendix E to state this claim precisely.

W2.6 "All teacher models belong to the same family and share a common base model—a prerequisite for ExpertAlign."

Shared base. Thank you for noting and understanding this. A shared vocabulary and family is already required by on-policy distillation itself, since teacher and student log-probabilities are compared on the same tokens, and teachers post-trained from a common base are the standard industrial setup. It is therefore a standard requirement of MOPD rather than an additional assumption introduced by our method. ExpertAlign adds only a forward pass of the base model (1.18× token-normalized cost).


We sincerely appreciate the reviewer's constructive suggestions and believe that the additional explanations significantly improve the quality of our submission. We hope that this provides sufficient reasons to raise the score.

Reviewer_Bn3XOpenReview ↗

Replying to authors

Thank you for the detailed rebuttal and additional experiments.

While I appreciate the authors' efforts, my main concern regarding the effectiveness of the proposed teacher selection mechanism remains unresolved. Therefore, I will maintain my original rating.

Thank you again for your responses.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers and the AC

We thank all reviewers for their time and constructive comments. We have uploaded a revised version of the paper, with all changes marked in blue. Below we summarize our contributions, the additions made during the rebuttal, and our responses to shared questions.

1. Contributions

  • MOPD-Router. To our best knowledge, this work is the first systematic study of token-level teacher routing in MOPD. MOPD-Router replaces prompt-level hard routing with label-free, token-level routing over the full teacher pool, without a separate router, and compares metrics from three information sources under one interface, objective and budget.
  • ExpertAlign. We propose a student-dependent teaching vector and route each token by its alignment with the teacher's expertise vector. Without domain labels, ExpertAlign achieves the best Overall score in all four settings (two data regimes × two distillation scenarios).
  • Complementary cross-domain supervision. Off-domain teachers provide complementary supervision: removing them, or fixing the routing at the response level, costs 1.97–3.62 Overall (Table 4), and the same-size student surpasses its RL teachers on math and code.

2. Additions in the revised version

  • Theoretical analysis (Appendix F.6). We added a new analysis, valid for any student, explains what the ExpertAlign gate and cosine weight measure, and why they identify the supervision that the student has not yet absorbed.
  • Two additional label-free token-level baselines (Appendix F.7), in all four settings. TokenProb routes by the teacher's log-probability of the sampled token, and ExpertNorm routes by the expertise magnitude ∥ei∥\|e_i\| without the teaching vector.
OverallUnlabeled 1.7BUnlabeled 4BLabeled 1.7BLabeled 4B
Standard MOPD36.3348.7637.5250.63
MOPD-Router w/ TokenProb34.8746.6837.8549.68
MOPD-Router w/ ExpertNorm36.6652.3537.7352.67
MOPD-Router w/ ExpertAlign38.5853.7640.1954.58

TokenProb stays at the level of Standard MOPD or below it, consistent with our finding that teacher confidence is not a reliable routing signal (Section 4.3.1). ExpertNorm improves over Standard MOPD, confirming that the expertise vector is informative, yet remains 1.41–2.46 points below ExpertAlign in every setting, showing that the student-dependent teaching vector is necessary.

  • Clarifications in the main text. We revised the contribution statement and related work (Sections 1 and 2.2), the description of Figure 2 (Section 4.3.4), the significance statements (Section 4.2 and Appendix E), the description of the Open-MOPD implementation (Section 4.1), and added teacher reliability as a future direction (Appendix A).

3. Responses to common questions

  • Novelty over adaptive OPD and the teacher–base delta (32tP, JpQd). Adaptive OPD methods reweight supervision from a single teacher, whereas MOPD-Router allocates supervision among multiple specialists at every token. Prior work uses the teacher–base delta only as a single-teacher distillation signal; ExpertAlign couples it with the student-dependent teaching vector to decide which teachers supervise each token.
  • Why ExpertAlign selects useful supervision, including in strong-to-weak distillation (Bn3X, 32tP). The goal of MOPD is to transfer each teacher's specialization into the student, so a teacher's supervision is useful at a token to the extent that its specialization there has not yet been absorbed, which is what the gate tests. The capacity gap lies in the component shared by all teachers and is transmitted by the standard OPD signal under any routing rule; it does not affect which teacher is selected, since eie_i compares each teacher with the shared base (Appendix F.6).
  • Fairness of comparison (JpQd, Bn3X). All methods share the objective, teacher pool, rollout configuration and optimization budget; Open-MOPD is trained with its officially released framework. Label-based baselines use the true domain labels on the labeled set and human-validated LLM labels on the unlabeled set (Appendix C). The evaluation pipeline, decoding parameters, benchmark items and bootstrap procedure are identical across methods (Appendix E).

We hope these updates address the reviewers' concerns. Since the discussion period is limited, we would greatly appreciate it if the remaining reviewers could let us know whether our responses resolve their concerns, and we are happy to discuss any remaining questions.

Dear reviewers,

It seems that the authors have carefully responded to your reviews.

If you haven't already, please take your time to go over them and let authors know what you think, to your earliest convenience.

Best,
AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.