acceptodds
Under review as a conference paper at ICLR 2027

Why Better Expert Choices Do Not Always Make Better MoE Routers

10 views

Abstract

Sparse mixture-of-experts (MoE) routing determines which experts contribute to each next-token prediction in language models. A router selects a small set of experts from the available prefix before the next token is observed. Yet a search conducted after observing that token can find expert choices that predict it much better than the native route, leaving unclear how much of this gain a router could achieve. Recent methods use such hindsight choices to supervise router training. Because the search sees the answer that the router cannot, part of its gain may be unavailable to any input-only decision. Conversely, a trained router's failure to match the search cannot show whether it has also missed useful choices that were available before the answer. We separate these effects within fixed candidate sets by using independent human completions of the same prefix to lower-bound the value of answer information without assuming an optimal learner. We then use a frozen reference model's complete answer distribution to compute both the hindsight and best input-only values and test whether trained selectors recover the latter on new prefixes. Across three frozen MoEs, answer information accounts for a provable part of the search gain, while the tested selectors leave input-only value unrecovered; in the controlled reference setting, about 70% of the search gain requires the answer.

Primary area:
interpretability and explainable AI
Keywords:
mixture-of-experts, expert routing, oracle routing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2XamOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work concentrates on rethinking recent work that enhance trained MoE routers via “hindsight choices” and “search gain”. The authors propose a paired-answer measurement, and a controlled environment that could get both the hindsight value and the best input-only value. They claimed that nearly 70% of the search gain requires the information of the answer, showing why better expert choices based on the answer do not always result in a better router.

Critical strengths
  1. Overall, the proposed idea is sound, and the authors have conducted sufficient experiments to draw their conclusion.
Critical weaknesses
  1. This paper focuses on rethinking methods that adopt “hindsight choices” to supervise router training, which is not a mainstream approach in the broader MoE field. Its effectiveness has not been widely acknowledged, and we have not seen industrial-scale models deploying similar strategies. This weakens the contribution of this paper. In addition, it is not entirely clear what insights this study can offer for MoE in pre-training — for instance, regarding dynamic MoE routing strategies in pre-training.
  2. The 70% figure claimed in the paper is derived solely from limited experimental results on OLMoE, which cannot establish that this ratio still holds in real-world scenarios. Whether the same conclusion applies to other backbones (with different MoE routing strategies) also warrants further experimental verification. Moreover, would the task type, difficulty, and context length of different evaluation benchmarks affect the validity of the experimental conclusions?
Questions for authors
  1. Is the single-expert swap setting overly restrictive, thereby affecting the conclusions?
Reviewer_n3AZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies routing in mixture-of-experts (MoE) language models. The main question is how much of the gain from selecting experts after observing the next token can be recovered without knowing that token. The paper divides the routing gap into two parts: the value that depends on knowing the next token, and the value that is available without it but may not be recovered by a trained selector. The authors bound the former using paired human completions and compute both parts exactly under a known reference distribution. They then run several evaluations to see whether trained selectors can recover the available value.

Critical strengths

S1. The question, whether hindsight expert choices can be used to supervise router training, the paper seize to answer is interesting

S2. The reasoning process and the evaluation seem sound to me

Critical weaknesses

My main concern is that the scope of the paper seems rather narrow.

W1 the paper only studies the last layer. From a high-level perspective, expert selection can serve different purposes, such as load balancing or specialization by domain and topic. However, the paper mainly focuses on expert selection in relation to next-token prediction. I am not sure how well the conclusions generalize to other aspects of expert routing.

W2. the amount of training data seems quite limited. For example, the input dimension of the selector (2,112/2,176; line 731) is larger than the number of training samples (Sec 5.1: UCL 205 sentences, and up to 9893 documents in FineWeb-Edu). I wonder whether the results would change with more training data.

W3. I also found the paper difficult to follow. One issue is that some sentences are overly complicated and could be simplified. Another issue is that some terms are used without being explained when they are first introduced. For example, "paired-answer" first appears in line 76 without any explanation.

Questions for authors

Q1. Given that the experiments only cover the last layer and short English prefixes, what are the broader implications of the findings?

Q2. Could the authors include some downstream evaluations that reflect other aspects of expert selection, such as domain or topic specialization, or use the AIME and HMMT evaluations of Yoon et al. (2026), to extend the generality of the findings beyong the current setting?

Q3. Could the authors report results with more training data?

Q4. I think there is no page limit in appendix. Why adding previous version of the manuscript in supplement and then pointing to it through appendix? page 22 line 1187.

Reviewer_ADrgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

There is some benefit that you get when using a router to route a token to experts. We can consider these to be hindsight routing gains. The authors separate hindsight routing gains into the value of knowing the answer and gains available to an input-only router. The human completions establish a lower bound on the former term. The authors run evals to see whether a trained selector can recover this

Critical strengths

There is a very interesting question being asked by the paper which gets at the heart of the value of the router in MoEs. And I think the evals are good. The paper contains careful statistics at a respondent-level. There are extensive controls for fixed routing rules, expert weights, capitalization, and calibration strengthen the analysis.

Critical weaknesses

This 70% number just comes from OLMoE; OLMoE is just not a realistic model to study. Is there a cheap way to see whether this can apply to other MoEs?

Questions for authors

This 70% number just comes from OLMoE; OLMoE is just not a realistic model to study. Is there a cheap way to see whether this can apply to other MoEs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.