acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Routing Exploration in Modern Mixture-of-Experts

Abstract

Mixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained expert segmentation and shared experts. The former increases expressiveness by enabling more expert combinations at a fixed activation ratio, and the latter introduces experts that are active for every token. We identify an *exploration dilemma* between these designs: shared experts can make routing less exploratory by reducing the variability of token representations. This conflicts with the greater need for exploration induced by fine-grained expert segmentation due to the combinatorially enlarged routing space. Consequently, the additional expressiveness offered by fine-grained segmentation may not be fully realized in practice. MirrorMoE is proposed to reconcile this exploration dilemma by shrinking the exploration space through token-adaptive suppression of unnecessary experts. MirrorMoE consistently improves performance across model scales.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MoE, expressiveness, shared experts, pretraining efficiency, exploration

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_8TsdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies routing exploration in MoEs with fine-grained and shared experts. The authors argue that shared experts reduce the effect of routing decisions, making the larger routing space harder to explore. They propose MirrorMoE, which suppresses some selected experts' contributions early in training and gradually returns to standard routing. The paper gives sufficient conditions for expert suppression and preservation, and reports better pretraining PPL and downstream performance across 100M-2.4B models.

Critical strengths

1. Consistent improvements. MirrorMoE improves pretraining PPL across four model scales (100M-2.4B) and different activation budgets, with downstream gains. This shows that its benefits extend beyond one configuration.

2. Simple method. MirrorMoE changes expert selection and aggregation using existing router logits, with no extra parameters. Annealing returns it to standard routing, so inference stays unchanged.

Critical weaknesses

1. The analysis does not clearly show poor exploration (Section 2.2). I am not convinced that a smaller PPL increase under random routing means poor exploration. Routed experts may help specific tokens or contexts, which average PPL can hide. ExpertVar and RouterVar may also decrease because shared experts increase the normalization denominator, even when absolute variation stays the same. The bound in Appendix C.1 does not establish exploration effectiveness during training. I would like to see absolute variation and NLL changes on relevant token subsets before accepting this explanation.

2. Routed experts may simply need more training (Section 2.2, Appendix B.1). The main analysis uses a 498M model with a limited training budget. Shared experts see every token, while routed experts only see assigned tokens. This may partly explain their smaller influence. Do these observations persist with longer training or on independently pretrained MoEs? Without this check, it is unclear whether they reflect an architectural limitation or a stage of training.

Questions for authors

1. Average PPL may hide useful specialization. (Section 2.2) Routed experts may mainly help specific tokens or contexts. Random routing also leaves shared experts unchanged, while it may disrupt specific knowledge carried by routed experts in the model without shared experts. Could you report NLL changes on specific token groups to separate these explanations?

2. ExpertVar and RouterVar may mainly decrease because of the denominator. Write the output as y=s+ry=s+r, where ss is the shared-expert output and rr is the combined routed-expert output. Let rer_e be the output of routed expert ee, and let r(m)r^{(m)} be the routed output under sampled route mm. Define the mean routed output as: rˉ=1M∑mr(m)\bar{r}=\frac{1}{M}\sum_m r^{(m)}. The metrics then become: ExpertVar=1R∑e∥re−r∥2∥s+r∥2\mathrm{ExpertVar}=\frac{\frac{1}{R}\sum_e\|r_e-r\|^2}{\|s+r\|^2} and RouterVar=1M∑m∥r(m)−rˉ∥2∥s+rˉ∥2\mathrm{RouterVar}=\frac{\frac{1}{M}\sum_m\|r^{(m)}-\bar{r}\|^2}{\|s+\bar{r}\|^2}. The shared output cancels in both numerators but remains in the denominators. Both metrics could therefore decrease even when absolute variation stays the same. Could you report the numerators and denominators separately? I would also like to see token NLL variance across sampled routes to check whether output variation reflects changes in predictions.

3. Could routed experts simply need more training? With roughly balanced routing, each routed expert sees only 6/64 of the tokens, while each shared expert sees every token. This may partly explain their different influence. Does the routing-sensitivity gap between the two models shrink across the 20%, 50%, and 100% checkpoints? Does it persist with longer training?

4. Do these findings hold beyond text? All experiments use text, so its use beyond text remains unclear. A small multimodal experiment would help check whether shared experts show the same dominance and whether MirrorMoE still helps.

5. Could you report random-guessing accuracy for each downstream task? This would help show how far the models perform above chance and put the reported gains in context.

4 comments

We thank the reviewer for the effort devoted to reviewing our paper. We have revised the draft, with changes highlighted in blue. Please also find our responses below.

W1. Motivation

  • PPL increase vs. exploration. We believe there is a misunderstanding on Fig.3(a), possibly due to its caption. We do not claim that PPL increase indicates insufficient exploration. The message, emphasized in the italicized statement in Lines 177–181, is that shared experts dominate the model output. We have updated the caption to avoid further confusion.
  • ExpVar and RouterVar. As requested, we re-evaluated RouterVar and ExpertVar by normalizing only with respect to the outputs of routed experts. Both metrics still have smaller variance even when shared experts are excluded from the normalization; see the dotted lines in Figs.3 (a) and (b).
  • Our original choice of normalization with shared experts is motivated by the fact that this is the actual representation passed to the next layer, which in turn influences routing decisions and exploration in subsequent layers. In either case, both metrics remain lower with shared experts, further supporting our motivation.

W2. Limited tokens, routed experts need longer training?

First, we would like to clarify that our training budget follows the compute-optimal scaling laws established in [R1]. A more conservative approach is to apply the Chinchilla scaling law to the number of activated parameters, under which our training token budget is already about 3x than the prescribed amount. We therefore believe that our training budget would not be characterize as limited.

Second, as requested, we doubled the training budget, and report the ExpVar and RouterVar for both MoEs with or without shared expert at checkpoints 60%, 80% and 100% of the extended training schedule. Results are shown in Fig.9 in App.E2. We observe that the reduced variance associated with shared experts persists with longer training, suggesting that additional training alone may not be the solution. Intuitively, this is because routing patterns become stable in the later stages of training.

Third, if routed experts require longer training when combined with shared experts than when trained without them, this itself represents a limitation of shared experts. The reviewer's explanation is in aligning with our motivation.

Q1. evaluation on specific token groups

We appreciate the reviewer's suggestion to investigate specific token groups and would welcome further guidance on how best to define them.

  • The definition of relevant token groups may depend on the task. For example, tokens important for coding may differ substantially from those relevant to mathematics.
  • Different routing mechanisms may naturally have their own strengths and weaknesses, much like students who excel in different subjects. For instance, one router may perform better on math tasks, while another may be stronger at coding. Consequently, the choice of token groups could influence the conclusions.

Q2. ExpertVar and RouterVar, and NLL variance

More on ExpertVar and RouterVar, please see our responses to W2.

Regarding NLL variance, we conducted the requested evaluation across sampled routes at the final checkpoint. The NLL variance is 0.2446 with shared experts, compared with 0.3329 without shared experts. This further confirms that shared experts reduce the sensitivity of model predictions to routing variations, consistent with our observations from ExpertVar and RouterVar.

Q3. longer training

If introducing shared experts causes routed experts to require more training to develop their contributions, this itself represents a limitation of the shared-expert architecture. Thus, the reviewer's explanation is aligning with our motivation.

More details see W2.

Q4. Extension to multi-modality models

A common approach to building multimodal models is to pretrain an LLM backbone and then align it with other modalities. Our contribution focuses precisely on the first stage, where MirrorMoE can be applied to pretrain a stronger MoE backbone, potentially benefiting downstream multimodal fine-tuning.

We appreciate the reviewer’s suggestion and agree that extending MirrorMoE to multimodal models is a valuable direction for future work. Our paper makes no claims on multimodality, and focusing on text is well-established practice in MoE research, as exemplified by DeepSeek-V3, and Qwen1.5-MoE. We therefore hope our contributions can be evaluated within their intended scope.

Q5. Average guess on downstream tasks

ARC-EOBQAHSPIQABoolQSIQAAvg
Random~252525505033.334.72

This confirms that the performance reported in Table 2 is above random-guessing levels.

[R1] Flame-MoE: A transparent end-to-end research platform for mixture-of-experts language models. arXiv:2505.20225.

Thank you again for your valuable feedback. Have our responses addressed your concerns? We would be happy to provide further clarification and look forward to hearing your thoughts!

Thanks for the rebuttal and the new experiments. My main concern is still only partly addressed: I asked for the numerators and denominators of ExpertVar and RouterVar separately, but got a different normalization instead, and the NLL variance differs only modestly (0.2446 vs. 0.3329), unlike the roughly 10x gap in Fig. 3c. The training-stage explanation is also still open, since routing criticality under longer training wasn't checked and the bound in Appendix C.1 wasn't discussed. So I'll keep my rating.

Thank you for your additional response.

1. Asked for numerators and denominators of ExpertVar and RouterVar separately, but got a different normalization instead,

Reporting the asked quantities separately is challenging. Taking ExpertVar as an example, both the numerator and denominator are token-dependent and vary across tokens. Moreover, the numerator involves all 64 experts in each layer. Therefore, reporting these quantities token-by-token, layer-by-layer, and expert-by-expert would be infeasible for rebuttal. Similar considerations apply to RouterVar.

This motivated us to provide an alternative normalization in response to your W1, which you also raised in Q2. It shows that even only considering routed experts in normalization, ExpertVar and RouterVar are still smaller with shared experts. We appreciate your understanding of these practical difficulties.

Nevertheless, we can still report the averaged numbers across different tokens, experts, and layers for your reference below. Please also note that the ratio of the averaged numerator to the averaged denominator does not reproduce the reported ExpertVar or RouterVar, since averaging and division does not equal to division then average.

w/ shared expertNumeratorDenominatorDenominator (Routed Only)
RouterVar0.59376.60094.1304
ExpertVar25.46158.01256.3387
w/o shared expertNumeratorDenominator
RouterVar0.66671.2240
ExpertVar36.1063.1560

2. NLL variance differs only modestly (0.2446 vs. 0.3329), unlike the roughly 10x gap in Fig. 3c

We agree that NLL variance differs less than RouterVar in Fig.3(c). However, the observed reduction still supports our claim, and we see no theoretical reason to expect proportional changes in two distinct quantities. In particular,

  • First, the decrease from 0.3329 to 0.2446 still represents a 26.5% reduction, which is not negligible. Our claim is that exploration, as reflected by variance, is reduced, rather than that exhibiting a 10x reduction. Therefore, this observation is consistent with our claim and provides additional empirical support.
  • Second, scientifically, relative changes of two distinct quantities should not necessarily be proportional. For example, a 10% increase in outdoor temperature does not necessarily lead to a 10% increase in electricity consumption. Similarly, there seems to be no theoretical basis requiring the relative changes of RouterVar and NLL variance to be proportional.

3. routing criticality under longer training wasn't checked

As requested, we have further included the routing criticality results in Appendix E.3, Fig. 9(c). The results exhibit a trend similar to that observed in Fig. 3(a). Moreover, routing criticality remains relatively stable throughout extended training, consistent with the trends observed for ExpertVar and RouterVar in Figs. 9(a) and (b). These additional results suggest that simply extending the training duration may not be sufficient to address the underlying issue.

4. the bound in Appendix C.1.

The original concern in W1 from the reviewer was:

The bound in Appendix C.1 does not establish exploration effectiveness during training.

There might be a misunderstanding of the intended purpose of the bound in Appendix C.1, as we have never claimed that it establishes effectiveness of exploration. Rather, as stated in Line 201, it provides an upper bound on the relative change induced by arbitrary routing policies. Indeed, MirrorMoE is only introduced in Section 3, whereas this bound belongs to the analysis in Section 2 (with its derivation put in Appendix C.1 to save space). Therefore, it is not intended to explain effectiveness from a method that has not yet been introduced!

Instead, MirrorMoE is supported by our extensive empirical comparisons with standard MoE routing in Section 4.



Overall, we would like to highlight that both the original review and the remaining concerns appear to focus primarily on the motivation. We believe that an equally important, if not more compelling, validation of our motivation lies in the concrete and consistent empirical improvements of MirrorMoE. These results, in retrospect, further support our underlying motivation.

Reviewer_WhoxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the interaction between two now-standard MoE design choices — fine-grained expert segmentation  (large E, K at fixed activation ratio) and shared experts  (always-on experts) — through the lens of  routing exploration during pretraining. The central claim is there exists an exploration dilemma, where fine-grained segmentation combinatorially enlarges the routing space, while shared experts dominate the layer output and thereby reduce the variability of token representations (so routing becomes less exploratory). The two pull in opposite directions, so the expressiveness promised by larger K is not realized in shared-expert MoEs. The diagnosis is supported by several complementary measurements on a 498M model: (i) expert-ablation showing shared experts matter disproportionately; (ii) routing criticality: replace router with uniform routing showing early/middle layers of shared-expert MoEs are robust to random routing; (iii) expert variance: output change if only one routed expert fired and (iv) router variance: output spread under stochastic routing / Gumbel noise both ~an order of magnitude smaller with shared experts. Three downstream "inefficiencies" are then shown across 100M–2.4B models (Fig. 1): shared-expert benefits erode, larger K can hurt, and routing specialization is slower. Based on these observations, the authors propose MirrorMoE, which shrink exploration. The method contains: Mirror routing that mixes the standard softmax with a sign-negated softmax (favoring low-logit experts). Mirror aggregation then weights the selected experts, so activated low-logit experts receive negligible weight and are "functionally suppressed," reducing the number of effective experts K_eff. In validation, four shared-expert MoEs (100M–2.4B total, 38M–270M activated) are conducted on DCLM via Megatron-LM / FLAME-MoE, six commonsense downstream tasks and ablations are done.

Critical strengths

1.A well-instrumented, convincing diagnosis of a real phenomenon. The exploration dilemma is substantiated by four independent, mutually reinforcing measurements (expert ablation, routing criticality, expert variance, router variance across Gumbel \beta), and its consequences are shown to scale. This diagnostic rigor is the paper's strongest asset: it establishes that shared-expert dominance measurably suppresses routing variability, which is a non-obvious and useful finding independent of the proposed fix.

2.A simple, cheap method tightly coupled to the diagnosis, with the right controls. MirrorMoE is a few lines, adds negligible/no runtime.

Critical weaknesses

1.Modest gains without uncertainty quantification. The PPL improvements are ~0.5-1.0 point and downstream averages +0.21 to +1.64; the 100M downstream gain (+0.21) is within plausible noise, and several individual tasks regress (HellaSwag, BoolQ, SIQA in places). Under such small improvement, no variance or CIs are reported for PPL or downstream, so it is hard to judge which gains are significant.

2.A meaningful fraction of the gain is from the load-balancing change, not mirror routing. Table 6 shows MirrorMoE with vanilla balancing reaches 52.94 vs. 51.77 with the modified balance (baseline 54.78). So the load-balancing modification contributes ~1.2 of the total ~3.0 PPL gain — this is a somewhat separable trick bundled into the method, and its interaction with the exploration narrative is under-analyzed.

3.Max 2.4B total / 270M activated with 4.4–12.8B training tokens is small relative to the modern MoEs the motivation invokes (DeepSeek/Kimi/Qwen). Whether the dilemma and the fix persist at larger E, more shared experts, and longer schedules is untested. Is there any result at a larger scale or with more shared experts (S>2) / larger activation ratios, even partial, indicating the dilemma and the fix persist where the motivation locates them?

4.For the w/ vs w/o shared-expert comparison (Fig. 1), how much does the differing batch size / expert-parallel layout affect the curves? Can you match these more tightly?

Questions for authors

See above weakness.

2 comments

We thank the reviewer for the effort devoted to reviewing our paper. We have revised the draft, with changes highlighted in blue. Please also find our responses below.

W1. seeds

Thanks for pointing this out. We conducted additional pretraining experiments across all four model sizes using three independent random seeds. We report the mean ± standard deviation and the corresponding p-values below.

ModelVanillaMirrorMoEp-value
MoE-100M-A38M53.73 ± 1.0651.84 ± 0.440.0746
MoE-498M-121M28.47 ± 0.4527.31 ± 0.160.0349
MoE-625M-A100M28.67 ± 0.2627.94 ± 0.240.0236
MoE-2.4B-A270M21.38 ± 0.2220.34 ± 0.200.0039

MirrorMoE consistently outperforms vanilla MoE, with statistically significant improvements (p<0.05) on larger models. Notably, the improvements observed on larger models are even more pronounced when averaged across multiple seeds than in our original single-seed experiments.

W2. Load balancing loss

We would like to clarify that the improvement from 52.94 to 51.77 should not be interpreted as a gain from load balancing. Rather, conventional load balancing conflicts with MirrorMoE's design for reasons explained below. Even under this burden, MirrorMoE still improves performance from 54.78 to 52.94. This further showcases the effectiveness of its design.

Specifically, MirrorMoE deliberately assigns negligible aggregation weights to certain selected experts, effectively suppressing their contributions. However, conventional load balancing encourages more uniform routing probabilities, thereby increasing the importance of experts that MirrorMoE intentionally suppresses. This directly counteracts MirrorMoE's design objective.

Therefore, we believe the result highlighted by the reviewer further supports the effectiveness of MirrorMoE, rather than revealing a limitation.

W3. More experiments

We agree that experiments on larger MoE architectures would further strengthen our findings. However, training models at the scale of DeepSeek, Kimi, or Qwen is currently beyond our computational budget. To partially address this concern, we have already extended our experiments on the 2.4B model to different values of KK, examining the effectiveness of MirrorMoE under varying activation ratios. Regarding the number of shared experts SS, our intuition suggests that increasing SS may further exacerbate the exploration dilemma. To investigate this hypothesis, we have conducted additional experiments on the 38M model with varying numbers of shared experts.

# Shared ExpertsMoEMirrorMoE
S=254.7851.77
S=452.9449.26

Increasing SS makes MirrorMoE enlarges the improvement. This is aligning with the intuition.

W4. w/ vs w/o shared-expert configuration

The slight mismatch arises from internal constraints in Megatron-LM related to expert parallelism, which prevent certain configurations from passing internal consistency checks. Addressing these framework limitations is an interesting direction for future community efforts.

Nevertheless, when comparing the performance of different architectures, matching training FLOPs and the number of training tokens is a common practice in the literature; see e.g., [R1]. Both are controlled in our experiments.

[R1] Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation

Thank you again for your valuable feedback. Have our responses addressed your concerns? We would be happy to provide further clarification and look forward to hearing your thoughts!

Reviewer_AhbTOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Modern MoE architectures typically combine fine-grained expert segmentation with shared experts; however, the tension between these two designs prevents the capacity gains of activating more experts from being realized, even leading to performance degradation.

Taking the opposite approach, the authors propose a strategy that adaptively shrinks the exploration space: routing logits are negated during expert selection, whereas the original positive probabilities are retained for final representation aggregation. This drives the contribution of weak experts chosen by the mirror mechanism close to zero, achieving token-adaptive "functional suppression" of unnecessary experts and thereby reducing the number of effective experts that must be explored. As training progresses, a cosine annealing schedule gradually diminishes this suppression to reopen the exploration space, balancing early-stage training stability with late-stage model expressiveness.

In pretraining experiments on MoE models ranging from 100M to 2.4B parameters, MirrorMoE consistently reduces validation perplexity and improves downstream commonsense reasoning tasks by an average of 1.64 points, successfully restoring the scaling benefits where performance steadily improves as the number of activated experts increases.

Critical strengths
  1. From the dynamical perspective of "exploration versus exploitation," the paper identifies the inherent conflict between two prevailing design trends: fine-grained expert segmentation leads to a combinatorial explosion of the routing space, creating an urgent demand for broader routing exploration; meanwhile, shared experts—which are constantly activated for every token—overwhelmingly dominate representation learning, drastically diminishing the representational variance across different routing choices and directly stifling exploration.

  2. Lightweight and elegant algorithmic design: By negating the routing logits to select experts (mirror routing) while retaining the original positive probabilities for aggregation, the method achieves token-adaptive "functional suppression". It offers high practical versatility and engineering viability.

Critical weaknesses
  1. With a maximum scale of 2.4B parameters and only 12.8B tokens used for training, the model remains at an early stage by modern pre-training standards; it remains to be seen whether the identified "exploration dilemma" and the performance gains offered by MirrorMoE will persist over longer training durations.
  2. Some current open-source MoE models have shifted toward a routing architecture that relies on independent Sigmoid-based scoring—omitting auxiliary losses—and employs dynamic routing bias to balance load. The paper does not discuss how this approach might be adapted for a Sigmoid-based architecture.
Questions for authors
  1. Suppressed experts contribute negligibly during aggregation, yet they still execute full FFN forward and backward passes, consuming computation and memory. Have the authors considered implementing a dynamic threshold to physically prune these suppressed experts during execution, thereby translating the reduced exploration space into actual wall-clock speedup and training throughput gains?
  2. Recent open-source MoE architectures have largely shifted toward using independent Sigmoid-based scoring combined with dynamic routing biases. However, the mathematical derivation of MirrorMoE fundamentally relies on the normalization property of Softmax and the properties of its reciprocal. Could the authors clarify whether a "mirror mechanism" can be constructed within a Sigmoid-based routing framework? If so, how might this be implemented in practice?
3 comments

We thank the reviewer for the effort devoted to reviewing our paper. We have revised the draft, with changes highlighted in blue. Please also find our responses below.

C1. Longer run

We would like to clarify that our training budget follows the compute-optimal scaling laws established in [R1]. A more conservative approach is to apply the Chinchilla scaling law to the number of activated parameters, under which our training token budget is already approximately 3× larger than the prescribed amount. We therefore believe that the convergence would not be characterize as early stage.

At the same time, we agree with the reviewer that evaluating MirrorMoE under longer training schedules would further strengthen our findings. We doubled the training iterations for both vanilla MoE and MirrorMoE.

ModelVanillaMirrorMoE
MoE-100M-A38M42.5940.16
MoE-498M-A121M24.6524.05

The perplexity results show that MirrorMoE's performance advantage persists.

The exploration dilemma also holds on doubled training budget; see Fig.9 in the updated draft, where we evaluate RouterVar and ExpertVar on checkpoints of 60%, 80% and 100% of the extended training schedule.

C2 & Q2. Extension to sigmoid routing.

Thanks for raising this great point.

First, we would like to clarify that recent models, such as Qwen3-Next and Qwen3.5, continue to adopt the architectural settings studied in our work. Thus, our method is directly applicable to modern MoE architectures, even without further extensions.

Second, we agree that extending MirrorMoE to sigmoid-based routing is a valuable direction. The underlying principle of MirrorMoE can be adapted with a simple modification. Let zez_e be the logit output of the Gh\mathbf{G} \mathbf{h}. Then, one can adapt MirrorMoE routing as: pe=max⁡((1−ρ)σ(ze), ρσ(−ze)).p_e = \max\left((1-\rho)\sigma(z_e),\ \rho\sigma(-z_e)\right).

The aggregation is still the same to use original routing scores. More detailed discussions is provided in App.D in the updated draft. Please also see Fig.8 for the similarity between MirrorMoE with sigmoid and softmax routing. We are currently queueing for the GPUs for experiments, and will update once there are responses.

Q1.Hard suppression

Thanks for bringing it out. So far we are employing a soft suppression, yet a hard suppression is a natural extension that could reduce runtime. We run an experiment by removing experts whose \xi_e < \tau, where \tau is a threshold. We set τ=0.05\tau = 0.05, and compare the perplexity, as well as runtime below.

MethodPPLIter. Time (ms)
Vanilla MoE54.787871
MirrorMoE51.777844
MirrorMoE (τ=0.05\tau=0.05)52.347520

Encouragingly, this achieves a 4.1% reduction in iteration time while retaining most of MirrorMoE's PPL improvement. We view this as a promising direction for future work, while the present paper focuses on the best performance.

[R1] Hao Kang, Zichun Yu, and Chenyan Xiong. Flame-moe: A transparent end-to-end research platform for mixture-of-experts language models. arXiv preprint arXiv:2505.20225, 2025.

Thank you again for your valuable feedback. Have our responses addressed your concerns? We would be happy to provide further clarification and look forward to hearing your thoughts!

Thanks to the authors for running the additional experiments and answering my questions.

The doubled-iteration runs on the 100M and 498M models address my concern. The hard-suppression experiment also shows real speedup with minimal PPL impact. I look forward to seeing the results for sigmoid routing once your GPU runs finish.

Overall, my main concerns have been well addressed, and I am happy to maintain my rating.

Reviewer_5zmhOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies the interaction between shared experts and fine-grained experts in MoE models. The authors argue that shared experts dominate token representations, limiting routing exploration and making it difficult to benefit from larger expert combinations (as the expert count increases, returns diminish). To tackle this, the authors propose mirrorMoE, which selects experts using a mixture of standard and sign-reversed routing probabilities and assignsx negligible weights to low-probability experts.

Critical strengths
  • A problem setting is of interest to the community.
  • Interesting observations about shared experts.
Critical weaknesses
  • The main motivation is not fully convincing. The paper treats lower representation variance as evidence of insufficient routing exploration. However, the proposed metrics are normalized by the overall output, which includes shared experts. A dominant shared component can naturally reduce these values without necessarily indicating poor exploration. The connection between this observation and optimization performance remains unclear.
  • MirrorMoE essentially selects some low-probability experts and gives them negligible weights, followed by annealing. This seems straightforward, and the paper does not convincingly explain why sign-reversed routing is preferable to simpler alternatives. The theoretical results do not explain why it improves learning; rather, they mainly formalize when suppression occurs.
  • The baselines are limited. Comparisons with entropy regularization, temperature annealing, simple optimization tricks, and stronger adaptive routing approaches would help establish the proposed method's actual benefit. Moreover, the top-p comparison is not compute-matched. It remains unclear whether the sign-reversed routing mechanism is actually necessary to achieve the reported gains.
  • On load balancing: it seems that the gain from the modified load balancing in the case of MirrorMoE (Appendix B.5) is substantial: 52.94 -> 51.77. Can better load balancing alone help the stated problem? The paper does not sufficiently disentangle the effects of routing, annealing, and load balancing.
  • The effective expert count is based on the entropy of the aggregation weights, so its reduction is largely expected from the proposed design. Moreover, the suppressed experts are still activated and computed. The method therefore does not actually reduce computational cost, and it is unclear why this is preferable to directly adjusting the number of activated experts.
  • On one of the claims, the paper claims that mirrorMoE consistently benefits from increasing K, but Fig. 1a with the 100M model shows the opposite trend; it degrades performance. Also, comparisons between shared/non-shared experiments involve different batch configurations, which could affect the conclusions; this needs to be taken into account.
  • On statistical significance, the experiments do not report results across multiple seeds, and several downstream improvements are relatively small. Evaluation on larger or established MoE architectures would make the findings more convincing.
Questions for authors

Please refer to the weaknesses above.

4 comments

We thank the reviewer for the effort devoted to reviewing our paper. We have revised the draft, with changes highlighted in blue. Please also find our responses below.

W1. Shared experts in the normalization of Expert/Router Var

We re-evaluated RouterVar and ExpertVar by normalizing only with respect to the outputs of routed experts. Both metrics still have smaller variance even when shared experts are excluded from the normalization; see the dotted lines in Figs.3 (b) and (c). We hope these additional results strengthen our motivation.

W2&W3. Comparison with other methods

We hope to clarify that the original paper already includes several baselines:

  • Methods with more explorative routing, i.e., stochastic routing in Tab.5
  • Other methods that shrink routing space in Tab.3
  • Adaptive routing such as Top-p (Tab.8)

Following the suggestion, we add entropy regularization with weight λ\lambda.

MoEEntReg(0.01)EntRef(0.001)MirrorMoE
54.755.1554.8651.77

MirrorMoE outperforms these methods. The reasons have been discussed in the original paper, which we summarize below.

  • Exploration-enhancing methods (sto. routing and entropic reg.). Increasing exploration introduces larger variance in training, which can negatively impact convergence of optimizers.
  • Routing-space reduction methods (Tab.3). While reducing routing space, they do so uniformly across tokens. In contrast, MirrorMoE achieves token-level adaptivity, as characterized in Props. 3.2 and 3.3 and shown in Fig.7.
  • Top-p routing (Tab.8). Primarily designed for settings without shared experts, and hence can also suffer from exploration dilemma. Even with more computational cost, top-p routing is outperformed by MirrorMoE. Though not compute-matched, the comparison favors Top-p, which uses more compute. We view this as a merit of MirrorMoE than a limitation.

W2. necessity of sign reversal

To the best of our knowledge, we are the first to characterize exploration dilemma and propose a solution. Sign reversal is a design choice to provide a simple, theoretically and empirically characterized starting point for this previously overlooked issue. Analogously, many successful architectures involve choices that may be not uniquely optimal, e.g., gaps of GPT and Llama models.

W4. Load balancing.

We hope to clarify that the improvement should not be interpreted as a gain from load balancing (LB). Instead, conventional LB conflicts with MirrorMoE's design for reasons below. Even under this burden, MirrorMoE still improves from 54.78 to 52.94. This reversely showcases our effectiveness.

Specifically, MirrorMoE assigns negligible aggregation weights to suppress the contributions for certain experts. However, conventional LB encourages more uniform routing probabilities, increasing the importance of experts that MirrorMoE suppresses. This counteracts MirrorMoE's design objective.

We believe that the result highlighted by the reviewer is a strength than a limitation.

W5.1 reduction of effective experts is expected

While its reduction is consistent with MirrorMoE's design, the magnitude of the reduction of effective experts depends on the learned routing. The magnitude of the curves vary substantially across layers; see Fig.7.

We see both (1) the agreement between the design and empirical observation and (2) the ability to reveal differences in dynamics as merits rather than limitations.

W5.2 comparison to adjusting the number of R

  • Reducing R, e.g., from 8 to 6, throughout. This reduces the exploration space at the cost of limiting model expressiveness, as 8 experts can represent at least what 6 experts can. Empirically, Fig.1 shows that this tradeoff between exploration and expressiveness does not yield a consistent trend. And simply reducing R does not match the performance of MirrorMoE; see Fig.1

  • Gradually increasing R at training. This is exactly the setting in Tab.3. It lacks the token and layer-level adaptivity, hence does not match the performance of MirrorMoE.

W6. (non)shared experiments configs

The slight mismatch on the configuration of (non-)shared experts arises from constraints in internal checks of the code framework, Megatron-LM. Addressing the framework limitations is an interesting direction for future community efforts.

Nevertheless, when comparing different architectures, matching training FLOPs and the number of tokens is a common practice; see e.g., [R1]. Both are controlled in our experiments.

W7. seeds

MoEMirrorMoEp-value
MoE-100M-38M53.73±1.0651.84±0.440.0746
MoE-498M-121M28.47±0.4527.31±0.160.0349
MoE-625M-100M28.67±0.2627.94±0.240.0236
MoE-2.4B-270M21.38±0.2220.34±0.200.0039

We conduct experiments with 3 seeds. MirrorMoE has statistically significant improvements (p<0.05) on larger models. Moreover, the gain on larger models are more pronounced than the original single-seed result.

[R1] arxiv 2507.10524

Thank you again for your valuable feedback. Have our responses addressed your concerns? We would be happy to provide further clarification and look forward to hearing your thoughts!

Thank you for the additional experiments; I will raise my soundness score but keep my overall rating at 2. My remaining concerns are (i) the unclear link between lower routing variance and poorer optimization, (ii) missing simpler controls such as temperature annealing, and (iii) the batch-size confound in the shared/non-shared comparison.

AuthorsOpenReview ↗

Responses to the follow-up questions

We are glad that our responses have addressed most of your concerns, and we are happy to provide further clarification.

(i) the unclear link between lower routing variance and poorer optimization

We would like to clarify the intuition step by step.

  • Step 1. In MoE pretraining, “optimization” can be equivalently translated into “finding the optimal experts for every token.” The routing mechanism thus calls for both exploration (to tryout other experts) and exploitation (to stay in current routed experts).

  • Step 2. Variance (of a policy) can serve as an indicator of exploration, consistent with what is commonly studied in bandit and reinforcement learning literature. Since the outcomes of a routing policy are reflected in the learned features, RouterVar is defined through them to quantify variance/exploration. In other words, lower RouterVar suggests less exploration in MoEs with shared experts compared to their non-shared counterparts.

  • Step 3. The lack of exploration hinders this exploration-exploitation tradeoff, therefore making “optimization” less effective.

More detailed explanations are provided in Sections 1 and 2.2, while the practical consequences can be found in Section 2.3.

(ii) missing simpler controls such as temperature annealing

Following your suggestion, we add temperature annealing as an additional baseline (Temp in the table below). Specifically, we initialize the routing temperature at 2 to encourage exploration and gradually anneal it to 1 using a cosine schedule. The perplexity results show that it does not improve over standard MoE training.

MoEMirrorMoETemp
54.7851.7755.43

(iii) the batch-size in the shared/non-shared comparison

We would first like to clarify that the batch-size difference arises from practical constraints of Megatron-LM, one of the most widely adopted frameworks for MoE training.

  • Consider an MoE with 64 routed experts and 2 shared experts. Under Megatron-LM's expert parallelism, the 64 routed experts can be evenly distributed across 4 GPUs, with 16 routed experts per GPU, while the 2 shared experts are replicated across GPUs. Under our training configuration, a batch size of 1024 can be evenly divided across 4 GPUs, allowing each GPU to process 256 samples.
  • For the non-shared MoE, our primary goal is to match the total number of experts and trainable parameters, resulting in 66 routed experts. However, evenly distributing 66 experts requires a different expert-parallel configuration in Megatron-LM, such as 3 GPUs. Since 1024 is not divisible by 3, we slightly adjust the batch size to 1056 (= 32 × 3 GPUs × 11 gradient accumulation steps). This represents only a tiny difference, which we would not expect to substantially affect training outcomes.

Indeed, comparing different architectures under perfectly identical configurations is often infeasible, and minor adjustments to accommodate architectural differences are common in large-scale research. For example, LLaMA 3-8B has 8.03B parameters, while Qwen3-8B has 8.19B, i.e., a 0.16B difference. Moreover, their pretraining batch sizes are not necessarily identical. Despite these differences, their performance is routinely compared in the literature. In the same principle, we believe that a small batch-size difference (1024 vs. 1056, approximately 3.1%) should not invalidate an otherwise carefully controlled (#params, FLOPs, #tokens) comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.