- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies routing exploration in MoEs with fine-grained and shared experts. The authors argue that shared experts reduce the effect of routing decisions, making the larger routing space harder to explore. They propose MirrorMoE, which suppresses some selected experts' contributions early in training and gradually returns to standard routing. The paper gives sufficient conditions for expert suppression and preservation, and reports better pretraining PPL and downstream performance across 100M-2.4B models.
Critical strengths
1. Consistent improvements. MirrorMoE improves pretraining PPL across four model scales (100M-2.4B) and different activation budgets, with downstream gains. This shows that its benefits extend beyond one configuration.
2. Simple method. MirrorMoE changes expert selection and aggregation using existing router logits, with no extra parameters. Annealing returns it to standard routing, so inference stays unchanged.
Critical weaknesses
1. The analysis does not clearly show poor exploration (Section 2.2). I am not convinced that a smaller PPL increase under random routing means poor exploration. Routed experts may help specific tokens or contexts, which average PPL can hide. ExpertVar and RouterVar may also decrease because shared experts increase the normalization denominator, even when absolute variation stays the same. The bound in Appendix C.1 does not establish exploration effectiveness during training. I would like to see absolute variation and NLL changes on relevant token subsets before accepting this explanation.
2. Routed experts may simply need more training (Section 2.2, Appendix B.1). The main analysis uses a 498M model with a limited training budget. Shared experts see every token, while routed experts only see assigned tokens. This may partly explain their smaller influence. Do these observations persist with longer training or on independently pretrained MoEs? Without this check, it is unclear whether they reflect an architectural limitation or a stage of training.
Questions for authors
1. Average PPL may hide useful specialization. (Section 2.2) Routed experts may mainly help specific tokens or contexts. Random routing also leaves shared experts unchanged, while it may disrupt specific knowledge carried by routed experts in the model without shared experts. Could you report NLL changes on specific token groups to separate these explanations?
2. ExpertVar and RouterVar may mainly decrease because of the denominator. Write the output as , where is the shared-expert output and is the combined routed-expert output. Let be the output of routed expert , and let be the routed output under sampled route . Define the mean routed output as: . The metrics then become: and . The shared output cancels in both numerators but remains in the denominators. Both metrics could therefore decrease even when absolute variation stays the same. Could you report the numerators and denominators separately? I would also like to see token NLL variance across sampled routes to check whether output variation reflects changes in predictions.
3. Could routed experts simply need more training? With roughly balanced routing, each routed expert sees only 6/64 of the tokens, while each shared expert sees every token. This may partly explain their different influence. Does the routing-sensitivity gap between the two models shrink across the 20%, 50%, and 100% checkpoints? Does it persist with longer training?
4. Do these findings hold beyond text? All experiments use text, so its use beyond text remains unclear. A small multimodal experiment would help check whether shared experts show the same dominance and whether MirrorMoE still helps.
5. Could you report random-guessing accuracy for each downstream task? This would help show how far the models perform above chance and put the reported gains in context.
4 comments
Follow-up
Thank you again for your valuable feedback. Have our responses addressed your concerns? We would be happy to provide further clarification and look forward to hearing your thoughts!
Thanks for the rebuttal and the new experiments. My main concern is still only partly addressed: I asked for the numerators and denominators of ExpertVar and RouterVar separately, but got a different normalization instead, and the NLL variance differs only modestly (0.2446 vs. 0.3329), unlike the roughly 10x gap in Fig. 3c. The training-stage explanation is also still open, since routing criticality under longer training wasn't checked and the bound in Appendix C.1 wasn't discussed. So I'll keep my rating.
Thank you for your additional response.
1. Asked for numerators and denominators of ExpertVar and RouterVar separately, but got a different normalization instead,
Reporting the asked quantities separately is challenging. Taking ExpertVar as an example, both the numerator and denominator are token-dependent and vary across tokens. Moreover, the numerator involves all 64 experts in each layer. Therefore, reporting these quantities token-by-token, layer-by-layer, and expert-by-expert would be infeasible for rebuttal. Similar considerations apply to RouterVar.
This motivated us to provide an alternative normalization in response to your W1, which you also raised in Q2. It shows that even only considering routed experts in normalization, ExpertVar and RouterVar are still smaller with shared experts. We appreciate your understanding of these practical difficulties.
Nevertheless, we can still report the averaged numbers across different tokens, experts, and layers for your reference below. Please also note that the ratio of the averaged numerator to the averaged denominator does not reproduce the reported ExpertVar or RouterVar, since averaging and division does not equal to division then average.
| w/ shared expert | Numerator | Denominator | Denominator (Routed Only) |
|---|---|---|---|
| RouterVar | 0.5937 | 6.6009 | 4.1304 |
| ExpertVar | 25.4615 | 8.0125 | 6.3387 |
| w/o shared expert | Numerator | Denominator |
|---|---|---|
| RouterVar | 0.6667 | 1.2240 |
| ExpertVar | 36.106 | 3.1560 |
2. NLL variance differs only modestly (0.2446 vs. 0.3329), unlike the roughly 10x gap in Fig. 3c
We agree that NLL variance differs less than RouterVar in Fig.3(c). However, the observed reduction still supports our claim, and we see no theoretical reason to expect proportional changes in two distinct quantities. In particular,
- First, the decrease from 0.3329 to 0.2446 still represents a 26.5% reduction, which is not negligible. Our claim is that exploration, as reflected by variance, is reduced, rather than that exhibiting a 10x reduction. Therefore, this observation is consistent with our claim and provides additional empirical support.
- Second, scientifically, relative changes of two distinct quantities should not necessarily be proportional. For example, a 10% increase in outdoor temperature does not necessarily lead to a 10% increase in electricity consumption. Similarly, there seems to be no theoretical basis requiring the relative changes of RouterVar and NLL variance to be proportional.
3. routing criticality under longer training wasn't checked
As requested, we have further included the routing criticality results in Appendix E.3, Fig. 9(c). The results exhibit a trend similar to that observed in Fig. 3(a). Moreover, routing criticality remains relatively stable throughout extended training, consistent with the trends observed for ExpertVar and RouterVar in Figs. 9(a) and (b). These additional results suggest that simply extending the training duration may not be sufficient to address the underlying issue.
4. the bound in Appendix C.1.
The original concern in W1 from the reviewer was:
The bound in Appendix C.1 does not establish exploration effectiveness during training.
There might be a misunderstanding of the intended purpose of the bound in Appendix C.1, as we have never claimed that it establishes effectiveness of exploration. Rather, as stated in Line 201, it provides an upper bound on the relative change induced by arbitrary routing policies. Indeed, MirrorMoE is only introduced in Section 3, whereas this bound belongs to the analysis in Section 2 (with its derivation put in Appendix C.1 to save space). Therefore, it is not intended to explain effectiveness from a method that has not yet been introduced!
Instead, MirrorMoE is supported by our extensive empirical comparisons with standard MoE routing in Section 4.
Overall, we would like to highlight that both the original review and the remaining concerns appear to focus primarily on the motivation. We believe that an equally important, if not more compelling, validation of our motivation lies in the concrete and consistent empirical improvements of MirrorMoE. These results, in retrospect, further support our underlying motivation.
We thank the reviewer for the effort devoted to reviewing our paper. We have revised the draft, with changes highlighted in blue. Please also find our responses below.
W1. Motivation
W2. Limited tokens, routed experts need longer training?
First, we would like to clarify that our training budget follows the compute-optimal scaling laws established in [R1]. A more conservative approach is to apply the Chinchilla scaling law to the number of activated parameters, under which our training token budget is already about 3x than the prescribed amount. We therefore believe that our training budget would not be characterize as limited.
Second, as requested, we doubled the training budget, and report the ExpVar and RouterVar for both MoEs with or without shared expert at checkpoints 60%, 80% and 100% of the extended training schedule. Results are shown in Fig.9 in App.E2. We observe that the reduced variance associated with shared experts persists with longer training, suggesting that additional training alone may not be the solution. Intuitively, this is because routing patterns become stable in the later stages of training.
Third, if routed experts require longer training when combined with shared experts than when trained without them, this itself represents a limitation of shared experts. The reviewer's explanation is in aligning with our motivation.
Q1. evaluation on specific token groups
We appreciate the reviewer's suggestion to investigate specific token groups and would welcome further guidance on how best to define them.
Q2. ExpertVar and RouterVar, and NLL variance
More on ExpertVar and RouterVar, please see our responses to W2.
Regarding NLL variance, we conducted the requested evaluation across sampled routes at the final checkpoint. The NLL variance is 0.2446 with shared experts, compared with 0.3329 without shared experts. This further confirms that shared experts reduce the sensitivity of model predictions to routing variations, consistent with our observations from ExpertVar and RouterVar.
Q3. longer training
If introducing shared experts causes routed experts to require more training to develop their contributions, this itself represents a limitation of the shared-expert architecture. Thus, the reviewer's explanation is aligning with our motivation.
More details see W2.
Q4. Extension to multi-modality models
A common approach to building multimodal models is to pretrain an LLM backbone and then align it with other modalities. Our contribution focuses precisely on the first stage, where MirrorMoE can be applied to pretrain a stronger MoE backbone, potentially benefiting downstream multimodal fine-tuning.
We appreciate the reviewer’s suggestion and agree that extending MirrorMoE to multimodal models is a valuable direction for future work. Our paper makes no claims on multimodality, and focusing on text is well-established practice in MoE research, as exemplified by DeepSeek-V3, and Qwen1.5-MoE. We therefore hope our contributions can be evaluated within their intended scope.
Q5. Average guess on downstream tasks
This confirms that the performance reported in Table 2 is above random-guessing levels.
[R1] Flame-MoE: A transparent end-to-end research platform for mixture-of-experts language models. arXiv:2505.20225.