- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies why MoE models route each token to multiple experts. The authors challenge the common view of geometric complementarity and propose coherent overlap as an alternative explanation. They introduce a candidate-by-context factorial test and conduct experiments across six MoE models. The results show that selected experts have overlapping input subspaces and negative interaction effects, while still providing useful contributions to model predictions.
Critical strengths
(1) This paper provides an interesting and insightful perspective on understanding routing in MoE
(2) The authors have put considerable effort into conducting extensive experiments
Critical weaknesses
(1) Concerns regarding the geometric analysis. The router selects experts based on the input representations and its learned routing weights. The input subspace of each expert is then fitted using the tokens assigned to that expert by the same router. Finally, these router-conditioned subspaces are used to characterize geometric overlap among experts and explain the router's selection behavior. This creates a potentially self-referential relationship between how the expert subspaces are constructed and the routing patterns they are used to analyze.
In other words, The expert input subspaces are derived from router-selected tokens, so the observed geometric patterns may largely reflect the routing process itself rather than provide independent evidence of expert complementarity.
(2) The authors' arguments and supporting experiments is unconvincing to me.
Taking experiments in Section 6 as examples, I would argue that a model's dependence on its original computation path does not necessarily imply that each expert has a distinct function. While the router is not explicitly trained with optimal expert labels, its routing behavior is still shaped by the NLL objective. Therefore, many of the observed NLL changes under expert-patching interventions are largely expected and do not necessarily support the authors' proposed explanation.
Questions for authors
See weaknesses.