Towards Understanding Specialization in MoE: A Mean-Field Perspective on Routing-Expert Dynamics

Mixture of Experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given sample unchanged. Their benefits depend on a stable specialization among experts, which is shaped by the coupled dynamics of routing and expert modules. However, this specialization can become unstable or collapse, while existing theories struggle to capture this bidirectional feedback. We study a sparsely gated MoE model through mean-field theory, retaining each expert as a finite interacting unit while its overparameterized parameters evolve as a probability measure. This yields a hybrid finite dimensional and Wasserstein dynamical system. We provide a general decomposition of the expert and router dynamics and derive the evolution of comparative advantage. Our analyses further establish sufficient conditions for linear growth of specialization over a finite interval and routing collapse as an invariant set of the joint flow. Finally, we conduct experiments to verify our theoretical results. Our analysis offers a unified macroscopic account of specialization in MoE and provides guidance for scaling sparse models without sacrificing stability.