acceptodds
Withdrawn from ICLR 2027

Towards Understanding Specialization in MoE: A Mean-Field Perspective on Routing-Expert Dynamics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mixture of Experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given sample unchanged. Their benefits depend on a stable specialization among experts, which is shaped by the coupled dynamics of routing and expert modules. However, this specialization can become unstable or collapse, while existing theories struggle to capture this bidirectional feedback. We study a sparsely gated MoE model through mean-field theory, retaining each expert as a finite interacting unit while its overparameterized parameters evolve as a probability measure. This yields a hybrid finite dimensional and Wasserstein dynamical system. We provide a general decomposition of the expert and router dynamics and derive the evolution of comparative advantage. Our analyses further establish sufficient conditions for linear growth of specialization over a finite interval and routing collapse as an invariant set of the joint flow. Finally, we conduct experiments to verify our theoretical results. Our analysis offers a unified macroscopic account of specialization in MoE and provides guidance for scaling sparse models without sacrificing stability.

Primary area:
learning theory
Keywords:
Mixture of Experts, Mean Field Theory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5JvGOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper develops a mean-field theory for specialization in mixture-of-experts models by coupling a finite-dimensional trainable router with expert populations represented as probability measures. The framework decomposes router and expert dynamics, derives the evolution of pairwise comparative advantage, gives sufficient conditions for sustained specialization, characterizes routing collapse, and studies how load-balancing regularization can destabilize collapsed states. Synthetic numerical experiments and a solvable radial-Gaussian example are used to validate the theoretical predictions.

Critical strengths
  1. The paper explicitly models the bidirectional feedback between expert performance and routing allocation rather than treating either component as fixed.
  2. The derived dynamics connect specialization to task allocation and gradient geometry, giving interpretable sufficient conditions for sustained specialization and collapse.
Critical weaknesses
  1. The main validation is synthetic, while modern Transformer MoEs involve top-k routing, optimizer effects, finite-width experts, and data structures not directly captured by the framework.
  2. The quasi-static reductions, invariant-collapse analysis, and solvable example require conditions whose practical validity in real MoE training is not established.
Questions for authors

Q1. Which of the key sufficient conditions in Proposition 4.6 can be measured or approximately verified in a realistic Transformer MoE? Q2. How do the conclusions change under hard top-k routing with discrete expert selection rather than the smooth routing formulation used in the main analysis? Q3. Can the authors provide even a small-scale Transformer-MoE experiment showing that the proposed comparative-advantage dynamics predict specialization or collapse beyond the synthetic setting?

Reviewer_bNVJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the emergence and stability of expert specialization in Mixture-of-Experts (MoE) models from a mean-field perspective. The authors formulate a coupled dynamical system in which the router evolves through Euclidean gradient flow, while each expert is represented by a probability measure evolving through Wasserstein gradient flow. Based on this framework, they derive decompositions of the router and expert dynamics and introduce comparative advantage to characterize the relative performance of different experts on individual tasks.

The main theoretical results include an exact evolution equation for comparative advantage, sufficient conditions for its linear growth over a finite interval, and a characterization of routing collapse as an invariant set. The authors further analyze how load-balancing regularization may destabilize collapsed routing states. A two-task radial Gaussian example is introduced to obtain more explicit analytical results, and numerical experiments on synthetic tasks are conducted to support the theoretical predictions.

Critical strengths
  1. An interesting perspective on coupled routing–expert dynamics.

The paper attempts to study expert specialization by explicitly modeling the interaction between routing allocation and expert optimization, rather than treating them separately. The hybrid Euclidean–Wasserstein formulation offers a potentially useful perspective on this problem. In particular, the decomposition of comparative-advantage dynamics provides an interpretable connection between expert performance, task allocation, and gradient alignment.

  1. A reasonably structured theoretical framework with supporting numerical experiments.

The paper presents several analytical results, including an energy dissipation identity and an exact decomposition of comparative-advantage dynamics, which appear mathematically consistent under the stated model. The authors also construct a radial Gaussian example and conduct controlled numerical experiments to examine the theoretical predictions. These efforts help illustrate the framework, although the extent to which the experiments validate the stronger claims remains limited.

Critical weaknesses
  1. Several mathematical errors and inconsistencies in the theoretical derivations, particularly in the Gaussian example.

My primary concern is the mathematical correctness of several results in the appendix. First, the infinite-width risk defined in Eq. (43) is not generally equivalent to the expected squared loss of the original MoE model. It measures the squared discrepancy between conditional mean predictions and target labels, while the actual expected squared loss also contains conditional prediction variance terms. The authors do not justify eliminating these terms, meaning that zero-risk solutions of Eq. (43) need not be zero-risk solutions of the original problem. Moreover, the treatment of task mixtures introduces another inconsistency: Eqs. (41)–(43) square the averaged task responses, whereas Eq. (53) averages task-wise squared losses. The difference is generally nonzero and equals w(1−w)2[(m1+−m2+)2+(m1−−m2−)2]\frac{w(1-w)}{2}[(m_1^+-m_2^+)^2+(m_1^--m_2^-)^2], where mi±=∫qi±(r)ρ(dr)m_i^\pm=\int q_i^\pm(r)\rho(dr). Consequently, Proposition A.12 characterizes an approximation error for a surrogate risk rather than the original mixture risk.

Second, Proposition A.11 contains an incorrect equality in Eq. (74). The authors define βi\beta_i as the difference between an expert's residual vector and that of a reference single-point solution, but then identify the task risk with ∣βi∣22/2|\beta_i|2^2/2. The correct expression is 2Ri∞(ρj)=∣βi+αi(δri∗)∣222R_i^\infty(\rho_j)=|\beta_i+\alpha_i(\delta{r_i^*})|_2^2, which contains additional baseline and cross terms. These terms are generally nonzero, particularly when Δi<Δ∞\Delta_i<\Delta^\infty, where Lemma D.1 states that the minimum task risk is strictly positive. The expression involving ∣γi∣22|\gamma_i|_2^2 in the same proposition has a similar problem, as γi\gamma_i measures a difference between expert responses rather than the actual prediction residual. The proof in Appendix C.6 does not establish the claimed equalities.

Third, the zero-risk claims in Lemma D.1 and Proposition A.10 are false at an explicitly permitted boundary value. The assumptions allow Δ=1\Delta=1 and require q(0)<−1q(0)<-1. However, at Δ=1\Delta=1, we have q−(r;1)=q(0)q^-(r;1)=q(0) for every rr, implying R∞(ρ;1)≥12(1+q(0))2>0R^\infty(\rho;1)\geq\frac12(1+q(0))^2>0 for every probability measure ρ\rho. This contradicts the claimed existence of zero-risk measures for all Δ≥Δ∞\Delta\geq\Delta^\infty. The underlying proof incorrectly uses γ(+∞;Δ)=(q(+∞),q(+∞))\gamma(+\infty;\Delta)=(q(+\infty),q(+\infty)) at Δ=1\Delta=1, and the claimed set equality D=FD=F in Proposition A.9 consequently fails as stated. There are also issues with other boundary cases: the proof of Lemma D.1 uses a strict monotonicity assumption that does not cover Δ=0\Delta=0, and Proposition A.5 does not properly address the probability mass at the origin when Δi=1\Delta_i=1 in the hard-routing limit.

Finally, Proposition A.14 does not establish the existence of a finite optimal routing threshold under its stated assumptions. The routing cost may approach its infimum only as s→∞s\to\infty. The proof introduces an additional non-degeneracy condition to exclude this possibility, but the condition is missing from the proposition itself. Taken together, these errors are concerning because the Gaussian example serves as the main tractable instantiation of the proposed framework, and several subsequent conclusions rely on its risk characterization.

  1. The main conclusions about specialization and routing collapse are not sufficiently justified by the assumptions or experiments.

A particularly serious issue concerns the quasi-static approximation used to derive the collapse-stability threshold in Proposition A.6. This analysis relies on Proposition 4.2, which assumes a uniformly exponentially attracting expert equilibrium branch. However, as s→0+s\to0^+, the inactive expert's load satisfies m1(s)=O(s2d)m_1(s)=O(s^{2d}), meaning that its optimization speed vanishes near collapse. At s=0s=0, every distribution ρ1\rho_1 is stationary because the expert receives no training signal. Therefore, the required uniform attraction to a particular equilibrium measure cannot hold at the collapsed boundary. Furthermore, the existence of multiple zero-risk expert measures in some regimes raises additional questions about the uniqueness and stability of the equilibrium branch. Consequently, the reduction underlying the proposed stability threshold Cλ=Λ(0)+2λ(κ1+κ2)C_\lambda^=\Lambda^(0)+2\lambda(\kappa_1+\kappa_2) is not justified near the very state it aims to characterize.

The collapse-related propositions also support weaker conclusions than the paper suggests. Proposition A.5 describes s=0s=0 as a collapsed fixed point, although the active expert continues evolving; it is an invariant set rather than a fixed point of the complete joint system. Proposition 4.7 obtains near-collapse velocity bounds only by additionally assuming that the routing gradients are already O(ϵ)O(\epsilon), which does not follow from concentrated routing probabilities alone. Proposition 4.8 proves a conditional local instability result but does not guarantee that load balancing restores specialization. Under the exact-collapse conditions of Lemma D.3, ∇θπj=0\nabla_\theta\pi_j=0 implies ∇θA(θc)=0\nabla_\theta A(\theta_c)=0, so the first destabilization mechanism in Proposition 4.8 cannot apply to that collapsed state. The second mechanism requires a negative-curvature direction that is assumed rather than established. More importantly, destabilizing a collapsed routing state does not necessarily imply convergence to a specialized equilibrium.

Another concern is the gap between the analyzed model and practical MoE training. Eq. (5) optimizes a weighted sum of individual expert losses, which differs from the objective of a conventional soft-output MoE that combines expert predictions before evaluating the loss. The Gaussian example introduces an additional simplification by replacing input-dependent routing with an auxiliary router that is conditionally independent of the input given task identity. For overlapping task distributions, this behavior is generally not realizable by the original input-only router. These simplifications remove important interactions between routing and expert learning, making it unclear whether the theoretical conclusions extend to practical MoE architectures. The paper also does not rigorously establish convergence from finite-width joint training to the proposed mean-field dynamics, including the required parameter and learning-rate scaling.

The experimental evidence is similarly limited. Most experiments verify algebraic identities or conditional inequalities of the assumed gradient-flow system rather than demonstrate why specialization emerges or remains stable. In particular, the load-balancing experiments freeze the experts, so they cannot demonstrate recovery of expert specialization through joint routing–expert feedback. The experiments are restricted to synthetic tasks, and the numerical scales in Table 2 appear inconsistent with Figure 1 unless an additional normalization factor is applied. Overall, the paper provides an interesting formal description of routing–expert interactions, but it has not convincingly established that the proposed dynamics explain the emergence, stability, or recovery of specialization in actual MoE training.

Questions for authors
  1. Regarding the Gaussian risk formulation: How is Eq. (43) obtained from the original expected squared-loss objective? What assumptions justify removing conditional prediction variance? Furthermore, why is the risk of a mixture defined by squaring the averaged responses rather than averaging the task-wise squared losses? How does this affect Proposition A.12 and the subsequent specialization analysis?

  2. Regarding the incorrect mathematical identities: In Proposition A.11, how do the authors justify Eq. (74)'s expressions involving βi\beta_i and γi\gamma_i, given that these quantities represent differences between expert responses rather than residuals relative to the target? Can the authors provide corrected expressions and explain whether the downstream conclusions remain valid?

  3. Regarding the zero-risk claims: How can Lemma D.1 and Proposition A.10 hold at Δ=1\Delta=1, where q−(r;1)=q(0)<−1q^-(r;1)=q(0)<-1 makes zero risk impossible? Does correcting this boundary case also require revising Proposition A.9 and the hard-routing collapse analysis?

  4. Regarding the quasi-static approximation: How can the uniform exponential-attraction assumption required by Proposition 4.2 hold near s=0s=0, where the inactive expert's training load vanishes and every measure of that expert is stationary? How is the equilibrium branch selected in the presence of multiple stationary or zero-risk expert distributions? Can the stability threshold in Proposition A.6 be established without this assumption?

  5. Regarding collapse and load balancing: Proposition 4.8 only provides a conditional local instability result. What guarantees that its negative-curvature condition holds at an actual collapsed routing state, and what ensures that escaping collapse results in meaningful expert specialization rather than merely balanced routing? Could the authors provide experiments in which both the router and experts remain trainable after introducing regularization?

  6. Regarding the relationship to practical MoEs: To what extent do the results extend to standard Top-k or soft-output MoEs? In particular, how do the derivations change when the loss is applied to the weighted combination of expert outputs, or when routing decisions must be made from inputs rather than task identity? Can the authors justify the finite-width-to-mean-field limit for the coupled system?

  7. Regarding auxiliary results and reproducibility: Could the authors clarify the finite-minimizer assumption in Proposition A.14, the boundary case Δ=0\Delta=0 in Lemma D.1, and the numerical scaling used in Table 2? More broadly, can the authors provide experiments that evaluate the emergence and stability of specialization itself, rather than primarily verifying gradient-flow identities?

Reviewer_Z2XfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the coupled training dynamics of a finite collection of experts and a differentiable router, taking a mean-field representation within each expert. Its central technical contribution is an exact evolution equation for pairwise task-specific comparative advantage: the performance difference between two experts is decomposed into contributions from task allocation, task-gradient geometry, and within-task sample-selection covariance. This formulation is developed through Propositions 4.4 and 4.5, rather than through a new training algorithm.

The subsequent results provide a sufficient condition for finite-interval advantage growth, bounds on suppressed learning near routing collapse, and conditional criteria for a load-balancing penalty to displace or destabilize a routing equilibrium. A radial Gaussian example supplies reduced dynamics and a more explicit stability analysis. Experiments use synthetic regression with four tasks and four experts, supplemented by the two-task radial example.

The numerical evidence is reasonably convincing for the exact decomposition and the conditional scaling near an already collapsed state. It is less convincing for the stronger interpretation that the framework explains the formation of specialization through bidirectional feedback and its recovery through regularization. The decisive issue is whether the experiments distinguish these mechanisms and validate the assumptions connecting the reduced model to the joint dynamics.

Critical strengths

S1. The comparative-advantage decomposition makes the mechanism more precise.

Proposition 4.4 separates expert training into the effects of load, conditional task composition, and within-task selection, while expressing router updates through relative expert performance and a covariance residual. Proposition 4.5 then identifies how task allocations and inner products between task gradients govern changes in comparative advantage, with an explicit bound on the selection residual (pages 4–6). This gives a concrete account of when receiving more examples from a task can improve an expert’s relative performance and when interference or selective routing can offset that effect.

Retaining the covariance contribution is particularly useful: a router can select systematically easier or harder examples within the same task, so marginal assignment probabilities alone need not explain expert learning. The framework accommodates this effect instead of assuming it away in the general result. Proposition 4.6 translates the decomposition into interpretable sufficient conditions. This is the paper’s clearest technical contribution, even though its implications remain conditional.

S2. Several experiments directly examine the quantities appearing in the theory.

The experiments do more than report aggregate loss. Figure 1(b) compares a finite-difference estimate of advantage growth with both terms in the decomposition, and Figure 1(d) compares the measured derivative with the lower bound. Figure 2 reports routing and task-risk matrices together. In its strong-bias setting, diagonal routing probabilities are approximately 0.61–0.66 and corresponding task risks are approximately 0.34–0.39, compared with off-diagonal risks around 0.61–0.66. Thus the observed routing structure is accompanied by task-specific performance differences (page 9).

The supplementary evidence is also relevant. Table 3 reports router-velocity scaling slopes of approximately 0.95–0.96 and non-dominant-expert velocity slopes near 1.00 across four task separations, supporting Proposition 4.7’s predicted near-collapse scaling (page 22). Table 1 varies routing temperature; Table 2 includes two task-similarity settings; and Figure 5(b) examines expert widths from 16 to 256 against a width-512 reference. Section 5.1 reports five seeds for the general experiments. These checks deserve credit and make a blanket criticism of “insufficient experiments” too imprecise.

Critical weaknesses

W1. The specialization result is conditional, and the experiments do not isolate the feedback mechanism.

Scope of the result. Proposition 4.6 assumes a positive task-specific routing margin, controlled gradient magnitudes and cross-task interactions, and a favorable residual-adjusted lower bound at every time in the interval under consideration (page 6). It establishes advantage growth while those conditions hold. It does not establish that joint training creates the margin or preserves the required conditions from initialization. Likewise, Proposition 4.7 bounds velocities under an already concentrated router and small gate derivatives; it does not prove attraction into collapse from a broader set of initial conditions. These are meaningful local or conditional conclusions, but they are weaker than a general account of how the coupled system generates its specialization and collapse regimes.

What the specialization experiment establishes. The main intervention is the strength of an imposed task-specific routing bias. Table 4 shows that, from the weakest to the strongest bias condition, final diagonal assignment rises from approximately 0.289 to 0.642 and the reported advantage rises from approximately 0.025 to 0.266, after accounting for the table’s scale (page 23). This supports amplification of an existing allocation preference. However, training experts under different fixed task mixtures can also produce differentiated performance. The paper does not include a matched fixed-router control that isolates the additional contribution of adaptive routing to this outcome. This is consequential because bidirectional feedback is the central explanatory claim.

Predictive value of the criterion. Table 2 usefully reports where the sufficient condition is satisfied: at task similarity zero, the fraction of sampled trajectory points with a positive bound rises from 1.63% to 82.72% across the listed bias settings (page 22). However, the bound is evaluated using quantities from the realized trajectory. The table does not establish an interval of guaranteed persistence from initial conditions or demonstrate prediction on a separate configuration. The two similarity settings are a useful start, but a controlled comparison reporting the actual task-gradient geometry, while holding initial allocation fixed, would better test the proposed mechanism. Failure of a sufficient condition should, of course, not be interpreted as a prediction that specialization cannot occur.

Scope of the mean-field and routing claims. Figure 5(b) presents one representative seed and reports qualitative agreement across seeds, but does not provide aggregate width-dependent errors or establish that width 512 accurately represents the infinite-width limit (pages 22 and 24). This is illustrative support rather than a quantitative finite-width validation. More fundamentally, Equation 5 averages individual-expert losses under routing weights. For nonlinear losses, this generally differs from evaluating the loss after combining several expert outputs. The statement on page 3 that the weights can describe Top-k routing therefore needs qualification. The applicability to common multi-expert aggregation objectives is not established by the current experiments.

W2. The collapse-recovery claim has incomplete experimental and theoretical support.

Redistribution is demonstrated; recovery of expert specialization is not. Appendix B.1 explicitly freezes the experts in the load-balancing experiment and updates only the router (page 20, lines 1064–1073). Table 5 shows substantial redistribution: for task separation 2.0, final imbalance decreases from approximately 0.3276 without regularization to 0.0093 at regularization strength 8, while probability assigned away from the initially dominant expert rises from approximately 0.049 to 0.643 (page 23). These results support the narrow claim that the penalty changes routing. They cannot demonstrate renewed expert learning or recovery of task-specific advantages. Proposition 4.8 is itself a conditional local stationarity or curvature argument, not a guarantee of useful specialization after escape; Appendix C.17 also makes the additional requirements for interpreting a routing instability as a joint-system instability explicit (page 35).

The quantitative stability prediction is not directly validated. Figure 5(a) supplies qualitative radial trajectories at several regularization strengths. Appendix B.2 explicitly states that the critical value is not drawn (pages 21–22). The paper does not compare a threshold calculated from the theory with an empirically located transition across several task settings. A nonzero routing plateau also requires task-performance evidence before it can be interpreted as useful specialization. The current results therefore support a qualitative regularization effect more strongly than a validated quantitative transition criterion.

The radial reduction has unresolved prerequisites. Proposition 4.2 requires uniform exponential attraction of the frozen-router expert dynamics to an equilibrium branch. Proposition A.6 and its proof invoke this reduction uniformly near the collapsed boundary (pages 16 and 26). Yet Proposition A.5 states that the unused expert is stationary at that boundary because its load is zero (page 15). A frozen, unloaded expert cannot uniformly contract from neighboring expert states to a single equilibrium branch. A separate argument, restricted domain, or revised reduction is needed to justify applying the fast-expert approximation there. This does not invalidate a conditional stability calculation for an assumed reduced model, but it leaves its connection to the original coupled system unestablished.

There is a second issue in the static expert-response calculation. Proposition A.12 defines an averaged task response, and the risk immediately preceding Equation 77 squares the residual of that averaged response (pages 18–19). The actual risk under a mixture distribution averages the component task losses. The latter includes an additional nonnegative between-task response-variance term that generally depends on the expert parameters. Consequently, the difference need not disappear from the expert gradients or minimizers. The response approximation checked in Figure 6 does not, by itself, validate an approximation to the original mixture objective. The authors should establish which conclusions survive after retaining this term or explicitly restrict the analysis to the surrogate objective.

Table 5 contains a conditional reporting inconsistency. Appendix B.1 describes the final column as the initial regularized routing-gradient norm and describes the experiment as starting from the same collapsed condition. If the router, experts, and empirical data are identical across regularization strengths within each seed, the gradient depends affinely on regularization strength, and its norms must obey the corresponding triangle-inequality constraint. For task separation 2.0, the reported means at strengths 0 and 2 imply that the mean norm at strength 4 must be at least 907.02 in the table’s displayed units; the table reports 416.25. For separation 3.8, the corresponding lower bound is 616.17, versus a reported 159.36. These constraints also hold after averaging paired seeds. Different initial states, a different evaluation time, or a different metric could alter this interpretation, so clarification is necessary before using this column as evidence.

Questions for authors

Q1. What does joint learning add beyond a fixed routing preference?

Clarification and targeted validation for W1: Which result establishes persistence of Proposition 4.6’s conditions from initialization? A matched comparison of joint training and fixed-router training, using the same initial allocations and expert states, would help isolate the contribution of adaptive routing. Please also report the measured task-gradient geometry under controlled task-similarity changes and clarify the applicability of Equation 5 to stochastic single-expert selection versus losses applied to aggregated Top-k outputs.

Q2. Does load balancing restore useful specialization, and does the predicted threshold match the transition?

Targeted validation for W2: Please show recovery from a near-collapsed state with both experts and router updating, reporting task-risk and routing matrices alongside load imbalance. For the radial model, can the theoretical critical regularization strength be calculated and compared with an empirically located transition across multiple task settings, with variation across seeds where relevant? This would directly test the claimed transition mechanism.

Q3. How are the radial reductions justified near collapse?

Technical clarification for W2: Please reconcile the uniform attraction required by Proposition 4.2 with the zero-load expert in Proposition A.5, particularly the uniform reduction invoked in Appendix C.3. Separately, please reconcile the risk preceding Equation 77 with the mixture of the component task risks, account for the between-task response-variance term, and identify any resulting changes to the expert response map and stability interpretation.

Q4. What exactly is measured in Table 5?

Reporting clarification for W2: Are the gradient norms evaluated at initialization, with identical router parameters, expert parameters, and data across regularization strengths within each seed? Please explain the apparent inconsistency quantified in W2, specify any normalization, and correct the metric label or protocol description if the measurements are taken elsewhere along the trajectory.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.