- Several mathematical errors and inconsistencies in the theoretical derivations, particularly in the Gaussian example.
My primary concern is the mathematical correctness of several results in the appendix. First, the infinite-width risk defined in Eq. (43) is not generally equivalent to the expected squared loss of the original MoE model. It measures the squared discrepancy between conditional mean predictions and target labels, while the actual expected squared loss also contains conditional prediction variance terms. The authors do not justify eliminating these terms, meaning that zero-risk solutions of Eq. (43) need not be zero-risk solutions of the original problem. Moreover, the treatment of task mixtures introduces another inconsistency: Eqs. (41)–(43) square the averaged task responses, whereas Eq. (53) averages task-wise squared losses. The difference is generally nonzero and equals 2w(1−w)[(m1+−m2+)2+(m1−−m2−)2], where mi±=∫qi±(r)ρ(dr). Consequently, Proposition A.12 characterizes an approximation error for a surrogate risk rather than the original mixture risk.
Second, Proposition A.11 contains an incorrect equality in Eq. (74). The authors define βi as the difference between an expert's residual vector and that of a reference single-point solution, but then identify the task risk with ∣βi∣22/2. The correct expression is 2Ri∞(ρj)=∣βi+αi(δri∗)∣22, which contains additional baseline and cross terms. These terms are generally nonzero, particularly when Δi<Δ∞, where Lemma D.1 states that the minimum task risk is strictly positive. The expression involving ∣γi∣22 in the same proposition has a similar problem, as γi measures a difference between expert responses rather than the actual prediction residual. The proof in Appendix C.6 does not establish the claimed equalities.
Third, the zero-risk claims in Lemma D.1 and Proposition A.10 are false at an explicitly permitted boundary value. The assumptions allow Δ=1 and require q(0)<−1. However, at Δ=1, we have q−(r;1)=q(0) for every r, implying R∞(ρ;1)≥21(1+q(0))2>0 for every probability measure ρ. This contradicts the claimed existence of zero-risk measures for all Δ≥Δ∞. The underlying proof incorrectly uses γ(+∞;Δ)=(q(+∞),q(+∞)) at Δ=1, and the claimed set equality D=F in Proposition A.9 consequently fails as stated. There are also issues with other boundary cases: the proof of Lemma D.1 uses a strict monotonicity assumption that does not cover Δ=0, and Proposition A.5 does not properly address the probability mass at the origin when Δi=1 in the hard-routing limit.
Finally, Proposition A.14 does not establish the existence of a finite optimal routing threshold under its stated assumptions. The routing cost may approach its infimum only as s→∞. The proof introduces an additional non-degeneracy condition to exclude this possibility, but the condition is missing from the proposition itself. Taken together, these errors are concerning because the Gaussian example serves as the main tractable instantiation of the proposed framework, and several subsequent conclusions rely on its risk characterization.
- The main conclusions about specialization and routing collapse are not sufficiently justified by the assumptions or experiments.
A particularly serious issue concerns the quasi-static approximation used to derive the collapse-stability threshold in Proposition A.6. This analysis relies on Proposition 4.2, which assumes a uniformly exponentially attracting expert equilibrium branch. However, as s→0+, the inactive expert's load satisfies m1(s)=O(s2d), meaning that its optimization speed vanishes near collapse. At s=0, every distribution ρ1 is stationary because the expert receives no training signal. Therefore, the required uniform attraction to a particular equilibrium measure cannot hold at the collapsed boundary. Furthermore, the existence of multiple zero-risk expert measures in some regimes raises additional questions about the uniqueness and stability of the equilibrium branch. Consequently, the reduction underlying the proposed stability threshold Cλ=Λ(0)+2λ(κ1+κ2) is not justified near the very state it aims to characterize.
The collapse-related propositions also support weaker conclusions than the paper suggests. Proposition A.5 describes s=0 as a collapsed fixed point, although the active expert continues evolving; it is an invariant set rather than a fixed point of the complete joint system. Proposition 4.7 obtains near-collapse velocity bounds only by additionally assuming that the routing gradients are already O(ϵ), which does not follow from concentrated routing probabilities alone. Proposition 4.8 proves a conditional local instability result but does not guarantee that load balancing restores specialization. Under the exact-collapse conditions of Lemma D.3, ∇θπj=0 implies ∇θA(θc)=0, so the first destabilization mechanism in Proposition 4.8 cannot apply to that collapsed state. The second mechanism requires a negative-curvature direction that is assumed rather than established. More importantly, destabilizing a collapsed routing state does not necessarily imply convergence to a specialized equilibrium.
Another concern is the gap between the analyzed model and practical MoE training. Eq. (5) optimizes a weighted sum of individual expert losses, which differs from the objective of a conventional soft-output MoE that combines expert predictions before evaluating the loss. The Gaussian example introduces an additional simplification by replacing input-dependent routing with an auxiliary router that is conditionally independent of the input given task identity. For overlapping task distributions, this behavior is generally not realizable by the original input-only router. These simplifications remove important interactions between routing and expert learning, making it unclear whether the theoretical conclusions extend to practical MoE architectures. The paper also does not rigorously establish convergence from finite-width joint training to the proposed mean-field dynamics, including the required parameter and learning-rate scaling.
The experimental evidence is similarly limited. Most experiments verify algebraic identities or conditional inequalities of the assumed gradient-flow system rather than demonstrate why specialization emerges or remains stable. In particular, the load-balancing experiments freeze the experts, so they cannot demonstrate recovery of expert specialization through joint routing–expert feedback. The experiments are restricted to synthetic tasks, and the numerical scales in Table 2 appear inconsistent with Figure 1 unless an additional normalization factor is applied. Overall, the paper provides an interesting formal description of routing–expert interactions, but it has not convincingly established that the proposed dynamics explain the emergence, stability, or recovery of specialization in actual MoE training.