SPECIALIZATION AND SLOW RECOVERY IN JOINTLY TRAINED MIXTURES OF EXPERTS

Marginal load balance alone does not determine whether experts specialize; the prediction objective also matters. We study a family of squared losses that interpolates between selected-expert and averaged-output training. For two jointly trained scalar linear experts with a binary routing cue, we prove a global phase diagram under positive balancing. When $0<q<1$, the thresholds $k=q{g}^2$ and $k=g^2/q$ separate full specialization, stable partial specialization, and homogenization. Here $k=\lambda \chi$ combines balancing strength with batch-cue imbalance, and $g$ is the cue-conditional regression signal. For every $q>0$, expert parameters remain bounded, asymptotic starvation is excluded, and every trajectory from a finite initialization converges to a single state. At $q=1$, each trajectory at the critical threshold selects one point on a stationary continuum. For multiple smooth experts with fixed-feature softmax routing, we derive a local Hessian criterion that separates expert-separation curvature, routing–residual coupling, and balancing curvature. At $q=0$, the binary linear objective has no finite minimizer when the routing-contrast penalty is positive. For initial conditions in a specified invariant set, expert parameters diverge while routing approaches uniformity and cue-dependent prediction persists. For nonlinear experts, curvature can alter stability at the $q=0$ boundary. For selected-expert training, we also prove matching population load-recovery bounds of order $e^{M/T}$ from an initial logit gap $M$. For simultaneous no-baseline categorical updates, a corrected potential controls shared router–expert noise and yields a recovery-time second-moment bound under explicit bounded-data and step-size conditions.