acceptodds
Under review as a conference paper at ICLR 2027

High-Dimensional Dynamics of Mixture-of-Experts: Phase Transitions, Routing, and Specialization

Abstract

Mixture-of-Experts (MoE) architectures have been pivotal in scaling modern language models, yet the coupled evolution of their routing mechanisms and expert parameters remains underexplored. We give a high-dimensional analysis of the training dynamics of an MoE layer with experts in the feature-learning regime, using dynamical mean-field theory (DMFT) to derive a closed system of integro-differential equations for the router and expert order parameters. The routing nonlinearity yields two differences from single-pathway DMFT: a non-Gaussian effective action, which we close by treating the routing fields as Gaussian, and a router-specific Onsager correction, needed because the router is trained on the examples it routes. We validate the equations against gradient flow at , and and show that their error grows as the routing temperature falls. For softmax routing we identify the gate with a Random Energy Model in a signal-dependent field, and use it to separate two distinct processes: a dynamical onset, where a symmetric saddle amplifies the initialization bias, and a thermodynamic condensation of the router. In gradient flow, cold routers condense onto poorly aligned experts while warm routers stay diffuse and align, and the specialization reached grows with the initialization bias, which in our longest runs is amplified by a finite gain.

Primary area:
learning theory
Keywords:
dynamical mean field theory, mixture of experts, training dynamics, random energy model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_quftOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies joint router-expert gradient-flow dynamics in a high-dimensional MoE with EE experts trained on an EE-cluster Gaussian teacher task. It derives a symmetry-reduced DMFT with memory kernels and a router-specific Onsager correction, and interprets softmax routing as a signal-dependent REM with a noise-driven freezing scale and a signal threshold for matched-expert condensation. Experiments compare DMFT and REM predictions with gradient-flow simulations for E=2,4,8E=2,4,8, plus bare-softmax statistics at larger EE. They support the proposed mechanisms within the synthetic model; DMFT agreement is strongest at high temperature and weak alignment, and worsens as routing cools. The work does not establish robustness in less symmetric data or more practical MoE settings.

Critical strengths
  1. A technically substantial coupled-dynamics derivation. Extending DMFT to jointly trained experts and routers, including a router self-reaction term for reused examples, is a meaningful theoretical contribution. Width-scaling comparisons at E=2E=2 provide evidence that parts of the reduced dynamics track the specified model.
  2. A quantitative REM analysis. The signal-dependent REM formulation yields explicit freezing and matched-condensation criteria and is checked against fresh-input routing statistics in the synthetic experiments. This is more substantive than the qualitative observation that concentrated routing can be unhelpful.
Critical weaknesses
  1. The main qualitative claim has limited novelty, and the setting leaves the quantitative novelty narrow. The observation that a concentrated gate need not correspond to semantically useful expert specialization is intuitive. The paper's more novel mathematical claims are the DMFT/Onsager equations and REM thresholds, but they are derived and tested under balanced, orthogonal, isotropic clusters with one expert per cluster. The paper does not establish whether those results persist with correlated directions, imbalanced feature/group frequencies, or anisotropic representations. Language data provide one example of such a mismatch. Sparse top-k routing is also not part of the main validation.
  2. The key dynamical approximation is uncontrolled where the main phenomenon occurs. The second-cumulant closure has no stated small parameter, and its error increases as temperature falls. The freezing diagrams include limited replication, including one-seed cells, so the robustness of the proposed phase boundaries near the transition is uncertain.
Questions for authors
  1. Can the authors provide a robustness experiment with correlated teacher directions, unequal cluster priors, or anisotropic input covariance? Does the predicted distinction between freezing and matched-expert condensation, including the quantitative thresholds, survive?
  2. Can the authors report repeated-seed results and direct closure-error diagnostics near and below the predicted freezing temperature, where the central phase claim is made?
  3. What is the precise advance of the qualitative claim that routing concentration need not imply useful specialization over prior theoretical or empirical observations of router collapse? Is the intended novelty instead the DMFT/Onsager derivation and the REM thresholds, and can the paper sharpen that distinction?
3 comments
AuthorsOpenReview ↗

Response Part 1

We thank the reviewer for their strong reading of the paper and valuable feedback. We are glad that they find our DMFT extension technically substantial, and our quantitative REM analysis more substantial than prior qualitative observations.

W1 - Limited novelty and narrow setting Novelty. We agree that the qualitative point is intuitive and has been made before. The revised contributions cite it as prior work and lead with the quantitative results. However, we provide a detailed response to the exact novelty of our work in the answer to Q3 below.

Less symmetric settings. The REM part holds up in each case you name. With correlated teachers, unequal cluster priors or anisotropic inputs, the REM still predicts routing on fresh inputs to within 0.013 (Appendix L.9), and hard top-1 and top-2 gates show the same two regimes as softmax (Appendix L.7, now also stated in Section 5.2). On language data, the REM with signal and noise measured from the logits of trained Switch Transformers predicts held-out routing to within 0.05 (Appendix L.11).

Not covered. We kept one expert per cluster throughout and did not test more experts than clusters. The dynamical DMFT is still derived only for the symmetric setting, and we name this as the main limitation.

W2 - Uncontrolled closure and limited replication The revision measures the closure directly where the phenomenon occurs and replicates the freezing diagram over four seeds.

Closure. The closure has no small parameter, and Section 4.2 now states what it assumes and what it approximates. Most of the low-temperature error in the submission came from evaluating the router's self-reaction in its linear-gate form, and Section 4.3 now evaluates it with the softmax gate. The new bottom row of Figure 3 shows the closure measured in gradient flow and the DMFT's error across temperature and EE.

Replication. Figure 4 now averages four seeds, and Figure 13 shows how little they spread; the crossover sits at the same temperature in every seed (Appendix L.3).

Q1 - Robustness to less symmetric data Yes, it survives in all three cases, provided the REM inputs are measured from the router-logit noise.

Results. We repeated the test of Appendix F.8, in which the REM predicts each trained router's routing on fresh inputs from its head start ss and effective temperature TeffT_{\mathrm{eff}}, with correlated teachers, unequal cluster priors and a power-law input covariance (Appendix L.9). The error on the matched-expert weight stays at most 0.013 in every case, against 0.0036 in the symmetric model.

What changes. Under anisotropy the router aligns with high-variance directions, so ss and TeffT_{\mathrm{eff}} must be measured from the logit noise; the overlap formula in the paper assumes isotropic inputs and is off by up to 0.08. With unequal priors, frequent clusters align faster, so each cluster has its own head start. That gives a prediction with no counterpart in the symmetric model: rare clusters can fail to condense while common ones specialize. In our runs at κ=3\kappa = 3, the most frequent cluster's head start is well above the threshold while the rarest one's sits right at it.

Scope of the DMFT. The DMFT itself is derived for the symmetric setting, because its reduction to a few order parameters uses expert exchangeability and isotropic inputs. Without them the equations still close, but over per-cluster overlap matrices and spectral functions of the covariance.

Q2 - Repeated seeds and closure diagnostics Seeds. We ran the E=8E = 8 freezing grid of Figure 4 with four seeds (Appendix L.3). Figure 4 now shows their mean and Figure 13 the spread across seeds. The seeds barely differ: the seed-to-seed error in each cell is under 1% of the range across the grid, and the alignment crossover sits at the same temperature, about 0.73 Tc(0)0.73\,T_c^{(0)}, in every seed.

Closure diagnostics. We measured the routing fields directly in softmax gradient flow at E=2E = 2, 4 and 8, from 8 Tc(0)8\,T_c^{(0)} down to 0.35 Tc(0)0.35\,T_c^{(0)} (Appendix L.1). On fresh inputs they are Gaussian throughout (excess kurtosis at most 0.09 in absolute value), as they must be, since a fresh input is independent of the trained weights. On training inputs they are not (excess kurtosis down to −0.80-0.80), and adding each example's self-reaction to the Gaussian fields accounts for the difference: the reconstructed router drive is within 3.0–6.4% of the measured one at Tc(0)T_c^{(0)} and 0.7 Tc(0)0.7\,T_c^{(0)}, where Gaussian fields alone are off by 18–124%, and within 9% at 0.5 Tc(0)0.5\,T_c^{(0)} (Figure 3a).

DMFT error. The DMFT built on this closure is within 1.7% of gradient flow on the router alignment at 0.7 Tc(0)0.7\,T_c^{(0)} (Figure 3b). Below that temperature we did not compare the DMFT with gradient flow, so the phase claims there rest on gradient flow and the REM.

AuthorsOpenReview ↗

Response Part 2

Q3 - Precise advance over prior work Yes, the novelty is the quantitative theory, and the revised contributions now say so, citing the qualitative observation as prior work (Shazeer et al., 2017; Chi et al., 2022). Here, we state the exact contributions we have over prior work.

REM thresholds. Identifying softmax weights with REM Boltzmann weights is known for dense associative memories (Lucibello & Mézard, 2024) and for attention at initialization (Giorlandino & Goldt, 2026), but to our knowledge has not been explored for MoE models, and has not been explored for settings where the logits change during training. A learned router differs because its logits carry a cluster signal that grows during training. This adds a second threshold, condensation onto the matched expert at s∗=2ln⁡(E−1)s_\ast = \sqrt{2\ln(E-1)}, and the bound s≤κs \leq \kappa ties the head start to the data. Together these identify a specific failure, frozen collapse, in which each input is routed confidently and the load stays balanced but the router is poorly aligned with the clusters, so load metrics miss it and a balancing loss leaves it unchanged (Appendix L.6). They also predict when it occurs: at κ=1\kappa = 1 the router never condenses onto the matched expert for E≥3E \geq 3, which holds in gradient flow up to E=32E = 32 (Appendix L.8), and annealing cannot remove the collapse (Appendix L.5).

Dynamics. The DMFT tracks expert specialization and router alignment together, including the router's self-reaction on reused examples, which has no counterpart in single-pathway networks. Kang & Oh (1996) analyze the equilibrium of a mixture of experts, so the question of when training reaches the specialized state does not arise there. The recent DMFTs of MoE layers (Jiang et al., 2026; Vankadara et al., 2026) target hyperparameter transfer and have no specialization order parameter, and Chen et al. (2022) and Kawata et al. (2025) prove that SGD learns the cluster structure without tracking the routing regimes. Finally, the onset of specialization is an instability of the symmetric state, and in gradient flow its gain is finite, so the initial bias sets how far experts specialize.

Thank you again for your time and thoughtful feedback on our submission. As we are approaching the end of the discussion period, we wanted to kindly ask whether our responses and additional experiments have addressed your concerns, and we are happy to answer any remaining questions. If our rebuttal has resolved your concerns, we would be grateful if you would consider reassessing our work given the new results.

Reviewer_hkQVOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper develops a Dynamical Mean-Field Theory (DMFT) for jointly trained routers and experts in a Gaussian-mixture teacher–student MoE model, with model dimensions and sample size growing proportionally. To tracks expert specialization, it introduces a router-specific self-reaction correction to account for differences between routing training data and fresh inputs. It also connects softmax routing to the random energy model, explaining how router temperature and signal strength distinguish useful specialization from concentration onto poorly aligned experts.

Critical strengths

The paper illustrates a clear distinction between expert specialization/alignment versus routing concentration. The REM model provides asymptotic router temperature and signal strength thresholds separating diffuse routing, concentration, and noise-driven collapse. This clarifies questions such as why neither confident routing nor balanced expert loads necessarily imply meaningful specialization.

Critical weaknesses

The central DMFT derivations relies on uncontrolled closures, and there seems to be a nonzero (arguable unjustified) gap between the derived equations and the numerical implementation. As a result, the numerical simulations supports the implemented approximation as opposed to the theoretical derivation themselves.

Questions for authors

The existing analysis starts from a O(1) planted alignment with the teacher, whereas random init produces overlaps of order d^{-1/2}. Given the observed finite amplification of small biases, can one provide any analysis of how nonvanishing specialization may emerge naturally in the high dimensional limit?

2 comments

We thank the reviewer for their effort and understanding of our paper, and are glad that they find our paper strong. We address the weakness and question provided, and have uploaded a revised version of our paper:

W1 - Uncontrolled closures and implementation gap Most of this gap is now closed.

Implementation. Appendix D gives the equations exactly as we solve them, with the router memory in its derived form and the noise pairing derived in every equation but one, and for softmax the router side, including its self-reaction, is derived and evaluated with the softmax gate (Appendix D.9). In the softmax runs the noise pairing and memory weights keep their earlier forms, and switching them changes every order parameter by less than 0.1%. Three approximations made at E=2E = 2 remain: the amplitude kernels leave out the self-reaction; for softmax, the expert-output noise that is correlated with the router noise enters only the population averages; and the two-time correlation between different experts' weights keeps the noise pairing chosen at E=2E = 2, which affects only its values at two different times.

Closure. The routing fields are exactly Gaussian on inputs the router was not trained on. On training inputs the departure from Gaussianity is each example's self-reaction, which the theory includes, and gradient flow confirms that this reproduces the router drive (Section 4.2, Figure 3a). With these equations the softmax DMFT is within 4.5% of gradient flow on every order parameter at eight new settings with E=4E = 4, 6 and 8, down to 0.7 Tc(0)0.7\,T_c^{(0)} (Figure 3c).

Q1 - Specialization from random initialization Our analysis and experiments indicate that on O(1)O(1) time scales it does not: specialization from random initialization vanishes as dd grows, so the planted O(1)O(1) alignment does matter.

Argument. A small bias ε\varepsilon is amplified by a gain that is finite in every run we made (Appendix L.4), so specialization is about the gain times ε\varepsilon. Random initialization gives ε=O(d−1/2)\varepsilon = O(d^{-1/2}), so specialization should vanish in the proportional limit unless the initial bias is of order one or the training time grows with dd.

Experiment. We trained from random initialization at d=1000d = 1000, 2000 and 4000 (three seeds each, to t=1200t = 1200), measuring specialization under the best expert-to-cluster assignment (Appendix L.10). Specialization shrinks with dd, with a log–log slope between −0.53-0.53 and −0.63-0.63 at every time from t=100t = 100 to 1200, close to d−1/2d^{-1/2} and far from the zero slope that order-one specialization would give. It is still growing slowly at the end of these runs, though, so they do not rule out a slower route on time scales that grow with dd.

Thank you again for your time and thoughtful feedback on our submission. As we are approaching the end of the discussion period, we wanted to kindly ask whether our responses and additional experiments have addressed your concerns, and we are happy to answer any remaining questions.

Reviewer_BQ8bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper builds a theory of how the router and the experts in a mixture-of-experts (MoE) layer learn together during training. The setting is a two-layer MoE with E experts and one router, trained by gradient flow on Gaussian-mixture data with one cluster per expert, in the high-dimensional limit. Using dynamical mean-field theory (DMFT), the authors reduce training to a few summary quantities, such as how strongly each expert and the router align with their own cluster, and derive closed equations for how these evolve. Two new pieces are needed for MoE. The first is an approximation that treats the routing fields as Gaussian. The second is a “router Onsager correction” that accounts for the router routing its own training data more confidently than fresh data. The paper then shows that softmax routing is a Random Energy Model. This gives a freezing temperature T_c = 1/√(2 ln E), below which the router confidently picks experts chosen by noise, and a minimum signal √(2 ln(E−1)) needed to lock onto the correct expert. Together these define three regimes: democratic, specialization and frozen collapse.

Critical strengths
  • The Random Energy Model connection is clean and gives simple, interpretable results: a closed-form freezing temperature, a signal threshold, and a phase diagram that explain the known temperature and initialization sensitivity of MoE routing. The predictions are checked against gradient flow.
  • Extending DMFT to a jointly trained router and experts is a real technical step, and the router Onsager correction is a useful insight. Adding it fixes the rise-and-fall shape of the cross-cluster alignment and its predicted routing bias matches gradient flow within 3%.
Critical weaknesses
  • The theory is not fully closed or predictive. The Gaussian closure for the routing fields has no small parameter, and several pieces were chosen by matching gradient flow at E = 2: the router memory kernels and their step size, the noise-term pairing and sign (which differ from the derived form). The router Onsager correction is derived only for the linear router and applied to softmax in its linear-gate form. Accuracy also drops where the main story lives: the softmax error grows from under 1% at high temperature to 4.9–11.8% at 0.7 T_c. So it seems this is partly a calibrated model rather than a fully derived one.
  • The setting is narrow, and the link to practical MoE training is weak. It uses a two-layer model with orthogonal, equally likely clusters, gradient flow, no load-balancing loss, and training dynamics only up to E = 8, with no hard top-k training results and no real MoE or data. At κ = 1 the specialization regime is unreachable, so it appears only at κ = 3 in the appendix. Key claims such as the finite gain of about 17 and long-time frozen collapse rest on single-seed runs.
Questions for authors
  • How well does the DMFT predict settings where nothing was tuned (a different ᾱ, κ or larger E), and with the derived noise pairing and full memory terms?
  • The router self-reaction is applied to softmax in linear-gate form, which overstates the router force at low temperature. How much of the low-temperature error comes from this, and can it be derived for softmax directly?
  • Is frozen collapse permanent? At κ = 1 the router stays frozen through t = 3200, while at κ = 3 it escapes. Would starting warm and annealing, or learning the temperature, avoid it?
  • With 64–256 fine-grained experts, T_c shrinks slowly but the signal threshold keeps growing. Does the theory give a practical rule for router temperature or initialization scale at large E?
  • Real collapse usually shows up as dead experts, while frozen collapse here keeps the load balanced. With the shared-logit component, does training cause load collapse, and would load balancing (an auxiliary loss or bias-based balancing) help?
  • Can the predictions be checked on an open MoE model, for example by measuring router logit spread and Y₂ against E during training?
  • Do the three regimes apply to hard top-k, the most common router in practice, and are there gradient-flow results for it?
  • Can you add seeds for the finite-gain result, and longer runs at ᾱ = 1 and 4, where the gain is still growing at t = 800?
4 comments
AuthorsOpenReview ↗

Response Part 1

We thank the reviewer for their effort and detailed response, and are glad that they find the REM connection interpretable and clean, as well as appreciating the technical contributions on the DMFT side. We address the weaknesses and questions, and have uploaded a revised version of our paper:

W1 - Closure and calibrated pieces

The revised theory evaluates the router side with the softmax gate, its closure can be measured directly in gradient flow, and the DMFT now stays within a few percent of gradient flow down to 0.7 Tc(0)0.7\,T_c^{(0)}. You identified the source of the low-temperature error correctly: we were using the linear-gate approximation of the self-reaction, which fails at low temperatures.

What the closure assumes. On inputs the router was not trained on, the routing fields are exactly Gaussian given the weights, since within a cluster a fresh input is Gaussian and independent of the weights. On training inputs they are not, and the difference is each example's reaction to its own past router forces, the router Onsager correction of Section 4.3, which the theory includes. What we approximate is how this reaction and the expert outputs are evaluated at finite width. Section 4.2 now states this.

Measured in gradient flow. Gaussian fields plus the self-reaction reproduce the router drive on the training set to within 3.0–6.4% at Tc(0)T_c^{(0)} and 0.7 Tc(0)0.7\,T_c^{(0)}, while Gaussian fields alone are off by 18–124% (Figure 3a, Appendix L.1). The resulting DMFT is within 1.7% of gradient flow on the router alignment at 0.7 Tc(0)0.7\,T_c^{(0)} for E=2E = 2, 4 and 8 (Figure 3b). We did not compare the DMFT with gradient flow below 0.7 Tc(0)0.7\,T_c^{(0)}, and the paper rests its claims there on gradient flow and the REM.

Calibrated pieces. Section 4.3 and Appendix D.9 derive the self-reaction for softmax directly, and Appendix D now gives the system exactly as we implement it. For the soft-discriminant router, the router memory takes its derived form, every memory term enters at full weight, and the noise pairing and its sign take their derived forms in every equation but one. The softmax runs keep the earlier pairing and memory weights; switching them to the derived forms changes every order parameter by less than 0.1% (Appendix D). The results no longer depend on the integration step (Appendix L.2). Three choices made at E=2E = 2 remain: the amplitude kernels are evaluated at the population fields without the self-reaction; for softmax, the part of the expert-output noise that is correlated with the router noise enters the population averages but not each example's path; and the two-time correlation between different experts' weights keeps the noise pairing chosen at E=2E = 2. Its equal-time value is fixed by the order parameters, so this pairing acts only on its values at two different times.

W2 - Narrow setting

We have now run most of the cases you list (Appendix L), and we answer them in detail below: seeds and longer runs in Q3 and Q8, larger EE in Q4, load balancing in Q5, trained Switch Transformers in Q6 and hard top-kk in Q7. Figure 4 also now averages four seeds (Appendix L.3).

Less symmetric data. The REM thresholds, which carry the phase-diagram claims, hold with correlated teachers, unequal cluster priors and anisotropic inputs, where the REM predicts routing on fresh inputs to within 0.013 (Appendix L.9). We did not go beyond two-layer experts or gradient flow; the DMFT assumes both, as well as symmetric clusters, and the Limitations list this.

Specialization at κ=1\kappa = 1. Our theory predicts that this regime does not exist. The head start is at most κ\kappa, and condensing onto the matched expert needs a head start above 2ln⁡(E−1)\sqrt{2\ln(E-1)}, which exceeds 1 for every E≥3E \geq 3. At κ=3\kappa = 3 the threshold can be crossed, and gradient flow does specialize (Appendix F.8).

Q1 - Untuned settings Well: at every new softmax setting, every order parameter is within 4.5% of gradient flow.

Softmax, larger EE. We ran the softmax DMFT at eight settings beyond those in the submission: E=4E = 4, 6 and 8 at Tc(0)T_c^{(0)} and 0.7 Tc(0)0.7\,T_c^{(0)}, and E=4E = 4 and 8 at αˉ=4\bar\alpha = 4 and Tc(0)T_c^{(0)} (Figure 3c, Appendix L.2). The gradient-flow references are converged in dd, so these are errors of the theory rather than of finite size. These runs keep the earlier noise pairing and memory weights; switching them to the derived forms changes every order parameter by less than 0.1% (Appendix D).

Other αˉ\bar\alpha and κ\kappa. We varied κ\kappa only for the linear router, at E=2E = 2, over six new (αˉ,κ)(\bar\alpha, \kappa) pairs. The errors stay within 2.3%, except on PcrossP_{\mathrm{cross}}, the smallest quantity, at αˉ=4\bar\alpha = 4, where the error is 9.2% of its peak (but remains very small).

AuthorsOpenReview ↗

Response Part 2

Q2 - Linear-gate self-reaction at low temperature Most of it, and yes.

How much. Against the same gradient-flow runs, the DMFT with the router side, including the self-reaction, in its linear-gate form is off by 12–49% on Δ\Delta and 20–77% on PcrossP_{\mathrm{cross}} at the eight settings of Figure 3c (Appendix L.2).

Derivation for softmax. The revision derives the reaction for softmax directly (Section 4.3, Appendix D.9). A training example's router logits are the logits it would have had without being trained on, plus its accumulated reaction ∫R−v(t,s) cμ(s) ds\int R^v_-(t,s)\, c_\mu(s)\, ds to its own router force cμc_\mu. The linear-gate form linearizes cμc_\mu around the population gate, which overstates it once the gate on the training examples saturates. We now evaluate cμc_\mu with the full softmax at the example's own state, including its own expert outputs. The response R−vR^v_- in this reaction agrees with the response measured directly in gradient flow (Appendix L.1).

Q3 - Permanence of frozen collapse At κ=1\kappa = 1 it does not end in any of our runs, and annealing makes it milder without preventing it, both as the REM predicts.

Mechanism. As in W2, the head start is at most κ\kappa, so at κ=1\kappa = 1 it stays below the condensation threshold (1.97 at E=8E = 8), while at κ=3\kappa = 3 it can cross it, which is why the run in Appendix F escapes.

Longer runs. We continued the three frozen cells to t=3200t = 3200 with two more seeds each, and no run escapes (Appendix L.5).

Annealing. We cooled TT geometrically from 1.5 to 0.10. The router still does not condense: the matched expert gets about a third of the routing weight, below the half that condensation needs. But much of the alignment built while the router was warm survives, and the final router alignment is two to two and a half times what it reaches at constant T=0.10T = 0.10.

Learned temperature. We did not try this, but the same criterion applies: changing TT rescales the signal and the noise together, so it cannot create a head start, only delay freezing until one has grown.

Q4 - Large E Yes. In units of σ\sigma, the noise part of the router logits, the router freezes when σ/T\sigma/T exceeds 2ln⁡E\sqrt{2\ln E}, and it can condense onto the correct expert only if its head start exceeds 2ln⁡(E−1)\sqrt{2\ln(E-1)}:

EE8163264256
2ln⁡E\sqrt{2 \ln E} (freezing scale)2.042.352.632.883.33
2ln⁡(E−1)\sqrt{2 \ln(E-1)} (required head start)1.972.332.622.883.33

The rule. Start in the democratic regime, with σ0/T\sigma_0/T below 2ln⁡E\sqrt{2\ln E}, which a small router initialization gives. Then do not let the temperature schedule or the growing router norm push σ/T\sigma/T past that value before the head start has crossed 2ln⁡(E−1)\sqrt{2\ln(E-1)}. Both thresholds grow only as ln⁡E\sqrt{\ln E}, so going from 8 to 256 experts raises the required signal-to-noise ratio by about 70%.

Gradient flow at E=16E = 16 and 32. Our gradient-flow runs go up to E=32E = 32, so beyond that the rule rests on the thresholds alone. At E=16E = 16 and 32 (Appendix L.8), the crossover to concentrated routing moves with Tc(0)(E)T_c^{(0)}(E), and the REM predicts routing on fresh inputs to within 0.006 in every run. Alignment behaves differently: it is lost at a nearly fixed absolute temperature, so at large EE it is worth monitoring the router's alignment and not only its concentration.

Q5 - Dead experts and load balancing In our setting training does not cause load collapse, and load balancing does not cure frozen collapse.

Shared logits. In the static model of Appendix F.7, where each logit has a part shared across inputs, the load imbalance freezes at the same temperature as per-input routing, and a balancing loss acts only on the shared part, not on the per-input freezing. We did not train with a shared component.

No load collapse in training. Across the E=8E = 8 grid, no expert dies under softmax, Switch top-1 or top-2 routing, and even in the frozen cells the largest load is at most 1.43 times the smallest. Each input is routed confidently, but to an expert picked by the noise, and those picks are spread evenly over inputs.

Balancing losses. For the same reason an auxiliary balancing loss has nothing to act on (Appendix L.6). At weights λ\lambda from 0.01 to 1 it changes the router alignment by at most 0.0012, and the frozen cell stays frozen. We did not test bias-based balancing, which also acts on the load.

Closest analogue of dead experts. With unequal cluster frequencies, rare clusters can fall below the condensation threshold while common ones specialize (Appendix L.9).

AuthorsOpenReview ↗

Response Part 3

Q6 - Open MoE model Partly: we could check them on final checkpoints but not during training.

Setup. We measured the final routers of the google/switch-base-E encoders, which differ only in EE (8, 16 and 32), on WikiText-2 (Appendix L.11). Text has no ground-truth clusters, so we used the word as the cluster: the part of a token's router logits shared by all occurrences of its word is the signal, and the rest is the noise.

Logit spread and Y2Y_2 against EE. The logit spread on its own turns out to be misleading. It is close to the freezing scale 2ln⁡E\sqrt{2\ln E} in most layers, but most of it is signal from the word. In 14 of the 18 layers the noise part is well below the freezing scale and the head start is above s∗s_\ast, so these layers are in the specialization regime, and their Y2Y_2 falls with EE, from about 0.6 at E=8E = 8 to about 0.4 at E=32E = 32. The other four layers carry far more noise and have a head start close to s∗s_\ast. They route confidently, yet about a third of tokens go to an expert other than their word's preferred one, as in frozen collapse.

Prediction. With each word's signal estimated on half of its occurrences, the REM predicts the weight on the preferred expert in the other half to within 0.05 in every layer, while in the noisy layers the signal alone would miss by about a third.

Q7 - Hard top-k Yes for the two regimes reachable at κ=1\kappa = 1: both hard gates behave like softmax (Appendix L.7).

Setup. We ran gradient flow on the E=8E = 8 grid of Figure 4 with Switch top-1 routing, which weights the selected expert by its softmax probability, and with top-2 routing, which renormalizes the softmax over the two largest logits. Both router forces agree with automatic differentiation.

Results. Both gates show the same democratic and frozen regimes as softmax. At T=0.10T = 0.10 the router is almost fully concentrated, but its alignment is less than half of what it reaches at T=1.5T = 1.5, and the loads stay balanced.

Specialization. We expect the third regime to behave as with softmax, since the head start is a property of the logits, which all three gates share; we did not run the hard gates at κ=3\kappa = 3.

Q8 - Seeds and longer runs Yes. We ran five seeds at αˉ=8\bar\alpha = 8 and three at the other settings, and continued every run to t=2400t = 2400 (Appendix L.4, Figure 14). The gain is finite in every seed. At αˉ=8\bar\alpha = 8 it is 13.5±1.113.5 \pm 1.1 (standard error), between 10.7 and 16.8; the value of about 17 we reported came from a single run, which turned out to be the highest of the five. At αˉ=1\bar\alpha = 1 and 4 the growth has stopped by t=2400t = 2400, at smaller gains.

Thank you again for your time and thoughtful feedback on our submission. As we are approaching the end of the discussion period, we wanted to kindly ask whether our responses and additional experiments have addressed your concerns, and we are happy to answer any remaining questions. If our rebuttal has resolved your concerns, we would be grateful if you would consider reassessing our work given the new results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.