- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies joint router-expert gradient-flow dynamics in a high-dimensional MoE with experts trained on an -cluster Gaussian teacher task. It derives a symmetry-reduced DMFT with memory kernels and a router-specific Onsager correction, and interprets softmax routing as a signal-dependent REM with a noise-driven freezing scale and a signal threshold for matched-expert condensation. Experiments compare DMFT and REM predictions with gradient-flow simulations for , plus bare-softmax statistics at larger . They support the proposed mechanisms within the synthetic model; DMFT agreement is strongest at high temperature and weak alignment, and worsens as routing cools. The work does not establish robustness in less symmetric data or more practical MoE settings.
Critical strengths
- A technically substantial coupled-dynamics derivation. Extending DMFT to jointly trained experts and routers, including a router self-reaction term for reused examples, is a meaningful theoretical contribution. Width-scaling comparisons at provide evidence that parts of the reduced dynamics track the specified model.
- A quantitative REM analysis. The signal-dependent REM formulation yields explicit freezing and matched-condensation criteria and is checked against fresh-input routing statistics in the synthetic experiments. This is more substantive than the qualitative observation that concentrated routing can be unhelpful.
Critical weaknesses
- The main qualitative claim has limited novelty, and the setting leaves the quantitative novelty narrow. The observation that a concentrated gate need not correspond to semantically useful expert specialization is intuitive. The paper's more novel mathematical claims are the DMFT/Onsager equations and REM thresholds, but they are derived and tested under balanced, orthogonal, isotropic clusters with one expert per cluster. The paper does not establish whether those results persist with correlated directions, imbalanced feature/group frequencies, or anisotropic representations. Language data provide one example of such a mismatch. Sparse top-k routing is also not part of the main validation.
- The key dynamical approximation is uncontrolled where the main phenomenon occurs. The second-cumulant closure has no stated small parameter, and its error increases as temperature falls. The freezing diagrams include limited replication, including one-seed cells, so the robustness of the proposed phase boundaries near the transition is uncertain.
Questions for authors
- Can the authors provide a robustness experiment with correlated teacher directions, unequal cluster priors, or anisotropic input covariance? Does the predicted distinction between freezing and matched-expert condensation, including the quantitative thresholds, survive?
- Can the authors report repeated-seed results and direct closure-error diagnostics near and below the predicted freezing temperature, where the central phase claim is made?
- What is the precise advance of the qualitative claim that routing concentration need not imply useful specialization over prior theoretical or empirical observations of router collapse? Is the intended novelty instead the DMFT/Onsager derivation and the REM thresholds, and can the paper sharpen that distinction?
3 comments
Response Part 2
Q3 - Precise advance over prior work Yes, the novelty is the quantitative theory, and the revised contributions now say so, citing the qualitative observation as prior work (Shazeer et al., 2017; Chi et al., 2022). Here, we state the exact contributions we have over prior work.
REM thresholds. Identifying softmax weights with REM Boltzmann weights is known for dense associative memories (Lucibello & Mézard, 2024) and for attention at initialization (Giorlandino & Goldt, 2026), but to our knowledge has not been explored for MoE models, and has not been explored for settings where the logits change during training. A learned router differs because its logits carry a cluster signal that grows during training. This adds a second threshold, condensation onto the matched expert at , and the bound ties the head start to the data. Together these identify a specific failure, frozen collapse, in which each input is routed confidently and the load stays balanced but the router is poorly aligned with the clusters, so load metrics miss it and a balancing loss leaves it unchanged (Appendix L.6). They also predict when it occurs: at the router never condenses onto the matched expert for , which holds in gradient flow up to (Appendix L.8), and annealing cannot remove the collapse (Appendix L.5).
Dynamics. The DMFT tracks expert specialization and router alignment together, including the router's self-reaction on reused examples, which has no counterpart in single-pathway networks. Kang & Oh (1996) analyze the equilibrium of a mixture of experts, so the question of when training reaches the specialized state does not arise there. The recent DMFTs of MoE layers (Jiang et al., 2026; Vankadara et al., 2026) target hyperparameter transfer and have no specialization order parameter, and Chen et al. (2022) and Kawata et al. (2025) prove that SGD learns the cluster structure without tracking the routing regimes. Finally, the onset of specialization is an instability of the symmetric state, and in gradient flow its gain is finite, so the initial bias sets how far experts specialize.
Thank you again for your time and thoughtful feedback on our submission. As we are approaching the end of the discussion period, we wanted to kindly ask whether our responses and additional experiments have addressed your concerns, and we are happy to answer any remaining questions. If our rebuttal has resolved your concerns, we would be grateful if you would consider reassessing our work given the new results.
Response Part 1
We thank the reviewer for their strong reading of the paper and valuable feedback. We are glad that they find our DMFT extension technically substantial, and our quantitative REM analysis more substantial than prior qualitative observations.
W1 - Limited novelty and narrow setting Novelty. We agree that the qualitative point is intuitive and has been made before. The revised contributions cite it as prior work and lead with the quantitative results. However, we provide a detailed response to the exact novelty of our work in the answer to Q3 below.
Less symmetric settings. The REM part holds up in each case you name. With correlated teachers, unequal cluster priors or anisotropic inputs, the REM still predicts routing on fresh inputs to within 0.013 (Appendix L.9), and hard top-1 and top-2 gates show the same two regimes as softmax (Appendix L.7, now also stated in Section 5.2). On language data, the REM with signal and noise measured from the logits of trained Switch Transformers predicts held-out routing to within 0.05 (Appendix L.11).
Not covered. We kept one expert per cluster throughout and did not test more experts than clusters. The dynamical DMFT is still derived only for the symmetric setting, and we name this as the main limitation.
W2 - Uncontrolled closure and limited replication The revision measures the closure directly where the phenomenon occurs and replicates the freezing diagram over four seeds.
Closure. The closure has no small parameter, and Section 4.2 now states what it assumes and what it approximates. Most of the low-temperature error in the submission came from evaluating the router's self-reaction in its linear-gate form, and Section 4.3 now evaluates it with the softmax gate. The new bottom row of Figure 3 shows the closure measured in gradient flow and the DMFT's error across temperature and E.
Replication. Figure 4 now averages four seeds, and Figure 13 shows how little they spread; the crossover sits at the same temperature in every seed (Appendix L.3).
Q1 - Robustness to less symmetric data Yes, it survives in all three cases, provided the REM inputs are measured from the router-logit noise.
Results. We repeated the test of Appendix F.8, in which the REM predicts each trained router's routing on fresh inputs from its head start s and effective temperature Teff, with correlated teachers, unequal cluster priors and a power-law input covariance (Appendix L.9). The error on the matched-expert weight stays at most 0.013 in every case, against 0.0036 in the symmetric model.
What changes. Under anisotropy the router aligns with high-variance directions, so s and Teff must be measured from the logit noise; the overlap formula in the paper assumes isotropic inputs and is off by up to 0.08. With unequal priors, frequent clusters align faster, so each cluster has its own head start. That gives a prediction with no counterpart in the symmetric model: rare clusters can fail to condense while common ones specialize. In our runs at κ=3, the most frequent cluster's head start is well above the threshold while the rarest one's sits right at it.
Scope of the DMFT. The DMFT itself is derived for the symmetric setting, because its reduction to a few order parameters uses expert exchangeability and isotropic inputs. Without them the equations still close, but over per-cluster overlap matrices and spectral functions of the covariance.
Q2 - Repeated seeds and closure diagnostics Seeds. We ran the E=8 freezing grid of Figure 4 with four seeds (Appendix L.3). Figure 4 now shows their mean and Figure 13 the spread across seeds. The seeds barely differ: the seed-to-seed error in each cell is under 1% of the range across the grid, and the alignment crossover sits at the same temperature, about 0.73Tc(0), in every seed.
Closure diagnostics. We measured the routing fields directly in softmax gradient flow at E=2, 4 and 8, from 8Tc(0) down to 0.35Tc(0) (Appendix L.1). On fresh inputs they are Gaussian throughout (excess kurtosis at most 0.09 in absolute value), as they must be, since a fresh input is independent of the trained weights. On training inputs they are not (excess kurtosis down to −0.80), and adding each example's self-reaction to the Gaussian fields accounts for the difference: the reconstructed router drive is within 3.0–6.4% of the measured one at Tc(0) and 0.7Tc(0), where Gaussian fields alone are off by 18–124%, and within 9% at 0.5Tc(0) (Figure 3a).
DMFT error. The DMFT built on this closure is within 1.7% of gradient flow on the router alignment at 0.7Tc(0) (Figure 3b). Below that temperature we did not compare the DMFT with gradient flow, so the phase claims there rest on gradient flow and the REM.