acceptodds
Under review as a conference paper at ICLR 2027

Square-Root Gradient Routing: Geometrical Understanding of Router Optimization in Sparse MoEs

Abstract

Sparse Mixture-of-Experts (MoE) is widely adopted to scale modern large language models. The router activates a small fraction of experts for each token, playing a critical role towards efficient and effective scaling. However, the router's dynamics during MoE training remains poorly understood. In this work, we rigorously study the routing dynamics through a tractable piecewise regression problem, where we simplify the routing with a boundary partition model. Our theory analytically proves that the loss objective is locally cubic in boundary displacement, yielding a quadratically vanishing gradient. Both gradient descent and Adam consequently perform polynomial convergence , while a signed square-root gradient update (SqrtGD) achieves geometric convergence with exponential loss decay . Inspired by the mechanistic understanding, we propose the square-root gradient routing (SR) on MoE-based LLM and evaluate in LLM pretraining. Our experiments on FineWeb demonstrate that the benefits from SR increase with the sparsity of routing adopted in MoE, while its effectiveness is also coupled with the decision on gating mechanism and base optimizer choices. Our findings establish a rigorous mechanistic understanding of the routing dynamics in mixture-of-experts models, which further inspires practical optimizer design for MoE-based foundation models with high sparsity.

Primary area:
optimization
Keywords:
Mixture-of-Expert, Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PWYtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies router optimization in sparse MoE models. It analyzes a toy regression problem with a hard routing boundary, affine experts, and a continuous kinked target. Experts are refitted after each boundary move.

The loss grows as ∣δ∣3|\delta|^3 near the optimum. The gradient therefore vanishes quadratically. Gradient descent and Adam with fixed epsilon converge slowly. The authors propose SqrtGD. It applies a signed square root to the gradient and gives geometric convergence in the toy setting.

The practical method is SR. It keeps the base optimizer for most parameters. For the router, it takes momentum utu_t and applies a signed square root elementwise. The update is about −η sign⁡(ut)∣ut∣.-\eta \, \operatorname{sign}(u_t)\sqrt{|u_t|}. SR helps with top 2 routing in 1.3B MoE LLM experiments. It gives little gain with top 4 or top 8. It fails with AdamW when router gates are renormalized.

Critical strengths

S1. Clear theory for a simple routing problem.

The paper explains why router gradients can vanish near a boundary. Theorem 1 gives a clean cubic loss mechanism. The proofs look correct. The simulations match the predicted rates. This is the main contribution.

S2. Simple method with strong top 2 results.

SR is cheap to implement. With top 2 routing, AdamW+SR reaches validation loss 3.203. The best AdamW baseline reaches 3.320. The gain is 0.117. SR is also stable where AdamW diverges. The authors honestly report failures with renormalized gates.

Critical weaknesses

W1. The theory does not fully explain the LLM setup.

The Adam analysis depends strongly on epsilon. The t−3t^{-3} regime assumes v^\sqrt{\hat v} is much smaller than epsilon. Synthetic tests use epsilon 0.1. Standard AdamW uses 10−810^{-8}. With 10−810^{-8}, Adam can become much faster in practice.

A simple scalar simulation shows this. After 1000 steps, Adam with epsilon 10−810^{-8} reaches ∣δ∣|\delta| around 2×10−102\times10^{-10}. Adam with epsilon 0.1 reaches around 5×10−45\times10^{-4}. SqrtGD is still faster, but the gap is not simply polynomial versus geometric convergence.

The toy assumptions are also strong. The target must be continuous at the boundary. Experts must be refitted to least squares optima. LLM routers use soft gates and top k selection is non differentiable. Experts are updated throughout training. The paper does not show that LLM router gradients follow Theorem 1.

W2. The LLM experiments do not isolate why SR helps.

Table 1 uses one learning rate for all runs. The AdamW top 2 baseline diverges at that rate. Muon also seems poorly tuned. With tuned learning rates, the top 2 gain is 0.117 for AdamW but only 0.029 for Muon. Gains at top 4 and top 8 are small or absent. No random seeds or variance are reported.

The key missing baseline is a separately tuned router learning rate. SR is not scale invariant. Its update size is η∣u∣\eta\sqrt{|u|}. It may simply reduce the effective router learning rate. This could explain both lower loss and better stability.

The paper should compare SR with AdamW using a tuned router learning rate. It should also test sign updates or plain momentum for the router.

The renormalized gate failure is serious. With Mixtral style gates, AdamW+SR diverges under top 2 routing. Under top 4 it is 0.41 worse than the baseline. The paper does not analyze this enough.

Questions for authors
  1. What is the effective router learning rate under SR? Please report typical ∣ut∣|u_t| values and update magnitudes. Compare SR with AdamW using a separately tuned router learning rate. Also test identity, sign, and square root transforms.

  2. How many random seeds were used? Please report variance for Tables 1 and 5. Were top 1 and E∈{8,16}E \in \{8,16\} runs trained? Top 1 is closest to the hard routing theory.

  3. Which learning rates were used for Muon? Table 1 says η=10−3\eta=10^{-3}. The Muon numbers match Table 5 at matrix learning rate 0.02. Please clarify router and non router learning rates.

  4. Is the mechanism visible in LLM training? Do router gradients shrink as routing stabilizes? What happens before divergence in renormalized gate runs?

  5. Does the t−3t^{-3} rate hold for Adam with epsilon 10−810^{-8}? If not, please limit the scope of Theorem 4 and revise the claims.

  6. How does Theorem 1 relate to change point regression? Please clarify the novelty relative to breakpoint estimation literature.

6 comments
AuthorsOpenReview ↗

Rebuttal 1/3

We sincerely thank the reviewer for the insightful feedback, and we really appreciate your close reading through our theory and carefully assessing our experiments! We respond to each of your concerns below.

Q1: Independent learning rate sweep on the router

We conducted experiments with E=32E=32 top-2 routing setup with an independently tuned router learning rate. All non-router parameters use a peak learning rate 3×10−43\times 10^{-4}, as the optimum selected for plain AdamW according to Table 5 in the paper.

We report the results in our paper revision (Appendix E.3, Figure 9, Table 8). Firstly, we do observe that the untuned learning rate on the router is not optimal for both AdamW and AdamW+SR; second, SR still outperforms AdamW at the optimum of the router's learning rate. We also provide the statistics below.

Router updaterouter LRval. loss
AdamW9×10^{-6}$3.355
AdamW3×10−53×10^{-5}3.332
AdamW9×10−59×10^{-5}3.323
AdamW3×10−43×10^{-4}3.342
AdamW9×10−49×10^{-4}3.338
SR9×10−59×10^{-5}3.318
SR3×10−43×10^{-4}3.319
SR9×10−49×10^{-4}3.307
SR3×10−33×10^{-3}3.312
SR9×10−39×10^{-3}3.373

##Q2: Top-1 hard-routing results and multiple random seeds

Top-1 Results. We conducted experiments with E=16 and E=32 experts, top-1 routing. In both cases, SR outperforms plain AdamW and Muon, where SRSR consistently outperforming AdamW and Muon.

Router updateE=16E=32
AdamW3.6085.909
SR (on AdamW)3.2883.297
Muon3.4853.483
SR (on Muon)3.4583.465

Random Seeds We run E=32E=32, top-2 experiments with three random seeds and report the statistics in the paper revision (Appendix E.2, Table 6). According to the results, AdamW with SR is consistently more stable than plain AdamW and achieves lower validation losses than AdamW with renormalization. We present the statistics below.

Recipeseed 0seed 1seed 2
AdamW6.5505.7417.416
SR3.2033.1733.218
AdamW, renormalized gate3.2553.2933.282

It’s hard to get all setups (38 pretraining setups x 3 seeds) to run with multiple random seeds during the rebuttal period, but we promise to provide the variance number in our main Table in our later revision.

Q3: Learning rate on Muon

Thanks for pointing it out, and it’s our mistake that caused the confusion! 10−310^{-3} and 0.020.02 are relatively applied on the non-matrices parameter groups and matrix parameter groups in Muon. We have clarified it in the revision.

AuthorsOpenReview ↗

Rebuttal 2/3

Q4: Is the mechanism visible in LLM training?

(1) What happens on divergence

In Appendix E.5, we measure the router dynamics on the plain AdamW run with un-renormalized gating at top-2 to analyze the routing mechanism behind the model divergence. The first loss spike (around step 8500) comes with a routing collapse in layer 0 and layer 1 (Figure 10 (a) and (b)): the load entropy values fall from around 2.2 (layer 1) / 1.4 (layer 0) to close to log 2, the value when two experts receive all tokens. Around step 14500, the loss diverges with a second spike, and at the same evaluation, every layer routes all tokens to two experts (Figure 10, (b)). Therefore, the router collapse corresponds clearly with the loss divergence.

Observational mechanistic comparison between AdamW and AdamW+SR

We run a set of experiments with a larger number of experts E=128E=128 and top-4 routing with 10B tokens. We present the training curves with AdamW and SR at two different learning rates in our paper revision (Appendix E.6, Figure 11 and Table 10), where SR demonstrates a consistent improvement over AdamW.

Besides, we also conducted an analysis of the Router’s behavior under AdamW and AdamW with SR in Appendix E.6. According to Figure 12 and Table 11, adding SR would use more experts in the top layers (ones are close to the output), while AdamW seems to activate more experts in early layers (ones are close to the embedding layer). We don’t have a theoretical explanation for that yet, but this aligns with our intuition that in top layers, the LLM needs to acquire more sophisticated semantic features and requires more capacity and specification among various experts. The number of experts in use per MoE layer and corresponding validation loss at 25k steps (3.3B training tokens) is presented below:

MoE layer01234567all (of 1,024)val. loss
AdamW, peak LR 10−410^{-4}11253441371716161973.560
SR, peak LR 10−410^{-4}4961628606681674183.507
AdamW, peak LR 10−310^{-3}181019583863930204723.514
SR, peak LR 10−310^{-3}17161388205569683463.475

Router gradient steps

We measure and present the router’s gradient steps in each run in Appendix E.6.1, Figure 13. We track the router gradient norm (Equation 53) every 500 steps. The router gradient does not decay or spike extremely under any rule. Therefore, the router gradient does not vanish in these runs, while SRSR does encourage more stable gradient steps compared to plain AdamW.

AuthorsOpenReview ↗

Rebuttal 3/3

Q5: Does the rate hold on Adam with ϵ=10−8\epsilon=10^{-8}?

We present the error curves with various values of ϵ\epsilon in our paper revision, Appendix E.1 and Figure 8. The performance shows that SqrtGD is faster than Adam for all ϵ=10−1,10−2,10−4,10−8\epsilon=10^{-1}, 10^{-2}, 10^{-4}, 10^{-8}.

W2: Explanation on the gate renormalization

We analyze the router gradients under gate renormalization. According to Appendix E.4, with renormalization, only the router logits from selected experts have gradients. Therefore, the language-model loss gives exactly zero gradient to unselected logits, so those coordinates are driven by the auxiliary losses only; therefore, SR amplifies the wrong gradients. It falls apart from our theoretical setup, where router logits for all experts will get gradients. We also validate that AdamW with SR under non-renormalized gating outperforms AdamW with renormalization in Table 6.

Recipeseed 0seed 1seed 2
AdamW6.5505.7417.416
SR3.2033.1733.218
AdamW, renormalized gate3.2553.2933.282

Q6. Relation to change-point regression.

Thanks for pointing us to the related works we missed! We have added the related literature to the related work section in the paper revision. The breakpoint literature has three relevant cases: A jump at the change point gives a criterion linear in ∣δ∣|\delta| and a super-consistent estimator ([1, 2]). A continuous model fitted with the continuity constraint gives a quadratic criterion ([3, 4]). A continuous truth fitted without the constraint, which is our setting of independently refit experts, gives a cube-root rate for the threshold estimate (Hidalgo, Lee and Seo 2019), which we consider as the statistical counterpart of our established cubic population criterion in Theorem 1. Therefore, the cubic geometry is not new in the change-point regression literature, and we will say so and cite those related works. Our novelty lies in its optimization consequence: the rates of GD and Adam on this criterion, the square-root update that restores geometric convergence, the extension to coupled boundaries, and the relevance to a hard MoE router.

—----------------

Sorry for posting the responses late, since the experiments take time. We hope our answer and additional results can resolve your concerns, and you can consider raising the score accordingly. If you have any further questions, we are happy to address them!

[1] David V. Hinkley. Inference about the change-point in a sequence of random variables. Biometrika, 57(1):1–17, 1970.

[2] Jushan Bai and Pierre Perron. Estimating and testing linear models with multiple structural changes. Econometrica, 66(1):47–78, 1998.

[3] Paul I. Feder. On asymptotic distribution theory in segmented regression problems—identified case. The Annals of Statistics, 3(1):49–83, 1975

[4] Kung-Sik Chan and Ruey S. Tsay. Limiting properties of the least squares estimator of a continuous threshold autoregressive model. Biometrika, 85(2):413–426, 1998.

[5] Javier Hidalgo, Jungyoon Lee, and Myung Hwan Seo. Robust inference for threshold regression models. Journal of Econometrics, 210(2):291–309, 2019.

Reviewer_STP9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether routers in sparse MoEs, typically trained with the same optimizer as experts, have distinct optimization geometry. The authors simplify routing to a single boundary in piecewise-linear regression with affine experts and exact expert refitting. When the target is continuous at the boundary, they show that the local loss is cubic in boundary displacement and the gradient vanishes quadratically. Consequently, GD and Adam exhibit asymptotic polynomial loss decay of O(t⁻³), whereas signed square-root gradient descent (SqrtGD) achieves exponential decay. Based on this analysis, they propose Square-root Gradient Routing (SR), which applies square-root updates only to router parameters, and evaluate it in 32-expert MoE LLM pretraining with AdamW and Muon.

Critical strengths
  1. Interpretable theoretical insight. The analysis connects target continuity to cubic boundary-loss geometry and explains how square-root updates overcome the resulting slowdown. Extensions to coupled boundaries and high-dimensional inputs with a fixed routing normal, supported by toy experiments, strengthen this contribution.
Critical weaknesses
  1. The practical relevance of the continuity assumption needs further support. Assumption 2 is central to the cubic-loss result, but it is unclear whether an analogous property holds for learned MoE experts. In the high-dimensional affine setting, Appendix B.3 shows that continuity across the interface requires identical tangential coefficients, allowing the two target experts to differ only along the routing normal. This is a substantive restriction whose relevance to practical MoE training is not established. With a nonzero target jump, the same regression setting instead yields a leading loss term proportional to |δ\delta|, so the cubic-loss mechanism no longer applies. Measuring expert-output differences near routing boundaries and discussing this assumption explicitly would strengthen the practical motivation.
  2. Additional controls and broader validation would strengthen the empirical conclusions. The top-2 learning-rate sweeps provide encouraging evidence. However, independently tuning the baseline router learning rate for the configurations in Table 1 would help clarify whether SR’s gains persist under stronger baselines. Repeated-seed results would also make the smaller improvements easier to assess. Finally, the LLM experiments are limited to a single model architecture and scale on FineWeb, leaving the generality of the gains across architectures, scales, and datasets uncertain.
Questions for authors
  1. Prior work analyzes output discontinuities at top-k switching boundaries [r1]. How does the target-continuity assumption in your analysis relate to this behavior, and is there evidence that learned experts approximately agree near routing boundaries?
  2. Does the toy Adam slowdown remain observable with ε=10⁻⁸, and does ε dominate the router’s Adam denominator during LLM training?
  3. Prior work uses separate learning rates for routers and experts [r2, r3]. For the configurations in Table 1, does SR retain its advantage when the baseline router learning rate is tuned independently while keeping the other parameters’ learning rates fixed? Are these gains consistent across seeds?

[r1] Huu, Tho Tran, et al. "Geometric and Stochastic Analysis of Discontinuities in Sparse Mixture-of-Experts." arXiv preprint arXiv:2606.19036 (2026).

[r2] Chen, Zixiang, et al. "Towards understanding the mixture-of-experts layer in deep learning." Advances in neural information processing systems 35 (2022): 23049-23062.

[r3] Dikkala, Nishanth, et al. "On the benefits of learning to route in mixture-of-experts models." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.


minor coment

  • Learning-rate description (§4.2.1, p. 8). The statement that η=10−3η = 10^{-3} is used across all configurations should distinguish parameter groups. Table 4 specifies a learning rate of 0.02 for Muon matrices, consistent with the Muon results in Tables 1 and 5.

  • Typo (line 035). “DeepSeek-V4 only activate” → “DeepSeek-V4 only activates.”

4 comments
AuthorsOpenReview ↗

Rebuttal 1/2

We sincerely thank the reviewer for the insightful feedback, and we really appreciate your close reading through our theory and carefully assessing our experiments! We respond to each of your concerns below.

Q1 & W1: Discontinuity assumption and its realistic interpretation

We thank the reviewer for pointing us to this related work [r1], we added it to the related work section in our paper revision! We admit the gap between our continuous theoretical setup and the real-world MoE training landscape, where the ground-truth “boundary” is hard to determine in its high-dimensional landscape. We would like to make it clear that in our theory, we only assume the ground-truth functions are continuous but did NOT constrain the learned experts to be continuous at the boundary until they converge to the ground-truth. Therefore, it does not contradict the reference [r1]’s results that smoothing the experts to be continuous can push them quickly to the ground-truth. A discontinuity at the ground-truth boundary will lead to an infinite error when we integrate at the breakpoint, which makes the theoretical analysis intractable. Intuitively, it should be held in practice, since if one token can be routed into two experts in a convergence MoE model (as approached to ground-truth), we do not expect their output to differ too much; otherwise, there would be a training spike there. We have clarified the gaps between our toy setup for theoretical analysis and practical LLM pretraining in Section 5.

Q2: Adam with various ϵ\epsilon, and practical evidence from LLM training

We present the error curves with various values of ϵ\epsilon in our paper revision, Appendix E.1 and Figure 8. The performance shows that SqrtGD is faster than Adam for all ϵ=10−1,10−2,10−4,10−8\epsilon=10^{-1}, 10^{-2}, 10^{-4}, 10^{-8}.

**Does ϵ\epsilon dominate the optimization during LLM training? ** No. we measure the value of vtvt∣+ϵ\frac{\sqrt{v_t}}{\sqrt{v_t}|+\epsilon} value, which determines how much ϵ=10−8\epsilon=10^{-8} can impact the second-order momentum. The range of measured vtvt∣+ϵ\frac{\sqrt{v_t}}{\sqrt{v_t}|+\epsilon} is around 0.9996, which indicates ϵ=10−8\epsilon=10^{-8} does not dominate the optimization. Since it is a minor point, we did not add it to the paper revision.

Q3 (I): Independent learning rate sweep on the router

a correction on your [r2] reference: it should point to the “Towards Understanding the Mixture-of-Experts Layer in Deep Learning” work?

We conducted experiments with E=32E=32 top-2 routing setup with separately tuned router learning rates. All non-router parameters use a peak learning rate 3×10−43\times 10^{-4}, as the optimum selected for plain AdamW according to Table 5 in the paper.

We report the results in our paper revision (Appendix E.3, Figure 9, Table 8). Firstly, we do observe that the untuned learning rate on the router is not optimal for both AdamW and AdamW+SR; second, SR still outperforms AdamW at the optimum of the router's learning rate. We also provide the statistics below

Router updaterouter LRval. loss
AdamW9×10^{-6}$3.355
AdamW3×10−53×10^{-5}3.332
AdamW9×10−59×10^{-5}3.323
AdamW3×10−43×10^{-4}3.342
AdamW9×10−49×10^{-4}3.338
SR9×10−59×10^{-5}3.318
SR3×10−43×10^{-4}3.319
SR9×10−49×10^{-4}3.307
SR3×10−33×10^{-3}3.312
SR9×10−39×10^{-3}3.373

Q3 (II): Robustness concern and multiple random seeds

We rerun E=32E=32, top-2 experiments with three random seeds and report the statistics in the paper revision (Appendix E.2, Table 6). According to the results, AdamW with SR are consistently more stable than plain AdamW, with lower validation losses.

AuthorsOpenReview ↗

Rebuttal 2/2

W3: Experiments with other MoE architectures with interpretation on the routing mechanism

We run a set of experiments with a larger number of experts E=128E=128 and top-4 routing with 10B tokens. We present the training curves with AdamW and SR at two different learning rates in our paper revision (Appendix E.6, Figure 11, Table 10), where SR demonstrates a consistent improvement over AdamW.

Besides, we also conducted an analysis of the Router’s behavior under AdamW and AdamW with SR in Appendix E.6. According to Figure 12 and Table 11, adding SR would use more experts in the top layers (ones are close to the output), while AdamW seems to activate more experts in early layers (ones are close to the embedding layer). We don’t have a theoretical explanation for that yet, but this aligns with our intuition that in top layers, the LLM needs to acquire more sophisticated semantic features and requires more capacity and specification among various experts. The number of experts in use per MoE layer is presented below:

MoE layer01234567all (of 1,024)val. loss
AdamW, peak LR 10−410^{-4}11253441371716161973.560
SR, peak LR 10−410^{-4}4961628606681674183.507
AdamW, peak LR 10−310^{-3}181019583863930204723.514
SR, peak LR 10−310^{-3}17161388205569683463.475

We hope our answer and additional results can resolve your concerns, and you can consider raising the score accordingly. If you have any further questions, we are happy to address them!

I sincerely thank the authors for the detailed response and the additional experiments, which I believe address my concerns regarding router learning-rate tuning and seed variability. However, my concern regarding the practical relevance of the theoretical analysis remains.

The clarification that continuity is assumed for the target rather than the learned experts is helpful, but the relevance of this assumption to practical MoE training remains insufficiently supported. Moreover, given the authors’ observation that ε does not dominate Adam’s denominator in their LLM experiments, it remains unclear whether the slowdown characterized by the theory explains the practical gains from SR.

Nevertheless, I recognize the value of the theoretical contribution. The analysis gives an interpretable account of how target continuity leads to cubic boundary-loss geometry and how square-root updates overcome the resulting slowdown. I view the remaining concern as a limitation rather than a reason for rejection, and I will raise my score.

Finally, I thank the authors for the correction regarding [r2]. The intended reference is indeed “Towards Understanding the Mixture-of-Experts Layer in Deep Learning.”

We thank the reviewer for the response! We fully understand your concerns and appreciate your carefulness on the theoretical-practical gap between multi-segment regression and practical situation on LLMs. However, since the ground-truth boundaries in an LLM MoE model are intractable, we are not able to provide further theoretical studies or verifications on LLM landscape. We make some more clarifications below:

  • We admit the theory-practical gap between our multi-segment regression setup and practical LLM training. In real LLM training, the model sits in a high-dimensional, non-convex and stochastic landscape, so it is hard to find the ground-truth boundary and quantify how much benefit comes from a more accurate routing mechanism. Also, some common implementation tricks like load-balancing loss would add additional gradients upon routers and experts. Although these additional gradients prevent the router inside LLM to perform a gradient vanishing (Figure 13), those gradients are only used for regularization. That's also the reason why we conduct our theoretical analysis upon the canonical setup, as previous literatures on MoE understanding [1]. Our theory provide a tractable understanding on the dynamic of router during optimization which inspires us to apply SR in LLM training, but the theory it is not capable to fully explain LLM, where the ground-truth boundary is not clear.

  • In Appendix E.6, Figure 12 and Table 11, we provide the practical interpretation on the routing mechanism from SR and plain AdamW. (i.e. what SR did differently and why it can be helpful) Adding SR would use more experts in the top layers (ones are close to the output), while AdamW activates more experts in early layers (ones are close to the embedding layer). The theoretical explanation for that is intractable, but this aligns with our intuition that in top layers, the LLM needs to acquire more sophisticated semantic features and requires more capacity and specification among various experts.


[1] Towards Understanding Mixture of Experts in Deep Learning, 2022.

Reviewer_BSSEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies router optimization in sparse MoEs. It starts with some simplified assumptions such as hard routing, continuous piecewise-affine targets, and exact least-squares expert refitting. In this simplified and synthetic setting, it shows that router gradients vanish quadratically as the routing boundary approaches its optimum, leading to slow convergence. Signed square-root updates counteract this gradient shrinkage and achieve geometric convergence under these assumptions. The practical method, SR, applies a signed square-root transformation to router gradient momentum. Synthetic experiments examine the predicted convergence rates. Emperically, the1.3B-parameter, 8-layer MoE models are pretrained on 3.3B FineWeb tokens. Practical gains vary across gating and top-k settings; SR improves learning-rate tolerance in selected configurations but this large lr can degrade or diverge with renormalized AdamW.

Critical strengths
  1. A well-supported optimization mechanism in a synthetic setting. The theory explains why router gradients shrink near the optimal boundary and how square-root updates address this issue. These predictions are well-supported by the synthetic experiments, which show faster convergence with square-root updates.

  2. Improved tolerance to larger learning rates. Figure 3 shows that SR remains stable at larger learning rates where the baseline diverges in the unrenormalized top-2 setting. This may suggest a practical benefit for router optimization in the chosen configuration.

Critical weaknesses
  1. Gaps between the simplified routing theory and practical MoE training. First, the analysis treats routing as hard partitioning. In actual top-k MoEs, discrete expert selection changes mainly at the top-k boundary, and the router also learns continuous mixture weights when the selected experts stay the same. Learning these weights may play a larger role in practice, yet this is not captured by the boundary analysis. The vanishing-gradient result also relies on target branches meeting continuously at the boundary, while real experts need not produce equal outputs there. Finally, the proposed Adam slowdown requires epsilon to dominate its denominator. The toy experiments use epsilon = 0.1, versus 1e-8 in LLM training; whether the latter actually enters this regime is not shown.

  2. Limited robustness across gating, top-k, and MoE architectures. SR performs worse and can diverge with renormalized gates under AdamW, and gains are small or absent at larger top-k. It is unclear why the benefit depends so strongly on these setups. Furthermore, the introduction states that “its benefit grows as routing becomes sparser” (line 72). However, changing top-k also changes activated capacity and computation, so the results do not isolate sparsity itself as the reason for the gains.

  3. The experiments only use a 32-expert GPT-2-style architecture. Validation on models combining shared experts and fine-grained routed experts is needed to establish broader applicability.

Questions for authors
  1. Please address the theoretical assumptions and robustness concerns detailed in W1 and 2.

  2. Can you compare AdamW and SR after independently tuning the router learning rate for each method, while keeping the optimization settings of all other parameters fixed? Please include both gating conventions.

  3. Could you provide LLM training/validation loss curves and check whether the main results hold with longer training and independent seeds, if possible?

4 comments
AuthorsOpenReview ↗

Rebuttal 1

We thank the reviewer for the careful reading of our theory and for locating the gap between the theory and real-world LLM training. We provide the responses below.

W1: Adam with various ϵ\epsilon and the other theoretical assumptions

We present the boundary error curves with various ϵ\epsilon in our paper revision Appendix E.1. According to Figure 8, SqrtGD is faster than Adam across all ϵ=10−1,10−2,10−4,10−8\epsilon=10^{-1}, 10^{-2}, 10^{-4}, 10^{-8}.

We admit the gap between our convex, continuous theoretical setup and the real-world MoE training landscape. However, the discontinuity at the boundary will lead to an infinite error when we integrate at the breakpoint, which makes the theoretical analysis intractable. We also would like to make it clear that the expert outputs are not restricted to continuous until it converges to the ground-truth functions. We have already clarified the gaps between our toy setup for theoretical analysis and practical LLM pretraining in Section 5.

W2: Clarification on sparsity and vanishing gain with increasing top-k activation

We agree with the reviewer that our original claim “its benefit grows as routing becomes sparser” (line 72) is not proper since the activated parameters are changed. We propose to rephrase it in our paper revision as “Remarkably, it can effectively stabilize MoE model training with sparse top-kk activation.”

Secondly, grounded in our theory, the benefit from Square-root Routing is expected to vanish with large top-kk activation, since our theory is derived with a hard-routing scenario, which motivates that in a sparsely activated MoE model, applying Square-root Routing can lead to a more accurate routing strategy. We also provide both E=16 and E=32 expert results in the paper revision (Appendix E.2, Table 7) to support this trend across various MoE architectures.

E=16 top-1E=16 top-2E=16 top-4E=16 top-8E=32 top-1E=32 top-2E=32 top-4E=32 top-8
AdamW3.6086.631 ✗3.1763.1435.909 ✗6.550 ✗3.1403.086
SR3.2883.2493.1753.1433.2973.2033.1293.094

Q2: Independent learning rate running on the router

We conducted experiments with E=32E=32 top-2 routing setup with separately tuned router learning rates. All non-router parameters use a peak learning rate 3×10−43\times 10^{-4}, as the optimum selected for plain AdamW according to Table 5 in the paper.

We report the results in our paper revision (Appendix E.3, Figure 9, Table 8). Firstly, we do observe that the untuned learning rate on the router is not optimal for both AdamW and AdamW+SR; second, SR still outperforms AdamW at the optimum of the router's learning rate. We also provide the statistics below

Router updaterouter LRval. loss
AdamW9×10^{-6}$3.355
AdamW3×10−53×10^{-5}3.332
AdamW9×10−59×10^{-5}3.323
AdamW3×10−43×10^{-4}3.342
AdamW9×10−49×10^{-4}3.338
SR9×10−59×10^{-5}3.318
SR3×10−43×10^{-4}3.319
SR9×10−49×10^{-4}3.307
SR3×10−33×10^{-3}3.312
SR9×10−39×10^{-3}3.373

Q3 & W3: Larger-scale training curves

Yes, we ran a set of experiments with a larger number of experts E=128E=128, and top-4 routing with 10B tokens. We present the training curves with AdamW and SR at two different learning rates in our paper revision (Appendix E.6, Figure 11, Table 10), where SR demonstrates a consistent improvement over AdamW.

Besides, we also analyzed the router’s behavior under AdamW and AdamW with SR in Appendix E.6. According to Figure 12 and Table 11, adding SR would use more experts in the top layers (ones are close to the output), while AdamW seems to activate more experts in early layers (those close to the embedding layer). We don’t have a theoretical explanation for that yet, but this aligns to our intuition that in top layers, the LLM needs to acquire more sophisticated semantic features and requires more capacity and specification among various experts. The number of experts in use per MoE layer is presented below:

MoE layer01234567all (of 1,024)val. loss
AdamW, peak LR 10−410^{-4}11253441371716161973.560
SR, peak LR 10−410^{-4}4961628606681674183.507
AdamW, peak LR 10−310^{-3}181019583863930204723.514
SR, peak LR 10−310^{-3}17161388205569683463.475

We hope our answer and additional results can resolve your concerns, and you can consider raising the score accordingly. If you have any further questions, we are happy to address them!

Reviewer_BSSEOpenReview ↗

Response to author's response

Thanks for the additional experiments. My concerns are partially addressed.

My remaining question is whether SR’s gains diminish as more experts are selected per token, even at the same activation fraction. The theory explains how SR may help with expert selection, but do not explain whether it helps learn the weights of the selected experts. Could you extend the fine-grained expert experiments to compare, for example, top-2 out of 32 with top-8 out of 128? First, fine-grained experts are widely used in modern MoEs, and then it make it possible to increase the number of selected experts while keeping the activation fraction fixed. This would help test whether SR’s gains disappear as more experts contribute to each token, even when routing remains equally sparse. I think gains in such setting could strenthen the paper a lot.

Thanks a lot for the quick response and the insightful suggestions, we really appreciate it! :) Also, your suggestions really helped us to strengthen the manuscript a lot! We now understand the setting under the “granularity” comparison, and we provide more results on granularity (E128-top8 v.s. E32-top2) and sparsity (E128-top4 V.S. E128-top4) in our new paper revision:

(1) Granularity: In Figure 15, we compare the coarse experts (E32 with top-2 routing) and fine-grained experts (E128 with top-8 routing) at the same sparsity level (1/16 activated). With the coarse experts, SR is ahead of plain AdamW by 0.028 by 3.15B tokens; while with the fine-grained experts, SR outperformed by 0.082 at 2.10B tokens and 0.085 at 3.15B tokens. The advantage is consistent along the training towards 8B tokens. Comparing to the AdamW run on E128 with top-8 routing, our proposed SR achieves a 1.56×1.56\times acceleration.

(2) Sparsity: In Figure 16, we compare the performance of AdamW and SR with E128 with top-4 and top-8 routing, with various activated experts (1/32 v.s. 1/16). The results show that the benefit from SR is significant in both cases. Also, SR effectively improves the stability and can apply a larger learning rate safely (Table 12 and Figure 16 (b)).

The runs have not been completed (SR is around 8B out of 10B for E128-top8) but we can already observe a consistent and significant pattern from our current evidence. We promise to include the full run’s statistics and plot in our final manuscript.

We hope our additional responses can resolve more of your concerns!

Thanks for your results. I will raise my score. The rebuttal shows that SR improves validation loss with top-8 routing at the same sparsity, addressing my main empirical concern.

I would also highlight to AC that it is still unclear how much of the gain will persist over longer training. Furthermore, the theory also analyzes expert selection boundaries, while the experiments leave open whether the practical gains come from expert selection, mixture weighting, or both. The conclusion for this paper could also be limited and I hope AC could take this into account when evaluating this paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the three reviewers for their careful reading through our theory and corresponding empirical evidence, especially, we sincerely thank to their understanding of each theorem, including considerations on the theory-practical gaps of our assumptions. All the reviewers have recognized our theoretical contributions but have concerns on the gap to the practical LLM training. Our theory aims to push an optimization-grounded understanding of router’s dynamics in MoE model training, which inspires a better router design in real LLM-MoE training. However, since the router only occupies <0.1% parameters, it's hard to expect a strikingly large improvement on LLM experiments. In the paper revision, we provide empirical evidence on LLM training with more architectures (E=16, 32, 128) to confirm its value in practical values with visualized analysis on the router dynamics for better interpretability.

We hereby provide a roadmap of our revisions here and we hope they can mitigate the reviewers' concerns. Each reviewer's response below is self-contained.

Main paper

Related work. A new paragraph on change-point and threshold regression [PWYt Q6]; [r1] on discontinuities of the learned MoE map [STP9 W1]; Chen et al. (2022) and Dikkala et al. (2023) for separate router learning rates [BSSE Q2].

Section 4.1. Point to the experiments with various \epsilon in Appendix E.1.

Introduction and Section 4.2.1: fix the typo “activate” → “activates”; clarify the learning rate differences on different parameter groups with Muon.

Appendix E: additional figures and results

  • E.1, Adam with various ϵ\epsilon. We report the boundary error on Adam with various ϵ\epsilon from 0.1 to 1e-8 in Figure 8. With all values of ϵ\epsilon, Sqrt-GD consistently converges faster than Adam.

  • E.2, Multiple seeds and sparsity grid. Table 6: We present results with three random seeds across three methods (AdamW, SR, AdamW+renormalization); Table 7: sparsity grid on number of experts E=16 and E=32 with top-k=1, 2, 4, 8.

  • E.3, separately tuned router learning rate. Figure 9, final loss against the router learning rate for AdamW, SR, and the sign update; Table 8, learning rate sweep statistics on E=32, top2.

  • E.4, router-logit gradients under the two gating conventions: Present a gradient analysis on why the renormalization case can be far from our theory.

  • E.5, router dynamics before a divergence. Figure 10, share of tokens per expert over training when divergence happens.

  • E.6, fine-grained experts with E=128, TOP-4, 10B tokens. Table 10 and Figure 11, validation loss values and detailed statistics; Figure 12 and Table 11, routing per layer and experts in use and mechanistic analysis of why SR helps; E.6.1 and Figure 13, the size of the router gradient over training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.