acceptodds
Under review as a conference paper at ICLR 2027

GradRouter: Decompose Gradient Before Assigning Learning Responsibility in Sparse Mixture-of-Experts

Abstract

Sparse Mixture-of-Experts (MoE) models enable efficient scaling by activating only a subset of experts for each token. Their effectiveness largely depends on how experts divide and coordinate their responsibilities. Modern MoEs use routers to determine which experts are activated and how much their outputs contribute to the aggregated output. Accordingly, existing algorithms promote expert specialization primarily through forward computation, e.g, differentiating the token distributions and output representations across experts, while leaving backward learning responsibility directly inherited from forward pass. Specifically, for expert outputs are linearly combined, the co-activated experts will receive collinear output-side gradients. In this paper, we raise the attention that such collinear gradient might drive co-activated experts towards the same representation, implicitly hindering their functional differentiation. Moreover, backward learning signals are inherently high-dimensional, with different experts potentially better suited to different directions. This suggests that scalar routing weights alone might not fully characterize how learning responsibility should be divided. Therefore, in this paper, we propose GradRouter, a backward learning responsibility assignment framework that enables differentiated learning direction of co-activated experts. GradRouter first identifies the principal directions of the output-gradient space and then assigns the each direction to co-activated experts according to their capabilities, meanwhile preserving original gradient on aggregated expert outputs. Experiments demonstrate that GradRouter consistently improves model performance across diverse MoE architectures with 7B–30B total parameters, highlighting the effectiveness and scalability of directional learning responsibility assignment.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Expert Specialization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iBP6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GradRouter, a framework that decouples backward learning responsibility from forward routing in sparse MoE models. It decomposes output gradients into principal directions and assigns them to co-activated experts according to their direction-specific capabilities, while preserving the aggregate gradient.

Critical strengths
  1. The paper provides a novel distinction between prediction responsibility and learning responsibility, revealing a fundamental limitation of standard MoE backpropagation.
  2. GradRouter offers a principled approach to assigning direction-specific learning signals while preserving the aggregate output gradient.
  3. The method is validated across multiple MoE architectures and scales, showing consistent improvements, particularly on code-generation tasks.
Critical weaknesses
  1. The benefits of expert specialization are expected to emerge in multi-task learning and generalization, as discussed in the paper, but the experiments do not directly evaluate either aspect.
  2. The empirical improvements are not fully consistent across architectures and tasks, and GradRouter occasionally underperforms the strongest baseline, making its overall effectiveness less conclusive.
  3. GradRouter introduces additional PCA, affinity estimation, and optimal-transport computations, but the paper provides limited empirical analysis of the resulting training-time and memory overhead at scale.
Questions for authors

please refer to Weakness

3 comments

W1.Direct evidence for multitask learning and generalization benefits is missing.

R1. Our training and evaluation cover multiple task types. Our training dataset includes coding, mathematics, and reasoning tasks. We also evaluate downstream generalization on diverse benchmarks spanning these task types, including HumanEval and MBPP for coding, GSM8K and MATH500 for mathematics, and BBH and GPQA for reasoning.

W2. The empirical improvements are not fully consistent across architectures and tasks, and GradRouter occasionally underperforms the strongest baseline, making its overall effectiveness less conclusive.

Thank you for pointing this out. We agree that GradRouter trails the best-performing baseline in some settings. The baselines are competitive methods that target aspects of MoE training such as load balancing and expert specialization. GradRouter addresses how learning credit is assigned to active experts. The mixed results suggest that the gains from each intervention may depend on the backbone and task. We will qualify our discussion accordingly. We includes vanilla SFT in Table R1 as a reference for training with standard backpropagation.

MethodMMLUMMLU-ProGSM8KMATH500HumanEvalHumanEval+MBPPMBPP+BBHGPQALiveBench
OLMoE
SFT46.8717.3045.9418.0014.6310.9830.9525.939.0621.729.46
GradRouter47.2517.3446.9319.0017.0714.6330.6925.408.6521.219.41
DeepSeek-V2-Lite
SFT52.6220.0535.1013.4026.8321.3445.2438.6235.1431.829.95
GradRouter56.4725.0544.9620.6036.5930.4959.5249.2147.3030.3012.86

W3.The paper provides limited empirical analysis of the resulting training-time and memory overhead at scale.

R3. Corrected measurements show approximately 1.2× baseline runtime. The unexpectedly large cost also prompted us to revisit Figure 4 after submitting the manuscript. We notice that GradRouter's “Base” component in Figure 4 alone exceeded the total baseline runtime. We found that in Figure 4, baselines used ZeRO-2 while GradRouter used ZeRO-3, so that sharding and implementation differences are included. With matched ZeRO stages, GradRouter takes 1.163–1.185× baseline runtime under ZeRO-2 and 1.254–1.271× under ZeRO-3, as reported in Table R1 . We appreciate the reviewer's concern about practical efficiency and will update Figure 4 and the corresponding runtime analysis. And the updated per-component time cost of GradRouter is reported in Table R2.

Table R1. Per-step training time (/senconds) with matched ZeRO stages across 3 MoE models. Parentheses report GradRouter time divided by the corresponding baseline time. These profiles measure runtime, not downstream performance at matched compute.

ArchitectureConfigurationSFTAuxLossLoss-FreeGradRouter
OLMoEZeRO-241.32 (1.185×)41.77 (1.173×)42.12 (1.163×)48.98
OLMoEZeRO-362.73 (1.271×)63.34 (1.259×)63.56 (1.254×)79.73
DeepSeek-V2-LiteZeRO-269.26 (1.164×)70.47 (1.144×)69.69 (1.157×)80.61
DeepSeek-V2-LiteZeRO-3108.66 (1.268×)110.89 (1.243×)111.04 (1.241×)137.82
Qwen3-30B-A3B-BaseZeRO-3377.95 (1.241×)385.72 (1.216×)386.56 (1.214×)469.12

Table R2. GradRouter runtime breakdown on OLMoE (/seconds). “Regular” denotes a step without PCA/affinity refresh; “refresh” includes those updates; “amortized” averages one refresh step and four regular steps.

GradRouter componentZeRO-2 regularZeRO-2 refreshZeRO-2 amortizedZeRO-3 regularZeRO-3 refreshZeRO-3 amortized
Base38.4837.7138.3267.3465.6667.00
PCA0.0020.294.060.0020.884.18
Affinity0.003.600.720.006.611.32
Assignment0.570.420.540.620.490.59
Correction5.385.145.336.646.636.64
Total44.4367.1648.9874.60100.2679.73

Thank you for your rebuttal. It has addressed some of my concerns, and I will maintain my evaluation of "Weak accept."

Thank you for your positive assessment and constructive feedback. We are glad that our rebuttal addressed some of your concerns.

Reviewer_8purOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GradRouter, a method for assigning learning responsibility among activated experts in sparse Mixture-of-Experts (MoE) models. The main observation is that, under standard backpropagation, co-activated experts receive output-side gradients that point in the same direction and differ only through their forward routing weights. The authors argue that tying learning responsibility so closely to prediction responsibility may discourage experts from developing distinct capabilities.

GradRouter separates these two roles. It first uses PCA to build a low-rank basis from recent output gradients. It then estimates how well each expert can respond along each basis direction using Jacobian-based affinity scores and applies optimal transport to distribute directional responsibility across the activated experts. A residual correction is added so that the expert-specific output gradients still sum to the original task gradient. Experiments on three MoE backbones, ranging from 7B to 30B parameters, report improvements on a selection of reasoning, mathematics, and code-generation benchmarks.

Critical strengths

The paper offers a fresh perspective on MoE credit assignment. The distinction between responsibility for the forward prediction and responsibility for learning is useful and, to my knowledge, less explored than forward-routing design. Because GradRouter changes only the backward signal, it may also be compatible with existing routing and load-balancing methods.

The evaluation covers several architectures and task families. The experiments use OLMoE-1B-7B, DeepSeek-V2-Lite, and Qwen3-30B-A3B across 11 benchmarks. The clearest gains appear on code-generation tasks. The reported training curves are also reassuring, as they suggest that the custom backward signals do not introduce obvious optimization instability.

Critical weaknesses

The motivation does not yet demonstrate that standard backpropagation prevents expert specialization. Co-activated experts do receive collinear output-side gradients, but their parameter gradients are JiTgJ_i^T g, and each expert has a different Jacobian. Their actual parameter updates may therefore differ substantially even when the incoming output gradients are aligned. The paper does not measure parameter-gradient alignment or show that output-gradient collinearity leads to weak functional specialization. The proposed failure mode is plausible, but it remains largely unverified.

Preserving the summed output gradient is not the same as preserving the original optimization objective. GradRouter ensures that the modified expert-specific gradients add up to the original output gradient. However, those signals are generally not the derivatives of the forward computation in Equation 1. The method should therefore be viewed as a custom-gradient or surrogate-update procedure rather than ordinary gradient descent on the stated task loss. The paper would benefit from clarifying whether this update field corresponds to any explicit objective, or at least identifying conditions under which it remains a descent direction for the task loss.

The empirical evidence is not yet statistically robust. The gains vary considerably by task, and GradRouter trails the strongest baseline on several benchmarks. The appendix notes that the hyperparameter results come from single runs, while the main tables do not report variance, confidence intervals, or significance tests. AFBM also performs unexpectedly poorly in several settings, which raises questions about implementation details or baseline tuning.

The computational overhead makes the scalability claim less convincing. The reported end-to-end training cost increases by roughly 1.3x to 2.2x, with PCA and gradient correction accounting for much of the additional time. Given that the downstream improvements are often modest and inconsistent, the practical tradeoff is difficult to assess. The evaluation is also restricted to LoRA fine-tuning on about 20K examples, so it is not clear whether the method remains effective for full-parameter MoE training at larger scale.

Minor comments:

The manuscript is generally readable, but several phrases should be revised. For example:

“we raise the attention that” -> “we highlight that” (p. 1);

“with different experts might be potentially better suited” -> “different experts may be better suited” (p. 1);

“we parallel compute ... in parallel” -> “we compute ... in parallel” (p. 5);

The notation for “Top-K” and “top-k.” should be standardized

Questions for authors
  1. Despite receiving collinear output-side gradients, do co-activated experts under standard backpropagation also exhibit highly aligned parameter gradients? How does GradRouter affect this alignment and measurable expert specialization?

  2. Does the modified backward update correspond to the gradient of an explicit objective? If not, can the authors provide conditions under which it is still a descent direction for the task loss?

  3. Can the authors report results across multiple random seeds, including confidence intervals and paired comparisons with standard backpropagation under identical forward routing?

  4. How does GradRouter compare with the baselines under matched training compute or wall-clock time, given its reported 1.3x to 2.2x overhead?

5 comments

W1&Q1. The proposed failure mode is plausible, but it remains largely unverified. Do Co-activated experts under standard backpropagation exhibit highly aligned parameter gradients?

R1. GradRouter is to promote expert differentiation when Jacobian differences alone are insufficient to drive it. Thank you for this insightful and important question. Your intuition is right: when expert Jacobians differ, collinear output gradients do not necessarily make expert representations more similar. While we did not claim such colinear gradient would inevitably lead to representation collapse and our concern focuses that, when experts are frequently co-activated, their Jacobians may not differ enough to drive distinct representations of the shared tokens. Repeatedly receiving collinear gradients may then reinforce an existing tendency toward representation convergence. In such cases, GradRouter aims to encourage differentiation by decomposing the output gradient and allocating its components across the co-activated experts.

Thanks for your thoughtful question again, and in the following, we organize the evidence of our motivation through 3 parts: (1) theoretically illustrate that when a shared update would contract co-activated expert representations, (2) repeated shared updates can accumulate aligned components despite independent initialization of expert Jacobian, and (3)the prevalence of representation contraction during actual MoE training, despite the difference in experts' Jacobian.

1. Brief explaination of the collapse condition. Let ei,eje_i,e_j denote the expert outputs and gi,gjg_i,g_j their output-side gradients. For a SGD step, we have first-order approximation of the update of experts' outputs as the following:

e_j^+=e_j-\eta J_jJ_j^\top g_j.$$ Here, $J_i,J_j$ are the Jacobians with respect to the parameter blocks being updated, and $K_i=J_iJ_i^\top$, $K_j=J_jJ_j^\top$ are their response kernels. Consequently, $\langle e_i-e_j,K_i g_i-K_j g_j\rangle>0$ gives a sufficient first-order condition for local contraction at a sufficiently small step size: $$\|e_i^+-e_j^+\|^2-\|e_i-e_j\|^2=-2\eta\left\langle e_i-e_j, K_i g_i-K_j g_j\right\rangle.$$ For SwiGLU like experts, we have \$K_i(x)=\|h_i(x)\|^2 I+\|x\|^2 W_{d,i} \operatorname{diag}\!\bigl(s_i(x)^2+m_i(x)^2\bigr)W_{d,i}^{\top}\$ Interpreting from the form, we could learn that for co-activated experts i,j, both of their kernels use the same $x$, which sets the common factor $\|x\|^2$ and enters each expert's gate and up activations. The expert-specific weights determine the resulting response. Then we will demonstrate the revealence of such common information of shared input token $x$ will gradually appear with a toy example. **2. Repeated shared updates can accumulate weight alignment despite distinct feature maps.** **Setup of toy example.** Two independently initialized experts receive the same input through $d\in\{0,1,2,3\}$ frozen Linear–ReLU layers; only the final linear weights $A_i,A_j$ are updated. The $d=0$ case uses the raw input. We initialize weights from $\mathcal N(0,0.1^2)$, use input/hidden/output dimensions 12/16/8, and set slightly different fixed ReLU thresholds for about 30% activation. Each step independently draws $x_t\sim\mathcal N(0,I_{12}/12)$ and a synthetic gradient $g_t\sim\mathcal N(0,I_8/8)$; both experts share these draws and have mixing coefficient $1/\sqrt{2}$. For $e_k=A_kh_k(x)$, the trainable parameter Jacobian is $J_{A,k}=h_k(x)^\top\otimes I$. Its cross-expert Frobenius cosine equals the feature cosine. Shared nonzero output gradients also give the same parameter-gradient cosine. The SGD increments satisfy $$\langle\Delta A_{i,t},\Delta A_{j,t}\rangle_F =\frac{\eta_d^2}{2}\|g_t\|^2h_{i,d}(x_t)^\top h_{j,d}(x_t).$$ Overlapping ReLU activations contribute positively to within-step alignment. Independent zero-mean synthetic gradients cancel cross-step terms in expectation, allowing aligned components to accumulate. With one to three frozen layers, mean weight cosine increases from approximately zero to 0.088–0.114 by step 3 and 0.144–0.153 by step 10. **This example demonstrates an accumulation of shared components across different initial feature maps.** **Table R1. Final-layer weight alignment under repeated shared updates.** Values are Frobenius cosine similarities in the original neuron order.**Depth-specific learning rates compensate for decreasing feature magnitude.** | Frozen Linear–ReLU layers | Learning rate | Step 0 | Step 3 | Step 10 | Step 100 | | -| -| -| -| - | - | | 0 | 1 | −0.0016 ± 0.1047 | 0.5915 ± 0.1098 | 0.8343 ± 0.0427 | 0.9809 ± 0.0041 | | 1 | 5 | 0.0045 ± 0.0888 | 0.0862 ± 0.1161 | 0.1398 ± 0.1185 | 0.1745 ± 0.1126 | | 2 | 30 | −0.0037 ± 0.0868 | 0.0940 ± 0.1233 | 0.1431 ± 0.1226 | 0.1692 ± 0.1192 | | 3 | 200 | 0.0006 ± 0.0861 | 0.1097 ± 0.1473 | 0.1469 ± 0.1406 | 0.1562 ± 0.1248 |

3. Empirical prevalence of representation collapse during training. We modify Qwen3-0.6B as Qwen3-0.6B-A0.1B (6 layers, 128 experts per layer, Top-8), and trained it from random initialization for 5,000 optimizer steps on FineWeb. Each run processed 4,096 input tokens per step, about 20.48M tokens in total. We separately measure actual checkpoint changes Δκ=κ(T)−κ(100)\Delta\kappa=\kappa(T)-\kappa(100) along the full training trajectory. In practice, the extent of angular convergence among experts may vary across tasks, which could in turn affect how much GradRouter can mitigate it.

Table R1b. Net angular convergence in the exploratory L0 census. Fixed-input evaluation holds initial layer inputs constant across expert-weight checkpoints; current-input evaluation includes changes in upstream representations.

Input evaluationCheckpointsBalance coefficient λ\lambdaΔκ>10−4\Delta\kappa>10^{-4}Δκ>0.01\Delta\kappa>0.01Δκ>0.1\Delta\kappa>0.1
Fixed initial inputs100 → 50000.01137,446/157,852 (87.1%)84.0%42.3%
Fixed initial inputs100 → 50000.1130,584/157,852 (82.7%)79.4%37.6%

W2. Preserving the summed output gradient is not the same as preserving the original optimization objective, and providing the descent condition of GradRouter.

R2. The core operator of GradRouter is positive definite and therefore admitting a descent direction, and there is not an explicit loss for such gradient intervention. The GradRouter operator in Equation 11 can be written directly as

\qquad B_{t,i}=\alpha_{t,i}I+qV\operatorname{diag}\!\left(p_{t,i}-\mathbf{1}\right)V^\top,$$ where $\alpha_{t,i}>0$ is the routing weight of active expert $i$ for token $t$. Here, $p_{t,i}$ is auxiliary notation for the directional allocation shares, rather than an additional model parameter. Thus, $q<\alpha_{t,i}$ is a sufficient condition for positive definiteness. During our training , q is control strength of GradRouter which is smaller than 0.1. Therefore, for most activated experts, $B_{t,i}$ is positive definite and this gradient is a descent direction. > **W3 & Q3. The empirical evidence is not yet statistically robust and the main tables do not report variance, confidence intervals, or significance tests. The authors should report results across multiple random seeds, including confidence intervals and paired comparisons with standard backpropagation under identical forward routing.** **R3. We report GradRouter’s cross-seed mean and variance on OLMoE.** We report mean (sample variance) across these seeds (n=3). Means are percentages; variances use denominator n−1 and are in squared percentage points. A dash denotes incomplete three-seed evaluation. **The reproduced performance gap of AFBM might be attributed to used fine-tuning dataset and hyper-tuned coefficient of their regularization term.** Thanks you for raising this important problem, and we also notice such performance gap and had already carefully hyper-tuned for AFBM. In summary, our setup differs from the original AFBM study in two main respects: 1. **Fine-tuning data.** AFBM did not release their manually curated, high-quality dataset, so we constructed our fine-tuning dataset by sampling from open-source datasets. 2. **Regularization strength.** With both regularization coefficients set to the recommended $10^{-3}$ in the original paper, we observed substantial fluctuations in the expert variance-suppression term, unstable optimization, and performance considerably below even our currently reported results. Reducing the coefficient of this term stabilized training and yielded the reported AFBM results. If needed, we would be happy to provide our code and training configurations to facilitate verification and reproduction of these AFBM results. **Table R2. GradRouter on OLMoE across seeds 42, 43 and 44 (mean ± sample variance).** | Statistic | MMLU | MMLU-Pro | GSM8K | MATH500 | HumanEval | HumanEval+ | MBPP | MBPP+ | BBH | GPQA | LiveBench | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | GradRouter | 47.42 ± 0.859 | 17.24 ± 0.022 | 47.51 ± 0.268 | 17.67 ± 1.173 | 15.65 ± 1.612 | 12.60 ± 3.081 | 30.51 ± 0.094 | 24.60 ± 0.632 | 9.55 ± 5.360 | 21.04 ± 1.608 | 9.74 ± 0.084 |

W4 & Q4.The computational overhead makes the scalability claim of GradRouter less convincing.

R4. Corrected measurements show approximately 1.2× baseline runtime. The unexpectedly large cost also prompted us to revisit Figure 4 after submitting the manuscript. We notice that GradRouter's “Base” component in Figure 4 alone exceeded the total baseline runtime. We found that in Figure 4, baselines used ZeRO-2 while GradRouter used ZeRO-3, so that sharding and implementation differences are included. With matched ZeRO stages, GradRouter takes 1.163–1.185× baseline runtime under ZeRO-2 and 1.254–1.271× under ZeRO-3, as reported in Table R3 . We appreciate the reviewer's concern about practical efficiency and will update Figure 4 and the corresponding runtime analysis. And the updated per-component time cost of GradRouter is reported in Table R4.

Table R3. Per-step training time (/senconds) with matched ZeRO stages across 3 MoE models. Parentheses report GradRouter time divided by the corresponding baseline time. These profiles measure runtime, not downstream performance at matched compute.

ArchitectureConfigurationSFTAuxLossLoss-FreeGradRouter
OLMoEZeRO-241.32 (1.185×)41.77 (1.173×)42.12 (1.163×)48.98
OLMoEZeRO-362.73 (1.271×)63.34 (1.259×)63.56 (1.254×)79.73
DeepSeek-V2-LiteZeRO-269.26 (1.164×)70.47 (1.144×)69.69 (1.157×)80.61
DeepSeek-V2-LiteZeRO-3108.66 (1.268×)110.89 (1.243×)111.04 (1.241×)137.82
Qwen3-30B-A3B-BaseZeRO-3377.95 (1.241×)385.72 (1.216×)386.56 (1.214×)469.12

Table R4. GradRouter runtime breakdown on OLMoE (/seconds). “Regular” denotes a step without PCA/affinity refresh; “refresh” includes those updates; “amortized” averages one refresh step and four regular steps.

GradRouter componentZeRO-2 regularZeRO-2 refreshZeRO-2 amortizedZeRO-3 regularZeRO-3 refreshZeRO-3 amortized
Base38.4837.7138.3267.3465.6667.00
PCA0.0020.294.060.0020.884.18
Affinity0.003.600.720.006.611.32
Assignment0.570.420.540.620.490.59
Correction5.385.145.336.646.636.64
Total44.4367.1648.9874.60100.2679.73

Minor Comments about typos.

Thanks for pointing out these typos and we will perform careful proofreading to correct them.

Reviewer_Lgv5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes GradRouter, a novel router optimization framework to address an overlooked limitation in MoE training. Typically, co-activated experts in traditional MoE receive collinear output gradients during training. The authors argue that this reflects an undesirable coupling between forward prediction responsibility and backward learning responsibility, and propose GradRouter to address this issue through gradient redistribution.

Specifically, GradRouter first constructs a shared gradient basis by applying PCA to recent output gradients. It then estimates an expert–direction affinity matrix to measure each expert's ability to change its output along different principal directions. Based on this affinity matrix, the Top-K activated experts, and their routing weights, GradRouter optimizes a learning responsibility assignment matrix to decompose and redistribute the output gradient into expert-specific directional gradients.

The authors conduct experiments to evaluate the effectiveness of GradRouter.

Critical strengths

(1) The observation that co-activated experts receive collinear output gradients and has been largely overlooked in prior research. The authors identify this issue and propose a novel and carefully designed approach to address.

(2) The paper is mostly well-written and easy to follow.

Critical weaknesses

(1) Issues with the main experiments. The authors compare GradRouter against routing designs orthogonal to their design, which may fail to provide sufficient evidence of its effectiveness. (See Questions)

(2) GradRouter introduces excessive complexity and has limited practicality. Such overhead is unacceptable for practical MoE training, as training the baselines with the same compute budget could yield comparable or greater gains.

I appreciate the authors' insight, these two critical weaknesses call the soundness of the paper into question.

Questions for authors

Q1. Could the authors extend the experiments in Appendix C.1 to evaluate GradRouter combined with the other routing designs, across all MoE backbones?

Q2. Can the authors evaluate GradRouter in MoE pretraining rather than only LoRA-based instruction tuning? For example, a scaling-ladder experiment across different model sizes and training budgets would help.

Q3. Does GradRouter actually accelerate convergence under the same computational budget?

Figure 3 only compares training loss againststeps, while Figure 4 shows substantial training overhead. Could the authors provide compute-matched comparisons, such as validation loss and downstream performance against GPU hours, to demonstrate it offers a meaningful advantage.


Some other comment: I'm not convinced by the choice of baselines in the main experiments. AuxLoss, Loss-Free Balancing, and AFBM mainly address load balancing or expert specialization, which are largely orthogonal to MoE router optimization. It would make more sense to test GradRouter's compatibility with these methods, rather than treating them as direct competitors. I also find the use of expert bias in the Loss-Free baseline questionable. The pretrained backbones do not originally use expert-wise routing biases, yet the authors introduce them during SFT. This seems particularly odd for OLMoE, which uses softmax routing (which is not so compatible with expert bias). Such a change could affect the pretrained routing behavior and make the comparison less reliable. The authors should explain why this setup is reasonable and how it affects the results.

4 comments

W1.1 & Additional Comment. Provided baselines focus on load balancing or expert specialization, which are largely orthogonal to GradRouter and should not be considered as direct competitors.

R1.1. "Baselines" and additional SFT reference. Thank you for raising these questions about the baselines. As you noted, the problem of collinear output gradients has been largely overlooked, making it difficult to identify suitable baselines. To the best of our knowledge, there is no existing method likewisely investigating how to allocate each token’s task-gradient components across co-activated experts like GradRouter. We therefore include representative MoE training methods, AuxLoss, Loss-Free Balancing, and AFBM for they also modify how experts learn in a broader sense. We add vanilla SFT as a more direct comparison with GradRouter in Table R1, hoping this comparison helps illustrate the benefit of decomposing output gradients in MoE training.

Table R1. Vanilla SFT and GradRouter on OLMoE and DeepSeek-V2-Lite.

MethodMMLUMMLU-ProGSM8KMATH500HumanEvalHumanEval+MBPPMBPP+BBHGPQALiveBench
OLMoE
SFT46.8717.3045.9418.0014.6310.9830.9525.939.0621.729.46
GradRouter47.2517.3446.9319.0017.0714.6330.6925.408.6521.219.41
DeepSeek-V2-Lite
SFT52.6220.0535.1013.4026.8321.3445.2438.6235.1431.829.95
GradRouter56.4725.0544.9620.6036.5930.4959.5249.2147.3030.3012.86

Compatibility with existing methods (Q1). Thank you for your thoughtful suggestion. GradRouter and the other methods we compare address different aspects of MoE training. Thus we also think it is worth testing them in combination. We tested these combinations both during the development of GradRouter and again after receiving your suggestion, demonstrating the combined results on DeepSeek-V2-Lite in Table R2. GradRouter is designed to allocate directional task-gradient signals. AuxLoss introduces an objective shared across experts, and that GradRouter’s allocation scheme may not be appropriate for this objective. Nevertheless, we find these combinations worth investigating further and will revisit them in future work.

Table R2. GradRouter combined with AuxLoss or Loss-Free Balancing on DeepSeek-V2-Lite.

MethodMMLUMMLU-ProGSM8KMATH500HumanEvalHumanEval+MBPPMBPP+BBHGPQALiveBench
AuxLoss56.4624.4345.2617.8036.5930.4959.7949.7446.8930.8112.32
Loss-Free56.2924.3845.9419.2035.3731.1058.2048.4146.9229.2912.71
GradRouter56.4725.0544.9620.6036.5930.4959.5249.2147.3030.3012.86
GradRouter + AuxLoss53.0019.5935.3314.2025.0020.7346.0340.2138.5528.7910.48
GradRouter + Loss-Free51.4020.2432.9010.8027.4423.1743.9239.1537.4830.3010.25

W2&Q3. GradRouter introduces excessive complexity and has limited practicality.

R2. Corrected measurements show approximately 1.2× baseline runtime. The unexpectedly large cost also prompted us to revisit Figure 4 after submitting the manuscript. We notice that GradRouter's “Base” component in Figure 4 alone exceeded the total baseline runtime. We found that in Figure 4, baselines used ZeRO-2 while GradRouter used ZeRO-3, so that sharding and implementation differences are included. With matched ZeRO stages, GradRouter takes 1.163–1.185× baseline runtime under ZeRO-2 and 1.254–1.271× under ZeRO-3, as reported in Table R3 . We appreciate the reviewer's concern about practical efficiency and will update Figure 4 and the corresponding runtime analysis. And the updated per-component time cost of GradRouter is reported in Table R4.

Table R3. Per-step training time (/senconds) with matched ZeRO stages across 3 MoE models. Parentheses report GradRouter time divided by the corresponding baseline time. These profiles measure runtime, not downstream performance at matched compute.

ArchitectureConfigurationSFTAuxLossLoss-FreeGradRouter
OLMoEZeRO-241.32 (1.185×)41.77 (1.173×)42.12 (1.163×)48.98
OLMoEZeRO-362.73 (1.271×)63.34 (1.259×)63.56 (1.254×)79.73
DeepSeek-V2-LiteZeRO-269.26 (1.164×)70.47 (1.144×)69.69 (1.157×)80.61
DeepSeek-V2-LiteZeRO-3108.66 (1.268×)110.89 (1.243×)111.04 (1.241×)137.82
Qwen3-30B-A3B-BaseZeRO-3377.95 (1.241×)385.72 (1.216×)386.56 (1.214×)469.12

Table R4. GradRouter runtime breakdown on OLMoE (/seconds). “Regular” denotes a step without PCA/affinity refresh; “refresh” includes those updates; “amortized” averages one refresh step and four regular steps.

GradRouter componentZeRO-2 regularZeRO-2 refreshZeRO-2 amortizedZeRO-3 regularZeRO-3 refreshZeRO-3 amortized
Base38.4837.7138.3267.3465.6667.00
PCA0.0020.294.060.0020.884.18
Affinity0.003.600.720.006.611.32
Assignment0.570.420.540.620.490.59
Correction5.385.145.336.646.636.64
Total44.4367.1648.9874.60100.2679.73

Q2. Apply GradRouter to pre-training beyond LoRA fine-tuning.

RQ2. Pretraining is an important direction for future work. Thank you for your interest in extending GradRouter beyond fine-tuning. Full pretraining requires substantially more compute and training time than LoRA fine-tuning and is not feasible within our limited computational resources. We are sorry that our paper would still foucus on fine-tuning without claiming pretraining results, leaving such extension to future work.

Reviewer_Lgv5OpenReview ↗

Rebuttal Acknowledgement

Thanks for the authors' quick response.

However, I remain unconvinced by the authors' explanation that "AuxLoss introduces an objective shared across experts, and that GradRouter’s allocation scheme may not be appropriate for this objective."

This explanation seems speculative and does not adequately explain why the two methods cannot be effectively combined. Moreover, the lack of rigor in the efficiency evaluation makes me even more concerned about this work.

Given these considerations, I will maintain my current score and further communicate my concerns to the AC.

Thank you for your thoughtful review and kind follow-up. We are sorry for our earlier speculative explanation about the worse performance of GradRouter after combination. We would explain this in more detail.

AuxLoss may reduce the consistency of GradRouter’s directional credit allocation over training. GradRouter uses each expert’s routing weight to constrain its total credit. While AuxLoss would regularize the routing weights of heavily used experts. If AuxLoss lowers the routing weight of an expert that has repeatedly received credit along a particular gradient direction for similar tokens, some of that credit may shift to other experts even if the original expert retains high affinity for that direction. This redistribution could weaken the cumulative effect of repeated updates along that direction within the same expert. We think this might be a possible explanation for the weaker performance observed when combining the two methods.

Concern about runtime evaluation. We sincerely apologize for the error in our original runtime comparison. We understand that the incorrect timing comparison may have raised broader concerns about the fairness and credibility of our evaluation. The computational cost of the additional PCA operations can be analyzed using standard results from the literature, and we expected this part of the overhead to be modest. In the paper, we reported the timings as measured at the time, including the results that were unfavorable to our method. After carefully reviewing the timing setup, we identified the error and corrected the comparison. We respect your assessment and hope our correction could reassure you that we reported the timings in good faith.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.