- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes GradRouter, a framework that decouples backward learning responsibility from forward routing in sparse MoE models. It decomposes output gradients into principal directions and assigns them to co-activated experts according to their direction-specific capabilities, while preserving the aggregate gradient.
Critical strengths
- The paper provides a novel distinction between prediction responsibility and learning responsibility, revealing a fundamental limitation of standard MoE backpropagation.
- GradRouter offers a principled approach to assigning direction-specific learning signals while preserving the aggregate output gradient.
- The method is validated across multiple MoE architectures and scales, showing consistent improvements, particularly on code-generation tasks.
Critical weaknesses
- The benefits of expert specialization are expected to emerge in multi-task learning and generalization, as discussed in the paper, but the experiments do not directly evaluate either aspect.
- The empirical improvements are not fully consistent across architectures and tasks, and GradRouter occasionally underperforms the strongest baseline, making its overall effectiveness less conclusive.
- GradRouter introduces additional PCA, affinity estimation, and optimal-transport computations, but the paper provides limited empirical analysis of the resulting training-time and memory overhead at scale.
Questions for authors
please refer to Weakness
3 comments
Thank you for your rebuttal. It has addressed some of my concerns, and I will maintain my evaluation of "Weak accept."
Thank you for your positive assessment and constructive feedback. We are glad that our rebuttal addressed some of your concerns.
R1. Our training and evaluation cover multiple task types. Our training dataset includes coding, mathematics, and reasoning tasks. We also evaluate downstream generalization on diverse benchmarks spanning these task types, including HumanEval and MBPP for coding, GSM8K and MATH500 for mathematics, and BBH and GPQA for reasoning.
Thank you for pointing this out. We agree that GradRouter trails the best-performing baseline in some settings. The baselines are competitive methods that target aspects of MoE training such as load balancing and expert specialization. GradRouter addresses how learning credit is assigned to active experts. The mixed results suggest that the gains from each intervention may depend on the backbone and task. We will qualify our discussion accordingly. We includes vanilla SFT in Table R1 as a reference for training with standard backpropagation.
R3. Corrected measurements show approximately 1.2× baseline runtime. The unexpectedly large cost also prompted us to revisit Figure 4 after submitting the manuscript. We notice that GradRouter's “Base” component in Figure 4 alone exceeded the total baseline runtime. We found that in Figure 4, baselines used ZeRO-2 while GradRouter used ZeRO-3, so that sharding and implementation differences are included. With matched ZeRO stages, GradRouter takes 1.163–1.185× baseline runtime under ZeRO-2 and 1.254–1.271× under ZeRO-3, as reported in Table R1 . We appreciate the reviewer's concern about practical efficiency and will update Figure 4 and the corresponding runtime analysis. And the updated per-component time cost of GradRouter is reported in Table R2.
Table R1. Per-step training time (/senconds) with matched ZeRO stages across 3 MoE models. Parentheses report GradRouter time divided by the corresponding baseline time. These profiles measure runtime, not downstream performance at matched compute.
Table R2. GradRouter runtime breakdown on OLMoE (/seconds). “Regular” denotes a step without PCA/affinity refresh; “refresh” includes those updates; “amortized” averages one refresh step and four regular steps.