- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the distinction between balanced expert utilization and meaningful expert specialization in Mixture-of-Experts (MoE) models. Then, this paper proposes Gradient-Aligned Routing (GAR), which formulates expert routing as a gradient-space partitioning problem. Specifically, GAR introduces a load-normalized objective that encourages observations with aligned gradients to be assigned to the same experts. This method employs detached gradient observations to optimize the router without introducing second-order derivatives. This paper also provides theoretical connections to k-means clustering and spectral partitioning. Experiments on multiple text classification mixtures using RoBERTa, DeBERTa, and Qwen3 demonstrate improvements in validation accuracy and several routing-related metrics over existing baselines.
Critical strengths
- This paper proposes an important problem with conventional load-balancing objectives in MoE models: balanced expert utilization does not necessarily imply meaningful specialization. Formulating routing from the perspective of gradient compatibility provides an interesting direction beyond conventional load balancing.
- GAR introduces a relatively simple gradient-alignment objective and employs detached gradients to avoid second-order optimization. The connections to k-means clustering and spectral partitioning provide additional intuition for the proposed routing criterion.
- The experiments cover multiple backbone architectures, frozen and trainable backbones, different expert parameterizations, and top-1 and top-4 routing configurations. The comparisons include several gradient-based and load-balancing baselines, with multiple random seeds and additional analyses of routing behavior.
Critical weaknesses
- The writing of this paper should be improved, for example:
-
The motivation of this paper in the Introduction section should be clear. The introduction of the paper does not clearly state what problems with existing methods the GAR was designed to address, nor does it explain what advantages it offers over other methods.
-
The layout of the chart on page 9 is awkward, with large areas of blank space.
-
On page 7, when discussing baseline methods and models, the references were not cited.
-
Some of the formulas in the main paper have numbers, while others do not. Please standardize them.
-
The framework figure (Figure 1:) is too simple. The authors might consider adding more images to the main paper to make it more visually engaging.
-
The novelty of this paper is limited. The central objective is closely related to normalized gradient clustering, while the theoretical connections to k-means and spectral partitioning follow established results under restrictive assumptions. Although applying this criterion to MoE routing is reasonable, this paper cannot sufficiently establish what is fundamentally new beyond existing gradient-based clustering and conflict-aware routing methods.
-
The experiments are restricted to supervised English text classification. The paper lacks experiments on a wider variety of datasets.
-
Although the framework is motivated by token-level gradient compatibility, the reported experiments construct gradient observations from same-task groups of up to eight examples. These aggregated observations may obscure within-task gradient conflicts and do not directly represent the individual tokens being routed. Furthermore, the theoretical continuous-assignment analysis does not fully reflect the optimization behavior of top-k masked routing, where unselected experts receive no direct gradient through the masked probabilities.
Questions for authors
- How much of GAR's improvement comes from cross-observation gradient alignment rather than implicit routing sparsification?
- How reliable are gradient similarities computed from independently parameterized experts? In particular, how sensitive is GAR to equivalent reparameterizations of the LoRA factors or to differences in expert parameter initialization?
- Can the authors provide additional results on more realistic MoE language modeling or instruction-tuning settings, together with training-time and memory-overhead measurements on larger backbones?