acceptodds
Withdrawn from ICLR 2027

Routing in Gradient Space: Balanced Usage Is Not Expert Specialization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sparse expert models can distribute traffic evenly while still grouping incompatible training signals within the same experts. We study routing as a gradient-partitioning problem and introduce gradient-aligned routing (GAR), whose load-normalized router objective rewards grouping observations with aligned gradients. On five multi-task text-classification mixtures, we compare GAR with task-loss-only routing, gradient-combination and gradient-conflict methods, and load-balancing losses. With a fully trainable RoBERTa backbone and classification-head experts, GAR has the highest aggregate validation accuracy, percentage points above task-loss-only routing. With frozen DeBERTa and Qwen3-1.7B backbones and low-rank adapter experts, it again ranks first, points above task-loss-only routing, with better-balanced expert load and higher gradient-mass purity, the share of each expert's gradient-norm mass from its dominant task; the load-balancing losses flatten load further but leave this purity near its task-loss-only level. Top-1 routing, trainable full-parameter feed-forward network (FFN) experts, and a larger backbone also show positive aggregate gains. The results distinguish expert-load balance from gradient-based routing organization and indicate the predictive value of gradient-informed routing in multi-task text classification.

Primary area:
optimization
Keywords:
mixture of experts, expert routing, multi-task learning, gradient-based routing, expert specialization, load balancing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_q7p2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the distinction between balanced expert utilization and meaningful expert specialization in Mixture-of-Experts (MoE) models. Then, this paper proposes Gradient-Aligned Routing (GAR), which formulates expert routing as a gradient-space partitioning problem. Specifically, GAR introduces a load-normalized objective that encourages observations with aligned gradients to be assigned to the same experts. This method employs detached gradient observations to optimize the router without introducing second-order derivatives. This paper also provides theoretical connections to k-means clustering and spectral partitioning. Experiments on multiple text classification mixtures using RoBERTa, DeBERTa, and Qwen3 demonstrate improvements in validation accuracy and several routing-related metrics over existing baselines.

Critical strengths
  1. This paper proposes an important problem with conventional load-balancing objectives in MoE models: balanced expert utilization does not necessarily imply meaningful specialization. Formulating routing from the perspective of gradient compatibility provides an interesting direction beyond conventional load balancing.
  2. GAR introduces a relatively simple gradient-alignment objective and employs detached gradients to avoid second-order optimization. The connections to k-means clustering and spectral partitioning provide additional intuition for the proposed routing criterion.
  3. The experiments cover multiple backbone architectures, frozen and trainable backbones, different expert parameterizations, and top-1 and top-4 routing configurations. The comparisons include several gradient-based and load-balancing baselines, with multiple random seeds and additional analyses of routing behavior.
Critical weaknesses
  1. The writing of this paper should be improved, for example:
  • The motivation of this paper in the Introduction section should be clear. The introduction of the paper does not clearly state what problems with existing methods the GAR was designed to address, nor does it explain what advantages it offers over other methods.

  • The layout of the chart on page 9 is awkward, with large areas of blank space.

  • On page 7, when discussing baseline methods and models, the references were not cited.

  • Some of the formulas in the main paper have numbers, while others do not. Please standardize them.

  • The framework figure (Figure 1:) is too simple. The authors might consider adding more images to the main paper to make it more visually engaging.

  1. The novelty of this paper is limited. The central objective is closely related to normalized gradient clustering, while the theoretical connections to k-means and spectral partitioning follow established results under restrictive assumptions. Although applying this criterion to MoE routing is reasonable, this paper cannot sufficiently establish what is fundamentally new beyond existing gradient-based clustering and conflict-aware routing methods.

  2. The experiments are restricted to supervised English text classification. The paper lacks experiments on a wider variety of datasets.

  3. Although the framework is motivated by token-level gradient compatibility, the reported experiments construct gradient observations from same-task groups of up to eight examples. These aggregated observations may obscure within-task gradient conflicts and do not directly represent the individual tokens being routed. Furthermore, the theoretical continuous-assignment analysis does not fully reflect the optimization behavior of top-k masked routing, where unselected experts receive no direct gradient through the masked probabilities.

Questions for authors
  1. How much of GAR's improvement comes from cross-observation gradient alignment rather than implicit routing sparsification?
  2. How reliable are gradient similarities computed from independently parameterized experts? In particular, how sensitive is GAR to equivalent reparameterizations of the LoRA factors or to differences in expert parameter initialization?
  3. Can the authors provide additional results on more realistic MoE language modeling or instruction-tuning settings, together with training-time and memory-overhead measurements on larger backbones?
Reviewer_dwQYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Load balancing in MoE models does not ensure that items sharing an expert have compatible gradients. The paper proposes gradient-aligned routing (GAR), a load-normalized auxiliary loss that trains the router to group items with aligned gradients. Under idealized conditions, the objective is equivalent to k-means on gradients.

On multi-task text classification, GAR improves accuracy over task-loss-only routing and load-balancing baselines. The results also show that more balanced expert usage does not necessarily lead to greater expert specialization.

Critical strengths

1. Consistent gains with careful evaluation. GAR improves accuracy over task-loss-only routing in all frozen backbone–mixture combinations. Table 1a also reports positive paired confidence intervals across trainable and frozen backbones, head and FFN experts, and top-1 and top-4 routing. Hyperparameters are selected on held-out training data, and Baseline receives a larger search budget. Results use five seeds. These checks make the small gains more credible.

2. A low-cost loss that trains only the router. GAR reuses task gradients and requires no second-order derivatives. The auxiliary loss sends gradients only to the router and adds little step-time overhead in the reported DeBERTa runs. This makes the method practical, although whether cross-item gradient alignment explains the gains remains unclear.

Critical weaknesses

1. The affinity includes other effects that are not ablated. The affinity ⟨g~i,g~j⟩\langle\tilde{g}_i,\tilde{g}_j\rangle includes cross-expert terms that compare independently parameterized experts coordinate by coordinate (Section 3.3). These terms can change under a function-preserving permutation of one LoRA expert. It also includes self-terms that can favor concentrated routing without cross-item alignment (Appendix B.1). I do not see ablations removing these components separately. The denominator ablation in Table 2b does not separate them. Comparing concatenated expert gradients and removing self-terms would help show whether cross-item alignment drives the gains.

2. Same-task grouping may explain part of the gains. All reported experiments use same-task gradient groups (Appendix G.2). This may give the router a task-related signal, but I do not see comparisons with task-aware methods such as Mod-Squad [1] or Task-MoE [2]. The single-task controls show gains without task identity, but still retain self-terms. A task-aware baseline and GAR with mixed-task or per-example observations would help separate these effects. Without them, it is unclear whether the multi-task gains come from gradient alignment or task grouping.

References:

[1] Chen et al. Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learners. CVPR 2023.

[2] Kudugunta et al. Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference. Findings of EMNLP 2021.

Questions for authors

These checks could use the fixed-configuration protocol of Table 2, with a separate coefficient sweep for each objective.

1. Cross-expert terms: Do the accuracy and purity gains remain when per-expert gradients are concatenated instead of summed? At a fixed checkpoint, how much do the auxiliary assignment scores change after a function-preserving rank permutation of one LoRA expert?

2. Self-terms: How do the full, diagonal-only, and off-diagonal-only objectives compare in accuracy, load variance, utilization, and gradient-mass purity? Please also test the single-task setting to check whether its gains remain without self-terms.

3. Task identity: How does GAR compare with a task–expert mutual-information loss or task-level routing, as used in Mod-Squad and Task-MoE? Do the gains remain with mixed-task groups or per-example observations?

These checks would help show whether cross-item gradient alignment explains the gains.

Reviewer_ZTQtOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose Gradient-Aligned Routing (GAR), reformulating routing as a partitioning problem in gradient space. The method designs a load-normalized objective that encourages assigning observations with aligned, compatible gradient directions to the same expert, while utilizing gradient detachment to update the router using only first-order derivatives, thereby avoiding expensive second-order computational overhead. Across multi-task benchmark mixtures containing 5 to 8 English text-classification tasks, GAR achieves an accuracy improvement of approximately 1.1 percentage points over the standard baseline, while attaining higher "gradient-mass purity" and balanced expert utilization.

Critical strengths
  1. Overcoming the limitation of conventional MoEs that rely solely on hidden feature representations for routing, this paper formalizes MoE routing as a soft partitioning problem in parameter gradient space, directly tackling the central pain point of multi-task learning—gradient conflict.
  2. The proposed algorithm is elegantly designed and highly practical, achieving both computational efficiency and plug-and-play modularity.
Critical weaknesses
  1. The authors position their method largely by contrasting it against load balancing, contending that balanced routing fails to guarantee expert specialization—a premise that is conceptually inaccurate. Load balancing is fundamentally an operational mechanism to avoid routing collapse and maximize throughput under expert parallelism, not a design principle meant to induce functional specialization. Treating load balancing as an ineffective route to expert specialization attacks a recognized engineering trade-off rather than dispelling a genuine misconception.
  2. Bypassing realistic distributed communication overhead and lacking efficiency comparisons against load-balancing baselines.
Questions for authors
  1. Given that standard load-balancing objectives are strictly intended for hardware efficiency rather than functional specialization, how does GAR ensure that training throughput does not degrade in a real distributed expert-parallel setup when load variance increases?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.