acceptodds
Under review as a conference paper at ICLR 2027

SPECIALIZATION AND SLOW RECOVERY IN JOINTLY TRAINED MIXTURES OF EXPERTS

Abstract

Marginal load balance alone does not determine whether experts specialize; the prediction objective also matters. We study a family of squared losses that interpolates between selected-expert and averaged-output training. For two jointly trained scalar linear experts with a binary routing cue, we prove a global phase diagram under positive balancing. When , the thresholds and separate full specialization, stable partial specialization, and homogenization. Here combines balancing strength with batch-cue imbalance, and is the cue-conditional regression signal. For every , expert parameters remain bounded, asymptotic starvation is excluded, and every trajectory from a finite initialization converges to a single state. At , each trajectory at the critical threshold selects one point on a stationary continuum. For multiple smooth experts with fixed-feature softmax routing, we derive a local Hessian criterion that separates expert-separation curvature, routing–residual coupling, and balancing curvature. At , the binary linear objective has no finite minimizer when the routing-contrast penalty is positive. For initial conditions in a specified invariant set, expert parameters diverge while routing approaches uniformity and cue-dependent prediction persists. For nonlinear experts, curvature can alter stability at the boundary. For selected-expert training, we also prove matching population load-recovery bounds of order from an initial logit gap . For simultaneous no-baseline categorical updates, a corrected potential controls shared router–expert noise and yields a recovery-time second-moment bound under explicit bounded-data and step-size conditions.

Primary area:
learning theory
Keywords:
mixture-of-experts, expert specialization, load balancing, softmax routing, joint training, load recovery

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_avKnOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the relationship between load balancing and expert specialization in mixture-of-experts (MoE) models. The authors introduce a family of squared-loss objectives that interpolate between selected-expert and averaged-output training.

  • The main motivation of the paper is that balanced routing does not necessarily imply expert specialization. However, this is already a known observation and is not really surprising.
  • The authors should clarify what is fundamentally new compared with previous work on MoE specialization and load balancing.
  • The proposed interpolation between selected-expert and averaged-output losses is based on an already known squared-loss decomposition.
  • It is not clear why this interpolation is interesting or useful in practical MoE training.
  • The paper does not clearly explain why intermediate values of the interpolation parameter (q) should be considered.
  • The exponential dependence of the recovery time on the initial logit gap seems to follow naturally from softmax probabilities.
Critical strengths
Critical weaknesses
  • The paper has a very poor formatting. The paper is very dense, with too many mathematical formulations and technical details that are difficult to undestand. it looks more like a draft or unfinished work.
  • The goal of the paper is not clearly defined and ooks like a concatenation of different theories, ideas, and formulations without a clear connection between them and to identify one clear central contribution
  • The main theoretical analysis is based on a very simplified model with only two scalar linear experts and a binary routing cue.
  • It is not clear how the results generalize to actual MoE models and high-dimensional inputs.
  • Some results seem to be consequences of the simplified model rather than general properties of MoE training.
  • The assumptions and limitations of the different theoretical results are not sufficiently discussed.
Questions for authors
  • See above
Reviewer_ujMyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies theoretical guarantees of MoE training under a convex combination of average-expert loss and selected-expert loss. The setup considered by the paper is rather simple, with a binary latent task emitted to the experts' observation through a noisy cue. Under a load-balancing regualarization, the paper analyzes the phase diagram of which mixing level introduces no, partial, or ful specialization. In the meantime, the paper also considers the case of multi-task scenario, and show that gradient-based training takes on average exponential time for the router to converge. The paper accompanied the results with experiments on 4-layer Transformer-MoE.

Critical strengths
  1. The paper tackles an interesting problem of when specialization happens. Under a simple setup, the paper provides quite comprehensive theoretical result to decribe different phases of expert specialization.

  2. The paper's analysis on the recovery time of expert load is quite interesting, and the analysis holds under a multi-task scenario.

Critical weaknesses
  1. The theoretical model is quite simplified. The expert is assumed to be linear, which is rarely used in practice. In the meantime, there has been prior work (Chen et al. 2022) that shows the importance of a non-linear expert might induce a different dynamic. Also, whether it is safe to assume that the task signal is emitted to the expert in the simple form of a conditional random variable is not something well-justified.

  2. The phase diagram is limited to the case of two-tasks. There does not seem to be a comparable result in the multi-task case. While Theorem 2 provides a coupling critera, the result relies on the positive definiteness of matrices that involves the Hessian, whose information is hard to extract and control.

  3. Experiments does not seem to verify the main theoretical results, which would be the phase diagram or the load recovery speed.

  4. Notations are unclear:

a). gg, w±w_{\pm}, β\beta, and α\alpha are not explicitly defined (e.g. there should be an explicit statement that β\beta and α\alpha are router parameters).

b). Loss should be given a comprehensive form instead of specifying load-balacing term separately.

c). Not sure what is λX\lambda_{\mathcal{X}} and kk and what's there relationship, except for the equation that k=λXk = \lambda_{\mathcal{X}}. It is unclear what these two notations represent.

  1. Some places are not justified sufficiently. For instance, it is unclear how Eq. (5) is derived.
Questions for authors

Isn't bb in Section 2.2 guaranteed to be 00? Why do we care about its dynamic?

What do you mean by "Boundedness and the barrier confine trajectories; dissipation and boundary signs make the classification global"?

Why is it that "Along these trajectories, the marginal expert loads remain 1/2 and both conditional routing probabilities approach 1/2, while the expert parameters diverge with opposite signs."

What is DD in Eq. (18)?

How to guarantee that sigmoid$(-M/T) < r$?

Reviewer_fiWZOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This is an interesting paper because it provides theoretical explainations for training problems in MoE models and shows that load balancing alone may not be enough to ensure effective expert specialization. It studies why experts sometimes take a long time to recover. Through theoretical analysis and neural network experiments, the authors show how prediction loss, load-balancing strength, and batch composition affect expert specialization, and demonstrate that recovery time under Softmax routing can grow exponentially with the initial routing gap.

Critical strengths
  1. The authors use theoretical analysis to explain a problem that are often treated as practical training issues. They explains why some expert fail to specialize and slow to recover and how they relate to the loss function and load balancing. This makes the finding useful for understanding MoE training.

  2. The authors show an overlooked problem with softmax routing. They found that even when an expert is underused, it may not recover quickly because the initial routing gap can make recovery extremely slow. I think this is an important insight because it suggests that balancing expert useage is not just encouraging the router to select each expert more often.

Critical weaknesses

As a non-theory researcher, I have concerns on the practical aspects:

First, the authors shows their understanding on how batch composition and balancing strength affect expert specialization. I was hoping to see the paper discuss on how we can use these findings to improve MoE training in practice? For example, can we make the underused experts to recover faster by changing the balancing loss or batch composition? (I would appreciate to see some of the discussion about how their finding can help on the MoE training)

Second, I understand the author create the theory based on a simple model with two linear experts for making the analysis easier. But, I am not sure how well these findings generalize to modern MoE models with many experts and complicated routing. Especially, I am curious if the same slow recovery problem happens in large scale MoE models.

Questions for authors

I understand the rebuttal time period is short and compute constraint is there. So, I do not expect the author to give me a perfect rebuttal answer.

Similar to what I discussed in the weakness part, I understand the theory shows that the underused experts can take exponentially long to recover under softmax routing. But, in real world, the MoE models often use top-k routing and not sampling experts from a softmax distribution. Do you expect the same slow recovery problem in these modern MoE models? What is your intuition on it?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.