- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies the relationship between load balancing and expert specialization in mixture-of-experts (MoE) models. The authors introduce a family of squared-loss objectives that interpolate between selected-expert and averaged-output training.
- The main motivation of the paper is that balanced routing does not necessarily imply expert specialization. However, this is already a known observation and is not really surprising.
- The authors should clarify what is fundamentally new compared with previous work on MoE specialization and load balancing.
- The proposed interpolation between selected-expert and averaged-output losses is based on an already known squared-loss decomposition.
- It is not clear why this interpolation is interesting or useful in practical MoE training.
- The paper does not clearly explain why intermediate values of the interpolation parameter (q) should be considered.
- The exponential dependence of the recovery time on the initial logit gap seems to follow naturally from softmax probabilities.
Critical strengths
Critical weaknesses
- The paper has a very poor formatting. The paper is very dense, with too many mathematical formulations and technical details that are difficult to undestand. it looks more like a draft or unfinished work.
- The goal of the paper is not clearly defined and ooks like a concatenation of different theories, ideas, and formulations without a clear connection between them and to identify one clear central contribution
- The main theoretical analysis is based on a very simplified model with only two scalar linear experts and a binary routing cue.
- It is not clear how the results generalize to actual MoE models and high-dimensional inputs.
- Some results seem to be consequences of the simplified model rather than general properties of MoE training.
- The assumptions and limitations of the different theoretical results are not sufficiently discussed.
Questions for authors
- See above