- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes MoUE, a method for improving parameter utilization in MoE architectures. Specifically, it introduces a new scaling dimension, Virtual Width, which builds on MoE’s parameter efficiency through cross-layer expert sharing. The authors propose three novel components: Staggered Rotational Topology, Universal Expert Load Balance, and Universal Router. To further enhance scalability, they also describe how pretrained MoE models can be expanded along the virtual-width dimension. This offers a promising approach to continued training, as it can improve parameter utilization without increasing the physical parameter count.
Critical strengths
- The MoUE passage proposed gives another dimension to scaling the MoE Model.
- Without increasing total parameters or activated parameters, scaling virtual parameters increases model capability with no more devices or inference cost.
- The paper proposes a series of supporting facilities inclusive of Staggered Rotational Topology, Universal Expert Load Balance and Universal Router, making MoUE training stable and load balancing.
- The passage not only limit to train a MoUE from scrath, proposing curriculum routing warmup method to help trained MoE transfering to MoUE, which is quiet promising in this model explosion era.
Critical weaknesses
- The main text is not fully self-contained. For example, π in Eq. (1) and the Sim function used in the dual-pathway routing formulation in Section 4.4 are not clearly defined where they first appear. Although some details are provided in the appendix, these definitions should be included at their points of use in the main text. Additionally, the experimental section introduces depth expansion without sufficiently explaining how it is implemented.
- The improvement shown in the experiment is limited: only average +0.7 with double VP(MoUE 36A2/MoUE 18A2) in Table 1 and +1.1 after MoUE adaption in JetMoE. This result shows the risk of cost-inefficiency for MoUE training or adaption.
- The experiment is constraint in a small volume model and outdated model architecture.
- MoUE introduces cross-layer expert sharing. However, training and inference often employ pipeline parallelism. When shared experts reside on different GPUs or nodes, cross-layer access may incur additional inter-GPU or inter-node communication, potentially reducing training and inference efficiency
Questions for authors
- MoUE’s gains over MoE appear modest. In Table 1, approximately doubling the virtual parameter count yields only a 0.7-point improvement. Similarly, in the JetMoE conversion experiments, increasing VP from 8B to 14.6B—nearly twofold—results in only a 1.1-point improvement. Could the authors discuss prior methods that similarly leverage expert parameter reuse and compare their gains under a comparable degree of reuse, such as an approximately twofold increase in virtual parameters?
- Compared with MoE, MoUE requires cross-layer expert access, which may introduce additional inter-GPU or inter-node communication under pipeline parallelism. The MoUE architecture also appears to incur additional computational overhead. Could the authors conduct training and inference experiments with pipeline parallelism enabled and shared experts accessed across GPUs or nodes, and compare training and inference throughput against MoE under matched hardware and parallelism settings?
- Could the authors provide a more detailed explanation of how depth expansion is implemented?
- The MoUE conversion experiments primarily use JetMoE and OLMoE, both released in 2024, whose architectural designs may no longer reflect the state of the art. Could the authors also evaluate MoUE conversion on more recent MoE models, such as Qwen3.5? (I'm delight to increase score if this concern is solved.)
- If computational resources permit, could the authors also evaluate MoUE conversion on larger-scale MoE models? (Not Necessary)
Thank you for the additional experiments, particularly the comparison at matched depth and compute and the expanded coefficient sweep. These address my main concerns, so I will raise my rating; I am waiting for the revised paper, which should clearly reflect these.