- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents LigerMoE, an efficient expert-parallel execution framework for large-scale Mixture-of-Experts (MoE) models. The paper focuses on the communication and computation bottlenecks in distributed MoE execution, where expert parallelism introduces expensive all-to-all communication and many small expert matrix multiplications. The authors propose persistent fused MoE kernels that leverage warp specialization to overlap communication with expert computation. Specifically, LigerMoE integrates communication and computation within persistent CTAs, uses one-sided RDMA through NVSHMEM for remote token access, and introduces symmetric memory management to support CUDA Graph execution. Extensive experiments on Hopper and Blackwell GPUs demonstrate significant throughput improvements over existing MoE implementations, including Transformer Engine, DeepEP, Comet, and FlashMoE.
Critical strengths
- The paper addresses an important bottleneck in large-scale MoE systems.
- The paper is overall well written.
- The experimental evaluation is comprehensive.
Critical weaknesses
- The symmetric memory mechanism simplifies communication management and enables CUDA Graph execution, but it reserves memory according to worst-case token counts on each GPU. The impact of this design under highly imbalanced expert routing or extremely large-scale deployment is not fully evaluated. 2.The performance advantage is workload dependent. Although LigerMoE achieves strong improvements in many communication-bound settings, the paper also shows that performance advantages can decrease or reverse in compute-bound workloads. 3.The evaluation is mainly limited to NVIDIA GPU platforms and specific hardware features. 4.The novelty mainly lies in the engineering optimization of GPU kernels rather than introducing fundamentally new execution principles. 5.The reported performance gains may partially depend on extensive architecture-specific kernel tuning and carefully selected hyper-parameters. It is unclear how much improvement comes from the proposed design itself versus optimization effort for specific hardware and workloads.
Questions for authors
1.The proposed method relies on NVIDIA-specific features such as NVSHMEM and TMA. How do the authors expect the design principles of LigerMoE to transfer to other GPU architectures or AI accelerators? 2.How does LigerMoE behave under severe expert imbalance where token distributions differ significantly across GPUs? 3.Could the authors provide more analysis on workload characteristics that determine whether LigerMoE provides advantages over existing kernels?
3 comments
I think my concerns have been partially addressed, and I will keep my original rating.
Query
We thank reviewer for their swift reply, and kindly ask the reviewer what remaining concerns that were not addressed are grounds for rejection?
Response to Reviewer Comments
Comment: The symmetric memory mechanism simplifies communication management and enables CUDA Graph execution, but it reserves memory according to worst-case token counts on each GPU. The impact of this design under highly imbalanced expert routing or extremely large-scale deployment is not fully evaluated.
Response: Authors respectfully disagree with this notion that it reserves memory for the worst case. Our memory consumption is T x K where T is the local tokens and K is the number of routed tokens which is static under all conditions. Worst case is EP x T x K (all tokens routed to one expert) which is ~EP times larger. There is no need to measure the memory of the LigerMoE kernel since it’s constant and independent of the router skew and it’s handled by the ring buffers as explained in the kernel sections.
Comment: The performance advantage is workload dependent. Although LigerMoE achieves strong improvements in many communication-bound settings, the paper also shows that performance advantages can decrease or reverse in compute-bound workloads.
Response: Authors agree with this notion and this is also the reason why we present a significant number of different workloads. In multi-GPU settings Liger is very competitive but can lose to other cases in single GPU settings. However, it’s not intended to be used in single GPU cases and there are more suitable choices available in literature for that (i.e. SonicMoE).
Comment: The evaluation is mainly limited to NVIDIA GPU platforms and specific hardware features.
Reponse: Most if not all recent MoE kernels are hardware specific. Different hardware platforms such as TPUs require a very different mindset for optimizing performance. We also like to remind that we considered two different generations in this work to showcase that the ideas are not bound to single specific GPU generation.
Comment: The novelty mainly lies in the engineering optimization of GPU kernels rather than introducing fundamentally new execution principles.
Response: Authors agree that work is in the boundary of engineering and science rather than changing the MoE structure itself. However, this is a very relevant problem for training current state-of-the models which are mostly MoEs.
Comment: The reported performance gains may partially depend on extensive architecture-specific kernel tuning and carefully selected hyper-parameters. It is unclear how much improvement comes from the proposed design itself versus optimization effort for specific hardware and workloads.
Response: To shed light on this we compare multiple different kernels and in a wide range of different settings. It is easy to tune the kernel to win in a single setup but not in hundreds of different workloads which we present.
Comment: The proposed method relies on NVIDIA-specific features such as NVSHMEM and TMA. How do the authors expect the design principles of LigerMoE to transfer to other GPU architectures or AI accelerators?
Response: Other GPUs feature similar features (for instance NeuraLink in Trainium) so some aspects of the work are generalizable. We agree that our work is GPU centric and cannot be applied on CPUs for instance. However, most ML training workloads do run on GPUs today.
Comment: How does LigerMoE behave under severe expert imbalance where token distributions differ significantly across GPUs? 3.Could the authors provide more analysis on workload characteristics that determine whether LigerMoE provides advantages over existing kernels?
Response: There are two aspects to this: i) memory; and ii) compute time. Regarding the memory consumption, router skew has no impact on LigerMoE which is advantageous over any kernel doing all-to-all type communication that requires storing the dispatched tokens. ii) LigerMoE follows the same router evaluation framework as others and the expert weights are stored in a specific GPU. In this setting, if all the tokens are routed to a single expert the computations have to be performed in a single GPU. The advantage of LigerMoE in this case comes from pipelining the communication with the computation but the computation is serialized to single GPU as in other works compared in the manuscript.