acceptodds
Under review as a conference paper at ICLR 2027

LigerMoE: Efficient Expert Parallelism via Warp-Specialized Communication-Compute Fusion

Abstract

State-of-the-art large language models rely on Mixture-of-Experts (MoE) layers to scale model capacity while maintaining computational efficiency. At scale, MoE models rely on expert parallelism, whose efficiency is limited by many small expert matrix multiplications and expensive all-to-all communication. To address these bottlenecks, we present persistent, fused MoE kernels for both the forward and backward passes that leverage warp specialization to overlap communication with computation. The kernels use remote direct memory access (RDMA) to access data on remote GPUs without bidirectional sender-receiver coordination, while preallocated symmetric buffers support CUDA Graph execution without requiring a predefined per-expert token capacity. On the Qwen3-30B-A3B MoE model, our forward kernel achieves and higher throughput than the strongest available baselines on H200 and B300 GPUs, respectively. For the backward pass, our method improves throughput by and on the same systems. In end-to-end BF16 training on H200 GPUs, our implementation reduces training step time by compared with Megatron's default MoE implementation.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM, MoE

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ifSSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LigerMoE, an efficient expert-parallel execution framework for large-scale Mixture-of-Experts (MoE) models. The paper focuses on the communication and computation bottlenecks in distributed MoE execution, where expert parallelism introduces expensive all-to-all communication and many small expert matrix multiplications. The authors propose persistent fused MoE kernels that leverage warp specialization to overlap communication with expert computation. Specifically, LigerMoE integrates communication and computation within persistent CTAs, uses one-sided RDMA through NVSHMEM for remote token access, and introduces symmetric memory management to support CUDA Graph execution. Extensive experiments on Hopper and Blackwell GPUs demonstrate significant throughput improvements over existing MoE implementations, including Transformer Engine, DeepEP, Comet, and FlashMoE.

Critical strengths
  1. The paper addresses an important bottleneck in large-scale MoE systems.
  2. The paper is overall well written.
  3. The experimental evaluation is comprehensive.
Critical weaknesses
  1. The symmetric memory mechanism simplifies communication management and enables CUDA Graph execution, but it reserves memory according to worst-case token counts on each GPU. The impact of this design under highly imbalanced expert routing or extremely large-scale deployment is not fully evaluated. 2.The performance advantage is workload dependent. Although LigerMoE achieves strong improvements in many communication-bound settings, the paper also shows that performance advantages can decrease or reverse in compute-bound workloads. 3.The evaluation is mainly limited to NVIDIA GPU platforms and specific hardware features. 4.The novelty mainly lies in the engineering optimization of GPU kernels rather than introducing fundamentally new execution principles. 5.The reported performance gains may partially depend on extensive architecture-specific kernel tuning and carefully selected hyper-parameters. It is unclear how much improvement comes from the proposed design itself versus optimization effort for specific hardware and workloads.
Questions for authors

1.The proposed method relies on NVIDIA-specific features such as NVSHMEM and TMA. How do the authors expect the design principles of LigerMoE to transfer to other GPU architectures or AI accelerators? 2.How does LigerMoE behave under severe expert imbalance where token distributions differ significantly across GPUs? 3.Could the authors provide more analysis on workload characteristics that determine whether LigerMoE provides advantages over existing kernels?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Comments

Comment: The symmetric memory mechanism simplifies communication management and enables CUDA Graph execution, but it reserves memory according to worst-case token counts on each GPU. The impact of this design under highly imbalanced expert routing or extremely large-scale deployment is not fully evaluated.

Response: Authors respectfully disagree with this notion that it reserves memory for the worst case. Our memory consumption is T x K where T is the local tokens and K is the number of routed tokens which is static under all conditions. Worst case is EP x T x K (all tokens routed to one expert) which is ~EP times larger. There is no need to measure the memory of the LigerMoE kernel since it’s constant and independent of the router skew and it’s handled by the ring buffers as explained in the kernel sections.

Comment: The performance advantage is workload dependent. Although LigerMoE achieves strong improvements in many communication-bound settings, the paper also shows that performance advantages can decrease or reverse in compute-bound workloads.

Response: Authors agree with this notion and this is also the reason why we present a significant number of different workloads. In multi-GPU settings Liger is very competitive but can lose to other cases in single GPU settings. However, it’s not intended to be used in single GPU cases and there are more suitable choices available in literature for that (i.e. SonicMoE).

Comment: The evaluation is mainly limited to NVIDIA GPU platforms and specific hardware features.

Reponse: Most if not all recent MoE kernels are hardware specific. Different hardware platforms such as TPUs require a very different mindset for optimizing performance. We also like to remind that we considered two different generations in this work to showcase that the ideas are not bound to single specific GPU generation.

Comment: The novelty mainly lies in the engineering optimization of GPU kernels rather than introducing fundamentally new execution principles.

Response: Authors agree that work is in the boundary of engineering and science rather than changing the MoE structure itself. However, this is a very relevant problem for training current state-of-the models which are mostly MoEs.

Comment: The reported performance gains may partially depend on extensive architecture-specific kernel tuning and carefully selected hyper-parameters. It is unclear how much improvement comes from the proposed design itself versus optimization effort for specific hardware and workloads.

Response: To shed light on this we compare multiple different kernels and in a wide range of different settings. It is easy to tune the kernel to win in a single setup but not in hundreds of different workloads which we present.

Comment: The proposed method relies on NVIDIA-specific features such as NVSHMEM and TMA. How do the authors expect the design principles of LigerMoE to transfer to other GPU architectures or AI accelerators?

Response: Other GPUs feature similar features (for instance NeuraLink in Trainium) so some aspects of the work are generalizable. We agree that our work is GPU centric and cannot be applied on CPUs for instance. However, most ML training workloads do run on GPUs today.

Comment: How does LigerMoE behave under severe expert imbalance where token distributions differ significantly across GPUs? 3.Could the authors provide more analysis on workload characteristics that determine whether LigerMoE provides advantages over existing kernels?

Response: There are two aspects to this: i) memory; and ii) compute time. Regarding the memory consumption, router skew has no impact on LigerMoE which is advantageous over any kernel doing all-to-all type communication that requires storing the dispatched tokens. ii) LigerMoE follows the same router evaluation framework as others and the expert weights are stored in a specific GPU. In this setting, if all the tokens are routed to a single expert the computations have to be performed in a single GPU. The advantage of LigerMoE in this case comes from pipelining the communication with the computation but the computation is serialized to single GPU as in other works compared in the manuscript.

I think my concerns have been partially addressed, and I will keep my original rating.

We thank reviewer for their swift reply, and kindly ask the reviewer what remaining concerns that were not addressed are grounds for rejection?

Reviewer_in1dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LigerMoE fuses the forward and backward passes of expert-parallel MoE into persistent kernels. Warp specialization lets every SM handle both communication and computation, and one-sided NVSHMEM pulls remove the need for a per-expert capacity, enabling CUDA Graphs. On Qwen3-30B-A3B, it reports 49%/143% (forward) and 31%/121% (backward) speedups over the strongest baselines on 8×H200/8×B300, and 47% faster end-to-end training than Megatron's default.

Critical strengths

Covers both forward and backward passes, including a fused implementation of the cross-token weight-gradient reduction in the backward pass. The pull-based design needs neither sender-receiver coordination nor per-expert capacity, which simplifies CUDA Graph execution. Experiments span Hopper and Blackwell and include end-to-end training; the authors report the compute-bound cases where their backward kernel is slower than TE, and code is provided.

Critical weaknesses

Central claim untested. No experiment isolates the effect of warp-level fusion. Within a node, the gap to TE is largest on one GPU (no communication) and shrinks with more GPUs (Fig. 3a: 3.3× → 2.1×; Fig. 3b: 5.4× → 1.5×), suggesting much of the gain comes from expert computation itself. Comet's argument against placing communication warps in the same block is not addressed. Timing and end-to-end baseline. The timing scope of each method is not stated. The 47% end-to-end gain is against Megatron's default rather than the strongest baseline; per Table 1, LigerMoE is only ~4% faster than TE. Related work. ParallelKittens [1], which compares intra- vs. inter-SM overlap, is not cited. UniEP [2] (HPDC 2026) already fuses forward and backward MoE training into a megakernel, so the "first work" claim should be revised. As an SM-level design covering both passes, UniEP is the closest existing counterpart to this work and deserves discussion. [1] S. H. Sul, S. Arora, B. F. Spector, and C. Ré. ParallelKittens: Systematic and Practical Simplification of Multi-GPU AI Kernels. In Proceedings of Machine Learning and Systems (MLSys), 2026. [2] S. Zheng, X. Zheng, L.-w. Chang, and J. Zhai. UniEP: Unified Expert-Parallel MegaKernel MoE for LLM Training. In Proceedings of the 35th ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC '26), 2026. https://doi.org/10.1145/3806645.3807818

Questions for authors

Could you add two comparisons within the same codebase: a version that runs dispatch, computation, and combine sequentially, and a version that assigns communication and computation to separate CTAs? This would separately quantify the contributions of overlap and of warp-level specialization. In multi-GPU runs, does the TE baseline use NCCL all-to-all for dispatch and combine? Which kernels are included in the timing of each method? Were the baselines tuned per shape, as LigerMoE is? What routing distribution is used in the kernel experiments? Was CUDA Graph actually used in any experiment? How does LigerMoE's peak memory compare with the baselines?

3 comments
AuthorsOpenReview ↗

Response to Reviewers Comments

Comment: Central claim untested. No experiment isolates the effect of warp-level fusion.

Response: We have compared against multiple other works using CTA specialization to establish the manuscript's claim.

Comment: Comet's argument against placing communication warps in the same block is not addressed.

Response: We compare directly against Comet and out-perform it virtually in every single case on every hardware and GPU count considered. Authors will revise the manuscript to include the following counter argument to comets claims which can be divided into the following sub-arguments. Claim i: Limited communication parallelism

Using every SM to perform communication does not limit parallelism but enables it. Being able to run a few more warps per SM in a CTA specialized setting does not compare to using every SM. Also, libraries such as NCCL GIN or NVHSMEM do not use all NVLink channels if the instructions are issued from a very small number of CTAs and usually at least 16 SMs is required for good throughput and higher number like 32 is preferred.

Claim ii: Communication warps interfere with computation warps

GPUs FLOP output scales linearly with the number of tensor cores hence not using some tensor cores (SM used for computation) does not increase the throughput but reduces it. Utilizing a tensor core has always higher throughput than not using it therefore using the communication CTAs for computation will have a positive effect on the overall throughput. The only situation where this could be true is in a register bound kernel with communication fabric that does not require much GPU resources (InfiniBand, EFA, etc.). However, as previously noted this was not the case for the hundred of test cases we used.

Comment: The 47% end-to-end gain is against Megatron's default rather than the strongest baseline

Response: Other frameworks were not integrated with Megatron and our aim was to showcase what gains can be expected in end-to-end training as supplemental rather than main result. We have performed multiple isolated kernel experiments hence stronger baselines are indeed presented in the manuscript.

Comment: Related work. ParallelKittens [1], which compares intra- vs. inter-SM overlap, is not cited. UniEP [2] (HPDC 2026) already fuses forward and backward MoE training into a megakernel, so the "first work" claim should be revised. As an SM-level design covering both passes, UniEP is the closest existing counterpart to this work and deserves discussion.

Response: We like to thank the reviewer for pointing out these works and agree that they should be cited. ParallelKittens employs CTA specialization that is also used in the other baselines we compare against. It also does not appear to work with InfiniBand/EFA and replies in uniform NVLink connections which are available in intra-host or in NVL36/72 clusters. UniEP is more similar to LigerMoE with an exception that the communication is not pipelined from end-to-end in the same kernel. The advantage of that is the whole GPU compute phase can be overlapped with the communication stream without any breaks or under utilizing the hardware.

Comment: Could you add two comparisons within the same codebase: a version that runs dispatch, computation, and combine sequentially, and a version that assigns communication and computation to separate CTAs? Response: The ring-buffer structure of LigerMoE would make this hard as it is looped over, but the transformer-engine baseline does exactly this and runs the phases serially in an optimal manner. It uses NCCL to perform the all-to-all communication phases as optimally as possible and then uses a highly performant kernel (TE) to perform the routed expert computations.

For the warp specialization we show multiple comparisons to other works that employ CTA specialization instead of the warp specialization. In general, floating point throughput of Nvidia GPUs scale with SM count and warp specialization improves the throughput by the amount of SMs spent on communication purposes in CTA specialization. EFA and InfiniBand require very few warps but NVLink benefits from higher instruction rates and usually needs 16-32 SMs for good link utilization. Hence, the warp specialization on a B200 for instance will improve the throughput by utilizing the 32 SMs otherwise reserved for NVLink which gives ~25% improvement in the tensor core throughput.

AuthorsOpenReview ↗

Response to Reviewers (Continued)

In multi-GPU runs, does the TE baseline use NCCL all-to-all for dispatch and combine?

Response: yes

Which kernels are included in the timing of each method?

Response: All the kernels responsible for the work are included into the kernel times and they all do exactly the same work. Router computation is excluded from all the times of all the kernels.

Were the baselines tuned per shape, as LigerMoE is?

Response: Yes and many other shapes. We used a collection of ~200 different shapes for the tuning. Appendix has detailed results for a wide range of shapes and models.

What routing distribution is used in the kernel experiments?

Response: Kernels use a random uniform distribution while the training experiments use the distribution arising naturally.

Was CUDA Graph actually used in any experiment?

Response: End-to-end training used CUDA graphs but we excluded them from the kernel experiments as some of the methods (DeepEP) do not support them. This biases the results slightly against our work.

How does LigerMoE's peak memory compare with the baselines?

Response: Liger has a flat memory requirement and which does not change with skew. Most of it is used in the symmetric buffers holding the token tiles. Other methods need to reserve memory for the tokens they receive which can cause significant memory consumption when there are large skews. In particular, the GPU memory does not grow with the number of EP world size which is not true for any all-to-all approach.

Reviewer_in1dOpenReview ↗

Thank you for the detailed rebuttal

Thank you for the detailed rebuttal. Your answers on the TE baseline, timing, routing, and CUDA Graphs cleared up several of my questions, and I’m glad you plan to cite ParallelKittens and UniEP and to discuss Comet’s argument. One small correction: ParallelKittens covers both intra-SM and inter-SM overlap, not just CTA specialization. I’d also suggest qualifying the “first work” claim with respect to UniEP.

That said, two concerns remain.

Attribution to warp specialization: I understand that Comet and FlashMoE only provide forward-pass baselines on H200, but this leaves every backward and B300 kernel result compared only against TE and DeepEP, and even on H200 the comparison changes more than the specialization scheme (e.g., the GEMM kernels). Your SonicMoE numbers make me more cautious here: on a single GPU for Qwen3-30B-A3B, SonicMoE reaches 839/1350 TFLOPS (fwd/bwd), while TE reaches 151/49 on B300 in Fig. 4 (the GPU used for SonicMoE isn’t stated). So a good part of the speedup over TE may come from expert computation rather than overlap. Since a same-codebase ablation is hard with the ring-buffer design, simply reporting compute-only, communication-only, and full-kernel times for LigerMoE in the revision would already help.

End-to-end headline: TE is already in Table 1, and LigerMoE is only about 4% faster than it on Qwen3-30B-A3B, so I think the abstract should say so rather than only report the 47% against Megatron’s default. It would also be worth stating whether the Megatron and TE runs used CUDA Graphs as well.

Since the main attribution question is still open, I’ll keep my rating at 2.

Reviewer_Z3hoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose LigerMoE which is a fused MoE kernel for training. They implment both forward and backward passes for their approach combining compute and communication into fused kernels. They demonstrate significant speedups for forward (49% on H100) and (143% on B300).

Critical strengths
  1. The authors propose fused communication and compute kernels for MoE training.
  2. The kernels are useful, however, its unclear how they compare against MoE kernels from Cursor.
Critical weaknesses
  1. There is no comparison to SonicMoE (current SOTA kernel) in the single GPU setting.
  2. The authors don't say how they compute the backward. SonicMoE saves significant amount of memory using the reparameterization trick (router can be applied after down projection and treated as if it was applied before down projection in backward since down projection is a linear op). This saves activation memory requirement.
Questions for authors

see weaknesses

1 comment

Comment: There is no comparison to SonicMoE (current SOTA kernel) in the single GPU setting.

Response: We did not compare against SonicMoE as the work is about expert parallelism and SonicMoE is not an expert parallel kernel. We have presented below results for the SonicMoE that shows the cost of running an expert parallel kernel in a single GPU setting with 8192 tokens. The throughput is higher than LigerMoE in every case as expected and highlights why the expert-parallel kernels should not be used in single GPU case.

ModelE/KFwd msFwd TFLOPSBwd msBwd TFLOPS
Qwen3-30B-A3B128/80.7388391.2221,350
Qwen3-235B-A22B128/82.1001,1784.0471,630
Qwen3.5-35B-A3B256/80.6456401.0911,008
Qwen3.5-122B-A10B256/81.3818962.7991,179
Mixtral-8x7B8/23.8841,4878.5531,800
Mixtral-8x22B8/26.6561,48714.7161,793
Llama-4-Scout-17B-16E16/11.6711,2343.3971,618

Comment:The authors don't say how they compute the backward. SonicMoE saves significant amount of memory using the reparameterization trick (router can be applied after down projection and treated as if it was applied before down projection in backward since down projection is a linear op). This saves activation memory requirement.

Response: Authors have dedicated a full section and appendix on outlining how the backward is computed. Hence, we respectfully disagree with the notion that we haven’t explained this, and like to ask what specific detail reviewer is looking for that is not explained in the manuscript? We did not use the reparametrization trick as that will require sending the router scores to the target GPUs increasing communication load while not reducing the memory in the manuscripts setting. It works in sonic as there is no cross-GPU communication that needs to be worried about.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.