- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
FluidMoE is a scheduler for MoE training with Ulysses context parallelism and expert parallelism. It jointly schedules attention communication, expert communication, and gradient AllReduce. It splits work in the forward pass. In the backward pass, it delays weight gradient computation and uses this work to hide communication. It also inserts gradient AllReduce when gradients are ready. The paper reports gains over DeepSpeed and Megatron baselines, with ablations and tests across several model shapes and hardware settings.
Critical strengths
-
The paper has a clear motivation. Attention communication, expert communication, and gradient synchronization can compete for the same resources. Scheduling them together is a useful direction. The backward schedule goes beyond overlap within a single MoE operator.
-
The paper includes useful ablations and reports cases where the method helps less or becomes slower. These results help explain both the source of the gains and the cost of the scheduler.
Critical weaknesses
-
The paper does not discuss or compare with MegaMoE. This is a major gap in both the positioning and the evaluation. DeepSeek-V4 describes MegaMoE as an open-source part of DeepGEMM. It divides experts into waves and overlaps dispatch, expert computation, and combine within a fused kernel. This shares a key idea with FluidMoE: starting useful computation before all communication has finished. The paper discusses COMET and MegaScale-MoE, but these do not replace a discussion of MegaMoE. FluidMoE has a broader training schedule, so the methods are not identical. However, the paper does not establish what gains remain once expert communication is already hidden by a strong fused backend. Results against standalone kernels do not resolve this issue. Given the reported public implementation, the authors should provide a comparison where the workloads and interfaces permit one, or explain the concrete barriers. I do not expect a direct comparison between MegaMoE inference speedups and FluidMoE training speedups. I do expect evidence that separates the benefit of better expert overlap from the added benefit of the joint training schedule.
-
The current tests provide limited evidence for the full training setting where this added benefit matters. The main experiments use reduced-depth models. The implementation requires matched context and expert parallel sizes and full gradient AllReduce. It does not test pipeline parallelism or integration with a distributed optimizer. DeepSeek-V4 uses different context communication, gradient synchronization, and pipeline schedules. These differences do not make FluidMoE invalid, but they leave its value on an already optimized training stack unclear. Together with the missing MegaMoE comparison, this limits my confidence in the scope of the claimed gains and leads me to a score of 2.
Questions for authors
-
What are the precise differences between FluidMoE and MegaMoE in scheduling scope and overlap opportunities? Which parts of FluidMoE remain useful when dispatch, expert computation, and combine are already fused and overlapped?
-
Can you provide a matched comparison on a supported workload, or an ablation that measures the extra benefit of FluidMoE's joint schedule on top of a strong fused expert backend? If MegaMoE cannot support the required training path, please identify the missing interfaces and explain how you can test the same concern with a suitable backend.
-
Which parts of the schedule depend on full AllReduce and the absence of pipeline parallelism? Would gradient sharding or an existing pipeline schedule remove work that FluidMoE currently uses to hide communication?
4 comments
Response (2/2)
W2 / Q3: Which parts depend on full AllReduce and the absence of pipeline parallelism; would gradient sharding or a pipeline schedule remove the work FluidMoE uses to hide communication
A3: Gradient sharding does not remove the dW computation that FluidMoE uses to hide communication. A pipeline schedule that also moves dW would compete for it, so the two schedules would need to be coordinated.
Of the four mechanisms, AR placement is the one built on full AllReduce. The forward tournament, dW deferral, and chunking involve only A2A and computation. With gradient sharding or a distributed optimizer, the gradient of every parameter is still computed in the backward pass, so the dW computation that covers communication remains, while gradient synchronization and buffer management change. On the cluster, dW deferral and chunking account for 84% of the time saved (A3 of our response to Reviewer hWQq).
The part that interacts with pipeline parallelism is dW deferral. Pipeline schedules such as Zero Bubble also move dW to fill pipeline bubbles. Any dW placed in a bubble is unavailable to cover an exchange. A pipeline schedule uses dW when a bubble opens between stages, while FluidMoE uses it at the four exchanges inside each microbatch.
We did not measure a distributed optimizer or pipeline parallelism. The implementation has only been run without pipeline parallelism, and supporting the distributed optimizer means reworking AR placement and the gradient buffers, more than we could finish and validate in the discussion period. The closest setting we measured is gradient accumulation, which also runs several microbatches per step: on the 4090 node, FluidMoE is 1.096×, 1.104×, and 1.125× faster than Megatron with a tuned bucket size at 1, 2, and 4 microbatches per step.
W2: Reduced-depth models, and the value of FluidMoE on an already optimized training stack
A: After tuning Megatron's gradient bucket size, FluidMoE remains 1.073× to 1.135× faster on the 4090 node. On the A100 cluster with groups spanning nodes, it is 1.073× faster than the fastest Megatron configuration with Ulysses CP, 1.072× and 1.083× faster at 4 and 8 layers, and 1.097× faster in 8-layer end-to-end training. With Ring CP, a different context-parallel scheme, Megatron is faster than with Ulysses CP in this layout, and FluidMoE is 1.049× faster than that configuration. The tables are in A2 and A4 of our response to Reviewer 5AWc. The requirement cp = ep is a limitation of the current implementation and is listed in Section 6.
These results are in Table 2 and Appendix H.2, H.4, and H.8 of the revision. Thank you again for your time. We would be glad to answer any further questions.
Thank you for the detailed responses and additional experiments. The clarification of MegaMoE’s scope and implementation barriers addresses my concern about the missing comparison. The tuned baselines, gradient accumulation experiments, and deeper end-to-end evaluation also strengthen the evidence for FluidMoE’s practical value.
Some limitations remain: the stock-kernel ablations do not establish the incremental benefit on top of a strong fused expert backend, and integration with distributed optimizers and pipeline schedules remains unverified. The topology experiments also show that the gains depend on the communication setting. I appreciate the authors’ transparency about these limitations and encourage keeping the corresponding scope restrictions explicit in the paper.
Overall, the revisions sufficiently address my main concerns to support acceptance. I am raising my rating from 2 to 3.
Sincere appreciation
Thank you for reading our responses carefully and for raising your rating. Your comments, in particular the pointer to MegaMoE, have improved the paper.
The scope restrictions you mention are stated in the revision: the dependence on the communication layout in Section 4.2, and the settings that remain untested in Section 6 and Appendix B.4. We will keep them explicit in the final version.
We truly appreciate your time and effort.
Response (1/2)
Thank you for recognizing the motivation for joint scheduling and the part of the backward schedule that goes beyond a single MoE operator, and for pointing us to MegaMoE. We should have discussed it in the submission, and we apologize for the omission. Section 5 and Appendix B.4 of the revision now cover it.
W1 / Q1: Differences between FluidMoE and MegaMoE in scheduling scope; which parts of FluidMoE remain useful once dispatch, expert computation, and combine are fused
A1: MegaMoE is a fused kernel for the expert forward pass: dispatch, expert computation, and combine. FluidMoE is a training-time scheduler with a wider scope. Besides the expert forward pass, it schedules the expert backward pass, the attention exchanges in both passes, and gradient AllReduce, none of which a fused expert-forward kernel covers. The two share the idea of starting computation before an exchange has finished.
The per-region measurements in the paper show how much of the exposed communication lies in the expert forward pass. Of the eight A2A regions per block (Appendix B.3), forward dispatch and combine are two. In the Megatron baseline they expose 23.7 ms on the 4090 node, 10% of the baseline's total exposure, and 52.0 ms on the A100 cluster, 20%.
The table shows exposed communication before and after scheduling, in ms. "Before" is Megatron without overlap.
FluidMoE already removes most of the expert-forward exposure by scheduling stock kernels. The exposure it removes outside the expert forward pass is 85% of the total removed on the 4090 node and 76% on the cluster.
W1 / Q2: A matched comparison, or an ablation measuring the extra benefit of the joint schedule on top of a fused expert backend; the missing interfaces if MegaMoE cannot support training
A2: MegaMoE cannot be used in our training path at present, for two reasons that can be checked in the public DeepGEMM code:
We looked into running MegaMoE on our hardware but could not get around either limitation. The closest evidence we can offer is an ablation on our own implementation: we add the schedule in steps and measure how much of the saved time comes from outside the expert side. All rows use the same stock kernels. On the 4090 node with Mixtral at 16K, the first three rows reduce gradients at the tail and differ only in which exchanges are scheduled:
Of the 86.6 ms saved, the expert-side schedule accounts for about one third, and the attention-side schedule and AR placement for about two thirds.
On the A100 cluster at (2,8,8) and 32K, Megatron without overlap takes 629.1 ± 0.3 ms. Adding the forward schedule gives 628.5 ± 1.3 ms, adding backward dW deferral and chunking gives 592.0 ± 1.4 ms, and the full FluidMoE takes 585.5 ± 1.1 ms. Of the 43.6 ms saved, the forward schedule accounts for 0.6 ms and the backward schedule and AR placement for 43.0 ms. In this cumulative ablation the net saving appears almost entirely when the backward schedule and AR placement are added, and these are the parts outside a fused forward kernel.
In a separate comparison on this cluster, scheduling the attention exchanges as well shortens the backward pass by 12 ms, but the forward tournament on the attention side costs about 6 ms and about 6 ms of AR moves to the tail, so the block time is unchanged.
We have not been able to measure the combination itself. The two approaches are complementary in scope: a fused kernel would handle the two expert-forward regions, while FluidMoE would schedule the rest. These results are in Appendix H.5 and H.7 of the revision.