acceptodds
Under review as a conference paper at ICLR 2027

FluidMoE: An Asymmetric Scheduling Framework for Context-Parallel Mixture-of-Experts Training

Abstract

Mixture-of-Experts (MoE) training with Ulysses context parallelism (CP) and expert parallelism (EP) uses AllToAll (A2A) for attention and expert exchanges and AllReduce (AR) for gradient synchronization. Existing overlap methods largely optimize these paths separately. We introduce FluidMoE, an asymmetric scheduler that coordinates these exchanges and computation under a common critical-path communication exposure (CCE) objective. In forward, it decomposes each A2A into rounds of point-to-point exchange and overlaps data transfer with matrix multiplication. In backward, it schedules ready weight-gradient tasks at runtime to overlap A2A, then pipelines communication and activation-gradient chunks where exposure remains. It schedules AR on ready gradient slices within the available time between A2A exchanges, overlapping reduction with computation. Implemented as a scheduling layer in Megatron, FluidMoE reuses compute kernel implementations and preserves communication volume and training semantics. Across three public MoE families at 16K tokens, block-level speedups reach 1.2× against Megatron with gradient-reduce overlap enabled and 1.7× against DeepSpeed-MoE. Measured CCE, comprising A2A exposure and full tail-AR duration, falls by up to 67% relative to non-overlap Megatron. End-to-end loss trajectories remain aligned, and multi-node evaluation on data-center GPUs confirms the performance gains.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
mixture-of-experts, distributed training, computation-communication overlap

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TYEeOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

FluidMoE is a scheduler for MoE training with Ulysses context parallelism and expert parallelism. It jointly schedules attention communication, expert communication, and gradient AllReduce. It splits work in the forward pass. In the backward pass, it delays weight gradient computation and uses this work to hide communication. It also inserts gradient AllReduce when gradients are ready. The paper reports gains over DeepSpeed and Megatron baselines, with ablations and tests across several model shapes and hardware settings.

Critical strengths
  • The paper has a clear motivation. Attention communication, expert communication, and gradient synchronization can compete for the same resources. Scheduling them together is a useful direction. The backward schedule goes beyond overlap within a single MoE operator.

  • The paper includes useful ablations and reports cases where the method helps less or becomes slower. These results help explain both the source of the gains and the cost of the scheduler.

Critical weaknesses
  • The paper does not discuss or compare with MegaMoE. This is a major gap in both the positioning and the evaluation. DeepSeek-V4 describes MegaMoE as an open-source part of DeepGEMM. It divides experts into waves and overlaps dispatch, expert computation, and combine within a fused kernel. This shares a key idea with FluidMoE: starting useful computation before all communication has finished. The paper discusses COMET and MegaScale-MoE, but these do not replace a discussion of MegaMoE. FluidMoE has a broader training schedule, so the methods are not identical. However, the paper does not establish what gains remain once expert communication is already hidden by a strong fused backend. Results against standalone kernels do not resolve this issue. Given the reported public implementation, the authors should provide a comparison where the workloads and interfaces permit one, or explain the concrete barriers. I do not expect a direct comparison between MegaMoE inference speedups and FluidMoE training speedups. I do expect evidence that separates the benefit of better expert overlap from the added benefit of the joint training schedule.

  • The current tests provide limited evidence for the full training setting where this added benefit matters. The main experiments use reduced-depth models. The implementation requires matched context and expert parallel sizes and full gradient AllReduce. It does not test pipeline parallelism or integration with a distributed optimizer. DeepSeek-V4 uses different context communication, gradient synchronization, and pipeline schedules. These differences do not make FluidMoE invalid, but they leave its value on an already optimized training stack unclear. Together with the missing MegaMoE comparison, this limits my confidence in the scope of the claimed gains and leads me to a score of 2.

Questions for authors
  • What are the precise differences between FluidMoE and MegaMoE in scheduling scope and overlap opportunities? Which parts of FluidMoE remain useful when dispatch, expert computation, and combine are already fused and overlapped?

  • Can you provide a matched comparison on a supported workload, or an ablation that measures the extra benefit of FluidMoE's joint schedule on top of a strong fused expert backend? If MegaMoE cannot support the required training path, please identify the missing interfaces and explain how you can test the same concern with a suitable backend.

  • Which parts of the schedule depend on full AllReduce and the absence of pipeline parallelism? Would gradient sharding or an existing pipeline schedule remove work that FluidMoE currently uses to hide communication?

4 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for recognizing the motivation for joint scheduling and the part of the backward schedule that goes beyond a single MoE operator, and for pointing us to MegaMoE. We should have discussed it in the submission, and we apologize for the omission. Section 5 and Appendix B.4 of the revision now cover it.

W1 / Q1: Differences between FluidMoE and MegaMoE in scheduling scope; which parts of FluidMoE remain useful once dispatch, expert computation, and combine are fused

A1: MegaMoE is a fused kernel for the expert forward pass: dispatch, expert computation, and combine. FluidMoE is a training-time scheduler with a wider scope. Besides the expert forward pass, it schedules the expert backward pass, the attention exchanges in both passes, and gradient AllReduce, none of which a fused expert-forward kernel covers. The two share the idea of starting computation before an exchange has finished.

The per-region measurements in the paper show how much of the exposed communication lies in the expert forward pass. Of the eight A2A regions per block (Appendix B.3), forward dispatch and combine are two. In the Megatron baseline they expose 23.7 ms on the 4090 node, 10% of the baseline's total exposure, and 52.0 ms on the A100 cluster, 20%.

The table shows exposed communication before and after scheduling, in ms. "Before" is Megatron without overlap.

Where the exposure sitsRTX 4090, 16KA100 cluster, (2,8,8), 32K
Expert forward (covered by a fused kernel)23.7 → 0.552.0 → 13.5
Expert backward25.3 → 8.053.8 → 4.5
Attention A2A, forward and backward49.6 → 32.1106.7 → 75.0
Tail AR127.8 → 35.049.9 → 8.0

FluidMoE already removes most of the expert-forward exposure by scheduling stock kernels. The exposure it removes outside the expert forward pass is 85% of the total removed on the 4090 node and 76% on the cluster.

W1 / Q2: A matched comparison, or an ablation measuring the extra benefit of the joint schedule on top of a fused expert backend; the missing interfaces if MegaMoE cannot support training

A2: MegaMoE cannot be used in our training path at present, for two reasons that can be checked in the public DeepGEMM code:

  • It supports only the SM100 (Blackwell) architecture. Its two entry points, FP8×FP4 and BF16, report every other architecture as unsupported. A100 is SM80 and RTX 4090 is SM89.
  • It is forward-only. The public interface exposes the fused forward function and has no backward or weight-gradient entry, so it cannot be used in backpropagation.

We looked into running MegaMoE on our hardware but could not get around either limitation. The closest evidence we can offer is an ablation on our own implementation: we add the schedule in steps and measure how much of the saved time comes from outside the expert side. All rows use the same stock kernels. On the 4090 node with Mixtral at 16K, the first three rows reduce gradients at the tail and differ only in which exchanges are scheduled:

ConfigurationBlock timeSaved vs. previous row
Megatron, no overlap460.4 ± 0.8
Expert exchanges scheduled432.2 ± 1.828.2
All four exchanges scheduled422.3 ± 0.69.9
Full FluidMoE, AR placed inside the backward pass373.8 ± 0.548.5

Of the 86.6 ms saved, the expert-side schedule accounts for about one third, and the attention-side schedule and AR placement for about two thirds.

On the A100 cluster at (2,8,8) and 32K, Megatron without overlap takes 629.1 ± 0.3 ms. Adding the forward schedule gives 628.5 ± 1.3 ms, adding backward dW deferral and chunking gives 592.0 ± 1.4 ms, and the full FluidMoE takes 585.5 ± 1.1 ms. Of the 43.6 ms saved, the forward schedule accounts for 0.6 ms and the backward schedule and AR placement for 43.0 ms. In this cumulative ablation the net saving appears almost entirely when the backward schedule and AR placement are added, and these are the parts outside a fused forward kernel.

In a separate comparison on this cluster, scheduling the attention exchanges as well shortens the backward pass by 12 ms, but the forward tournament on the attention side costs about 6 ms and about 6 ms of AR moves to the tail, so the block time is unchanged.

We have not been able to measure the combination itself. The two approaches are complementary in scope: a fused kernel would handle the two expert-forward regions, while FluidMoE would schedule the rest. These results are in Appendix H.5 and H.7 of the revision.

AuthorsOpenReview ↗

Response (2/2)

W2 / Q3: Which parts depend on full AllReduce and the absence of pipeline parallelism; would gradient sharding or a pipeline schedule remove the work FluidMoE uses to hide communication

A3: Gradient sharding does not remove the dW computation that FluidMoE uses to hide communication. A pipeline schedule that also moves dW would compete for it, so the two schedules would need to be coordinated.

Of the four mechanisms, AR placement is the one built on full AllReduce. The forward tournament, dW deferral, and chunking involve only A2A and computation. With gradient sharding or a distributed optimizer, the gradient of every parameter is still computed in the backward pass, so the dW computation that covers communication remains, while gradient synchronization and buffer management change. On the cluster, dW deferral and chunking account for 84% of the time saved (A3 of our response to Reviewer hWQq).

The part that interacts with pipeline parallelism is dW deferral. Pipeline schedules such as Zero Bubble also move dW to fill pipeline bubbles. Any dW placed in a bubble is unavailable to cover an exchange. A pipeline schedule uses dW when a bubble opens between stages, while FluidMoE uses it at the four exchanges inside each microbatch.

We did not measure a distributed optimizer or pipeline parallelism. The implementation has only been run without pipeline parallelism, and supporting the distributed optimizer means reworking AR placement and the gradient buffers, more than we could finish and validate in the discussion period. The closest setting we measured is gradient accumulation, which also runs several microbatches per step: on the 4090 node, FluidMoE is 1.096×, 1.104×, and 1.125× faster than Megatron with a tuned bucket size at 1, 2, and 4 microbatches per step.

W2: Reduced-depth models, and the value of FluidMoE on an already optimized training stack

A: After tuning Megatron's gradient bucket size, FluidMoE remains 1.073× to 1.135× faster on the 4090 node. On the A100 cluster with groups spanning nodes, it is 1.073× faster than the fastest Megatron configuration with Ulysses CP, 1.072× and 1.083× faster at 4 and 8 layers, and 1.097× faster in 8-layer end-to-end training. With Ring CP, a different context-parallel scheme, Megatron is faster than with Ulysses CP in this layout, and FluidMoE is 1.049× faster than that configuration. The tables are in A2 and A4 of our response to Reviewer 5AWc. The requirement cp = ep is a limitation of the current implementation and is listed in Section 6.

These results are in Table 2 and Appendix H.2, H.4, and H.8 of the revision. Thank you again for your time. We would be glad to answer any further questions.

Thank you for the detailed responses and additional experiments. The clarification of MegaMoE’s scope and implementation barriers addresses my concern about the missing comparison. The tuned baselines, gradient accumulation experiments, and deeper end-to-end evaluation also strengthen the evidence for FluidMoE’s practical value.

Some limitations remain: the stock-kernel ablations do not establish the incremental benefit on top of a strong fused expert backend, and integration with distributed optimizers and pipeline schedules remains unverified. The topology experiments also show that the gains depend on the communication setting. I appreciate the authors’ transparency about these limitations and encourage keeping the corresponding scope restrictions explicit in the paper.

Overall, the revisions sufficiently address my main concerns to support acceptance. I am raising my rating from 2 to 3.

AuthorsOpenReview ↗

Sincere appreciation

Thank you for reading our responses carefully and for raising your rating. Your comments, in particular the pointer to MegaMoE, have improved the paper.

The scope restrictions you mention are stated in the revision: the dependence on the communication layout in Section 4.2, and the settings that remain untested in Section 6 and Appendix B.4. We will keep them explicit in the final version.

We truly appreciate your time and effort.

Reviewer_5AWcOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

FluidMoE is a scheduling framework for MoE training with Ulysses context parallelism and expert parallelism. It jointly schedules attention-side and expert-side AllToAll exchanges together with gradient AllReduce. The forward pass pipelines point-to-point exchanges with matrix multiplication slices. The backward pass defers weight-gradient computation to cover communication, selectively chunks remaining exchanges, and places gradient reductions between exchanges. Experiments demonstrate speedups over Megatron and DeepSpeed-MoE, including measurements on a multi-node A100 cluster.

Critical strengths

The motivation for joint scheduling is supported by direct evidence. Table 3 shows that introducing inline AllReduce increases backward AllToAll exposure but reduces total block time. This illustrates why optimizing individual communication paths independently can be suboptimal.

The evaluation includes meaningful controlled comparisons. FluidMoE reuses Megatron’s compute kernel implementations and compares against gradient-reduction overlap. At 16K, block-level speedups are 1.13–1.20×, while the two-layer end-to-end training experiment achieves approximately 1.10× speedup over the overlap-enabled baseline. The paper also includes measured multi-node results.

The implementation and ablations are reasonably detailed. The paper examines individual scheduling components, execution overhead, memory usage, numerical differences, and training loss agreement. It also explicitly reports configurations where scheduling overhead eliminates the benefit.

Critical weaknesses

1. The benefit under key large-scale training configurations remains unverified.

All experiments use a single microbatch per optimizer step and full AllReduce. Gradient accumulation, cross-microbatch expert overlap, pipeline parallelism, and distributed optimizers are not evaluated. These are substantive gaps because they change the available scheduling opportunities: accumulation reduces the relative contribution of AllReduce placement, while other overlap mechanisms compete for the same computation and communication windows. Appendix B.4 discusses these interactions, but does not experimentally establish the incremental benefit of FluidMoE when they are enabled.

2. The model-level evidence primarily comes from reduced-depth workloads.

The main benchmarks and multi-node experiments use two-layer stacks retaining the model families’ per-layer dimensions. End-to-end training also uses a two-layer Mixtral-shaped model with a reduced vocabulary. Although the appendix includes depth sweeps, the 16K sweep reaches only four layers, while deeper evaluations primarily use shorter sequences. Full-depth execution can change the deferred-work queue, activation residency, memory pressure, and scheduling opportunities. The current evidence establishes useful layer-level improvements more clearly than full-model training benefits.

3. Fixed-capacity routing limits the evaluation of scheduling under dynamic expert workloads.

The primary experiments use capacity factor 1.0, dropping overflow tokens and padding remaining slots to preserve fixed communication and compute shapes. Chunk counts, deferred-task limits, and AllReduce budgets are selected offline. Under dropless routing or substantial load imbalance, exchange sizes and expert computation times can vary, potentially invalidating these settings. Runtime operand-readiness checks do not by themselves resolve this configuration problem. The paper does not establish effectiveness under these conditions or with unequal context- and expert-parallel degrees.

4. The incremental advantage over sufficiently tuned existing overlap configurations remains unclear.

The Megatron overlap baseline uses its default gradient bucket size, whereas FluidMoE profiles chunk counts and communication budgets. A comparison after comparable baseline tuning would help isolate the benefit of the scheduling design. The static weight-gradient placement control is useful, but represents a policy class rather than a complete implementation of the related system. Comparisons with Ring context parallelism also cover limited settings. The results establish improvements over the selected configurations, but leave uncertainty about the advantage over stronger workload-specific alternatives.

Questions for authors
  1. How much incremental benefit remains with gradient accumulation and Megatron’s cross-microbatch expert overlap enabled? How are deferred weight-gradient tasks and communication resources coordinated between the two scheduling levels?

  2. Can the authors report end-to-end results for a deeper model and a distributed-optimizer configuration, including the effect of weight-gradient deferral on activation residency and peak memory?

  3. How effective are the fixed chunk counts and AllReduce budgets under dropless routing or imbalanced expert loads? What triggers reprofiling, and is there a runtime fallback when the profile becomes unsuitable?

  4. How much improvement remains after comparable tuning of Megatron’s gradient buckets and overlap settings? How does FluidMoE compare with Ring or hybrid context parallelism in the multi-node setting?

2 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for recognizing the direct evidence for joint scheduling, the controlled comparisons, and the implementation details. We added experiments and address your four points below.

W1 / Q1: Benefit under gradient accumulation and cross-microbatch expert overlap; coordination between the two scheduling levels

A1: We implemented gradient accumulation and measured it: with up to 4 microbatches per step, the speedup does not drop.

As in Megatron, gradients are accumulated in the earlier microbatches and reduced only during the last one. With two accumulated microbatches, the estimated relative L2 difference from the reference without backward scheduling is at most 3.4e-3, below the paper's 0.02 threshold. Block times per step for Mixtral at 16K on the 4090 node:

Microbatches per stepMegatron, overlap, tuned bucketFluidMoESpeedup
1409.4 ± 0.3373.6 ± 0.51.096×
2751.9 ± 13.5681.3 ± 2.91.104×
41434.0 ± 36.31274.7 ± 0.81.125×

At (2,8,8) and 32K on the A100 cluster we add Megatron without overlap, the faster baseline there:

Microbatches per stepMegatron, no overlapMegatron, overlap, tuned bucketFluidMoESpeedup vs. fastest Megatron
1630.4 ± 3.7643.3 ± 4.2591.8 ± 11.51.065×
21210.6 ± 0.41226.4 ± 6.21129.5 ± 1.01.072×
42378.2 ± 1.32392.6 ± 3.32217.6 ± 1.51.072×

You are right that accumulation lowers the relative contribution of AR placement. The speedup holds because the four exchanges, and their schedule, recur in every microbatch.

We could not measure cross-microbatch expert overlap, because it is not yet integrated with FluidMoE. That mechanism pairs the forward pass of one microbatch with the backward pass of another, while FluidMoE schedules inside one microbatch. Both use expert dW to cover communication, so a combined system needs one policy for where each expert dW runs and for the order of exchanges on the communication stream (Appendix B.4). Cross-microbatch overlap covers only the expert exchanges, so the attention-side schedule and AR placement lie outside it.

See Appendix H.4 of the revision.

W2 / Q2: End-to-end results for a deeper model, the distributed optimizer, and the effect of dW deferral on activation residency and peak memory

A2: We extended the model to 8 layers on the A100 cluster at (2,8,8), 32K, microbatch size 4. The speedup does not drop with depth: at 8 layers it is 1.083× in the block benchmark and 1.097× in end-to-end training. At 32K, 8 layers is close to the 80 GB memory limit (peak 75,327 MiB with Megatron). Block benchmark, five interleaved rounds:

LayersMegatron, no overlapMegatron, overlap, tuned bucketFluidMoESpeedup vs. fastest Megatron
2628.6 ± 0.4642.4 ± 8.6585.6 ± 1.11.073×
41258.4 ± 1.31275.4 ± 10.51174.1 ± 0.81.072×
82519.9 ± 1.62522.0 ± 6.22326.6 ± 3.31.083×

End-to-end training runs on SlimPajama and includes the embedding, output projection, loss, and optimizer. It keeps the paper's 16,384-token vocabulary, which does not change the blocks FluidMoE schedules. Each system trains for 150 steps in three interleaved rounds, and we time steps 101 to 150. Memory is the peak in MiB, Megatron without overlap / FluidMoE.

LayersMegatron, no overlapMegatron, overlapFluidMoESpeedup vs. fastest MegatronPeak memory
2778.1 ± 20.8842.6 ± 10.6712.5 ± 2.01.092×28,171 / 27,577
82801.0 ± 2.32903.9 ± 20.82553.7 ± 13.41.097×75,327 / 61,873

On memory: a deferred dW keeps the activations and upstream gradients it needs until it runs. The queue carries tasks across blocks but empties within each microbatch, and all depths reuse the two-layer profile. In the training runs above, FluidMoE's peak memory does not exceed Megatron's. In separate 16K runs at (2,8,8) with 8 to 16 layers, FluidMoE uses 1,000 to 1,434 MiB more than Megatron with overlap, at most 3.1% of the baseline peak, and this share does not grow with depth.

We did not evaluate the distributed optimizer or pipeline parallelism. The current implementation keeps full gradient buffers and is compatible with ZeRO-1, which shards only optimizer state. The distributed optimizer replaces AR with reduce-scatter followed by a parameter all-gather (Appendix B.4). Supporting it means reworking AR placement and the gradient buffers, which we could not finish and validate in time. The forward tournament, dW deferral, and chunking involve only A2A and computation, and the latter two account for 84% of the time saved on the cluster. For pipeline parallelism, please see A3 of our response to Reviewer TYEe.

See Appendix H.8 and H.2 of the revision.

AuthorsOpenReview ↗

Response (2/2)

W3 / Q3: Fixed chunk counts and AR budgets under dropless routing or imbalanced loads; what triggers reprofiling; runtime fallback

A3: Under the fixed-capacity routing used in the paper, the profiled chunk counts and budgets can be reused whatever the load imbalance. With capacity factor 1.0, overflow tokens are dropped and empty slots are padded, so the size of every exchange and the shape of every expert matmul are fixed, and only the token assignment changes (Appendix D.7). Under dropless routing the exchange sizes change every step. This is not supported yet, and Section 6 lists it as a limitation.

We also added a sensitivity study of the offline-selected parameters. On the 4090 node with Mixtral at 16K, we change one profiled parameter at a time and keep the others fixed. Megatron with a tuned bucket size takes about 409 ms in the same sessions.

Parameter changedSetting and block time (ms)
AR byte budgets, scaled together×0.5: 393.5, ×1: 373.7, ×2: 372.3
Cap on deferred dW tasks per exchange1: 379.0, 2: 373.7, 4: 379.9
Chunk counts in the four backward regionsall 1: 396.3, profiled 8/1/8/1: 373.7, all 4: 375.8, all 8: 382.5

Bold marks the profiled value. In each row it is either the fastest setting or within 0.4% of the fastest. The slowest setting, no chunking at 396.3 ms, is still 3% faster than Megatron.

Reprofiling is needed when the sequence length, CP/EP degrees, microbatch size, compute kernels, hardware, routing policy, or capacity setting changes. It is not needed when the dataset, optimizer hyperparameters, or loss scaling changes (Appendix D.5). One profile takes a few seconds and runs before training starts.

On runtime fallback: the profile stays fixed during a run, and the scheduler does not switch schedules or reprofile automatically. An unsuitable profile reduces the overlap but does not affect correctness, because a dW task runs only once its operands are ready and the task cap only limits how many run at each exchange. No chunking is one of the profiler's candidates, so a region that does not benefit from chunking stays unchunked. Unequal cp and ep are not supported at present either, as listed in Section 6.

The sensitivity results are in Appendix H.6 of the revision.

W4 / Q4: Gain after tuning Megatron's gradient buckets and overlap settings; Ring and hybrid CP in the multi-node setting

A4: After tuning Megatron's gradient bucket size, FluidMoE remains 1.073× to 1.135× faster on the 4090 node. The submission compared against Megatron's default bucket size while FluidMoE's parameters were profiled, and we agree that the baseline should be tuned as well.

The candidates range from one quarter to four times the 40M-parameter default. Each is timed for two interleaved rounds, and the fastest is then run against FluidMoE for five interleaved rounds.

Model, RTX 4090, 16KMegatron, overlap, default bucketMegatron, overlap, tuned bucketFluidMoESpeedup vs. tuned / vs. default
Mixtral431.1 ± 0.7412.7 ± 3.1 (80M)376.2 ± 6.91.097× / 1.146×
Phi-3.5-MoE334.4 ± 1.7323.4 ± 3.9 (80M)301.5 ± 2.21.073× / 1.109×
MiniCPM-MoE179.2 ± 0.7169.0 ± 0.7 (20M)148.9 ± 0.51.135× / 1.204×

Tuning makes Megatron 3% to 6% faster. Megatron without overlap is slower than with overlap on this node (Table 2). At (2,8,8) on the A100 cluster we measured Megatron without overlap, with the default bucket, and with the tuned bucket, and FluidMoE is 1.073× faster than the fastest of the three (table in A1 of our response to Reviewer hWQq).

We also measured Ring CP in the multi-node setting. At (2,8,8) and 32K, Megatron with overlap takes 666.2 ± 6.8 ms with Ulysses CP and 614.0 ± 4.3 ms with Ring CP. FluidMoE takes 585.2 ± 0.3 ms, 1.049× faster than Ring (Appendix H.2). We did not measure hybrid CP: our Megatron baseline does not support it yet, and we could not add it in time.

On the static-placement control, we agree with your reading. It represents Lancet's policy class, which fixes dW positions at compile time and leaves backward operators unpartitioned, and Appendix E.1 describes it in these terms. Lancet's public code is built on the RAF compiler and does not support context parallelism, so a direct port would compare two software stacks. On the same workload FluidMoE is 1.149× faster than this control.

The tuned baseline is now in Table 2, with details in Appendix H.1. Thank you again for the careful review.

Reviewer_hWQqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FluidMoE, a scheduling framework for overlapping communication and computation in MoE training with Ulysses context parallelism (CP) and expert parallelism (EP).

FluidMoE combines several techniques: The forward scheduling decomposes AllToAll operations into point-to-point communication rounds and pipelines them with GEMM slices; The backward scheduling defers weight-gradient computation (dW) to overlap with AllToAll communication, and selectively chunks AllToAll and activation-gradient computation (dX) to hide remaining communication latency; The gradient synchronization schedules AllReduce operations in the gaps between AllToAll exchanges, using profiled communication budgets to reduce the gradient synchronization tail.

The authors also introduce critical-path communication exposure (CCE) to analyze the remaining communication overhead and provide theoretical bounds for the proposed scheduling strategies.

The implementation builds on Megatron without modifying the underlying compute kernels. Experiments on three MoE model families show 1.13–1.20× block-level speedups over Megatron with gradient-reduce overlap enabled at 16K sequence length. The paper also reports end-to-end training results and multi-node experiments on A100 GPUs.

Critical strengths

The overall approach is practical and well motivated. Rather than optimizing attention communication, expert communication, and gradient synchronization separately, FluidMoE considers their interactions and schedules them together. The paper also provides a detailed analysis of communication exposure, along with theoretical bounds and ablation studies that help explain the contribution of each scheduling technique. The end-to-end training experiments and multi-node evaluation also help demonstrate that the improvements are not limited to isolated communication microbenchmarks.

Critical weaknesses

My main concern is how well the reported performance gains generalize to practical large-scale MoE training setups. The main experiments use RTX 4090 GPUs with relative higher communication overhead, which differs substantially from the high-bandwidth GPU interconnects commonly used in large-scale training systems.

While the authors also evaluate FluidMoE on a multi-node A100 cluster, the configuration uses four nodes with four GPUs each and (dp, cp, ep) = (2, 8, 8), which necessarily places CP and EP communication across nodes. I wonder why CP=EP=8 was chosen instead of a configuration such as CP=EP=4, which could keep expert communication within the high-bandwidth NVLink domain. It would be helpful to understand whether the larger parallelism degree is necessary for the evaluated workloads, or whether it provides sufficient computational or memory benefits to justify the additional cross-node communication.

A related concern is the decision to serialize AllToAll and gradient AllReduce on a single communication stream. In configurations where expert communication is primarily intra-node while gradient synchronization spans nodes, these operations may use partially different network resources and could potentially benefit from concurrent execution. Although some GPU resource contention may remain, it is unclear whether serialization would still be preferable in such settings.

Overall, I would like to better understand how sensitive FluidMoE's performance gains are to the parallelism configuration and communication topology, especially given that the improvements on the A100 cluster are smaller than those reported on the RTX 4090 setup.

Therefore, I am very much on the borderline between accept and reject of this paper.

Questions for authors
  • What motivated the choice of (dp, cp, ep) = (2, 8, 8) on four nodes with four A100 GPUs each? Would (dp, cp, ep) = (4, 4, 4), which could keep CP/EP communication within nodes, be feasible for the evaluated workloads? If so, how would the performance of FluidMoE compare under this configuration?

  • Have the authors experimented with separate communication streams for AllToAll and gradient AllReduce? A comparison would help clarify the tradeoff between communication concurrency and contention, particularly when these operations use different process groups or communication paths.

  • Could the authors provide a more detailed breakdown of communication exposure in the multi-node experiments? Table A.16 reports reductions of 158.4 ms and 239.4 ms in exposed communication at 32K and 64K, respectively, but does not provide the original exposure or the residual exposure after optimization. A breakdown by attention AllToAll, expert AllToAll, and gradient AllReduce would help explain the smaller speedups observed in these experiments and identify which scheduling techniques remain most effective.

5 comments
AuthorsOpenReview ↗

Response (1/3)

Thank you for recognizing the motivation for joint scheduling, the exposure analysis, and the ablations, and for stating your concerns so concretely. We ran the experiments you suggested for all three questions and respond to each below.

W1 / Q1: Generalization beyond RTX 4090; why (2,8,8); is (4,4,4) feasible, and how does FluidMoE perform there?

A1: Yes, (4,4,4) is feasible for the two-layer model of the paper, and we have now measured it. When the CP/EP groups span nodes, at (2,8,8), FluidMoE is 1.073× faster than the fastest Megatron configuration. When they stay inside one NVLink node, at (4,4,4), Megatron with a tuned bucket size is faster. The submission should have included this layout and explained the choice of (2,8,8). We do both below. Block times for Mixtral with two layers at 32K and microbatch size 4, in ms:

System(2,8,8)(4,4,4)
Megatron, no overlap628.6 ± 0.41027.3 ± 0.6
Megatron, overlap, default bucket663.0 ± 3.8963.9 ± 1.1
Megatron, overlap, tuned bucket642.4 ± 8.6919.4 ± 4.6
FluidMoE585.6 ± 1.11020.7 ± 1.8

At (2,8,8), both AR and A2A cross nodes. Enabling gradient-reduce overlap makes them run concurrently, and Megatron becomes slower than without overlap. FluidMoE is 1.073× faster than the no-overlap row and 1.097× to 1.132× faster than the two overlap rows, in line with the 1.096× that the submission reported against Megatron with overlap.

At (4,4,4), A2A stays on NVLink inside a node. Megatron's exposed A2A time is only 7% of its block time, against 34% at (2,8,8), which leaves little for FluidMoE to hide. We now state this as a scope condition in Section 4.2.

On the choice of (2,8,8): we used it to measure the case in which the CP/EP groups span nodes, which deeper models require once they no longer fit at (4,4,4). The table below gives peak GPU memory in end-to-end training at 16K, including optimizer state, in MiB. Each cell is Megatron with overlap / FluidMoE.

Layers(2,8,8)(4,4,4)
831,857 / 32,85755,209 / 55,337
1244,223 / 45,60177,099 / 76,903
1656,551 / 57,985OOM / OOM

At (4,4,4) each GPU holds twice the tokens and twice the expert parameters, so the model exceeds 80 GB at 16 layers, and the full Mixtral model has 32. A deeper model therefore needs larger groups, and with four GPUs per node a group of 8 already spans two nodes. Large-scale training also runs expert parallelism across nodes: DeepSeek-V3 uses 64-way expert parallelism across 8 nodes and reports a computation-to-communication ratio of about 1:1 (Section 3.2 of its technical report).

On whether the RTX 4090 results carry over to high-bandwidth interconnects: the gain depends on how much communication, A2A plus AR, the baseline exposes on the critical path. On the 4090 node this is 49% to 59% of the block time, and FluidMoE is 1.073× to 1.135× faster than Megatron with a tuned bucket size. At (2,8,8) on the A100 cluster it is 42% (34% A2A plus 8% AR), and FluidMoE is 1.073× faster at 32K and 1.064× faster at 64K. The gains therefore carry over to the A100 cluster when the CP/EP groups span nodes.

These results are in Table 6 and Appendix H.2 of the revision.

AuthorsOpenReview ↗

Response (2/3)

W2 / Q2: Separate communication streams for AllToAll and gradient AllReduce

A2: We had not tried this before submission, and following your suggestion we implemented a separate stream for AR. The results agree with your intuition: when A2A stays inside a node and AR crosses nodes, separate streams are 2.7% faster, and when both cross nodes they bring no gain. Thank you for the suggestion.

Only the stream changes. AR keeps its submission points, slices, and byte budgets, and the gradients match the reference implementation. Block times in ms:

Platform and layoutA2A pathAR pathShared streamSeparate AR stream
A100, (4,4,4), 32KNVLink, intra-nodeRDMA, inter-node1021.2 ± 1.0993.8 ± 1.9
A100, (2,8,8), 32KRDMA, inter-nodeRDMA, inter-node585.7 ± 0.2589.7 ± 6.1
RTX 4090, (2,4,4), 16KPCIePCIe373.3 ± 0.3368.6 ± 1.0

On a shared stream, an A2A waits behind any AR submitted before it. On separate streams this queueing disappears, but the two collectives transfer at the same time and the A2A itself slows down. On the 4090 node, the time the backward pass spends waiting on communication, queueing behind AR included, falls from 45.6 to 37.8 ms, while the part spent waiting on the A2A transfers themselves rises from 27.3 to 35.0 ms. Overall, block time decreases by 4.7 ms. At (2,8,8) the two collectives compete for the same inter-node link: the exposed A2A time, forward and backward together, rises from 92.1 to 124.4 ms, which cancels the benefit of not queueing.

With a separate stream the byte budgets still matter, because they set how much AR runs concurrently with A2A. At (4,4,4), reducing gradients in buckets without budgets takes 1024.2 ± 2.3 ms on the separate stream, against 993.8 ± 1.9 ms with the budgets. On the 4090 node the two numbers are 377.0 ± 0.4 and 368.6 ± 1.0 ms. Doubling the budgets helps further there: in one session, the shared stream takes 373.7 ± 0.5 ms, the separate stream 368.9 ± 1.3 ms, and the separate stream with doubled budgets 355.6 ± 1.1 ms.

Following your suggestion, FluidMoE's block time on the 4090 node falls from 373.7 to 355.6 ms. Section 3.4 of the revision now describes the stream as a scheduler option, and the results are in Appendix H.3.

AuthorsOpenReview ↗

Response (3/3)

W3 / Q3: Exposure breakdown in the multi-node experiments, why the A100 speedup is smaller, and which techniques remain most effective

A3: At (2,8,8), FluidMoE removes 62% of the exposed communication at 32K and 54% at 64K, and backward dW deferral and chunking are the most effective techniques. We apologize that Table A.16 gave only the reductions and not the values before and after. The two reductions correspond to exposed A2A time falling from 257.8 to 99.4 ms at 32K and from 448.8 to 209.4 ms at 64K. Those traces did not separate the attention side from the expert side, so we re-measured with a per-region breakdown. As noted in the general response, the cluster nodes changed after submission, so the absolute values below differ from these. The baseline is Megatron without overlap, and values sum the forward and backward passes, in ms.

LayoutSystemAttention A2AExpert A2ATail ARTotal
(2,8,8), 32KMegatron106.7105.749.9262.4
FluidMoE75.018.08.0100.9
(2,8,8), 64KMegatron217.6217.950.1485.7
FluidMoE159.851.811.9223.5
(4,4,4), 32KMegatron44.031.6140.4216.0
FluidMoE25.68.412.446.4

At (2,8,8), expert A2A and tail AR each fall by about 80% and attention A2A by about 30%, the same pattern as on the 4090 node (Appendix B.3).

The speedup on A100 is smaller for two reasons. First, exposure is a smaller share of the baseline: 42% of the block time at (2,8,8) and 32K, against 49% to 59% on the 4090 node. Second, only part of the reduction in exposure turns into a shorter block time: at (2,8,8) and 32K, exposure falls by about 160 ms and block time by about 43 ms. Concurrent computation and communication interfere with each other, and scheduling adds its own overhead. Appendix B.2 breaks down the same gap on the 4090 node.

To see which techniques remain effective, we repeated the cumulative ablation of Table 3 on the cluster. Block times at (2,8,8), 32K:

ConfigurationBlock timeSaved vs. previous row
Megatron, no overlap629.1 ± 0.3
+ forward tournament628.5 ± 1.30.6
+ backward dW deferral604.6 ± 0.323.9
+ backward chunking592.0 ± 1.412.6
+ inline AR (full FluidMoE)585.5 ± 1.16.5

Backward dW deferral and chunking together save 36.5 ms, 84% of the 43.6 ms saved in total.

The cluster also leaves room for tuning. The Table 3 control that disables cross-region dW takes 558.2 ± 1.2 ms there, 1.127× faster than the fastest Megatron configuration. The traces show why: dW deferred from an earlier region runs concurrently with a cross-node A2A, which lengthens that A2A from 20.0 to 34.0 ms and leaves its exposed time unchanged. Cross-region dW is a scheduler switch, and on this cluster it is better turned off. The 1.073× in A1 was measured with it on.

At (4,4,4), where the exchanges stay on NVLink, inline AR is the technique that remains effective, and it saves 79.0 ms.

The breakdown is in Appendix H.2 and the full ablation for both layouts in Appendix H.7. Thank you again for these suggestions. We are happy to clarify anything further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers and the AC for their time and for the detailed, constructive feedback. We are glad that the reviewers find the joint scheduling of attention exchanges, expert exchanges, and gradient synchronization well motivated, and that they value the exposure analysis and the ablations.

The reviews ask for more evidence on multi-node layouts, tuned baselines, and settings closer to full-scale training. We apologize that the submission did not cover these, and we used the discussion period to run the additional experiments. The revised PDF is uploaded, with changes marked in blue. New material is collected in Appendix H, so the numbering of the existing appendices is unchanged. The main additions are:

  1. A comparison of (dp, cp, ep) = (2,8,8) and (4,4,4) on the A100 cluster (Table 6, Appendix H.2). At (2,8,8), where the CP/EP groups span nodes, FluidMoE is 1.073× faster than the fastest Megatron configuration with the same Ulysses CP and 1.097× to 1.132× faster than Megatron with gradient-reduce overlap.
  2. Megatron with a tuned gradient bucket size (Table 2, Appendix H.1). FluidMoE remains 1.073× to 1.135× faster on the RTX 4090 node.
  3. Gradient accumulation (Appendix H.4). The speedup does not drop with up to 4 microbatches per step.
  4. Deeper models (Appendix H.8). At 8 layers on the cluster, the speedup is 1.083× in the block benchmark and 1.097× in end-to-end training.
  5. An exposure breakdown and a component ablation on the cluster (Appendix H.2 and H.7).
  6. A separate communication stream for AllReduce as a scheduler option (Section 3.4, Appendix H.3).
  7. Expert-side versus joint scheduling, and sensitivity to the profiled parameters (Appendix H.5 and H.6).
  8. A discussion of MegaMoE (Section 5, Appendix B.4).

Block-time comparisons follow the paper's protocol: 10 warm-up and 40 timed iterations, five interleaved rounds per system, mean ± 95% confidence interval, in ms. The A100 cluster nodes available to us changed after submission, and all systems run faster on the current nodes, so absolute times differ from Table A.16. Every comparison is between systems measured in the same interleaved session on the same nodes. We respond to each reviewer in the comments below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.