acceptodds
Under review as a conference paper at ICLR 2027

DisagMoE: Computation-Communication overlapped MoE Training via Disaggregated AF-Pipe Parallelism

Abstract

Mixture-of-experts (MoE) architectures enable trillion-parameter LLMs with sparsely activated experts. Expert parallelism (EP) is a widely adopted MoE training strategy, but it suffers from severe all-to-all communication bottlenecks, exacerbated by limited inter-node bandwidth as growing model sizes require distributing experts across GPU nodes. Prior work focuses on overlapping these communications with feed-forward network (FFN) and self-attention computations, which often leaves residual network-bound stalls due to the imbalance in attention and FFN layers' computation–communication ratios. We present DisagMoE, a disaggregated MoE training system that jointly optimizes model placement and scheduling for efficient training. DisagMoE separates attention and FFN layers into disjoint GPU groups, introduces AF-Pipe, a multi-stage pipeline with a unidirectional forward path and many-to-many communication, and employs a network–compute roofline model to guide GPU and network resource allocation between the groups. Implemented on Megatron-LM, DisagMoE achieves up to 1.80x throughput over Megatron-1F1B across multiple MoE models in controlled cross-node EP experiments on 16-node clusters with eight H800 GPUs per node.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, MoE training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 5 reviews, above 27% of the others.
Reviewer_fpj2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

DisagMoE addresses communication bottlenecks in MoE training by separating attention and FFN computation across GPU groups. AF-Pipe overlaps computation and communication across microbatches and fuses expert combine with the transfer to the next attention stage, while a roofline model guides resource allocation. Overall, the contribution appears primarily focused on system integration and engineering optimization, with relatively limited methodological novelty.

Critical strengths

AF-Pipe separates attention and FFN computation across device groups, overlaps work across microbatches, and fuses combine with the transfer to the next attention stage.The Megatron-LM implementation is evaluated on up to 512 H800 GPUs across three model families, with comparisons against five baselines, communication breakdowns, and ablation studies.

Critical weaknesses

HeterMoE already separates attention and expert computation, and can be combined with pipeline parallelism. Therefore, component disaggregation and pipelining across microbatches alone do not constitute contributions unique to this work. The added value of the roofline model beyond subsequent profiling is not clearly demonstrated. The system uses the model to generate an initial allocation and then refines it through profiling, but the paper does not report the quality of the initial choice or how much profiling improves it.

Questions for authors

Since HeterMoE already separates attention and expert FFN computation and overlaps work across microbatches, what is the substantive technical novelty of the first contribution? Is the depth pipelining in the second contribution essentially conventional pipeline parallelism with layers distributed across multiple attention and FFN groups?

Reviewer_9JmjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes separating attention and expert computation onto different GPU groups, scheduling them through AF-Pipe, and choosing resource allocations using a network–compute model

Critical strengths

The paper deals with a real problem: in distributed MoE training, communication often becomes the main bottleneck, so it makes sense to treat attention, expert computation, and network traffic as one balancing problem.

Critical weaknesses

HeterMoE already separates attention from the expert computation and overlaps compute with communication. It even notes that Zebra parallelism can be combined with pipeline parallelism. Splitting attention and experts isn’t new, then, and adding depth pipelining on top isn’t new either. I’d like the paper to spell out what it contributes beyond that.

The dependency description makes sense to me, but it’s a description, not a validation. The appendix itself admits there are no numerical comparisons. I’d want to see, at the very least, forward outputs and gradients checked against a reference implementation. After that, a few optimizer steps with the same initialization, inputs, routing, and RNG states.

The headline claims 30–80% over Megatron-1F1B. Once both systems are tuned in Figure 9, though, the gains for DeepSeek-MoE and GPT-OSS shrink to 4–14%. That’s still a useful improvement, but it’s the fairer comparison, and the overall performance claim should be based more on it than on the headline numbers.

Questions for authors

Does AF-Pipe actually match a reference implementation on forward outputs, gradients, and optimizer updates? This should include runs that use re-send and stochastic operators such as dropout.

If AF-Pipe and Zebra use the same placement, batch size, kernels, and parallel settings, how much of the 1.20–2.55× gain is still there?

With a fixed GPU and NIC budget, how close does the allocator’s choice come to the best feasible configuration?

Reviewer_koqyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a distributed training sytem designed to mitgate communication bottlenecks in large scale MoE models. The main contributions include: a disagregated component placement strategy, a multi-stage piepline scheduling mechanism, and a network-compute roofline model. The paper implements DisagMoE on Megatron-LM and evaluate it using several MoE models, including DeepSeek-MoE, GPT-OSS, and Qwen3.

Critical strengths
  • The paper addresses an important performance bottleneck in large scale MoE training.
  • The system is implemented in Megatron-LM and evaluated using several MoE model configurations, sequence lengths and gpu cluster sizes.
Critical weaknesses
  • The contributions of indivdiual optimisations are not clear. DisagMoE combines several optimizations, including attention-ffn disaggregation, AF-Pipe scheduling, communication hand-off fusion and adaptive resource allocation. The ablation studies do not fully isolate the contributions of the individual techniques to end-to-end performance.

  • There is no sufficient evidence for the effectiveness of generality of the adaptive allocation strategy. Adaptive worker allocation is one of the paper's main contributions. However, the evaluation does not establish its effectiveness over alternative allocation strategies.

Questions for authors
  • could the authors provide an end-to-end ablation that progressively introduce i) attention-ffn disaggregation, ii) AF-Pipe scheduling, iii) communication hand-off fusion and iv) adaptive worker allocation, while keeping the total gpu budget and other parallel configurations fixed?

  • How accurately does the proposed roofline model predict the relative performance of different attention-ffn allocations? Could the authors compare their selected allocations against balanced allocations and empirically tuned alternatives under the same gpu budget?

  • Could the authors provide more detailed information about the parallel configurations, microbatch sizes, gradient accumulation settings, and tuning procedures used for Disagmoe and the baselines? For example, how much of the reported improvement in the scalability experiments remains when the compared systems use matched resource budgets and controlled configurations?

Reviewer_vSGVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DisagMoE, a distributed MoE training system designed to improve the training efficiency by overlapping computation with expert communication. The key idea is to disaggregate attention and FFN computations onto separate GPU groups, enabling independent resource allocation and more flexible computation–communication overlap. The system introduces AF-Pipe, a multi-stage pipeline scheduling mechanism that overlaps attention, FFN, and All2All communication across microbatches. It also employs a network–compute roofline model to guide GPU and network resource allocation between attention and FFN workers. DisagMoE is implemented on Megatron-LM and evaluated on H800 GPU clusters with up to 512 GPUs. Experimental results show up to 1.80× training throughput over Megatron-1F1B and up to 1.17× over DualPipe.

Critical strengths

S1: The communication bottlenecks in large-scale MoE training is an important and practical problem.

S2: DisagMoE is implemented on Megatron-LM and evaluated on clusters containing up to 512 H800 GPUs. The experiments cover multiple MoE architectures and sequence lengths, providing useful evidence of scalability.

S3: The system demonstrates throughput gains over several relevant baselines.

Critical weaknesses

The fundamental idea of disaggregating attention and FFN computations across separate GPU groups has already been explored in several prior training and inference systems like MegaScale-Infer and MegaScale-Infer. Although DisagMoE extends this approach with AF-Pipe scheduling and resource allocation, the distinction is not sufficiently clear.

Questions for authors
  1. Could the authors clarify the fundamental differences between DisagMoE and prior AF-disaggregated MoE systems?

  2. Could the authors provide controlled ablations that progressively enable each optimization (e.g., attention–FFN disaggregation, AF-Pipe scheduling)?

  3. How were the parallel configurations of Megatron-LM, DualPipe, and Zebra tuned?

  4. Could the authors provide comparisons with matched GPU counts, CP/TP/EP settings, and pipeline configurations to better isolate the gains from the proposed scheduling mechanism?

  5. What are the additional memory and network costs introduced by attention–FFN disaggregation, especially during backward propagation?

Reviewer_svjqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes DisagMoE, an MoE training scheme that assigns attention and expert FFN computation to separate GPU groups. It overlaps computation and communication while fusing expert-output combination with pipeline hand-off, and a roofline-guided resource-allocation technique.

Critical strengths

-AF-Pipe aligns attention–FFN separation with routed communication and pipeline hand-off, creating additional opportunities to overlap computation and communication. This provides a technically plausible approach to reducing exposed communication in MoE training.

-Substantial implementation and large-scale evaluation. The Megatron-LM implementation is evaluated with multiple MoE workloads on up to 512 H800 GPUs.

Critical weaknesses

-Performance comparisons do not isolate the contribution. Headline gains rely on fixed cross-node EP; improvements over tuned Megatron shrink to 4–14% for two workloads. Comparisons that also change parallel configurations cannot clearly attribute gains to AF-Pipe.

-Resource allocation is insufficiently validated and reproducible. The roofline-guided allocator lacks prediction-error measurements and fixed-budget optimum comparisons. Missing rank maps, tuning grids, and repeated-trial statistics prevent independent verification of configuration quality and reported gains. Contributions are not validated against recent baselines FlowMoE, BalanceMoE, etc.

-Appendix B.1 acknowledges missing output, gradient, and multi-step update comparisons despite changes to residual handling and backward communication. Throughput gains alone cannot establish that the system preserves the intended training computation.

Questions for authors

-Have you compared outputs, parameter gradients, including router gradients, and multi-step optimizer updates against Megatron under identical inputs, weights, and RNG states?

-What gains remain against equally tuned Megatron, DualPipe, and Zebra with matched GPU budgets, global batches, CP settings, and memory policies?

-How accurately does the roofline model predict measured iteration times, and how close are its selected allocations to the best feasible fixed-budget configurations? Please specify the executed rank/NIC mapping for the 248:264, p=2248:264,\ p=2 configuration and how shared-node contention is modeled.

-After including residual transfers, routing metadata, backward traffic, and re-send, what are the measured inter-node bytes per iteration?

-Some ablation studies is necessary. How much of the observed benefit comes from reduced traffic versus improved overlap or fewer synchronization operations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.