- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
DisagMoE addresses communication bottlenecks in MoE training by separating attention and FFN computation across GPU groups. AF-Pipe overlaps computation and communication across microbatches and fuses expert combine with the transfer to the next attention stage, while a roofline model guides resource allocation. Overall, the contribution appears primarily focused on system integration and engineering optimization, with relatively limited methodological novelty.
Critical strengths
AF-Pipe separates attention and FFN computation across device groups, overlaps work across microbatches, and fuses combine with the transfer to the next attention stage.The Megatron-LM implementation is evaluated on up to 512 H800 GPUs across three model families, with comparisons against five baselines, communication breakdowns, and ablation studies.
Critical weaknesses
HeterMoE already separates attention and expert computation, and can be combined with pipeline parallelism. Therefore, component disaggregation and pipelining across microbatches alone do not constitute contributions unique to this work. The added value of the roofline model beyond subsequent profiling is not clearly demonstrated. The system uses the model to generate an initial allocation and then refines it through profiling, but the paper does not report the quality of the initial choice or how much profiling improves it.
Questions for authors
Since HeterMoE already separates attention and expert FFN computation and overlaps work across microbatches, what is the substantive technical novelty of the first contribution? Is the depth pipelining in the second contribution essentially conventional pipeline parallelism with layers distributed across multiple attention and FFN groups?