- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents HiPipe, an input pipeline for the shared-expert branch of mixture-of-experts (MoE) models with sequence-sharded inputs. HiPipe first projects the local shard, then overlaps copy-engine transfers of remote shards with projections of ready shard groups, avoiding the need to wait for a complete all-gather before computation begins. The group size trades earlier execution against GEMM efficiency, and the paper argues that grouping and pipeline enablement should be selected according to joint completion of the shared and routed branches. The evaluation covers workloads derived from eleven architectures on H200 and A800, synthetic replays, native Megatron MoE layers, and reduced training stacks.
Critical strengths
- Clear and concrete motivation. The paper identifies avoidable input waiting within an already concurrent shared/routed schedule. The row-wise independence of the first projection directly motivates early execution.
- Broad coverage of workload shapes and input lengths. The experiments include eleven architecture-derived shapes, multiple input lengths, and two GPU platforms. The repeated A800 experiments provide evidence of long-input projection benefits across different shapes.
- Useful distinction between local and joint performance. The measured preferred grouping differs between input projection and joint completion in 27 of 55 workloads. The paper also reports regressions, includes conventional NCCL as an option in the offline selection comparison, and explicitly acknowledges the limitations of its native-layer and training results.
Critical weaknesses
- Practical system-level acceleration remains unestablished. The main benefits are demonstrated for input projection and synthetic joint replay. Native-layer measurements are generally slower than or near parity with the baseline. Although the authors appropriately acknowledge these limitations, the current evidence supports a local scheduling optimization more strongly than an effective integrated MoE acceleration system.
- The practical choice of grouping and pipeline enablement remains unclear. The results show that the preferred group size depends on the workload and the measurement objective, while shorter inputs can favor conventional execution. Although the paper demonstrates the importance of selecting by joint latency, it does not establish how to make these choices reliably in native execution. Clearer guidance on when to enable HiPipe and how to select its group size would strengthen its practical applicability.
- Sensitivity to realistic routed workloads is insufficiently characterized. The main joint replay uses a dense proxy that does not fully capture actual routing, token permutation, expert-load imbalance, and distributed expert execution. While the paper also evaluates native MoE layers, those experiments do not systematically isolate how routing configurations and token-load distributions affect the observed gains or regressions.
Questions for authors
- With shared-expert dimensions and input length held fixed, how do top-k, expert-load imbalance, routing policy, and routed communication volume affect joint gains and the preferred grouping? Could the authors provide controlled native-layer experiments that vary these factors?
- Could the authors provide a quantitative breakdown of the overhead introduced by input publication, weight-copy maintenance, additional GEMM launches, synchronization, and reduced GEMM efficiency from splitting? Which costs dominate in native cases with speedups below one, and how do they explain the gap between projection-level improvements and native-layer performance?
- How should the group size and pipeline enablement be selected in practice? Could the authors provide a decision guideline based on input length, shared-expert dimensions, and the relative durations of the shared and routed branches, and demonstrate whether this guideline improves native-layer latency across different workloads?