- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the communication bottlenecks of Expert Parallelism in distributed MoE training. The authors observed that many expert pairs are frequently co-activated by the same tokens. Inspire by this, they propose COBALT, a framework that using node-level token deduplication to reduce inter-node traffic while balancing GPU workloads. COBALT combines a two-stage layout planner, which performs periodic global placement via Mixed-Integer Linear Programming and per-step intra-node replica swapping via greedy bin-packing, with a GPU-accelerated minimum set-cover task assignment heuristic. Evaluations across three large MoE architectures on up to 32 NVIDIA B200 GPUs demonstrate notable end-to-end training speedup over several commonly used baselines.
Critical strengths
-
Using expert co-activation help node-level token deduplication is well-motivated. The results also show that incorporating this insight into experts placement can largely reduce the communication volume.
-
Decoupling global planning from intra-node rebalancing is well aligned with bandwidth hierarchy. Updating the global layout periodically over slower inter-node link while adjusting intra-node link at every step over nvlink keeps the adaptation overhead low.
-
The results on 32 B200 GPUs is solid and shows clear end-to-end speedups across three mordern MoE architectures.
Critical weaknesses
- The evaluation covers only 50 steps after an initial 200-step training period. Since the gains depend on routing patterns, it remains unclear whether similar improvements persist at later training stages.
- The paper does not explain how optimizer states are handled when replicas are dynamically adjusted. How gradients and optimizer states remain consistent?
- The bitwise enumeration in Algorithm 1 scales exponentially with the number of nodes. Will it become a bottleneck if the parallelism spans a larger number of nodes?
- Although all baselines use DeepEP, the evaluation only compares against per-expert load balancers (SmartMoE and LAER-MoE) and lacks comparisons with communication-aware placement baselines such as HierMoE and Occult.
Questions for authors
- Have you evaluated COBALT at later training stages? How do changes in expert co-activation affect its speedup?
- How are gradients aggregated and optimizer states maintained?