- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes CAER, a system for improving multimodal MoE training throughput through expert replication. It combines a layer-specific adaptive imbalance gate, whose threshold is adjusted by AIMD feedback, with a greedy scheduler that estimates computation, communication, and replication costs. CAER preserves the router's original token-expert assignments and consolidates replica gradients at the canonical expert owners. Experiments on three models and seven datasets show throughput improvements over vanilla training and EPLB, and over LPLB on the language-only workloads. The paper also includes ablations on the gate and cost-aware selection.
Critical strengths
1. CAER speeds up training without changing the original routing. It moves some expert computation to less-loaded GPUs and sends replica gradients back to the original experts. Each token still uses the experts selected by the router.
2. The results are consistent across several models and datasets. The paper tests three multimodal MoE architectures on both multimodal and language-only data. Appendix B.1 also shows similar gains across different measurement windows.
Critical weaknesses
1. The improvement over prior methods is unclear. FlexMoE [1] already uses an imbalance threshold and a cost model to guide expert adjustment. FasterMoE [2] also uses a performance model to guide replication. CAER uses a per-layer adaptive gate and temporary replicas, but what these changes add is not clear enough. The experiments only compare with EPLB and LPLB, without related training methods such as LAER-MoE [3] or LLEP [4]. I would like to see these comparisons. Without them, it is unclear whether CAER improves over existing training methods.
2. The ablations do not clearly show why the specific design helps. Removing cost-aware selection shows that considering cost helps, but does not show whether the tile/wave model is better than a simpler cost model. I also do not see a comparison between predicted savings and measured savings. The AIMD ablation uses only one fixed threshold, so a tuned fixed threshold may give different results. In addition, all main experiments use one node, leaving the cross-node communication term untested. These checks would help explain how much the proposed cost model and adaptive gate contribute to the gains.
References:
[1] Nie et al. FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement
[2] He et al. FasterMoE: modeling and optimizing training of large-scale dynamic pre-trained models
[3] Liu et al. LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training
[4] Nguyen et al. Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts
Questions for authors
1. Clarification: What does CAER add beyond FlexMoE's threshold and cost-aware scheduling, and FasterMoE's model-guided replication? The differences in the gate, cost model, and the scheduler are not clear enough.
2. Baseline comparison: Could the authors compare with LLEP or LAER-MoE under the same hardware and memory budget?
3. Cost-model validation: Does the tile/wave model select faster plans than a simpler token-count-based model? Could the authors also compare predicted savings with measured savings?
4. Gate validation: Does AIMD still improve throughput over a fixed threshold tuned on separate calibration workloads?