acceptodds
Under review as a conference paper at ICLR 2027

CAER: Cost-Aware Expert Replication for Load Balancing in Mixture-of-Experts Multimodal Large Language Model Training

Abstract

Mixture-of-Experts (MoE) has become a key architecture for scaling multimodal large language models, yet its input-dependent routing can cause load imbalance in expert-parallel (EP) training, leading to stragglers and reduced training throughput. Expert replication is widely used to mitigate such imbalance, but historical load estimates may fail to track rapid workload shifts, while fine-grained, frequent replication introduces additional costs that can offset its benefits. We propose CAER, a cost-aware expert replication method for efficient multimodal MoE (MMoE) training. CAER selectively triggers rebalancing through an adaptive imbalance gate and uses a cost-aware greedy scheduler to identify minimum-cost replication plans under a calibrated cost model that jointly captures computation, communication, and replication overhead. It then dynamically replicates overloaded experts onto underloaded devices while preserving the original routing patterns. Across diverse MMoE architectures and datasets, CAER consistently outperforms state-of-the-art methods, improving end-to-end training throughput by up to 31.8% over EPLB and 31.0% over LPLB while maintaining comparable or better load balance. These results demonstrate the effectiveness and robustness of CAER across diverse training settings.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
MoE models, Load Balancing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_99HYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CAER, a system for improving multimodal MoE training throughput through expert replication. It combines a layer-specific adaptive imbalance gate, whose threshold is adjusted by AIMD feedback, with a greedy scheduler that estimates computation, communication, and replication costs. CAER preserves the router's original token-expert assignments and consolidates replica gradients at the canonical expert owners. Experiments on three models and seven datasets show throughput improvements over vanilla training and EPLB, and over LPLB on the language-only workloads. The paper also includes ablations on the gate and cost-aware selection.

Critical strengths

1. CAER speeds up training without changing the original routing. It moves some expert computation to less-loaded GPUs and sends replica gradients back to the original experts. Each token still uses the experts selected by the router.

2. The results are consistent across several models and datasets. The paper tests three multimodal MoE architectures on both multimodal and language-only data. Appendix B.1 also shows similar gains across different measurement windows.

Critical weaknesses

1. The improvement over prior methods is unclear. FlexMoE [1] already uses an imbalance threshold and a cost model to guide expert adjustment. FasterMoE [2] also uses a performance model to guide replication. CAER uses a per-layer adaptive gate and temporary replicas, but what these changes add is not clear enough. The experiments only compare with EPLB and LPLB, without related training methods such as LAER-MoE [3] or LLEP [4]. I would like to see these comparisons. Without them, it is unclear whether CAER improves over existing training methods.

2. The ablations do not clearly show why the specific design helps. Removing cost-aware selection shows that considering cost helps, but does not show whether the tile/wave model is better than a simpler cost model. I also do not see a comparison between predicted savings and measured savings. The AIMD ablation uses only one fixed threshold, so a tuned fixed threshold may give different results. In addition, all main experiments use one node, leaving the cross-node communication term untested. These checks would help explain how much the proposed cost model and adaptive gate contribute to the gains.

References:

[1] Nie et al. FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement

[2] He et al. FasterMoE: modeling and optimizing training of large-scale dynamic pre-trained models

[3] Liu et al. LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training

[4] Nguyen et al. Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts

Questions for authors

1. Clarification: What does CAER add beyond FlexMoE's threshold and cost-aware scheduling, and FasterMoE's model-guided replication? The differences in the gate, cost model, and the scheduler are not clear enough.

2. Baseline comparison: Could the authors compare with LLEP or LAER-MoE under the same hardware and memory budget?

3. Cost-model validation: Does the tile/wave model select faster plans than a simpler token-count-based model? Could the authors also compare predicted savings with measured savings?

4. Gate validation: Does AIMD still improve throughput over a fixed threshold tuned on separate calibration workloads?

Reviewer_RhyxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CAER, a cost-aware expert replication framework to mitigate load imbalance in multimodal MoE training. The key ideas of CAER are selective triggering rebalancing through an adaptive imbalance gate, and identifying cost-effective replication plans using a calibrated cost model that jointly captures the computation, communication, and replication overhead. Experimental results demonstrate the effectiveness of CAER.

Critical strengths
  1. CAER is well motivated and provides a correct and useful insight: better load balance does not always mean higher throughput.
  2. The design of CAER is clean with two key techniques to address when and how to do rebalancing, respectively.
  3. Evaluation results demonstrate the effectiveness of CAER.
Critical weaknesses
  1. All experiments run on a single 8-GPU H100 node, and thus cannot evaluate the cross-node communication overhead. Large-scale MoE training is typically multi-node where cross-node communication is a key constraint and must be evaluated.

  2. It is unclear why LPLB is absent from the multimodal experiments (Table 1) and appears only in the language-only ones (Table 2).

  3. Run-time overheads are not fully evaluated, particularly the memory cost of the replication.

  4. The cost model introduces some hardware-dependent parameters which must be calibrated but no sensitivity analysis was provided.

Questions for authors
  1. Can you provide multi-node experimental results to evaluate the performance of CAER in typical MOE training settings?
  2. Why are LPLB results provided in only Table 2 but not Table 1?
  3. What is the memory overhead and how will CAER perform in memory-constrained scenarios (e.g., larger model or less GPU memory)?
  4. How are the parameters of the cost model calibrated and how the calibration may impact the performance?
Reviewer_FCf7OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CAER, a cost-aware expert replication for MMoE model training. The proposed method includes an adaptive imbalance gate with a cost-aware scheduler, which can simulate the hadwre behavior during the training.

Critical strengths
  1. How to improve the training efficiency of the MMoE model is a very important topic.
  2. The proposed methods are straightforward and rational.
Critical weaknesses
  1. The proposed method seems to have little contribution to the overall performance (from the experiments, the improvements are always less than 1.1x, although in the training case, the improvement is not important). I think the main reason is that with a large batch, the workload tends to balance out.
  2. The experiment setting is relatively small. For example, the experiment only considers 200 steps for end-to-end performance evaluation and 8 GPUs for training. In fact, I think more GPUs would improve performance, since the workload may become more imbalanced with more hardware resources.
Questions for authors
  1. Does MMoE have a significant difference from MoE models?
  2. Does the batch selection (for example some group method for different request) affect the expert selection and training efficiency?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.