acceptodds
Under review as a conference paper at ICLR 2027

Cobalt: Leveraging Expert Co-activation for Efficient Distributed MoE Training

10 views

Abstract

Mixture-of-Experts (MoE) has increasingly become a mainstream approach for scaling large language models, as it expands model capacity while keeping computation cost nearly constant. Training large-scale MoE models relies on Expert Parallelism (EP), which distributes expert replicas across GPUs and exchanges tokens through all-to-all communication. The efficiency of EP is often constrained by two system bottlenecks: cross-node token transfers are limited by inter-node bandwidth, while skewed expert workloads lead to imbalanced computation across GPUs. Prior work mitigates these bottlenecks based on per-expert workload statistics, but overlooks the fact that experts could share the communication. In this work, we empirically present the observation that many pairs of experts are frequently co-activated by individual tokens. Motivated by this, we present Cobalt, an efficient MoE training framework that leverages expert co-activation to reduce cross-node traffic and workload imbalance. Cobalt adopts a two-stage expert layout planner that adapts expert layout to the evolving expert co-activation and workload conditions. It periodically co-locates frequently co-activated experts on the same node to reduce the cross-node communication, and performs per-step intra-node adjustment to rebalance the workloads. Subsequently, we develop a communication-aware task assignment method that routes tokens to fewer remote nodes based on the current expert layout. Experiments on 32 B200 GPUs show that Cobalt achieves up to 1.53-2.41 times (1.28-1.89 times on average) of speedup compared to existing MoE training frameworks, while reducing cross-node token traffic by 75.74%-99.26%. Source code is available.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Expert Parallelism, Distributed Training, Mixture-of-Experts, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_espsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the communication bottlenecks of Expert Parallelism in distributed MoE training. The authors observed that many expert pairs are frequently co-activated by the same tokens. Inspire by this, they propose COBALT, a framework that using node-level token deduplication to reduce inter-node traffic while balancing GPU workloads. COBALT combines a two-stage layout planner, which performs periodic global placement via Mixed-Integer Linear Programming and per-step intra-node replica swapping via greedy bin-packing, with a GPU-accelerated minimum set-cover task assignment heuristic. Evaluations across three large MoE architectures on up to 32 NVIDIA B200 GPUs demonstrate notable end-to-end training speedup over several commonly used baselines.

Critical strengths
  1. Using expert co-activation help node-level token deduplication is well-motivated. The results also show that incorporating this insight into experts placement can largely reduce the communication volume.

  2. Decoupling global planning from intra-node rebalancing is well aligned with bandwidth hierarchy. Updating the global layout periodically over slower inter-node link while adjusting intra-node link at every step over nvlink keeps the adaptation overhead low.

  3. The results on 32 B200 GPUs is solid and shows clear end-to-end speedups across three mordern MoE architectures.

Critical weaknesses
  1. The evaluation covers only 50 steps after an initial 200-step training period. Since the gains depend on routing patterns, it remains unclear whether similar improvements persist at later training stages.
  2. The paper does not explain how optimizer states are handled when replicas are dynamically adjusted. How gradients and optimizer states remain consistent?
  3. The bitwise enumeration in Algorithm 1 scales exponentially with the number of nodes. Will it become a bottleneck if the parallelism spans a larger number of nodes?
  4. Although all baselines use DeepEP, the evaluation only compares against per-expert load balancers (SmartMoE and LAER-MoE) and lacks comparisons with communication-aware placement baselines such as HierMoE and Occult.
Questions for authors
  1. Have you evaluated COBALT at later training stages? How do changes in expert co-activation affect its speedup?
  2. How are gradients aggregated and optimizer states maintained?
Reviewer_HvgKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper targets cross-node all-to-all traffic in expert-parallel (EP) MoE training under node-level token deduplication, where a token's cross-node cost is the number of distinct remote nodes that hold its selected experts. Motivated by frequent expert co-activation (the tracked pairs' Ochiai similarity rises from 0.41 to 0.89 over the first 200 steps of GLM-4.5-Air training), Cobalt (i) re-plans expert replication and placement every 10 steps with an MILP that maximizes the co-activation of co-located experts under a replica budget ρ\rho and an imbalance bound ε\varepsilon, (ii) rebalances replicas within each node at every step with a greedy partition, and (iii) assigns each token's experts to the fewest remote nodes through a small set-cover enumeration. On 32 B200 GPUs with truncated Hunyuan3, GLM-4.5-Air and DeepSeek-V3 models, it reports a 1.28× average (up to 1.53×) speedup over LAER-MoE and 75.7–99.3% lower cross-node volume than EP+FSDP.

Critical strengths
  • The cost model matches the communication primitive in use, and the problem decomposition is clean. Eq. (1) counts distinct remote nodes per token, which is what DeepEP-style node-level deduplication actually pays for. Eq. (2)–(4) split the joint problem into a periodic MILP over layouts and a per-token minimum set cover that is cheap at the evaluated scale (2N−1=82^{N-1} = 8 masks on four nodes). This framing explains why per-expert load statistics cannot capture the achievable savings, which is the paper's central argument.
  • The measurements use current hardware and models, with variance, ablation and overhead accounting. All frameworks use DeepEP on 32 B200 GPUs across three architectures and two EP degrees, with five runs and error bars (Figure 6). Figure 7 attributes the gain to intra-node rebalancing (1.22×), periodic global planning (1.43×) and task assignment (1.53×), and Table 1 reports relayout costs of 2.0–2.4 s every 10 steps, about 2% of the 11–13 s GLM-4.5-Air step. Code is released.
  • The gain grows with the EP degree. Against LAER-MoE the speedup rises from 1.07–1.12× at EP16 to 1.30–1.53× at EP32, in line with MoE communication taking 58–83% of step time (Figure 2). If the trend holds beyond four nodes, the method addresses the main scaling bottleneck of EP training.
Critical weaknesses
  1. All evidence comes from the first ~250 steps of from-scratch training, where routing is unusually concentrated. Every framework starts from a checkpoint after 200 steps and is timed over the next 50 (Section 4.1), and Figure 3 covers the same 200 steps. Routing here is far from balanced: the EP+FSDP imbalance ratio is 2.65–6.66 (Figure 6), and at EP16 Cobalt removes 98–99% of cross-node traffic, so nearly every token finds all eight of its experts on its own node. The paper does not state the training load-balancing objective, or whether the Figure 3 averages cover all expert pairs or only the tracked ones. Step time and traffic at a later checkpoint or with the released routers, and the all-pair Ochiai distribution against its independent-routing level (about K/EK/E), are needed.
  2. The novelty claim and the description of the closest prior work need correction, and that work is not compared. The paper presents itself as the first to use expert co-activation to reduce cross-node communication in MoE training. Occult (ICML 2025) already treats co-activated experts as collaborating, places them on one device and sends one copy of a token per device; the paper describes it instead as clustering experts "with similar workloads". HierMoE (INFOCOM 2026) combines hierarchical deduplication with duplicate-aware expert swaps for training, and Semantic Parallelism (ICLR 2026) groups co-activated experts for inference. The new elements appear to be periodic re-planning with replication under a balance constraint and the set-cover assignment. A comparison with HierMoE, or with a placement computed once at step 200, would isolate their value.
  3. The memory and state-migration costs of replication are not reported. With ρ=0.5\rho = 0.5, each rank holds 50% more expert slots than under EP+FSDP, while the models were already truncated "due to the memory constraint". Peak memory per GPU is not reported for any framework, and the paper does not describe how optimizer states and replica gradients are handled when the global layout changes every 10 steps. Peak memory at matched configurations, the relayout traffic including optimizer state, and MILP solve times for larger expert counts and more nodes (the formulation has O(E2N)O(E^2 N) pair variables) would establish the practical cost.
Questions for authors
  1. Clarification. Which load-balancing objective (auxiliary-loss coefficient or bias-based) is used in training, and how do the co-activation statistics and the speedup over LAER-MoE change at a later checkpoint, for example after 5k steps?
  2. Clarification. Are the averages in Figure 3(a)(d) taken over all expert pairs or over the pairs shown in the heatmaps? What is the all-pair average, and what would independent routing give?
  3. Clarification. What is the peak per-GPU memory of each framework in Figure 6, and are optimizer states migrated together with expert parameters during global relayout?
  4. Clarification. Figure 6 shows LAER-MoE reducing cross-node volume by 39–71%, while Section 4.2 states that it does not address cross-node communication. Could the text reconcile these?
Reviewer_M32uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Cobalt, a distributed MoE training framework that reduces cross-node communication and load imbalance by leveraging expert co-activation patterns. Cobalt combines periodic global expert placement, per-step intra-node load rebalancing, and communication-aware task allocation strategies. Experiments on up to 32 NVIDIA B200 GPUs show that this approach has certain performance advantages compared to existing systems.

Critical strengths

S1. The paper addresses communication overhead and load imbalance in distributed MoE training. The analysis of expert co-activation patterns provides good motivation for system design. S2. Cobalt combines global expert placement, local load rebalancing, and communication-aware task allocation, balancing cross-node communication costs and GPU load. S3. The system implementation and experiments cover multiple MoE models and baseline methods. Ablation studies and cost analysis help verify the practical effect of each component.

Critical weaknesses

W1. Innovation is not clear compared to Occult. Occult has already used expert co-activation patterns to optimize expert placement and reduce communication overhead. The paper needs to further explain the algorithm-level differences in Cobalt, not just the addition of expert replicas and load balancing mechanisms. W2. The scalability of the MILP optimization algorithm is unclear. Global layout planning relies on mixed-integer linear programming (MILP), but there is no scalability analysis for more experts, nodes, and replicas. Further evaluation of planning time and its impact on periodic layout updates is needed. W3. The generality of expert co-activation patterns is not well validated. Figure 3 only analyzes two layers of GLM-4.5-Air, and it’s unclear whether similar patterns exist in other layers or models, and whether these patterns remain stable during training. W4. The distinction from Libra’s method is insufficiently explained. Although the two systems target different execution scenarios, both use expert activation information for load balancing. The paper should clarify their methodological differences, especially between historical EMA-based statistics and prediction-based scheduling. W5. The analysis of communication overhead is incomplete. The reported cross-node communication does not include that caused by FSDP and expert layout updates. Parameter transfers and replica synchronization can also add extra cost, so it’s recommended to evaluate the overall communication cost of the system.

Questions for authors

Q1. What are the main algorithmic differences between Cobalt and Occult? Could a direct experimental comparison be provided? Q2. How scalable is the MILP planning algorithm with larger models and EP parallelism? If the planning time exceeds the update cycle, how does the system respond? Q3. Are the expert co-activation patterns consistent across different layers and training stages? When co-activation relationships are weak or unstable, does Cobalt still perform effectively? Q4. How does Cobalt's EMA-based load estimation differ from Libra's method? Why not adopt a prediction-based scheduling strategy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.