acceptodds
Under review as a conference paper at ICLR 2027

J-MoE: Attention-Aware Joint Sample–Expert Placement for MoE RL Training

Abstract

Reinforcement learning (RL) post-training has become central to improving large language models, which increasingly adopt mixture-of-experts (MoE) architectures. However, variable-length rollout samples and skewed expert routing cause substantial load imbalance across GPUs in three major cost components: dense attention computation, sparse expert computation, and routed All-to-All communication. Balancing these costs requires coordinating two tightly coupled decisions: sample placement and expert placement. Sample placement determines both attention workloads and the sources of routed traffic, while expert placement determines both expert workloads and traffic destinations. Existing load-balancing techniques address only subsets of these costs, without jointly optimizing all three through coordinated sample and expert placement. We present J-MoE, an MoE RL training system that jointly plans sample and expert placement to trade off among all three costs at once. J-MoE leverages routing replay, in which training reuses the routing decisions recorded during rollout, making each sample's expert demand known exactly before training begins. Guided by this information and a unified cost model of all three components, a two-stage solver first assigns the samples of each optimizer step to microbatches and GPUs, and then jointly optimizes sample and expert placement for each microbatch. To execute these plans efficiently, J-MoE redistributes samples to their assigned GPUs and overlaps expert reconfiguration with attention computation, without altering routing or dropping tokens. Across multiple MoE models and RL datasets, J-MoE achieves up to training speedup over existing training systems.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture-of-Experts, Reinforcement Learning Post-Training, Distributed Training, Load Balancing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_duAVOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to enable multiple GPUs to complete MoE reinforcement learning training more efficiently. The authors utilized the routing information recorded during the generation process to arrange the training samples and the positions of the experts simultaneously, considering attention, expert computation, and communication overhead. During execution, they overlapped expert transfers with attention computation to reduce waiting time between GPUs. Experiments compared the training speed and load distribution under different arrangements.

Critical strengths

S1. Arranging the samples and experts together is a relatively straightforward approach. The positions on both sides would affect communication, and optimizing only one side would indeed lead to trade-offs.

S2. The experiments not only report the acceleration but also examine the changes in attention, expert load, and cross-machine communication. These results can clearly illustrate where the speed improvement comes from.

S3. The effect of overlapping expert transfers and attention computation has been measured in practice. Compared to only estimating the transfer time, this provides a more reliable indication that the solution can actually run in the test environment.

Critical weaknesses

W1. The communication cost is mainly estimated based on the total amount of cross-machine transmission, but even with the same total amount, speed can be affected by which GPUs have the concentrated traffic. It is not clear yet whether this estimation can reliably select the arrangement with faster communication.

W2. The baseline comparison did not compare the effects of sequential arrangement and alternating adjustment under the same consideration of the three types of costs. Therefore, it is not easy to distinguish how much of the benefit comes from the joint adjustment itself and how much from the more complete consideration of costs.

Questions for authors

Q1. Can the predicted values and actual measured values of communication times for several arrangements be compared to confirm whether the cost model can correctly judge their speed?

Q2. Can the sequential arrangement and alternating adjustment be compared under the same cost model, initial arrangement, and search budget, separately looking at the benefits brought by the joint adjustment?

3 comments
AuthorsOpenReview ↗

Official Comment by Authors [1/1]

Thank you for your encouraging comments and insightful suggestions. We address each point below.

W1 & Q1. Communication Cost Model Accuracy

We have added Appendix D and Fig. 8 in the revised paper to compare predicted and measured All-to-All communication times and identify a suitable model for our configuration:

Cost modelMAPE
Attention computation3.03%
Expert computation0.89%
Communication: total inter-node traffic5.81%
Communication: maximum per-node load5.93%
Communication: maximum per-GPU load7.49%

Fig. 8(c)–(e) compare three All-to-All communication models fitted using total inter-node traffic, maximum per-node communication load, and maximum per-GPU communication load, respectively. On our cluster, the total-volume model achieves the lowest prediction error (5.81%), closely matching the maximum-node model (5.93%) and outperforming the more variable maximum-GPU model (7.49%). Given this comparable (and slightly better) accuracy, we adopt the total inter-node traffic model in Eq. 4 for two reasons: (1) smoother optimization guidance: the maximum-node model yields gain only when reducing traffic on the currently busiest node and is insensitive to traffic reductions among other nodes, whereas the additive total-volume model directly rewards any reduction in inter-node traffic and more effectively guides the search toward communication locality; and (2) lighter incremental evaluation: each candidate swap simply updates a local inter-node traffic delta without maintaining and recomputing per-node send/receive maxima at every trial.

W2 & Q2. Ablations on Solver and Objective Components

We have expanded Fig. 6 (§4.2) in the revised paper with additional ablations to demonstrate the benefit of each component (all speedups are relative to Megatron-Balanced):

VariantCodeforcesMixed
E→S1.15×1.21×
S→E1.19×1.19×
w/o Stage 11.21×1.26×
w/o Stage 21.15×1.22×
w/o Comm1.22×1.17×
w/o Attention1.15×1.22×
w/o Cross-MB1.19×1.27×
J-MoE1.25×1.30×

These ablations quantify the sources of J-MoE's improvement along three dimensions:

  1. Contribution of communication awareness and the unified objective: As shown in Fig. 6(b), removing the communication term (w/o Comm) reduces the speedup from 1.25× to 1.22× on Codeforces and from 1.30× to 1.17× on Mixed, confirming the substantial contribution of communication awareness. Similarly, removing the attention term (w/o Attention: 1.15× / 1.22×) worsens attention-computation stragglers and degrades performance. The expert-computation term is retained across all variants, as expert placement relies on it for guidance.
  2. Contribution of joint vs. sequential optimization: We compare against sequential optimization at two levels. First, at the overall workflow level, we have also added ReLibra, ForeMoE, and RoutePack to our main evaluation in Fig. 5 and Tab. 4 (§4); all these baselines essentially follow one-way sequential pipelines (e.g., EPLB+ and ForeMoE can be viewed as balancing samples by length first and then reconfiguring experts, whereas ReLibra, MBR, and RoutePack determine expert placement first and then adjust samples), and J-MoE consistently outperforms all of them (by up to 1.07×–1.13×, and up to 1.93× over Megatron-LM). Second, within the Stage 2 solver, to strictly isolate the benefit of alternating iterations, we construct two sequential variants, E→S and S→E, which execute Expert-KL and Sample-KL in a single pass under either order. As shown in Fig. 6(a), starting from the exact same initial placement (Megatron-Balanced), three-component objective, and search budget, alternating Sample-KL and Expert-KL (w/o Stage 1: 1.21× / 1.26×) significantly outperforms both one-pass sequential schemes, E→S (1.15× / 1.21×) and S→E (1.19× / 1.19×), because each placement can re-adapt after the other changes traffic sources or destinations. Combining this alternating refinement with our coupled Stage 1 initialization (w/o Stage 2: 1.15× / 1.22×) yields the highest speedup in full J-MoE (1.25× / 1.30×).
  3. Contribution of cross-microbatch sample flexibility: Prohibiting samples from moving across microbatches (w/o Cross-MB: 1.19× / 1.27×) likewise degrades performance, because fixing microbatch membership limits the solver's ability to rebalance aggregate expert demand across microbatches.

Thank you for the response. It addresses most of my concerns, and I will keep my current positive rating.

Thank you for your thoughtful review and feedback. We’re glad that our response helped address your concerns and appreciate your valuable insights.

Reviewer_KjgXOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes J-MoE, a system for jointly optimizing sample and expert placement in Mixture-of-Experts (MoE) reinforcement learning training for LLMs. The proposed framework addresses load imbalance arising from variable-length sequences, uneven expert routing, and inter-node communication. By leveraging routing replay, J-MoE obtains sample-level expert demands in advance and introduces a unified cost model that considers attention computation, expert computation, and All-to-All communication. A two-stage solver is developed to determine sample and expert placement, while the runtime overlaps expert reconfiguration with attention computation. Experiments across multiple MoE configurations and RL datasets demonstrate training speedups of up to 1.93× over the standard Megatron-LM baseline.

Critical strengths
  1. This paper addresses a significant efficiency bottleneck in MoE RL training. Jointly considering sequence-length imbalance, expert-load imbalance, and communication overhead is well motivated and relevant to large-scale distributed training.
  2. J-MoE integrates routing replay, a unified cost model, a two-stage placement solver, and asynchronous expert reconfiguration into a coherent system. The interaction between sample placement and expert placement is clearly explained.
  3. The evaluation covers multiple MoE architectures, RL datasets, and expert-parallel configurations. The reported speedups, load-balancing improvements, and reduced inter-node communication demonstrate the practical potential of the proposed J-MoE.
Critical weaknesses
  1. The paper discusses RoutePack and ForeMoE as closely related approaches, but neither is directly included in the experimental comparison. Instead, the evaluation mainly relies on Megatron-LM, EPLB+, and a microbatch-level adaptation of ReLibra. Since RoutePack already considers attention-aware sample and expert placement, a direct comparison is particularly important for establishing the incremental contribution of J-MoE. Moreover, the improvements over the stronger implemented baselines are relatively modest, reaching up to 1.10× over EPLB+ and Microbatch-Reordering.
  2. The communication model in Eq. (4) estimates latency primarily from aggregate inter-node token traffic, without explicitly accounting for per-node communication hotspots, link contention, or topology-dependent bandwidth utilization. These factors can substantially influence distributed All-to-All latency. Furthermore, the model does not explicitly incorporate expert reconfiguration costs, assuming that these costs can be hidden by attention computation.
  3. Figure 6 evaluates the two solver stages, but it does not separately examine the contributions of the three cost components. In particular, the paper lacks experiments removing the communication term, comparing joint refinement with sequential placement optimization, and isolating the benefits of cross-microbatch sample assignment. These experiments are important for determining whether the improvements primarily arise from the unified objective, the increased optimization flexibility, or the heuristic initialization.
Questions for authors
  1. Could the authors provide direct experimental comparisons with RoutePack and ForeMoE under identical hardware, model, and workload configurations? How much of J-MoE's improvement can be attributed specifically to communication-aware joint optimization?
  2. How accurately does the proposed cost model predict measured per-layer execution times? Could the authors report prediction errors and evaluate the impact of removing the communication component from the optimization objective?
4 comments
AuthorsOpenReview ↗

Official Comment by Authors [1/2]

We greatly appreciate the time and effort you have devoted to reviewing our work. Your thoughtful suggestions have helped us improve both its clarity and completeness.

W1 & Q1.1. Direct Comparisons with Related Systems

In the revised paper (§4), we have added ReLibra, ForeMoE, and RoutePack as baselines, and updated the baseline descriptions, end-to-end performance comparison (Fig. 5), and case-study analysis (Tab. 4).

End-to-end training performance (Fig. 5). J-MoE achieves the highest speedup across all evaluated configurations, with maximum speedups over each baseline summarized below:

BaselineMegatron-LMMegatron-BalancedEPLB+ReLibraForeMoEMBRRoutePack
J-MoE speedup1.93×1.32×1.10×1.13×1.07×1.10×1.08×

J-MoE outperforms the newly added ReLibra, ForeMoE, and RoutePack by up to 1.13×, 1.07×, and 1.08×, respectively. Specifically, ReLibra and RoutePack fix expert placement at batch granularity and cannot adapt to expert-demand fluctuations across microbatches; although ForeMoE reconfigures experts per microbatch, it keeps sample placement fixed and therefore cannot jointly coordinate traffic sources and microbatch expert demand.

Case study (Tab. 4). On Mixed with Qwen3 Instruct (8K, EP16), Tab. 4 reports the max-to-mean GPU load imbalance (1 = ideal) and total inter-node traffic:

MethodAttn. Imbal.Expert Imbal.Inter-node Traffic (GiB)
Megatron-LM2.7161.7722795.04
Megatron-Balanced1.0341.8152789.96
EPLB+1.0341.2592787.66
ReLibra1.0381.2492758.78
MBR1.1071.1022605.01
ForeMoE1.0341.1262761.86
RoutePack1.0431.2582787.36
J-MoE1.0151.0572238.28

ReLibra optimizes under a fixed batch-level expert placement, resulting in higher overall expert imbalance and limited communication reduction; RoutePack likewise fixes expert placement at batch granularity and omits communication cost from its objective, leading to higher expert imbalance and leaving inter-node traffic nearly unchanged; ForeMoE optimizes expert placement without adjusting sample placement, which limits its ability to reduce inter-node communication.

W2.1 & Q2.1. Cost-Model Accuracy

We have added Appendix D and Fig. 8 in the revised paper to compare predicted and measured per-layer execution times for attention computation, expert computation, and routed-token communication. The mean absolute percentage errors (MAPE) of each model are summarized below:

Cost modelMAPE
Attention computation3.03%
Expert computation0.89%
Communication: total inter-node traffic5.81%
Communication: maximum per-node load5.93%
Communication: maximum per-GPU load7.49%

The attention and expert computation models closely match the measured times (for Qwen3.5, linear-attention and full-attention layers are calibrated separately as described in Appendix A). To examine potential node/GPU communication hotspots and link contention, Fig. 8(c)–(e) compare three All-to-All communication models fitted using total inter-node traffic, maximum per-node communication load, and maximum per-GPU communication load, respectively. On our cluster, the total-volume model achieves the lowest prediction error (5.81%), closely matching the maximum-node model (5.93%) and outperforming the more variable maximum-GPU model (7.49%). Given this comparable (and slightly better) accuracy, we adopt the total inter-node traffic model in Eq. 4 for two reasons: (1) smoother optimization guidance: the maximum-node model yields gain only when reducing traffic on the currently busiest node and is insensitive to traffic reductions among other nodes, whereas the additive total-volume model directly rewards any reduction in inter-node traffic and more effectively guides the search toward communication locality; and (2) lighter incremental evaluation: each candidate swap simply updates a local inter-node traffic delta without maintaining and recomputing per-node send/receive maxima at every trial.

AuthorsOpenReview ↗

Official Comment by Authors [2/2]

W2.2. Expert-Reconfiguration Overlap

Fig. 7(b) and (c) show that forward parameter prefetch and backward gradient return are completely overlapped with attention computation in our evaluated configurations, which is why we omit a separate expert-reconfiguration cost term from the objective. In the revised Appendix C, we further removed the unnecessary conservative bound assumption in the original derivation and expanded the analysis of this overlap condition, showing that an effective bandwidth of BWa2a≥\mathrm{BW}_{\mathrm{a2a}} \ge 2.68 GB/s (Qwen3) and 3.11 / 5.73 GB/s (Qwen3.5 full-/linear-attention layers) suffices for complete overlap:

  • Bandwidth requirement: On our cluster, the profiled BWa2a\mathrm{BW}_{\mathrm{a2a}} is 22 GB/s, well above the requirement for complete overlap. Moreover, as inter-node bandwidth decreases (while still meeting the overlap threshold), per-layer All-to-All activation communication accounts for a larger share of step time, which further amplifies the speedup from J-MoE's inter-node traffic reduction.
  • Sequence-length requirement: Conversely, at 22 GB/s, complete overlap requires a mean complete sequence length of only 2,258 (Qwen3) and 2,402 (Qwen3.5) tokens, meaning only 210 and 354 response tokens on top of a 2,048-token prompt. This condition is easily met in practice, as modern RL post-training operates well above these short-sequence thresholds: mathematical/STEM reasoning uses maximum response lengths up to 128K tokens (UloRL [1], MAI-Thinking-1 [2]), and agentic RL trains with contexts of 256K to 1M tokens (Nemotron-Cascade 2 [3], MiMo-V2.6 [4]), providing even larger attention windows to hide expert reconfiguration.

References

[1] Du et al. UloRL: An Ultra-Long Output Reinforcement Learning Approach for Advancing Large Language Models' Reasoning Abilities. 2025.

[2] The Microsoft AI Team. MAI-Thinking-1: Building a Hill-Climbing Machine. 2026.

[3] Yang et al. Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation. 2026.

[4] Xiaomi LLM-Core Team. MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement. 2026.

W3, Q1.2 & Q2.2. Ablations on Solver and Objective Components

We have expanded Fig. 6 (§4.2) in the revised paper with additional ablations to demonstrate the benefit of each component (all speedups are relative to Megatron-Balanced):

VariantCodeforcesMixed
E→S1.15×1.21×
S→E1.19×1.19×
w/o Stage 11.21×1.26×
w/o Stage 21.15×1.22×
w/o Comm1.22×1.17×
w/o Attention1.15×1.22×
w/o Cross-MB1.19×1.27×
J-MoE1.25×1.30×

These ablations quantify the sources of J-MoE's improvement along three dimensions:

  1. Contribution of communication awareness and the unified objective: As shown in Fig. 6(b), removing the communication term (w/o Comm) reduces the speedup from 1.25× to 1.22× on Codeforces and from 1.30× to 1.17× on Mixed, confirming the substantial contribution of communication awareness. Similarly, removing the attention term (w/o Attention: 1.15× / 1.22×) worsens attention-computation stragglers and degrades performance. The expert-computation term is retained across all variants, as expert placement relies on it for guidance.
  2. Contribution of joint vs. sequential optimization: We compare against sequential optimization at two levels. First, at the overall workflow level, as discussed in W1 & Q1.1, existing baselines essentially follow one-way sequential pipelines (e.g., EPLB+ and ForeMoE can be viewed as balancing samples by length first and then reconfiguring experts, whereas ReLibra, MBR, and RoutePack determine expert placement first and then adjust samples), and J-MoE consistently outperforms all of them. Second, within the Stage 2 solver, to strictly isolate the benefit of alternating iterations, we construct two sequential variants, E→S and S→E, which execute Expert-KL and Sample-KL in a single pass under either order. As shown in Fig. 6(a), starting from the exact same initial placement (Megatron-Balanced), three-component objective, and search budget, alternating Sample-KL and Expert-KL (w/o Stage 1: 1.21× / 1.26×) significantly outperforms both one-pass sequential schemes, E→S (1.15× / 1.21×) and S→E (1.19× / 1.19×), because each placement can re-adapt after the other changes traffic sources or destinations. Combining this alternating refinement with our coupled Stage 1 initialization (w/o Stage 2: 1.15× / 1.22×) yields the highest speedup in full J-MoE (1.25× / 1.30×).
  3. Contribution of cross-microbatch sample flexibility: Prohibiting samples from moving across microbatches (w/o Cross-MB: 1.19× / 1.27×) likewise degrades performance, because fixing microbatch membership limits the solver's ability to rebalance aggregate expert demand across microbatches.
Reviewer_KjgXOpenReview ↗

Response to Authors

Thank you for your detailed additional description and results, such as direct comparisons with related systems and cost-model accuracy, which address my main concerns. I agree to accept this paper.

Thank you for your detailed review and feedback. We’re pleased to have addressed your main concerns and appreciate your valuable insights.

Reviewer_p9TqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

J-MoE jointly plans sample and expert placement for MoE reinforcement-learning training using routing decisions recorded during rollout. It combines a cost model for attention, expert computation, and communication with greedy initialization, alternating pair-swap refinement, and a runtime that overlaps expert transfers with attention computation. Across the evaluated Qwen models and workloads on 32 H20 GPUs, the paper reports training-stage speedups of up to 1.93× over Megatron-LM and up to 1.10× over the stronger placement baselines.

Critical strengths
  1. The proposed approach addresses the computation and communication bottlenecks.
  2. Experiments across multiple models and workloads demonstrate consistent training speedups.
Critical weaknesses
  1. The methodological novelty could be clarified further. Routing replay and sample/expert placement have been explored in prior MoE RL systems, and the solver builds on established LPT and Kernighan–Lin heuristics. The main distinction appears to be their coordination under an objective covering attention, expert computation, and communication. Clarifying what is new in the formulation and solver relative to ReLibra and RoutePack would help establish the methodological contribution.

  2. The experimental evidence could be strengthened through closer system comparisons and broader operating conditions. The evaluation uses one 32-GPU H20 cluster with 8K/16K response settings, and does not directly compare against ForeMoE or RoutePack. Comparisons under a common runtime and sensitivity tests with shorter sequences or lower inter-node bandwidth would help establish the advantage over related systems and the conditions under which planning and expert-transfer overheads remain hidden.

Questions for authors
  1. Could you clarify the principal methodological distinction from ReLibra and RoutePack? In particular, which placement decisions or interactions are newly captured by the joint formulation, and which parts of the solver are specifically designed for this coupling rather than standard applications of LPT and Kernighan–Lin? An explanation of whether the main contribution lies in the formulation, solver adaptations, or runtime integration would help make the novelty more explicit.

  2. Could you provide comparisons with ForeMoE and RoutePack under a common runtime and hardware setup, or clarify which of their mechanisms are covered by the current baselines? In addition, how do the net speedups and exposed overheads change with shorter sequences or lower inter-node bandwidth? Measurements that include any unhidden planning and expert-transfer costs would help establish whether the gains remain consistent beyond the evaluated configurations and identify the practical operating range.

4 comments
AuthorsOpenReview ↗

Official Comment by Authors [1/2]

We sincerely thank you for your careful review and constructive feedback. Your comments have provided valuable guidance for improving our manuscript.

W1 & Q1. Core Contribution and Differences from Existing Methods

Core contribution. The core contribution of J-MoE is the joint optimization of sample and expert placement for MoE RL training. Problem formulation, solver adaptations, and runtime integration complement each other toward this goal: Problem formulation (§3.1) couples cross-microbatch sample placement and microbatch-level expert placement under a unified objective (Eq. 1) covering attention computation, expert computation, and All-to-All communication; Solver adaptations (§3.2) make this coupled problem tractable before each optimizer step; and Runtime integration (§3.3) executes the resulting plan without exposing overhead on the critical path by redistributing samples before each optimizer step and overlapping expert reconfiguration with attention computation (Fig. 7, Appendix C).

Comparison with existing methods. We have added Tab. 1 and revised §1 and §2.2 to clarify these differences:

MethodESJointAttn.ExpertComm.
ReLibra✔✔✘✘✔✔
ForeMoE✔✘✘✘✔✔
RoutePack✔✔✘✔✔✘
J-MoE✔✔✔✔✔✔

E/S: expert/sample placement; Joint: joint optimization of sample and expert placement. Cost awareness (Attn./Expert/Comm.) indicates inclusion in the optimization objective.

Specifically, ReLibra first fixes a batch-level expert placement and then swaps samples within microbatches, omitting attention cost; ForeMoE reconfigures experts at microbatch granularity while keeping sample placement fixed; and RoutePack first fixes a batch-level expert placement and then assigns samples based on attention and expert-computation workloads, omitting communication cost. None of the three jointly considers attention, expert, and communication costs, and all three solve placement in a one-pass, non-iterative manner without revisiting one placement after changing the other. Only J-MoE jointly and iteratively optimizes both placements under a unified three-component objective.

Differences from standard LPT and Kernighan–Lin. Standard LPT and KL optimize a single type of item under fixed costs. In our problem, however, sample and expert costs depend on each other. Without knowing the expert placement, one cannot evaluate the expert-computation bottleneck and communication cost incurred by a sample, and moving either placement changes the costs seen by the other. Accordingly, our solver (§3.2, Alg. 1, Appendix B) introduces two concrete adaptations:

  1. Stage 1 vs. standard LPT: Standard LPT assigns items using fixed weights (e.g., sequence length alone). In J-MoE, whenever we tentatively place a sample into a candidate microbatch, that microbatch's aggregate expert demand changes; we immediately run LPT on the updated demand to obtain a conditional expert placement, and score both placements together under FmF_m. Fig. 6(a) confirms the effectiveness of this adaptation: when followed by the same Stage 2 refinement, replacing sequence-length-only sample initialization (w/o Stage 1) with our coupled Stage 1 initialization improves training speedup from 1.21× / 1.26× to 1.25× / 1.30× (on Codeforces / Mixed, relative to Megatron-Balanced).
  2. Stage 2 vs. standard KL: Standard KL swaps items of a single type under static costs. Here, swapping samples changes attention loads and the source GPUs of routed tokens, whereas swapping experts changes expert loads and token destinations, so updating one placement alters the communication costs of the other. J-MoE therefore alternates Sample-KL and Expert-KL under the same objective FmF_m, allowing both placements to repeatedly adapt to each other. Fig. 6(a) likewise confirms this adaptation: starting from the same initial placement (Megatron-Balanced), objective, and search budget, alternating refinement (w/o Stage 1: 1.21× / 1.26×) consistently outperforms one-pass sequential optimization E→S (1.15× / 1.21×) and S→E (1.19× / 1.19×), demonstrating the necessity of alternating refinement.
AuthorsOpenReview ↗

Official Comment by Authors [2/2]

W2 & Q2. Comparisons with Related Systems and Overhead Overlap Analysis

Direct Comparisons with Related Systems

In the revised paper (§4), we have added ReLibra, ForeMoE, and RoutePack as baselines, and updated the baseline descriptions, end-to-end performance comparison (Fig. 5), and case-study analysis (Tab. 4).

End-to-end training performance (Fig. 5). J-MoE achieves the highest speedup across all evaluated configurations, with maximum speedups over each baseline summarized below:

BaselineMegatron-LMMegatron-BalancedEPLB+ReLibraForeMoEMBRRoutePack
J-MoE speedup1.93×1.32×1.10×1.13×1.07×1.10×1.08×

J-MoE outperforms the newly added ReLibra, ForeMoE, and RoutePack by up to 1.13×, 1.07×, and 1.08×, respectively. Specifically, ReLibra and RoutePack fix expert placement at batch granularity and cannot adapt to expert-demand fluctuations across microbatches; although ForeMoE reconfigures experts per microbatch, it keeps sample placement fixed and therefore cannot jointly coordinate traffic sources and microbatch expert demand.

Case study (Tab. 4). On Mixed with Qwen3 Instruct (8K, EP16), Tab. 4 reports the max-to-mean GPU load imbalance (1 = ideal) and total inter-node traffic:

MethodAttn. Imbal.Expert Imbal.Inter-node Traffic (GiB)
Megatron-LM2.7161.7722795.04
Megatron-Balanced1.0341.8152789.96
EPLB+1.0341.2592787.66
ReLibra1.0381.2492758.78
MBR1.1071.1022605.01
ForeMoE1.0341.1262761.86
RoutePack1.0431.2582787.36
J-MoE1.0151.0572238.28

ReLibra optimizes under a fixed batch-level expert placement, resulting in higher overall expert imbalance and limited communication reduction; RoutePack likewise fixes expert placement at batch granularity and omits communication cost from its objective, leading to higher expert imbalance and leaving inter-node traffic nearly unchanged; ForeMoE optimizes expert placement without adjusting sample placement, which limits its ability to reduce inter-node communication.

Overhead Overlap Analysis

Planning time. Fig. 7(a) shows that placement planning is overlapped and does not extend the training critical path. Because the planner operates on scalar sample lengths and aggregated routing demands, with fixed batch, model, parallelism, and search settings, planning time does not depend on sequence length, and planning runs only once per optimizer step, remaining lightweight for shorter sequences.

Expert-reconfiguration overlap. Fig. 7(b) and (c) show that forward parameter prefetch and backward gradient return are completely overlapped with attention computation in our evaluated configurations. In the revised Appendix C, we further removed the unnecessary conservative bound assumption in the original derivation and expanded the analysis of this overlap condition, showing that an effective bandwidth of BWa2a≥\mathrm{BW}_{\mathrm{a2a}} \ge 2.68 GB/s (Qwen3) and 3.11 / 5.73 GB/s (Qwen3.5 full-/linear-attention layers) suffices for complete overlap:

  • Under lower inter-node bandwidth: On our cluster, the profiled BWa2a\mathrm{BW}_{\mathrm{a2a}} is 22 GB/s, well above the requirement for complete overlap. Moreover, as inter-node bandwidth decreases (while still meeting the overlap threshold), per-layer All-to-All activation communication accounts for a larger share of step time, which further amplifies the speedup from J-MoE's inter-node traffic reduction.
  • Under shorter sequences: Conversely, at 22 GB/s, complete overlap requires a mean complete sequence length of only 2,258 (Qwen3) and 2,402 (Qwen3.5) tokens, meaning only 210 and 354 response tokens on top of a 2,048-token prompt. This condition is easily met in practice, as modern RL post-training operates well above these short-sequence thresholds: mathematical/STEM reasoning uses maximum response lengths up to 128K tokens (UloRL [1], MAI-Thinking-1 [2]), and agentic RL trains with contexts of 256K to 1M tokens (Nemotron-Cascade 2 [3], MiMo-V2.6 [4]), providing even larger attention windows to hide expert reconfiguration.

References

[1] Du et al. UloRL: An Ultra-Long Output Reinforcement Learning Approach for Advancing Large Language Models' Reasoning Abilities. 2025.

[2] The Microsoft AI Team. MAI-Thinking-1: Building a Hill-Climbing Machine. 2026.

[3] Yang et al. Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation. 2026.

[4] Xiaomi LLM-Core Team. MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement. 2026.

Thank you for the response. Most of my concerns are addressed. Given that my original score is positive, I will keep the score unchanged.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your detailed review and feedback. We’re pleased to have addressed your questions and appreciate your valuable insights.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.