acceptodds
Under review as a conference paper at ICLR 2027

Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling

Abstract

Mixture-of-Experts (MoE) layers replace the feed-forward block of a Transformer with expert networks, and each token is routed to of these experts. Under expert parallelism (EP) the experts are distributed across GPUs, and every MoE layer runs all-to-all collectives in the forward and backward passes to dispatch tokens to their experts and then combine the results. On nodes of eight MI300X GPUs, these collectives can take 45 % of the training step at EP32 with top-2 routing and 60 % with top-6 routing. We find that early in pretraining routers have already learned to assign tokens to experts in correlated patterns, both within a layer and across layers. At top-2, 0.8 % of the expert pairs in a layer are selected together by 45 % of tokens, and the experts a token selects at one layer predict the experts it selects at the next layer. We use these correlations to keep more token–expert assignments on the token's own GPU, which reduces communication across GPUs and across nodes. Correlated expert placement puts experts that are often selected together on the same GPU. Combined with a dispatcher that sends each token to each GPU once, it removes up to 58 % of dispatched rows. Token shuffling applies when sequence parallelism shards tokens across the EP group. It moves each token to the GPU predicted to hold its next-layer experts during the reduce-scatter that follows attention. On one node this raises the share of token–expert assignments served on the token's GPU from 12.5 % to 59 %. In Megatron-LM, across EP degrees from 8 to 64 with top-2 and top-6 routing, the two methods reduce all-to-all time by 1.16–2.63 and end-to-end step time by up to 1.41. Neither method changes the models' underlying routing decisions or expert parameters.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture of Experts, All-to-All communication, Distributed training, Expert Correlation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_d43rOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper measures routing correlations during MoE pretraining and uses them to cut expert-parallel all-to-all traffic without changing routing decisions or parameters. On a 12-layer, 128-expert model (top-2 and top-6) trained for 2.1B tokens, it finds strong within-layer co-selection (64 of 8,128 expert pairs carry 42% of top-2 tokens, against 1.6% under independent routing) and cross-layer predictability (112 of 128 experts send at least 30% of their tokens to a single next-layer expert). Correlated expert placement permutes experts with a Kernighan–Lin search so that co-selected experts share a GPU, combined with a dispatcher that sends one row per (token, GPU). Token shuffling, for layouts where TP equals EP, predicts each token's next-layer owner and folds that permutation into the post-attention reduce-scatter. On MI300X nodes with Megatron-LM, all-to-all time falls by 1.16–2.63× and step time by up to 1.41× (top-6, EP16), and an appendix repeats the analysis for three load-balancing methods.

Critical strengths
  • The routing characterization is careful and tied directly to the system decisions. Section 2.2, Table 1 and Section 4.1 compare co-selection with the independent-routing level, measure next-layer predictability, show that 1,024 tokens suffice to fit the tables (Pearson 0.93 and 0.98 with the full fit at 4,096 tokens), and show that tables from step 1000 are within 2–3 points of the final ones. Appendix A.4 repeats the measurements for a 10× stronger auxiliary loss and an auxiliary-loss-free router. Together these give a usable rule for how much routing data to record and how often to refit.
  • Both mechanisms cost almost nothing in memory or communication. Placement is a per-layer permutation with exactly E/EPE/\text{EP} experts per GPU, so no replicas are needed, and token shuffling reuses the bytes that the tensor-parallel reduce-scatter and all-gather already move, with the planner hidden on a side stream (Figure 6). This makes the approach easy to combine with replication-based load balancing.
  • Results are reported per configuration, including where the gains vanish. Top-2 step speedups on TP = EP layouts are 1.01–1.04×, and placement alone is 0.98× at TP16 EP16; Appendix A.3 explains both by the all-to-all share of the step. This makes the operating envelope of the method clear.
Critical weaknesses
  1. The closest prior work is not cited, and the main mechanisms overlap with it. Occult (ICML 2025) places co-activated experts on the same device and sends one copy of a token per device, which is the combination evaluated here as correlated placement with deduplication. Semantic Parallelism (ICLR 2026) clusters co-activated experts offline and reorders tokens toward their expert groups under attention tensor parallelism, NetMoE (ICLR 2025) moves training samples across devices to reduce all-to-all traffic, and HierMoE (INFOCOM 2026) combines deduplication with expert swaps in Megatron-LM training. None of these is discussed. The contribution relative to them appears to be the token-level shuffle inside the training-time reduce-scatter and the characterization across balancing methods; the paper should state this and compare against at least one prior placement method.
  2. The evidence comes from one small model at one checkpoint, but the conclusions are stated generally. All experiments use one 12-layer model (dmodel=4096d_{\text{model}} = 4096, 128 experts of width 1024) trained for 2,000 steps and evaluated at step 2000, with each timing taken as the median of 14 steps from a single run. Section 4 states that the expert correlation does not change with model size or routing, yet Appendix A.4 shows the deduplication rate falling from 48.8% to 35.9% across routers. Co-selection and next-layer hit rates can be measured offline from router traces of released MoE models (for example OLMoE, Qwen3-30B-A3B or DeepSeek-V2-Lite) at low cost; such measurements, a top-8 fine-grained configuration and repeated timing runs would support the generalization.
  3. The baseline omits deduplication, which inflates the headline speedups relative to current practice. Configuration B sends one row per (token, expert), although node- or rank-level deduplication is standard in DeepEP-style dispatchers, as the paper itself notes. Relative to deduplication alone (D), placement adds 1.02–1.10× in step time at top-2 and 1.07–1.32× at top-6 (Figure 9), and the abstract's 1.16–2.63× range mixes placement-only and shuffling results across layouts. Using D as the primary reference and moving the end-to-end results from Appendix A.3 into the main text would give a more accurate picture.
Questions for authors
  1. Clarification. How do correlated placement and token shuffling differ from Occult's collaboration-aware placement and from Semantic Parallelism's expert grouping and token scheduling, and how does the method compare with them on the same routing traces?
  2. Clarification. What within-layer co-selection and next-layer hit rates does a released production-scale MoE (top-8, 64–256 experts) show when measured offline from router traces?
  3. Clarification. How much do the 14-step medians vary across runs, and do the step-time gains hold over longer windows?
  4. Clarification. Is the loss curve unchanged, up to floating-point reordering, with token shuffling enabled? A short loss comparison would confirm that the residual permutation does not affect training.
4 comments
AuthorsOpenReview ↗

Response to Reviewer d43r (1/3)

We thank the reviewer for the careful reading.

Q1. Relation to prior work

Clarification. How do correlated placement and token shuffling differ from Occult's collaboration-aware placement and from Semantic Parallelism's expert grouping and token scheduling, and how does the method compare with them on the same routing traces?

Occult and Semantic Parallelism. We will cite and discuss these works. All three placements build on pairwise expert co-activation. Occult's Algorithm 1, run on our routing traces, removes the same share of non-local dispatch rows as ours, within 1.4 points (top-2 EP16 / EP32: -20.8 / -16.6% vs. -20.6 / -17.3%; top-6: -40.5 / -30.7% vs. -41.9 / -31.8%). Placement is thus a shared building block; the differences are in the setting and in what is built around it:

OccultSemantic ParallelismOurs
Settingfine-tuning + inferenceinferencepretraining from scratch
Model unchangedonly without pruning (headline speedups use pruning + fine-tuning)yesyes
Placementoffline, fixedoffline, periodicrefitted online
Token movenoneinside the inference reduce-scatterinside the training SP reduce-scatter, with backward pass
Predictor-per-token-id tableinter-layer expert affinity
Megatron-LM implementationnonoyes, to be released

NetMoE and HierMoE. NetMoE notes that sample placement does not help when a sample's tokens are balanced over experts; on our traces, moving whole sequences removes only 1.4% of inter-node rows. HierMoE's dedup-aware expert swap, implemented on the same captures, removes 3.6% (top-2) and 15.3% (top-6) of non-local rows against 19.8% and 40.2% for ours.

In summary, our contribution is the training-time methodology around the placement, with the model unchanged: the dedup dispatcher in Megatron-LM, the online placement refresh, and the inter-layer token shuffle inside the training tensor-parallel collectives.

AuthorsOpenReview ↗

Response to Reviewer d43r (2/3)

Q2. Released production-scale checkpoints

Clarification. What within-layer co-selection and next-layer hit rates does a released production-scale MoE (top-8, 64–256 experts) show when measured offline from router traces?

Co-selection and next-layer hit on released checkpoints. We measured both quantities offline from the router traces of two released top-8 checkpoints with the Table I protocol (placement and affinity table fitted on 75% of the forward passes, scored on held-out passes, 120k test tokens; no training and no change to the models): Qwen3-30B-A3B (128 experts, 48 MoE layers, 0.8M fineweb-edu tokens) and GLM-5.3-Flash (288 experts, 42 MoE layers, 0.79M tokens). Within-layer co-selection is strong on both: when an expert is selected, its most frequent partner is selected with it 52% of the time on Qwen3 and 26% on GLM, against 5.5% and 2.4% if routing were random (25% against 5.5% for our top-8 model trained from scratch).

The table gives the within-layer dedup rate (fraction of a token's k expert rows that land on a GPU the token already sends to) under contiguous and co-activation placement, the resulting distinct GPUs per token, and the next-layer hit of the shuffle predictor (fraction of next-layer rows that are local to the GPU the shuffle picks; unbalanced and load-balanced owner assignment; chance = 1/EP):

modelEPdedup %, contiguous -> co-activationdistinct GPUs per tokennext-layer hit %, unbalanced / balanced (chance)
Qwen3-30B-A3B833.2 -> 60.95.3 -> 3.151.4 / 23.2 (12.5)
1617.5 -> 47.96.6 -> 4.237.4 / 15.6 (6.2)
327.9 -> 33.97.4 -> 5.325.9 / 10.0 (3.1)
GLM-5.3-Flash833.9 -> 50.75.3 -> 4.037.2 / 25.0 (12.5)
1618.6 -> 37.86.5 -> 5.027.6 / 17.6 (6.2)
329.3 -> 27.17.3 -> 5.820.5 / 12.4 (3.1)

On both released checkpoints, correlated placement raises the dedup rate at EP8 from 33% to 61% (Qwen3) and from 34% to 51% (GLM), so a token reaches 3.1 and 4.0 GPUs instead of 5.3. This matches our Section 4 top-6 model (25% to 58%), so the within-layer structure and the payload reduction carry over to 128- and 288-expert production routers. The next-layer predictor hits 51% (Qwen3) and 37% (GLM) at EP8, against 12.5% for a random choice, so the cross-layer correlation is present as well.

AuthorsOpenReview ↗

Response to Reviewer d43r (3/3)

Q3. Variance of the 14-step medians and longer windows

Clarification. How much do the 14-step medians vary across runs, and do the step-time gains hold over longer windows?

14-step medians vs 200-iteration runs. We re-ran the Section 4 model (top-2 and top-6) from the same checkpoints for 200 iterations of 1.05M tokens each instead of 14, on the paper's layouts. The table gives the 200-iteration median (paper's 14-step number in parentheses); the last column is the largest deviation of any 50-iteration window's gain from the full-run gain:

model, layout, nodesplain, 200-step (paper 14-step)dedupdedup + placement+ shufflewindow drift
top-2 TP1 EP16, 2 nodes2359 (2338)-3.1% (-3.1%)-11.6% (-11.2%)<= 0.1 pt
top-6 TP1 EP16, 2 nodes4929 (4878)-7.6% (-6.7%)-29.4% (-29.0%)<= 0.1 pt
top-2 TP1 EP8, 1 node3258 (3249)-2.9% (-3.0%)-4.7% (-4.7%)<= 0.1 pt
top-2 TP8 EP8, 1 node5284 (5331)-2.7% (-2.1%)-2.3% (-2.0%)-2.2% (-1.1%)<= 0.9 pt
top-2 TP16 EP16, 2 nodes5128 (5253)-1.8% (-4.6%)-3.0% (-4.8%)-3.7% (-7.4%)<= 0.6 pt
top-6 TP1 EP8, 1 node5208 (5294)-3.6% (-3.7%)-11.9% (-12.5%)<= 0.1 pt
top-6 TP8 EP8, 1 node7354 (7278)-3.5% (-3.1%)-8.3% (-7.3%)-7.7% (-7.0%)<= 0.4 pt

The 200-iteration gains match the 14-step numbers to within 1 point on every layout except top-2 TP16 EP16, where the full stack gives -3.7% instead of the paper's -7.4%; we will replace that entry with the 200-iteration value. Within each run the gain moves by at most 0.9 points between 50-iteration windows, and the largest gains (top-6 and top-2 TP1 EP16) are flat to 0.1 points. Across interleaved repeats of the 14-step measurement (3 to 5 rounds per arm), the step time varies by 0.1 to 1.1% and the gain by 0.6 to 1.6 points.

Why the gain holds. The gain depends on how many rows per token survive dedup under the placement, which is set by the routing correlations within and across layers, and these stop changing once the router has specialised. Top-6, EP16; placement and affinity table fitted at step s and scored on the routes of step 1999:

fit at step (tokens seen)placement dedup ratenext-layer predictor hit
200 (0.2B)32.4%29.5%
600 (0.6B)44.1%43.6%
1000 (1.0B)46.7%46.8%
1400 (1.5B)48.4%48.8%
1800 (1.9B)48.4%49.3%
1990 (2.1B, used in the paper)48.8%49.4%

Tables fitted at 1.0B tokens are within 2 points of the end-of-run tables, and from 1.5B tokens within 0.5 points. A placement and predictor fitted once after that point therefore hold for the rest of the run; the online refresh in the paper covers the early-training regime.


Q4. Loss with token shuffling enabled

Clarification. Is the loss curve unchanged, up to floating-point reordering, with token shuffling enabled? A short loss comparison would confirm that the residual permutation does not affect training.

Loss and gradient norm with dedup and with shuffling enabled. Yes, both are unchanged up to floating-point reordering. Dedup and shuffling are exact rewrites of the same computation, so the only difference they can introduce is the order of bf16 sums. Same checkpoint, same data order, 300 iterations; differences are arm minus plain, per iteration:

armlm loss, step 1grad norm, step 1max |loss diff|mean loss diffmean grad norm, 300 steps
TP8 EP8 plain4.6105594.6474.344
plain, repeated4.6105594.6486.6e-3-1.7e-44.336
dedup + placement4.6105734.6486.4e-3-1.1e-44.345
dedup + placement + shuffle4.6106214.6476.5e-3+0.5e-44.334
TP1 EP8 plain4.6072884.6334.339
dedup4.6073784.6336.7e-3+0.2e-44.369
dedup + placement4.6073784.6338.7e-3+0.5e-44.342

At step 1, before any optimizer step, the loss agrees to 9e-5 or better and the gradient norm to its printed precision (1e-3): this is the reduction-order effect alone. Later steps differ by up to 6.4e-3 to 8.7e-3 in loss because each optimizer step carries the rounding difference into the weights, and two runs of the unmodified code differ by the same amount (6.6e-3), since the fused attention backward is not deterministic in bf16. The mean loss differences are below 2e-4 and of both signs, the per-step loss curves correlate above 0.999, and the mean gradient norm of every arm is within 0.7% of plain (0.2% between the two plain runs), so neither dedup nor shuffling biases training. The 200-iteration replication above also ends at the same loss for every arm (4.6359 to 4.6361 at top-2 TP1 EP8, 4.4397 to 4.4401 at top-6).

Thank you for the thorough response. The measurements on released checkpoints, the 200-iteration reruns, and the comparison with Occult, HierMoE and NetMoE on the same traces address my concerns about evidence and positioning. I have raised my rating to 3.

Reviewer_JzLHOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a pre-training system for MoE models that reduce the overhead of token shuffling. The motivation is that a large fraction of time is spent on token shuffling in MoE because all-to-all communication is expensive. The paper notes that expert coupling occurs across experts where tokens often go to the same expert pairs. So by placing them together on the GPU it can reduce the communication. In addition, the paper also notes that experts selected in the previous layer serves as a good indication for the next layer. So the system predicts the expert needed in the next layer and overlaps communication and computation. The paper also shows optimization that piggybacks token communication with SP and even applies kernel-level fusion to further reduce the communication overhead.

Critical strengths

The idea of expert coupling is very interesting. The results are good and the implemetation details are also interesting.

Piggybacking on existing SP collectives is really interesting and cute. The paper actually have shown this can be done through implementation. The paper also shows the details of how this done, such as the fused kernel. I think the paper can highlight the challenges upfront and then show the solution. Currently, it is written in a plain way just show what you did.

Critical weaknesses
  1. Fraction of time spent on communication.

Doesn’t this depend on the GPU-to-GPU bandwidth? I would like to see whether the similar pattern holds for NVIDIA GPUs, where the GPU network bandwidth tend to be larger (e.g., 400G/800G InfiniBand) and the network scales better with NVSwitch. I think when k=6 pairwise affinity becomes less effective because of the probability. As k increases there is combinatorial explosion of expert pair selection. But your data shows otherwise. Can you explain why this is not the case? Please provide a deeper analysis, reasoning, or intuition as to why this is more effective when k is 6. What happens when k gets larger like 8 or 16? Justify the choice of 2 and 6.

2 Assumption on Expert-to-GPU Ratio (E/EP >= 2):

  • The proposed correlated placement and deduplicating dispatcher fundamentally rely on the premise that each GPU hosts multiple experts (E/EP >= 2). However, when scaling to large GPU clusters where the number of GPUs exceeds the number of experts, the opportunity to co-locate multiple co-selected experts on a single device disappears. Frameworks like Tutel employ intra-expert parallelism via TP in this scenario.

  • The authors should clarify the applicability and boundaries of this technique when E/EP <= 1, as the benefits of deduplication and placement would diminish entirely under such configurations.

Questions for authors

Thank you for submitting your work to ICLR. The idea of expert coupling is interesting.

  1. Fraction of time spent on communication.

Doesn’t this depend on the GPU-to-GPU bandwidth? I would like to see whether the similar pattern holds for NVIDIA GPUs, where the GPU network bandwidth tend to be larger (e.g., 400G/800G InfiniBand) and the network scales better with NVSwitch. I think when k=6 pairwise affinity becomes less effective because of the probability. As k increases there is combinatorial explosion of expert pair selection. But your data shows otherwise. Can you explain why this is not the case? Please provide a deeper analysis, reasoning, or intuition as to why this is more effective when k is 6. What happens when k gets larger like 32 or 64?

2 Assumption on Expert-to-GPU Ratio (E/EP >= 2):

  • The proposed correlated placement and deduplicating dispatcher fundamentally rely on the premise that each GPU hosts multiple experts (E/EP >= 2). However, when scaling to large GPU clusters where the number of GPUs exceeds the number of experts, the opportunity to co-locate multiple co-selected experts on a single device disappears. Frameworks like Tutel employ intra-expert parallelism via TP in this scenario.

  • The authors should clarify the applicability and boundaries of this technique when E/EP <= 1, as the benefits of deduplication and placement would diminish entirely under such configurations.

  1. The two points above are all related to when your system is effective. You need to clarify them and discuss whether there are any other factors that make the system more/less effective.

  2. Piggybacking on existing SP collectives. I think this is really interesting and cute. The paper actually have shown this can be done through implementation. The paper also shows the details of how this done, such as the fused kernel. I think the paper can highlight the challenges upfront and then show the solution. Currently, it is written in a plain way just show what you did.

  3. I don't think figure 2 is a good way to show this. The dots are too small and blue are scattered all over. Please think of a way to represent the idea of expert coupling in a more intuitive way.

4 comments
AuthorsOpenReview ↗

Response to Reviewer JzLH (1/3)

We thank the reviewer for the careful reading.

Q1. Fraction of time spent on communication

Fraction of time spent on communication. Doesn't this depend on the GPU-to-GPU bandwidth? I would like to see whether the similar pattern holds for NVIDIA GPUs, where the GPU network bandwidth tend to be larger (e.g., 400G/800G InfiniBand) and the network scales better with NVSwitch.

Communication fraction on NVIDIA hardware. We agree the all-to-all share depends on the network-to-compute ratio, so we repeated Section 4.2 on NVIDIA DGX H100 nodes (8 H100 SXM per node on NVSwitch, 8 × 400 Gb/s NDR InfiniBand per node, 4× the per-NIC bandwidth of our MI300X cluster) with the same model as Section 4 (12 layers, d_model 4096, 128 experts of FFN 1024, top-6, sequence-level auxiliary loss 1e-2), the same Megatron-LM implementation and the same measurement protocol, at EP8 to EP32 (1 to 4 nodes).

EPA2A share, MI300X / H100Dedup alone (step / A2A / bytes), MI300XDedup alone, H100Dedup + placement, MI300XDedup + placement, H100
824% / 21%1.04 / 1.24 / 1.331.03 / 1.49 / 1.331.14 / 1.58 / 2.361.05 / 1.60 / 1.79
1660% / 48%1.07 / 1.19 / 1.141.03 / 1.15 / 1.141.41 / 1.95 / 1.951.12 / 1.35 / 1.50
3260% / 58%1.04 / 1.13 / 1.061.00 / 1.06 / 1.051.25 / 1.48 / 1.541.08 / 1.22 / 1.29

The all-to-all share is essentially the same on H100 (21 to 58%) as on MI300X (24 to 60%) over EP8 to EP32 and grows with EP in the same way: the faster fabric is matched by faster compute, and once the EP group spans nodes the inter-node all-to-all dominates on both. Deduplication alone removes the same fraction of traffic on both platforms (1.33×, 1.14×, 1.05× at EP8 to EP32), because it depends only on how often a token's experts share a rank under contiguous placement. Its kernels are the same Triton code on both vendors; on H100 a profile shows they add 77 ms per step while replacing 85 ms of stock permutation kernels, for a net compute cost of 1 to 3% of the step, consistent with the unchanged MoE-compute bar in Figure 9. Deduplication with correlated placement reduces the all-to-all time and the step time on both platforms (step speedup 1.05 to 1.12× on H100, 1.14 to 1.41× on MI300X). The step speedup on both platforms is the all-to-all share times the all-to-all reduction, with non-MoE time unchanged.

AuthorsOpenReview ↗

Response to Reviewer JzLH (2/3)

Q2. Effect of k

I think when k=6 pairwise affinity becomes less effective because of the probability. As k increases there is combinatorial explosion of expert pair selection. But your data shows otherwise. Can you explain why this is not the case? Please provide a deeper analysis, reasoning, or intuition as to why this is more effective when k is 6. What happens when k gets larger like 32 or 64?

What happens when k gets larger like 8 or 16? Justify the choice of 2 and 6.

The combinatorial argument is correct for whole expert sets, and our data confirms it: the share of tokens whose entire set lands on one GPU falls from 52.9% at top-2 to 18.0% at top-6 at EP8, and to 0% at EP32 (first row of the table below). Our method does not rely on that quantity. It relies on pairs, for three reasons:

  1. The saving is per pair. Dedup removes a row whenever two of a token's experts share a rank. A top-2 token has one pair and can save at most 1 of its 2 rows; a top-6 token has C(6,2) = 15 pairs and can save up to 5 of 6. This is why the contiguous placement, which ignores routing, already deduplicates 25.2% of top-6 slots against 5.9% of top-2 slots.
  2. The planner's input does not grow with k. It partitions the 128×128 expert co-activation matrix (8,128 pairs); the number of possible k-sets never enters. Each top-6 token contributes 15 pair observations per layer where a top-2 token contributes one, so the same captured data samples the matrix better at top-6.
  3. The trained router concentrates pair mass. Experts specialise and co-fire, so the matrix is far from uniform, and placement only has to keep the heavy pairs on one rank. Placement therefore removes 2.0 rows per token at top-6 against 0.4 at top-2 (EP8), and top-6 still reaches 35.1% dedup at EP32. The structure is stable: tables fit at step 1000 are within 2 to 3 points of the final ones, and from step 1400 the fitted and final co-selection matrices correlate above 0.93.

What erodes the pairwise signal is a flatter router, not k: our three top-6 routers give 57.7 / 51.1 / 46.7% dedup at EP8. The combinatorial penalty does apply to whole-set objectives such as a per-token home GPU, which we report separately.

Choice of 2 and 6; k = 8 or 16. Top-2 and top-6 bracket the range used by sparse-routing MoE models (top-2 in Mixtral, top-6 in DeepSeek-V2), and top-2 is the hardest case for our method since a token has a single pair. For top-8 we have measurements on released checkpoints (128-expert Qwen3-30B-A3B, EP8): contiguous placement deduplicates 33.2% of slots and co-activation placement 60.9%, continuing the trend from top-2 and top-6. The counting estimate below gives 33% for k = 8 and 55% for k = 16 before any placement.

k = 32 or 64. We did not train such models, so the following is a counting estimate, not a measurement. A token can occupy at most one row per rank, so with R ranks dedup removes at least 1 − R/k of the slots under any placement. With 128 experts placed without regard to routing, the expected dedup is 75% (k = 32) and 87.5% (k = 64) at EP8, and 31% and 53% at EP32; the same formula gives 5.9% and 25.5% for top-2 and top-6 at EP8, matching the measured contiguous values. Dedup therefore becomes more effective as k grows, while placement has less left to add: once a token selects a quarter or half of all experts it reaches nearly every rank under any placement, and the exchange approaches one row per token per rank, as for a dense layer.

top-2top-6
Tokens with all experts on one GPU, placed (EP8 / 16 / 32)52.9 / 45.3 / 35.8%18.0 / 8.3 / 0.0%
Dedup'd slots, contiguous → placed (EP8)5.9 → 26.4%25.2 → 57.7%
Dedup'd slots, contiguous → placed (EP16 / EP32)2.7 → 22.7 / 1.1 → 17.9%12.7 → 48.6 / 5.5 → 35.1%
AuthorsOpenReview ↗

Response to Reviewer JzLH (3/3)

Q3. Assumption on Expert-to-GPU Ratio (E/EP >= 2)

2 Assumption on Expert-to-GPU Ratio (E/EP >= 2):

The proposed correlated placement and deduplicating dispatcher fundamentally rely on the premise that each GPU hosts multiple experts (E/EP >= 2). However, when scaling to large GPU clusters where the number of GPUs exceeds the number of experts, the opportunity to co-locate multiple co-selected experts on a single device disappears. Frameworks like Tutel employ intra-expert parallelism via TP in this scenario.

The authors should clarify the applicability and boundaries of this technique when E/EP <= 1, as the benefits of deduplication and placement would diminish entirely under such configurations.

The two points above are all related to when your system is effective. You need to clarify them and discuss whether there are any other factors that make the system more/less effective.

Thank you for the note. We confirm that deduplication and correlated placement require EP > 1 (otherwise there is no all-to-all) and at least two experts per expert-parallel rank, E/EP ≥ 2, so that two of a token's experts can share a destination. This is the usual configuration in large-scale pretraining (e.g. DeepSeek-V3: 256 experts at EP64, 4 per GPU; our runs: 128 experts at EP8 to EP64, 16 to 2 per GPU), and our EP64 results at E/EP = 2 still show a 10 to 18% step-time gain. Our method targets the EP + DP regime, where each expert lives whole on one rank and the all-to-all carries token rows; it is the row count per token that we reduce. When the GPU count exceeds E and experts are sharded across devices with intra-expert model parallelism (EP + MP, as in Tutel), the exchange is organised around weight shards rather than token-to-expert rows, every shard is a separate destination, and neither technique applies. We will state these boundaries explicitly in the paper.


Q4. Piggybacking on existing SP collectives

Piggybacking on existing SP collectives. I think this is really interesting and cute. The paper actually have shown this can be done through implementation. The paper also shows the details of how this done, such as the fused kernel. I think the paper can highlight the challenges upfront and then show the solution. Currently, it is written in a plain way just show what you did.

Thank you, we agree. We will restructure the section to state three challenges upfront, each followed by its solution:

  1. The token owner must be decided before the router output exists, solved by the inter-layer affinity predictor, which runs hidden behind the preceding attention layer.
  2. A naive token move adds a second collective and a fully exposed return hop, solved by piggybacking on the existing sequence-parallel reduce-scatter and all-gather.
  3. Gathered rows must arrive in the order the attention kernels expect, solved by the fused assignment kernel.

Q5. Figure 2

I don't think figure 2 is a good way to show this. The dots are too small and blue are scattered all over. Please think of a way to represent the idea of expert coupling in a more intuitive way.

Thanks for the note; Figure 2 is revised with a magnified inset of the densest 16x16 window in each panel, every pair above 0.3 outlined and labelled.

Thank you for taking my feedback.

Reviewer_ZuJeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper describes a method of improving MoE training efficiency by reducing communication overhead in token routing, by detecting patterns in token-expert pairs and using this to place correlated experts together and shuffle tokens accordingly to minimize cross-shard token movement in expert parallel training.

Critical strengths

The paper has solid speed improvements (improving end-to-end training time by up to 1.41x). It also has very clear descriptions of the motivating problems and analysis of token/expert correlation.

Critical weaknesses

The abstract is confusing and lacks the clarity of the main paper. Re-reading it after reading the paper it becomes clearer, but as a first introduction to the paper coming in blind, it needs work.

Questions for authors

n/a

1 comment

The abstract is confusing and lacks the clarity of the main paper. Re-reading it after reading the paper it becomes clearer, but as a first introduction to the paper coming in blind, it needs work.

Thank you for the assessment. We will revise the abstract to improve its clarity and avoid any potential confusion. If anything specific was unclear, we would welcome the reviewer's pointers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.