- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper measures routing correlations during MoE pretraining and uses them to cut expert-parallel all-to-all traffic without changing routing decisions or parameters. On a 12-layer, 128-expert model (top-2 and top-6) trained for 2.1B tokens, it finds strong within-layer co-selection (64 of 8,128 expert pairs carry 42% of top-2 tokens, against 1.6% under independent routing) and cross-layer predictability (112 of 128 experts send at least 30% of their tokens to a single next-layer expert). Correlated expert placement permutes experts with a Kernighan–Lin search so that co-selected experts share a GPU, combined with a dispatcher that sends one row per (token, GPU). Token shuffling, for layouts where TP equals EP, predicts each token's next-layer owner and folds that permutation into the post-attention reduce-scatter. On MI300X nodes with Megatron-LM, all-to-all time falls by 1.16–2.63× and step time by up to 1.41× (top-6, EP16), and an appendix repeats the analysis for three load-balancing methods.
Critical strengths
- The routing characterization is careful and tied directly to the system decisions. Section 2.2, Table 1 and Section 4.1 compare co-selection with the independent-routing level, measure next-layer predictability, show that 1,024 tokens suffice to fit the tables (Pearson 0.93 and 0.98 with the full fit at 4,096 tokens), and show that tables from step 1000 are within 2–3 points of the final ones. Appendix A.4 repeats the measurements for a 10× stronger auxiliary loss and an auxiliary-loss-free router. Together these give a usable rule for how much routing data to record and how often to refit.
- Both mechanisms cost almost nothing in memory or communication. Placement is a per-layer permutation with exactly experts per GPU, so no replicas are needed, and token shuffling reuses the bytes that the tensor-parallel reduce-scatter and all-gather already move, with the planner hidden on a side stream (Figure 6). This makes the approach easy to combine with replication-based load balancing.
- Results are reported per configuration, including where the gains vanish. Top-2 step speedups on TP = EP layouts are 1.01–1.04×, and placement alone is 0.98× at TP16 EP16; Appendix A.3 explains both by the all-to-all share of the step. This makes the operating envelope of the method clear.
Critical weaknesses
- The closest prior work is not cited, and the main mechanisms overlap with it. Occult (ICML 2025) places co-activated experts on the same device and sends one copy of a token per device, which is the combination evaluated here as correlated placement with deduplication. Semantic Parallelism (ICLR 2026) clusters co-activated experts offline and reorders tokens toward their expert groups under attention tensor parallelism, NetMoE (ICLR 2025) moves training samples across devices to reduce all-to-all traffic, and HierMoE (INFOCOM 2026) combines deduplication with expert swaps in Megatron-LM training. None of these is discussed. The contribution relative to them appears to be the token-level shuffle inside the training-time reduce-scatter and the characterization across balancing methods; the paper should state this and compare against at least one prior placement method.
- The evidence comes from one small model at one checkpoint, but the conclusions are stated generally. All experiments use one 12-layer model (, 128 experts of width 1024) trained for 2,000 steps and evaluated at step 2000, with each timing taken as the median of 14 steps from a single run. Section 4 states that the expert correlation does not change with model size or routing, yet Appendix A.4 shows the deduplication rate falling from 48.8% to 35.9% across routers. Co-selection and next-layer hit rates can be measured offline from router traces of released MoE models (for example OLMoE, Qwen3-30B-A3B or DeepSeek-V2-Lite) at low cost; such measurements, a top-8 fine-grained configuration and repeated timing runs would support the generalization.
- The baseline omits deduplication, which inflates the headline speedups relative to current practice. Configuration B sends one row per (token, expert), although node- or rank-level deduplication is standard in DeepEP-style dispatchers, as the paper itself notes. Relative to deduplication alone (D), placement adds 1.02–1.10× in step time at top-2 and 1.07–1.32× at top-6 (Figure 9), and the abstract's 1.16–2.63× range mixes placement-only and shuffling results across layouts. Using D as the primary reference and moving the end-to-end results from Appendix A.3 into the main text would give a more accurate picture.
Questions for authors
- Clarification. How do correlated placement and token shuffling differ from Occult's collaboration-aware placement and from Semantic Parallelism's expert grouping and token scheduling, and how does the method compare with them on the same routing traces?
- Clarification. What within-layer co-selection and next-layer hit rates does a released production-scale MoE (top-8, 64–256 experts) show when measured offline from router traces?
- Clarification. How much do the 14-step medians vary across runs, and do the step-time gains hold over longer windows?
- Clarification. Is the loss curve unchanged, up to floating-point reordering, with token shuffling enabled? A short loss comparison would confirm that the residual permutation does not affect training.
4 comments
Response to Reviewer d43r (2/3)
Q2. Released production-scale checkpoints
Clarification. What within-layer co-selection and next-layer hit rates does a released production-scale MoE (top-8, 64–256 experts) show when measured offline from router traces?
Co-selection and next-layer hit on released checkpoints. We measured both quantities offline from the router traces of two released top-8 checkpoints with the Table I protocol (placement and affinity table fitted on 75% of the forward passes, scored on held-out passes, 120k test tokens; no training and no change to the models): Qwen3-30B-A3B (128 experts, 48 MoE layers, 0.8M fineweb-edu tokens) and GLM-5.3-Flash (288 experts, 42 MoE layers, 0.79M tokens). Within-layer co-selection is strong on both: when an expert is selected, its most frequent partner is selected with it 52% of the time on Qwen3 and 26% on GLM, against 5.5% and 2.4% if routing were random (25% against 5.5% for our top-8 model trained from scratch).
The table gives the within-layer dedup rate (fraction of a token's k expert rows that land on a GPU the token already sends to) under contiguous and co-activation placement, the resulting distinct GPUs per token, and the next-layer hit of the shuffle predictor (fraction of next-layer rows that are local to the GPU the shuffle picks; unbalanced and load-balanced owner assignment; chance = 1/EP):
| model | EP | dedup %, contiguous -> co-activation | distinct GPUs per token | next-layer hit %, unbalanced / balanced (chance) |
|---|---|---|---|---|
| Qwen3-30B-A3B | 8 | 33.2 -> 60.9 | 5.3 -> 3.1 | 51.4 / 23.2 (12.5) |
| 16 | 17.5 -> 47.9 | 6.6 -> 4.2 | 37.4 / 15.6 (6.2) | |
| 32 | 7.9 -> 33.9 | 7.4 -> 5.3 | 25.9 / 10.0 (3.1) | |
| GLM-5.3-Flash | 8 | 33.9 -> 50.7 | 5.3 -> 4.0 | 37.2 / 25.0 (12.5) |
| 16 | 18.6 -> 37.8 | 6.5 -> 5.0 | 27.6 / 17.6 (6.2) | |
| 32 | 9.3 -> 27.1 | 7.3 -> 5.8 | 20.5 / 12.4 (3.1) |
On both released checkpoints, correlated placement raises the dedup rate at EP8 from 33% to 61% (Qwen3) and from 34% to 51% (GLM), so a token reaches 3.1 and 4.0 GPUs instead of 5.3. This matches our Section 4 top-6 model (25% to 58%), so the within-layer structure and the payload reduction carry over to 128- and 288-expert production routers. The next-layer predictor hits 51% (Qwen3) and 37% (GLM) at EP8, against 12.5% for a random choice, so the cross-layer correlation is present as well.
Response to Reviewer d43r (3/3)
Q3. Variance of the 14-step medians and longer windows
Clarification. How much do the 14-step medians vary across runs, and do the step-time gains hold over longer windows?
14-step medians vs 200-iteration runs. We re-ran the Section 4 model (top-2 and top-6) from the same checkpoints for 200 iterations of 1.05M tokens each instead of 14, on the paper's layouts. The table gives the 200-iteration median (paper's 14-step number in parentheses); the last column is the largest deviation of any 50-iteration window's gain from the full-run gain:
| model, layout, nodes | plain, 200-step (paper 14-step) | dedup | dedup + placement | + shuffle | window drift |
|---|---|---|---|---|---|
| top-2 TP1 EP16, 2 nodes | 2359 (2338) | -3.1% (-3.1%) | -11.6% (-11.2%) | <= 0.1 pt | |
| top-6 TP1 EP16, 2 nodes | 4929 (4878) | -7.6% (-6.7%) | -29.4% (-29.0%) | <= 0.1 pt | |
| top-2 TP1 EP8, 1 node | 3258 (3249) | -2.9% (-3.0%) | -4.7% (-4.7%) | <= 0.1 pt | |
| top-2 TP8 EP8, 1 node | 5284 (5331) | -2.7% (-2.1%) | -2.3% (-2.0%) | -2.2% (-1.1%) | <= 0.9 pt |
| top-2 TP16 EP16, 2 nodes | 5128 (5253) | -1.8% (-4.6%) | -3.0% (-4.8%) | -3.7% (-7.4%) | <= 0.6 pt |
| top-6 TP1 EP8, 1 node | 5208 (5294) | -3.6% (-3.7%) | -11.9% (-12.5%) | <= 0.1 pt | |
| top-6 TP8 EP8, 1 node | 7354 (7278) | -3.5% (-3.1%) | -8.3% (-7.3%) | -7.7% (-7.0%) | <= 0.4 pt |
The 200-iteration gains match the 14-step numbers to within 1 point on every layout except top-2 TP16 EP16, where the full stack gives -3.7% instead of the paper's -7.4%; we will replace that entry with the 200-iteration value. Within each run the gain moves by at most 0.9 points between 50-iteration windows, and the largest gains (top-6 and top-2 TP1 EP16) are flat to 0.1 points. Across interleaved repeats of the 14-step measurement (3 to 5 rounds per arm), the step time varies by 0.1 to 1.1% and the gain by 0.6 to 1.6 points.
Why the gain holds. The gain depends on how many rows per token survive dedup under the placement, which is set by the routing correlations within and across layers, and these stop changing once the router has specialised. Top-6, EP16; placement and affinity table fitted at step s and scored on the routes of step 1999:
| fit at step (tokens seen) | placement dedup rate | next-layer predictor hit |
|---|---|---|
| 200 (0.2B) | 32.4% | 29.5% |
| 600 (0.6B) | 44.1% | 43.6% |
| 1000 (1.0B) | 46.7% | 46.8% |
| 1400 (1.5B) | 48.4% | 48.8% |
| 1800 (1.9B) | 48.4% | 49.3% |
| 1990 (2.1B, used in the paper) | 48.8% | 49.4% |
Tables fitted at 1.0B tokens are within 2 points of the end-of-run tables, and from 1.5B tokens within 0.5 points. A placement and predictor fitted once after that point therefore hold for the rest of the run; the online refresh in the paper covers the early-training regime.
Q4. Loss with token shuffling enabled
Clarification. Is the loss curve unchanged, up to floating-point reordering, with token shuffling enabled? A short loss comparison would confirm that the residual permutation does not affect training.
Loss and gradient norm with dedup and with shuffling enabled. Yes, both are unchanged up to floating-point reordering. Dedup and shuffling are exact rewrites of the same computation, so the only difference they can introduce is the order of bf16 sums. Same checkpoint, same data order, 300 iterations; differences are arm minus plain, per iteration:
| arm | lm loss, step 1 | grad norm, step 1 | max |loss diff| | mean loss diff | mean grad norm, 300 steps |
|---|---|---|---|---|---|
| TP8 EP8 plain | 4.610559 | 4.647 | 4.344 | ||
| plain, repeated | 4.610559 | 4.648 | 6.6e-3 | -1.7e-4 | 4.336 |
| dedup + placement | 4.610573 | 4.648 | 6.4e-3 | -1.1e-4 | 4.345 |
| dedup + placement + shuffle | 4.610621 | 4.647 | 6.5e-3 | +0.5e-4 | 4.334 |
| TP1 EP8 plain | 4.607288 | 4.633 | 4.339 | ||
| dedup | 4.607378 | 4.633 | 6.7e-3 | +0.2e-4 | 4.369 |
| dedup + placement | 4.607378 | 4.633 | 8.7e-3 | +0.5e-4 | 4.342 |
At step 1, before any optimizer step, the loss agrees to 9e-5 or better and the gradient norm to its printed precision (1e-3): this is the reduction-order effect alone. Later steps differ by up to 6.4e-3 to 8.7e-3 in loss because each optimizer step carries the rounding difference into the weights, and two runs of the unmodified code differ by the same amount (6.6e-3), since the fused attention backward is not deterministic in bf16. The mean loss differences are below 2e-4 and of both signs, the per-step loss curves correlate above 0.999, and the mean gradient norm of every arm is within 0.7% of plain (0.2% between the two plain runs), so neither dedup nor shuffling biases training. The 200-iteration replication above also ends at the same loss for every arm (4.6359 to 4.6361 at top-2 TP1 EP8, 4.4397 to 4.4401 at top-6).
Thank you for the thorough response. The measurements on released checkpoints, the 200-iteration reruns, and the comparison with Occult, HierMoE and NetMoE on the same traces address my concerns about evidence and positioning. I have raised my rating to 3.
Response to Reviewer d43r (1/3)
We thank the reviewer for the careful reading.
Q1. Relation to prior work
Occult and Semantic Parallelism. We will cite and discuss these works. All three placements build on pairwise expert co-activation. Occult's Algorithm 1, run on our routing traces, removes the same share of non-local dispatch rows as ours, within 1.4 points (top-2 EP16 / EP32: -20.8 / -16.6% vs. -20.6 / -17.3%; top-6: -40.5 / -30.7% vs. -41.9 / -31.8%). Placement is thus a shared building block; the differences are in the setting and in what is built around it:
NetMoE and HierMoE. NetMoE notes that sample placement does not help when a sample's tokens are balanced over experts; on our traces, moving whole sequences removes only 1.4% of inter-node rows. HierMoE's dedup-aware expert swap, implemented on the same captures, removes 3.6% (top-2) and 15.3% (top-6) of non-local rows against 19.8% and 40.2% for ours.
In summary, our contribution is the training-time methodology around the placement, with the model unchanged: the dedup dispatcher in Megatron-LM, the online placement refresh, and the inter-layer token shuffle inside the training tensor-parallel collectives.