- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MegaFlux, which accelerates multi-GPU Mixture-of-Experts LLM training and inference by dynamically creating replicas for busy experts on less busy GPUs and balancing token workloads. It comprises a planner that selects replica placements and token block distribution, and an executor that pipelines weight transfers and gradient reductions with expert computation.
Critical strengths
This paper identifies and addresses a meaningful problem. It provides a way to execute MoE LLMs on multi-GPU platforms more efficiently, and validated the latency reductions through experiments.
Critical weaknesses
The practical significance of the backward mechanism is not verified in a complete training process. This paper's end-to-end experiment covers inference prefill, but end-to-end training is not reported. Additionally, UltraEP documented that training integration overlaps reduction with router and attention backward, but the isolated MoE-layer comparison excluded it. This limits how strongly MegaFlux's advantage in training is established.
Questions for authors
Section 4.4 measures UltraEP's performance within the MoE layer, so the non-MoE overlap is excluded. Would the comparison change when UltraEP's asynchronous reduction overlaps router and attention backward?
-
How is backward latency's sensitivity to λ? Can you include results with reduction-aware pruning disabled?
-
How much skew would remain in practice? The 7.3–41.0% additional latency was measured from the P95 routes of OLMoE and Qwen3 workloads, while modern MoE models like DeepSeek-V3 employs auxiliary-loss-free balancing, and may result in less skew. How would such models benefit from MegaFlux?
-
How general is this approach? It seems to be tied to Blackwell features and a single NVLink domain. Can some parts transfer to other megakernels like DeepGEMM Mega MoE? And would some parts break on other scenarios like cross-node EP, where weight transfer is more expensive?
-
Compared with approaches that prefetch ahead of time, like PROBE, what are the advantages and disadvantages of planning from the current routing, especially when the token count is small?
5 comments
2. Sensitivity to λ and experiment with reduction-aware pruning disabled (Q1)
We thank the reviewer for this suggestion. We provide additional experiments on 8×B200 (EP8, E256, top-8) to evaluate the sensitivity of execution latency and replica placement to λ, as well as the impact of disabling reduction-aware pruning.
(a) Backward execution latency sensitivity to λ. We sweep λ ∈ {0, 4, 8, 12, 16, 20, 32, 48} on real-high routes at 1K/4K/8K/16K tokens/GPU, with 8 spare experts per GPU. The table reports the geometric-mean and maximum latency ratios (compared to λ=12, lower is better) across these four evaluated configurations.
| λ | Geomean ratio | Worst-case ratio |
|---|---|---|
| 0 | 1.0256 | 1.1053 |
| 4 | 1.0332 | 1.1258 |
| 8 | 0.9980 | 1.0033 |
| 12 | 1.0000 | 1.0000 |
| 16 | 1.0019 | 1.0097 |
| 20 | 1.0095 | 1.0241 |
| 32 | 1.0143 | 1.0252 |
| 48 | 1.0220 | 1.0552 |
Backward latency is stable around λ=12: λ=8–16 changes geometric-mean latency by at most 0.20%, whereas λ=0 increases average latency by 2.56%, with a 10.53% worst-case regression at 1K. Larger coefficients provide no consistent benefit: λ=32 and 48 increase average latency by 1.43% and 2.20%, respectively. These results indicate a relatively flat operating region near λ=12.
We also evaluate forward plus backward (F+B) combined latency. Relative to λ=12, λ=0 increases geometric-mean F+B latency by 4.07% (worst-case regression 49.26%). Increasing λ to 16–20 yields only 0.63–0.86% average improvement, while λ=20 introduces a 1.71% worst-case regression. Full F+B table is in our response to Reviewer sGfX.
(b) Replica placement plan sensitivity to λ. We further evaluate 12 high-skew routes (1K/4K/8K/16K tokens/GPU, three seeds each). The table reports the total remote replicas across eight GPUs, with each entry corresponding to three seeds.
| Tokens/GPU | λ=0 | λ=8 | λ=12 | λ=16 |
|---|---|---|---|---|
| 1K | 6 / 7 / 7 | 1 / 5 / 5 | 1 / 0 / 0 | 0 / 0 / 0 |
| 4K | 8 / 6 / 8 | 6 / 6 / 8 | 5 / 5 / 8 | 4 / 4 / 7 |
| 8K | 8 / 6 / 9 | 7 / 6 / 9 | 5 / 6 / 9 | 6 / 6 / 9 |
| 16K | 8 / 7 / 8 | 8 / 6 / 8 | 7 / 6 / 8 | 6 / 6 / 8 |
Both λ=8 and λ=16 change 6 of 12 complete placement plans relative to λ=12. The effect is especially pronounced at small token counts: at 1K, λ=0 selects 6–7 replicas, whereas λ=12 retains only 0–1. For one route, removing seven replicas increases the maximum compute load from 82 to 131 blocks but reduces the estimated reduction-aware cost from 154 to 131 block equivalents, illustrating why compute-load balancing alone can be insufficient.
(c) Reduction-aware pruning ablation (λ=12 vs. λ=0). We compare λ=12 against λ=0, which disables the reduction-aware penalty while retaining replication and all required gradient reductions. Both use identical high-skew OLMoE training routes. For each token size, we evaluate three routing seeds, reporting the median backward latency ratio and range across measurements.
| Tokens/GPU | Backward latency (λ=0 / λ=12) | Range |
|---|---|---|
| 1K | 1.2551× | 1.045–1.373 |
| 2K | 1.1215× | 1.018–1.192 |
| 4K | 0.9976× | 0.982–1.033 |
| 8K | 0.9968× | 0.988–1.001 |
| 16K | 0.9996× | 0.990–1.004 |
| 32K | 0.9995× | 0.998–1.002 |
Disabling reduction awareness increases backward latency by 25.5% at 1K and 12.2% at 2K tokens/GPU, consistently across all measurements at each size. At 4K–32K, the median latency differences remain within 0.32%. These results demonstrate that reduction-aware pruning is particularly important for small workloads, where replica-gradient reduction costs are less easily amortized by computation.
3. Practical skewness in inference and training (Q2)
Training-time balancing reduces average skew, but does not remove per-batch skew at inference or within individual training steps. We add two measurements below.
1. Inference skew in DeepSeek-V4-Pro. We evaluate DeepSeek-V4-Pro-0813 (auxiliary-loss-free balancing, 384 experts, top-6). We analyze inference routing traces from four workload sources: ShareGPT (chat), InstructCoder (code), AIMO (math), and Sonnet (long-context). We collect routing statistics across all 58 learned-routing layers and report the median per-layer skew metrics:
| Workload | Median κ | Expert max/mean |
|---|---|---|
| Chat / ShareGPT | 2.77 | 14.54 |
| Code / InstructCoder | 6.83 | 18.52 |
| Math / AIMO | 12.63 | 46.45 |
| Long-context / Sonnet | 12.99 | 55.96 |
These reveal substantial workload-dependent concentration despite auxiliary-loss-free balancing.
We additionally show the routing skewness across layers for one 41,378-token request in LongCodeBench. Owner-rank means work within each GPU.
| Layer | κ | Expert max/mean | Owner-rank max/mean |
|---|---|---|---|
| L15 | 3.06 | 13.83 | 1.34 |
| L30 | 3.27 | 19.37 | 1.39 |
| L45 | 4.63 | 27.33 | 1.50 |
| L60 | 2.44 | 13.00 | 1.18 |
The original expert ownership produces 18–50% higher assignment load on the busiest GPU than the rank average in these sampled layers. This confirms that expert concentration can translate into cross-GPU imbalance.
2. Training Skew in Qwen3-30B-A3B. We analyze 3,000 full-parameter training updates × 48 MoE layers. This table shows the routing skew throughout the training trajectory.
| Training updates | Median κ | P95 κ | Expert max/mean | Top-8 share |
|---|---|---|---|---|
| 1–100 | 2.922 | 3.987 | 8.192 | 31.31% |
| 501–600 | 2.098 | 2.773 | 6.219 | 24.45% |
| 1,001–1,100 | 1.919 | 2.598 | 5.838 | 22.86% |
| 1,501–1,600 | 1.874 | 2.515 | 5.758 | 22.59% |
| 2,001–2,100 | 1.800 | 2.415 | 5.555 | 21.84% |
| 2,501–2,600 | 1.803 | 2.436 | 5.491 | 21.84% |
| 2,901–3,000 | 1.781 | 2.445 | 5.538 | 21.72% |
| All 3,000 updates | 1.924 | 2.814 | 5.833 | 22.99% |
Although routing concentration decreases over training trajectory, 91.69% of all layer-step observations still have κ > 1.5, and 45/48 layers have a per-step median κ above 1.5. The median remains 1.781 even in the final 100 updates.
3. Implications for MegaFlux. Residual skew persists in both practical inference and full-model training. On the training routes, MegaFlux reduces the median rank max/mean load from 1.359 to 1.010. On DeepSeek-V4-Pro inference, it yields 1.13–1.26× prefill speedups with only two spare slots/GPU.
4. MegaFlux port to DeepGEMM MegaMoE (Q3)
Yes. MegaFlux’s planner and pipelined replication are independent of the underlying megakernel. DeepGEMM Mega MoE uses a persistent kernel that overlaps token dispatch with tiled FC1/FC2 computation, allowing MegaFlux to integrate at the same points: align the planner’s block size beta with DeepGEMM’s token tile, reuse existing dispatch warps for replica-weight transfers, and gate FC1/FC2 weight readiness. Kernel-specific tuning primarily involves dispatch-warp scheduling and transfer chunk sizes to maximize overlap with existing communication and computation.
5. Generality (Q3)
The core principles of MegaFlux are not specific to Blackwell, which we chose for implementation and evaluation.
Planner. The planner operates on integer token-block counts without architecture-specific features; its reduction-aware penalty λ requires recalibration.
MoE kernel. Key hardware enablers include GPU-initiated peer-memory access, cross-GPU synchronization, and persistent execution with specialized communication and compute roles. Comparable capabilities exist on NVIDIA Hopper (NVLink/NVSHMEM, system-scope synchronization, warp specialization) and AMD Instinct (xGMI/rocSHMEM, fine-grained synchronization, persistent HIP kernels). Both also support accelerated matrix multiplication (WGMMA/MFMA) and asynchronous data movement.
Porting requires architecture-specific kernel implementations and tuning, potentially including numerical formats, but not redesigning MegaFlux's core algorithms.
6. Cross-node EP (Q3)
MegaFlux's planning and fine-grained pipelining principles extend across nodes, but require a different transport and topology-aware cost model. A hierarchical planner would prioritize NVLink-based replication within nodes and allow cross-node replication only when straggler reduction outweighs additional communication costs. A similar cross-node replica cost calibration can also be done for the cross-node planner. GPU-initiated RDMA and node-local weight distribution and gradient aggregation could further reduce cross-node overhead. We leave this extension to future work.
We thank the reviewer for recognizing the importance of the problem and for the constructive feedback. We conducted additional experiments and expanded our discussion to clarify the key concerns and strengthen the paper.
1. End-to-end training comparison between MegaFlux and UltraEP (W & Q0)
We thank the reviewer for highlighting the need to validate MegaFlux's backward optimizations in complete training, particularly when UltraEP's replica-gradient reduction can overlap with non-MoE backward computation. We agree that the isolated MoE-layer comparison in §4.4 does not account for this overlap. To address this concern, we added end-to-end training-step measurements and a 3,000-step training and convergence study, explicitly retaining UltraEP's asynchronous reduction schedule.
Experimental setup. We perform full-parameter training of Qwen3-30B-A3B (48 MoE layers, 128 experts/layer, top-8 routing) on WikiText-103, using eight NVIDIA B200 GPUs. Unlike our layer-level benchmarks with reconstructed routing distributions, these experiments use real training data and live model routing. We use EP=8, dense DP=8, TP=PP=CP=1, a sequence length of 4,096, and a global batch of eight sequences. All methods share identical initialization. Training uses BF16 weights and activations, with FP32 router/SwiGLU arithmetic, parameter gradients, reductions, and Adam states. MegaFlux and UltraEP each provision four spare expert slots/GPU/layer. Fixed-placement MegaFlux uses the same MegaFlux kernel with replication disabled.
UltraEP integration and overlap. We preserve UltraEP's expert GEMMs, replica placement, weight synchronization, and asynchronous replica-gradient reduction. Unlike the isolated MoE-layer comparison in §4.4, replica-gradient reduction is launched after expert backward and synchronized only after router and attention backward, allowing it to overlap with non-MoE computation as designed in UltraEP.
End-to-end training performance. We measure 30 complete training steps after 10 warmup steps, reporting the median CUDA-synchronized, rank-maximum wall-clock latency. Timing includes full forward/backward computation, activation recomputation, planning, communication, gradient reduction/clipping, and Adam updates.
MegaFlux achieves 1.10× speedup over fixed placement and 2.02× over UltraEP, even when UltraEP's asynchronous replica-gradient reduction is allowed to overlap non-MoE backward computation. This shows that MegaFlux's advantage is not limited to the isolated MoE-layer comparison, but also remains substantial in complete training under UltraEP's asynchronous reduction schedule.
Training convergence. We additionally train the model with MegaFlux and UltraEP for 3,000 steps each, processing 98.3 million training tokens per method under identical initialization. Both runs use a 100-step linear learning-rate warmup to 10−5, followed by cosine decay to 10−6, with evaluation on the same validation set at initialization and every 100 training steps.
Both methods exhibit closely matched convergence behavior. The final five validation evaluations average 1.67537/1.67516 (MegaFlux/UltraEP), and both methods reach their lowest observed validation losses at step 3,000: 1.67436/1.67472 (from approximately 2.111 initially). Across all 31 validation evaluations, the mean and maximum absolute relative cross-entropy differences are only 0.0209% and 0.0744%, respectively. The mean absolute relative training-loss difference across all 3,000 steps is 0.0516%, providing further evidence of correct end-to-end training integration and closely matched optimization trajectories.
Importantly, the training acceleration is sustained throughout these longer runs: MegaFlux averages 546.5 ms/step versus 1,164.7 ms/step for UltraEP (2.131× speedup), and completes the entire training/evaluation workload in 30.45 versus 61.34 minutes (2.015× speedup), including setup, input staging, and periodic validation.
Together, these results establish that MegaFlux's performance benefits extend to complete full-model training, even when UltraEP overlaps replica-gradient reduction with non-MoE backward computation, while achieving convergence behavior and final validation loss comparable to the training using UltraEP.