acceptodds
Under review as a conference paper at ICLR 2027

MegaFlux: Skew-Resilient MoE Megakernels via Pipelined Expert Replication

Abstract

Mixture-of-experts (MoE) megakernels fuse expert-parallel communication with expert computation. However, under fixed expert placement, routing skew creates GPU stragglers: overloaded GPUs determine layer latency while others sit idle. Replicating hot experts can shift work to underloaded GPUs, but dynamic replicas introduce additional work: replicas must receive expert weights to execute and, during training, their partial weight gradients must be reduced at the expert owners. We present **MegaFlux**, which makes expert replication a runtime decision and pipelines the communication induced by replication within persistent MoE execution. An on-device planner jointly selects replica locations and assigns tile-aligned token blocks under a per-GPU replica budget, leaving router outputs unchanged. The forward and backward megakernels realize pipelined expert replication: replicas begin computation as their required weights arrive, while backward overlaps replica-gradient reduction with ongoing expert computation. MegaFlux extends TensorRT-LLM's CuTeDSL MegaMoE forward kernel and introduces a new backward MoE megakernel. Across 147 configurations per direction on eight NVIDIA B200 GPUs, MegaFlux achieves geometric-mean speedups of for forward and for backward over the same megakernels with fixed placement, peaking at and . In ablations, pipelining hides –% of replica-weight transfer cost in forward and –% of combined weight-transfer and replica-gradient-reduction cost in backward, yielding up to % and % additional layer-latency reductions over the same replication plans with these operations executed separately. Integrated into vLLM for DeepSeek-V4-Pro prefill, MegaFlux delivers – median end-to-end speedups over fixed placement.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
mixture of experts, expert parallelism, load balancing, dynamic expert replication, GPU megakernels, communication-computation overlap, efficient training and inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wyrBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MegaFlux, which accelerates multi-GPU Mixture-of-Experts LLM training and inference by dynamically creating replicas for busy experts on less busy GPUs and balancing token workloads. It comprises a planner that selects replica placements and token block distribution, and an executor that pipelines weight transfers and gradient reductions with expert computation.

Critical strengths

This paper identifies and addresses a meaningful problem. It provides a way to execute MoE LLMs on multi-GPU platforms more efficiently, and validated the latency reductions through experiments.

Critical weaknesses

The practical significance of the backward mechanism is not verified in a complete training process. This paper's end-to-end experiment covers inference prefill, but end-to-end training is not reported. Additionally, UltraEP documented that training integration overlaps reduction with router and attention backward, but the isolated MoE-layer comparison excluded it. This limits how strongly MegaFlux's advantage in training is established.

Questions for authors

Section 4.4 measures UltraEP's performance within the MoE layer, so the non-MoE overlap is excluded. Would the comparison change when UltraEP's asynchronous reduction overlaps router and attention backward?

  1. How is backward latency's sensitivity to λ? Can you include results with reduction-aware pruning disabled?

  2. How much skew would remain in practice? The 7.3–41.0% additional latency was measured from the P95 routes of OLMoE and Qwen3 workloads, while modern MoE models like DeepSeek-V3 employs auxiliary-loss-free balancing, and may result in less skew. How would such models benefit from MegaFlux?

  3. How general is this approach? It seems to be tied to Blackwell features and a single NVLink domain. Can some parts transfer to other megakernels like DeepGEMM Mega MoE? And would some parts break on other scenarios like cross-node EP, where weight transfer is more expensive?

  4. Compared with approaches that prefetch ahead of time, like PROBE, what are the advantages and disadvantages of planning from the current routing, especially when the token count is small?

5 comments

We thank the reviewer for recognizing the importance of the problem and for the constructive feedback. We conducted additional experiments and expanded our discussion to clarify the key concerns and strengthen the paper.

1. End-to-end training comparison between MegaFlux and UltraEP (W & Q0)

We thank the reviewer for highlighting the need to validate MegaFlux's backward optimizations in complete training, particularly when UltraEP's replica-gradient reduction can overlap with non-MoE backward computation. We agree that the isolated MoE-layer comparison in §4.4 does not account for this overlap. To address this concern, we added end-to-end training-step measurements and a 3,000-step training and convergence study, explicitly retaining UltraEP's asynchronous reduction schedule.

Experimental setup. We perform full-parameter training of Qwen3-30B-A3B (48 MoE layers, 128 experts/layer, top-8 routing) on WikiText-103, using eight NVIDIA B200 GPUs. Unlike our layer-level benchmarks with reconstructed routing distributions, these experiments use real training data and live model routing. We use EP=8, dense DP=8, TP=PP=CP=1, a sequence length of 4,096, and a global batch of eight sequences. All methods share identical initialization. Training uses BF16 weights and activations, with FP32 router/SwiGLU arithmetic, parameter gradients, reductions, and Adam states. MegaFlux and UltraEP each provision four spare expert slots/GPU/layer. Fixed-placement MegaFlux uses the same MegaFlux kernel with replication disabled.

UltraEP integration and overlap. We preserve UltraEP's expert GEMMs, replica placement, weight synchronization, and asynchronous replica-gradient reduction. Unlike the isolated MoE-layer comparison in §4.4, replica-gradient reduction is launched after expert backward and synchronized only after router and attention backward, allowing it to overlap with non-MoE computation as designed in UltraEP.

End-to-end training performance. We measure 30 complete training steps after 10 warmup steps, reporting the median CUDA-synchronized, rank-maximum wall-clock latency. Timing includes full forward/backward computation, activation recomputation, planning, communication, gradient reduction/clipping, and Adam updates.

MethodMedian ms/stepSlowdown vs. MegaFlux
UltraEP1,126.12.02×
Fixed-placement MegaFlux612.91.10×
MegaFlux557.61.00×

MegaFlux achieves 1.10× speedup over fixed placement and 2.02× over UltraEP, even when UltraEP's asynchronous replica-gradient reduction is allowed to overlap non-MoE backward computation. This shows that MegaFlux's advantage is not limited to the isolated MoE-layer comparison, but also remains substantial in complete training under UltraEP's asynchronous reduction schedule.

Training convergence. We additionally train the model with MegaFlux and UltraEP for 3,000 steps each, processing 98.3 million training tokens per method under identical initialization. Both runs use a 100-step linear learning-rate warmup to 10−510^{-5}, followed by cosine decay to 10−610^{-6}, with evaluation on the same validation set at initialization and every 100 training steps.

Both methods exhibit closely matched convergence behavior. The final five validation evaluations average 1.67537/1.67516 (MegaFlux/UltraEP), and both methods reach their lowest observed validation losses at step 3,000: 1.67436/1.67472 (from approximately 2.111 initially). Across all 31 validation evaluations, the mean and maximum absolute relative cross-entropy differences are only 0.0209% and 0.0744%, respectively. The mean absolute relative training-loss difference across all 3,000 steps is 0.0516%, providing further evidence of correct end-to-end training integration and closely matched optimization trajectories.

Importantly, the training acceleration is sustained throughout these longer runs: MegaFlux averages 546.5 ms/step versus 1,164.7 ms/step for UltraEP (2.131× speedup), and completes the entire training/evaluation workload in 30.45 versus 61.34 minutes (2.015× speedup), including setup, input staging, and periodic validation.

Together, these results establish that MegaFlux's performance benefits extend to complete full-model training, even when UltraEP overlaps replica-gradient reduction with non-MoE backward computation, while achieving convergence behavior and final validation loss comparable to the training using UltraEP.

2. Sensitivity to λ and experiment with reduction-aware pruning disabled (Q1)

We thank the reviewer for this suggestion. We provide additional experiments on 8×B200 (EP8, E256, top-8) to evaluate the sensitivity of execution latency and replica placement to λ, as well as the impact of disabling reduction-aware pruning.

(a) Backward execution latency sensitivity to λ. We sweep λ ∈ {0, 4, 8, 12, 16, 20, 32, 48} on real-high routes at 1K/4K/8K/16K tokens/GPU, with 8 spare experts per GPU. The table reports the geometric-mean and maximum latency ratios (compared to λ=12, lower is better) across these four evaluated configurations.

λGeomean ratioWorst-case ratio
01.02561.1053
41.03321.1258
80.99801.0033
121.00001.0000
161.00191.0097
201.00951.0241
321.01431.0252
481.02201.0552

Backward latency is stable around λ=12: λ=8–16 changes geometric-mean latency by at most 0.20%, whereas λ=0 increases average latency by 2.56%, with a 10.53% worst-case regression at 1K. Larger coefficients provide no consistent benefit: λ=32 and 48 increase average latency by 1.43% and 2.20%, respectively. These results indicate a relatively flat operating region near λ=12.

We also evaluate forward plus backward (F+B) combined latency. Relative to λ=12, λ=0 increases geometric-mean F+B latency by 4.07% (worst-case regression 49.26%). Increasing λ to 16–20 yields only 0.63–0.86% average improvement, while λ=20 introduces a 1.71% worst-case regression. Full F+B table is in our response to Reviewer sGfX.

(b) Replica placement plan sensitivity to λ. We further evaluate 12 high-skew routes (1K/4K/8K/16K tokens/GPU, three seeds each). The table reports the total remote replicas across eight GPUs, with each entry corresponding to three seeds.

Tokens/GPUλ=0λ=8λ=12λ=16
1K6 / 7 / 71 / 5 / 51 / 0 / 00 / 0 / 0
4K8 / 6 / 86 / 6 / 85 / 5 / 84 / 4 / 7
8K8 / 6 / 97 / 6 / 95 / 6 / 96 / 6 / 9
16K8 / 7 / 88 / 6 / 87 / 6 / 86 / 6 / 8

Both λ=8 and λ=16 change 6 of 12 complete placement plans relative to λ=12. The effect is especially pronounced at small token counts: at 1K, λ=0 selects 6–7 replicas, whereas λ=12 retains only 0–1. For one route, removing seven replicas increases the maximum compute load from 82 to 131 blocks but reduces the estimated reduction-aware cost from 154 to 131 block equivalents, illustrating why compute-load balancing alone can be insufficient.

(c) Reduction-aware pruning ablation (λ=12 vs. λ=0). We compare λ=12 against λ=0, which disables the reduction-aware penalty while retaining replication and all required gradient reductions. Both use identical high-skew OLMoE training routes. For each token size, we evaluate three routing seeds, reporting the median backward latency ratio and range across measurements.

Tokens/GPUBackward latency (λ=0 / λ=12)Range
1K1.2551×1.045–1.373
2K1.1215×1.018–1.192
4K0.9976×0.982–1.033
8K0.9968×0.988–1.001
16K0.9996×0.990–1.004
32K0.9995×0.998–1.002

Disabling reduction awareness increases backward latency by 25.5% at 1K and 12.2% at 2K tokens/GPU, consistently across all measurements at each size. At 4K–32K, the median latency differences remain within 0.32%. These results demonstrate that reduction-aware pruning is particularly important for small workloads, where replica-gradient reduction costs are less easily amortized by computation.

3. Practical skewness in inference and training (Q2)

Training-time balancing reduces average skew, but does not remove per-batch skew at inference or within individual training steps. We add two measurements below.

1. Inference skew in DeepSeek-V4-Pro. We evaluate DeepSeek-V4-Pro-0813 (auxiliary-loss-free balancing, 384 experts, top-6). We analyze inference routing traces from four workload sources: ShareGPT (chat), InstructCoder (code), AIMO (math), and Sonnet (long-context). We collect routing statistics across all 58 learned-routing layers and report the median per-layer skew metrics:

WorkloadMedian κExpert max/mean
Chat / ShareGPT2.7714.54
Code / InstructCoder6.8318.52
Math / AIMO12.6346.45
Long-context / Sonnet12.9955.96

These reveal substantial workload-dependent concentration despite auxiliary-loss-free balancing.

We additionally show the routing skewness across layers for one 41,378-token request in LongCodeBench. Owner-rank means work within each GPU.

LayerκExpert max/meanOwner-rank max/mean
L153.0613.831.34
L303.2719.371.39
L454.6327.331.50
L602.4413.001.18

The original expert ownership produces 18–50% higher assignment load on the busiest GPU than the rank average in these sampled layers. This confirms that expert concentration can translate into cross-GPU imbalance.

2. Training Skew in Qwen3-30B-A3B. We analyze 3,000 full-parameter training updates × 48 MoE layers. This table shows the routing skew throughout the training trajectory.

Training updatesMedian κP95 κExpert max/meanTop-8 share
1–1002.9223.9878.19231.31%
501–6002.0982.7736.21924.45%
1,001–1,1001.9192.5985.83822.86%
1,501–1,6001.8742.5155.75822.59%
2,001–2,1001.8002.4155.55521.84%
2,501–2,6001.8032.4365.49121.84%
2,901–3,0001.7812.4455.53821.72%
All 3,000 updates1.9242.8145.83322.99%

Although routing concentration decreases over training trajectory, 91.69% of all layer-step observations still have κ > 1.5, and 45/48 layers have a per-step median κ above 1.5. The median remains 1.781 even in the final 100 updates.

3. Implications for MegaFlux. Residual skew persists in both practical inference and full-model training. On the training routes, MegaFlux reduces the median rank max/mean load from 1.359 to 1.010. On DeepSeek-V4-Pro inference, it yields 1.13–1.26× prefill speedups with only two spare slots/GPU.

4. MegaFlux port to DeepGEMM MegaMoE (Q3)

Yes. MegaFlux’s planner and pipelined replication are independent of the underlying megakernel. DeepGEMM Mega MoE uses a persistent kernel that overlaps token dispatch with tiled FC1/FC2 computation, allowing MegaFlux to integrate at the same points: align the planner’s block size beta with DeepGEMM’s token tile, reuse existing dispatch warps for replica-weight transfers, and gate FC1/FC2 weight readiness. Kernel-specific tuning primarily involves dispatch-warp scheduling and transfer chunk sizes to maximize overlap with existing communication and computation.

5. Generality (Q3)

The core principles of MegaFlux are not specific to Blackwell, which we chose for implementation and evaluation.

Planner. The planner operates on integer token-block counts without architecture-specific features; its reduction-aware penalty λ requires recalibration.

MoE kernel. Key hardware enablers include GPU-initiated peer-memory access, cross-GPU synchronization, and persistent execution with specialized communication and compute roles. Comparable capabilities exist on NVIDIA Hopper (NVLink/NVSHMEM, system-scope synchronization, warp specialization) and AMD Instinct (xGMI/rocSHMEM, fine-grained synchronization, persistent HIP kernels). Both also support accelerated matrix multiplication (WGMMA/MFMA) and asynchronous data movement.

Porting requires architecture-specific kernel implementations and tuning, potentially including numerical formats, but not redesigning MegaFlux's core algorithms.

6. Cross-node EP (Q3)

MegaFlux's planning and fine-grained pipelining principles extend across nodes, but require a different transport and topology-aware cost model. A hierarchical planner would prioritize NVLink-based replication within nodes and allow cross-node replication only when straggler reduction outweighs additional communication costs. A similar cross-node replica cost calibration can also be done for the cross-node planner. GPU-initiated RDMA and node-local weight distribution and gradient aggregation could further reduce cross-node overhead. We leave this extension to future work.

Reviewer_sGfXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MegaFlux addresses routing-induced load imbalance in expert-parallel MoE execution by integrating dynamic expert replication into persistent MoE megakernels. An on-device planner determines replica placement and tile-aligned token-block assignments under per-GPU capacity constraints, with reduction-aware pruning for training. The forward executor tracks the readiness of the two expert weight matrices separately, while the backward executor coordinates gradient-production priorities to enable earlier tile-level replica-gradient reduction. MegaFlux extends the CuTeDSL MegaMoE forward kernel and implements a backward megakernel. On eight NVIDIA B200 GPUs, the authors report geometric-mean forward and backward speedups of 1.45× and 1.28× over fixed expert placement, respectively, and 1.13–1.26× end-to-end prefill speedups after integration into vLLM.

Critical strengths
  1. Useful execution mechanisms for reducing exposed replication overhead. Routing-induced imbalance can leave GPUs underutilized even when token communication and expert computation are overlapped. MegaFlux integrates load redistribution into persistent execution while preserving expert selections and routing weights. Per-projection weight readiness allows useful computation before replica materialization finishes, and coordinated backward gradient production enables earlier reduction. These mechanisms address concrete dependencies that can otherwise limit the benefit of replication and are the most interesting technical aspects of the work.

  2. Measurable performance benefits with some separation of planning and execution effects. The sameplan comparison between separate-stage and pipelined replication isolates the aggregate effect of pipelining from load redistribution. Comparisons with UltraEP and Mixture-of-Kittens, together with end-to-end prefill improvements, provide evidence of practical value. The reported planner overhead and discussion of workloads where replication is unhelpful also help contextualize the results.

Critical weaknesses
  1. The incremental value of the individual scheduling mechanisms is insufficiently isolated. Expert replication, constrained load balancing, and fine-grained communication–computation overlap have substantial precedent. The contribution therefore depends on the specific integration and schedules. The current ablation evaluates pipelining as a whole, without separately quantifying per-projection weight readiness, communication-warp reuse, and coordinated gradient-production priorities. It remains unclear which mechanisms are essential and how much each contributes. The reduction-aware planner also uses a fixed penalty of lambda = 12 without explicitly modeling transfer time or overlap; regressions on small backward workloads make sensitivity to this choice relevant to the reliability of the scheduling contribution.

  2. The practical training benefit and the scope of the UltraEP comparison remain uncertain. The main experiments use synthetic weights and activations with reconstructed routing distributions, while the end-to-end evaluation covers inference prefill. This establishes layer-level benefits but does not demonstrate an improvement in complete training-step latency. In addition, UltraEP is required to complete replica-gradient reduction within the measured MoE layer, whereas its broader training integration supports overlap with non-MoE work. The reported layer-level advantage may therefore differ from the advantage under the respective intended training schedules. This uncertainty limits the strength of the evidence for the backward contribution.

Questions for authors
  1. Validation of the claimed training benefit: Could you provide a focused end-to-end training comparison for a representative model such as Qwen3-30B-A3B, measuring at least 30 steady-state steps after warmup? One representative configuration would already be useful. Please compare with fixed expert placement and UltraEP under matched precision and parallelism settings, use the same replica budget for MegaFlux and UltraEP, and report training-step latency together with exposed replica-weight transfer and gradient-reduction costs.

  2. Clarification of baseline performance: How does the UltraEP comparison change when replica-gradient reduction can overlap with non-MoE backward computation? A profiling timeline would help explain the performance differences from UltraEP and Mixture-of-Kittens in Figure 6 and identify which costs remain on the critical path.

  3. Validation of the incremental scheduling mechanisms: Can you quantify the contribution of per-projection weight readiness and coordinated backward gradient-production priorities separately, while keeping the replica plan fixed where applicable? This would clarify which mechanisms account for the improvement beyond replication and aggregate pipelining.

  4. Clarification of planner robustness: How was lambda = 12 selected, and how sensitive are replica decisions and execution latency to this value and to a smaller replica budget? A limited sensitivity check around the evaluated settings would help explain the small-workload regressions.

6 comments

We thank the reviewer for the thoughtful evaluation and for recognizing the value of our execution mechanisms. We strengthened our experimental evaluation and analysis to address the reviewer's questions and suggestions.

1. Practical training benefit and UltraEP comparison (W2 & Q1 & Q2)

We thank the reviewer for highlighting the need to evaluate MegaFlux in complete training, particularly when UltraEP's replica-gradient reduction can overlap with non-MoE backward computation. We added end-to-end training-step measurements and a 3,000-step training and convergence study.

Experimental setup. We perform full-parameter training of Qwen3-30B-A3B (48 MoE layers, 128 experts/layer, top-8 routing) on WikiText-103, using eight NVIDIA B200 GPUs. Unlike our layer-level benchmarks with reconstructed routing distributions, these experiments use real training data and live model routing. We use EP=8, dense DP=8, TP=PP=CP=1, a sequence length of 4,096, and a global batch of eight sequences. All methods share identical initialization. Training uses BF16 weights and activations, with FP32 router/SwiGLU arithmetic, parameter gradients, reductions, and Adam states. MegaFlux and UltraEP each provision four spare expert slots/GPU/layer. Fixed-placement MegaFlux uses the same native MegaFlux kernel with replication disabled.

UltraEP integration and overlap. We preserve UltraEP's expert GEMMs, replica placement, weight synchronization, and asynchronous replica-gradient reduction. Unlike the isolated MoE-layer comparison in §4.4, replica-gradient reduction is launched after expert backward and synchronized only after router and attention backward, allowing it to overlap with non-MoE computation as designed in UltraEP.

End-to-end training performance. We measure 30 complete training steps after 10 warmup steps, reporting the median CUDA-synchronized, rank-maximum wall-clock latency. Timing includes full forward/backward computation, activation recomputation, planning, communication, gradient reduction/clipping, and Adam updates.

MethodMedian ms/stepSlowdown vs. MegaFlux
UltraEP1,126.12.02×
Fixed-placement MegaFlux612.91.10×
MegaFlux557.61.00×

MegaFlux achieves 1.10× speedup over fixed placement MegaFlux and 2.02× over UltraEP, even when UltraEP's replica-gradient reduction is allowed to overlap non-MoE backward computation. Thus, the advantage observed in the isolated MoE-layer evaluation persists at the complete training-step level, without restricting UltraEP's reduction overlap.

Exposed replica-weight transfer and gradient-reduction costs. We follow the separate-stage replication methodology in §4.3, comparing MegaFlux with variants that externalize weight transfers or gradient reductions into separate GPU stages. All configurations use the above training setup, running 100 steps (10 warmup). The estimated exposed fractions are 44.30% for forward weight transfer and 6.04% for backward gradient reduction, indicating that MegaFlux hides approximately 55.70% and 93.96% of their respective costs through pipelining.

Training convergence. We additionally train the model with MegaFlux and UltraEP for 3,000 steps each, under identical initialization. Both runs use a 100-step linear learning-rate warmup to 10−510^{-5}, followed by cosine decay to 10−610^{-6}, with evaluation on the same validation set at initialization and every 100 training steps.

Both methods exhibit closely matched convergence behavior, with validation cross-entropy decreasing substantially throughout training. The final five validation evaluations average 1.67537/1.67516 (MegaFlux/UltraEP), and both methods reach their lowest observed validation losses at step 3,000: 1.67436/1.67472 (from approximately 2.111 initially). Across all 31 validation evaluations, the mean and maximum absolute relative cross-entropy differences are only 0.0209% and 0.0744%, respectively. The mean absolute relative training-loss difference across all 3,000 steps is 0.0516%. These results provide additional end-to-end evidence that MegaFlux is correctly integrated into the training process and preserves comparable optimization behavior and validation quality.

Over these 3,000 training steps, MegaFlux averages 546.5 ms/step versus 1,164.7 ms/step for UltraEP (2.131× speedup), and completes the entire training/evaluation workload in 30.45 versus 61.34 minutes (2.015× speedup), including setup, input staging, and periodic validation.

These results demonstrate that MegaFlux's training benefits extend beyond isolated MoE layers to complete, sustained full-model training, even when UltraEP is permitted to overlap replica-gradient reduction with non-MoE backward computation, while achieving closely matched training convergence and final validation quality.

2. Separation of scheduling mechanisms (W1 & Q3)

We thank the reviewer for suggesting a finer-grained decomposition. The three mechanisms are the finest-grained layer of MegaFlux. They further refine our pipelined replication design on top of a block-granular planner, whose reduction-aware variant (details in Response 4) decides which replicas are worth creating. Here we isolate each mechanism.

Experimental setup. We use eight B200 GPUs (EP=8), 256 experts, top-8 routing, hidden/intermediate dimensions of 7,168/2,048, and eight spare expert slots per GPU. We evaluate: real-high skewness at 1K/4K/8K/16K tokens/GPU and real-extreme skewness at 2K/4K/6K. Within each comparison, we change only the mechanism under study. We report median rank-maximum latency.

1. Per-projection weight readiness. We compare independent W1/W2 readiness against waiting for both W1 and W2 before computation. Forward latency improves in 11/12 real-high and 9/9 real-extreme configurations. At 8K tokens/GPU, the real-high routes improve by 1.36–1.68%; at 6K, the real-extreme routes improve by 0.93–2.44%. The mechanism matters more when skew is concentrated enough that underloaded GPUs execute mostly replica work. On six routes of this kind, enabling per-projection readiness reduces forward latency by 1.3–4.9%.

2. Coordinated gradient-production priority. We separately ablate the production order of replicated-expert gradients. Specifically, we prioritize intermediate-gradient requantization and W1-gradient production for replicated experts; the baseline uses the default expert order. On real-high routing, coordinated priority reduces backward latency by up to 3.62% in 4K tokens/rank, 5.96% at 8K, and 3.66% at 16K. We further evaluate prioritizing gradient production on owners only, replicas only, or both. Interestingly, on one 8K input, prioritizing replicas alone achieves an 8.37% latency reduction, compared with 5.96% when prioritizing both, suggesting that the best production order is workload-dependent. We leave workload-adaptive prioritization to future work.

3. Communication-warp reuse. We compare this mechanism with issuing the same transfers from epilogue warps. Across three real-extreme routes per size, communication-warp reuse reduces latency by 2.50–15.32% at 2K and 2.88–11.02% at 4K tokens/GPU, with median reductions of 13.63% and 3.59%, respectively. This demonstrates the advantage of assigning replica transfers to existing communication warps rather than competing with compute-side epilogue work.

3. Finer-grained Profiling of MegaFlux, UltraEP, and Mixture-of-Kittens (Q2)

We thank the reviewer for suggesting further profiling. We collected forward and backward execution timelines for all three systems on 8×B200. We analyze a representative case with 4K tokens/GPU, real-high routing, and BF16. The measured backward latencies are 21.93/11.64/7.62 ms for UltraEP/MoK/MegaFlux, respectively.

Analysis:

1. UltraEP: Reduction dominates backward latency but does not explain the entire gap. Replica-gradient reduction forms a 12.68 ms tail (58% of the profiled execution span). Even excluding this tail, the preceding operations span 9.25 ms, versus 7.62 ms for MegaFlux's complete backward execution.

2. MoK: Load imbalance creates a long execution tail despite kernel fusion. MoK already overlaps communication and computation, but its busiest rank remains constrained by routed work. In backward, WGrad continues for 4.30 ms after internal input-gradient combine finishes. The per-rank profiles further illustrate the imbalance:

DirectionMethodFastest rank (ms)Slowest rank (ms)
ForwardMoK1.6277.051
ForwardMegaFlux2.9883.532
BackwardMoK5.68911.638
BackwardMegaFlux7.0787.623

MoK's slowest-to-fastest rank ratio reaches 4.33×/2.05× in forward/backward, compared with 1.18×/1.08× for MegaFlux.

3. MegaFlux: Fine-grained overlap reduces the separate reduction tail. Replica-gradient reduction overlaps WGrad production (3.98–6.93 ms) instead of forming a separate post-computation phase.

4. Selection of λ, and sensitivity to λ and the replica budget (W1 & Q4)

We thank the reviewer for these detailed questions and answer each part separately. All experiments use 8×B200 (EP8), E=256, top-8, H/I = 7168/2048, and 128-row token blocks.

(a) How was λ = 12 selected?

We express the extra cost of each replica in units of token-block computation time. Our backward profiling showed that a replica's cost depends on where its gradients are reduced:

Backward costTimeCost in token blocks
One token block computation≈28 μs1
One replica, partials spread across owners≈110–200 μs≈4–7
One replica, six replicas reduced at the same owner≈570 μs≈20

We therefore charge each remote replica λ = 12 blocks at its owner. This is a moderate value between the dispersed and concentrated regimes.

(b) Sensitivity of execution latency to λ.

We sweep λ ∈ {0, 4, 8, 12, 16, 20, 32, 48} across balanced, dispersed, and concentrated routing patterns, with 1K/4K/16K/32K tokens/GPU. The table reports the geometric-mean and worst-case forward-plus-backward latency ratios relative to λ=12.

λAverage ratioWorst-case ratio
01.04071.4926
41.03421.3713
81.01581.1977
121.00001.0000
160.99371.0000
200.99141.0171
321.00671.0919
481.03331.1901

Increasing λ to 12 substantially reduces average latency and worst-case regressions. Further increasing λ to 16–20 yields only 0.63–0.86% average improvement, while λ=20 introduces a 1.71% worst-case regression. Larger values degrade performance. Thus, λ=12 provides a robust operating point near diminishing returns, without requiring workload-specific tuning.

(c) Sensitivity of replica decisions to λ.

We evaluate replica placement across 12 OLMoE training routes (1K–16K tokens/GPU, three seeds each). The table reports the total number of remote replicas across eight GPUs, with each entry corresponding to three seeds.

Tokens/GPUλ=0λ=8λ=12λ=16
1K6 / 7 / 71 / 5 / 51 / 0 / 00 / 0 / 0
4K8 / 6 / 86 / 6 / 85 / 5 / 84 / 4 / 7
8K8 / 6 / 97 / 6 / 95 / 6 / 96 / 6 / 9
16K8 / 7 / 88 / 6 / 87 / 6 / 86 / 6 / 8

Compared with λ=12, both λ=8 and λ=16 change 6 of 12 complete placement plans. The impact is most pronounced at small token counts: at 1K, λ=12 eliminates all seven replicas selected by λ=0 in 2 cases. Although the maximum compute load increases from 82 to 131 blocks, the estimated reduction-aware cost decreases from 154 to 131 block equivalents. This illustrates why minimizing compute imbalance alone can lead to suboptimal replication decisions.

(d) Reduction-aware pruning ablation (λ=12 vs. λ=0).

We compare λ=12 against λ=0, which disables the reduction-aware penalty while retaining replication and all required gradient reductions. Both use identical high-skew OLMoE training routes. For each token size, we evaluate three routing seeds, reporting the median backward latency ratio and range across measurements.

Tokens/GPUBackward latency (λ=0 / λ=12)Range
1K1.2551×1.045–1.373
2K1.1215×1.018–1.192
4K0.9976×0.982–1.033
8K0.9968×0.988–1.001
16K0.9996×0.990–1.004
32K0.9995×0.998–1.002

Disabling reduction awareness increases backward latency by 25.5% at 1K and 12.2% at 2K tokens/GPU, consistently across all measurements at each size. At 4K–32K, the median latency differences remain within 0.32%. These results demonstrate that reduction-aware pruning is particularly important for small workloads, where replica-gradient reduction costs are less easily amortized by computation.

Even with λ=12, replica overhead can still outweigh load-balancing benefits on some small backward workloads. This motivates exploring a runtime gating mechanism that falls back to fixed placement when the predicted benefits of replication do not justify its overhead.

Reviewer_toowOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

MegaFlux addresses routing imbalance in MoE megakernels by dynamically replicating experts and pipelining weight transfers and replica-gradient reductions with computation. On eight B200 GPUs, it reports geometric-mean forward/backward speedups of 1.45×/1.28× over fixed placement and 1.13–1.26× end-to-end prefill speedups.

Critical strengths
  • Paper address practical limitations of fused moe by handling expert skewness and handle EP within a persistent kernel.
  • Paper easy to follow and understand
  • Results are encouraging
Critical weaknesses
  • full training-step speedups and convergence are absent. “Numerical correctness checks” lack reported tolerances or error measurements, particularly relevant to low-precision computation and reordered gradient accumulation.
  • Mixture-of-Kittens performs additional work; adding separately measured costs to MegaFlux is a useful approximation but does not replace an identical-workload comparison
  • The main sweep assumes 25% spare expert capacity without reporting total peak memory, including replica-gradient buffers
  • The method is not novel. Because the concept of persistent kernel is not new, this is an engineering optimization that the previous work (UltraEP, LLEP) didn't optimize for, so it is trivial if someone wants to do this..
  • Open-source codes missing. Though I know this can be vibe-coded away easily.
Questions for authors

Can you report full training-step performance, quantitative numerical errors, and peak memory?

5 comments

We thank the reviewer for the careful evaluation and constructive feedback, and for recognizing the practical relevance of our work. We conducted additional experiments and provided detailed clarifications to address the concerns raised.

1. Full training-step speedups and convergence (W1 & Q1)

We thank the reviewer for highlighting the need to evaluate MegaFlux in complete training. We added end-to-end training-step measurements and a 3,000-training-step convergence study.

Experimental setup. We perform full-parameter training of Qwen3-30B-A3B (48 MoE layers, 128 experts/layer, top-8 routing) from its official checkpoint on WikiText-103, using eight NVIDIA B200 GPUs. We use EP=8, dense DP=8, TP=PP=CP=1, a sequence length of 4,096, and a global batch of eight sequences. All methods share identical initialization. Training uses BF16 weights and activations, with FP32 router/SwiGLU arithmetic, parameter gradients, reductions, and Adam states. MegaFlux and UltraEP each provision four spare expert slots/GPU/layer. Fixed-placement MegaFlux uses the same MegaFlux kernel with replication disabled.

End-to-end training performance. We measure 30 complete training steps after 10 warmup steps, reporting the median CUDA-synchronized, rank-maximum wall-clock latency. Timing includes full forward/backward computation, activation recomputation, planning, communication, gradient reduction/clipping, and Adam updates.

MethodMedian ms/stepSlowdown vs. MegaFlux
UltraEP1,126.12.02×
Fixed-placement MegaFlux612.91.10×
MegaFlux557.61.00×

MegaFlux achieves 1.10× speedup over fixed placement MegaFlux and 2.02× over UltraEP, even with UltraEP's replica-gradient reduction overlapping non-MoE backward computation.

Training convergence. We additionally train the model with MegaFlux and UltraEP for 3,000 steps each. Both runs use a 100-step linear learning-rate warmup to 10−510^{-5}, followed by cosine decay to 10−610^{-6}, with evaluation on the same validation set and every 100 training steps.

Both methods exhibit similar convergence behavior, with validation cross-entropy loss decreasing substantially and leveling off toward the end of training. The final five validation evaluations average 1.67537/1.67516 (MegaFlux/UltraEP), and both methods reach their lowest observed validation losses at step 3,000: 1.67436/1.67472 (from 2.111 at the beginning). Across all 31 validation evaluations during training, the mean and maximum absolute relative cross-entropy differences between the two methods are only 0.0209% and 0.0744%, respectively. The mean absolute relative training-loss difference across all 3,000 steps is 0.0516%, indicating closely matched training trajectories.

Over these 3,000 training steps, MegaFlux averages 546.5 ms/step versus 1,164.7 ms/step for UltraEP (2.131× speedup), and completes the entire training/evaluation workload in 30.45 versus 61.34 minutes (2.015× speedup).

These results demonstrate that MegaFlux significantly accelerates complete MoE training while achieving convergence behavior and final validation loss comparable to the training using UltraEP.

2. Numerical correctness: tolerances and measured errors (W1 & Q2)

We thank the reviewer for highlighting numerical correctness, particularly under low-precision computation and reordered gradient accumulation. We provide the following tolerances and measured errors.

Replica-gradient accumulation. Replication changes the accumulation of expert weight gradients: each owner and replica computes partial dW1dW_1 and dW2dW_2, which the owner sums in FP32 in a fixed order (owner first, then replicas by GPU rank).

MXFP8 correctness checks. We compare against an unfused, non-replicated MoE reference with matched precision and quantization rules, measuring relative L2 errors and taking the maximum across eight GPUs. We require ≤3% error for dXdX and final dW1/dW2dW_1/dW_2, and <0.5% for per-expert partial gradients against references. In our experiments across E=128/256, 4K–128K tokens/GPU, and multiple routing realizations, all checks pass; maximum final gradient errors are only 0.265% (dW1dW_1) and 0.253% (dW2dW_2).

Independent PyTorch validation. We additionally compare against a PyTorch MoE reference using BF16 weights/activations and FP32 gradients. Across eight test cases and six Adam updates (EP8, E=128, 4K tokens/GPU), the maximum relative L2 errors are:

QuantityFixed placement MegaFlux vs. PyTorchMegaFlux vs. PyTorch
dXdX0.0101%0.0100%
dW1dW_10.0039%0.0038%
dW2dW_20.0039%0.0038%

These results show that replication preserves numerical accuracy comparable to fixed placement.

3. Mixture-of-Kittens identical-workload comparison (W2)

We thank the reviewer for this point and agree that adding separately measured costs is an approximation. We added an identical workload comparison experiment against MoK with the same inputs. In MegaFlux, shared expert's tiles enter the persistent forward and backward task queues as compute-only tasks and reuse the existing MMA and weight-gradient pipelines.

Setup. We test 1k/8k/16k tokens/rank, 3 seeds, and real-high and Zipf-κ6 routes; other settings follow the experiments in Figure 6. Each experiment runs 3 warmup and 8 timed iterations. We take the per-iteration maximum across ranks, then the median within each seed and across seeds. Speedup is defined as MoK latency divided by MegaFlux latency.

Routing skewTokens/rankForward speedupBackward speedup
Real-high1K1.654×1.103×
Real-high8K1.802×1.045×
Real-high16K1.873×1.278×
Zipf-κ61K1.668×1.153×
Zipf-κ68K1.901×1.880×
Zipf-κ616K1.955×2.100×

All cases pass checks of y, dX and routed/shared weight gradients, against the BF16 PyTorch reference. The six configurations retain 1.654–1.955× forward and 1.045–2.100× backward speedups.

4. Peak memory and replica-gradient buffers (W3 & Q3)

We thank the reviewer for pointing out this omission. We report measured peak memory, including replica-gradient buffers. We also clarify that the 25% setting refers to spare expert slots, not total GPU memory: at E=256 and EP8, each GPU preallocates buffers for its 32 owned experts plus 8 spare expert slots.

Replica-gradient buffers. Each expert slot reserves 84 MiB of BF16 partial gradients (56 MiB for dW1 and 28 MiB for dW2). With 8 spare slots, replica gradients therefore occupy 672 MiB/GPU of a 3,360 MiB/GPU gradient bank (32 owned experts plus 8 spared). The spare slots also hold 178.5 MiB of replica weights in forward and 346.5 MiB in backward. These buffers are preallocated.

Measured memory peaks. We use eight B200 GPUs, E=256, H/I=7168/2048, top-8, real-high routing, the precision of the main sweep, and 64K tokens/rank. After warmup, we record whole-device memory occupancy (total minus free) as reported by NVML during five executions, and report the maximum across all eight GPUs. With 25% spare experts, MegaFlux peaks at 25.9 GiB/GPU in forward and 43.5 GiB/GPU in backward. To isolate the cost of the spare capacity itself, we vary the number of expert slots actually allocated on each GPU (32, 34, or 40), which shrinks the preallocated weight and gradient buffers accordingly:

ConfigurationSpare slots/GPUdW buffers (MiB/GPU)Peak memory (GiB/GPU)Backward latency (ms)
Fixed placement02,68843.96120.65
MegaFlux2 (6.25%)2,85642.46106.60 (1.13×)
MegaFlux8 (25%)3,36043.46106.82 (1.13×)

Two observations:

  1. Replication does not raise the observed peak. MegaFlux’s peak stays below that of the zero-spare baseline, partly because saved-activation buffers are sized before execution based on each configuration’s token assignment, and MegaFlux’s more balanced assignment reduces their required capacity. Changes in weight storage and allocator reservation can also contribute to the measured difference.
  2. A smaller budget suffices. The 25% budget is simply the setting used in our sweep; in practice, the planner needs less in this case. With two spare slots, MegaFlux selects the same placements and token assignments as with eight. This reduces the measured peak by 1.0 GiB/GPU, keeps latency within 0.21%.

5. Novelty and complexity (W4)

We thank the reviewer for this comment. Persistent kernels and expert replication are each established, but combining them effectively is not straightforward. Our contribution is a coordinated planning and execution design, guided by insights demonstrated through controlled experiments (EP8, B200, E=256):

  1. Replication has its own cost; pipelining decides whether it pays off. A better-balanced plan does not automatically yield faster execution. Weight transfer and replica-gradient reduction can outweigh the balancing benefit, especially when workloads are small or skew is low. With the identical replica plan, separate-stage execution makes real-high 2K backward 23.8% slower than fixed placement, while MegaFlux's integrated pipeline makes it 9.3% faster.

  2. Planning must account for replication overhead, not only compute balance. Each replica sends full-size gradient partials to the owner, regardless of its token count. At high-skew 1K tokens/GPU, a planner that only balances compute reduces the busiest GPU's load from 131 to 82 blocks for one specific input. Yet it routes six replicas' gradients to one owner, making backward 33–37% slower. Our reduction-aware planner accounts for this owner-side cost and avoids the slowdown.

We sincerely appreciate your careful review and insightful comments. Your feedback has motivated us to conduct substantial additional experiments and further clarify our technical contributions.

We hope these new results and explanations have addressed your concerns. If anything remains unclear or unresolved, we would be more than happy to discuss it further.

Thank you again for your valuable time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.