acceptodds
Under review as a conference paper at ICLR 2027

Mixture-of-Kittens: MoE Megakernel for NVL72s

Abstract

AI accelerator systems are rapidly consolidating into scale-up architectures, where tens to thousands of GPUs communicate over high-bandwidth, single-hop fabrics. We find that existing Mixture-of-Experts (MoE) training systems, optimized for conventional scale-out networks, transfer poorly to this setting, often running slower than a naive baseline built with PyTorch and NCCL. With industry roadmaps pointing toward even larger scale-up domains, understanding the performance tradeoffs of this hardware regime is increasingly important. We present Mixture-of-Kittens (MoK), an MoE training system designed for Nvidia NVL72. MoK builds on three insights that unlock performance on scale-up domains: (1) choosing push- or pull-based communication per operator, (2) restructuring the computation-communication overlap, and (3) fully eliminating CPU-GPU synchronization. MoK distills these insights into a single deterministic training megakernel that fuses token dispatch, shared and routed expert FFNs, and token combine. Across MoE layer shapes from four widely used open-weight models, MoK delivers up to the throughput of the strongest publicly available baseline. In a production run on 512 GPUs spanning multiple GB300 NVL72 racks, MoK improves end-to-end training throughput by .

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture-of-Experts, expert parallelism, distributed training, scale-up architectures, NVL72, megakernel

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FrryOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Mixture-of-Kittens (MoK), an MoE training system for NVIDIA NVL72. Based on three insights (push/pull communication, restructured computation–communication overlap, and eliminating CPU–GPU synchronization), it builds a single deterministic training megakernel.

Critical strengths
  • The paper identifies key limitations of existing MoE training systems on NVL72, derives three hardware-specific insights to address them, and explains the proposed design with clear, intuitive figures.
  • The system combines these insights in a single deterministic training megakernel.
  • The evaluation demonstrates substantial speedups across MoE layer shapes from widely used open-weight models and analyzes how the gains vary with the communication-to-computation balance, precision, pass, and local token count.
Critical weaknesses
  1. The paper’s applicability to other MoE architectures is unclear.\textbf{The paper’s applicability to other MoE architectures is unclear.}
    Can MoK be applied to architectures such as LatentMoE [1,2], Mixture-of-Head Attention[3], or SoftMoE[4]? If the proposed design is limited to a specific class of MoE architectures, such as FFN-based MoEs, the paper should clarify this scope such and reflect it in the title.

  2. The rationale for choosing communication directions in Section 3.1 is not sufficiently clear.\textbf{The rationale for choosing communication directions in Section 3.1 is not sufficiently clear.}
    Figure 4(right) shows that pull-based dispatch has much lower signaling overhead than push-based dispatch, with the gap increasing at higher EP degrees. This may lead readers to infer that pull is generally preferable. However, MoK uses different communication directions across operators (dispatch/combine) and passes (forward/backward). Although the paper states that each direction is chosen so that the GPU receiving the signal also issues the remote loads or stores, the connection between this criterion, the Figure 4 (right) results, and the operator-specific choices could be explained more explicitly. In particular, a clearer account of how the signaling costs differ across operators and between forward and backward passes would make the design rationale easier to follow.

  3. The relationship between EP degree and the NVL72 setting needs clarification.\textbf{The relationship between EP degree and the NVL72 setting needs clarification.}
    Figure 4(right) varies the EP degree on NVL72, while the text argues that scale-up platforms allow the entire EP group to fit within a scale-up domain. Explicitly connecting the evaluated EP degrees to the NVL72 configuration and explaining why this makes the comparison relevant to the proposed design would strengthen the motivation.

  4. The evaluation does not cover worst-case routing scenarios.\textbf{The evaluation does not cover worst-case routing scenarios.}
    In particular, it does not evaluate MoK under highly skewed routing, where a disproportionate share of tokens is assigned to a small subset of experts or ranks. Such workloads could cause buffer stalls on overloaded ranks and may increase activation recomputation during the backward pass due to ring buffer overwrite, but the resulting overhead and worst-case latency are not quantified.

[1] LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Expert
[2] Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
[3] MoH: Multi-Head Attention as Mixture-of-Head Attention
[4] SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs

Questions for authors
  1. Can MoK’s design be applied to scale-up systems beyond NVL72?\textbf{Can MoK’s design be applied to scale-up systems beyond NVL72?}
    The paper motivates its design through communication costs and CPU–GPU synchronization overheads on NVL72, but it is unclear which hardware characteristics are essential for MoK to be effective. Clarifying these requirements would help establish the design’s applicability to other scale-up systems.
8 comments
AuthorsOpenReview ↗

Response to Reviewer Frry [1/5]

Thank you for your thoughtful review, and we are very grateful for the time you put into providing detailed and thorough feedback. We also appreciate that you found our explanation to be clear and intuitive, and that our results show substantial speedups. We provide our response to your concerns below.

On applicability to other MoE architectures (weakness 1)

Thank you for raising this concern! We acknowledge that we were not sufficiently clear about MoK’s applicability to other MoE architectures in our paper. To clarify, MoK can indeed be applied to other types of MoE architectures. This is actually one of MoK’s strengths compared to previous works. Because compute work is decoupled from the computation-communication overlap schedule and control signals (Section 3.2), MoK can accommodate different expert computations while keeping its overall design.

To demonstrate this, we ported LatentMoE and SoftMoE to MoK’s computation pipeline, and we compare the performance against their original implementations below. We note that we excluded Mixture-of-Head Attention because attention introduces dependencies across tokens. Supporting these dependencies would require changes to the communication structure and scheduling assumptions used in MoK. While this is certainly possible and something we would do in production, such changes would make it difficult to isolate the effectiveness of the methods presented in MoK.

For each experiment, we select the largest model shape used in the original paper, use 2048 local tokens before routing, and use the highest EP degree possible. We report the median slowest-rank throughput in TFLOP/s over 100 iterations after 500 warmup iterations. Specifically, for LatentMoE, we use 512 routed experts and 2 shared experts with D = 4096, I = 2688, K = 24, and a compression ratio of 4, along with the squared ReLU activation function and EP degree 64. For SoftMoE, we use 32 routed experts with D = 640, I = 2560, and an average K = 1.5, along with EP degree 32. The rest of the benchmark setup is identical to what we described in Section 4.

ArchitecturePrecisionOriginal forwardMoK forwardOriginal backwardMoK backward
LatentMoEMXFP8291.8597.4702.21049.4
LatentMoEBF16300.0711.8507.3953.2
SoftMoEMXFP89.748.724.3109.9
SoftMoEBF1612.250.026.8116.3

Here, we can see that MoK is 1.49-2.37x faster than the original implementation of LatentMoE, and 4.10-5.02x faster for SoftMoE. This shows that MoK is not only efficient across MoE variants, but can also be easily used to significantly speed up less-optimized prototype architectures. We note that SoftMoE's TFLOP/s numbers are generally low because of its small expert dimensions and the low average number of active experts per token. As we stated in Section 4.1, MoK’s advantages grow in this regime, which is why the performance benefits are most significant for SoftMoE.

AuthorsOpenReview ↗

Response to Reviewer Frry [2/5]

On the rationale for choosing communication directions (weakness 2)

Thank you for your thoughtful feedback and for bringing this to our attention! We also agree that our description could misleadingly suggest that pull is generally preferable, which was not our intended claim. We clarify the rationale for choosing communication directions below, and will incorporate this clarification into the final revision.

MoK’s rationale for choosing the communication direction is that, for every communication operator in distributed MoE execution, the direction must keep signalling local. For signalling to stay local, the GPU performing the computation must also issue the transfers (i.e., remote loads or stores).

To elaborate, let us consider the two communication operators in the MoE forward pass: dispatch and combine. For dispatch, we must transfer each token from the source GPU to the GPU that owns the expert, to which the token is routed. We must make a decision here. Should the source GPU initiate the transfer (i.e., push)? Or should the expert-owning GPU initiate the transfer (i.e., pull)? We make this choice based on which GPU needs to receive the completion signal. Since the expert-owning GPU must receive the completion signal before performing expert computation on the token, choosing pull keeps the signalling local. That is, the expert-owning GPU initiates the transfer.

For combine, we must transfer tokens from the expert-owning GPU back to the source GPUs. Again, which GPU should initiate the transfer? Since the expert-owning GPU sends the completion signals after finishing all expert computation, making that same GPU send the tokens back to the source GPUs (i.e., push) keeps the signalling local.

The backward pass reverses these operations. Since reverse-combine requires the expert-owning GPUs to receive completion signals before performing computation, we must pull. Since reverse-dispatch requires completion signals from the expert-owning GPUs before initiating the transfers, we must push.

Figure 4 serves as an example to illustrate this rationale for only one of the four communication operators (i.e., dispatch). Because the signalling cost is symmetric across the four communication operators, we initially believed it would be redundant to present figures and microbenchmarks for all four of them. However, we acknowledge that this intention was insufficiently clear in our work. In the final revision, we will clarify the purpose of Figure 4 and further explain how this rationale leads to the specific direction choices in other communication operators.

AuthorsOpenReview ↗

Response to Reviewer Frry [3/5]

On the relationship between EP degree and the NVL72 setting (weakness 3)

We acknowledge that the relationship between EP degree and the NVL72 setup is not clearly specified in our paper, and we appreciate you giving us the opportunity to clarify this matter. We will first provide a brief background on the selection of EP degrees in NVL72 platforms, and then directly address your concerns.

Because there are 72 GB300 GPUs connected via a high-bandwidth interconnect inside a single NVL72 rack, we can theoretically increase the EP degree up to 72 within a single rack, and this would be the best way to fully utilize these platforms. However, in real-world settings, we find that it is often rare for all 72 GPUs inside an NVL72 to be fully functional. Thus, for fault-tolerance purposes, EP64 is usually selected as the highest power-of-two configuration, and the remaining GPUs in the rack are allocated for other uses (e.g., small research experiments, backups in case one of the selected 64 fails). This is why we choose EP64 as the highest EP configuration in all of our microbenchmarks and main evaluations. In addition, while the largest production training deployments usually use the highest EP degree possible for best performance, model research requires frequent, smaller runs, where a smaller EP degree is more appropriate. To demonstrate that MoK gives optimal performance for both small, quick research iterations and large training runs, we chose EP16 and EP64 as representative configurations in our main benchmarks.

To specifically address your concerns, the evaluated EP degrees in Figure 4 (i.e., 4, 8, 16, 32, 64) are common EP degree configurations that a researcher can select while keeping the entire EP group within a single scale-up domain. We evaluated various EP degrees in Figure 4 to show that the control overhead of push-based fine-grained communication increases with EP degree. As shown in the table, the overhead is worst at EP64, and we can extrapolate that it will become worse on future scale-up platforms with even more accelerators per domain. Thus, MoK’s proposed design becomes more effective as EP degree grows, which is critical because the largest training deployments will always try to utilize the entire scale-up domain.

We certainly agree that clarifying these points would strengthen the motivation of our paper, and we will incorporate these clarifications into the final revision.

Reviewer_KoGLOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper observes that large "scale-up" architectures, e.g., the NVL72, significantly alleviate the bandwidth bottlenecks typical of traditional "scale-out" systems. Consequently, performance bottlenecks have shifted increasingly toward control overheads, such as synchronization and scheduling. To address this shift, the authors propose a design focusing on three areas: communication control, the granularity of compute-communication coordination, and VRAM cache management. The authors provide a full system implementation, demonstrating improvements in end-to-end throughput. The experimental results are quite compelling.

Critical strengths
  • The paper identifies opportunities presented by new hardware and addresses a significant, valuable problem.
  • It includes ablation studies for each individual module, justifying the design choices made.
  • The system implementation is comprehensive, and the end-to-end experiments offer practical value.
Critical weaknesses
  • The paper lacks cross-module ablation studies.
  • The performance model is primarily explanatory and offers only limited predictive capability.
Questions for authors

Q1: From my understanding, the core shift addressed in this paper is the transition from a bandwidth-constrained regime to a control-constrained one. Since all three proposed modules target the control flow, explicitly stating this might improve clarity.

Q2: The effects of different modules may dependent of one another. For instance, the communication strategy and the mini-batch size could be coupled or serve as substitutes for one another. Could you provide cross-module ablations? This would make the findings more convincing.

Q3: In Fig. 5, the connections across mini-batches should be interpreted as control flow rather than data flow.

5 comments
AuthorsOpenReview ↗

Response to Reviewer KoGL [1/3]

Thank you for your positive and very insightful review! We sincerely appreciate your feedback, and we are also glad that you found our work to address a significant problem and provide comprehensive, practical value. We provide our response to your concerns and questions below.

On cross-module ablation studies (weakness 1 and question 2)

We appreciate you raising this concern! We acknowledge that the limited scope of ablation studies is one of the weaknesses of our paper. The main reason behind the shortage of cross-module studies is because each module is heavily entangled with the megakernel fusion, and we believed the cleanest way to show the effectiveness of each module is through micro-benchmarks in Sections 3.1, 3.2, and 3.3.

However, as you pointed out, the effects of each module could depend on one another, and it is indeed valuable to provide better understanding of such dependency in our work. We present our cross-module ablation studies here, and we have uploaded a revised version of our paper that includes them in Appendix H.

For each experiment, we used Kimi K2.7 model shape, MXFP8 precision, and EP degree of 64. We report the median slowest-rank TFLOP/s of 100 forward passes after 500 warmups. Unless otherwise noted, the remaining configurations are identical to that of Section 4.1. Please note that, because the underlying hardware drivers have changed since submission, we also remeasured MoK, so its numbers differ slightly from the paper.

Communication Direction and Overlap Scheduling (module 1 & 2)

Here, we analyze the dependence between the choice of communication direction and overlap schedules.

For communication direction, MoK selects pull-based dispatch and push-based combine for the forward pass, and pull-based reverse-combine and push-based reverse-dispatch for the backward pass. However, we could have chosen other directions, namely (1) push-based communication for all operators (“Push-only”), (2) pull-based communication for all operators (“Pull-only”), and (3) the opposite direction for all operators (“MoK-opposite”).

For computation-communication overlap scheduling, MoK uses the maximally deferred schedule described in our Figure 5, where communication occurs breadth-first and computation happens depth-first. Instead, we could have chosen other scheduling strategies, namely (1) “Comet-inspired”, where computation-communication overlap is coupled to specific operations (i.e., dispatch-GEMM1 and GEMM2-combine), and (2) “DeepEP-inspired”, where we always overlap the communication (dispatch or combine) of one microbatch with the computation of the other in lockstep.

We compare the performance of all unique combinations of communication direction and overlap scheduling below. Each row denotes a communication direction, and each column denotes a scheduling strategy. We select the best overlap granularity for each combination. The number after “@” refers to the selected granularity.

Direction \ SchedulingComet-inspiredDeepEP-inspiredMoK
Push-only615.3 @8192840.8 @4096679.6 @8192
Pull-only949.7 @2048900.4 @40961015.7 @2048
MoK-opposite562.6 @4096762.8 @4096613.6 @8192
MoK1147.7 @20481059.4 @40961247.4 @4096

We can make two observations from the above results. First, MoK's selection of communication directions (i.e., the last row) performs best regardless of the overlap schedule. This tells us that MoK's choice of direction is the more consistently beneficial component. Second, the optimal schedule depends on the direction. Specifically, we can see that the DeepEP-inspired schedule performs best in the Push-only and MoK-opposite rows. This is due to suboptimal communication directions increasing the control overhead, making lockstep execution more efficient than continuous fine-grained communication. Regardless, the best combination remains the current set of choices MoK makes for both communication direction and overlap scheduling (i.e., the bottom-right cell).

AuthorsOpenReview ↗

Response to Reviewer KoGL [2/3]

Independence of device-side buffer management (module 3)

Here, we show that the effects of device-side buffer management are largely independent of the other two modules.

To reiterate our Section 3.3, the effects of device-side ring buffering are twofold: (1) the CPU-GPU synchronization required for router-dependent memory allocation is eliminated. This is the positive effect, as avoiding CPU-GPU synchronization is critical for large-scale training on NVL72. (2) Ring buffer management and additional synchronization when wrapping around ring boundaries introduce overhead. This is the negative effect. We quantified this overhead in Figure 6. Ring buffering adds roughly 1.7% latency on geometric mean compared to using overallocated buffers, which are impractical but provide an ideal performance baseline.

The above two effects are largely independent of the other two modules. Specifically, the choices of communication direction and overlap schedule directly affect how computation and communication are performed. On the other hand, device-side ring buffering mainly concerns how memory is allocated and reused. To show this independence, we report the performance of every unique combination of variants of the three modules below, with host-side exact allocation (“Host-alloc”) serving as a replacement for device-side ring buffering (“Ring buffers”).

Overlap schedulingCommunication directionHost-allocRing buffers (MoK)
Comet-inspiredPush-only562.5615.3
Comet-inspiredPull-only851.4949.7
Comet-inspiredMoK-opposite510.8562.6
Comet-inspiredMoK995.31147.7
DeepEP-inspiredPush-only743.6840.8
DeepEP-inspiredPull-only799.6900.4
DeepEP-inspiredMoK-opposite672.2762.8
DeepEP-inspiredMoK937.01059.4
MoKPush-only607.6679.6
MoKPull-only886.21015.7
MoKMoK-opposite568.2613.6
MoKMoK1082.71247.4

Across all 24 combinations of overlap scheduling, communication direction, and buffer management scheme, device-side ring buffering is consistently 1.08-1.15x faster than host-side exact allocation. We can also see that the relative ordering of scheduling strategies and communication directions with regard to performance remains the same under both buffer management schemes. Thus, the benefit of device-side buffering is largely independent of the other two modules. We also note that the effects of host allocation get much worse as deployment scales to tens of thousands of GPUs.

On performance model (weakness 2)

Thank you for raising this point! We would like to first clarify that our cost model at the start of Section 3 is primarily there to provide readers who are not acquainted with GPU systems with a broad framing of how to break down GPU performance and which targets we are optimizing for. Our hope was that it would help them navigate through Section 3 by consistently connecting the explanations and methods with specific terms in the model. For example, it would be clearer to say that a specific method aims to minimize T_sync, rather than stating the same goal in words with slight variations in phrasing throughout the section.

To reiterate, the cost model is not intended to serve as a predictive model, and we do not claim it as a contribution of our work. In fact, building and formalizing a comprehensive model to predict the performance of GPU kernel designs is an area of research we are excited about going forward. We have made this distinction explicit in a footnote in Section 3 of the uploaded revision.

AuthorsOpenReview ↗

Response to Reviewer KoGL [3/3]

On the shift to control-constrained regime (question 1)

Your observation is correct, and we are grateful for your insightful comment, which helped us strengthen the framing of our work. We would like to further distinguish between (1) reducing control overhead and (2) optimizing hardware utilization through new control/data flow. Our understanding is that your point on the control-constrained regime leans more towards (1), which is certainly one bottleneck in scale-up architectures that we aim to address through MoK. Specifically, minimizing the signalling overhead through the selection of communication direction (Section 3.1) and eliminating CPU-GPU synchronization through ring buffers (Section 3.3) directly target this goal.

In addition, MoK also addresses the latter point (2) on optimizing hardware utilization through control/data flow. Specifically, traditional scale-out platforms often make distributed MoE execution primarily network-bound, whereas scale-up platforms place MoE near the boundary between being network-bound and compute-bound. For instance, consider running DeepSeek V4 Pro (i.e., 384 routed experts with D = 7168, I = 3072, K = 6) with 2,048 tokens per GPU before routing and EP 64. Using the theoretical compute throughput and network bandwidth on GB300s (i.e., 5 PFLOP/s for routed experts in FP8, 2.5 PFLOP/s for the shared expert in BF16, and 900 GB/s unidirectional network bandwidth), and assuming balanced routing across experts and ranks, the theoretical time for compute is 0.433 ms, and for network communication 0.385 ms. Because the actual wall-clock times for computation and communication can vary substantially depending on how the control/data flow is designed, this design becomes noticeably more significant, which is why we present the scheduling algorithm (Appendix B), as well as a design for fine-grained computation-communication overlap (Section 3.2).

We are certain that this largely aligns with your understanding, just described in a slightly different manner. We have clarified this distinction in the conclusion of the uploaded revision, to give readers a clearer unifying theme for the regime shift MoK is targeting.

On Figure 5 (question 3)

Thank you for bringing this to our attention. This is indeed a mistake on our side, and the solid arrows between minibatches should have been labelled “control flow” rather than “data flow”. We have fixed this in the uploaded revision.

Reviewer_vwYMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the training bottleneck of MoE under NVL72 single-hop scale-up fabric and proposes MoK, which fuses dispatch, routed/shared expert FFN, gated activation, and combine into a single deterministic megakernel. It selects push/pull direction per operator, makes overlap granularity a tunable parameter, and uses a device-side ring buffer to eliminate CPU-GPU synchronization. Evaluated on four real-world MoE shapes over GB300 NVL72, significant improvements are observed in both single-layer MXFP8 forward pass and 512-card throughput.

Critical strengths
  1. Most critically, the work presented in this paper is open-source, has been integrated into NeMo AutoModel, and is already in use for production training. The evaluation spans up to 512 GPUs, ensuring strong reproducibility and practical value at a considerable scale. The experiments cover real-world shapes and multiple models, including Kimi K2.7, GLM-5.2, Qwen3.5, and DeepSeek-V4-Pro. The system implementation is also comprehensive.
  2. This paper makes an interesting observation that, under NVL72, legacy MoE communication libraries sometimes underperform compared to PyTorch+NCCL, which carries practical significance. It also identifies that under scale-up fabric, pull-based dispatch can keep completion signaling local, which is a clear insight that contrasts with the conventional wisdom favoring push-based approaches under scale-out RDMA.
Critical weaknesses
  1. In my view, the most critical weakness is that the core contribution of this work essentially amounts to a hardware re-adaptation of existing components. All related components have mature implementations already; this paper primarily performs systems-level engineering integration rather than proposing new methods, theories, or models.
  2. The baseline comparisons are neither fair nor sufficiently complete. For instance, there is a lack of platform-matched comparisons against fused kernels such as FlashMoE, Comet, and MegaMoE. The three claimed insights are not isolated via ablation studies, and the fairness of the controlled comparisons in Section 4.2 is difficult to assess.
Questions for authors
  1. Compared to existing works such as FlashMoE, Comet, DeepGEMM, MegaMoE, and MegaBlocks, aside from system integration, does this paper present any definite innovation?
  2. Can additional experiments be added to provide a platform-matched comparison against Comet, FlashMoE, or MegaMoE-style kernels?
  3. Can the three claimed insights each be ablated independently (e.g., pull-only, fixed granularity, host-side allocation)?
  4. Can the bitwise consistency of Ring-aware forward replay be demonstrated either experimentally or theoretically?
  5. (Minor issue) The latter part of Table 2 appears to be identical to Table 1; a direct reference would suffice.
6 comments
AuthorsOpenReview ↗

Response to Reviewer vwYM [1/5]

Thank you for the critical and constructive review of our work! We are grateful for your feedback, which has helped us revisit and improve our work significantly. We carefully provide our response to your concerns below.

On novelty (weakness 1 and question 1)

We understand the concern and acknowledge that our paper did not sufficiently distinguish the methods introduced by MoK from the existing techniques it builds upon. We appreciate you raising this concern and giving us the opportunity to clarify.

We would like to first clarify that the methods we introduce in MoK would actually not work in non-scale-up systems, and to our knowledge, MoK is the first research work that specifically targets scale-ups for distributed MoE training. As briefly discussed in the paper, scale-up platforms introduce new opportunities as well as a different cost structure, such as pull being as cheap as push, flexibility in overlap granularity, and CPU-GPU synchronization being much more expensive. In previous generations of hardware, exploiting these characteristics in large-scale MoE training was not possible; all of the cited works either evaluate small-scale deployments or are inference-only. Scale-ups opened up these opportunities, and through MoK we introduced new primitives that exploit them.

To illustrate this, we provide some concrete distinctions from previous works below. We note that this is not a comprehensive list.

MegaMoE

  • Most importantly, MegaMoE is an inference kernel, whereas MoK presents methods for MoE training. MoK provides a set of methods for supporting backward execution, which requires a separate schedule from the forward pass due to the additional computation and communication dependencies (e.g., activation and weight gradients, collecting router gradients, quantization, and forward replay).
  • We also emphasize that supporting training changes the design requirements of the forward pass as well. For instance, training forward requires additional work that inference forward does not, and MegaMoE cannot accommodate this without introducing resource contention. As an example, MoK’s training MXFP8 forward generates both normal and transposed quantized activations with separately computed block scales, for numerically stable weight-gradient computation. In MegaMoE’s warp-specialized design, adding this work to dispatch would compete with the colocated GEMM workers for registers and shared memory. MoK instead schedules dispatch and its fused quantization on dedicated communication SMs, separating their resource requirements from the GEMM pipelines.
  • Furthermore, MoK separates communication-computation overlap granularity from GEMM tile granularity through its minibatch schedule (Section 3.2), whereas MegaMoE fixes communication granularity to the GEMM block size. Because of this, only MoK can tune the amount of work between communication-computation handoffs (Figure 5) without changing the underlying GEMM block granularity. As shown in Figure 5, the optimal computation-communication overlap granularity is workload-dependent, so this flexibility is critical.
  • Finally, memory pressure is much higher during training forward than inference forward due to all the training states (e.g., optimizer states, master weights) and intermediate tensors that have to be kept for the backward pass. Depending on the setup, memory usage during training forward could be roughly 5x to 10x that of inference forward. This is why the set of techniques MoK provides, such as fused forward replay or reversed macrobatch processing, is valuable specifically for the training setup.
AuthorsOpenReview ↗

Response to Reviewer vwYM [2/5]

Comet

  • Comet’s scheduling algorithm (1) uses tile-level overlap, which our results show is not always optimal for scale-up workloads (Section 3.2), and (2) separates MoE execution into two distinct stages, each fusing one expert computation stage with communication, which confines computation-communication overlap to the respective stages. For instance, the widely used gated-activation FFNs become ineffective because the additional computation in the up and gate projections cannot be used to hide communication in the subsequent combine stage. On the other hand, MoK’s own minibatch schedule, detailed in Section 3.2, (1) supports any overlap granularity, and (2) enables overlap across stage boundaries (e.g., by allowing the combine of an earlier minibatch to overlap with computation on later minibatches). Furthermore, because MoK’s schedule does not assume a fixed internal structure for expert computation, MoK can also accommodate a wide range of MoE architectures. We show how all of these differences lead to performance degradation with additional experiments below.
  • Comet’s execution path includes routing-dependent host-side scheduling, which becomes costly on NVL72 given the slower host execution measured in Section 3.3 and further shown by the additional experiments below. Resolving this requires moving scheduling and buffer management onto the GPU. Only MoK enables this through its own device-side scheduling algorithm detailed in Appendix B, its device-managed token ring buffer (Section 3.3), and the reconstruction of overwritten activations fused into the backward kernel. These methods collectively enable fully GPU-resident execution of both MoE forward and backward on NVL72s.

FlashMoE

  • Again, FlashMoE does not support backward execution, and its forward design limits its usability in large-scale training. This is because FlashMoE requires worst-case memory overallocation to guarantee dropless execution (i.e., tokens exceeding the pre-configured capacity are dropped). MoK bounds memory usage without dropping tokens through its device-side ring buffering algorithm (Section 3.3). Furthermore, FlashMoE couples output-tile transfers with signals to a receiving rank; however, as shown in our Section 3.1, signalling overhead becomes critical on scale-up architectures. Only MoK addresses this by choosing communication directions that keep communication-computation handoffs local.

MegaBlocks

  • Most importantly, MegaBlocks executes expert computation sequentially with dispatch and combine, and thus does not perform any computation-communication overlap. In addition, MegaBlocks supports dropless execution at the cost of routing-dependent CPU-GPU synchronization and host-side exact memory allocation. Such a scheme, as shown in Section 3.3 and the additional experiments below, degrades performance on NVL72 platforms and was in fact our core motivation for developing MoK. MoK completely avoids this CPU-GPU synchronization while still preserving every token.

DeepGEMM

  • For DeepGEMM, we respectfully disagree that the work is directly comparable to MoK. The main contribution of the original DeepGEMM work, excluding the recently added MegaMoE, is optimized kernels for expert-grouped GEMMs, without other components of distributed MoE execution such as route scheduling, dispatch, combine, quantization, overlap, and more. DeepGEMM’s contributions are actually complementary to MoK. In fact, if we wish to utilize DeepGEMM’s 128x128 FP8 block-scaling format, we can use DeepGEMM as MoK’s expert computation engine.

Finally, we respect the maturity and contributions of the previous systems. However, we respectfully disagree that all related components of our work already have mature implementations for the following three reasons: (1) The cited works, except for DeepGEMM/MegaMoE, do not run on NVL72s, and systems that do support NVL72 have substantially suboptimal performance, as shown in our Section 4. (2) Although not discussed in our main text, we provide production-specific features that are not present in the cited works, such as RDMA overlap or MXFP8 quantization fused into the megakernel (Appendix D). (3) We validated our work on large-scale setups, with a 64-GPU single-rack setup for all standalone benchmarks and 512-GPU large-scale runs.

AuthorsOpenReview ↗

Response to Reviewer vwYM [3/5]

On platform-matched baselines (weakness 2 and question 2)

Thank you for raising this concern! We would first like to explain the reasons why we did not include platform-matched baselines in our paper. There are two main reasons:

  1. In GPU systems work such as Comet, FlashMoE, or DeepGEMM, the presented methods themselves are tightly coupled to the hardware platform they are targeting, because each hardware generation is very different in its execution model. For instance, one of the key methods in the original DeepGEMM work is overlapping the FP8 partial dequantization with matrix multiplication, which relies on Hopper’s asynchronous tensor core execution accumulating results in register tiles. However, the newer Blackwell GPUs accumulate in separate on-chip memory that does not support software arithmetic operations, and we found last year that naively porting the original DeepGEMM to Blackwell results in no speedup despite Blackwell tensor cores having twice the compute throughput. Likewise, FlashMoE and Comet rely on pipeline schedules designed for earlier hardware (e.g., Hopper’s ping-pong asynchronous wgmma execution or Ampere’s synchronous warp-level tensor core execution). Adapting these works to Blackwell’s execution model would involve arbitrary choices about buffering, synchronization, and overlap between compute and epilogue/communication work. It would rely heavily on our own judgment about what is a faithful extrapolation of their methods to newer hardware.

  2. Even for systems that do support Blackwell (e.g., MegaMoE), the comparison is not fair unless they are intended for MoE training. As explained in the previous section, training forward requires additional compute/memory work and is more memory-limited than inference forward. This constrains the set of optimizations that can be applied to the training forward kernel, and directly comparing the two works would thus mix the differences in workload requirements with differences in the methods. If we want to align the comparisons, we would have to make changes to the baseline such that either (1) it no longer preserves its core methods (e.g., abandoning the warp-specialized design to accommodate the additional work), or (2) it is significantly disadvantaged by the additional work (e.g., keeping the warp-specialized design and causing register spilling). Furthermore, we would have to implement a new method for backward, which would again depend largely on our judgment about what would be a faithful extrapolation of baseline’s forward methods.

In addition, we would like to emphasize that new hardware generations have opened new opportunities while closing existing ones, which led to many widely used research works in the past. One famous example is FlashAttention. FlashAttention 3 (built for Hopper) was a hardware-specific adaptation of the same overall algorithm described in FlashAttention 2 (built for Ampere), exploiting Hopper’s asynchronous tensor cores and TMA. The same applies to FlashAttention 4 (built for Blackwell). These works were valuable because new generations of AI accelerators introduce substantial architectural changes (e.g., Hopper’s wgmma instructions versus Blackwell’s tcgen05 instructions), and adapting existing systems to new hardware is not trivial, often requiring new primitives and methods for mapping the same operator to the new hardware.

To conclude, we respectfully do not believe that platform-matched baseline comparisons would add meaningful value. Instead, we believe end-to-end system ablations of each method that MoK presents, against other variants including those drawn from previous works, would better serve this purpose. We provide such additional experiments below.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response to All Reviewers

We sincerely thank all the reviewers for their time and effort reviewing our work and for their constructive comments and questions, which have helped us strengthen our paper. We also appreciate the reviewers’ recognition of the “significant, valuable problem” MoK addresses and the “practical significance” of our insights into the opportunities presented by scale-up hardware [KoGL, vwYM]. Reviewers further highlighted MoK’s “significant”, “substantial”, and “compelling” performance improvements across widely used MoE models [vwYM, KoGL, Frry]. We are also happy that they recognized MoK’s “practical value”, emphasizing our open-source release, integration into Nvidia AutoModel, use in production training, and “strong reproducibility” at a “considerable scale” [vwYM, KoGL]. Finally, we are grateful that the reviewers found the presentation of our work easy to follow and our figures “clear and intuitive” [KoGL, Frry].

In this common response, we reiterate our core contributions to clarify the overall purpose of our work and provide a reference point for discussing individual concerns.

Summary of contributions

We study how the emerging scale-up hardware platforms change the design tradeoffs of MoE training, as well as why legacy systems often underperform in this setting. We identify three key insights to exploit new hardware capabilities and translate them into MoK, a deterministic MoE training megakernel that fuses token dispatch, expert computation, and token combine. Our contributions are:

  1. Demonstration of new opportunities in emerging hardware platforms. As described in Section 1, major vendors (Nvidia, AMD, Google, AWS) are moving toward increasingly large scale-up architectures. These platforms fundamentally change the memory and networking characteristics assumed by existing AI systems, making it important to revisit established design choices. Our study of MoE training on NVL72 provides an early, concrete demonstration of these opportunities, with measured performance improvements in large-scale deployments. These results offer insight into how systems should be built for the next generation of frontier AI infrastructure.

  2. Identifying 3 reusable insights and demonstrating their effects in a fully working system. We identify communication direction, computation-communication overlap granularity, and device-managed buffering as 3 key insights for unlocking performance on scale-up architectures. For each insight, we (1) develop novel algorithms and scheduling designs that materialize the insight, (2) demonstrate its effectiveness through microexperiments, and (3) integrate it into a complete, production-ready system with significant end-to-end speedups.

  3. End-to-end MoE training megakernel validated in production. We emphasize that turning GPU system designs into effective implementations at scale is a substantial challenge on its own. Ideas that appear promising in principle often do not lead to practical gains. We open-source the full implementation of MoK, which has been (1) validated and used in large-scale production training of a frontier AI model, (2) integrated into the broader open-source ecosystem through Nvidia AutoModel, and (3) evaluated using open-source benchmarks, showing up to 2.37x the throughput of the strongest baseline and a 1.41x end-to-end training speedup on 512 GPUs.

Responses to individual reviewers

Because all 3 reviews raise distinct concerns and questions, we address each reviewer’s specific points separately. Please find our detailed responses in the respective review threads.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.