- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Mixture-of-Kittens (MoK), an MoE training system for NVIDIA NVL72. Based on three insights (push/pull communication, restructured computation–communication overlap, and eliminating CPU–GPU synchronization), it builds a single deterministic training megakernel.
Critical strengths
- The paper identifies key limitations of existing MoE training systems on NVL72, derives three hardware-specific insights to address them, and explains the proposed design with clear, intuitive figures.
- The system combines these insights in a single deterministic training megakernel.
- The evaluation demonstrates substantial speedups across MoE layer shapes from widely used open-weight models and analyzes how the gains vary with the communication-to-computation balance, precision, pass, and local token count.
Critical weaknesses
-
Can MoK be applied to architectures such as LatentMoE [1,2], Mixture-of-Head Attention[3], or SoftMoE[4]? If the proposed design is limited to a specific class of MoE architectures, such as FFN-based MoEs, the paper should clarify this scope such and reflect it in the title. -
Figure 4(right) shows that pull-based dispatch has much lower signaling overhead than push-based dispatch, with the gap increasing at higher EP degrees. This may lead readers to infer that pull is generally preferable. However, MoK uses different communication directions across operators (dispatch/combine) and passes (forward/backward). Although the paper states that each direction is chosen so that the GPU receiving the signal also issues the remote loads or stores, the connection between this criterion, the Figure 4 (right) results, and the operator-specific choices could be explained more explicitly. In particular, a clearer account of how the signaling costs differ across operators and between forward and backward passes would make the design rationale easier to follow. -
Figure 4(right) varies the EP degree on NVL72, while the text argues that scale-up platforms allow the entire EP group to fit within a scale-up domain. Explicitly connecting the evaluated EP degrees to the NVL72 configuration and explaining why this makes the comparison relevant to the proposed design would strengthen the motivation. -
In particular, it does not evaluate MoK under highly skewed routing, where a disproportionate share of tokens is assigned to a small subset of experts or ranks. Such workloads could cause buffer stalls on overloaded ranks and may increase activation recomputation during the backward pass due to ring buffer overwrite, but the resulting overhead and worst-case latency are not quantified.
[1] LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Expert
[2] Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
[3] MoH: Multi-Head Attention as Mixture-of-Head Attention
[4] SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs
Questions for authors
-
The paper motivates its design through communication costs and CPU–GPU synchronization overheads on NVL72, but it is unclear which hardware characteristics are essential for MoK to be effective. Clarifying these requirements would help establish the design’s applicability to other scale-up systems.
8 comments
Response to Reviewer Frry [2/5]
On the rationale for choosing communication directions (weakness 2)
Thank you for your thoughtful feedback and for bringing this to our attention! We also agree that our description could misleadingly suggest that pull is generally preferable, which was not our intended claim. We clarify the rationale for choosing communication directions below, and will incorporate this clarification into the final revision.
MoK’s rationale for choosing the communication direction is that, for every communication operator in distributed MoE execution, the direction must keep signalling local. For signalling to stay local, the GPU performing the computation must also issue the transfers (i.e., remote loads or stores).
To elaborate, let us consider the two communication operators in the MoE forward pass: dispatch and combine. For dispatch, we must transfer each token from the source GPU to the GPU that owns the expert, to which the token is routed. We must make a decision here. Should the source GPU initiate the transfer (i.e., push)? Or should the expert-owning GPU initiate the transfer (i.e., pull)? We make this choice based on which GPU needs to receive the completion signal. Since the expert-owning GPU must receive the completion signal before performing expert computation on the token, choosing pull keeps the signalling local. That is, the expert-owning GPU initiates the transfer.
For combine, we must transfer tokens from the expert-owning GPU back to the source GPUs. Again, which GPU should initiate the transfer? Since the expert-owning GPU sends the completion signals after finishing all expert computation, making that same GPU send the tokens back to the source GPUs (i.e., push) keeps the signalling local.
The backward pass reverses these operations. Since reverse-combine requires the expert-owning GPUs to receive completion signals before performing computation, we must pull. Since reverse-dispatch requires completion signals from the expert-owning GPUs before initiating the transfers, we must push.
Figure 4 serves as an example to illustrate this rationale for only one of the four communication operators (i.e., dispatch). Because the signalling cost is symmetric across the four communication operators, we initially believed it would be redundant to present figures and microbenchmarks for all four of them. However, we acknowledge that this intention was insufficiently clear in our work. In the final revision, we will clarify the purpose of Figure 4 and further explain how this rationale leads to the specific direction choices in other communication operators.
Response to Reviewer Frry [3/5]
On the relationship between EP degree and the NVL72 setting (weakness 3)
We acknowledge that the relationship between EP degree and the NVL72 setup is not clearly specified in our paper, and we appreciate you giving us the opportunity to clarify this matter. We will first provide a brief background on the selection of EP degrees in NVL72 platforms, and then directly address your concerns.
Because there are 72 GB300 GPUs connected via a high-bandwidth interconnect inside a single NVL72 rack, we can theoretically increase the EP degree up to 72 within a single rack, and this would be the best way to fully utilize these platforms. However, in real-world settings, we find that it is often rare for all 72 GPUs inside an NVL72 to be fully functional. Thus, for fault-tolerance purposes, EP64 is usually selected as the highest power-of-two configuration, and the remaining GPUs in the rack are allocated for other uses (e.g., small research experiments, backups in case one of the selected 64 fails). This is why we choose EP64 as the highest EP configuration in all of our microbenchmarks and main evaluations. In addition, while the largest production training deployments usually use the highest EP degree possible for best performance, model research requires frequent, smaller runs, where a smaller EP degree is more appropriate. To demonstrate that MoK gives optimal performance for both small, quick research iterations and large training runs, we chose EP16 and EP64 as representative configurations in our main benchmarks.
To specifically address your concerns, the evaluated EP degrees in Figure 4 (i.e., 4, 8, 16, 32, 64) are common EP degree configurations that a researcher can select while keeping the entire EP group within a single scale-up domain. We evaluated various EP degrees in Figure 4 to show that the control overhead of push-based fine-grained communication increases with EP degree. As shown in the table, the overhead is worst at EP64, and we can extrapolate that it will become worse on future scale-up platforms with even more accelerators per domain. Thus, MoK’s proposed design becomes more effective as EP degree grows, which is critical because the largest training deployments will always try to utilize the entire scale-up domain.
We certainly agree that clarifying these points would strengthen the motivation of our paper, and we will incorporate these clarifications into the final revision.
Response to Reviewer Frry [1/5]
Thank you for your thoughtful review, and we are very grateful for the time you put into providing detailed and thorough feedback. We also appreciate that you found our explanation to be clear and intuitive, and that our results show substantial speedups. We provide our response to your concerns below.
On applicability to other MoE architectures (weakness 1)
Thank you for raising this concern! We acknowledge that we were not sufficiently clear about MoK’s applicability to other MoE architectures in our paper. To clarify, MoK can indeed be applied to other types of MoE architectures. This is actually one of MoK’s strengths compared to previous works. Because compute work is decoupled from the computation-communication overlap schedule and control signals (Section 3.2), MoK can accommodate different expert computations while keeping its overall design.
To demonstrate this, we ported LatentMoE and SoftMoE to MoK’s computation pipeline, and we compare the performance against their original implementations below. We note that we excluded Mixture-of-Head Attention because attention introduces dependencies across tokens. Supporting these dependencies would require changes to the communication structure and scheduling assumptions used in MoK. While this is certainly possible and something we would do in production, such changes would make it difficult to isolate the effectiveness of the methods presented in MoK.
For each experiment, we select the largest model shape used in the original paper, use 2048 local tokens before routing, and use the highest EP degree possible. We report the median slowest-rank throughput in TFLOP/s over 100 iterations after 500 warmup iterations. Specifically, for LatentMoE, we use 512 routed experts and 2 shared experts with D = 4096, I = 2688, K = 24, and a compression ratio of 4, along with the squared ReLU activation function and EP degree 64. For SoftMoE, we use 32 routed experts with D = 640, I = 2560, and an average K = 1.5, along with EP degree 32. The rest of the benchmark setup is identical to what we described in Section 4.
Here, we can see that MoK is 1.49-2.37x faster than the original implementation of LatentMoE, and 4.10-5.02x faster for SoftMoE. This shows that MoK is not only efficient across MoE variants, but can also be easily used to significantly speed up less-optimized prototype architectures. We note that SoftMoE's TFLOP/s numbers are generally low because of its small expert dimensions and the low average number of active experts per token. As we stated in Section 4.1, MoK’s advantages grow in this regime, which is why the performance benefits are most significant for SoftMoE.