- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors show that sparse MoEs do not have to sacrifice performance compared with dense models when the total parameter count and training data are kept the same. This does require changes to the standard MoE architecture. Instead of using Top-k routing, it uses ReLU routing which allows different number of experts to be activated for each token and is almost differentiable everywhere, and then stabilize this with NormSiLU (RMS + intra-expert normalization), and an adaptive sparsity penalty that keeps the average activation ratio around a chosen target (20% in their case).
The main result is that DECO matches dense-model performance across a range of model sizes while activating only about 20% of its routed experts, and it consistently outperforms the other sparse MoE baselines they compare against. They also show that the sparsity leads to real inference speedups on hardware, not just lower theoretical compute. This kind of suggests that dense models (esp those using gated FFNs) have inherent sparsity and we can speed up inference if we always train MoEs.
Critical strengths
1 - Clear motivation - Fix gap between param / data matched dense and MoEs which means we could speed up inference if always train MoEs 2 - Thorough experimentation - Compared to several MoE baseliens across scales with ablations showing the motivation behind normsilu and adaptive sparsity
Critical weaknesses
No significant weakness in my opinion I think they could make it clear that training DECO is still more expensive because it requires extra compute due to the 'averaged' expert computation. Also, please make it clear if you are suggesting that non-gated MLPs are better in YOUR design or always better because that kind of contradicts the last several years of experimental validation saying the opposite.
Questions for authors
- How much more expensive is the training for DECO compared to Top-k with similar sparsity ratio
2 comments
Thank you for responding to my concerns. I will maintain my current score.
Rebuttal
Thank you for your excellent review! We will carefully address your concerns with detailed experiments and analyses.
We've already updated our manuscript to incorporate suggested revisions.
Weakness 1 & Question: Training cost versus sparse TopK
During training, our current implementation materializes the up-projection states of all routed experts so that inter-expert mean normalization can be computed. Therefore, DECO has higher theoretical expert-side FLOPs during training than genuinely sparse TopK at a comparable routed-expert activation ratio. However, this does not necessarily cause higher wall-clock training time.
We examined the training logs of DECO and genuinely sparse DeepSeek-V3 TopK baselines. All settings have a routed-expert activation ratio of around 20%, and are trained under the same Megatron-LM training stack. The training is conducted on NVIDIA A800 GPU machines, and the same number of GPUs is used within each scale. The median step times are:
These are wall-clock observations collected over 981 steady-state steps within the 5000–6000-step window, with matching logged batch size, sequence length, precision, world size, and parallel configuration within each scale. As shown in the table, the step time of DECO is about 0.43×–0.52× the step time of the sparse TopK baselines. In other words, under our training framework, DECO has higher theoretical expert FLOPs, but it is not more expensive in wall-clock training time.
A system-level explanation is that DECO converts all-expert computation to regular dense matrix multiplications over concatenated experts. By contrast, the sparse TopK execution must perform token dispatch, permutation, and grouped-GEMM operations. Gated experts of DeepSeek-V3 also add an additional linear projection and a multiplicative gate. These effects may outweigh the nominal FLOP advantage of sparse execution in this regime. Thus, DECO trains faster in wall-clock time despite materializing more theoretical expert FLOPs.
Notably, this is a hardware-specific and implementation-specific wall-clock observation. It should not be extrapolated to arbitrary hardware, expert counts, or parallel configurations. We do not claim that DECO can always be trained faster than sparse TopK, especially when the number of experts is extremely large.
At inference, the inter-expert mean is computed by one fixed average-weight projection rather than by evaluating every expert. Let Wi∈Rde×dh denote the up-projection matrix of expert i, then:
=\left(\frac{1}{E}\sum_i W_i\right)x =\bar W x.$$ The average weight $\bar W$ is precomputed and fixed during inference. The implementation therefore needs only one fixed average-weight projection and the selected experts' projections. **It does not evaluate any inactive expert outputs.** Therefore, the inference FLOPs of the fixed average-weight projection do not scale with the number of routed experts. Intra-expert RMS normalization is only applied to selected experts. # Weakness 2: Are non-gated MLPs generally better? As clarified in Section 4.4, our design choice is architecture-dependent: non-gated experts are particularly compatible with ReLU-based routing, as observed for DECO and ReMoE. The claim is supported by the ablation experiments in Table 4 and **is not a claim that non-gated MLPs universally replace gated MLPs**. For example, according to Table 4, DeepSeek-V3, which has a fixed routed-expert activation ratio per token, still favors gated MLP experts. Section 4.4 provides an empirical explanation: gated experts produce a less stable activation-ratio trajectory under ReLU routing, which requires stronger sparsity regularization. Figure 7 visualizes this behavior.