acceptodds
Under review as a conference paper at ICLR 2027

DECO: Closing the Gap Between Sparse MoE and Dense at Equal Total Parameters

Abstract

While sparse Mixture-of-Experts (MoE) scales model capacity without proportionally increasing computation, its massive total parameter footprint creates significant storage and memory-access bottlenecks, which hinder efficient edge-device deployment that simultaneously requires high performance, low computational cost, and small storage overhead. To achieve these properties, we present **DECO**, a **DE**nse-**CO**mpetitive sparse MoE architecture that closes the gap between existing MoE designs and dense Transformers, under a matched total parameter budget and an identical number of training tokens. We argue that this gap is not an inherent cost of sparsity but an artifact of how existing designs enforce it: some impose a hard TopK selection that blocks gradients and fixes one activation ratio for every token, others apply a heavy sparsity regularization that costs performance. DECO instead lets sparsity arise from the architecture itself through differentiable ReLU-based routing, along with learnable expert-wise scaling that balances routed against shared experts. Its experts are non-gated MLPs, and their activation function, NormSiLU, keeps the routed-expert activation ratio low and stable, so only light regularization is needed. Experiments demonstrate that the established MoE baselines consistently leave a performance gap to an equally-budgeted dense model, and DECO removes it while activating only 20% of routed experts across six parameter scales from 0.11B to 7B. Our specialized acceleration kernel delivers a 2.93 speedup on Jetson AGX Orin compared with dense inference. Code and checkpoints will be released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture-of-experts, edge-side LLM, inference acceleration

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_8YbXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors show that sparse MoEs do not have to sacrifice performance compared with dense models when the total parameter count and training data are kept the same. This does require changes to the standard MoE architecture. Instead of using Top-k routing, it uses ReLU routing which allows different number of experts to be activated for each token and is almost differentiable everywhere, and then stabilize this with NormSiLU (RMS + intra-expert normalization), and an adaptive sparsity penalty that keeps the average activation ratio around a chosen target (20% in their case).

The main result is that DECO matches dense-model performance across a range of model sizes while activating only about 20% of its routed experts, and it consistently outperforms the other sparse MoE baselines they compare against. They also show that the sparsity leads to real inference speedups on hardware, not just lower theoretical compute. This kind of suggests that dense models (esp those using gated FFNs) have inherent sparsity and we can speed up inference if we always train MoEs.

Critical strengths

1 - Clear motivation - Fix gap between param / data matched dense and MoEs which means we could speed up inference if always train MoEs 2 - Thorough experimentation - Compared to several MoE baseliens across scales with ablations showing the motivation behind normsilu and adaptive sparsity

Critical weaknesses

No significant weakness in my opinion I think they could make it clear that training DECO is still more expensive because it requires extra compute due to the 'averaged' expert computation. Also, please make it clear if you are suggesting that non-gated MLPs are better in YOUR design or always better because that kind of contradicts the last several years of experimental validation saying the opposite.

Questions for authors
  • How much more expensive is the training for DECO compared to Top-k with similar sparsity ratio
2 comments

Thank you for your excellent review! We will carefully address your concerns with detailed experiments and analyses.

We've already updated our manuscript to incorporate suggested revisions.

Weakness 1 & Question: Training cost versus sparse TopK

During training, our current implementation materializes the up-projection states of all routed experts so that inter-expert mean normalization can be computed. Therefore, DECO has higher theoretical expert-side FLOPs during training than genuinely sparse TopK at a comparable routed-expert activation ratio. However, this does not necessarily cause higher wall-clock training time.

We examined the training logs of DECO and genuinely sparse DeepSeek-V3 TopK baselines. All settings have a routed-expert activation ratio of around 20%, and are trained under the same Megatron-LM training stack. The training is conducted on NVIDIA A800 GPU machines, and the same number of GPUs is used within each scale. The median step times are:

ScaleDECOSparse TopK, gated expertsSparse TopK, non-gated experts
0.24B1566.5 ms3106.2 ms3045.1 ms
0.53B2822.1 ms6219.9 ms6515.0 ms

These are wall-clock observations collected over 981 steady-state steps within the 5000–6000-step window, with matching logged batch size, sequence length, precision, world size, and parallel configuration within each scale. As shown in the table, the step time of DECO is about 0.43×–0.52× the step time of the sparse TopK baselines. In other words, under our training framework, DECO has higher theoretical expert FLOPs, but it is not more expensive in wall-clock training time.

A system-level explanation is that DECO converts all-expert computation to regular dense matrix multiplications over concatenated experts. By contrast, the sparse TopK execution must perform token dispatch, permutation, and grouped-GEMM operations. Gated experts of DeepSeek-V3 also add an additional linear projection and a multiplicative gate. These effects may outweigh the nominal FLOP advantage of sparse execution in this regime. Thus, DECO trains faster in wall-clock time despite materializing more theoretical expert FLOPs.

Notably, this is a hardware-specific and implementation-specific wall-clock observation. It should not be extrapolated to arbitrary hardware, expert counts, or parallel configurations. We do not claim that DECO can always be trained faster than sparse TopK, especially when the number of experts is extremely large.

At inference, the inter-expert mean is computed by one fixed average-weight projection rather than by evaluating every expert. Let Wi∈Rde×dhW_i\in\mathbb{R}^{d_e\times d_h} denote the up-projection matrix of expert ii, then:

=\left(\frac{1}{E}\sum_i W_i\right)x =\bar W x.$$ The average weight $\bar W$ is precomputed and fixed during inference. The implementation therefore needs only one fixed average-weight projection and the selected experts' projections. **It does not evaluate any inactive expert outputs.** Therefore, the inference FLOPs of the fixed average-weight projection do not scale with the number of routed experts. Intra-expert RMS normalization is only applied to selected experts. # Weakness 2: Are non-gated MLPs generally better? As clarified in Section 4.4, our design choice is architecture-dependent: non-gated experts are particularly compatible with ReLU-based routing, as observed for DECO and ReMoE. The claim is supported by the ablation experiments in Table 4 and **is not a claim that non-gated MLPs universally replace gated MLPs**. For example, according to Table 4, DeepSeek-V3, which has a fixed routed-expert activation ratio per token, still favors gated MLP experts. Section 4.4 provides an empirical explanation: gated experts produce a less stable activation-ratio trajectory under ReLU routing, which requires stronger sparsity regularization. Figure 7 visualizes this behavior.

Thank you for responding to my concerns. I will maintain my current score.

Reviewer_vhviOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DECO, a sparse Mixture-of-Experts (MoE) architecture designed to match the performance of dense Transformers under equal total parameter budgets and training tokens while reducing inference computation. DECO combines ReLU-based routing, learnable expert-wise scaling, non-gated MLP experts with a proposed NormSiLU activation function, and adaptive entropy-based sparsity regularization. The authors argue that the performance gap between conventional MoEs and dense models primarily stems from non-differentiable TopK routing and excessive sparsity regularization. Experiments across six model scales (0.11B–7B) show that DECO achieves approximately dense-level performance while activating only 20% of routed experts, and consistently outperforms the evaluated MoE baselines. The paper also reports a 2.93× inference speedup over dense execution on Jetson AGX Orin.

Critical strengths
  1. Meaningful problem formulation. The paper investigates an important question regarding the performance–computation–storage tradeoff of sparse MoEs. Comparing architectures under matched total parameter budgets and training tokens provides a relevant perspective for resource-constrained deployment.

  2. Relatively comprehensive empirical evaluation. The paper evaluates DECO across six model scales and compares it against several established MoE architectures. The consistent improvements over the included baselines, together with ablation studies on architectural components, provide useful empirical evidence for the proposed design.

Critical weaknesses
  1. Insufficient justification of the proposed mechanisms and questionable theoretical support.

The paper argues that conventional MoEs underperform dense models because TopK routing blocks gradients and alternative sparse routing methods require excessive regularization. However, this causal explanation is not convincingly established. DECO simultaneously changes routing, expert architecture, activation functions, and regularization, making it difficult to attribute the improvements to the proposed mechanisms. Moreover, ReLU routing still produces zero gradients for inactive experts through their routing scores, so its advantage over TopK requires a more precise explanation. The theoretical analysis of NormSiLU in Appendix I is also concerning: the gradient bound is derived for global LayerNorm rather than the actual two-stage normalization used in DECO. In addition, inter-expert mean subtraction can introduce gradient coupling between active and inactive experts, an important property that is neither analyzed nor experimentally isolated. Consequently, the paper provides limited evidence explaining why its architectural choices lead to better performance, rather than merely demonstrating that their combination works.

  1. Experimental evidence does not fully support the central performance and efficiency claims.

Although DECO achieves competitive results, the evidence for dense-level parity remains statistically limited. Five-seed experiments are conducted only at 0.11B and 0.24B, and the averaged 0.11B perplexity actually favors the dense baseline, whereas larger-scale results rely on single runs. More importantly, the reported 2.93× inference speedup is obtained using a separate deployment-oriented configuration in which approximately 90% of non-embedding parameters belong to routed experts, unlike the architectures evaluated in the main experiments. Furthermore, the dense execution baseline in this benchmark computes all experts using the same MoE weights rather than evaluating the independently trained dense Transformer. Therefore, the speedup does not directly establish that DECO achieves dense-level performance with substantially lower inference latency under the same experimental configuration. These inconsistencies weaken the paper's central claim of simultaneously achieving high performance, low computation, and small storage overhead.

Questions for authors

Q1. Can the authors provide controlled ablations that isolate the effects of routing differentiability, sparsity regularization, and expert architecture? In particular, how can the observed performance gap be attributed to TopK gradient blocking rather than other architectural differences?

Q2. Does the theoretical gradient bound in Appendix I hold for the actual NormSiLU implementation? How does inter-expert mean normalization affect gradients to inactive experts, and is the cross-expert mean detached during training?

Q3. Can the authors clarify why the inference benchmark uses a different expert-parameter distribution from the main experiments? Would the reported speedups persist when comparing the actual trained DECO checkpoints against their corresponding dense Transformer baselines?

Q4. Can the authors provide stronger statistical evidence for dense-level performance, particularly at larger scales? How robust are the conclusions across training seeds and different data distributions?

3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Thank you for your excellent review! We will carefully address your concerns with detailed experiments and analyses.

We've already updated our manuscript to incorporate suggested revisions.

Weakness 1.1 & Q1: Controlled ablations and attribution to TopK

First, we want to emphasize that our central claim is that DECO achieves dense-comparable performance under matched total parameters and training tokens. We do not claim universal superiority over Dense, nor do we claim that TopK gradient blocking is the sole cause of the dense–MoE gap.

To address your question concerning routing and sparsity regularization, we conducted controlled experiments at 0.11B and 0.24B while retaining DECO's expert architecture at each scale: 42 or 57 routed experts of width 64, one shared expert of width 128, and non-gated NormSiLU experts. All runs use 15,000 training steps and the same training-token budget within each scale.

Setting0.11B C4 loss ↓0.11B C4 BPB ↓0.24B C4 loss ↓0.24B C4 BPB ↓
DECO (ours): ReLU + router entropy3.5367301.1880113.3228851.116179
DECO: ReLU + ReMoE-style L13.5451021.1908233.3432371.123016
DECO: hard TopK (activation 8/42 or 11/57)3.5475191.1916353.3477891.124545

As shown by the table, "ReLU + router entropy" has the lowest C4 loss at both scales. All settings have a routed-expert activation ratio of around 20%. The L1 variant retains DECO's ReLU routing and expert architecture while replacing only the sparsity regularizer. The hard-TopK variant replaces the routing mechanism with DeepSeekMoE-style hard selection.

The expert-architecture dimension is already covered by component-wise ablations in the submitted paper. Table 3 studies the contributions of inter-expert mean normalization and intra-expert RMS normalization. Table 4 compares gated and non-gated expert MLPs within DECO. Table 1 compares the standard Dense baseline with a Dense model that retains DECO's expert architecture after removing routing. Table 2 separately studies expert-wise router scaling. Together with the new L1 and hard-TopK comparisons, these results provide component-wise evidence for our main design choices.

Weakness 1.2 & Q2: NormSiLU theory and gradients to inactive experts

The mean is intentionally attached during training. Let AA be the active expert set and WjW_j be the up-projection weight of expert jj. Following Algorithm 1, let zj=Wjx∈Rdez_j=W_jx\in\mathbb{R}^{d_e} denote the up-projected state of expert jj. DECO computes the per-feature cross-expert mean m=1E∑j=1Ezj∈Rdem=\frac{1}{E}\sum_{j=1}^{E}z_j\in\mathbb{R}^{d_e}, and applies ui=zi−mu_i=z_i-m to each active expert ii. If gi=∂L/∂uig_i=\partial\mathcal{L}/\partial u_i for an active expert ii, then, coordinate-wise, the backward relation through the mean stage is:

=\mathbf{1}[j\in A]g_j-\frac{1}{E}\sum_{i\in A}g_i.$$ The subtraction is applied independently for each expert feature coordinate. Thus, an inactive expert's up-projection can receive an indirect gradient even when its ReLU score receives no direct CE gradient. Concerning the stability issue, **such an inter-expert mean operation does not violate the theoretical rationality of NormSiLU**. For each feature coordinate, cross-expert mean subtraction is equivalent to the following orthogonal projection: $$P_E=I_E-\frac{1}{E}\mathbf{1}\mathbf{1}^{\mathsf T}, \qquad \lVert P_E \rVert_2=1.$$ Across all $d_e$ feature coordinates, the corresponding operator is $I_{d_e}\otimes P_E$, which also has spectral norm one. With routing fixed, selecting the active experts gives the forward Jacobian: $J_A=S_AP$, where $S_A$ is the expert selection mask and $\lVert J_A \rVert_2\leq1$. **The mean stage shares gradients across experts without amplifying their joint Euclidean norm.** Therefore, the gradient scale stability still persists when the mean stage is considered. In addition, **we make a quality comparison between the mean-attached and mean-detached setting at the 0.24B checkpoint of DECO**. Both settings continue for 2,000 steps from that 15,000-step checkpoint with identical training tokens, optimizer initialization, and constant learning rate: | Mean treatment | C4 loss ↓ | C4 BPB ↓ | | :-------------- | :------------:| :------------:| | Attached | **3.405509** | **1.143933** | | Detached | 3.405972 | 1.144089 | As shown in the table, keeping the mean attached preserves the indirect gradient path and yields slightly lower C4 loss (by 0.000463) in this paired continuation. The two branches are close in quality over this short continual-training run.
AuthorsOpenReview ↗

Rebuttal (2/2)

Weakness 2.1 & Q4: Seeds and data-distribution robustness

While the averaged 0.11B perplexity favors the dense baseline: 34.43 (Dense) versus 34.69 (DECO), this does not violate our claim: DECO has dense-comparable quality, rather than universal superiority. For a more persuasive response, we provided an additional seed run for the 0.53B setting (the 2-run seeds are 1234 and 12345):

ModelOriginal run (Table 12)Added seed 123452-run mean ± std. ↓
Dense21.9321.6021.77 ± 0.17
DECO21.8721.6221.75 ± 0.13

These descriptive results are consistent with our claim of dense-comparable language-modeling performance at 0.53B.

In terms of the data-distribution issue, Section 6 shows why the data mixture matters: on the less heterogeneous FineWeb corpus, DECO has C4 PPL 30.14 versus Dense's 29.76 while retaining higher task accuracy, 38.65 versus 38.03. A likely explanation is that high-entropy, multi-domain data suits sparse MoE, where domain-specific inputs can be handled by activating only a specialized subset of experts. More studies are needed to verify this.

Weakness 2.2 & Q3: Deployment configuration and the dense baseline

In Table 5, we introduce a deployment-oriented systems experiment rather than another quality comparison. As shown in the table below, recent frontier MoE models commonly allocate a large fraction of their non-embedding parameters to routed experts. This expert-dominant regime is directly relevant to deployment scenarios in which inactive expert weights must be avoided during decoding. Therefore, we finally choose a setting with 90% of non-embedding parameters residing in routed experts, which is a conservative deployment-oriented stress test grounded in representative frontier LLM configurations instead of an arbitrary extreme.

ModelPublic architectureRouted-expert fraction
DeepSeek-V3671B; 58 MoE layers; 256 experts; d=7168,de=2048d=7168,d_e=204897.6%
Kimi K21.04T; 60 MoE layers; 384 experts; d=7168,de=2048d=7168,d_e=204897.7%
Qwen3-235B-A22B235B; 94 MoE layers; 128 experts; d=4096,de=1536d=4096,d_e=153697.0%

This deployment configuration is deliberately different from the main training configurations, which are designed for parameter-matched quality comparisons and activate approximately 50%–55% of their dense baseline's non-embedding parameters. The two experiments answer different questions: the main experiments test quality at an equal number of total parameters, while Table 5 tests whether the sparsity property of DECO translates into faster inference.

The "Dense" baseline executes the same expert weights, with up/down matrices concatenated into regular dense matrix multiplications. In other words, the "Dense" baseline adopts a dense-MLP-style execution control constructed from the same concatenated expert weights. It is therefore designed to isolate the cost of processing all stored experts from the cost of selecting a sparse subset. Thus, as our response to your question, the speedup can persist when compared with dense Transformer baselines with a vanilla MLP implementation.

Thanks for the detailed reply from authors. I think my concerns have been addressed and would like to raise my score.

Reviewer_4VZ3OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces DECO, a sparse MoE model structure that is comparable with dense models with the same total parameters. The authors argue that the performance gap between MoE and dense models is not an inherent cost of sparsity, but an artifact of how existing MoE models enforce sparsity. The experimental results demonstrate that DECO is comparable to dense models under delicate arch design.

Critical strengths
  • The experiments are solid, with promising results
  • The conclusion about norm silu in FFN experts is aligned with modern MoE models like K3
Critical weaknesses

I reviewed this paper before in another venue. A lot of new results are presented in this version. Overall, I think this paper is solid. Speaking of weaknesses:

  1. The authors believe the performance gap between dense and MoE is not the inherent sparsity, but the static top-k routing and hard regularization. But there’s also some regularization methods like router entropy or adaptive scaling in DECO. And top-p is less effective, which means the right arch combination matters instead of hard top-k and regularization I guess. If hard regularization matters, maybe we should add an ablation to use ReMoE’s sparsification loss but on a larger target (let’s say 20% from Figure 8).
  2. According to Fig 3a, the scaling property may be less effective since larger ladder points (7B model size) show less performance gain compared with dense models.
Questions for authors

What’s the scaling formula of DECO and other baselines in Figure 3a? I mean something like L(C) = A\times C^{-\alpha}

1 comment

Thank you for your excellent review! We will carefully address your concerns with detailed experiments and analyses.

We've already updated our manuscript to incorporate suggested revisions.

Weakness 1: Attribution to routing and sparsity regularization

Following your suggestion, we trained a DECO variant that replaces router entropy regularization with ReMoE-style L1 sparsification. We also trained a hard TopK routing variant. We experiment on 0.11B and 0.24B, and retain DECO's expert architecture at each scale: 42 / 57 routed experts of width 64, one shared expert of width 128, and non-gated NormSiLU experts. All runs use 15,000 steps and the same training token budget within each scale.

Setting0.11B C4 loss ↓0.11B C4 BPB ↓0.24B C4 loss ↓0.24B C4 BPB ↓
DECO (ours): ReLU + router entropy3.5367301.1880113.3228851.116179
DECO: ReLU + ReMoE-style L13.5451021.1908233.3432371.123016
DECO: hard TopK (activation 8/42 or 11/57)3.5475191.1916353.3477891.124545

As shown by the table, "ReLU + router entropy" has the lowest C4 loss at both scales. All settings have a routed-expert activation ratio of around 20%. The L1 variant retains DECO's ReLU routing and expert architecture while replacing only the sparsity regularizer. The hard-TopK variant replaces the routing mechanism with DeepSeekMoE-style hard selection.

Weakness 2 & Question: Scaling formula

At 7B, DECO and Dense differ by only 0.0001 BPB (0.8381 vs. 0.8382) and 0.003 PPL (15.690 vs. 15.693). This small gap is consistent with our claim: DECO achieves dense-comparable performance under matched total parameters and training tokens. We do not claim universal superiority over Dense. To show the scaling trends of DECO and other baselines underlying Figure 3a, we fit

BPB(N)=A⋅N−α,\mathrm{BPB}(N)=A\cdot N^{-\alpha},

where NN is the non-embedding parameter count. Fits use ordinary least squares in log-parameter/log-BPB space:

MethodAαR²Scale range
Dense5.8541460.0860620.99540.11B–7B
DECO5.8395140.0859400.99550.11B–7B
TopP7.3854550.0952310.95880.11B–2.4B
DeepSeek-V36.1696700.0884330.99030.11B–2.4B
ReMoE6.4440600.0905240.99180.11B–2.4B
BlockFFN6.1695870.0882810.98510.11B–2.4B

As shown by the table, Dense and DECO have nearly identical fitted trends over the measured range, which supports our conclusion: DECO maintains dense-comparable quality as the parameter budget grows, rather than relying on a transient small-model advantage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.