acceptodds
Under review as a conference paper at ICLR 2027

Sparser-MoE: Beyond Expert Sparsity with Heterogeneous Neuron-Level Sparsity

Abstract

Large language models (LLMs) have demonstrated strong performance across diverse domains, but their high inference costs remain a challenge. To address this issue, prior work has introduced methods that selectively activate neurons in Gated-MLP blocks based on intermediate activation magnitudes, reducing computational costs while largely preserving model performance. However, these methods do not account for the distinctive characteristics of mixture-of-experts (MoE) models, limiting their ability to preserve performance when applying activation sparsification to MoE. We propose Sparser-MoE, an MoE-tailored neuron sparsification framework featuring dynamic expert-wise sparsity and contribution-scale-aware neuron scoring. Motivated by the observation that higher-ranked experts contain more important neurons, Sparser-MoE performs global neuron selection across the top- experts, allowing heterogeneous sparsity to emerge naturally across experts. It further incorporates routing weights into neuron scores to better preserve the scale of the MoE layer output. Together, these designs identify a more effective set of active neurons than existing methods and thereby better preserve the original model performance under sparsification. Experiments across diverse models, model scales, and benchmarks show that Sparser-MoE retains more of the dense model's original performance than existing sparsification methods on nearly every individual benchmark at each evaluated sparsity level, while achieving the highest average benchmark performance at every evaluated sparsity level. To our knowledge, Sparser-MoE is the first Gated-MLP sparsification approach to jointly introduce MoE-specific neuron scoring and selection strategies, and we believe it can serve as a starting point for related research.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Activation Sparsity, Dynamic Neuron Selection, Routing-Aware Sparsification, Efficient LLM Inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NzgeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Sparser-MoE, a training-free neuron-level sparsification framework for Mixture-of-Experts (MoE) language models. The method combines two main ideas: (1) global neuron selection across the activated experts, allowing heterogeneous sparsity allocations, and (2) routing-weight-aware neuron scoring, which prioritizes neurons according to the routing weights of their corresponding experts.To improve computational efficiency, the authors further introduce a two-stage neuron selection procedure and custom Triton kernels for sparse up- and down-projections. Experiments on OLMoE-1B-7B, DeepSeek-V2-Lite-16B, and Qwen3.5-35B-A3B demonstrate improved downstream accuracy over several activation-based scoring baselines at 80% and 90% sparsity, together with decoding throughput improvements under certain batch-size configurations.

Critical strengths
  1. Well-motivated MoE-specific sparsification strategy. The paper identifies a meaningful limitation of applying uniform neuron sparsity independently to routed experts. Global neuron selection is a simple and intuitive way to accommodate heterogeneous activation distributions. Incorporating routing weights into neuron scores also provides a reasonable mechanism for accounting for differences in expert contribution scales. 2.Consistent accuracy improvements. The proposed method achieves higher average benchmark accuracy than the evaluated scoring baselines across all three models at both reported sparsity levels. The gains are particularly noticeable on OLMoE-1B-7B.
Critical weaknesses

Critical Weaknesses

  1. Ambiguity in the two-stage neuron selection formulation. Equation (8) appears inconsistent with the quantile definition in Equation (5). Specifically, (p'_i) is defined as a routing-weight-proportional fraction but is directly used as a sparsity quantile. Under the stated definition, this could result in retaining substantially more neurons than intended and assigning stricter thresholds to experts with larger routing weights, contrary to the proposed motivation. The authors should clarify whether (p'_i) represents a sparsity quantile or a candidate-retention ratio, and explain how the intended candidate budget is maintained.
  2. Inconsistent results for apparently equivalent configurations. Table 3 reports 58.0% average accuracy for "w/o r, global selection" at 90% sparsity, while Table 6 reports 58.4% for the gate-based baseline with global selection under the same model and sparsity level. These settings appear conceptually equivalent. The authors should clarify whether they differ in candidate filtering, selection procedures, or other experimental configurations, and explain the discrepancy.
Questions for authors

1.Clarification of the two-stage neuron selection formulation. Equation (8) appears inconsistent with the quantile definition in Equation (5). Could the authors clarify whether (p'_i) represents a sparsity quantile or a candidate-retention ratio, and explain how the proposed formulation ensures that experts with larger routing weights retain more candidate neurons while maintaining the intended candidate budget? 2.Discrepancy between Table 3 and Table 6. The "w/o r, global selection" setting in Table 3 and the gate-based baseline with global selection in Table 6 appear to be equivalent, but their reported average accuracies differ (58.0% vs. 58.4%). Could the authors clarify whether these experiments use identical configurations and explain the discrepancy? 3.Comparison with recent related work. Several recent studies, including Mixture of Neuron Experts (MoNE, 2025), MoSE: Mixture of Slimmable Experts (ICML 2026), DualSparse-MoE (ICML 2026), and TENP (ACL 2026 Findings), explore related ideas involving neuron-level sparsification and heterogeneous resource allocation across MoE experts. Their contributions appear to overlap conceptually with aspects of Sparser-MoE. Could the authors discuss these works and clarify the key methodological differences and novel contributions of Sparser-MoE? Additional experimental comparisons would be valuable where implementations are publicly available and experimental settings are compatible.

Reviewer_PWGHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Sparser-MoE introduces training-free neuron sparsification within routed MoE experts. It combines routing-weighted gate scores with global neuron selection, supported by two-stage filtering and custom Triton kernels. Experiments across three models and eight benchmarks show improved accuracy over the tested baselines and decoding speedups approaching 1.6x, but formulation inconsistencies and incomplete comparisons limit confidence in the conclusions.

Critical strengths
  • The method addresses a useful problem through a simple and training-free design that allows sparsity to vary across experts.
  • Accuracy gains are consistent across model families and benchmarks.
  • Custom sparse kernels demonstrate actual decoding acceleration, supported by latency breakdowns and batch-size analysis.
Critical weaknesses
  • Incomplete comparison: Simplified scoring baselines do not establish superiority over complete prior methods such as FloE. Moreover, baseline sparsity is calibration-dependent while Sparser-MoE uses exact budgets, without reporting realized sparsity or matched-compute comparisons.
  • Insufficient support for the mechanism: Routing-weighted gate magnitude omits up-projection activations and down-projection weights. Accuracy improvements alone do not establish the claimed preservation of FFN output scale, and direct reconstruction-error measurements are absent.
  • Inconsistent filtering formulation: Equation 8 assigns higher quantile thresholds to larger routing weights, contradicting the stated more permissive filtering. With uniform routing, it implies approximately 97.5% candidate retention rather than the stated 20%.
  • Incomplete quality-speed evaluation: Timings cover one GPU and sequence length, exclude prefill, and show limited gains or slowdowns at some batch sizes. Accuracy is evaluated only at very high sparsity, with substantial OLMoE degradation and no perplexity or generation-quality assessment.
Questions for authors
  1. What corrected threshold formula was actually implemented in Equation 8? Please explain its normalization and report candidate-retention rates relative to the intended preliminary budget.

  2. Do the improvements persist against faithful method adaptations and exact-budget baselines when realized sparsity and computation are matched? Can you provide corresponding accuracy–latency comparisons?

  3. How closely do the proposed scores track actual neuron contributions? Can you measure FFN output-scale distortion and reconstruction error across all three models?

  4. Can you provide quality–latency curves across 50–90% sparsity, including prefill and longer sequences? How does sparsification affect perplexity and generated responses?

1 comment

I appreciate the authors' response that addressed some of my concerns partially. Some other concerns still remain, e.g., the insufficient justification of the mechanism. Since the response provided very detailed information that is useful, the paper currently is better positioned to be 2.5 - but not 3. As a result, the rating will be unchanged.

Reviewer_dnVrOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Focusing on bringing sparsity to MoE models, this paper introduces a neuron-level top-k routing in experts. The motivation of sparsify MoE is to accelerate model inference (as the proposed method is training-free) while maximally retain the original model’s capabilities. With multi-stage top-k sparsification, MoE model could reach 90% expert sparsity while retaining 86% to 98% performance on OLMoE, DeepSeek, and Qwen-MoE.

Critical strengths
  • The method is straightforward and intuitive
  • The overall writing is clear.
  • The results are good, especially on Qwen, which obtain 98% downstream performance, and the speedup becomes a free lunch
Critical weaknesses
  • Relevant CATS/TEAL/FloE baselines are not compared. This paper is basically comparing to its variants, which is less convincing and we don’t know how does this method perform.
  • Besides the above baseline, If the target is to speedup model inference, pruning methods should be considered as baselines.
Questions for authors

see weaknesses above.

Comments on the paper

1 comment
Area_Chair_jGaVOpenReview ↗

Official Comment by Area Chair

Dear Reviewers

As we are now halfway through the discussion period, I encourage reviewers to engage in the responses and discuss should there be more questions.

---AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.