- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes Sparser-MoE, a training-free neuron-level sparsification framework for Mixture-of-Experts (MoE) language models. The method combines two main ideas: (1) global neuron selection across the activated experts, allowing heterogeneous sparsity allocations, and (2) routing-weight-aware neuron scoring, which prioritizes neurons according to the routing weights of their corresponding experts.To improve computational efficiency, the authors further introduce a two-stage neuron selection procedure and custom Triton kernels for sparse up- and down-projections. Experiments on OLMoE-1B-7B, DeepSeek-V2-Lite-16B, and Qwen3.5-35B-A3B demonstrate improved downstream accuracy over several activation-based scoring baselines at 80% and 90% sparsity, together with decoding throughput improvements under certain batch-size configurations.
Critical strengths
- Well-motivated MoE-specific sparsification strategy. The paper identifies a meaningful limitation of applying uniform neuron sparsity independently to routed experts. Global neuron selection is a simple and intuitive way to accommodate heterogeneous activation distributions. Incorporating routing weights into neuron scores also provides a reasonable mechanism for accounting for differences in expert contribution scales. 2.Consistent accuracy improvements. The proposed method achieves higher average benchmark accuracy than the evaluated scoring baselines across all three models at both reported sparsity levels. The gains are particularly noticeable on OLMoE-1B-7B.
Critical weaknesses
Critical Weaknesses
- Ambiguity in the two-stage neuron selection formulation. Equation (8) appears inconsistent with the quantile definition in Equation (5). Specifically, (p'_i) is defined as a routing-weight-proportional fraction but is directly used as a sparsity quantile. Under the stated definition, this could result in retaining substantially more neurons than intended and assigning stricter thresholds to experts with larger routing weights, contrary to the proposed motivation. The authors should clarify whether (p'_i) represents a sparsity quantile or a candidate-retention ratio, and explain how the intended candidate budget is maintained.
- Inconsistent results for apparently equivalent configurations. Table 3 reports 58.0% average accuracy for "w/o r, global selection" at 90% sparsity, while Table 6 reports 58.4% for the gate-based baseline with global selection under the same model and sparsity level. These settings appear conceptually equivalent. The authors should clarify whether they differ in candidate filtering, selection procedures, or other experimental configurations, and explain the discrepancy.
Questions for authors
1.Clarification of the two-stage neuron selection formulation. Equation (8) appears inconsistent with the quantile definition in Equation (5). Could the authors clarify whether (p'_i) represents a sparsity quantile or a candidate-retention ratio, and explain how the proposed formulation ensures that experts with larger routing weights retain more candidate neurons while maintaining the intended candidate budget? 2.Discrepancy between Table 3 and Table 6. The "w/o r, global selection" setting in Table 3 and the gate-based baseline with global selection in Table 6 appear to be equivalent, but their reported average accuracies differ (58.0% vs. 58.4%). Could the authors clarify whether these experiments use identical configurations and explain the discrepancy? 3.Comparison with recent related work. Several recent studies, including Mixture of Neuron Experts (MoNE, 2025), MoSE: Mixture of Slimmable Experts (ICML 2026), DualSparse-MoE (ICML 2026), and TENP (ACL 2026 Findings), explore related ideas involving neuron-level sparsification and heterogeneous resource allocation across MoE experts. Their contributions appear to overlap conceptually with aspects of Sparser-MoE. Could the authors discuss these works and clarify the key methodological differences and novel contributions of Sparser-MoE? Additional experimental comparisons would be valuable where implementations are publicly available and experimental settings are compatible.
I appreciate the authors' response that addressed some of my concerns partially. Some other concerns still remain, e.g., the insufficient justification of the mechanism. Since the response provided very detailed information that is useful, the paper currently is better positioned to be 2.5 - but not 3. As a result, the rating will be unchanged.