- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates training-free reduction of activated experts in fine-grained Mixture-of-Experts (MoE) models. The authors argue that directly reducing the number of activated experts causes performance degradation not only because fewer experts contribute to the output, but also because renormalizing the remaining routing probabilities changes the effective scale of the expert outputs.
To address this issue, the paper proposes decoupling the number of activated experts from the number of experts used to compute the normalization denominator . This introduces a single inference-time hyperparameter without modifying model weights or requiring additional training. The paper also analyzes expert routing distributions, domain specialization, and the relative importance of expert selection and weighting. Experiments on Qwen3.6-35B-A3B and Qwen3.5-397B-A17B evaluate different normalization settings using perplexity, MMLU, GSM8K, and C-Eval. The results show that an appropriately chosen substantially reduces the accuracy degradation caused by halving the number of activated experts. The authors further observe that perplexity and downstream accuracy may favor different normalization settings.
Critical strengths
- The proposed modification is easy to implement and requires no additional training or model parameter updates. By decoupling expert activation from routing normalization, it provides a lightweight way to reduce activated-expert computation. This simplicity makes the approach potentially useful for inference-time efficiency optimization.
- The experiments show that adjusting the normalization reference set can recover much of the performance lost when halving the number of activated experts. The improvements are observed on both Qwen models and across multiple downstream benchmarks. These results suggest that routing normalization is an important factor when reducing activated experts, although the broader applicability of the finding remains unclear.
Critical weaknesses
- The proposed method essentially introduces an additional hyperparameter (k_2) to adjust the normalization scale of the retained experts. This is mathematically equivalent to applying an input-dependent scaling factor to the standard top-(k_1) routing output. The underlying observation that reducing the number of activated experts changes the normalization scale is relatively straightforward and has also been discussed in prior work. Beyond this observation, the paper offers limited new algorithmic insight: (k_2) is selected through empirical search, without a principled strategy for determining its value. Moreover, the method relies entirely on the upstream expert selection strategy, as it only rescales the outputs of the selected experts without addressing which experts should be retained. Its effectiveness may therefore depend heavily on the quality and behavior of the upstream selection method, yet no experiments investigate its integration with existing expert pruning or selection algorithms. The lack of comparisons with simpler scaling-based calibration baselines further makes it difficult to establish the necessity or advantage of the proposed parameterization. These limitations substantially weaken the methodological contribution.
- The experiments are limited to two models from the same Qwen family, both relying on similar routing normalization mechanisms. However, fine-grained MoE architectures differ substantially in their routing and normalization designs. For instance, OLMoE does not use the same top-(k) renormalization, while DeepSeek-MoE adopts a different routing and scaling mechanism. The proposed approach therefore addresses a particular normalization behavior rather than a general property of fine-grained MoE models, and its applicability to other architectures remains unclear. Furthermore, although the paper positions its approach as complementary to existing expert pruning and selection methods, it provides no experiments demonstrating such integration or comparisons with these methods under matched settings. This limited scope makes it difficult to assess the broader relevance and practical impact of the contribution.
Questions for authors
- Since the proposed normalization is mathematically equivalent to applying an input-dependent scaling factor to the retained expert outputs, what advantages does the top-(k_2) parameterization offer over simpler alternatives, such as a fixed or layer-wise scaling factor? Could the authors provide comparisons under the same evaluation settings?
- How should (k_2) be determined for a new model or downstream task without tuning on the evaluation benchmark? Given that the optimal (k_2) differs across models and that perplexity does not reliably predict downstream accuracy, is there a principled selection criterion or a separate validation protocol that supports the method's practical applicability?
- The experiments only cover two models from the Qwen family. How would the proposed method apply to architectures with different routing and normalization mechanisms, such as OLMoE, which does not use the same top-(k) renormalization, or DeepSeek-MoE, which employs a different routing and scaling design? Could the authors provide cross-architecture results or clarify the precise architectural assumptions required by the method?
- The method only adjusts the weights of retained experts, while relying on the upstream routing strategy to determine which experts are activated. How sensitive are the reported improvements to the expert selection strategy? Could the authors evaluate the proposed correction in combination with existing expert reduction methods, such as LExI or ACE, to substantiate the claimed complementarity and determine whether the benefits persist beyond standard top-(k) selection?