acceptodds
Withdrawn from ICLR 2027

Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern fine-grained Mixture-of-Experts (MoE) models route each token to a small number of experts and renormalize their router probabilities. We show that this renormalization implicitly calibrates expert output gain to the training top-: reducing at inference changes not only which experts are used but also the strength of the expert branch. We separate these effects by activating the top experts while normalizing by the probability mass of the top experts, introducing one integer with no parameters, training, or measurable compute overhead. On Qwen3.6-35B-A3B, reducing from 8 to 4 experts causes a 4.65-point MMLU drop under standard renormalization but only 0.35 points with , while halving routed-expert compute. The result replicates on the larger Qwen3.5-397B-A17B, where reducing from 10 to 5 experts loses only 0.55 points with an appropriate reference set. Removing renormalization entirely is catastrophic, showing that preserving a suitable reference mass is crucial. We further find that perplexity and downstream accuracy favor different , cautioning against selecting MoE compression settings using unlabeled text alone. Analyses also show that expert identity matters substantially more than expert weighting, while balanced and domain-specialized routing leaves limited room for expert pruning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Mixture-of-Experts, Training-free inference efficiency, Router renormalization

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_WaM4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates training-free reduction of activated experts in fine-grained Mixture-of-Experts (MoE) models. The authors argue that directly reducing the number of activated experts causes performance degradation not only because fewer experts contribute to the output, but also because renormalizing the remaining routing probabilities changes the effective scale of the expert outputs.

To address this issue, the paper proposes decoupling the number of activated experts k1k_1 from the number of experts used to compute the normalization denominator k2k_2. This introduces a single inference-time hyperparameter without modifying model weights or requiring additional training. The paper also analyzes expert routing distributions, domain specialization, and the relative importance of expert selection and weighting. Experiments on Qwen3.6-35B-A3B and Qwen3.5-397B-A17B evaluate different normalization settings using perplexity, MMLU, GSM8K, and C-Eval. The results show that an appropriately chosen k2k_2 substantially reduces the accuracy degradation caused by halving the number of activated experts. The authors further observe that perplexity and downstream accuracy may favor different normalization settings.

Critical strengths
  1. The proposed modification is easy to implement and requires no additional training or model parameter updates. By decoupling expert activation from routing normalization, it provides a lightweight way to reduce activated-expert computation. This simplicity makes the approach potentially useful for inference-time efficiency optimization.
  2. The experiments show that adjusting the normalization reference set can recover much of the performance lost when halving the number of activated experts. The improvements are observed on both Qwen models and across multiple downstream benchmarks. These results suggest that routing normalization is an important factor when reducing activated experts, although the broader applicability of the finding remains unclear.
Critical weaknesses
  1. The proposed method essentially introduces an additional hyperparameter (k_2) to adjust the normalization scale of the retained experts. This is mathematically equivalent to applying an input-dependent scaling factor to the standard top-(k_1) routing output. The underlying observation that reducing the number of activated experts changes the normalization scale is relatively straightforward and has also been discussed in prior work. Beyond this observation, the paper offers limited new algorithmic insight: (k_2) is selected through empirical search, without a principled strategy for determining its value. Moreover, the method relies entirely on the upstream expert selection strategy, as it only rescales the outputs of the selected experts without addressing which experts should be retained. Its effectiveness may therefore depend heavily on the quality and behavior of the upstream selection method, yet no experiments investigate its integration with existing expert pruning or selection algorithms. The lack of comparisons with simpler scaling-based calibration baselines further makes it difficult to establish the necessity or advantage of the proposed parameterization. These limitations substantially weaken the methodological contribution.
  2. The experiments are limited to two models from the same Qwen family, both relying on similar routing normalization mechanisms. However, fine-grained MoE architectures differ substantially in their routing and normalization designs. For instance, OLMoE does not use the same top-(k) renormalization, while DeepSeek-MoE adopts a different routing and scaling mechanism. The proposed approach therefore addresses a particular normalization behavior rather than a general property of fine-grained MoE models, and its applicability to other architectures remains unclear. Furthermore, although the paper positions its approach as complementary to existing expert pruning and selection methods, it provides no experiments demonstrating such integration or comparisons with these methods under matched settings. This limited scope makes it difficult to assess the broader relevance and practical impact of the contribution.
Questions for authors
  1. Since the proposed normalization is mathematically equivalent to applying an input-dependent scaling factor to the retained expert outputs, what advantages does the top-(k_2) parameterization offer over simpler alternatives, such as a fixed or layer-wise scaling factor? Could the authors provide comparisons under the same evaluation settings?
  2. How should (k_2) be determined for a new model or downstream task without tuning on the evaluation benchmark? Given that the optimal (k_2) differs across models and that perplexity does not reliably predict downstream accuracy, is there a principled selection criterion or a separate validation protocol that supports the method's practical applicability?
  3. The experiments only cover two models from the Qwen family. How would the proposed method apply to architectures with different routing and normalization mechanisms, such as OLMoE, which does not use the same top-(k) renormalization, or DeepSeek-MoE, which employs a different routing and scaling design? Could the authors provide cross-architecture results or clarify the precise architectural assumptions required by the method?
  4. The method only adjusts the weights of retained experts, while relying on the upstream routing strategy to determine which experts are activated. How sensitive are the reported improvements to the expert selection strategy? Could the authors evaluate the proposed correction in combination with existing expert reduction methods, such as LExI or ACE, to substantiate the claimed complementarity and determine whether the benefits persist beyond standard top-(k) selection?
Reviewer_pMNhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free method to reduce the number of activated experts in fine-grained MoE models. The key idea is to decouple the number of activated experts, k1k_1, from the number of experts used for router probability renormalization, k2k_2. By adjusting k2k_2, the authors aim to compensate for the change in expert output magnitude caused by reducing k1k_1. Experiments on two Qwen MoE models report that approximately halving the activated experts can preserve most of the original performance on MMLU and GSM8K.

Critical strengths

The paper identifies an interesting effect of router renormalization when reducing the number of activated experts. The proposed correction is simple and training-free.

Critical weaknesses
  • Limited novelty and missing baselines. The paper does not compare against directly applying a calibrated scaling factor to the outputs of the activated experts, making the necessity of the proposed k2k_2-based formulation unclear.

  • Inappropriate evaluation protocol and limited experiments. The value of k2k_2 is selected based on MMLU performance, without a clearly separated validation and test set. Moreover, the main experiments are limited to two models from the same family and a few benchmarks with relatively small evaluation subsets. The generalizability of the method remains insufficiently validated.

  • Presentation and organization could be improved. Sections 4 and 5 contain extensive analyses whose relevance to the core method is not sufficiently clear. Some claims and wording could also be made more precise, and the paper would benefit from a tighter focus on its main contribution.

Questions for authors
  • How does the proposed k2k_2-based method compare with directly applying a calibrated scaling factor to the activated expert outputs?

  • How does the proposed method perform on other MoE models (e.g., GLM-4.7-Flash and gpt-oss-20b) and benchmarks (e.g., MMLU-Pro and LiveCodeBench)?

Reviewer_1cZgOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the performance degradation observed when reducing the number of activated experts (kk) in fine-grained Mixture-of-Experts (MoE) models at inference time. The authors argue that standard renormalization of router probabilities implicitly couples the "gain" (magnitude of expert contribution) to the training-time kk. Simply reducing kk and renormalizing over the remaining experts inflates their weights beyond the distribution seen during training, causing significant accuracy drops. To address this, the authors propose a training-free method called "Training-Free Halving," which decouples the selection of active experts (k1k_1) from the reference set used for normalization (k2k_2). Specifically, they activate the top-k1k_1 experts but normalize their weights by the sum of probabilities of the top-k2k_2 experts (where k2≥k1k_2 \ge k_1). Empirical results on Qwen3.6-35B and Qwen3.5-397B show that setting k2k_2 appropriately (e.g., k2=16k_2=16 when k1=4k_1=4) allows halving the compute cost with negligible loss in MMLU accuracy, whereas standard renormalization (k2=k1k_2=k_1) leads to significant degradation. The paper also analyzes router behavior, highlighting that expert identity is more critical than precise weighting and that perplexity is a poor proxy for downstream task performance in this context.

Critical strengths

None

Critical weaknesses
  1. Limited Benchmark Diversity: The evaluation is heavily reliant on MMLU, GSM8K, and C-Eval. While these are standard benchmarks, they do not fully represent the diversity of real-world tasks (e.g., long-context reasoning, code generation beyond short snippets, or multi-turn dialogue). The claim that "perplexity is a poor proxy" is interesting but limited to WikiText/CodeParrot; broader validation across diverse domains would strengthen this claim.
  2. Lack of Comparison with Strong Baselines: The paper compares primarily against the "naive" baseline (standard renormalization with reduced kk). It lacks comprehensive comparisons with other state-of-the-art training-free compression methods, such as expert pruning , merging, or dynamic routing strategies like LExI or ACE mentioned in related work. Without these comparisons, it is unclear if the proposed method offers superior trade-offs compared to existing techniques that might reduce memory and compute or offer better accuracy retention.
  3. Narrow Model Scope: The experiments are conducted exclusively on the Qwen series (fine-grained MoEs). While the appendix includes a preliminary study on Gemma, the generalizability to other architectures (e.g., Mixtral, DeepSeek-V2/V3) or coarse-grained MoEs is not thoroughly explored. The assertion that this method is specifically beneficial for fine-grained MoEs needs more robust cross-architecture validation.
Questions for authors

How does your method compare quantitatively with recent training-free expert pruning or merging techniques (e.g., REAP, Merge-then-Compress) in terms of both accuracy retention and actual inference latency/memory savings? A direct comparison would help position your work within the broader landscape of MoE compression.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.