acceptodds
Withdrawn from ICLR 2027

No Expert Replaces Another: Pruned Experts' Outputs Are Not Recovered by the Remaining Ones

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Expert pruning, merging and replacement in sparse Mixture-of-Experts (MoE) models are commonly motivated by expert redundancy, inferred from output similarity on shared inputs, from the small loss change after zeroing a single expert, or from replacing an expert by its mean output. None of these measurements tests whether the remaining experts can reproduce a pruned expert's output on the tokens routed to it, which is where pruning changes the layer output. We measure this on four MoE models. The remaining experts, combined linearly, recover only 2 to 28% of the pruned output energy, whereas a noisy copy of the pruned expert recovers 98%. Pruning still works at moderate budgets because this energy is concentrated in few experts: keeping half of them removes only 4.5 to 18% of it, and within a budget the energy a criterion removes predicts its perplexity. Three consequences follow. Pruning scores should be summed over an expert's routed tokens rather than averaged, which lowers perplexity in 15 of 18 settings on the three larger models. Modeling interactions between experts changes the selection little. Calibration data should match the deployment domain rather than be large: 2K matched tokens outperform two million mismatched ones.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MoE, Pruning, Compression

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xrA4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether kept experts can recover the outputs of pruned experts and argues that expert pruning works mainly by preserving deletion energy rather than through expert replacement.

Critical strengths

The paper attempts to understand the mechanism of MoE pruning and presents results with a limited scope.

Critical weaknesses
  1. The paper only tests fixed linear or affine combinations of the kept experts. This is an oversimplification of related pruning algorithms, where expert selection and gate weights can depend on each token. A low reconstruction score under a fixed linear combination does not establish that the kept experts cannot functionally replace a pruned expert after token-dependent rerouting. Therefore, the experiments do not fully support the broad claim that no expert can replace another.

  2. The paper has limited novelty. The proposed MEP ranking rule, $ w_j=\sum_{x\in X_j}g_j(x)^2\lVert f_j(x)\rVert_2^2, $ has already been explored in previous work [1]. The remaining procedure is standard per-layer expert pruning. The main contribution is therefore an empirical interpretation of an existing criterion within a limited scope.

  3. The frequency-weighted score depends strongly on the calibration distribution. The paper itself shows that mismatched calibration data can substantially reduce performance. This raises concerns about whether the proposed score can generalize to broad tasks beyond the calibration domain. The energy-preservation explanation is also not consistent across all experimental settings.

[1] How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle. Findings of EMNLP 2026.

Questions for authors

see weakness

Reviewer_bktiOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this paper, the author measure the recoverability of pruned experts in sparse mixture-of-experts (MoE) language models and challenges the common assumption that remaining experts can substitute for removed ones. Using a deletion-energy framework, the authors show that across four MoE models, a fixed linear or affine combination of the kept experts recovers only 2–28% of the deletion energy of pruned experts on their routed tokens, while a noisy copy of the pruned expert recovers 98%.

Critical strengths
  • The insights from this paper is useful and can help the research in MoE compression.
Critical weaknesses
  • Many sentences appear to be AI-generated and are difficult to understand; the writing needs substantial revision for clarity and readability.

  • The authors identify several problems but do not provide a method to address them, which limits the paper’s contribution.

Questions for authors

see weakness

Reviewer_ebQWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the recoverability of pruned experts in MoE and finds that their outputs are largely unrecoverable by fixed linear combinations of the remaining experts on their routed tokens. Motivated by this, the authors study MEP, a pruning criterion that retains experts with the largest total deletion energy. Experiments across several setups show that MEP often achieves lower perplexity than the evaluated baselines, but is also highly sensitive to the calibration data distribution.

Critical strengths

(1) The authors provide well-designed experiments to empirically show that the outputs of pruned experts cannot be well reconstructed by fixed linear combinations of the remaining experts.

(2) The proposed pruning strategy follows naturally from this observation. It is simple and generally effective when the calibration data matches the target domain.

Critical weaknesses

My main concern is MEP's dependence on calibration data matching the target domain, which the authors also acknowledge. While the authors have made efforts to address this issue (e.g., using mixed-domain calibration in Appendix G), I am not convinced that this sufficiently resolves the problem.

In practice, it is difficult to anticipate and cover all possible domains, and the input distribution may change over time. I do not think this issue can be easily addressed by simply mixing calibration data. While I find the authors' findings are solid, this raises concerns about the practical applicability of the resulting pruning method.

Questions for authors

Could the authors compare MEP with other pruning methods using the same equal-share mixed-domain calibration data across the domains in Figure 8?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.