acceptodds
Withdrawn from ICLR 2027

Experts Exhibit Structured Redundancy: Structure-Preserving Expert Pruning for Mixture-of-Experts Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sparse Mixture-of-Experts (MoE) large language models replace the dense feed-forward networks in standard Transformers with MoE layers consisting of multiple experts, activating only a small subset of experts for each token during the forward pass. This sparse activation mechanism significantly improves the trade-off between model capacity and inference efficiency. However, practical deployment still requires storing all experts, resulting in substantial memory overhead. In this work, we show that expert outputs in MoE layers exhibit structured redundancy. Across different models and task categories, the stable rank and entropy effective rank of expert outputs are consistently far below the nominal number of experts, while expert output distributions also exhibit fewer effective modes. Motivated by these observations, we reformulate expert pruning as a structure-preserving expert subset selection problem and efficiently optimize it via a greedy strategy, rather than only independently scoring and ranking individual experts or exhaustively searching over expert combinations. We propose two expert pruning methods under the unified structure-preserving framework: Projection Energy Expert Pruning (**PEEP**), which preserves the functional subspace by maximizing weighted projection energy, and Distributional Coverage Expert Pruning (**DCEP**), which preserves the distributional geometry by minimizing weighted coverage error. Both methods incorporate router-derived weights as priors, enabling them to jointly preserve the global structural relationships among experts while accounting for the local importance of individual experts. Experiments on DeepSeek-V2-Lite and Qwen3-30B-A3B demonstrate that, at the expert retention ratios of 75% and 50%, our methods outperform the state-of-the-art baselines on the majority of benchmarks and consistently achieve the best average performance across both general tasks and mathematical reasoning tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Expert Pruning, Mixture-of-Experts

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_No7POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies expert pruning in sparse MoE language models through the structure of expert outputs. It reports low effective dimensionality in expert mean representations and overlap between approximate output distributions, then proposes two greedy selection methods: PEEP preserves a weighted subspace of expert means, while DCEP preserves coverage of Gaussian output summaries. Both incorporate router-derived importance weights. Experiments on DeepSeek-V2-Lite and Qwen3-30B-A3B compare these methods with frequency pruning, EASY-EP and REAP on general and mathematical benchmarks. The results show competitive accuracy at 75% and 50% expert retention, supported by weighting ablations and calibration-size studies.

Critical strengths
  1. The central methodological contribution is a useful way to select complementary experts using their output relationships. PEEP measures the additional projection energy contributed by a candidate expert, while DCEP measures how much that expert improves coverage of the output distributions. Combining these objectives with router-derived weights gives a clear role to both subset diversity and individual importance. I find this a worthwhile extension of independent importance ranking: the selection explicitly depends on the experts already retained. The novelty lies primarily in this formulation for MoE pruning and its two concrete implementations, rather than in greedy optimization itself.

  2. The experiments provide meaningful evidence that the proposed selection criteria are useful across two model families and different task types. In Table 1, PEEP reaches a Qwen3 mathematical average of 78.6 at 75% retention, compared with 75.1 for REAP; at 50% retention, DCEP reaches 60.9 against 57.3. Either proposed method also achieves the highest general-task average in each model and retention setting. I appreciate the additional weighting ablations and the five-run calibration-size studies in Figures 4 and 5, which go beyond a single pruning configuration and help establish how performance changes with the available calibration data.

Critical weaknesses
  1. The connection between the structural summaries and preservation of the actual MoE function remains indirect. Section 4 computes effective ranks from mean expert outputs, so low rank describes these mean vectors rather than the full input-dependent expert functions. Experts can have similar means, or similar marginal Gaussian summaries, while behaving differently on the same token. Moreover, spanning a removed expert’s mean does not ensure that the retained experts reproduce its routed contribution. The downstream results make these summaries plausible selection heuristics, but a direct check of routed layer-output reconstruction would better support the stronger functional-preservation interpretation.

  2. The practical efficiency advantage is less thoroughly established than the accuracy results. Avoiding combinatorial search is a sensible motivation, but the paper does not report calibration time, expert-selection time or peak memory for the proposed methods and baselines. This matters because Section 4 describes collecting outputs from all experts, and Algorithm 1 forms a d-by-d matrix and repeatedly evaluates candidate directions. Those steps can have meaningful costs even with a small calibration set. A compact measurement on the existing models would help establish the accuracy–cost trade-off and support the claim that the approach offers an efficient way to account for expert relationships.

Questions for authors
  1. I would like to understand how closely the structural objectives track preservation of the routed MoE output. Have you measured layer-output reconstruction error on held-out tokens for the subsets selected by PEEP, DCEP and frequency pruning? Even a focused comparison on one model would help me judge whether lower projection or coverage error corresponds to better functional preservation. It would also clarify how much of the interpretation rests on the mean and Gaussian approximations.

  2. I find the greedy selection approach appealing, but I could not locate measurements of its computational cost. Could you report calibration time, selection time and peak memory for PEEP and DCEP alongside REAP on the existing hardware? I would also appreciate clarification on whether every expert is evaluated on every calibration token, and whether PEEP explicitly materializes the d-by-d matrix in Algorithm 1. These details would help me assess the practical cost of incorporating expert relationships.

  3. I noticed that Algorithm 2 z-score normalizes the expert means and standard deviations before applying Equation 11, whereas Section 4 defines that equation using the original Gaussian parameters. Could you clarify the distance actually used in the experiments?

Reviewer_oSb4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a structure-preserving expert pruning framework for sparse Mixture-of-Experts models. Authors report two observations: 1) Expert output representations have low stable rank and entropy effective rank relative to the number of experts. 2) Expert output distributions exhibit fewer effective modes than the total number of experts, according to a kernel-based participation ratio. Motivated by these observations, they propose two methods PEEP and DCEP, which both incorporate router-derived importance weights. Experiments on DeepSeek-V2-Lite and Qwen3-30B-A3B compare the methods against random pruning, frequency pruning, EASY-EP, and REAP at expert retention ratios of 75% and 50%.

Critical strengths
  1. Relatively simple methodology. The proposed methods use greedy selection strategies that are straightforward to implement, although their effectiveness over simpler alternatives is not fully established.

  2. Consideration of expert relationships. The methods account for inter-expert relationships rather than relying solely on individual importance scores, although this idea has been explored in prior work.

Critical weaknesses
  1. Insufficient Justification for the Connection Between Structural Redundancy and Expert Pruning.

The paper fails to establish a convincing connection between its structural observations and pruning effectiveness. Low effective rank or distributional similarity does not necessarily imply functional redundancy, as experts with similar output statistics may still perform distinct input-dependent functions. More importantly, the authors provide no direct evidence that their structural metrics or optimization objectives correlate with actual pruning-induced degradation. The effective-rank and effective-mode measurements are not directly used to guide expert selection or determine pruning budgets. Consequently, the central motivation remains insufficiently validated.

  1. Limited Novelty and Insufficient Experimental Differentiation.

The central observations regarding expert redundancy and structure-aware compression have already been explored in prior work, including MC-SMoE, HC-SMoE, and particularly STUN, which also exploits expert similarity for efficient representative selection. The proposed methods largely adapt conventional greedy subspace selection and weighted medoid selection, without clearly establishing their advantages over existing approaches. The omission of direct comparisons with relevant structure-aware methods further weakens the novelty claim, especially given that improvements over REAP on general benchmarks are typically below one percentage point. Overall, the paper does not convincingly establish a sufficiently distinct conceptual or methodological contribution.

Questions for authors
  1. How do the proposed structural metrics justify expert pruning? Is there empirical evidence that effective rank or the number of distributional modes correlates with pruning sensitivity or downstream performance degradation?

  2. Why are expert-output statistics sufficient to characterize functional redundancy? Experts with similar means or output distributions may still exhibit substantially different input-dependent behaviors. How do the proposed methods account for this?

  3. What is the key novelty compared with existing structure-aware MoE compression methods, particularly STUN, MC-SMoE, and HC-SMoE? What insights or capabilities do PEEP and DCEP provide beyond these approaches?

  4. Why are comparisons with closely related structure-aware selection methods missing? In particular, how do PEEP and DCEP compare with STUN's expert-pruning stage or simple similarity-based clustering under matched pruning budgets?

  5. Do the proposed optimization objectives reliably predict actual pruning quality? Can the authors demonstrate a correlation between projection/coverage errors and actual MoE output reconstruction errors or downstream accuracy?

  6. What are the actual computational advantages of the proposed methods? Can the authors report pruning time, calibration overhead, and memory consumption compared with existing approaches?

Reviewer_xmmcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PEEP and DCEP, which select MoE experts based on output subspace coverage and distributional coverage, respectively, with router-derived weights. The methods show promising pruning results on mathematical reasoning tasks across two models.

Critical strengths
  1. The selection objectives are clear, and the algorithms are straightforward and leave the retained experts' parameters unchanged.
  2. In Table 1, PEEP at 75% retention and DCEP at 50% retention improve Qwen3's average mathematical reasoning score over REAP by 3.5 and 3.6 percentage points, respectively. The appendix includes weight ablations and five runs with independently sampled calibration sets.
Critical weaknesses
  1. The novelty is not sufficiently clear. HC-SMoE already clusters experts using their mean outputs, while STUN uses expert cluster structure for greedy pruning without exhaustive search. The paper does not sufficiently explain how its specific objectives differ from existing methods that exploit expert structure. The redundancy observation alone is insufficient as the main contribution. I find the abstract's statement, “In this work, we show that expert outputs in MoE layers exhibit structured redundancy,” overstated, since this is already well established.
  2. Figure 2 appears to be AI-generated. I find Stage 2 unclear and difficult to follow. As an aside, more than half of the submissions I have reviewed seem to use AI-generated figures, which may make the papers harder to understand.
Questions for authors

What is specifically novel about the expert selection in PEEP/DCEP compared with existing methods such as STUN? With the same calibration data and the same number of retained experts, how do they compare with the simple approach of grouping similar experts and retaining one representative per group?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.