acceptodds
Under review as a conference paper at ICLR 2027

SetPrune: Beyond Expert Ranking for Mixture-of-Experts Pruning

Abstract

Expert pruning reduces the memory footprint of Mixture-of-Experts (MoE) models by removing redundant experts, thereby enabling more efficient deployment. Most existing methods assign scalar importance scores to individual experts and prune them according to the resulting ranking. However, pruning is inherently a set-selection problem: the utility of the retained expert set may depend not only on individual experts but also on their collective contributions, while direct subset search becomes combinatorially expensive as the expert pool grows. We propose SetPrune, a framework that moves beyond expert ranking by jointly modeling residual individual and collective coupling contributions and directly optimizing the subset of experts to retain. Building on the Möbius decomposition of retained-set utility, SetPrune distinguishes singleton contributions from collective effects associated with expert coalitions. Because exact recovery of the full contribution structure requires evaluating exponentially many coalitions, SetPrune samples coalition-conditioned marginal contributions through Monte Carlo expert permutations and summarizes their collective dependencies using centered marginal covariance. These observations are used to construct residual individual terms and collective couplings within a structured set-level objective, which directly optimizes the retained expert set under a prescribed pruning budget rather than collapsing coalition-dependent information into independent expert scores. We further introduce efficient truncated sampling strategies to reduce the calibration cost of contribution estimation. Experiments demonstrate that SetPrune effectively preserves both downstream and language modeling performance, particularly under aggressive expert pruning, paving the way toward practical set-level expert selection for MoE compression.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, MoE pruning, expert pruning, Shapley value, cooperative game theory, Möbius representation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rSYXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SetPrune formulates MoE expert pruning as cardinality-constrained set selection. It estimates marginal contributions using Monte Carlo expert permutations, summarizes coalition-dependent fluctuations with covariance, and combines unary terms with pairwise couplings in a binary optimization objective. Truncated sampling reduces calibration cost, and experiments compare downstream accuracy and perplexity across five MoE models.

Critical strengths
  1. Direct retained-set optimization is well motivated. It avoids reducing every expert to an independent ranking and provides a concrete, implementable way to use coalition-dependent observations.
  2. The results across several architectures show particularly useful improvements at aggressive pruning ratios. The language-modeling and downstream-task evaluations together make the empirical case stronger than a single-metric compression study.
Critical weaknesses
  1. The justification for covariance as a calibrated surrogate for signed coalition utility in Equation (23) remains incomplete. Appendix A.3 establishes dependence of centered marginals, not equivalence to utility loss. Covariance is quadratic in utility scale while the mean marginal is linear, so their relative weighting can change under an otherwise equivalent positive rescaling of the utility function.
  2. With only 20 permutations, covariance estimation and the shrinkage choice are important, but the reported threshold and representation-error ablations do not establish how stable the selected expert sets or downstream gains are to these choices.
Questions for authors
  1. How is the coupling scale calibrated, and is the selected set stable when the utility is multiplied by a positive constant?
  2. How do downstream performance and set overlap vary with the number of permutations and the shrinkage coefficient?
1 comment
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer rSYX,

Please forgive this brief follow-up. As the author–reviewer discussion period approaches its close, we wanted to kindly ask whether you have had an opportunity to review our rebuttal and the revised manuscript.

Your main concerns involved the calibration of covariance coupling and the stability of expert selection under different utility scales and sampling settings. In response, we clarified how rescaling affects the relative weight of the mean and covariance terms. We also added ablations on utility scaling, the shrinkage coefficient, and the number of sampled permutations, including downstream results and expert-set overlap across sampling budgets.

We hope these clarifications and results address your principal concerns. If you find the additional evidence convincing, we would be grateful if you could reconsider your overall assessment. If any concern remains, a brief indication would help us respond promptly.

Thank you for your time and thoughtful review.

Best regards,

Authors of SetPrune

Reviewer_35hiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents SetPrune, an MoE expert pruning method that formulates expert selection as a set optimization problem rather than ranking experts independently. The authors argue that expert-wise importance scores may overlook interactions among experts, leading to suboptimal pruning decisions. SetPrune estimates expert marginal contributions using Monte Carlo permutations. Motivated by the Möbius decomposition, it uses the mean and covariance of these contributions to approximate individual contributions and pairwise couplings. The resulting quadratic objective is optimized using mixed-integer linear programming under a fixed pruning ratio. Two sampling strategies, angle-based skipping and early truncation, are introduced to reduce the evaluation cost. Experiments are conducted on five MoE models with 25% and 50% expert pruning. The authors compare SetPrune against several ranking-based and pruning methods using seven zero-shot tasks and WikiText-2 perplexity. They also report ablations on sampling thresholds and analyses of representation distortion and inference efficiency.

Critical strengths
  1. The paper highlights a reasonable limitation of expert-wise ranking: selecting individually important experts does not necessarily produce the best retained expert set. Considering interactions among experts is relevant to MoE pruning, particularly at higher pruning ratios.
  2. The experiments cover five MoE models with different architectures and report both downstream task accuracy and language modeling perplexity. This provides a useful empirical basis for examining the proposed approach beyond a single model, although the completeness of the comparisons remains a concern.
  3. The paper is generally well-written and easy to follow. The motivation, method, and experimental setup are presented in a clear manner, making the overall approach accessible to readers.
Critical weaknesses
  1. My main concern is the theoretical basis of the covariance-based coupling. While the Möbius decomposition motivates considering expert interactions, the covariance of marginal contributions does not directly represent the sign or magnitude of these interactions. The paper does not sufficiently justify why these covariance terms, together with the residual individual contributions in Eq. (22), should provide a reliable approximation of the original set utility. The overall framework also combines multiple approximations and heuristic strategies without clearly establishing their necessity. Moreover, although skipping and truncation reduce the sampling overhead, the method still requires repeated model evaluations and mixed-integer optimization. The paper does not report the actual number of evaluations after truncation or the end-to-end pruning cost, making its practical scalability difficult to assess.
  2. The ablation studies focus mainly on sampling thresholds, but do not adequately isolate the effects of covariance modeling and set-level optimization. In particular, a comparison against ranking experts using the same estimated marginal contributions would be important to establish whether the proposed set-level objective provides meaningful benefits. The baseline selection is also incomplete, omitting recent relevant methods such as STEP and MoNE. Furthermore, while the average results favor SetPrune in several settings, the detailed per-task results show considerable variation, with many cases where other methods perform better. Without stronger baselines and more targeted ablations, I find it difficult to determine whether the reported gains come from the proposed interaction modeling or from the combination of sampling and optimization choices.
Questions for authors
  1. What is the justification for using the covariance of marginal contributions as pairwise coupling coefficients in Eq. (23)? Covariance captures statistical dependence but does not necessarily preserve the sign or magnitude of the underlying expert interactions. Is there any theoretical or empirical evidence that the resulting objective reliably approximates the original set utility or preserves the ranking of candidate expert subsets?
  2. How much improvement comes specifically from covariance modeling and set-level optimization? Could the authors compare SetPrune with a simpler approach that ranks experts using the same Monte Carlo marginal estimates, under identical calibration and sampling budgets? An ablation removing the pairwise coupling terms would also help clarify whether the added complexity is necessary.
  3. The method samples permutations over all experts across layers, but solves the final optimization independently for each layer. For a model with 128 MoE layers and 128 experts per layer, (M=20) would involve up to 327,680 model-value evaluations before skipping and truncation. How many evaluations are actually performed after applying these strategies? Could the authors report the end-to-end pruning time, GPU resources, and MILP solving time for each model? Moreover, since the final objective only considers within-layer couplings, how are cross-layer expert interactions accounted for? I am concerned that the method incurs substantial computational overhead without fully addressing the global set-selection problem it motivates.
  4. Why are recent and relevant MoE compression methods, particularly STEP and MoNE, missing from the comparisons? These methods have reported results on overlapping model architectures, making their omission important when assessing the claimed improvements. In addition, the detailed results in Appendix Tables 5 and 6 show that SetPrune is not consistently the best method across individual tasks, even when its average accuracy is higher. Could the authors provide comparisons against these missing baselines under matched settings and report variability across sampling seeds? Without these comparisons, I am not convinced that the reported gains are sufficiently robust or establish an advantage over recent methods.
4 comments
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer 35hi,

Please forgive this brief follow-up. As the author–reviewer discussion period approaches its close, we wanted to kindly ask whether you have had an opportunity to review our rebuttal and the revised manuscript.

Your main concerns included the theoretical basis of covariance-based coupling, the contribution of set-level optimization, and the completeness of the experimental comparisons. In response, we clarified the role and scope of the covariance surrogate and discussed how global sampling relates to cross-layer interactions. We also added a controlled comparison with marginal-contribution ranking, comparisons with STEP and MoNE, and a multi-seed evaluation on Qwen1.5-MoE-A2.7B.

We hope these additions help address your principal concerns. If you find the analysis and evidence convincing, we would be grateful if you could reconsider your overall assessment. If any concern remains, we would appreciate a brief indication so that we can clarify it promptly.

Thank you for your time and careful review.

Best regards,

Authors of SetPrune

Reviewer_35hiOpenReview ↗

Acknowledgment of Rebuttal and Remaining Concerns

Thank the authors for their detailed response and additional experiments. The new comparisons with STEP and MoNE, controlled ablations, multi-seed evaluations, and computational cost analysis substantially strengthen the empirical evaluation and address several of my original concerns. However, my primary concern regarding the theoretical justification of covariance-based coupling remains unresolved. While the authors clarify that covariance captures statistical dependencies induced by higher-order interactions, this does not establish why covariance should serve as a reliable pairwise coefficient for approximating the original set utility. In particular, covariance does not necessarily preserve the sign or magnitude of the underlying interactions. The exhaustive-search results are encouraging, but their limited scope does not fully establish the general fidelity of the proposed surrogate objective. Additionally, although the discussion of global sampling and layer-wise optimization provides useful clarification, the method still does not explicitly account for cross-layer expert interactions during final set selection. Overall, I acknowledge the improved empirical evidence and appreciate the authors' efforts. Nevertheless, since the central methodological concern remains insufficiently addressed, I will maintain my original score of 2.

AuthorsOpenReview ↗

Thank You for Your Continued Feedback: Our Further Response Part [1]

Dear Reviewer 35hi,

Please forgive this additional follow-up. Thank you for carefully reviewing our paper and previous response, and for continuing the discussion. Below, we address your remaining concerns about the surrogate objective, the scope of its experimental validation, and cross-layer interactions.

1. The role of covariance in the surrogate objective

SetPrune compresses the dependence induced by exponentially many higher-order expert interactions into a tractable quadratic objective. Mean marginals retain signed average contributions, while centered covariance extracts context dependence in collective contributions. Shrinkage regularizes the coupling estimates, and residual calibration makes the mean and coupling terms consistent within the set objective. We derive this construction below and validate its selection quality in Section 2.

1.1 Mean marginals and covariance capture complementary information

We use the calibration utility v(S)=1/PPL⁡(S)v(S)=1/\operatorname{PPL}(S). For exact marginals under uniform permutations of the global expert set, Möbius decomposition gives

\Delta_i(\pi)&=m(\{i\})+u_i(\pi),\\ \mu_i:=\mathbb E_{\pi}[\Delta_i(\pi)] &=m(\{i\})+ \sum_{\substack{T\ni i\\|T|\ge2}}\frac{m(T)}{|T|}. \end{aligned}$$ Here, $m(T)$ is the Möbius coefficient of coalition $T$, and $u_i(\pi)$ sums the collective terms allocated to $i$ under $\pi$. Each such term is allocated with probability $1/|T|$ and enters $\mu_i$ with its original sign. Centering removes the fixed singleton term and the mean collective allocation: $$\begin{aligned} \Delta_i(\pi)-\mu_i &=u_i(\pi)-\mathbb E[u_i(\pi)],\\ C_{ij} =\operatorname{Cov}_{\pi}(\Delta_i,\Delta_j) &=\operatorname{Cov}_{\pi}(u_i,u_j). \end{aligned}$$ Thus, **centered covariance provides a second-order representation of context dependence among collective contributions of all orders**. Equation (20) estimates this coupling from the sampled marginals, including the approximations from skipping and truncation. ### 1.2 Shrinkage and residual calibration produce the set-selection objective Within one layer (suppressing its index), Equations (21) and (22) use the sampled means $\bar\Delta_i$ and empirical covariance $\widehat\Sigma$ to define $$\begin{aligned} \widetilde\Sigma &=(1-\alpha)\widehat\Sigma +\alpha\operatorname{diag}(\widehat\Sigma),\\ \eta_i &=\bar\Delta_i-\sum_{j\ne i}\widetilde\Sigma_{ij}. \end{aligned}$$ Shrinkage attenuates noisy off-diagonal estimates, and **residual calibration makes the mean and coupling terms consistent within the set objective**. For a removed set $D$ in this layer, substituting Equation (22) into Equation (23) yields $$\widehat{\Delta V}(D) =\sum_{i\in D}\bar\Delta_i -\sum_{\substack{i<j\i,j\in D}}\widetilde\Sigma_{ij}.$$ Under Equation (22), **each singleton surrogate score satisfies $\widehat{\Delta V}(\{i\})=\bar\Delta_i$**, and the pairwise term adjusts joint removals. For an exactly evaluated additive utility, covariance vanishes and the objective equals the exact additive loss. ## 2. Empirical validation of the surrogate **Exhaustive search is computationally infeasible for large expert pools.** One Qwen3 layer alone has $\binom{128}{64}\approx2.4\times10^{37}$ ways to remove half its experts. The three experiments show agreement with the enumerated layer-wise references, gains from the coupled objective using identical marginal samples, and advantages over recent baselines across architectures.
AuthorsOpenReview ↗

Thank You for Your Continued Feedback: Our Further Response Part [2]

2.1 Selection accuracy against layer-wise exhaustive search

On Mixtral-8x7B, retaining four of eight experts gives (84)=70\binom{8}{4}=70 candidates per layer. We enumerate them under the budgets in Equation (25), evaluate their actual calibration PPL, and compare the best set with SetPrune's selection. Within this layer-wise search space, Table R2 confirms 100% expert-set overlap and identical PPL at all six pruning ratios from 12.5% to 75%.

2.2 Benefit of the coupled objective under identical samples

For Qwen1.5-MoE-A2.7B, Table R1 compares selection objectives while holding the marginal estimates and per-layer pruning budgets fixed. At α=1\alpha=1, Equation (23) reduces to independent ranking by mean marginals. Compared with this baseline, α=0.5\alpha=0.5 reduces PPL from 9.42 to 8.60 and increases seven-task average accuracy from 58.40% to 59.54% at 25% pruning. At 50%, PPL falls from 16.63 to 11.43 (31.3% lower), and accuracy rises from 55.15% to 57.46%.

2.3 Performance across architectures and pruning ratios

Table R5 evaluates five MoE architectures at two pruning ratios. SetPrune achieves lower PPL than STEP and MoNE in all ten settings. Across these settings, mean relative PPL reductions are 24.84% and 31.52%, and mean gains in seven-task average accuracy are 1.18 and 1.77 percentage points, respectively. SetPrune's average accuracy exceeds STEP's in all ten settings and MoNE's in nine. Tasks can favor different expert subsets, so we report all task results and use seven-task average accuracy to assess overall performance under task-agnostic calibration.

3. Cross-layer context and within-layer optimization

Global permutation sampling and full-model utility evaluations include cross-layer context in the marginal statistics. The MILP uses these statistics to select each layer's expert set, with explicit quadratic terms between experts in that layer. This limits estimation size and solver cost under the prescribed per-layer budgets.

3.1 Global sampling incorporates cross-layer context

During global removal trajectories, full-model utility evaluations reflect changes in hidden states, downstream routing, and subsequent expert contributions. Let π−l\pi_{-l} denote the expert context outside layer ll. The law of total covariance gives

\operatorname{Cov}(\Delta_ {l,i},\Delta_ {l,j}) ={}&\mathbb E_ {\pi_{-l}} \left[ \operatorname{Cov}(\Delta_ {l,i},\Delta_{l,j}\mid\pi_ {-l}) \right]\\ &+\operatorname{Cov}_ {\pi_ {-l}} \left( \mathbb E[\Delta_ {l,i}\mid\pi_ {-l}], \mathbb E[\Delta_{l,j}\mid\pi_{-l}] \right). \end{aligned}$$ The first term averages conditional within-layer covariance over the other-layer contexts; the second measures how mean marginals vary together across those contexts. Both contribute to the within-layer selection objective. ### 3.2 Block structure regularizes estimation under limited sampling For example, with 48 layers and 128 experts per layer, the global matrix has dimensions $6{,}144\times6{,}144$. With $M=20$ permutations, the raw global covariance and each individual within-layer block have rank at most $M-1=19$. Counting each unordered off-diagonal pair once gives $$\begin{aligned} \text{Global pairs:}\quad &\binom{6{,}144}{2}=18{,}871{,}296,\\ \text{Within-layer pairs:}\quad &48\binom{128}{2} =48\times8{,}128 =390{,}144. \end{aligned}$$ Within-layer blocks **reduce the number of modeled pairs by a factor of approximately 48.37**. The optimizer uses fewer uncertain coupling terms, and Equation (21) shrinks their magnitudes. ### 3.3 Layer-wise selection matches routing constraints and limits solver cost Within-layer coupling matches MoE routing: top-$k$ competition, probability renormalization, and load redistribution occur among experts in the same layer. Per-layer budgets keep pruning from concentrating in a few layers and preserve a regular structure for deployment. A global quadratic objective requires $O((\sum_l E_l)^2)$ coupling terms and one joint MILP. The layer-wise formulation uses $\sum_l O(E_l^2)$ terms, and each layer can be solved independently. The reported **millisecond-scale MILP times** measure solver cost. Repeated full-model utility evaluations remain the main pruning cost. --- --- Thank you again for your thoughtful questions and continued engagement. These explanations should help readers understand the method and reproduce its results using our planned open-source implementation. We hope the derivations and additional results address your remaining concerns. If you find this evidence convincing, we would be sincerely grateful if you could reconsider your overall assessment. Please let us know if further discussion or clarification would be helpful. With sincere thanks and best regards, Authors of SetPrune
Reviewer_EojYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper treats MoE expert pruning as a budget-constrained set selection problem instead of a score-then-rank problem. The Möbius decomposition of a set value function serves as the motivation and the value function is inverse perplexity. The method samples global expert permutations and removes experts one by one to collect marginal contributions. From these samples it computes the mean marginal and the covariance of centered marginals. These two quantities define a residual individual term and a collective coupling term, which form a layer-wise quadratic objective. The objective is solved under a cardinality constraint by an exactly linearized MILP. To cut the sampling cost, the authors add angle-based skipping on centered router weights and value-based early truncation.

Critical strengths
  • The problem framing is clear, and the optimization is well posed. The paper points out the gap between rank-based decisions and the utility of the retained set. It writes the pruning loss in Möbius multilinear form, truncates it to a second-order QUBO, and hands it to an MILP through a standard exact linearization. The path from a set function to a solvable problem is complete, and it goes further than another refinement of a scoring rule.
  • The gains under aggressive pruning are large and appear on several models. At 50% pruning, SetPrune beats the best baseline in average accuracy on Qwen1.5-MoE, Qwen3-30B, and DeepSeekMoE-16B in Table 1.
  • The authors pay attention to the sampling cost. Angle skipping absorbs the RMSNorm gain and centers the router weights, which is a sensible way to handle router distribution.
Critical weaknesses
  • The key ablation for the set-level part is missing, and the results are very sensitive to hyperparameters. The most direct control is ranking by Δˉ\bar\Delta from the same permutation samples, which equals SetPrune with alpha=1, but the paper does not report it.
  • Some baseline and evaluation results look abnormal, and a few claims are inaccurate. REAP is clearly worse than Random on Qwen1.5 and Qwen3 (for Qwen3 at 50%, 36.26 vs. 44.96 accuracy and 172.12 vs. 35.74 PPL in Tables 1 and 2). That result goes against the conclusions of the REAP paper and points to a problem in its calibration or implementation.
  • The paper gives no absolute numbers for calibration and solver cost. Every non-skipped removal step runs a full forward pass on the 128×512-token calibration set. The memory and throughput gains in Fig.3c hold for any method at the same pruning ratio and are not specific to SetPrune. By its title, the strongest baseline AIMER needs no calibration, and Frequency and REAP need a single statistics pass. Without GPU hours and MILP solve times, the claims of efficiency and tractability are not supported.
Questions for authors
  • What is the default alpha? Please report accuracy and PPL at 25% and 50% pruning on all five models for alpha=1 and for a few intermediate values. That experiment decides whether the gain comes from set-level optimization or from permutation sampling with a PPL-based value.
  • Please report the absolute calibration cost. The useful numbers are the actual forward-pass count for each model, the GPU type and GPU hours, the average fraction of each permutation traversed before truncation, and the average number of non-zero observations per expert.
  • Several evaluation results need explanation. Which calibration data and hyperparameters were used for REAP, and why does it fall below Random on Qwen1.5 and Qwen3? Why is the dense PPL of GPT-OSS-20B 252.12, and how can pruning lower it so much?
1 comment
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer EojY,

Please forgive this brief follow-up. As the author–reviewer discussion period approaches its close, we wanted to kindly ask whether you have had an opportunity to review our rebuttal and the revised manuscript.

Your main concerns included the contribution of set-level optimization, the evaluation results for REAP and GPT-OSS-20B, and the cost of calibration. In response, we added an ablation comparing SetPrune with ranking based on the same estimated marginal contributions, discussed the observed REAP and GPT-OSS-20B results, and provided additional pruning-time information.

We hope these clarifications and results address your principal concerns. If you find the additional evidence convincing, we would be grateful if you could reconsider your overall assessment. If any concern remains, a brief indication would help us provide further clarification promptly.

Thank you for your time and thoughtful review.

Best regards,

Authors of SetPrune

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.