SetPrune: Beyond Expert Ranking for Mixture-of-Experts Pruning

Expert pruning reduces the memory footprint of Mixture-of-Experts (MoE) models by removing redundant experts, thereby enabling more efficient deployment. Most existing methods assign scalar importance scores to individual experts and prune them according to the resulting ranking. However, pruning is inherently a set-selection problem: the utility of the retained expert set may depend not only on individual experts but also on their collective contributions, while direct subset search becomes combinatorially expensive as the expert pool grows. We propose SetPrune, a framework that moves beyond expert ranking by jointly modeling residual individual and collective coupling contributions and directly optimizing the subset of experts to retain. Building on the Möbius decomposition of retained-set utility, SetPrune distinguishes singleton contributions from collective effects associated with expert coalitions. Because exact recovery of the full contribution structure requires evaluating exponentially many coalitions, SetPrune samples coalition-conditioned marginal contributions through Monte Carlo expert permutations and summarizes their collective dependencies using centered marginal covariance. These observations are used to construct residual individual terms and collective couplings within a structured set-level objective, which directly optimizes the retained expert set under a prescribed pruning budget rather than collapsing coalition-dependent information into independent expert scores. We further introduce efficient truncated sampling strategies to reduce the calibration cost of contribution estimation. Experiments demonstrate that SetPrune effectively preserves both downstream and language modeling performance, particularly under aggressive expert pruning, paving the way toward practical set-level expert selection for MoE compression.