- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies expert pruning in sparse MoE language models through the structure of expert outputs. It reports low effective dimensionality in expert mean representations and overlap between approximate output distributions, then proposes two greedy selection methods: PEEP preserves a weighted subspace of expert means, while DCEP preserves coverage of Gaussian output summaries. Both incorporate router-derived importance weights. Experiments on DeepSeek-V2-Lite and Qwen3-30B-A3B compare these methods with frequency pruning, EASY-EP and REAP on general and mathematical benchmarks. The results show competitive accuracy at 75% and 50% expert retention, supported by weighting ablations and calibration-size studies.
Critical strengths
-
The central methodological contribution is a useful way to select complementary experts using their output relationships. PEEP measures the additional projection energy contributed by a candidate expert, while DCEP measures how much that expert improves coverage of the output distributions. Combining these objectives with router-derived weights gives a clear role to both subset diversity and individual importance. I find this a worthwhile extension of independent importance ranking: the selection explicitly depends on the experts already retained. The novelty lies primarily in this formulation for MoE pruning and its two concrete implementations, rather than in greedy optimization itself.
-
The experiments provide meaningful evidence that the proposed selection criteria are useful across two model families and different task types. In Table 1, PEEP reaches a Qwen3 mathematical average of 78.6 at 75% retention, compared with 75.1 for REAP; at 50% retention, DCEP reaches 60.9 against 57.3. Either proposed method also achieves the highest general-task average in each model and retention setting. I appreciate the additional weighting ablations and the five-run calibration-size studies in Figures 4 and 5, which go beyond a single pruning configuration and help establish how performance changes with the available calibration data.
Critical weaknesses
-
The connection between the structural summaries and preservation of the actual MoE function remains indirect. Section 4 computes effective ranks from mean expert outputs, so low rank describes these mean vectors rather than the full input-dependent expert functions. Experts can have similar means, or similar marginal Gaussian summaries, while behaving differently on the same token. Moreover, spanning a removed expert’s mean does not ensure that the retained experts reproduce its routed contribution. The downstream results make these summaries plausible selection heuristics, but a direct check of routed layer-output reconstruction would better support the stronger functional-preservation interpretation.
-
The practical efficiency advantage is less thoroughly established than the accuracy results. Avoiding combinatorial search is a sensible motivation, but the paper does not report calibration time, expert-selection time or peak memory for the proposed methods and baselines. This matters because Section 4 describes collecting outputs from all experts, and Algorithm 1 forms a d-by-d matrix and repeatedly evaluates candidate directions. Those steps can have meaningful costs even with a small calibration set. A compact measurement on the existing models would help establish the accuracy–cost trade-off and support the claim that the approach offers an efficient way to account for expert relationships.
Questions for authors
-
I would like to understand how closely the structural objectives track preservation of the routed MoE output. Have you measured layer-output reconstruction error on held-out tokens for the subsets selected by PEEP, DCEP and frequency pruning? Even a focused comparison on one model would help me judge whether lower projection or coverage error corresponds to better functional preservation. It would also clarify how much of the interpretation rests on the mean and Gaussian approximations.
-
I find the greedy selection approach appealing, but I could not locate measurements of its computational cost. Could you report calibration time, selection time and peak memory for PEEP and DCEP alongside REAP on the existing hardware? I would also appreciate clarification on whether every expert is evaluated on every calibration token, and whether PEEP explicitly materializes the d-by-d matrix in Algorithm 1. These details would help me assess the practical cost of incorporating expert relationships.
-
I noticed that Algorithm 2 z-score normalizes the expert means and standard deviations before applying Equation 11, whereas Section 4 defines that equation using the original Gaussian parameters. Could you clarify the distance actually used in the experiments?