- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether kept experts can recover the outputs of pruned experts and argues that expert pruning works mainly by preserving deletion energy rather than through expert replacement.
Critical strengths
The paper attempts to understand the mechanism of MoE pruning and presents results with a limited scope.
Critical weaknesses
-
The paper only tests fixed linear or affine combinations of the kept experts. This is an oversimplification of related pruning algorithms, where expert selection and gate weights can depend on each token. A low reconstruction score under a fixed linear combination does not establish that the kept experts cannot functionally replace a pruned expert after token-dependent rerouting. Therefore, the experiments do not fully support the broad claim that no expert can replace another.
-
The paper has limited novelty. The proposed MEP ranking rule, $ w_j=\sum_{x\in X_j}g_j(x)^2\lVert f_j(x)\rVert_2^2, $ has already been explored in previous work [1]. The remaining procedure is standard per-layer expert pruning. The main contribution is therefore an empirical interpretation of an existing criterion within a limited scope.
-
The frequency-weighted score depends strongly on the calibration distribution. The paper itself shows that mismatched calibration data can substantially reduce performance. This raises concerns about whether the proposed score can generalize to broad tasks beyond the calibration domain. The energy-preservation explanation is also not consistent across all experimental settings.
[1] How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle. Findings of EMNLP 2026.
Questions for authors
see weakness