- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SetPrune formulates MoE expert pruning as cardinality-constrained set selection. It estimates marginal contributions using Monte Carlo expert permutations, summarizes coalition-dependent fluctuations with covariance, and combines unary terms with pairwise couplings in a binary optimization objective. Truncated sampling reduces calibration cost, and experiments compare downstream accuracy and perplexity across five MoE models.
Critical strengths
- Direct retained-set optimization is well motivated. It avoids reducing every expert to an independent ranking and provides a concrete, implementable way to use coalition-dependent observations.
- The results across several architectures show particularly useful improvements at aggressive pruning ratios. The language-modeling and downstream-task evaluations together make the empirical case stronger than a single-metric compression study.
Critical weaknesses
- The justification for covariance as a calibrated surrogate for signed coalition utility in Equation (23) remains incomplete. Appendix A.3 establishes dependence of centered marginals, not equivalence to utility loss. Covariance is quadratic in utility scale while the mean marginal is linear, so their relative weighting can change under an otherwise equivalent positive rescaling of the utility function.
- With only 20 permutations, covariance estimation and the shrinkage choice are important, but the reported threshold and representation-error ablations do not establish how stable the selected expert sets or downstream gains are to these choices.
Questions for authors
- How is the coupling scale calibrated, and is the selected set stable when the utility is multiplied by a positive constant?
- How do downstream performance and set overlap vary with the number of permutations and the shrinkage coefficient?
Follow-up on Our Rebuttal
Dear Reviewer rSYX,
Please forgive this brief follow-up. As the author–reviewer discussion period approaches its close, we wanted to kindly ask whether you have had an opportunity to review our rebuttal and the revised manuscript.
Your main concerns involved the calibration of covariance coupling and the stability of expert selection under different utility scales and sampling settings. In response, we clarified how rescaling affects the relative weight of the mean and covariance terms. We also added ablations on utility scaling, the shrinkage coefficient, and the number of sampled permutations, including downstream results and expert-set overlap across sampling budgets.
We hope these clarifications and results address your principal concerns. If you find the additional evidence convincing, we would be grateful if you could reconsider your overall assessment. If any concern remains, a brief indication would help us respond promptly.
Thank you for your time and thoughtful review.
Best regards,
Authors of SetPrune