- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies expert pruning in MoE models by jointly evaluating expert subsets to account for correlations between expert outputs. RECOUP selects experts based on their optimal linear reconstruction capability, while RECOUPdeploy uses statistics collected under routing within a candidate pool to approximate deployment error. Both variants use greedy selection with swap refinement, require no recovery training, and demonstrate effectiveness across multiple models and pruning settings.
Critical strengths
- The empirical results support the value of joint expert selection. In some experimental settings, the methods substantially outperform independent expert importance ranking and surpass several existing approaches. These results suggest that jointly considering relationships between experts has practical value for expert pruning.
- The methods are computationally practical. The methods rely on calibration statistics and require no recovery training. Greedy selection with swap refinement avoids enumerating all expert combinations, reducing the computational cost of subset search and making the approach potentially useful in practice.
Critical weaknesses
- The distinction from prior work is insufficiently clear, and the novelty of the core design is not firmly established. SCHUR-A* already performs expert subset selection through unconstrained linear reconstruction of expert outputs, using an objective closely related to the base RECOUP objective. The paper should clarify its substantive contributions beyond SCHUR-A* and provide direct comparisons demonstrating the additional benefits of its proposed designs.
- Both variants have insufficiently validated gaps between their surrogate objectives and actual deployment behavior. The base method evaluates expert subsets through their optimal linear combination, but the compensation coefficients are not constrained to be nonnegative or normalized, and it is unclear whether they are used at deployment. If these coefficients are used only for selection, a better reconstruction score does not necessarily imply lower actual deployment error. The theoretical analysis mainly explains the least-squares formulation and the orthogonal case, without sufficiently establishing this connection in the general setting. RECOUPdeploy removes the free compensation coefficients, but its search uses routing statistics from a fixed candidate pool rather than the routing induced by each subset. Consequently, its surrogate error may differ from the actual deployment error defined in Eq. (2). The paper should validate whether the two objectives rank candidate subsets consistently.
- The benefits and applicability of RECOUPdeploy relative to the base method are unclear. RECOUPdeploy aims to reduce the mismatch between the scoring objective and actual deployment computation. However, the paper should more clearly summarize which models, retention ratios, and tasks benefit from this variant, and whether these advantages are consistent across calibration samples. If RECOUPdeploy does not consistently outperform the base method, the paper should explain why and provide guidance on choosing between the two variants.
- The solution quality of the subset search is insufficiently evaluated. Both variants use greedy selection with swap refinement without a global optimality guarantee. The comparison with random-restart local search in Appendix H is informative, but it does not quantify the gap to the global optimum. The absence of a global guarantee is not itself a decisive weakness; however, further evaluation is needed to characterize the search error and determine whether better optimization of the surrogate objective improves downstream performance.
[1] SCHUR-A*: Layer-wise Optimal Expert Pruning for MoEs via Schur-Complement Guided A* Search. ICML 2026.
Questions for authors
- What are RECOUP’s substantive contributions beyond the linear reconstruction framework of SCHUR-A*? Which experiments demonstrate the additional benefits of these contributions?
- Are the compensation coefficients of the base method used at deployment? Can you validate the relationship between each variant’s surrogate score and actual deployment error?
- Under which settings does RECOUPdeploy consistently outperform the base method? If its advantages are inconsistent, what explains this behavior, and how should practitioners choose between the two variants?
- On small-scale instances that can be solved exactly, what is the optimality gap of greedy selection with swap refinement? Do better surrogate objective values translate into better downstream performance?