- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper provides a theoretical analysis of the suboptimal learning procedure of MoE expert pruning. The analysis relies on formulating the problem of pruning into a submodular optimization problem and use a second-order Taylor expansion of the loss degradation to estimate pairwise expert interactions.
Based on this formulation, the authors propose a majorization-minimization (MM) algorithm that incorporates estimated second-order information into expert selection. They provide theoretical convergence claims and experimental results showing improved accuracy and fewer rounds to convergence compared with the selected baselines.
Critical strengths
1. Interesting optimization perspective on expert pruning.
I found the attempt to formulate expert selection through difference-of-submodular optimization particularly interesting. Expert pruning is already a well-studied area, but explicitly modeling the trade-off between expert redundancy and synergy provides a useful perspective on the problem. Although I have concerns about the precise formulation, as discussed in the weaknesses, I appreciate the direction of this analysis.
2. A principled approach to modeling expert dependencies.
Rather than relying solely on individual expert importance scores, the authors attempt to capture pairwise interactions using second-order information. I find this motivation compelling, particularly since the best subset of experts need not consist of the individually highest-ranked experts.
3. Interesting algorithmic approach.
The proposed MM procedure provides a tractable approach to an otherwise combinatorial selection problem, reducing each selection update to a Top- operation. I appreciate the effort to connect the theoretical formulation to a relatively simple selection algorithm, although I have concerns about the validity of some guarantees.
Critical weaknesses
1. Major concerns with the mathematical formulation (Equations 3–7)
Although I appreciate the mathematical perspective and proposed submodular optimization framework, I have significant concerns about whether the initial formulation represents the expert-pruning problem the authors aim to solve.
1.1. Ambiguity in the reference model.
The authors describe as a fully activated state and a theoretical upper bound on model capacity. However, Equation (1) already defines using Top- routing. Therefore, setting in Equation (3) simply recovers the original sparse MoE, rather than activating every expert.
Moreover, activating all experts would not necessarily improve performance (see, e.g., https://arxiv.org/abs/2509.21892). While the reference loss is constant with respect to and need not be optimal, its interpretation should be clarified.
1.2. Applying an additional mask to an already sparse router.
This is my main concern. The authors define the output as:
Since already performs Top- selection, the learned mask can only remove previously selected experts, not activate new ones.
This introduces potentially degenerate cases. For example, if the original router selects experts 1 and 2, but retains experts 3 and 4, Equation (3) produces a zero MoE-layer output. There is no apparent mechanism for rerouting or renormalization.
More importantly, Equation (7) estimates pairwise interactions using a term proportional to . Therefore, the method cannot capture interactions between experts that are never co-activated by the original router. How can the proposed approach discover new synergistic expert combinations beyond those already permitted by the original routing mechanism?
1.3. Consequences for the Taylor expansion.
This ambiguity directly affects the mathematical analysis:
-
If is fixed: Equation (5) is a valid local Taylor approximation, but it describes post-routing masking rather than conventional expert pruning with rerouting. Its accuracy for the large, discrete changes involved in pruning is also not established.
-
If the router is recomputed after applying : Then must depend on , and the gradient and Hessian derivations in Equations (6)–(7) no longer hold as written. Differentiation must account for this dependence, and hard Top- selection introduces potential nonsmoothness.
I believe the authors need to clarify which operation is actually implemented and establish that their mathematical surrogate corresponds to it.
2. Mathematical issues with the optimization guarantees
I have additional concerns about the theoretical analysis.
First, Equation (10) appears to rely on an invalid lower bound. The authors claim that the first-order Taylor expansion provides a global lower bound for the supermodular gain function . However, supermodularity alone does not guarantee this property.
For example, consider , which is supermodular. Its first-order Taylor approximation at evaluates to 1 at , while the true function value is 0. Therefore, the claimed lower bound fails.
Since this bound is used to construct the MM surrogate, the monotonic convergence guarantee in Theorem 1 is not established by the provided proof.
Are there missing assumption not stated in the paper that would let you get to this bound? If so please state them, and do they affect the rest of the analysis?
3. Unclear framing and insufficient pruning experiments
I find it difficult to determine whether the paper proposes a pruning method or a dynamic expert-selection method.
Equation (4) suggests a globally shared pruning mask, whereas the exposition frequently discusses dynamic routing and improvements during fine-tuning. These are different settings, and the distinction matters for both the theoretical analysis and the efficiency claims.
If this is primarily a pruning method, the experimental evaluation is insufficient. I would expect measurements of actual parameter reduction, GPU memory footprint, inference throughput, and computational cost. The paper evaluates , but this provides only a limited examination of the accuracy–compression trade-off, particularly without comparisons against the original unpruned model.
I was also surprised by the limited connection to the broader expert-pruning literature. Standard MoE routing and random selection are useful references, but they are not competitive pruning methods. More comparisons with established expert-pruning and expert-merging approaches would strengthen the evaluation.
As presented, the experiments do not sufficiently support the paper's claims regarding practical pruning efficiency.
4. Rounds to convergence do not establish computational efficiency
The authors repeatedly emphasize faster convergence in terms of training rounds. However, this is not necessarily a fair measure of computational efficiency.
The proposed method introduces additional computations to estimate expert interactions and perform MM optimization. Although the MM updates themselves may be inexpensive, constructing and updating the interaction matrix has a cost that is not adequately characterized.
In particular, it is unclear how frequently the Hessian approximation is recomputed during training.
Comparing convergence rounds alone is therefore insufficient. I would expect wall-clock training time, GPU-hours or total training FLOPs, and a breakdown of the additional computational overhead. Comparisons under equal computational budgets would be considerably more informative.
The definition of convergence rounds and the stationary validation-accuracy threshold also need clarification.
5. Downstream performance gains are insufficiently explained
Finally, I find the reported accuracy improvements difficult to interpret in light of the mathematical formulation.
If the method is designed to minimize the degradation caused by pruning, its primary objective is to preserve performance while reducing the number of experts. This does not rule out performance improvements, but the mathematical analysis does not explain why substantial gains over the original routing approach should consistently occur.
The experiments also involve fine-tuning experts and routers, making it difficult to isolate whether the improvements result from the proposed interaction-aware selection, additional optimization, or regularization effects.
I would like to see comparisons against the original unpruned MoE under matched fine-tuning conditions, together with an ablation using only diagonal Hessian information. The latter would help establish whether the gains arise specifically from modeling pairwise expert dependencies rather than simply incorporating second-order importance scores.
Overall, this paper although interesting in its submodular optimization attempt does not sufficiently answer any of the important question. If anything it opens up a lot of new threads that are hard to understand. I have some real concerns about the mathematical formulation and analysis, it doesnt seem to me that these are easy fixes either and will require some substantial work. If the authors are able to clarify the mathematical setup and provide substantial experiments I am willing to substantially increase my score. But in its current form it is a reject, as I dont see easy how you can easily fix the mathematical setup.
Questions for authors
1. Can the authors clarify the mathematical formulation in Equations (3)–(4)?
As discussed in my weaknesses, I find the interaction between the original Top- routing mask and the learned mask unclear. Does represent global pruning or input-dependent selection? Is routing recomputed after applying ? How does the actual implementation correspond to the Taylor expansion and Hessian derivation?
2. Can the authors explain the source of the downstream performance improvements?
The mathematical objective is motivated by minimizing the performance degradation associated with expert removal, but the reported results show substantial accuracy improvements. What explains these gains?
I would particularly appreciate comparisons with the original unpruned MoE under matched fine-tuning conditions. Furthermore, if the authors intend the fully activated state to represent dense expert activation and an upper bound on performance, can they empirically validate this assumption?
3. Can the authors provide training curves for Tables 1 and 2?
Rounds to convergence alone do not provide a complete picture of training behavior or computational efficiency. Training curves against both optimization steps and wall-clock time would be informative. How frequently is the Hessian approximation recomputed, and what is its computational overhead?
4. Can the authors provide additional experiments specifically evaluating pruning effectiveness?
I would like to see a more comprehensive evaluation of the accuracy–compression trade-off, including:
- A broader range of pruning ratios and comparisons with the original unpruned model.
- Actual parameter reduction and GPU memory footprint.
- Training/inference time and CPU/GPU throughput.
- Generalization to unseen tasks or out-of-distribution data.
These experiments would help establish whether the proposed approach offers practical advantages as an MoE pruning method.
5. Can the authors clarify the assumptions underlying the MM convergence guarantee?
As discussed in my weaknesses, Equation (10) appears to assume that a first-order Taylor expansion globally lower-bounds the supermodular gain function. Since supermodularity alone does not guarantee this property, I would appreciate clarification on the assumptions