acceptodds
Withdrawn from ICLR 2027

Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

While Mixture-of-Experts (MoE) models effectively scale model capacity through sparse activation, their deployment is often bottlenecked by prohibitive memory requirements. Extracting a compact subset of experts presents a promising solution. However, existing expert selection heuristics predominantly rely on Top-k ranking, which isolates the evaluation of individual experts and ignores the intricate inter-expert dependencies introduced by the MoE gating network. In this paper, we propose DS-MoE, a theoretically grounded framework that redefines expert selection via difference-of-submodular (DS) optimization. By analyzing the second-order Taylor expansion of the loss degradation, we reveal functional duality within expert combinations: redundancy (where experts encode overlapping representations) and synergy (where experts provide complementary error cancellation). To navigate this duality, we mathematically decouple redundancy reduction from synergy maximization by formulating the selection objective as a DS function. Furthermore, we devise a tailored majorization-minimization (MM) algorithm with provable guarantees to efficiently identify the optimal expert subset. Extensive experiments demonstrate that DS-MoE effectively preserves indispensable expert combinations, achieving superior performance compared to the state-of-the-art baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, mixture-of-experts, expert selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Uuw1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper provides a theoretical analysis of the suboptimal learning procedure of MoE expert pruning. The analysis relies on formulating the problem of pruning into a submodular optimization problem and use a second-order Taylor expansion of the loss degradation to estimate pairwise expert interactions.

Based on this formulation, the authors propose a majorization-minimization (MM) algorithm that incorporates estimated second-order information into expert selection. They provide theoretical convergence claims and experimental results showing improved accuracy and fewer rounds to convergence compared with the selected baselines.

Critical strengths

1. Interesting optimization perspective on expert pruning.

I found the attempt to formulate expert selection through difference-of-submodular optimization particularly interesting. Expert pruning is already a well-studied area, but explicitly modeling the trade-off between expert redundancy and synergy provides a useful perspective on the problem. Although I have concerns about the precise formulation, as discussed in the weaknesses, I appreciate the direction of this analysis.

2. A principled approach to modeling expert dependencies.

Rather than relying solely on individual expert importance scores, the authors attempt to capture pairwise interactions using second-order information. I find this motivation compelling, particularly since the best subset of experts need not consist of the individually highest-ranked experts.

3. Interesting algorithmic approach.

The proposed MM procedure provides a tractable approach to an otherwise combinatorial selection problem, reducing each selection update to a Top-kk operation. I appreciate the effort to connect the theoretical formulation to a relatively simple selection algorithm, although I have concerns about the validity of some guarantees.

Critical weaknesses

1. Major concerns with the mathematical formulation (Equations 3–7)

Although I appreciate the mathematical perspective and proposed submodular optimization framework, I have significant concerns about whether the initial formulation represents the expert-pruning problem the authors aim to solve.

1.1. Ambiguity in the reference model.

The authors describe m=1m=\mathbf{1} as a fully activated state and a theoretical upper bound on model capacity. However, Equation (1) already defines G(x)G(x) using Top-kk routing. Therefore, setting m=1m=\mathbf{1} in Equation (3) simply recovers the original sparse MoE, rather than activating every expert.

Moreover, activating all experts would not necessarily improve performance (see, e.g., https://arxiv.org/abs/2509.21892). While the reference loss is constant with respect to mm and need not be optimal, its interpretation should be clarified.

1.2. Applying an additional mask to an already sparse router.

This is my main concern. The authors define the output as:

y(x;m)=∑i=1NmiGi(x)Ei(x)y(x;m)=\sum_{i=1}^{N}m_iG_i(x)E_i(x)

Since G(x)G(x) already performs Top-kk selection, the learned mask mm can only remove previously selected experts, not activate new ones.

This introduces potentially degenerate cases. For example, if the original router selects experts 1 and 2, but mm retains experts 3 and 4, Equation (3) produces a zero MoE-layer output. There is no apparent mechanism for rerouting or renormalization.

More importantly, Equation (7) estimates pairwise interactions using a term proportional to Gi(x)Gj(x)G_i(x)G_j(x). Therefore, the method cannot capture interactions between experts that are never co-activated by the original router. How can the proposed approach discover new synergistic expert combinations beyond those already permitted by the original routing mechanism?

1.3. Consequences for the Taylor expansion.

This ambiguity directly affects the mathematical analysis:

  • If G(x)G(x) is fixed: Equation (5) is a valid local Taylor approximation, but it describes post-routing masking rather than conventional expert pruning with rerouting. Its accuracy for the large, discrete changes involved in pruning is also not established.

  • If the router is recomputed after applying mm: Then GG must depend on mm, and the gradient and Hessian derivations in Equations (6)–(7) no longer hold as written. Differentiation must account for this dependence, and hard Top-kk selection introduces potential nonsmoothness.

I believe the authors need to clarify which operation is actually implemented and establish that their mathematical surrogate corresponds to it.

2. Mathematical issues with the optimization guarantees

I have additional concerns about the theoretical analysis.

First, Equation (10) appears to rely on an invalid lower bound. The authors claim that the first-order Taylor expansion provides a global lower bound for the supermodular gain function ff. However, supermodularity alone does not guarantee this property.

For example, consider f(m1,m2)=m1m2f(m_1,m_2)=m_1m_2, which is supermodular. Its first-order Taylor approximation at (1,0)(1,0) evaluates to 1 at (0,1)(0,1), while the true function value is 0. Therefore, the claimed lower bound fails.

Since this bound is used to construct the MM surrogate, the monotonic convergence guarantee in Theorem 1 is not established by the provided proof.

Are there missing assumption not stated in the paper that would let you get to this bound? If so please state them, and do they affect the rest of the analysis?

3. Unclear framing and insufficient pruning experiments

I find it difficult to determine whether the paper proposes a pruning method or a dynamic expert-selection method.

Equation (4) suggests a globally shared pruning mask, whereas the exposition frequently discusses dynamic routing and improvements during fine-tuning. These are different settings, and the distinction matters for both the theoretical analysis and the efficiency claims.

If this is primarily a pruning method, the experimental evaluation is insufficient. I would expect measurements of actual parameter reduction, GPU memory footprint, inference throughput, and computational cost. The paper evaluates k∈1,2,3k\in\\{1,2,3\\}, but this provides only a limited examination of the accuracy–compression trade-off, particularly without comparisons against the original unpruned model.

I was also surprised by the limited connection to the broader expert-pruning literature. Standard MoE routing and random selection are useful references, but they are not competitive pruning methods. More comparisons with established expert-pruning and expert-merging approaches would strengthen the evaluation.

As presented, the experiments do not sufficiently support the paper's claims regarding practical pruning efficiency.

4. Rounds to convergence do not establish computational efficiency

The authors repeatedly emphasize faster convergence in terms of training rounds. However, this is not necessarily a fair measure of computational efficiency.

The proposed method introduces additional computations to estimate expert interactions and perform MM optimization. Although the MM updates themselves may be inexpensive, constructing and updating the interaction matrix has a cost that is not adequately characterized.

In particular, it is unclear how frequently the Hessian approximation is recomputed during training.

Comparing convergence rounds alone is therefore insufficient. I would expect wall-clock training time, GPU-hours or total training FLOPs, and a breakdown of the additional computational overhead. Comparisons under equal computational budgets would be considerably more informative.

The definition of convergence rounds and the stationary validation-accuracy threshold also need clarification.

5. Downstream performance gains are insufficiently explained

Finally, I find the reported accuracy improvements difficult to interpret in light of the mathematical formulation.

If the method is designed to minimize the degradation caused by pruning, its primary objective is to preserve performance while reducing the number of experts. This does not rule out performance improvements, but the mathematical analysis does not explain why substantial gains over the original routing approach should consistently occur.

The experiments also involve fine-tuning experts and routers, making it difficult to isolate whether the improvements result from the proposed interaction-aware selection, additional optimization, or regularization effects.

I would like to see comparisons against the original unpruned MoE under matched fine-tuning conditions, together with an ablation using only diagonal Hessian information. The latter would help establish whether the gains arise specifically from modeling pairwise expert dependencies rather than simply incorporating second-order importance scores.

Overall, this paper although interesting in its submodular optimization attempt does not sufficiently answer any of the important question. If anything it opens up a lot of new threads that are hard to understand. I have some real concerns about the mathematical formulation and analysis, it doesnt seem to me that these are easy fixes either and will require some substantial work. If the authors are able to clarify the mathematical setup and provide substantial experiments I am willing to substantially increase my score. But in its current form it is a reject, as I dont see easy how you can easily fix the mathematical setup.

Questions for authors

1. Can the authors clarify the mathematical formulation in Equations (3)–(4)?

As discussed in my weaknesses, I find the interaction between the original Top-kk routing mask and the learned mask mm unclear. Does mm represent global pruning or input-dependent selection? Is routing recomputed after applying mm? How does the actual implementation correspond to the Taylor expansion and Hessian derivation?

2. Can the authors explain the source of the downstream performance improvements?

The mathematical objective is motivated by minimizing the performance degradation associated with expert removal, but the reported results show substantial accuracy improvements. What explains these gains?

I would particularly appreciate comparisons with the original unpruned MoE under matched fine-tuning conditions. Furthermore, if the authors intend the fully activated state to represent dense expert activation and an upper bound on performance, can they empirically validate this assumption?

3. Can the authors provide training curves for Tables 1 and 2?

Rounds to convergence alone do not provide a complete picture of training behavior or computational efficiency. Training curves against both optimization steps and wall-clock time would be informative. How frequently is the Hessian approximation recomputed, and what is its computational overhead?

4. Can the authors provide additional experiments specifically evaluating pruning effectiveness?

I would like to see a more comprehensive evaluation of the accuracy–compression trade-off, including:

  • A broader range of pruning ratios and comparisons with the original unpruned model.
  • Actual parameter reduction and GPU memory footprint.
  • Training/inference time and CPU/GPU throughput.
  • Generalization to unseen tasks or out-of-distribution data.

These experiments would help establish whether the proposed approach offers practical advantages as an MoE pruning method.

5. Can the authors clarify the assumptions underlying the MM convergence guarantee?

As discussed in my weaknesses, Equation (10) appears to assume that a first-order Taylor expansion globally lower-bounds the supermodular gain function. Since supermodularity alone does not guarantee this property, I would appreciate clarification on the assumptions

Reviewer_fJFJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DS-MoE, a dependency-aware expert selection approach that considers both individual expert importance and its pairwise interactions. Specifically, the method uses loss-gradient information to model expert redundancy and synergy, and applies an MM-based optimization algorithm for expert selection. Experiments on several common-used MoE models report improvements in accuracy and convergence iterations over existing methods.

Critical strengths
  1. The research question is interesting and promising. The paper explicitly considers the dependencies among experts rather than evaluating each expert independently. This is a meaningful direction, as the value of an expert may depend on which other experts are selected.

  2. Reasonable formulation with loss-aware interaction. Modeling expert selection through binary masks keeps the optimization dimension much smaller than the number of model parameters. Moreover, compared with methods based on expert output similarity, the loss-aware interaction scores provide a more direct way to estimate how different experts jointly affect model performance.

Critical weaknesses

My main concern is about the potential issue with the theoretical guarantee in Eq. (10). The derivation appears to confuse supermodularity on binary variables with convexity in the continuous domain. A supermodular function does not necessarily satisfy the first-order lower-bound inequality used in Eq. (10). For example, consider

f(m1,m2)=m1m2.f(m_1,m_2)=m_1m_2.

This function is supermodular on binary variables, but its first-order approximation at (1,0)(1,0) gives a value of 11 at (0,1)(0,1), while the actual function value is 00. Therefore, the inequality in Eq. (10) does not generally hold. Since the MM updates and convergence analysis rely on this inequality, the claimed monotonic improvement and convergence guarantees are not sufficiently justified.

Another concern is about experiments: The main experiments report convergence iterations and acc, but actually fewer iterations do not necessarily translate into lower training costs. Collecting gradient statistics and optimizing expert combinations may introduce considerable overhead. I guess the more meaningful comparison would report the total wall-clock time required to reach the same accuracy, together with peak GPU memory usage under identical hardware settings.

Questions for authors

What exactly does the expert budget kk represent? The formulation describes kk as the number of experts that can fit within a storage budget, while the experiments appear to use it as the number of activated experts. Could the authors clarify whether the method performs expert pruning or per-token expert selection, and whether unselected experts remain in memory?

Other Questions please see weaknesses.

Reviewer_kUHaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The primary concept of this paper is to address a critical limitation in existing Mixture-of-Experts (MoE) routing mechanisms. Traditional Top-k selection evaluates individual experts in absolute isolation, implicitly treating the loss landscape’s Hessian as purely diagonal and failing to capture combinatorial inter-expert dependencies. To resolve this, the authors propose DS-MoE, a theoretically principled framework that models dynamic expert selection using a second-order Taylor expansion of loss degradation around the fully activated state. The key contributions are first, the authors provide a theoretical characterization of inter-expert interactions via the Gauss-Newton Hessian matrix, mathematically defining redundancy and synergy. Second, they formulate the cardinality-constrained quadratic binary optimization problem as a Difference-of-Submodular (DS) problem by decomposing the indefinite Hessian into two non-negative, supermodular components. Third, they design a tailored Majorization-Minimization (MM) algorithm with guaranteed monotonic convergence, which linearizes the NP-hard combinatorial problem into an efficient iterative Top-k scoring procedure. Finally, comprehensive empirical evaluations across Switch-Transformer and DeepSeek-MoE backbones show that DS-MoE achieves superior test accuracy and substantially faster convergence over state-of-the-art baselines.

Critical strengths
  • The paper presents an exceptionally cohesive storyline that bridges continuous optimization curvature with discrete combinatorial selection. Rather than relying on empirical heuristics or ad-hoc scoring tricks like prior sparse MoE literature, the authors rigorously formulate expert pruning from the second-order Taylor expansion.
  • Directly solving the cardinality-constrained quadratic binary optimization is fundamentally NP-hard. The authors introduce a tight Majorization-Minimization (MM) surrogate. The approach reduces to a simple Top-k sorting operation while mathematically guaranteeing monotonic convergence in a finite number of steps.
  • The empirical evaluation is thorough, systematically benchmarking DS-MoE against strong contemporary baselines such as SEER-MoE, DiEP, and standard Top-kk MoE across diverse tasks (AGNews, PIQA, HellaSwag, MMLU) and architectures (Switch-Base-32/64, DeepSeek-MoE-16B)
Critical weaknesses
  • The paper defines the selection vector m alongside token routing G(x), but leaves the operational scope ambiguous. It is unclear whether m is solved globally per layer as a static subset of active experts (pruning/expert-level routing) or dynamically recomputed per token/batch.
  • The paper exclusively reports "Convergence Rounds" to argue for training efficiency. However, calculating the Gauss-Newton Hessian approximation requires computing gradient inner products across expert representations over evaluation datasets. If each iteration step requires substantial computational overhead to assemble H_m and run the MM loop, fewer rounds might actually translate into longer overall wall-clock training time, completely offsetting the reported convergence gains.
  • The manuscript exhibits notable textual errors and mathematical notation mismatches. In Section 3.2, line 364 states: "DS-MoE converges in just 45 rounds reducing S-MoE’s required 100 rounds by more than half", yet Table 1 lists S-MoE on DeepSeek-MoE-16B as requiring 85 rounds. Furthermore, while the mathematical formulations throughout Section 2 specify strict equality constraints norm(m) = k, the prose interchangeably writes inequality constraints norm(m) <= k, introducing formal inconsistency.
Questions for authors
  • Explicitly clarify the exact execution frequency and granularity of the binary selection vector m?
  • What is the actual wall-clock training time (in GPU hours) compared to standard S-MoE and DiEP under identical hardware environments? Does the additional computation required to calculate curvature inner products outweigh the benefits gained from fewer convergence rounds, particularly when scaling to larger batch sizes or trillion-token datasets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.