acceptodds
Under review as a conference paper at ICLR 2027

Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models

Abstract

Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert parameters. We study post-training expert pruning using task-domain calibration data as a subset-selection problem: given a pretrained MoE layer, select a fixed number of experts that best reconstruct its original output. Exact subset search is combinatorial and becomes intractable for modern MoE models with many experts per layer. We develop OptEP, a scalable continuous relaxation that directly optimizes differentiable expert-selection variables, followed by a lightweight reconstruction-based update of the retained expert weights. Across multiple MoE architectures and tasks spanning mathematical reasoning and code generation, our approach improves over the strongest existing expert-pruning baselines, with absolute accuracy gains of up to 15.05% on Qwen3-30B-A3B, 14.55% on Qwen3-235B-A22B, 11.06% on GLM-4.5-Air, and 16.20% on Mixtral-8x7B. The improvements are particularly pronounced under aggressive pruning, demonstrating that reconstruction-driven expert selection and weight adaptation can preserve substantially more of the original model's performance while reducing the number of retained experts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts (MoE), Expert Pruning, Model Compression, Efficient Large Language Models, Post-Training Optimization, Reconstruction-Based Pruning, Continuous Optimization, Subset Selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dNCTOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OptEP, a layer-wise expert-pruning method that learns continuous expert-selection variables using local output reconstruction, followed by expert weight updates (EWU).

Critical strengths

The emperical performance of the paper is good and the evaluation covers several large MoE models like Qwen3-235B-A22B.

Critical weaknesses
  1. Expert selection is performed independently for each layer using inputs from the full, unpruned model. Consequently, pruning decisions in earlier layers do not affect the inputs used to select experts in later layers. This ignores the accumulation of pruning errors across layers and creates a mismatch between the selection stage and the final pruned model.

  2. The method uses a simple continuous relaxation of binary selection variables, optimized with SGD and clamped to ([0,1]). During optimization, it neither enforces the target number of retained experts nor uses the actual sparse Top-(K) routing. The final Top-(s) discretization may therefore differ substantially from the optimized continuous model. More advanced methods, such as Gumbel-Sigmoid with STE, should be included for comparison.

  3. The novelty over previous work is unclear. Differentiable subset selection guided by the outputs of the original model has already been explored. The paper should better distinguish OptEP from existing differentiable pruning [1] with knowledge-distillation methods [2] and explain which technical component constitutes the main contribution.

  4. EWU provides a large portion of the reported improvements, but it is only applied to experts selected by OptEP. Other baselines are evaluated without the same weight-update procedure. Applying EWU to REAP, EASY-EP, and other selection baselines is necessary to determine whether the gains come from the proposed expert-selection method or from the additional weight optimization.

[1] Differentiable Subset Pruning of Transformer Heads. TACL 2021.

[2] ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning. TMLR 2026.

Questions for authors

See weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer dNCT (1/4): Impact of Sequential vs Parallel Expert Pruning on Accuracy and Efficiency

We thank the reviewer for their constructive feedback and for recognizing the strong empirical performance of OptEP across large-scale MoE architectures, including Qwen3-235B-A22B. Below, we address the reviewer's concerns regarding layer-wise pruning and error accumulation, the continuous relaxation and its alignment with the discrete pruning objective, the novelty of our approach relative to prior work, and the independent contributions of expert selection and EWU.

Q1. Sequential vs. Parallel Expert Pruning. We thank the reviewer for the insightful question. As suggested, we further investigate whether sequential expert pruning provides an advantage over our proposed parallel pruning approach. Specifically, we compare sequential OptEP against parallel OptEP without expert weight updates (EWU) on GLM-4.5-Air.

As shown in Table R1 below, sequential and parallel pruning achieve comparable performance across all six settings on MATH-500 and LiveCodeBench. The accuracy differences are small (within 1% in five out of six settings) and are generally small relative to the reported standard errors. These results suggest that sequential pruning does not provide a consistent performance advantage over parallel pruning.

More importantly, sequential pruning requires a separate pruning procedure for each target number of retained experts, whereas our parallel approach performs pruning only once and obtains solutions for all pruning levels. Therefore, parallel pruning offers greater computational efficiency and flexibility while maintaining comparable downstream accuracy.

Table R1. Sequential vs parallel pruning results. Results are reported as mean accuracy (%) ± standard error over three calibration seeds. Δ denotes sequential minus parallel accuracy in percentage points (pp).

BenchmarkKeepParallel OptEP (uw)Sequential OptEP (uw)Δ (pp)
MATH-5003277.73 ± 1.8478.73 ± 0.44+1.00
MATH-5004892.53 ± 1.0992.93 ± 0.13+0.40
MATH-5006497.07 ± 0.0796.47 ± 0.73−0.60
LiveCodeBench6451.99 ± 1.3952.36 ± 0.45+0.37
LiveCodeBench8062.56 ± 2.0062.94 ± 0.97+0.38
LiveCodeBench9662.06 ± 2.3663.93 ± 0.33+1.87
AuthorsOpenReview ↗

Response to Reviewer dNCT (2/4): Validity of the Continuous Relaxation

Q2. We thank the reviewer for raising this important question. To investigate the alignment between our continuous relaxation and the exact discrete pruning objective, we conduct four complementary analyses on Mixtral.

a. Optimality Gap of the Continuous Relaxation (added as Appendix D.2). We compare the expert subsets selected by our continuous relaxation with the globally optimal subsets obtained through exhaustive search. Both are evaluated using the same exact reconstruction loss with binary expert selection and Top-K routing. As shown in Table R2 below, our method achieves reconstruction losses only 0.71–3.24% higher than the global optimum. Although it identifies the exact optimal subset in 28–43% of layer–seed pairs, the selected subsets have a mean rank of only 3.3–6.8 among 28 or 70 possible subsets. These results demonstrate that our relaxation consistently identifies near-optimal expert subsets under the exact discrete reconstruction objective.

Table R2. Comparison of expert subsets selected by exhaustive search and continuous optimization followed by discretization on Mixtral. Both are evaluated using the same exact reconstruction objective.

DatasetsGapOverlapAgreementRank
GSM8K40.71%81.1%68/1606.1/70
GSM8K62.58%89.0%56/1603.9/28
MBPP42.86%78.6%44/1606.8/70
MBPP63.24%89.5%63/1603.3/28

b. Effect of Top-K Routing During Continuous Optimization (added as Appendix D.3). We evaluate a variant that retains Top-K routing in the continuous optimization objective. This substantially degrades downstream performance: GSM8K accuracy drops from 12.89% to 2.23% with four retained experts and from 25.32% to 9.40% with six retained experts. We hypothesize that Top-K routing restricts gradient updates to the selected experts for each token, potentially reinforcing early selections and limiting exploration of alternative subsets. In contrast, our proposed relaxation allows all experts to receive gradient updates, leading to substantially better expert selection. Results are in Table R3 below.

c. Effect of Enforcing Binary Expert Selection During Optimization (added as Appendix D.4). We additionally evaluate a variant that applies hard thresholding after every optimization step, enforcing binary selection variables and retaining exactly the prescribed number of experts. This variant consistently underperforms our proposed relaxation, which maintains continuous selection scores and enforces the expert budget only after optimization. On GSM8K, accuracy drops from 12.89% to 9.17% with four retained experts and from 25.32% to 22.05% with six retained experts. We observe similar declines on MBPP. These results demonstrate the benefit of maintaining continuous selection scores throughout optimization and postponing discrete expert selection until the final step. Results are in Table R3 below.

d. Gumbel-STE Expert Selection. We additionally evaluate replacing our dense relaxation with Gumbel-Sigmoid selection trained via a straight-through estimator (STE), which keeps the forward pass identical to pruned inference by applying a hard Top-s expert mask and Top-K routing. We test 1, 8, and 16 noise samples per step; the 16-sample variant additionally anneals the noise scale and initializes the selection from our dense-relaxation subsets. All variants substantially degrade downstream performance. GSM8K accuracy drops from 12.89% to 3.85%, 4.26%, and 9.78% with four retained experts, and from 25.32% to 13.57%, 10.37%, and 10.87% with six retained experts. Results are in Table R3 below.

Table R3: Accuracy of Mixtral-8x7B after expert pruning with our approach (no top_k), hard top-2 routing during optimization (top_k = 2), and Gumbel-STE selection with 1, 8, or 16 noise samples per step. Values are mean ± standard error over five seeds: flexible-match accuracy (%) on GSM8K and standard pass@1 (%) on MBPP. Best result per row in bold.

BenchmarksOurs (no top_k)top_k = 2binary zzGumbel, 1 sampleGumbel, 8 samplesGumbel, 16 samples + anneal + init
GSM8K412.89 ± 0.002.23 ± 0.109.17 ± 0.883.85 ± 0.844.26 ± 1.439.78 ± 1.60
GSM8K625.32 ± 0.379.40 ± 1.8622.05 ± 2.2613.57 ± 3.2110.37 ± 3.3110.87 ± 4.34
MBPP430.31 ± 0.6121.08 ± 0.5926.66 ± 0.6117.14 ± 4.397.97 ± 4.7622.04 ± 4.71
MBPP647.61 ± 0.5940.35 ± 2.1342.66 ± 0.8238.10 ± 1.1439.25 ± 1.3532.31 ± 6.96

Together, these experiments demonstrate that our dense continuous relaxation identifies near-optimal expert subsets under the exact discrete reconstruction objective, while explicitly incorporating Top-K routing, an expert budget constraint with binary zz’s, or Gumbel-STE selection during optimization substantially degrades downstream performance.

AuthorsOpenReview ↗

Response to Reviewer dNCT (3/4): Main Contributions and Distinctions from Prior Differentiable Pruning Methods

Q3. We thank the reviewer for prompting us to clarify our contributions in relation to the cited works and broader literature. We have updated the Related Work section of the revised paper (Section 2) accordingly, and we elaborate below.

Main contribution and closest prior work. OptEP introduces the scalable approach to directly optimizing joint expert selection through MoE-layer output reconstruction. NAEE [3] is the closest prior work: it considers a similar reconstruction objective but exhaustively enumerates all (Es)\binom{E}{s} expert subsets, which becomes intractable for modern MoEs with 128 experts per layer. Our normalized continuous surrogate (Eqs. 4–5) instead enables gradient-based joint optimization of expert-selection variables, capturing redundancy and complementarity between experts rather than relying on fixed importance scores. Importantly, when exhaustive search is feasible (Mixtral), our relaxation achieves comparable downstream performance to brute-force optimization of the corresponding discrete objectives (Appendix Table 11), while scaling to substantially larger expert pools.

Comparison with DSP [1]. DSP selects attention heads by minimizing supervised downstream task loss using a Gumbel-top-KK relaxation. In contrast, OptEP jointly selects whole experts by directly minimizing original MoE-layer reconstruction error. DSP's relaxation requires KK sequential softmax operations, introduces stochasticity through Gumbel sampling, and relies on temperature annealing to approach discrete selection. Our deterministic relaxation instead requires a single normalization of the router-weighted expert mixture, avoiding sequential sampling and annealing, followed by top-ss selection after optimization. Thus, OptEP differs in both its reconstruction-based objective and its substantially simpler, more computationally efficient optimization procedure.

Comparison with ToMoE [2]. ToMoE addresses a fundamentally different problem: converting a dense model into an MoE by learning channel-selection masks that define new experts, jointly with a new router. These are optimized using KL distillation on the model's final output distribution, with regularization to reduce active computation per token. In contrast, OptEP compresses an existing pretrained MoE by jointly selecting a fixed subset of its original experts, preserving the pretrained router parameters and directly minimizing layerwise reconstruction error. Thus, although both optimize selection variables using gradients, ToMoE neither addresses reconstruction-based selection of existing experts nor provides our scalable optimization procedure for this problem.

Our second contribution is a lightweight, reusable expert-weight update (EWU). Our regularized reconstruction-based update can be applied after any expert-pruning method, adapting retained experts to compensate for those removed. Unlike the end-to-end task fine-tuning used by SEER-MoE [4], our method performs lightweight layerwise optimization using existing calibration data, without backpropagation through the full model. This provides a computationally inexpensive recovery mechanism that is independent of our expert-selection algorithm.

Both contributions are independently supported by strong empirical results. Without weight updates, at least one of our weighted and unweighted variants outperforms the strongest prior pure-pruning baseline in 21/22 settings, with absolute accuracy gains of up to 12.33% and 10.32% on Qwen3-30B-A3B and Qwen3-235B-A22B, respectively. Furthermore, across both OptEP variants, the best performance after expert-weight updates (EWU) exceeds the best performance before EWU in 20/22 settings, with additional absolute gains of up to 16.38% on Mixtral-8x7B.

References:

  1. Differentiable Subset Pruning of Transformer Heads. TACL 2021.

  2. ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning. TMLR 2026.

  3. Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models. ACL 2024.

  4. SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts. 2024.

Reviewer_mEepOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

In this paper, the authors introduces OptEP, a post-training expert pruning framework for Mixture-of-Experts (MoE) models that formulates expert selection as a joint reconstruction-based subset optimization problem. It employs a scalable continuous relaxation to directly optimize differentiable expert-selection variables, minimizing the layer-output reconstruction error to avoid the combinatorial cost of exhaustive search. Additionally, an inverse-frequency weighting scheme is incorporated to preserve crucial but infrequently activated experts that are often overlooked by traditional importance-based methods. The framework also includes a lightweight reconstruction-based expert weight update mechanism to adapt the retained experts and compensate for the capacity lost during pruning.

Critical strengths

None

Critical weaknesses
  • The manuscript shows signs of substantial AI assistance, and parts of the exposition are extremely difficult to follow.

  • The paper contains no figures, which makes the method and experimental pipeline hard to follow. This is unusual for ICLR submissions and significantly hinders readability.

  • The paper relies on an excessive amount of notation and complex formulas, especially in Section 3, without sufficient intuitive explanation.

Questions for authors

See weakness

6 comments
AuthorsOpenReview ↗

Response to Reviewer mEep: Clarifications on AI Assistance, Presentation, and Technical Soundness

Dear Reviewer,

Thank you for your feedback. We would like to address your concerns and clarify several points.

1. AI Assistance and Ethics Concerns

As disclosed in our submission, we used AI tools to assist with method implementation and manuscript editing to improve readability, which is permitted under ICLR's policies. The authors wrote the entire initial draft of the manuscript and carefully reviewed and edited all AI-generated writing suggestions before incorporating them. The authors take full responsibility for the methodology, experiments, and conclusions.

Could you provide concrete evidence suggesting AI use beyond what we have already disclosed, or clarify which aspect of our disclosed use raises ethical concerns? We believe it is important to distinguish permitted AI assistance from research-integrity violations.

2. Absence of Figures

We present our experimental results in tables because they allow direct numerical comparisons across pruning methods, models, and expert-retention budgets. This is consistent with prior MoE pruning work, including EASY-EP and REAP, which also report their results in tables.

3. Mathematical Presentation in Section 3

Section 3 introduces our layerwise optimization-based approach to MoE expert pruning. The main formulations can be summarized as follows:

  • Eqs. (1)–(2): Define the outputs of the original and pruned MoE layers.
  • Eq. (3): Formulates the exact reconstruction loss, whose combinatorial optimization is computationally prohibitive for large models.
  • Eqs. (4)–(5): Introduce a continuous relaxation enabling efficient optimization of expert-selection variables.
  • Eqs. (6)–(7): Extend the formulation with inverse-frequency weighting.
  • Eq. (8): Introduces an expert-weight update that adapts the retained experts after pruning.

In summary, our method consists of joint expert selection through continuous optimization, followed by optional adaptation of the retained expert weights. We appreciate the suggestion to make this intuition more explicit in the paper.

4. Soundness and Scientific Contribution

Finally, the review assigns the lowest soundness and informativeness ratings without identifying a specific methodological flaw, experimental issue, or incorrect result. Our paper introduces a scalable optimization-based expert-pruning method and provides extensive experiments across four large MoE models, including a 235B-parameter model, multiple benchmarks, expert-retention budgets, and competitive baselines. The results demonstrate substantial improvements over existing methods.

We would appreciate clarification on the concrete technical concerns motivating these ratings so that we can address them directly.

Thank you for your consideration.

Dear Authors,

Thank you for your explanation. My concerns have been partially addressed and I sincerely appreciate the extensive experiments conducted across four large MoE models.

I acknowledge that the initial draft of the manuscript was written entirely by the authors. However, as I understand it, many purely AI-generated papers lack figures. Including one or two clear and effective figures would help demonstrate the proposed method and improve the manuscript’s readability. I therefore suggest that the authors add simple but informative figures to illustrate the approach. Some of the tables could also be converted into figures to present the results more clearly.

AuthorsOpenReview ↗

Response to Reviewer mEep: Improved Presentation with Additional Figures

Dear Reviewer,

Thank you for your response and for reconsidering your initial assessment. We appreciate your constructive suggestions to improve the manuscript's clarity and readability through additional figures. In response to your feedback, we have added four figures to the revised manuscript:

1. Figure 1 (Overview of OptEP): We added a figure illustrating the three stages of our method: (a) optimizing expert-selection variables to reconstruct the original MoE layer output while keeping expert weights and the router fixed, (b) pruning experts based on the optimized selection variables, and (c) optionally adapting the retained expert weights through Expert Weight Update (EWU).

2. Figure 2 (Appendix D, Pruning Performance): We added a figure comparing OptEP with existing pruning baselines across all four models and different expert-retention budgets. The results show that OptEP (w) + EWU matches or outperforms competing methods at most budgets while generally preserving accuracy closest to that of the unpruned model.

3. Figures 3 and 4 (Appendix E, Inference Efficiency): We added two figures illustrating end-to-end generation time savings from expert pruning across different concurrency levels. For Qwen3-30B-A3B, pruning reduces generation time by up to 58.3% on math and 34.9% on coding, while Qwen3-235B-A22B achieves reductions of up to 31.7% and 39.2%, respectively. These results highlight the practical inference-efficiency benefits of our approach.

We hope these additions address your presentation concerns and make the proposed method, experimental results, and practical benefits easier to understand. Thank you again for your feedback and consideration.

Reviewer_GXwqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes OptEP, a reconstruction-based framework for post-training expert pruning in MoE models. Unlike conventional importance-based methods that independently rank experts, OptEP formulates pruning as a joint subset-selection problem that minimizes the reconstruction error of the original MoE layer. To avoid combinatorial search, the authors introduce a continuous relaxation of expert-selection variables and an inverse-frequency-weighted reconstruction objective to better preserve infrequently activated experts. After selecting the retained experts, an additional Expert Weight Update stage adapts their parameters through layerwise reconstruction. Experiments show substantial improvements over recent pruning baselines. Also, the comparisons with exhaustive subset selection on Mixtral further support the practical effectiveness of the proposed approximation.

Critical strengths
  1. The paper addresses an important limitation of current importance-based pruning. Individually important experts do not necessarily form the best subset because their contributions can be redundant or complementary. Optimizing layer-output reconstruction provides a more direct objective for selecting experts jointly. The continuous formulation makes this approach applicable to models with large expert pools, while inverse-frequency weighting offers an intuitive way to reduce the dominance of frequently activated experts.

  2. Strong experimental performance across diverse MoE architectures. The experiments cover four latest models with different expert configurations and two task domains. The improvements under aggressive pruning are particularly convincing.

Critical weaknesses

At first, the continuous surrogate is not fully aligned with the actual sparse pruning objective. In Eq. (4), the relaxed output is computed as a normalized combination of all experts, whereas the final pruned model in Eq. (2) still performs Top-K sparse routing. Consequently, even when the selection variables are binary, the relaxed output generally differs from the actual pruned-layer output:

\neq \widehat{y}^{(K)}(x;\mathbf{1}_S).$$ Moreover, the continuous optimization does not enforce the expert-retention budget, which is imposed only through post-hoc Top-s discretization. This creates a potentially significant mismatch between the optimized surrogate and the deployed architecture. Although the brute-force comparisons on Mixtral provide encouraging empirical evidence, the paper does not directly quantify how well the relaxed objective preserves the actual sparse Top-K reconstruction error after discretization. This weakens the justification for the proposed optimization formulation. In the experiments, the contribution of expert selection versus weight adaptation is not fully isolated in comparisons with existing methods. A main improvement comes from EWU, which can be applied independently of the expert-selection method. However, the main comparisons evaluate OptEP + EWU against pruning baselines without an equivalent reconstruction-based weight update. For example, on Mixtral GSM8K with four retained experts, OptEP (w) improves from 12.89% to 28.16% after EWU. While the pruning-only results are already competitive, applying the same EWU procedure to strong baselines such as REAP would provide a more convincing assessment of the independent contribution of OptEP's selection strategy. Additionally, although scalability is a central motivation, the paper does not report the practical pruning time or memory overhead relative to these baselines.
Questions for authors
  1. Since Eq. (4) uses dense expert aggregation while the final model uses sparse Top-K routing, how closely does the optimized surrogate objective correlate with the actual reconstruction error after Top-s discretization? Are the corresponding reconstruction errors available for the Mixtral experiments, particularly when compared with the exact discrete solutions?

  2. Could the auther apply the same EWU procedure to strong baselines such as REAP would provide a more convincing assessment of the independent contribution of OptEP's selection strategy.

  3. Could the authors provide approximate pruning time, GPU-hours, or peak memory requirements relative to importance-based baselines.

5 comments
AuthorsOpenReview ↗

Response to Reviewer GXwq (1/3): Validity of the Continuous Relaxation

We thank the reviewer for their constructive feedback and for recognizing the importance of joint expert selection, the scalability of our approach, and its strong empirical performance across diverse MoE architectures. Below, we address the reviewer's specific questions and concerns regarding the continuous relaxation, the independent contributions of expert selection and EWU, and computational overhead.

Q1. We thank the reviewer for raising this important question. To investigate the alignment between our continuous relaxation and the exact discrete pruning objective, we conduct three complementary analyses on Mixtral.

a. Optimality Gap of the Continuous Relaxation (added as Appendix D.2). We compare the expert subsets selected by our continuous relaxation with the globally optimal subsets obtained through exhaustive search. Importantly, both are evaluated using the same exact reconstruction loss with binary expert selection and Top-K routing. As shown in Table R1 below, our method achieves reconstruction losses only 0.71–3.24% higher than the global optimum. Although it identifies the exact optimal subset in 28–43% of layer–seed pairs, the selected subsets have a mean rank of only 3.3–6.8 among 28 or 70 possible subsets. These results demonstrate that our relaxation consistently identifies near-optimal expert subsets under the exact discrete reconstruction objective.

Table R1. Comparison of expert subsets selected by exhaustive search and continuous optimization followed by discretization on Mixtral. Both are evaluated using the same exact reconstruction objective.

DatasetsGapOverlapAgreementRank
GSM8K40.71%81.1%68/1606.1/70
GSM8K62.58%89.0%56/1603.9/28
MBPP42.86%78.6%44/1606.8/70
MBPP63.24%89.5%63/1603.3/28

b. Effect of Top-K Routing During Continuous Optimization (added as Appendix D.3). We evaluate a variant that retains Top-K routing in the continuous optimization objective. This substantially degrades downstream performance: GSM8K accuracy drops from 12.89% to 2.23% with four retained experts and from 25.32% to 9.40% with six retained experts. We hypothesize that Top-K routing restricts gradient updates to the selected experts for each token, potentially reinforcing early selections and limiting exploration of alternative subsets. In contrast, our proposed relaxation allows all experts to receive gradient updates, leading to substantially better expert selection. Results are in Table R2 below.

c. Effect of Enforcing Binary Expert Selection During Optimization (added as Appendix D.4). We additionally evaluate a variant that applies hard thresholding after every optimization step, enforcing binary selection variables and retaining exactly the prescribed number of experts. This variant consistently underperforms our proposed relaxation, which maintains continuous selection scores and enforces the expert budget only after optimization. On GSM8K, accuracy drops from 12.89% to 9.17% with four retained experts and from 25.32% to 22.05% with six retained experts. We observe similar declines on MBPP. These results demonstrate the benefit of maintaining continuous selection scores throughout optimization and postponing discrete expert selection until the final step. Results are in Table R2 below.

Table R2: Accuracy of Mixtral-8x7B after expert pruning with our approach (no top_k), hard top-2 routing during optimization (top_k = 2), and binary zz with expert budget constraint ∑ze=s\sum z_e = s at every optimization step. Values are mean ± standard error over five seeds: flexible-match accuracy (%) on GSM8K and standard pass@1 (%) on MBPP. Best result per row in bold.

BenchmarksOurs (no top_k)top_k = 2binary z with discrete expert constraint
GSM8K412.89 ± 0.002.23 ± 0.109.17 ± 0.88
GSM8K625.32 ± 0.379.40 ± 1.8622.05 ± 2.26
MBPP430.31 ± 0.6121.08 ± 0.5926.66 ± 0.61
MBPP647.61 ± 0.5940.35 ± 2.1342.66 ± 0.82

Together, these experiments demonstrate that our continuous relaxation identifies near-optimal expert subsets under the exact discrete reconstruction objective, while explicitly enforcing either Top-K routing or the discrete expert budget during optimization substantially degrades pruning performance. These findings support our design choice to optimize over all experts using continuous selection scores and impose the final sparsity constraints only after optimization.

AuthorsOpenReview ↗

Response to Reviewer GXwq (2/3): Comparison with Baselines After Expert Weight Updates (EWU)

Q2. In the main paper, we separately evaluate pruning-only methods without any model weight updates. Our weighted pruning only approach outperforms the baselines in 18 of 22 settings, with gains of up to 10.45%, while trailing by at most 0.5% in the remaining four settings. Considering both our weighted and unweighted variants, our method outperforms the baselines in 21 of 22 settings. Expert Weight Update (EWU) is an additional adaptation step that can be applied to any pruning method, not just ours.

To ensure a fair comparison, we apply the same EWU procedure to all pruning baselines on Qwen3-30B-A3B, with results reported in Table R3 below (also in Tables 5 and 6 of the paper Appendix).

  • OptEP (w) + EWU outperforms REAP + EWU in all settings, with absolute gains of up to 8.93% on math and 16.29% on coding.
  • Compared with EASY-EP + EWU, our method performs better in all but two settings, where it trails by only 0.54% on math (37.5% retained experts) and 3.11% on coding (50% retained experts; this difference goes down to 1.24% if we consider OptEP (uw) + EWU) but these two differences are not statistically significant due to high variance. In the remaining settings, OptEP (w) + EWU achieves gains of up to 21.14% over EASY-EP + EWU.
  • M-SMoE already changes expert weights but we apply our EWU on top of it as well. OptEP (w) + EWU outperforms M-SMoE + EWU in five of six settings, with gains of up to 85.93%. The only exception is coding with 75% of experts retained, where M-SMoE + EWU leads by 1.24%; however, this difference is not statistically significant, and both methods outperform the full model.
  • OptEP (w) + EWU also consistently outperforms Frequency + EWU across all settings.
  • We exclude Random + EWU from the comparison because its accuracy remains below 1% even after weight updates.

As discussed in the paper, methods with stronger pruning-only performance generally remain stronger after EWU. When pruning-only performance is similar, EWU can change the relative ranking of methods although they remain close and the differences are not statistically significant. However, larger performance gaps introduced during pruning tend to persist after adaptation. These findings highlight that EWU is broadly applicable across pruning methods, but its effectiveness still depends on the quality of the initial expert selection.

R3. Expert pruning with weight updates on Qwen3-30B-A3B after our proposed method and baselines. The best result in each column is shown in bold, and the second-best result is italicized.

MethodMath (s=32)Math (s=48)Math (s=64)Coding (s=48)Coding (s=64)Coding (s=96)
Frequency + EWU1.40 ± 0.2081.53 ± 2.2179.80 ± 3.930.00 ± 0.0018.41 ± 0.7661.19 ± 0.57
EASY-EP + EWU85.27 ± 0.5894.67 ± 0.8295.67 ± 0.5730.97 ± 0.3758.21 ± 0.8661.07 ± 0.87
REAP + EWU78.00 ± 2.5592.67 ± 0.3794.80 ± 0.1235.82 ± 2.0653.36 ± 0.5758.71 ± 0.54
M-SMoE + EWU1.00 ± 0.1283.27 ± 1.7979.53 ± 2.770.00 ± 0.0019.28 ± 1.0062.56 ± 1.40
OptEP (uw) + EWU90.73 ± 0.2793.40 ± 0.2095.60 ± 0.1253.11 ± 0.6656.97 ± 0.8759.95 ± 0.97
OptEP (w) + EWU86.93 ± 1.8394.13 ± 0.3795.87 ± 0.4752.11 ± 2.0055.10 ± 1.6361.32 ± 0.69
AuthorsOpenReview ↗

Response to Reviewer GXwq (3/3): Pruning and Evaluation Runtime Comparison with Baselines

Q3. We thank the reviewer for this question. Table R4 reports the mean pruning and evaluation runtimes for each method on Qwen3-30B-A3B, using both coding and math calibration sets.

Pruning Runtime. OptEP takes approximately 5–8× longer to prune than REAP and EASY-EP (roughly 6 hours vs. 1 hour). This difference is expected, as OptEP performs gradient-based optimization for every MoE layer, whereas REAP and EASY-EP require only a forward pass to collect expert usage statistics. However, it is important to note that pruning is a one-time cost across expert pruning budgets, while the resulting inference/eval savings accumulate over repeated deployments/evals.

Evaluation Runtime. OptEP models evaluate faster than REAP and EASY-EP models. For example, the difference with EASY-EP is substantial (0.71 vs. 1.78 hours on coding; 0.44 vs. 1.11 hours on math). This is because OptEP models achieve higher accuracy and produce shorter outputs.

Overall, OptEP trades higher one-time pruning costs for better downstream performance and lower evaluation runtimes. Since evaluation is repeated across multiple expert budgets, these runtime savings can accumulate and offset the additional pruning cost. We will include this runtime comparison in the revised paper.

Table R4. Mean pruning, evaluation, and total runtimes per method on Qwen3-30B-A3B (4× H100, averaged over 3 seeds and all sparsity levels)

MethodCoding: pruning (h)Coding: eval (h)Coding: pruning + eval (h)Math: pruning (h)Math: eval (h)Math: pruning + eval (h)
OptEP (uw)5.780.716.496.310.446.75
OptEP (w)5.820.736.556.060.316.36
REAP1.230.782.011.200.361.56
EASY-EP1.031.782.810.781.111.89
Frequency0.912.823.730.621.772.38
M-SMoE0.711.512.220.272.462.73
Random0.151.131.280.511.842.35

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We thank the reviewers for their thoughtful feedback and constructive discussion. In the revised manuscript and rebuttal, we have added experiments and clarifications that directly address the main concerns:

Validity of the continuous relaxation. New comparisons with exhaustive search on Mixtral show median reconstruction-loss gaps of only 0.71–3.24%. Additional comparisons with Top-K routing, binary selection, and Gumbel-STE support our choice of continuous relaxation.

Sequential versus parallel pruning. New experiments on GLM-4.5-Air show comparable accuracy between sequential and parallel expert selection, supporting our approach’s ability to obtain multiple pruning budgets from a single optimization run.

Fair comparisons and weight recovery. We expanded comparisons on Qwen3-30B-A3B using the same expert-weight update across competing methods. These results demonstrate that OptEP retains strong advantages under matched recovery, while the weight update also benefits other pruning methods.

Practical efficiency. We report the one-time pruning cost alongside downstream efficiency gains. On Qwen3-30B-A3B, OptEP (w) achieves lower mean evaluation time than all compared baselines, consistent with shorter generated outputs, alongside strong accuracy. Separate inference benchmarks on both Qwen models show generation-time reductions of up to 58.3% and lower memory requirements relative to the unpruned models.

Contributions and presentation. We identify NAEE as the closest prior work and clarify our distinctions from DSP, ToMoE, and SEER-MoE. We also added four figures illustrating the method, accuracy comparisons, and inference efficiency.

These additions strengthen the empirical support for OptEP and clarify its contributions to scalable, accurate MoE compression.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.