- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes OptEP, a layer-wise expert-pruning method that learns continuous expert-selection variables using local output reconstruction, followed by expert weight updates (EWU).
Critical strengths
The emperical performance of the paper is good and the evaluation covers several large MoE models like Qwen3-235B-A22B.
Critical weaknesses
-
Expert selection is performed independently for each layer using inputs from the full, unpruned model. Consequently, pruning decisions in earlier layers do not affect the inputs used to select experts in later layers. This ignores the accumulation of pruning errors across layers and creates a mismatch between the selection stage and the final pruned model.
-
The method uses a simple continuous relaxation of binary selection variables, optimized with SGD and clamped to ([0,1]). During optimization, it neither enforces the target number of retained experts nor uses the actual sparse Top-(K) routing. The final Top-(s) discretization may therefore differ substantially from the optimized continuous model. More advanced methods, such as Gumbel-Sigmoid with STE, should be included for comparison.
-
The novelty over previous work is unclear. Differentiable subset selection guided by the outputs of the original model has already been explored. The paper should better distinguish OptEP from existing differentiable pruning [1] with knowledge-distillation methods [2] and explain which technical component constitutes the main contribution.
-
EWU provides a large portion of the reported improvements, but it is only applied to experts selected by OptEP. Other baselines are evaluated without the same weight-update procedure. Applying EWU to REAP, EASY-EP, and other selection baselines is necessary to determine whether the gains come from the proposed expert-selection method or from the additional weight optimization.
[1] Differentiable Subset Pruning of Transformer Heads. TACL 2021.
[2] ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning. TMLR 2026.
Questions for authors
See weakness
5 comments
Response to Reviewer dNCT (2/4): Validity of the Continuous Relaxation
Q2. We thank the reviewer for raising this important question. To investigate the alignment between our continuous relaxation and the exact discrete pruning objective, we conduct four complementary analyses on Mixtral.
a. Optimality Gap of the Continuous Relaxation (added as Appendix D.2). We compare the expert subsets selected by our continuous relaxation with the globally optimal subsets obtained through exhaustive search. Both are evaluated using the same exact reconstruction loss with binary expert selection and Top-K routing. As shown in Table R2 below, our method achieves reconstruction losses only 0.71–3.24% higher than the global optimum. Although it identifies the exact optimal subset in 28–43% of layer–seed pairs, the selected subsets have a mean rank of only 3.3–6.8 among 28 or 70 possible subsets. These results demonstrate that our relaxation consistently identifies near-optimal expert subsets under the exact discrete reconstruction objective.
Table R2. Comparison of expert subsets selected by exhaustive search and continuous optimization followed by discretization on Mixtral. Both are evaluated using the same exact reconstruction objective.
| Dataset | s | Gap | Overlap | Agreement | Rank |
|---|---|---|---|---|---|
| GSM8K | 4 | 0.71% | 81.1% | 68/160 | 6.1/70 |
| GSM8K | 6 | 2.58% | 89.0% | 56/160 | 3.9/28 |
| MBPP | 4 | 2.86% | 78.6% | 44/160 | 6.8/70 |
| MBPP | 6 | 3.24% | 89.5% | 63/160 | 3.3/28 |
b. Effect of Top-K Routing During Continuous Optimization (added as Appendix D.3). We evaluate a variant that retains Top-K routing in the continuous optimization objective. This substantially degrades downstream performance: GSM8K accuracy drops from 12.89% to 2.23% with four retained experts and from 25.32% to 9.40% with six retained experts. We hypothesize that Top-K routing restricts gradient updates to the selected experts for each token, potentially reinforcing early selections and limiting exploration of alternative subsets. In contrast, our proposed relaxation allows all experts to receive gradient updates, leading to substantially better expert selection. Results are in Table R3 below.
c. Effect of Enforcing Binary Expert Selection During Optimization (added as Appendix D.4). We additionally evaluate a variant that applies hard thresholding after every optimization step, enforcing binary selection variables and retaining exactly the prescribed number of experts. This variant consistently underperforms our proposed relaxation, which maintains continuous selection scores and enforces the expert budget only after optimization. On GSM8K, accuracy drops from 12.89% to 9.17% with four retained experts and from 25.32% to 22.05% with six retained experts. We observe similar declines on MBPP. These results demonstrate the benefit of maintaining continuous selection scores throughout optimization and postponing discrete expert selection until the final step. Results are in Table R3 below.
d. Gumbel-STE Expert Selection. We additionally evaluate replacing our dense relaxation with Gumbel-Sigmoid selection trained via a straight-through estimator (STE), which keeps the forward pass identical to pruned inference by applying a hard Top-s expert mask and Top-K routing. We test 1, 8, and 16 noise samples per step; the 16-sample variant additionally anneals the noise scale and initializes the selection from our dense-relaxation subsets. All variants substantially degrade downstream performance. GSM8K accuracy drops from 12.89% to 3.85%, 4.26%, and 9.78% with four retained experts, and from 25.32% to 13.57%, 10.37%, and 10.87% with six retained experts. Results are in Table R3 below.
Table R3: Accuracy of Mixtral-8x7B after expert pruning with our approach (no top_k), hard top-2 routing during optimization (top_k = 2), and Gumbel-STE selection with 1, 8, or 16 noise samples per step. Values are mean ± standard error over five seeds: flexible-match accuracy (%) on GSM8K and standard pass@1 (%) on MBPP. Best result per row in bold.
| Benchmark | s | Ours (no top_k) | top_k = 2 | binary | Gumbel, 1 sample | Gumbel, 8 samples | Gumbel, 16 samples + anneal + init |
|---|---|---|---|---|---|---|---|
| GSM8K | 4 | 12.89 ± 0.00 | 2.23 ± 0.10 | 9.17 ± 0.88 | 3.85 ± 0.84 | 4.26 ± 1.43 | 9.78 ± 1.60 |
| GSM8K | 6 | 25.32 ± 0.37 | 9.40 ± 1.86 | 22.05 ± 2.26 | 13.57 ± 3.21 | 10.37 ± 3.31 | 10.87 ± 4.34 |
| MBPP | 4 | 30.31 ± 0.61 | 21.08 ± 0.59 | 26.66 ± 0.61 | 17.14 ± 4.39 | 7.97 ± 4.76 | 22.04 ± 4.71 |
| MBPP | 6 | 47.61 ± 0.59 | 40.35 ± 2.13 | 42.66 ± 0.82 | 38.10 ± 1.14 | 39.25 ± 1.35 | 32.31 ± 6.96 |
Together, these experiments demonstrate that our dense continuous relaxation identifies near-optimal expert subsets under the exact discrete reconstruction objective, while explicitly incorporating Top-K routing, an expert budget constraint with binary ’s, or Gumbel-STE selection during optimization substantially degrades downstream performance.
Response to Reviewer dNCT (3/4): Main Contributions and Distinctions from Prior Differentiable Pruning Methods
Q3. We thank the reviewer for prompting us to clarify our contributions in relation to the cited works and broader literature. We have updated the Related Work section of the revised paper (Section 2) accordingly, and we elaborate below.
Main contribution and closest prior work. OptEP introduces the scalable approach to directly optimizing joint expert selection through MoE-layer output reconstruction. NAEE [3] is the closest prior work: it considers a similar reconstruction objective but exhaustively enumerates all expert subsets, which becomes intractable for modern MoEs with 128 experts per layer. Our normalized continuous surrogate (Eqs. 4–5) instead enables gradient-based joint optimization of expert-selection variables, capturing redundancy and complementarity between experts rather than relying on fixed importance scores. Importantly, when exhaustive search is feasible (Mixtral), our relaxation achieves comparable downstream performance to brute-force optimization of the corresponding discrete objectives (Appendix Table 11), while scaling to substantially larger expert pools.
Comparison with DSP [1]. DSP selects attention heads by minimizing supervised downstream task loss using a Gumbel-top- relaxation. In contrast, OptEP jointly selects whole experts by directly minimizing original MoE-layer reconstruction error. DSP's relaxation requires sequential softmax operations, introduces stochasticity through Gumbel sampling, and relies on temperature annealing to approach discrete selection. Our deterministic relaxation instead requires a single normalization of the router-weighted expert mixture, avoiding sequential sampling and annealing, followed by top- selection after optimization. Thus, OptEP differs in both its reconstruction-based objective and its substantially simpler, more computationally efficient optimization procedure.
Comparison with ToMoE [2]. ToMoE addresses a fundamentally different problem: converting a dense model into an MoE by learning channel-selection masks that define new experts, jointly with a new router. These are optimized using KL distillation on the model's final output distribution, with regularization to reduce active computation per token. In contrast, OptEP compresses an existing pretrained MoE by jointly selecting a fixed subset of its original experts, preserving the pretrained router parameters and directly minimizing layerwise reconstruction error. Thus, although both optimize selection variables using gradients, ToMoE neither addresses reconstruction-based selection of existing experts nor provides our scalable optimization procedure for this problem.
Our second contribution is a lightweight, reusable expert-weight update (EWU). Our regularized reconstruction-based update can be applied after any expert-pruning method, adapting retained experts to compensate for those removed. Unlike the end-to-end task fine-tuning used by SEER-MoE [4], our method performs lightweight layerwise optimization using existing calibration data, without backpropagation through the full model. This provides a computationally inexpensive recovery mechanism that is independent of our expert-selection algorithm.
Both contributions are independently supported by strong empirical results. Without weight updates, at least one of our weighted and unweighted variants outperforms the strongest prior pure-pruning baseline in 21/22 settings, with absolute accuracy gains of up to 12.33% and 10.32% on Qwen3-30B-A3B and Qwen3-235B-A22B, respectively. Furthermore, across both OptEP variants, the best performance after expert-weight updates (EWU) exceeds the best performance before EWU in 20/22 settings, with additional absolute gains of up to 16.38% on Mixtral-8x7B.
References:
-
Differentiable Subset Pruning of Transformer Heads. TACL 2021.
-
ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning. TMLR 2026.
-
Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models. ACL 2024.
-
SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts. 2024.
Response to Reviewer dNCT (1/4): Impact of Sequential vs Parallel Expert Pruning on Accuracy and Efficiency
We thank the reviewer for their constructive feedback and for recognizing the strong empirical performance of OptEP across large-scale MoE architectures, including Qwen3-235B-A22B. Below, we address the reviewer's concerns regarding layer-wise pruning and error accumulation, the continuous relaxation and its alignment with the discrete pruning objective, the novelty of our approach relative to prior work, and the independent contributions of expert selection and EWU.
Q1. Sequential vs. Parallel Expert Pruning. We thank the reviewer for the insightful question. As suggested, we further investigate whether sequential expert pruning provides an advantage over our proposed parallel pruning approach. Specifically, we compare sequential OptEP against parallel OptEP without expert weight updates (EWU) on GLM-4.5-Air.
As shown in Table R1 below, sequential and parallel pruning achieve comparable performance across all six settings on MATH-500 and LiveCodeBench. The accuracy differences are small (within 1% in five out of six settings) and are generally small relative to the reported standard errors. These results suggest that sequential pruning does not provide a consistent performance advantage over parallel pruning.
More importantly, sequential pruning requires a separate pruning procedure for each target number of retained experts, whereas our parallel approach performs pruning only once and obtains solutions for all pruning levels. Therefore, parallel pruning offers greater computational efficiency and flexibility while maintaining comparable downstream accuracy.
Table R1. Sequential vs parallel pruning results. Results are reported as mean accuracy (%) ± standard error over three calibration seeds. Δ denotes sequential minus parallel accuracy in percentage points (pp).