acceptodds
Under review as a conference paper at ICLR 2027

Static Preservation Is Not Enough: Adaptation-Aware Structural Search for MoE Pruning

Abstract

Mixture-of-Experts (MoE) language models activate only a few experts per token but retain the full expert bank in memory during inference. Expert pruning reduces this deployment overhead by removing experts, but existing methods commonly evaluate retention decisions using layer-local proxies or a fixed model state. Such estimates can be unreliable because retention decisions interact across layers, while structures that preserve the current model well may not remain effective after subsequent adaptation. We therefore introduce MASS, a Model-Wide Adaptation-Aware Structural Search method, in which retention decisions remain revisable as the model evolves. Specifically, we evaluate candidate structures jointly across all MoE layers using full-model next-token prediction feedback. Under the target expert budget, a routing-aware objective guides expert allocation according to token-level native routing demand without favoring capacity expansion. We couple structural search with model adaptation throughout optimization, where current retention preferences define the structure used for model updates and feedback from the adapted model guides their subsequent refinement. Experiments on two distinct MoE architectures under two pruning severities show that MASS achieves the best overall recovered performance. Further analyses show that MASS exhibits favorable representation recovery while preserving native routing demand and limiting post-pruning representation distortion.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts (MoE), Model Compression, Efficient Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_rXQAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

MASS prunes MoE experts by iteratively updates to kept gates and model weights.

Critical strengths

Studying the combination of weight updates and expert pruning decisions is an important research direction.

Critical weaknesses
  1. The novelty over prior work is limited. Bilevel pruning has already been explored in CNNs [1,2], and differentiable pruning with budget regularization is also established [3]. The paper does not clearly demonstrate a substantial methodological advance beyond adapting these ideas to MoE expert pruning.

  2. The proposed solver reduces the bilevel formulation to alternating updates by dropping the response gradient through model adaptation. The paper neither discusses the consequences of this approximation nor provides controlled experiments to justify it.

  3. The gains over frequency-based pruning are marginal: only 0.41 and 0.47 percentage points in recovered average performance at 50% expert retention. These small gains raise questions about whether the additional search complexity and model updates are justified.

  4. The ablation study is insufficient. A comparison with expert pruning alone, without weight updates during search, is needed to isolate the benefit of adaptation. Moreover, removing the budget loss slightly improves recovered performance, leaving its benefit unclear.

[1] Advancing Model Pruning via Bi-level Optimization. NeurIPS 2022.

[2] BilevelPruning: Unified Dynamic and Static Channel Pruning for Convolutional Neural Networks. CVPR 2024.

[3] DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models. NeurIPS 2024.

Questions for authors

See weakness

3 comments

Q1: The novelty over prior work.

We thank the reviewer for pointing out these related lines of work. While prior studies have established important techniques for bilevel pruning and differentiable pruning with budget regularization, our MASS offers a distinct perspective on MoE expert pruning by explicitly accounting for how expert retention decisions affect model adaptation. Its methodological contributions differ from prior work in three aspects:

  • A distinct problem formulation: from static expert importance to adaptation-aware expert retention. Unlike vanilla parameters or Transformer channels, MoE experts are conditionally activated by the router for individual tokens. Their practical utility is therefore dependent on the model state and can change as the model continues to adapt under a pruned structure. As observed in our experiments, under the same pruning setting, Frequency achieves higher immediate pruning performance but poorer recovered performance. This illustrates that static importance does not reliably predict post-adaptation utility. MASS mitigates this limitation by evaluating expert retention decisions through their utility during model adaptation, instead of relying exclusively on static importance estimates.
  • A different optimization scope: from structural unit selection to model-wide expert configuration optimization. MASS treats expert retention as a model-wide structural optimization problem, where expert-level decisions are evaluated through their collective consequences for the entire MoE model. Retention decisions across experts and layers jointly determine the conditional computation structure, and thus influence routing choices, parameter updates, and the adaptation of the remaining experts. Accordingly, MASS evaluates candidate expert configurations based on the adaptation behavior of the model as a whole, rather than treating expert importance as a collection of independently assessed scores. This model-level perspective is implemented through structure-conditioned model updates, with the resulting model states providing feedback for subsequent structural refinement.
  • MoE-specific structural modeling: from generic structural masks to routing-aware expert retention. Generic differentiable masks and budget regularization do not explicitly capture the token-level routing dependencies induced by an MoE router. MASS incorporates Projected Native Top-K Coverage to account for the original model's top-k routing patterns when evaluating candidate expert structures. This connects expert retention directly to native token-to-expert assignments, making routing behavior an integral part of structural search rather than treating experts as interchangeable parameter blocks. Table 2 further demonstrates the contribution of this routing-aware mechanism.

Q2: The proposed solver reduces the bilevel formulation to alternating updates by dropping the response gradient through model adaptation.

A: The first-order alternating formulation provides a practical and computationally tractable solution for large-scale MoE models. Instead of explicitly differentiating through the inner model adaptation to compute the response gradient, MASS incorporates adaptation-induced information through iterative feedback: model parameters are updated under the current candidate expert structure, and structural variables are subsequently optimized based on the updated model state. This allows model adaptation to continuously inform expert retention decisions, while avoiding the substantial memory and computational overhead of differentiating through the entire adaptation trajectory.

Our controlled adaptation ablations demonstrate the effectiveness of this optimization design. Detailed results of the adaptation ablation can be found in our response to Reviewer 1RiY above. With model weights frozen during search, MASS-Frozen-W achieves 66.89% pruning performance and 68.65% recovered performance, compared with 68.47% and 70.31%, respectively, for MASS. Allowing model adaptation therefore improves both the quality of the selected expert structure (+1.58 points) and final recovery (+1.66 points). Furthermore, applying the MASS-selected mask to the original model weights yields only 64.32% pruning performance and 67.88% recovered performance, highlighting the importance of the compatibility between selected expert structures and adapted model parameters. Together, these results demonstrate that MASS effectively leverages structure-conditioned model adaptation to refine expert retention decisions and improve the resulting model.

Q3: The gains over frequency-based pruning are marginal.

A: We thank the reviewer for raising this question. We would like to clarify that expert-retention quality involves much more than the immediate preservation of the original model's capabilities. Considering the retained structure fundamentally determines how a pruned model evolves, its true utility must be assessed by how effectively it supports subsequent model adaptation. While Frequency is a strong baseline for preserving immediate performance, our experiments demonstrate that this advantage does not necessarily translate into better post-adaptation performance. Specifically:

  • MASS evaluates structural utility through model adaptation, beyond immediate preservation and the structural ranking reversals reveal the limitation of static preservation. The performance gaps highlighted by the reviewer illustrate precisely why one-shot performance alone is insufficient to assess structural quality. On Qwen3.5-35B-A3B at 50% retention, MASS starts 2.34 points below Frequency (66.51 vs. 68.85), but achieves a 6.44 point recovery gain, compared with 3.63 points for Frequency. This amounts to a 2.81 point advantage in recovery gain, more than compensating for the initial deficit and reversing the performance ranking after adaptation (72.95 vs. 72.48). A similar reversal occurs on Qwen3-30B-A3B at 75% retention, where MASS's recovery gain is 1.84 points compared with 0.54 for Frequency. These results demonstrate that structures favored by immediate preservation can be less effective after subsequent model adaptation, directly motivating our adaptation-aware structural search.

  • Representation-level analyses expose the consequences of static structural selection. At 75% retention on Qwen3-30B-A3B, Frequency achieves the highest routed-output similarity immediately after pruning, but this apparent preservation advantage deteriorates sharply during recovery: its recovered fraction of the pruning-induced representation gap is −81.7%, indicating further deviation from the original model. The layer-wise analysis reveals that this degradation occurs across all 48 MoE layers. In contrast, MASS recovers 25.9% of the representation gap and improves routed-output representations in 46/48 layers. At 50% retention, MASS similarly achieves 21.0% representation recovery, compared with 5.4% for Frequency and 12.2% for DiEP. These findings show that strong immediate representation preservation does not guarantee compatibility with subsequent model updates, whereas MASS identifies structures that consistently support representation recovery.

  • Consistent performance across model architectures and evaluation tasks. The benefits of MASS are not confined to a particular model or pruning configuration. Across two distinct MoE architectures and two retention ratios, MASS achieves the highest recovered average in three of four settings and achieves near-optimal performance in the remaining one. Furthermore, our evaluation extends beyond the 8 benchmarks in the main results to 52 task-level evaluations spanning English and multilingual knowledge, scientific reasoning, and commonsense reasoning. These results demonstrate the robustness of MASS across diverse model configurations and evaluation tasks, rather than gains tied to a specific benchmark.

We report the complete Search + Recovery cost (please refer to our response to Reviewer MsMZ05 above for details). We argue that the modest increase in offline GPU time is a worthwhile trade-off for discovering an expert structure with higher adaptation potential.

Q4: A comparison with expert pruning alone, without weight updates during search.

A: We thank the reviewer for this insightful question. First, DiEP provides a natural method-level contrast for examining adaptation. Both methods optimize differentiable structural variables, but DiEP evaluates expert importance under a fixed model state, whereas MASS continuously refines structural preferences as the model evolves. MASS consistently outperforms DiEP in recovered performance across all four model–retention settings (0.98–5.19 points), highlighting the benefit of adaptation-aware structural search.

To further examine the respective contributions of structural optimization and model adaptation within MASS, we conduct additional ablations, where MASS-Frozen-W denotes freezes model weights during search while retaining structural optimizationand MASS mask + baseline weights denotes transfers the expert set selected by MASS to baseline weights for recovery.

MethodPruned (%) ↑Recovered (%) ↑Routed-output cosine ↑RRM (%) ↓Top-1 affected rate (%) ↓
MASS68.4770.310.8281.79701.3574
MASS-Frozen-W66.8968.650.8271.83341.3808
MASS mask + baseline weights64.3267.880.801—*—*

The results are same as MASS.

These results reveal four key findings:

  • Structural optimization is effective on its own. MASS-Frozen-W achieves 68.65 recovered performance even with frozen weights, demonstrating the effectiveness of MASS's model-wide structural search.

  • Model adaptation further improves search outcomes. Allowing weights to evolve during search yields an additional 1.66-point recovery improvement over MASS-Frozen-W, despite nearly identical routing-preservation metrics. This highlights the benefit of adapting structural decisions to evolving model states.

  • Structural utility is model-state dependent. Transferring MASS's expert set to baseline weights reduces recovered performance to 67.88, while MASS-Frozen-W performs better under the same weight state. This demonstrates that structural quality depends on its compatibility with the corresponding model parameters.

  • Joint adaptation offers benefits beyond fixed-structure recovery. Despite substantial recovery under the transferred MASS mask, its final performance remains 2.43 points below the full MASS configuration. This shows that subsequent weight recovery alone cannot reproduce the benefit of continuously refining structural preferences during search.

Overall, these findings demonstrate that structural optimization and model adaptation serve complementary roles. By continuously refining structural preferences in response to evolving model states, MASS yields more effective model–structure configurations.

Q4: Removing the budget loss slightly improves recovered performance.

A: The budget loss is essential for aligning structural search with the target pruning budget. While we utilize Top-B discretization to guarantee the exact budget of the final pruned model, it does not constrain the soft retained capacity during search. The budget loss bridges this gap, keeping structural optimization and model adaptation aligned with the target deployment capacity.

Table 2 demonstrates three complementary benefits of the budget loss:

  • Improved budget adherence during search. Removing the budget loss nearly doubles the mean absolute budget gap, from 16.47 to 33.23, demonstrating substantially greater capacity drift throughout structural optimization.

  • Higher quality of the final discrete structure. Without the budget loss, pruned performance drops by 1.32 points (71.73 → 70.41), RRM increases from 1.89% to 3.98%, and Top-1 affected rate rises from 1.43% to 4.00%. Routed-output cosine also decreases from 0.835 to 0.803, indicating greater disruption to the original model's routing behavior and representations.

  • Complementary roles of capacity alignment and routing-aware allocation. The ablations reveal that the two objectives address distinct aspects of structural search.

    • With Lpcov\mathcal{L}_{\mathrm{pcov}} retained, removing Lbudget\mathcal{L}_{\mathrm{budget}} substantially increases the budget gap from 16.47 to 33.23, indicating that routing-aware allocation alone cannot ensure capacity alignment.
    • Conversely, without Lpcov\mathcal{L}_{\mathrm{pcov}}, incorporating the budget loss reduces the budget gap from 9.81 to 1.39 yet fails to avoid severe structural degradation. The pruned performance drops from 67.60 to 58.21.

This demonstrates that capacity alignment and routing-aware allocation are complementary: the former controls how much capacity is retained, while the latter guides which experts receive that capacity.

Thus, the marginally higher recovered performance without the budget loss does not diminish its contribution. The ablations establish its importance in maintaining budget consistency and improving the quality of the resulting expert structure, which cannot be adequately captured by recovered performance alone.

Reviewer_UNffOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MASS, a method for MoE expert pruning that combines model-wide structural search with model adaptation. It uses full-model next-token prediction and a routing-aware objective to select experts under a target budget, while allowing retention decisions to change during training. The authors report improved performance after recovery training and better representation recovery on two MoE architectures.

Critical strengths
  1. The paper addresses a relevant problem in MoE compression and provides a clear motivation for considering model adaptation during expert selection.

  2. The empirical evaluation is reasonably comprehensive, covering two MoE models, two retention ratios, and several pruning baselines, with supporting ablations and detailed task-level results.

Critical weaknesses
  1. The authors introduce expert-specific bias terms to the router logits in Eq. (3). The same parameters also control sigmoid gates that scale expert outputs in Eq. (4). These parameters are optimized using several loss functions and determine which experts are retained. However, the gains over the Frequency baseline are limited. In some cases, Frequency achieves higher accuracy immediately after pruning but lower accuracy after recovery training.

The following table reports averages over the subtasks listed within each dataset in Appendix F.

Dataset75%: Frequency75%: MASS50%: Frequency50%: MASS
MMLU77.63 / 76.4175.37 / 76.9767.77 / 70.0659.63 / 68.99
MMLU-LE59.41 / 59.9258.03 / 60.6952.31 / 56.3051.54 / 56.16
ArabicMMLU36.48 / 36.5434.30 / 38.1325.96 / 28.0726.22 / 29.85
IndoMMLU44.38 / 43.9842.49 / 44.3029.09 / 31.8827.66 / 32.07
JapaneseMMLU35.00 / 35.3433.34 / 39.6727.67 / 32.0027.00 / 31.00
ARC-Challenge50.53 / 52.7348.11 / 53.7828.20 / 36.0528.65 / 35.58
HellaSwag63.59 / 64.7162.66 / 65.0941.65 / 50.2041.30 / 49.66
  1. Introducing trainable structural parameters and auxiliary losses is a reasonable approach, but the methodological novelty appears limited.
Questions for authors
  1. Could the authors explain why Frequency achieves higher accuracy immediately after pruning but lower accuracy after recovery training? In Table 1, on Qwen3.5-35B-A3B, Frequency starts at 68.85%, higher than MASS’s 66.51%, but reaches 72.48% after recovery, below MASS’s 72.95%, despite using the same recovery training procedure.
2 comments

We thank the reviewer for recognizing the importance of our research problem, the motivation for considering model adaptation in expert pruning, and the breadth of our empirical evaluation. We appreciate the opportunity to further clarify the performance differences between MASS and Frequency, as well as the contributions of our approach.

Q1 & W1: Why Frequency achieves higher accuracy immediately after pruning but lower accuracy after recovery training? And the limited gains over the Frequency baseline.

We thank the reviewer for raising this question. The observed performance reversal highlights precisely the core motivation of MASS: preserving the immediate behavior of a frozen model does not guarantee that the retained structure will effectively support subsequent model adaptation.

  • Immediate pruning performance measures static preservation, whereas our goal is to identify structures that remain effective throughout model adaptation. Frequency selects experts based on historical routing statistics, which reflects the pretrained model's current usage patterns and naturally favors preserving its immediate behavior. However, expert importance estimated under a fixed model state does not necessarily reflect the utility of the retained structure after model parameters evolve. A structure that best preserves the current model is not necessarily the one that best supports its subsequent adaptation. This distinction is central to our motivation for moving beyond static expert importance estimation.

  • The performance reversal demonstrates that static preservation is insufficient to predict post-adaptation effectiveness. On Qwen3.5-35B-A3B at 50% retention, Frequency improves from 68.85% to 72.48% (+3.63 points), whereas MASS improves from 66.51% to 72.95% (+6.44 points). Despite initially trailing Frequency by 2.34 points, MASS gains an additional 2.81 points during recovery and ultimately surpasses it. A similar difference emerges at 75% retention, where MASS gains 3.44 points compared with only 0.18 for Frequency. Across the four model–retention settings, MASS achieves the best recovered performance in three and remains within 0.32 points of Frequency in the remaining setting. These results highlight that immediate preservation and post-adaptation utility are distinct criteria for evaluating expert retention.

  • Our representation recovery analysis further explains why expert retention requires dynamic evaluation. As shown in Figure 2, Frequency achieves the highest routed-output similarity immediately after pruning. However, it recovers only 5.4% of the representation gap at 50% retention and suffers a negative recovery of -81.7% at 75% retention. MASS, on the other hand, consistently recovers 21.0% and 25.9% in these respective settings. Furthermore, our layer-wise analysis in Appendix D shows that this recovery advantage is broadly distributed across the model depth and persists across different search checkpoints. This provides concrete evidence that strong static preservation does not automatically translate into better compatibility with future model updates.

Ultimately, the goal of MASS is not to maximize immediate post-pruning accuracy. The objective is to identify expert structures whose utility is continually reassessed as model parameters evolve. The performance reversal against Frequency illustrates exactly why static expert importance is insufficient. Meanwhile, our performance gains and representation analyses confirm the practical value of our adaptation-aware structural search.

W2: Introducing trainable structural parameters and auxiliary losses is a reasonable approach, but the methodological novelty appears limited.

We thank the reviewer for acknowledging the reasonableness of our structural parameters and auxiliary losses. We would like to clarify that our primary contribution lies in formulating expert pruning as model-wide structural search under evolving model states.

  • MASS moves beyond fixed-state expert importance estimation to evaluate retention decisions throughout model adaptation. Existing pruning methods primarily estimate expert importance from routing statistics, activation information, or reconstruction objectives under a fixed model state. However, expert retention decisions are coupled across layers: upstream pruning affects downstream representations and routing, while the utility of a retained structure changes as model parameters adapt. MASS addresses this by jointly optimizing retention decisions across all MoE layers and keeping them revisable as the model evolves, rather than treating expert importance as a fixed property of the pretrained model.

  • MASS makes structural decisions part of model computation and evaluates their joint predictive effects. The same structural parameters influence both router-logit shifts for expert selection (Eq. 3) and gates for expert-output aggregation (Eq. 4). Their effects are jointly evaluated through the full-model next-token prediction objective, going beyond local importance estimation or layer-wise reconstruction. Meanwhile, budget alignment and Projected Native Top-K Coverage guide the search toward the target retention capacity and allocate it according to the model's native token-level routing demand.

  • Alternating optimization enables structural decisions and model parameters to co-adapt throughout the search. The current candidate structure determines how model parameters are updated, while the resulting model state provides new predictive and routing feedback for subsequent structural refinement. This distinguishes MASS from differentiable pruning methods such as DiEP, which also employs trainable structural variables but estimates expert importance under a fixed model state. Empirically, MASS achieves higher recovered performance than DiEP across all four model–retention settings, with improvements of 0.98–5.19 percentage points. This comparison demonstrates the practical benefit of evaluating and revising structural decisions under evolving model states.

In conclusion, MASS provides a new perspective for MoE pruning by introducing an adaptation-aware structural search framework. This framework effectively overcomes the inherent limitations of fixed-state methods. It establishes that expert retention should be treated as a dynamic process that co-evolves with model adaptation. The consistent empirical gains fully validate the practical value and methodological novelty of this approach.

Reviewer_1RiYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a Model-Wide Adaptation-Aware Structural Search method, in which retention decisions remain revisable as the model evolves. Specifically, the authors evaluate candidate structures jointly across all MoE layers using full-model next-token prediction feedback. Under the target expert budget, a routing-aware objective guides expert allocation according to token-level native routing demand without favoring capacity expansion. They couple structural search with model adaptation throughout optimization, where current retention preferences define the structure used for model updates and feedback from the adapted model guides their subsequent refinement.

Critical strengths
  • MASS is a clever technique that fills a gap in MoE research (search that accounts for adaptation rather than a fixed model state).
  • Projected Native Top-K Coverage has an effect and Table 2 is good empirical evidence.
Critical weaknesses
  • The manuscript should report seeds or significance tests, and should make explicit what +/- measures. Also, the paper defines an Overall-52 average but as far as I can see never reports it.
  • The appendix A1 matches the number of weight updates, but not the conditioning. The baseline spends the 300 steps on the full model, while MASS spends them under its candidate structure. MASS is getting up to twice as much structure-conditioned training, so the P and recovery gain aren’t directly comparable with the baselines’. R also doesn’t separate selection from co-adapted weights.
  • There is no adaptation ablation. A proper ablation could more robustly defend the contributions of the manuscript.
  • What happens when you train with MASS but freeze the weight-updates? Does adapting during search improve selection?
  • What happens when you apply MASS’s expert set to the baseline weights, then recover? Does the selection help by itself?
  • What happens when you fix a baseline’s mask during its first 300 steps, then recover? Does structure-conditioned training explain any gain? Minor
  • More information is needed regarding why you chose the methods you actually benchmark against in section 2 or 4.
  • Figures 1, 3, and 4 text can and should be much larger
Questions for authors

Please see above.

2 comments

We sincerely thank the reviewer for the constructive feedback and valuable suggestions. We are encouraged that the reviewer recognizes MASS as a clever technique filling a critical gap in MoE research and acknowledges our strong empirical evidence demonstrating the effect of Projected Native Top-K Coverage. To address your questions, we provide our detailed clarifications and explanations as follows.

Q1: The manuscript should report seeds or significance tests, and should make explicit what +/- measures.

A: We thank the reviewer for the constructive suggestion. Our experiments were conducted using a random seed of 42. The ± values represent the standard evaluation error derived from the lm-evaluation-harness. Furthermore, the Overall-52 task macro metric will be included in the appendix.

Q2: The appendix A1 matches the number of weight updates, but not the conditioning.

A: We thank the reviewer for this insightful question. We would like to clarify that MASS investigates a fundamentally different pruning paradigm: evaluating and refining expert retention decisions through model adaptation, rather than estimating expert importance under a fixed model state. Our comparison is designed to evaluate the resulting end-to-end pruning pipelines under a matched total model-update budget.

Pre-pruning adaptation serves to evaluate structural utility rather than merely providing additional training. Both MASS and the baselines receive 300 pre-pruning steps followed by 300 recovery steps. It is worth noting that the conventional training updates model parameters without evaluating candidate structures, whereas post-pruning recovery adapts weights only after the retained structure has been finalized. Neither captures how candidate structures influence model adaptation and how adaptation changes their utility. In contrast, during search of MASS, candidate structures condition model updates, and the resulting model states provide feedback for subsequent structural refinement. These updates constitute the evaluation mechanism itself, rather than extra training under a fixed mask. Thus, MASS reallocates the same model-update budget to make structural decisions responsive to model adaptation, rather than increasing the total number of model updates.

The complementary question of how structural selection and co-adapted weights contribute to recovered performance is examined separately through targeted adaptation ablations.

Q3-Q5: Adaptation ablation: Does adapting during search improve selection and Does the selection help by itself?

A: We thank the reviewer for this insightful question. First, DiEP provides a natural method-level contrast for examining adaptation. Both methods optimize differentiable structural variables, but DiEP evaluates expert importance under a fixed model state, whereas MASS continuously refines structural preferences as the model evolves. MASS consistently outperforms DiEP in recovered performance across all four model–retention settings (0.98–5.19 points), highlighting the benefit of adaptation-aware structural search.

To further examine the respective contributions of structural optimization and model adaptation within MASS, we conduct additional ablations: (i) MASS, the full method jointly optimizing structural parameters and model weights; (ii) MASS-Frozen-W, which freezes model weights during search while retaining structural optimization; and (iii) MASS mask + baseline weights, which transfers the expert set selected by MASS to baseline weights for recovery.

MethodPruned (%) ↑Recovered (%) ↑Routed-output cosine ↑RRM (%) ↓Top-1 affected rate (%) ↓
MASS68.4770.310.8281.79701.3574
MASS-Frozen-W66.8968.650.8271.83341.3808
MASS mask + baseline weights64.3267.880.801—*—*

The RRM and Top-1 affected rate are reported as the same as MASS.

These results reveal four key findings:

  • Structural optimization is effective on its own. MASS-Frozen-W achieves 68.65 recovered performance even with frozen weights, demonstrating the effectiveness of MASS's model-wide structural search.

  • Model adaptation further improves search outcomes. Allowing weights to evolve during search yields an additional 1.66-point recovery improvement over MASS-Frozen-W, despite nearly identical routing-preservation metrics. This highlights the benefit of adapting structural decisions to evolving model states.

  • Structural utility is model-state dependent. Transferring MASS's expert set to baseline weights reduces recovered performance to 67.88, while MASS-Frozen-W performs better under the same weight state. This demonstrates that structural quality depends on its compatibility with the corresponding model parameters.

  • Joint adaptation offers benefits beyond fixed-structure recovery. Despite substantial recovery under the transferred MASS mask, its final performance remains 2.43 points below the full MASS configuration. This shows that subsequent weight recovery alone cannot reproduce the benefit of continuously refining structural preferences during search.

Overall, these findings demonstrate that structural optimization and model adaptation serve complementary roles. By continuously refining structural preferences in response to evolving model states, MASS yields more effective model–structure configurations.

Q6: What happens when you fix a baseline’s mask during its first 300 steps, then recover?

A: Fixed-mask adaptation serves a fundamentally different objective from MASS. Fixing a mask for the initial 300 steps amounts to continued training that merely adapts parameters to a pre-selected architecture and leaves the structural decisions completely static. In contrast, our MASS aims to discover the optimal structure through its dynamic interaction with the evolving model state, by utilizing model adaptation to continuously evaluate and refine the structure.

Q7-Q8: More information is needed regarding why you chose the methods and Figures 1, 3, and 4 text can and should be much larger.

A: We thank the reviewer for the constructive suggestion. We select representative expert-pruning methods covering complementary structural selection paradigms:

  • Random provides an uninformed reference;
  • Frequency represents routing-statistics-based selection;
  • REAP incorporates router-weighted activation magnitude for activation-aware expert importance estimation;
  • DiEP represents differentiable structural optimization of intra- and inter-layer expert importance.

Together, these methods span heuristic, routing-based, activation-aware, and optimization-based pruning approaches, providing diverse reference points for evaluating MASS.

We appreciate your suggestions to refine our work, and we will add these revisions to the final version of the manuscript and adjust the font size of the text in our figures.

Reviewer_MsMZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the issue of expert pruning in MoE models and highlights a key blind spot in existing methods: static importance evaluations based on a ‘fixed model state’ or ‘layer-local’ perspective are unable to accurately predict the true utility of the pruned structure during subsequent adaptation phases. To address this, the authors propose MASS (Model-Wide Adaptation-Aware Structural Search), a framework that combines structural search with model parameter adaptation. MASS evaluates the joint utility of cross-layer structures via a model-wide Next-Token Prediction (NTP) loss, and designs Capacity Alignment and Projected Native Top-K Coverage losses to ensure that continuous structural search remains faithful to the target budget whilst conforming to the model’s native routing requirements.

Critical strengths
  1. The motivation is clear: the paper is seeking a method to find a retained set of experts that is not only good after one-shot expert pruning but also shows good performance after post-compression recovery training.

  2. The experiments are somewhat sufficient, which includes both one-shot pruning result comparison and results after recovery retraining. This paper also includes detailed analysis experiments and ablation studies.

Critical weaknesses
  1. The first critical weakness of this work is the performance is not strong enough, no matter in the one-shot pruning and recovery retraining settings. For example, on Qwen3-30A3B model, MASS obtains 66.4 in the one-shot setting while the frequency-based method is 65.08. The conclusion is similar in retraining setting. Worse still, the one-shot performance of MASS (66.51) is worse than the frequency-based method(68.85) and REAP(68.9). After retraining, the improvement gap is still minor, with less than 0.5 points gains.

  2. Moreover, this paper utilizes 2.5B for retraining. I think this scale is hard to show the true adaptation-aware property. At this training scale, I think the one-shot initialization takes majority of roles for post-compression training.

  3. Although Table 5 shows that MASS’s search throughput is comparable to that of DiEP, this masks a significant hidden cost: DiEP optimises structural variables in a single pass under fixed weights, whereas MASS’s 300-step search actually involves 300 steps of forward and backward propagation across the full model’s weights (equivalent to 300 steps of continued pre-training). For MoE models of the 30B/35B scale, the GPU time and energy consumption associated with these 300 steps are extremely substantial. The paper does not explicitly report the total number of GPU hours for the entire Search + Recovery phase of MASS. If achieving a 1–2 percent improvement in recovery performance requires computational costs several times higher than the direct pruning + same number of steps for recovery baseline, its cost-effectiveness in practical industrial deployment will be significantly diminished.

  4. Some notations are not clearly written. For example, what does EE in eq. 7 mean? It means the expert number or the expert parameter? What does $ \epsilon $ in eq. 9 and 10 mean? It is not explained by the authors.

minor:

rr in eq.3 should be bold.

Questions for authors

See the weakness

4 comments

We sincerely thank the reviewer for the constructive feedback and recognizing the importance of identifying expert structures that perform well after recovery retraining. We also appreciate the positive feedback on the sufficiency of our experimental design, particularly our comprehensive before-and-after recovery comparisons and detailed analyses. We carefully address the specific concerns regarding model performance and computational cost below.

Q: Clarification on the one-shot performance comparisons and the significance of the recovery margins.

A: We thank the reviewer for raising this question. We would like to clarify that expert-retention quality involves more than immediate preservation of the original model's capabilities. Since the retained structure determines how the pruned model evolves during subsequent optimization, its utility must also be assessed by how effectively it supports model adaptation. Our results provide three complementary lines of evidence:

  • MASS evaluates structural utility through model adaptation, beyond immediate preservation and the structural ranking reversals reveal the limitation of static preservation. The performance gaps highlighted by the reviewer illustrate precisely why one-shot performance alone is insufficient to assess structural quality. On Qwen3.5-35B-A3B at 50% retention, MASS starts 2.34 points below Frequency (66.51 vs. 68.85), but achieves a 6.44 point recovery gain, compared with 3.63 points for Frequency. This amounts to a 2.81 point advantage in recovery gain, more than compensating for the initial deficit and reversing the performance ranking after adaptation (72.95 vs. 72.48). A similar reversal occurs on Qwen3-30B-A3B at 75% retention, where MASS's recovery gain is 1.84 points compared with 0.54 for Frequency. These results demonstrate that structures favored by immediate preservation can be less effective after subsequent model adaptation, directly motivating our adaptation-aware structural search.

  • Consistent performance across model architectures and evaluation tasks. The benefits of MASS are not confined to a particular model or pruning configuration. Across two distinct MoE architectures and two retention ratios, MASS achieves the highest recovered average in three of four settings and remains within 0.32 points of the best in the remaining one. Furthermore, our evaluation extends beyond the eight benchmarks in the main results to 52 task-level evaluations spanning English and multilingual knowledge, scientific reasoning, and commonsense reasoning. These results demonstrate the robustness of MASS across diverse model configurations and evaluation tasks, rather than gains tied to a specific benchmark.

  • Controlled comparisons isolate the benefit of structure-conditioned adaptation. To distinguish the effect of our search strategy from that of additional model updates, we match the model-update budget between MASS and DiEP. While both optimize differentiable structural variables, DiEP evaluates expert importance under fixed model weights, whereas MASS continuously refines retention decisions as model parameters adapt. Under this controlled protocol, MASS achieves higher recovered performance in all four model–retention settings, with improvements of 0.98–5.19 points. Notably, at 75% retention on Qwen3.5-35B-A3B, the two methods have nearly identical one-shot performance, yet MASS achieves a larger recovery gain (3.44 vs. 2.53 points). This demonstrates that the benefit stems from evaluating structural utility under evolving model states, rather than simply allocating more parameter updates.

  • Representation-level analyses expose the consequences of static structural selection. At 75% retention on Qwen3-30B-A3B, Frequency achieves the highest routed-output similarity immediately after pruning, but this apparent preservation advantage deteriorates sharply during recovery: its recovered fraction of the pruning-induced representation gap is −81.7%, indicating further deviation from the original model. The layer-wise analysis reveals that this degradation occurs across all 48 MoE layers. In contrast, MASS recovers 25.9% of the representation gap and improves routed-output representations in 46/48 layers. At 50% retention, MASS similarly achieves 21.0% representation recovery, compared with 5.4% for Frequency and 12.2% for DiEP. These findings show that strong immediate representation preservation does not guarantee compatibility with subsequent model updates, whereas MASS identifies structures that consistently support representation recovery.

These results establish that structural quality cannot be fully characterized by one-shot performance: the ability of a retained structure to support subsequent model adaptation is equally important. By explicitly optimizing this dimension, MASS achieves stronger post-adaptation utility and reveals a critical limitation of static expert-retention strategies.

Q2: The one-shot initialization may take majority of roles for post-compression training.

A: Regarding the 2.5B token recovery budget, this stage is not intended as full scale retraining to restore the original model. It serves specifically as a controlled adaptation horizon. This setup allows us to examine how effectively different retained structures support subsequent model updates. All methods undergo the exact same recovery schedule. This isolates the adaptation behavior associated with different pruning structures.

  • The loss trajectories during recovery demonstrate substantial model adaptation beyond one-shot initialization. We provide the recovery loss trajectories in Figure 8 to illustrate the adaptation process. MASS exhibits a rapid loss reduction, progressively closing the initial gap and reaching a comparable or slightly lower loss by the end of recovery. Its loss trajectory also becomes increasingly stable in the later stages, indicating diminishing optimization gains within the prescribed budget. This demonstrates that the recovery stage produces substantial changes to the model state, rather than merely preserving the one-shot initialization.

  • Performance reversals contradict the initialization dominant assumption. As detailed in our previous response, we observe clear performance ranking reversals: structures with lower immediate pruning performance can achieve better recovered performance after the same adaptation budget. This observation is inconsistent with the hypothesis that one-shot initialization predominantly determines the final outcome.

The representation-level analysis provides further structural evidence for this distinction. Although Frequency achieves the strongest immediate representation preservation, its representation recovery deteriorates during subsequent updates, resulting in a negative recovered fraction of the pruning-induced representation gap at 75% retention and degradation across all MoE layers. In contrast, MASS consistently recovers representations across model depth. These findings demonstrate that different retained structures support substantially different adaptation behaviors, and that their post-adaptation utility cannot be reliably inferred from their initial pruning quality.

Q3: The GPU hours for the entire Search + Recovery phase of MASS.

A: We report the GPU-hour consumption of all methods across their respective experimental stages to provide a complete view of computational requirements.

MethodMatched CPTSearch / Joint SearchRecoveryTotal (GPU-hours)
Random95.09—94.07189.16
REAP95.09—95.12190.21
Frequency95.09—97.78192.87
DiEP95.0998.0695.82288.97
MASS (Ours)Included in Joint183.1796.51279.68
  • Adaptation-aware structural search entails deliberate computational investment. The central objective of MASS is to identify expert-retention structures that remain effective as model parameters adapt, rather than evaluating structural quality solely under static model weights. This requires evaluating candidate structures through model updates under their respective structural configurations, allowing retention decisions to incorporate feedback from evolving model states. Accordingly, MASS jointly performs structural optimization and model-parameter updates during search, consuming 183.17 GPU-hours, followed by 96.51 GPU-hours for post-pruning recovery. The complete Search + Recovery pipeline therefore requires 279.68 GPU-hours. This investment directly enables the adaptation-aware structural evaluation that distinguishes MASS from static expert-selection strategies.

  • The computational investment translates into measurable structural and performance benefits. MASS achieves higher recovered performance than DiEP across all four model–retention settings, with improvements ranging from 0.98 to 5.19 points, while maintaining a comparable end-to-end computational cost (279.68 vs. 288.97 GPU-hours). This demonstrates that incorporating model adaptation into structural search improves the effectiveness of expert selection without increasing the overall computational budget relative to an alternative learnable structural-search method. As discussed in our response to Q1, these benefits are further reflected in improved adaptation behavior and stronger representation recovery. Moreover, structural search is performed offline during model compression, and the resulting pruned model can be deployed without additional structural-search overhead during inference.

Q4: Some notations are not clearly written.

A: We appreciate the reviewer for pointing out the missing definitions. In Eq. 7, EE denotes the number of experts. In Eq. 9 and 10, ϵ\epsilon is a small perturbation term set to 10−610^{-6}. We will clarify these notations in the revised manuscript.

Reviewer_MsMZOpenReview ↗

Response to Authors

Thank you for your detailed rebuttal and clarification. For Weakness 1, I accept the authors' view that we should focus more on the performance on the model after retraining instead of the one-shot model. However, that does not mean we should only care the recovery gap of different methods, especially with the more than 40% extra training cost. Therefore, I don't think it makes sense to achieve only 0.41 points (Freq vs. this MASS) higher accuracy with more GPU hours. Moreover, the downstream benchmark performance also includes randomness. It is unclear if the benefits come from the method or the randomness with such a minor improvement. Other concerns are addressed. So, I would like to respectfully maintain my original score.

We sincerely appreciate the reviewer's reply. However, we respectfully disagree with the view that MASS offers only marginal improvements.

First, we argue that different expert-retention strategies involve clear trade-offs across tasks, and no single pruning method performs best across all tasks. In our experiments, MASS shows stronger performance particularly on knowledge-related tasks, including factual knowledge, subject-specific understanding, and non-English knowledge tasks. This is consistent with the educational and multilingual nature of the data used during the search and adaptation process, highlighting the importance of model adaptation in identifying experts that remain useful as the model evolves.

Importantly, MASS maintains its advantages over Frequency even under the aggressive 50% expert retention setting, with gains of 10.37 pp on Arabic MMLU Anatomy, 8.27 pp on MMLU-LE Moral Scenarios, and 5.81 pp on MMLU-LE High-School Biology. Such task-specific improvements are valuable for deployment scenarios with different task requirements and should not be overlooked when considering only the overall average.

Expert RetentionModelBenchmark & TaskFrequencyMASSGain (pp)
50% (Extreme)Qwen3-30BArabic MMLU: Anatomy24.4434.81+10.37
Qwen3-30BMMLU-LE: Moral Scenarios41.3449.61+8.27
Qwen3.5-35BMMLU-LE: High-School Biology63.5569.36+5.81
Qwen3-30BArabic MMLU: Electrical Engineering26.2131.72+5.51
Qwen3.5-35BJapanese MMLU: Clinical Knowledge42.6748.00+5.33
Qwen3-30BMMLU: Professional Accounting48.2351.77+3.54
75% (Moderate)Qwen3.5-35BMMLU: Global Facts45.0053.00+8.00
Qwen3.5-35BMMLU: Business Ethics80.0086.00+6.00
Qwen3.5-35BArabic MMLU: Anatomy33.3339.26+5.93
Qwen3-30BMMLU: World Religions84.2190.06+5.85

More importantly, MASS offers a different perspective on expert importance in MoE pruning. It moves beyond evaluating expert importance based on a single, fixed model state and instead evaluates the utility of retained structures as model parameters and expert structures evolve together. This perspective highlights a limitation of existing static pruning methods and provides a potential approach for future MoE pruning research.

Thank you for your time and efforts. We hope our response addresses your concerns.

Comments on the paper

1 comment

We sincerely thank the reviewers for their constructive feedback. We are highly encouraged that the reviewers consistent recognize the significance and rigor of our work. Before addressing specific questions, we would like to highlight the core strengths acknowledged by the reviewers:

  • Clear Motivation & Novelty (Reviewer MsMZ, 1RiY, UNff): The paper addresses a highly relevant problem in MoE compression. Reviewers acknowledge MASS as a "clever technique" that fills a critical research gap by evaluating expert utility based on post-compression adaptation and retraining, rather than a static model state.
  • Methodological Soundness (Reviewer 1RiY): The proposed mechanism, Projected Native Top-K Coverage, is recognized as well-motivated and effective, supported by strong empirical evidence.
  • Comprehensive Evaluation (Reviewer MsMZ, UNff): The experimental setup is deemed sufficient and thorough. The evaluation rigorously covers multiple MoE architectures, varying pruning ratios, and strong baselines, backed by one-shot vs. retraining comparisons, detailed task-level analyses, and extensive ablation studies.

Below, we summarize our key responses addressing concerns regarding performance gains, adaptation ablations, and search cost:

  • Performance Improvements & Core Contribution:

    • Expert retention inherently involves trade-offs between tasks, and no single pruning method outperforms all benchmarks. Aligning with our educational and multilingual adaptation corpora, MASS achieves significant improvements in factual knowledge, subject-specific understanding, and non-English QA. Even at an aggressive 50% retention ratio, MASS outperforms Frequency by +10.37 pp on Arabic MMLU Anatomy, +8.27 pp on MMLU-LE Moral Scenarios and so on. Such task-specific improvements are valuable for deployment scenarios with different task requirements and should not be overlooked when considering only the overall average.
    • More importantly, our representation-level analyses and the observation of structural ranking reversals expose the limits of static pruning, demonstrating that expert utility should be evaluated by how effectively a retained structure supports subsequent adaptation.
  • Adaptation Ablations: To disentangle the roles of structural optimization and model adaptation, we conducted additional ablations establishing four key findings: (1) Structural optimization is effective on its own; (2) Model adaptation further improves search outcomes; (3) Structural utility is model-state dependent; and (4) Joint adaptation offers benefits beyond fixed-structure recovery.

  • Search Cost & Trade-offs: Compared to alternative learnable structural-search baselines (DiEP), incorporating adaptation improves expert selection without increasing the overall GPU-hours. Compared to static heuristics, the modest offline compute overhead represents a worthwhile one-time trade-off to identify expert structures with higher adaptation potential and stronger task-specific utility.

We will incorporate the above analyses into the final version of the manuscript. We hope these responses address the remaining concerns and clarify our contribution to highlighting the role of model adaptation in MoE pruning.

We sincerely thank all the reviewers for their time and suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.