- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
MASS prunes MoE experts by iteratively updates to kept gates and model weights.
Critical strengths
Studying the combination of weight updates and expert pruning decisions is an important research direction.
Critical weaknesses
-
The novelty over prior work is limited. Bilevel pruning has already been explored in CNNs [1,2], and differentiable pruning with budget regularization is also established [3]. The paper does not clearly demonstrate a substantial methodological advance beyond adapting these ideas to MoE expert pruning.
-
The proposed solver reduces the bilevel formulation to alternating updates by dropping the response gradient through model adaptation. The paper neither discusses the consequences of this approximation nor provides controlled experiments to justify it.
-
The gains over frequency-based pruning are marginal: only 0.41 and 0.47 percentage points in recovered average performance at 50% expert retention. These small gains raise questions about whether the additional search complexity and model updates are justified.
-
The ablation study is insufficient. A comparison with expert pruning alone, without weight updates during search, is needed to isolate the benefit of adaptation. Moreover, removing the budget loss slightly improves recovered performance, leaving its benefit unclear.
[1] Advancing Model Pruning via Bi-level Optimization. NeurIPS 2022.
[2] BilevelPruning: Unified Dynamic and Static Channel Pruning for Convolutional Neural Networks. CVPR 2024.
[3] DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models. NeurIPS 2024.
Questions for authors
See weakness
3 comments
Q3: The gains over frequency-based pruning are marginal.
A: We thank the reviewer for raising this question. We would like to clarify that expert-retention quality involves much more than the immediate preservation of the original model's capabilities. Considering the retained structure fundamentally determines how a pruned model evolves, its true utility must be assessed by how effectively it supports subsequent model adaptation. While Frequency is a strong baseline for preserving immediate performance, our experiments demonstrate that this advantage does not necessarily translate into better post-adaptation performance. Specifically:
-
MASS evaluates structural utility through model adaptation, beyond immediate preservation and the structural ranking reversals reveal the limitation of static preservation. The performance gaps highlighted by the reviewer illustrate precisely why one-shot performance alone is insufficient to assess structural quality. On Qwen3.5-35B-A3B at 50% retention, MASS starts 2.34 points below Frequency (66.51 vs. 68.85), but achieves a 6.44 point recovery gain, compared with 3.63 points for Frequency. This amounts to a 2.81 point advantage in recovery gain, more than compensating for the initial deficit and reversing the performance ranking after adaptation (72.95 vs. 72.48). A similar reversal occurs on Qwen3-30B-A3B at 75% retention, where MASS's recovery gain is 1.84 points compared with 0.54 for Frequency. These results demonstrate that structures favored by immediate preservation can be less effective after subsequent model adaptation, directly motivating our adaptation-aware structural search.
-
Representation-level analyses expose the consequences of static structural selection. At 75% retention on Qwen3-30B-A3B, Frequency achieves the highest routed-output similarity immediately after pruning, but this apparent preservation advantage deteriorates sharply during recovery: its recovered fraction of the pruning-induced representation gap is −81.7%, indicating further deviation from the original model. The layer-wise analysis reveals that this degradation occurs across all 48 MoE layers. In contrast, MASS recovers 25.9% of the representation gap and improves routed-output representations in 46/48 layers. At 50% retention, MASS similarly achieves 21.0% representation recovery, compared with 5.4% for Frequency and 12.2% for DiEP. These findings show that strong immediate representation preservation does not guarantee compatibility with subsequent model updates, whereas MASS identifies structures that consistently support representation recovery.
-
Consistent performance across model architectures and evaluation tasks. The benefits of MASS are not confined to a particular model or pruning configuration. Across two distinct MoE architectures and two retention ratios, MASS achieves the highest recovered average in three of four settings and achieves near-optimal performance in the remaining one. Furthermore, our evaluation extends beyond the 8 benchmarks in the main results to 52 task-level evaluations spanning English and multilingual knowledge, scientific reasoning, and commonsense reasoning. These results demonstrate the robustness of MASS across diverse model configurations and evaluation tasks, rather than gains tied to a specific benchmark.
We report the complete Search + Recovery cost (please refer to our response to Reviewer MsMZ05 above for details). We argue that the modest increase in offline GPU time is a worthwhile trade-off for discovering an expert structure with higher adaptation potential.
Q4: A comparison with expert pruning alone, without weight updates during search.
A: We thank the reviewer for this insightful question. First, DiEP provides a natural method-level contrast for examining adaptation. Both methods optimize differentiable structural variables, but DiEP evaluates expert importance under a fixed model state, whereas MASS continuously refines structural preferences as the model evolves. MASS consistently outperforms DiEP in recovered performance across all four model–retention settings (0.98–5.19 points), highlighting the benefit of adaptation-aware structural search.
To further examine the respective contributions of structural optimization and model adaptation within MASS, we conduct additional ablations, where MASS-Frozen-W denotes freezes model weights during search while retaining structural optimizationand MASS mask + baseline weights denotes transfers the expert set selected by MASS to baseline weights for recovery.
| Method | Pruned (%) ↑ | Recovered (%) ↑ | Routed-output cosine ↑ | RRM (%) ↓ | Top-1 affected rate (%) ↓ |
|---|---|---|---|---|---|
| MASS | 68.47 | 70.31 | 0.828 | 1.7970 | 1.3574 |
| MASS-Frozen-W | 66.89 | 68.65 | 0.827 | 1.8334 | 1.3808 |
| MASS mask + baseline weights | 64.32 | 67.88 | 0.801 | —* | —* |
The results are same as MASS.
These results reveal four key findings:
-
Structural optimization is effective on its own. MASS-Frozen-W achieves 68.65 recovered performance even with frozen weights, demonstrating the effectiveness of MASS's model-wide structural search.
-
Model adaptation further improves search outcomes. Allowing weights to evolve during search yields an additional 1.66-point recovery improvement over MASS-Frozen-W, despite nearly identical routing-preservation metrics. This highlights the benefit of adapting structural decisions to evolving model states.
-
Structural utility is model-state dependent. Transferring MASS's expert set to baseline weights reduces recovered performance to 67.88, while MASS-Frozen-W performs better under the same weight state. This demonstrates that structural quality depends on its compatibility with the corresponding model parameters.
-
Joint adaptation offers benefits beyond fixed-structure recovery. Despite substantial recovery under the transferred MASS mask, its final performance remains 2.43 points below the full MASS configuration. This shows that subsequent weight recovery alone cannot reproduce the benefit of continuously refining structural preferences during search.
Overall, these findings demonstrate that structural optimization and model adaptation serve complementary roles. By continuously refining structural preferences in response to evolving model states, MASS yields more effective model–structure configurations.
Q4: Removing the budget loss slightly improves recovered performance.
A: The budget loss is essential for aligning structural search with the target pruning budget. While we utilize Top-B discretization to guarantee the exact budget of the final pruned model, it does not constrain the soft retained capacity during search. The budget loss bridges this gap, keeping structural optimization and model adaptation aligned with the target deployment capacity.
Table 2 demonstrates three complementary benefits of the budget loss:
-
Improved budget adherence during search. Removing the budget loss nearly doubles the mean absolute budget gap, from 16.47 to 33.23, demonstrating substantially greater capacity drift throughout structural optimization.
-
Higher quality of the final discrete structure. Without the budget loss, pruned performance drops by 1.32 points (71.73 → 70.41), RRM increases from 1.89% to 3.98%, and Top-1 affected rate rises from 1.43% to 4.00%. Routed-output cosine also decreases from 0.835 to 0.803, indicating greater disruption to the original model's routing behavior and representations.
-
Complementary roles of capacity alignment and routing-aware allocation. The ablations reveal that the two objectives address distinct aspects of structural search.
- With retained, removing substantially increases the budget gap from 16.47 to 33.23, indicating that routing-aware allocation alone cannot ensure capacity alignment.
- Conversely, without , incorporating the budget loss reduces the budget gap from 9.81 to 1.39 yet fails to avoid severe structural degradation. The pruned performance drops from 67.60 to 58.21.
This demonstrates that capacity alignment and routing-aware allocation are complementary: the former controls how much capacity is retained, while the latter guides which experts receive that capacity.
Thus, the marginally higher recovered performance without the budget loss does not diminish its contribution. The ablations establish its importance in maintaining budget consistency and improving the quality of the resulting expert structure, which cannot be adequately captured by recovered performance alone.
We thank the reviewer for pointing out these related lines of work. While prior studies have established important techniques for bilevel pruning and differentiable pruning with budget regularization, our MASS offers a distinct perspective on MoE expert pruning by explicitly accounting for how expert retention decisions affect model adaptation. Its methodological contributions differ from prior work in three aspects:
A: The first-order alternating formulation provides a practical and computationally tractable solution for large-scale MoE models. Instead of explicitly differentiating through the inner model adaptation to compute the response gradient, MASS incorporates adaptation-induced information through iterative feedback: model parameters are updated under the current candidate expert structure, and structural variables are subsequently optimized based on the updated model state. This allows model adaptation to continuously inform expert retention decisions, while avoiding the substantial memory and computational overhead of differentiating through the entire adaptation trajectory.
Our controlled adaptation ablations demonstrate the effectiveness of this optimization design. Detailed results of the adaptation ablation can be found in our response to Reviewer 1RiY above. With model weights frozen during search, MASS-Frozen-W achieves 66.89% pruning performance and 68.65% recovered performance, compared with 68.47% and 70.31%, respectively, for MASS. Allowing model adaptation therefore improves both the quality of the selected expert structure (+1.58 points) and final recovery (+1.66 points). Furthermore, applying the MASS-selected mask to the original model weights yields only 64.32% pruning performance and 67.88% recovered performance, highlighting the importance of the compatibility between selected expert structures and adapted model parameters. Together, these results demonstrate that MASS effectively leverages structure-conditioned model adaptation to refine expert retention decisions and improve the resulting model.