- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Importance estimation guided component selection is essential for improving the efficiency of adaptive sparse models. However, there is an important but easily overlooked point is that the component's contribution to the model can differ from the model's dependence on its availability. This paper is therefore motivated to formalize the distinction between component ablated after routing (suppression) and that of before the routing (exclusion), which helps to establish the stability of the resulting attribution across different removal implementations. Specifically, this paper provide empirical evidence across mixtures of experts (MoE), dynamic token selection, and fitted sparse semantic fields to answer 3 questions: (1) Which conclusions about component function remain reliable across these responses? (2) When do changes in those conclusions affect component choices (3) How can uncertain choices be checked efficiently?
Critical strengths
- Different from existing works focusing on examining substitution, renormalization, or compensation for a specific intervention, this work focus on the resulting attribution stability across the suppression and exclusion interventions, which is meaningful to guide model expert pruning.
- The introduction and methodology sections are well written and easy to follow.
Critical weaknesses
- For the abstract part (line 18-24), it is hard to understand the motivation behind the empirical investigations as claimed "We derive a sufficient condition,...,that also account for the effect gaps of the remaining errors". The abstract needs some modifications to be reader friendly, which should concisely state the questions that the introduction section aims to answer.
- The experiment didn't consider the interactions between different components. For example, if we have two components A and B, they can serve as good substitutes for each another, however, the model is highly dependent on the presence of at least one of them. In this case, exclusion testing on only one component concluded that neither A nor B were the important components, so they were both removed during model pruning and results in collapsing system. I know the table 4 includes joint exclusion results but it take different components individually and still doesn't consider the component interaction.
Questions for authors
- The tables and figures in the experiment section should generally be placed at the top or the bottom.
- In table 1, will the conclusions still hold under larger-scale MoE model?
- In figure1 a, the spearman correlation for representation change is close to 1 and that of the signed loss effect is much lower, could you provide more analysis of the reason?
Thank you for the constructive and very helpful suggestions. Following them, we added experiments on the larger Qwen3-30B-A3B model and on pairs of experts that can substitute for each other, revised the paper and the figure layout, and added figures to make it easier to follow. We address each of your points below.
W1. We rewrote the abstract and added diagrams of the two interventions (Figures 1 and 2) and two MMLU examples (Figure 4) in which removing the same expert keeps the correct answer under one procedure but makes it wrong under the other, so the expert looks unnecessary in one case and important in the other.
W2. Component interactions and substitute pairs. Conclusion: Interactions between two removed experts are common at early layers, especially when one expert serves as a replacement for the other. Removing two individually minor experts rarely produces a large joint effect. Individual scores can therefore form candidate pruning sets, and the complete set should then be evaluated together.
We evaluated all pairs of 16 frequently routed experts in OLMoE and Qwen3 (Section 4.1, Figure 6, Appendix C.5). Under exclusion, pairs in which one expert replaces the other during individual removal interact more often than other pairs at every depth where interactions occur (Figure 6b), consistent with the replacement mechanism you describe. Two experts with small individual effects produce a large joint effect in fewer than 1% of evaluated pair removals (Appendix C.5). This is consistent with our pruning results, where candidate sets built from individual removal effects perform well under joint exclusion (Section 5.3, Table 1).
Q1. Figure and table placement. All figures and tables in the main text (Figures 1 to 8 and Table 1) are now placed at the top or bottom of the page.
Q2. Scaling to a larger MoE model. Conclusion: We added Qwen3-30B-A3B (48 layers, 128 experts per layer), and our main findings hold at this scale.
Within the rebuttal period, we evaluated five evenly spaced layers (0, 12, 24, 36, and 47) that span the full depth of the model. Suppression and exclusion give opposite effect signs in 24.5% of removals with a measurable effect and identify different top experts in 62.0% of groups (Figure 3, Table C.2). Later routing and pair interactions show the same qualitative patterns as in OLMoE (Figures 5 and 6, Appendix C.4). We also pruned the model at those five layers and evaluated it on 500 questions (Section 5.3, Table 1, Appendix G). Both criteria based on measured removal effects preserve the loss of the unpruned model more closely than REAP at both pruning budgets. We also report that routing frequency gives the lowest loss deviation when 16 experts are pruned per layer.
Q3. Why representation change correlations are near 1 while signed loss correlations are much lower. Conclusion: Representation change measures only the size of the output change, whereas the loss also depends on its direction.