- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on the expert pruning in fine-grained MoE LLMs. The authors conducted extensive pruning experiments via different strategies (2/3 and dynamic), attempting to answer these questions: do we need dynamic expert pruning? Is a simple "2/3 strategy" sufficient? And in which scenarios or settings might the dynamic expert pruning methods be particularly necessary? They discover that hard generative tasks require dynamic expert pruning, and multimodal models and smaller models are more vulnerable to pruning.
Critical strengths
- The authors have conducted extensive pruning experiments on lots of benchmarks to achieve robust conclusions.
- Overall, the conclusions are sound and the writing is clear.
Critical weaknesses
- Are existing benchmarks too easy to adequately reflect the capabilities of large language models? I suggest collecting more benchmarks on challenging generation tasks to enable more demanding evaluation. For example, more realistic long-context/agentic tasks. This would help uncover deeper insights.
- In Figure 2, some results show that some benchmark scores even improve after pruning. Does this suggest that the benchmarks exhibit considerable variance and thus are not that perfect for reflecting the actual capabilities of experts? A more detailed analysis is needed here.
Questions for authors
- Is the effectiveness of pruning correlated with the model size and the difficulty of the benchmark? One hypothesis is that larger models are trained with greater redundancy and can therefore naturally tolerate more aggressive pruning. The fact that these models employ a larger number of experts is, in essence, also intended to enable them to handle tasks far more difficult than the current benchmarks, where the dynamic strategies are required.
- It would be helpful to report the router scores of the top-k experts across different task types, together with a corresponding analysis.
- An interesting question: how large is the performance gap between the proposed method and a model natively pre-trained with 2/3 activated experts?
3 comments
Response to Reviewer UzNP (2/2)
Q1: does tolerance depend on model size and task difficulty? Yes, both matter, and this is one motivation of our work. Prior expert-pruning studies mostly evaluate small MoEs on simple QA tasks, which cannot fully show how pruning affects capability. We therefore extended the model range to MoEs with hundreds of billions of parameters (Hy3 with 295B, Qwen3-235B-A22B, MiniMax-M2.7 with 229B, DeepSeek-V4-Flash with 165B), evaluated long-form reasoning (AIME, LiveCodeBench, GPQA-Diamond), and added long-context and agentic tasks after submission.
- Difficulty matters strongly: at a relative budget of 0.375, retention is about 50% on AIME, 62% on LiveCodeBench, 80% on GPQA-Diamond, 84% on MATH-500 and 89% on GSM8K.
- Size matters within a family, in line with the reviewer's hypothesis: Qwen3-235B-A22B keeps 76.7% of its score on the four hardest datasets at k=3, against 62.3% for Qwen3-30B-A3B. Across architectures, size alone does not predict tolerance (Spearman correlation -0.45 to -0.15 between generative retention and total or active parameters), because design choices also matter: at a relative budget of 0.375, models with shared experts keep 79.1% of their generative scores on average, against 60.8% for those without.
- As the reviewer anticipates, the value of dynamic allocation is concentrated on the hardest generative tasks (Section 5.3).
Q2: router scores of the top-k experts by task type. We added this analysis (new Appendix G, pp. 22-24; Table 15, p. 25). We teacher-force seven models on their own stored responses and record, at every router, the share of the native top-K weight held by the k highest-weight experts.
- At ceil(2K/3) the kept experts hold 0.77-0.89 of the top-K weight on every task family.
- Routing is most concentrated on knowledge-QA prompts and least on generated tokens. On Qwen3-30B-A3B at k=3, the kept experts hold 0.59 of the top-K weight on QA prompts but 0.52 on generated mathematics and code, and the top-8 experts hold 45% versus 29% of the full softmax. Within every task family, generated tokens route more diffusely than prompt tokens on six of the seven models (on GPT-OSS-20B the two are equally concentrated).
- Truncation therefore discards the most routed weight on exactly the tokens whose errors carry forward through a long answer, which is consistent with long-form generation being the most sensitive family.
- Concentration explains some but not all differences between models: Qwen3-235B-A22B routes more concentrated than Qwen3-30B-A3B and is more tolerant, but Qwen3-30B-A3B Thinking and Instruct have virtually identical gate mass while the thinking model is more tolerant.
Q3: a model natively trained with two thirds of the experts. A very good question. A controlled answer would need the same architecture trained at both widths, which no public checkpoint provides and which is beyond our means. Our agentic runs do contain an interesting, if loose, contrast: Qwen3.8-Flash-Next is trained with K=10 and pruned to k=6, close to two thirds, while DeepSeek-V4-Flash is trained natively with K=6. Both were evaluated with the same agent frameworks and settings (DeepSeek-V4-Flash's Pro V2 run covers 633 of the 642 tasks):
| Benchmark | Qwen3.8-Flash-Next, K=10 (unpruned) | Qwen3.8-Flash-Next, pruned to k=6 | DeepSeek-V4-Flash, K=6 (native) |
|---|---|---|---|
| SWE-bench Verified | 79.2 | 76.4 | 89.4 |
| SWE-bench Pro V2 | 92.7 | 89.7 | 91.3 |
| Terminal-Bench 2.0 | 88.4 | 82.8 | 75.3 |
Pruning Qwen3.8 to the width DeepSeek-V4-Flash was trained with costs it 2.8-5.6 points, while the differences between the two models at that width are larger and go both ways (-13.0, -1.6 and +7.5). The pruned model does not fall visibly behind a model trained natively at the smaller width; it stays in its own capability range. This is consistent with the train-inference trade-off we describe under W1. We stress that the two models differ in architecture, training data and recipe, and overall capability, so this comparison cannot isolate the effect of training at the smaller width; we report it as an observation, not a conclusion.
Dear Reviewer UzNP,
As the discussion period draws within 4 hours, we wanted to kindly follow up on our rebuttal posted a few days ago above along with the revised paper, in which we addressed your questions in detail.
We would greatly appreciate it if you could review our responses and revisit your assessment. If any concerns remain, we would be more than happy to provide further analysis and explanations.
Thank you again for your time and review service!
Response to Reviewer UzNP (1/2)
We thank the reviewer for the positive assessment and the thoughtful questions. We think the two weaknesses partly come from a point our paper did not make clear enough, namely what the benchmarks are meant to show, and we address it below. All changes in the revised paper are marked in blue.
W1 and W2: are the benchmarks too easy or too noisy, and what about long-context and agentic tasks?
1. Agentic tasks.
We fully agree that realistic agentic tasks are an important test. Shortly after submission, in early October, we had come to the same view, since agentic use has quickly become central for these models, and we extended the study accordingly. We are glad to report the results here (new Appendix L, Table 18, p. 29). We evaluated uniform truncation on seven agentic benchmarks with five fine-grained MoEs: DeepSeek-V4-Flash and Gemma 4 26B-A4B from the main study, plus Qwen3.6-35B-A3B, Qwen3.8-Flash-Next and GLM-5.3-Flash. Benchmarks: SWE-bench Verified and Pro V2, tau3-bench, Terminal-Bench 2.0, DABstep, FeatureBench and SkillsBench, each pruned run paired by task with a same-session unpruned run.
2. Long-context tasks.
We also evaluated two long-context benchmarks (new Appendix M, Tables 19-20, p. 30). On RULER (all 13 tasks, 4K to 64K tokens), Qwen3-30B-A3B at ceil(2K/3) keeps 99.1-99.6% of its unpruned score at every length. On the 16 English tasks of LongBench, ceil(2K/3) keeps 98.1% on Qwen3-30B-A3B and 99.1% on Qwen3-Next-80B, in line with the core suite. Under aggressive pruning the loss grows with input length: at k=3, Qwen3-30B-A3B keeps 95.6% of its RULER score at 4K tokens but 84.1% at 64K, and 74.8% on LongBench.
3. What the benchmarks are for in our argument.
We do not argue that these benchmarks are insufficient, or sufficient, as measures of a model's capability. Our claim is about routing: per-token expert selection in fine-grained MoEs carries redundancy. The benchmarks can support that claim because they demonstrably register damage. The same benchmarks on which scores hold at two thirds of the native selection fall sharply under aggressive pruning: at a relative budget of 0.375, the nine models keep on average about 50% of their AIME scores and 62% on LiveCodeBench, against 89% on GSM8K. A benchmark too easy to reflect capability could not separate these budgets. Scores that stay level near ceil(2K/3) are therefore evidence about the router rather than about the benchmark, and the long-context and agentic results show the same.
4. Scores that rise after pruning.
We quantified this in the new Appendix H (p. 24). Item-sampling standard errors are about 6-6.6 points on AIME, 3.4-3.5 on LiveCodeBench and GPQA-Diamond, 1.4 on MATH-500, and under 1 on GSM8K and MMLU-Pro; repeated runs of the same configuration differ by up to 6 points on GPQA-Diamond and 4 on LiveCodeBench. Of the 25 dataset cells above the unpruned score at ceil(2K/3), 5 are significant in a paired test over items; at the task-family level only mathematics and code on GPT-OSS-20B is (+2.2 +- 1.7). The rises are thus mostly noise around an unchanged score, which is what redundancy predicts. Of two larger rises at k=2 on DeepSeek-V4-Flash, the GSM8K one (75.0 to 88.7) is an answer-extraction artifact (with permissive extraction, scores decline monotonically from 93.6 unpruned); we draw no conclusion from the LiveCodeBench one, whose cause we have not identified.
5. Why fine-grained MoEs carry this redundancy: our interpretation.
We believe the root cause is a trade-off between training and inference. With hundreds of experts, a small K would route too few tokens to each expert for all of them to be trained evenly: each expert receives on average a fraction K/E of the tokens, and balancing the load gets harder as that fraction shrinks. Activating more experts per token during training keeps every expert well trained, but an individual token may not need that many experts at inference. Part of the native selection is then a cost paid for balanced training rather than a requirement of each token, and it can be trimmed at inference. Our new gate-mass measurements are consistent with this view: at ceil(2K/3) the dropped experts carry only 11-23% of the top-K gate weight. We offer this as a hypothesis; a direct test would train the same architecture at different K (see Q3).