acceptodds
Under review as a conference paper at ICLR 2027

You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs

14 views

Abstract

Fine-grained mixture-of-experts (MoE) architectures have become a mainstream design for open-weight LLMs, with hundreds of experts and increasingly many selected per token. This shift makes dynamic expert pruning an attractive route to cheaper inference. Yet existing evidence comes largely from coarser architectures and likelihood-scored multiple-choice benchmarks, leaving three central questions open in the fine-grained regime: how redundant per-token expert selection is, how effectively existing pruning methods exploit that redundancy, and what governs a model's sensitivity to pruning. We fill this gap with a systematic empirical study of twelve fine-grained MoE checkpoints spanning nine architecture families, with a core suite of eleven benchmarks covering knowledge QA, mathematics, code generation, and general reasoning. We find that expert selection is far more redundant than the field's operating points assume: uniformly retaining about two thirds of the selected experts preserves 98.8% of unpruned performance on average, requiring only a one-integer change and delivering 1.2-1.7x measured speedup across two serving backends. This simple baseline leaves little room for dynamic allocation at conservative budgets: even the best published rules differ from it by under 1% at matched expert budgets. Their value emerges under aggressive pruning, where the best rules recover up to 3.0% over uniform truncation, with gains concentrated in the generative tasks that suffer the sharpest degradation. Sensitivity to aggressive pruning also depends on the model: larger and thinking models are more resilient, whereas multimodal models are more vulnerable. Together, these findings reveal how much expert computation fine-grained MoEs can dispense with, and establish when dynamic allocation earns its complexity, informing both practical deployment and future pruning methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
fine-grained moe, expert-pruning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UzNPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the expert pruning in fine-grained MoE LLMs. The authors conducted extensive pruning experiments via different strategies (2/3 and dynamic), attempting to answer these questions: do we need dynamic expert pruning? Is a simple "2/3 strategy" sufficient? And in which scenarios or settings might the dynamic expert pruning methods be particularly necessary? They discover that hard generative tasks require dynamic expert pruning, and multimodal models and smaller models are more vulnerable to pruning.

Critical strengths
  1. The authors have conducted extensive pruning experiments on lots of benchmarks to achieve robust conclusions.
  2. Overall, the conclusions are sound and the writing is clear.
Critical weaknesses
  1. Are existing benchmarks too easy to adequately reflect the capabilities of large language models? I suggest collecting more benchmarks on challenging generation tasks to enable more demanding evaluation. For example, more realistic long-context/agentic tasks. This would help uncover deeper insights.
  2. In Figure 2, some results show that some benchmark scores even improve after pruning. Does this suggest that the benchmarks exhibit considerable variance and thus are not that perfect for reflecting the actual capabilities of experts? A more detailed analysis is needed here.
Questions for authors
  1. Is the effectiveness of pruning correlated with the model size and the difficulty of the benchmark? One hypothesis is that larger models are trained with greater redundancy and can therefore naturally tolerate more aggressive pruning. The fact that these models employ a larger number of experts is, in essence, also intended to enable them to handle tasks far more difficult than the current benchmarks, where the dynamic strategies are required.
  2. It would be helpful to report the router scores of the top-k experts across different task types, together with a corresponding analysis.
  3. An interesting question: how large is the performance gap between the proposed method and a model natively pre-trained with 2/3 activated experts?
3 comments
AuthorsOpenReview ↗

Response to Reviewer UzNP (1/2)

We thank the reviewer for the positive assessment and the thoughtful questions. We think the two weaknesses partly come from a point our paper did not make clear enough, namely what the benchmarks are meant to show, and we address it below. All changes in the revised paper are marked in blue.

W1 and W2: are the benchmarks too easy or too noisy, and what about long-context and agentic tasks?

1. Agentic tasks.

We fully agree that realistic agentic tasks are an important test. Shortly after submission, in early October, we had come to the same view, since agentic use has quickly become central for these models, and we extended the study accordingly. We are glad to report the results here (new Appendix L, Table 18, p. 29). We evaluated uniform truncation on seven agentic benchmarks with five fine-grained MoEs: DeepSeek-V4-Flash and Gemma 4 26B-A4B from the main study, plus Qwen3.6-35B-A3B, Qwen3.8-Flash-Next and GLM-5.3-Flash. Benchmarks: SWE-bench Verified and Pro V2, tau3-bench, Terminal-Bench 2.0, DABstep, FeatureBench and SkillsBench, each pruned run paired by task with a same-session unpruned run.

  • Near ceil(2K/3) the agentic benchmarks keep 96.0% of the unpruned score on average over 21 model-benchmark pairs (median 96.8%), about two points below the core suite, as multi-step tasks compound small per-step losses.
  • Two pairs lose more than 10%: SkillsBench on Qwen3.6 (83.3%) and DABstep on Qwen3.8 (80.6% at k=6, 90.3% at k=8). Qwen3.6 and Gemma 4 keep their DABstep scores at k=6, so this sensitivity depends on the model as well as the task.

2. Long-context tasks.

We also evaluated two long-context benchmarks (new Appendix M, Tables 19-20, p. 30). On RULER (all 13 tasks, 4K to 64K tokens), Qwen3-30B-A3B at ceil(2K/3) keeps 99.1-99.6% of its unpruned score at every length. On the 16 English tasks of LongBench, ceil(2K/3) keeps 98.1% on Qwen3-30B-A3B and 99.1% on Qwen3-Next-80B, in line with the core suite. Under aggressive pruning the loss grows with input length: at k=3, Qwen3-30B-A3B keeps 95.6% of its RULER score at 4K tokens but 84.1% at 64K, and 74.8% on LongBench.

3. What the benchmarks are for in our argument.

We do not argue that these benchmarks are insufficient, or sufficient, as measures of a model's capability. Our claim is about routing: per-token expert selection in fine-grained MoEs carries redundancy. The benchmarks can support that claim because they demonstrably register damage. The same benchmarks on which scores hold at two thirds of the native selection fall sharply under aggressive pruning: at a relative budget of 0.375, the nine models keep on average about 50% of their AIME scores and 62% on LiveCodeBench, against 89% on GSM8K. A benchmark too easy to reflect capability could not separate these budgets. Scores that stay level near ceil(2K/3) are therefore evidence about the router rather than about the benchmark, and the long-context and agentic results show the same.

4. Scores that rise after pruning.

We quantified this in the new Appendix H (p. 24). Item-sampling standard errors are about 6-6.6 points on AIME, 3.4-3.5 on LiveCodeBench and GPQA-Diamond, 1.4 on MATH-500, and under 1 on GSM8K and MMLU-Pro; repeated runs of the same configuration differ by up to 6 points on GPQA-Diamond and 4 on LiveCodeBench. Of the 25 dataset cells above the unpruned score at ceil(2K/3), 5 are significant in a paired test over items; at the task-family level only mathematics and code on GPT-OSS-20B is (+2.2 +- 1.7). The rises are thus mostly noise around an unchanged score, which is what redundancy predicts. Of two larger rises at k=2 on DeepSeek-V4-Flash, the GSM8K one (75.0 to 88.7) is an answer-extraction artifact (with permissive extraction, scores decline monotonically from 93.6 unpruned); we draw no conclusion from the LiveCodeBench one, whose cause we have not identified.

5. Why fine-grained MoEs carry this redundancy: our interpretation.

We believe the root cause is a trade-off between training and inference. With hundreds of experts, a small K would route too few tokens to each expert for all of them to be trained evenly: each expert receives on average a fraction K/E of the tokens, and balancing the load gets harder as that fraction shrinks. Activating more experts per token during training keeps every expert well trained, but an individual token may not need that many experts at inference. Part of the native selection is then a cost paid for balanced training rather than a requirement of each token, and it can be trimmed at inference. Our new gate-mass measurements are consistent with this view: at ceil(2K/3) the dropped experts carry only 11-23% of the top-K gate weight. We offer this as a hypothesis; a direct test would train the same architecture at different K (see Q3).

AuthorsOpenReview ↗

Response to Reviewer UzNP (2/2)

Q1: does tolerance depend on model size and task difficulty? Yes, both matter, and this is one motivation of our work. Prior expert-pruning studies mostly evaluate small MoEs on simple QA tasks, which cannot fully show how pruning affects capability. We therefore extended the model range to MoEs with hundreds of billions of parameters (Hy3 with 295B, Qwen3-235B-A22B, MiniMax-M2.7 with 229B, DeepSeek-V4-Flash with 165B), evaluated long-form reasoning (AIME, LiveCodeBench, GPQA-Diamond), and added long-context and agentic tasks after submission.

  • Difficulty matters strongly: at a relative budget of 0.375, retention is about 50% on AIME, 62% on LiveCodeBench, 80% on GPQA-Diamond, 84% on MATH-500 and 89% on GSM8K.
  • Size matters within a family, in line with the reviewer's hypothesis: Qwen3-235B-A22B keeps 76.7% of its score on the four hardest datasets at k=3, against 62.3% for Qwen3-30B-A3B. Across architectures, size alone does not predict tolerance (Spearman correlation -0.45 to -0.15 between generative retention and total or active parameters), because design choices also matter: at a relative budget of 0.375, models with shared experts keep 79.1% of their generative scores on average, against 60.8% for those without.
  • As the reviewer anticipates, the value of dynamic allocation is concentrated on the hardest generative tasks (Section 5.3).

Q2: router scores of the top-k experts by task type. We added this analysis (new Appendix G, pp. 22-24; Table 15, p. 25). We teacher-force seven models on their own stored responses and record, at every router, the share of the native top-K weight held by the k highest-weight experts.

  • At ceil(2K/3) the kept experts hold 0.77-0.89 of the top-K weight on every task family.
  • Routing is most concentrated on knowledge-QA prompts and least on generated tokens. On Qwen3-30B-A3B at k=3, the kept experts hold 0.59 of the top-K weight on QA prompts but 0.52 on generated mathematics and code, and the top-8 experts hold 45% versus 29% of the full softmax. Within every task family, generated tokens route more diffusely than prompt tokens on six of the seven models (on GPT-OSS-20B the two are equally concentrated).
  • Truncation therefore discards the most routed weight on exactly the tokens whose errors carry forward through a long answer, which is consistent with long-form generation being the most sensitive family.
  • Concentration explains some but not all differences between models: Qwen3-235B-A22B routes more concentrated than Qwen3-30B-A3B and is more tolerant, but Qwen3-30B-A3B Thinking and Instruct have virtually identical gate mass while the thinking model is more tolerant.

Q3: a model natively trained with two thirds of the experts. A very good question. A controlled answer would need the same architecture trained at both widths, which no public checkpoint provides and which is beyond our means. Our agentic runs do contain an interesting, if loose, contrast: Qwen3.8-Flash-Next is trained with K=10 and pruned to k=6, close to two thirds, while DeepSeek-V4-Flash is trained natively with K=6. Both were evaluated with the same agent frameworks and settings (DeepSeek-V4-Flash's Pro V2 run covers 633 of the 642 tasks):

BenchmarkQwen3.8-Flash-Next, K=10 (unpruned)Qwen3.8-Flash-Next, pruned to k=6DeepSeek-V4-Flash, K=6 (native)
SWE-bench Verified79.276.489.4
SWE-bench Pro V292.789.791.3
Terminal-Bench 2.088.482.875.3

Pruning Qwen3.8 to the width DeepSeek-V4-Flash was trained with costs it 2.8-5.6 points, while the differences between the two models at that width are larger and go both ways (-13.0, -1.6 and +7.5). The pruned model does not fall visibly behind a model trained natively at the smaller width; it stays in its own capability range. This is consistent with the train-inference trade-off we describe under W1. We stress that the two models differ in architecture, training data and recipe, and overall capability, so this comparison cannot isolate the effect of training at the smaller width; we report it as an observation, not a conclusion.

Dear Reviewer UzNP,

As the discussion period draws within 4 hours, we wanted to kindly follow up on our rebuttal posted a few days ago above along with the revised paper, in which we addressed your questions in detail.

We would greatly appreciate it if you could review our responses and revisit your assessment. If any concerns remain, we would be more than happy to provide further analysis and explanations.

Thank you again for your time and review service!

Reviewer_okTbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies inference-time expert skipping in fine-grained MoE LLMs. It evaluates uniform top-k truncation across nine architecture families, compares four dynamic retention rules with fixed-k at approximately matched average expert budgets on four core models, and studies sensitivity using Qwen3 checkpoint pairs that differ in reasoning mode, scale, or modality. The paper reports that retaining roughly two thirds of the selected experts preserves 98.8% of unpruned performance on average, that dynamic allocation adds little at conservative budgets but helps under aggressive pruning, and that robustness varies across models and tasks. The evaluation includes both likelihood-scored QA and long-form generative tasks, with throughput measurements for fixed-k execution.

Critical strengths
  1. This is a broad and carefully assembled benchmark of expert skipping in modern fine-grained MoEs. The uniform sweep covers nine architecture families, the dynamic comparison includes four core models, and the evaluation goes beyond likelihood-scored QA to generative mathematics, code, and reasoning. The breadth and the explicit FIXED-K control make the paper useful as a measurement reference.

  2. The experimental reporting is unusually complete. The paper gives the actual average expert counts, handles shared experts separately, explains how published top-2 rules were adapted to wider routers, and includes detailed per-task results and fixed-k throughput. This is strong work for a benchmark or measurement paper, even if it does not establish a substantially new principle.

Critical weaknesses
  1. Part of the reported gain from dynamic routing comes from oracle-style rule selection, and the system budgets are not exactly matched. Table 3 picks the best rule for each model using the same eleven datasets on which the gain is then reported. Several winning settings also use more experts than the corresponding fixed-k baseline, even if the difference stays within the allowed 10% tolerance. Since latency or throughput is not measured for the dynamic rules, I cannot tell how much of the gain comes from better allocation and how much comes from post-hoc selection or extra computation.

  2. The paper is mainly a broad benchmark, and I found most of its qualitative conclusions fairly incremental. Prior work already showed that some expert computation can be skipped with limited degradation (NAEE, ACL 2024), that harder inputs can benefit from more dynamically allocated experts (DYNROUTE, ACL 2024), that a fixed active-expert count creates token- and layer-dependent redundancy (Ban&Pick), and that redundancy varies across layers (DiEP, NeurIPS 2025). The main added value here is the broader coverage and the FIXED-K control, rather than a new understanding of expert redundancy. The two-thirds result is also based on an unweighted mean without dispersion or worst-case results; Section 6 shows checkpoint-specific associations rather than causal effects; and the dynamic comparison uses adapted variants. Overall, this is a solid and useful empirical study, but I do not think the current methodological or conceptual contribution is strong enough for ICLR.

Questions for authors
  1. Using the existing result matrix, how much of the aggressive-budget gain remains if one dynamic rule is fixed in advance, or if the rule is selected in a leave-one-model-out setting, instead of choosing a different winner after seeing each model's evaluation results?

  2. At ceil(2K/3), what are the minimum, median, and range across the 27 normalized model-by-task-family retention values? Which model-task combination is the worst case?

  3. Which other factors differ within each checkpoint pair in Section 6—for example, post-training, generation length, training data and recipe, capacity, or evaluation suite? Which conclusions are meant only as checkpoint-specific associations? For the modality comparison, what happens if the text and vision-language checkpoints are evaluated on the same text-only prompts, with multimodal prompts considered separately?

  4. Using each rule's actual average expert count, how does every Table 3 winner compare with the two surrounding FIXED-K rows and an interpolated FIXED-K estimate, separately for generative and likelihood-scored evaluation? Do the aggressive-budget gains remain? What are the end-to-end prefill and decode results for BAN and DYNROUTE on the same backend and workloads as Table 2?

  5. Beyond broader model coverage and the FIXED-K control, what new scientific conclusion or method does the paper add over NAEE, DYNROUTE, Ban&Pick, and DiEP? Which findings are genuinely new, rather than broader confirmation of known expert redundancy?

4 comments
AuthorsOpenReview ↗

Response to Reviewer okTb (1/2)

We thank the reviewer for the careful and fair assessment, for recognising the breadth and completeness of the study, and for precise questions, which we could answer with the existing result matrix plus new analyses and experiments. All changes in the revised paper are marked in blue.

W1 / Q1 / Q4: rule selection and budget matching. What Table 3 is meant to show. We think the role of rule selection in Table 3 was not clear enough in our presentation. The paper does not propose a rule or claim that any one rule is best. Table 3 asks how much any of the published rules can add over uniform truncation at a given budget, so for each model and tier it reports the best of the four. This deliberately favours the dynamic rules and gives an upper bound on what allocation adds. That is why the conservative-tier conclusion is robust: even the best rule, chosen with hindsight, does not beat uniform truncation there. At the aggressive tier, where the bound is positive, the reviewer is right that selection and the small budget excess could inflate it, so we quantified both, and the gain remains.

(1) Matched budgets.

Exact matching is not possible in advance: a dynamic rule's knob is set before the run, but its average expert count depends on the tokens it routes, and since generations are sampled, this count fluctuates from run to run. The paper therefore allows up to 10% over the tier's budget. To remove this excess, we compare each Table 3 winner with FIXED-K interpolated linearly to the winner's measured kbar, separately for generation (seven datasets) and likelihood QA (four). Where a FIXED-K configuration was evaluated twice, the matched comparison uses the mean of the two readings. Aggressive tier, winner minus interpolated FIXED-K:

Modelrule, kbar (gen/QA)generationQAall 11
Qwen3-30B-A3BBan&Pick, 3.14/3.08+1.16+0.73+1.00
Qwen3-Next-80BBan&Pick, 4.27/4.18+1.69+1.53+1.63
Ling-lite-1.5Ban&Pick, 2.05+2.43+1.10+1.95
GPT-OSS-20BDynRoute, 2.07+1.86-1.37+0.68

At the conservative tier the matched margins are +0.21, -0.34, +0.29 and -0.44 (mean -0.07). The aggressive-tier advantage therefore remains on every model, at about 60% of the integer-budget figure (+2.26), and it sits in the generative datasets on all four models.

(2) Selection.

Choosing the rule on the other three models (leave-one-model-out) selects Ban&Pick for every held-out model at the aggressive tier: +1.00, +1.63, +1.95 and -1.13 (mean +0.86, positive on three of four). Fixing DynRoute in advance gives -0.37. At the conservative tier leave-one-model-out selects DynRoute every time (mean -0.08).

(3) Noise.

Paired 95% intervals over items, with the stored re-run on the FIXED-K side: aggressive tier +0.70 +- 1.01, +1.14 +- 0.83, +1.49 +- 1.23 (Qwen3-30B-A3B, Qwen3-Next-80B, Ling-lite), pooled +1.1 +- 0.6; conservative tier -0.25 +- 0.96, -0.45 +- 0.75, +0.33 +- 0.83. Table 3 has a new matched-budget column, and Section 5, the abstract and the new Appendix F (pp. 20-22) report the above.

(4) Prefill and decode of Ban&Pick and DynRoute on the Table 2 workload.

We implemented fused kernels for both rules, modeled on vLLM's fused MoE kernel; they add essentially no overhead, since the MoE layer is memory-bound and the rules are simple. On the seven configurations we measured (three models, both tiers), the rules then run at the throughput of uniform truncation at the same budget: at batch 4, the setting of Table 2, decode reaches 99.9-103.4% and prefill 98.7-103.7% of it, and decode stays at 97.8-103.4% from batch 1 to 256 (Table 16 in Appendix I, p. 26). The matched-budget gains above therefore hold at matched throughput as well: the aggressive-tier advantage is not bought with extra computation.

AuthorsOpenReview ↗

Response to Reviewer okTb (2/2)

W2 / Q5: contribution beyond NAEE, DynRoute, Ban&Pick and DiEP. We agree with the reviewer that expert redundancy was not first identified by us; our study builds directly on these works and credits them. Most of these rules, however, were designed for and evaluated on earlier MoEs with a few wide experts per layer (NAEE on Mixtral's top-2 of 8 experts, for example), and mostly on likelihood-scored or short-answer benchmarks, each at its own operating points and against the unpruned model rather than against uniform truncation at the same budget. Fine-grained MoEs with hundreds of experts and up to sixteen selected per token are now the mainstream design, and how these rules behave on them, and on the long-form tasks such models are used for, was not known.

Our contribution is to bring these methods together in one implementation, and to test, compare and summarise them on current fine-grained MoEs (nine architecture families, up to 295B parameters) and on a broader and more demanding suite (long-form mathematics, code and reasoning, and now long-context and agentic tasks), with uniform truncation at matched budgets as the control. This turns a set of separate claims into practical guidance. Near two thirds of the native selection, uniform truncation does as well as the best of the four rules, so a one-integer change is the right default. Dynamic allocation pays off only below about half of the native selection and mainly on long-form generation, so the rules should be judged on generative tasks and validated on the target model.

Frankly, at a time when AI applications are widespread and the number of papers is growing rapidly, we believe that a practical and objective assessment of how existing methods behave on the models deployed today can be more useful to the community than another method that reports state-of-the-art results but is hard to apply in practice. We therefore hope the reviewer can reconsider the contribution of our work.

The specific points in W2 are answered below: dispersion and worst cases (Q2) and the associational reading of Section 6 (Q3). On the adapted variants, our top-K form of DiEP is now described as our own generalisation and excluded from statements about published rules; it wins no tier, so no conclusion depends on it.

Q2: dispersion at ceil(2K/3). Over the 27 model-by-family values: median 98.4%, minimum 95.3%, maximum 103.6%, SD 2.0; 24 of 27 are at or above 97%. The three below 97% are knowledge QA on DeepSeek-V4-Flash (95.3%), Gemma 4 26B-A4B (95.3%) and Ling-lite-1.5 (96.7%). Section 4.1 now reports this.

Q3: what else differs within each pair.

  • Thinking vs Instruct (Qwen3-30B-A3B-2507): same architecture and pre-training; different post-training and output length (8.9K vs 3.7K tokens on average). Their routing concentration is virtually identical (retained gate mass 0.526 vs 0.524 at k=3; Appendix G, p. 23), so concentration does not explain the difference. The association also replicates on a second architecture: on Qwen3-Next-80B-A3B, the thinking checkpoint keeps 79.2% of its unpruned score on the four hardest generative datasets at three experts, against 73.0% for its Instruct counterpart (Table 14 in Appendix E, p. 24), so it is not specific to one checkpoint.
  • 235B vs 30B (Qwen3-2507 Instruct, both 128 experts with K=8): the larger model is deeper and wider (94 vs 48 layers, hidden size 4096 vs 2048, experts twice as wide), with more parameters and training compute.
  • VL vs text (Qwen3-VL-30B-A3B vs Qwen3-30B-A3B-2507): identical text-model configuration; different training, and in the submission a different evaluation suite. We have now removed the suite confound: on the same nine text QA datasets the VL checkpoint degrades like the text model (59.8% vs 62.0% at k=2), so the added sensitivity belongs to the multimodal tasks (Appendix K, p. 28). Section 6 now lists these factors and phrases each conclusion as an association; the reasoning-mode association now holds on two architectures.

Thanks for the response. I agree that a broad and practical evaluation is valuable, and I appreciate the effort put into this paper.

However, I still feel that the two main conclusions are already well-established observations in the model compression. Expert redundancy has been demonstrated by NAEE (ACL 2024), and the greater benefits of adaptive pruning at high compression ratios have also been observed in prior work such as DLP (ICML 2025). So while this paper provides a more comprehensive evaluation, I don't think its novel contributions are sufficient to meet the bar for ICLR. I think the paper's novelty would be stronger if the broad evaluation across different architectures could also provide deeper insights into how current pre-training or post-training practices contribute to expert redundancy.

Overall, I think this is a solid and well-executed paper, but I still have concerns about its novelty, similar to Reviewer tgzA. That said, if the other reviewers support acceptance, I would not object.

Best wishes to the authors.

Thank you very much for recognizing our paper as solid and well-executed, and for your constructive suggestions on strengthening its novelty. We respect your judgment and sincerely appreciate your thoughtful feedback. We are currently awaiting feedback and updated assessments from the other two reviewers.

Best wishes,

The authors

Reviewer_tgzAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies dynamic expert pruning in fine-grained MoE LLMs. The authors evaluate 12 checkpoints from 9 architecture families (Qwen3, Qwen3-Next, Ling, GPT-OSS, MiniMax-M2.7, DeepSeek-V2-Lite and V4-Flash, Hy3, Gemma 4) on 11 benchmarks, four of them likelihood-scored QA and seven generative math, code and knowledge tasks. The first finding is that uniformly keeping ⌈2K/3⌉ of the selected experts (FIXED-K) retains 98.8% of unpruned performance on average, with a measured 1.2 to 1.7× speedup on vLLM and HF Transformers.

Critical strengths
  1. Breadth and a fair comparison protocol: The study covers 9 architectures with K from 4 to 10, softmax, sigmoid and sqrt-softplus routers, and models with and without shared experts, and it includes long-form generative tasks. Each dynamic rule is compared with FIXED-K at the same measured k̄.

  2. A simple result practitioners can use: Setting a single integer to ⌈2K/3⌉ keeps 98.8% of performance and gives real speedups on two serving stacks (Table 2) without any calibration. Any new pruning method now has to beat this.

Critical weaknesses
  1. Dynamic rules are compared on accuracy but not on cost: The paper concludes that dynamic allocation is worth its complexity under aggressive pruning (+2.26 on average), but throughput is only measured for FIXED-K, which runs through the native config path without the router patch (Appendix F). BAN and DynRoute use a variable number of experts per token and layer and need calibration, and their kernel overhead is not reported. Two other factors could cancel a 2 to 3 point gain in deployment. Pruned models tend to generate longer outputs (Appendix A.1 notes that they "ramble towards the token limit", and the thinking model already averages 8.9k tokens), so per-token savings may not carry over to per-request cost. Speedups are also only reported at batch size 4, and the benefit of a smaller k changes a lot at serving batch sizes or with expert parallelism. Without accuracy vs. latency (or throughput) curves for FIXED-K and the best dynamic rules, the main conclusion of Section 5 is incomplete.

  2. Limited novelty, and the 2/3 rule is not explained: Reducing the number of activated experts at inference while keeping the full expert pool has been studied before. For example, [3] evaluates exactly this setting (k → k′) on GPT-OSS-20B and DeepSeek-Coder-V2-Lite and finds that keeping 3 of 4 experts on GPT-OSS-20B retains most of the math accuracy, while static expert pruning collapses. Huang et al. (2024) and Lu et al. (2024) also study reduced budgets. The contribution is therefore mainly breadth, yet the 2/3 rule is only explained qualitatively in Section 4.1. It would help to relate the routed gate mass retained at ⌈2K/3⌉ to relative accuracy across the nine models; at the moment that analysis is only done for the VL pair (Appendix G). The Section 6 sensitivity claims each rest on one model pair, and the modality pair is confounded: Qwen3-VL is scored on nine multimodal short-answer tasks while the text model is scored on nine likelihood-scored QA tasks, even though Section 4.2 shows that task format strongly affects pruning sensitivity.

  3. Missing related work: The paper should discuss [3], which measures the same reduced-k strategy and compares it with static expert pruning, and [6], which reports that MoE compression looks harmless on multiple-choice commonsense benchmarks but breaks GSM8K, HumanEval and MATH. That is the same pattern as Section 4.2 and supports the same argument against likelihood-only evaluation. Other methods that let the number of experts vary per token are also not discussed: null-expert routing in AdaMoE [1], threshold-based expert selection in XMoE [5], ReLU routing in ReMoE [2], and the online dynamic pruning in MC-MoE [4]. Most of these change training or calibration rather than only inference, but they are the obvious comparison points for the claim that adaptive allocation pays off only under aggressive budgets.

Minor: The DIEP rows implement only the skipping criterion, with a rank-K generalization chosen by the authors (Appendix B). The paper is upfront about this, but these rows say little about DIEP as published, so I would leave them out of general statements about "published rules".

References: [1] Zihao Zeng et al. AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models. Findings of EMNLP 2024.

[2] Ziteng Wang, Jianfei Chen, Jun Zhu. ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing. ICLR 2025.

[3] Haoze He et al. Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning. COLM 2026.

[4] Wei Huang et al. MC-MoE: Mixture Compressor for Mixture-of-Experts LLMs Gains More. ICLR 2025.

[5] Yuanhang Yang et al. XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection. Findings of ACL 2024.

[6] Haoze He, Xinkai Zou et al. Less is MoE: Trimming Experts in Domain-Specialist Language Models. EMNLP 2026.

Questions for authors
  1. What is the measured vLLM decode throughput of BAN and DynRoute at the aggressive tier, compared with FIXED-K at the same k̄? An accuracy vs. throughput plot for both would show whether the +2.26 gain holds once overhead is included. (Clarification; this could change my rating.)

  2. How does mean output length change under FIXED-K at ⌈2K/3⌉ and at the aggressive budget, especially for the thinking model and LiveCodeBench? Could the authors report end-to-end per-request latency in addition to per-token throughput?

  3. Is the safe budget better described as a fixed fraction of K or as a fixed amount of retained gate mass? Section 4.1 notes that k = 2 is catastrophic for Qwen3 but nearly free for GPT-OSS, which suggests gate mass may be the better predictor.

  4. Qwen3-VL accepts text-only input. Could the modality comparison be run on the same text benchmarks for both models, to separate a model property from a task-format effect?

  5. Suggestion for future work: do the Table 2 speedups hold at larger batch sizes (e.g. 32 to 256) and under expert parallelism, where all experts are loaded anyway?

The paper needs a more comprehensive related work. I would consider to raise my score if the answers to Q1 and Q2 show that the best dynamic rules keep their advantage at matched wall-clock cost.

4 comments
AuthorsOpenReview ↗

Response to Reviewer tgzA (1/3)

We sincerely thank the reviewer for the detailed and expert review, and in particular for pointing us to He et al. (2026a; 2026b). Both papers are directly relevant, and reading them sharpened how we position our study; we now discuss them prominently in the introduction and Section 2. The concerns about system cost and per-request cost are also well taken; addressing them required new code and measurements, which we report below. All changes in the revised paper are marked in blue.

W1 / Q1: cost of the dynamic rules. Thank you for this question. To be precise about the paper's claims: the speedups in Table 2 are for uniform truncation, which needs no custom kernel, since the model configuration simply declares fewer experts per token. For the dynamic rules, the paper compares accuracy at matched budgets and leaves their execution cost open; Section 7 notes that "kernels must execute a varying number of experts per token without spending the savings on overhead". We have now measured both: below we first confirm the speedups of uniform truncation, then report fused kernels for Ban&Pick and DynRoute and their throughput against uniform truncation at the same budget.

Uniform truncation. Re-measured with real evaluation prompts (vLLM 0.10.2 with CUDA graphs, HF Transformers 4.57), the speedups are consistent with Table 2: decode 1.18-1.65x in vLLM and 1.44-1.73x in HF Transformers, prefill 1.15-1.31x. They grow at the aggressive tier and persist at larger batches and under expert parallelism (see Q5).

Dynamic rules: fused kernels. The router patch used for the accuracy runs zeroes the weights of dropped experts, which saves no computation. We therefore implemented fused versions of Ban&Pick and DynRoute, modeled on vLLM's fused MoE kernel. One Triton routing kernel per layer computes the softmax, the top-K selection, the rule's per-token keep count and the renormalized weights, and the grouped expert GEMMs receive only the kept (token, expert) pairs, so dropped pairs are never loaded, computed or reduced. Because the MoE layer is memory-bound and both rules are a few operations on a token's top-K weights, fusing costs essentially nothing: with a rule that keeps every expert, the fused path runs at 1.00-1.04x the unpruned model on all three models and batch sizes 1 to 256. The kernels keep exactly as many experts per token as the router patch and reproduce its accuracy (mean difference +0.1 points for DynRoute and -0.5 for Ban&Pick over six generative datasets on Qwen3-30B-A3B). To be candid, we wrote these kernels largely with the help of an AI coding assistant, which is why we validated them against the router patch; they are in the updated supplementary material so that others can inspect and improve them.

Throughput of the fused rules, in % of FIXED-K interpolated to the rule's measured kbar (vLLM, 1024-token prompts, 256 generated tokens; b = batch size):

Model, rule (tier, kbar)decode b=1b=4b=32b=128b=256prefill b=4b=32
Qwen3-30B-A3B, DynRoute (conservative, 4.79)102.2100.999.8100.3100.0101.598.7
Qwen3-30B-A3B, Ban&Pick (aggressive, 3.12)101.7103.0100.9100.9100.0101.0100.5
Qwen3-Next-80B, Ban&Pick (conservative, 6.16)101.1103.4100.499.7100.399.399.4
Qwen3-Next-80B, Ban&Pick (aggressive, 4.23)99.899.999.899.3100.098.798.9
Qwen3-Next-80B, DynRoute (aggressive, 4.05)99.3101.997.898.398.298.999.6
Ling-lite, DynRoute (conservative, 4.11)102.7101.5100.3100.5100.5103.7100.7
Ling-lite, Ban&Pick (aggressive, 2.13)103.2101.2100.099.9100.3102.2101.8

Over the 35 decode cells the fused rules reach 97.8-103.4% of the throughput of uniform truncation at the same budget (median 100.3%), and prefill 98.7-103.7%. To Q1 directly: at the aggressive tier, the fused Ban&Pick decodes at 103.0%, 99.9% and 101.2% of FIXED-K at the same budget on Qwen3-30B-A3B, Qwen3-Next-80B and Ling-lite at batch 4 (100.9%, 99.8% and 100.0% at batch 32). The accuracy gains of the dynamic rules at matched budgets therefore carry over unchanged to matched throughput. Under the stricter accounting we added to Table 3, which interpolates FIXED-K to each rule's measured budget, the aggressive-tier gain is +1.32 points on average (+1.00, +1.63, +1.95 and +0.68 on the four models). This is smaller than the +2.26 at integer budgets but positive on every model, and the abstract and Section 5 now use it. The kernel design and these measurements are now in Appendix I (formerly Appendix F; pp. 25-26, Table 16), and we will add the accuracy-throughput plot the reviewer suggests.

AuthorsOpenReview ↗

Response to Reviewer tgzA (2/3)

Q2: output length and per-request cost. This is an important point, and we now report it (new Appendix J, Table 17, p. 26). Total output tokens over the seven generative datasets, relative to the unpruned model, and the implied routed-expert computation per response, (k/K) x length:

Model (K)ceil(2K/3): length, computeaggressive tier: length, compute
Qwen3-30B-A3B (8)0.96, 0.72 (k=6)1.64, 0.62 (k=3)
Qwen3-Next-80B (10)1.09, 0.77 (k=7)1.48, 0.59 (k=4)
Ling-lite-1.5 (6)0.73, 0.49 (k=4)0.80, 0.27 (k=2)
GPT-OSS-20B (4)0.99, 0.74 (k=3)1.58, 0.79 (k=2)
Qwen3-30B-A3B-Thinking (8)1.07, 0.81 (k=6)1.38, 0.52 (k=3)
Qwen3-235B-A22B (8)1.12, 0.84 (k=6)2.74, 1.03 (k=3)

For the thinking model, LiveCodeBench responses average 15.6K tokens unpruned, 16.8K at k=6 and 26.9K at k=3. The pattern supports the reviewer's concern and sharpens our guidance: near two thirds of the native selection the per-token saving carries over to the request, while under aggressive pruning longer outputs offset much of it (on Qwen3-235B, all of it). On agentic tasks the same effect appears at the level of whole trajectories (Appendix L, pp. 28-29). We also measured per-request latency directly: 190 prompts from the generative evaluation sets of Qwen3-30B-A3B (GSM8K, MATH-500, AIME 24, LiveCodeBench, MMLU-Pro, GPQA-D), submitted together to vLLM with at most 64 concurrent requests and the evaluation's sampling settings (T=0.7, up to 32K tokens). Mean latency per request, excluding the few requests that hit the token limit, averaged over three sampling seeds:

ConfigurationInstructThinking
unpruned (k=8)85.4 s488 s
FIXED-K k=6 (ceil(2K/3))77.1 s459 s
FIXED-K k=3 (aggressive)221.2 s1052 s

Near ceil(2K/3), the longer outputs eat into the speedup but do not cancel it: on these 190 prompts, outputs grow by 4-6%, and requests still finish 10% faster for Instruct and 6% faster for the thinking model. Under aggressive pruning, outputs roughly double (912K vs 476K tokens for Instruct) and requests take 2.2-2.6x longer despite cheaper tokens, so aggressive pruning clearly does not pay off per request. This supports the paper's recommendation to start at ceil(2K/3). Thank you for raising this point; Appendix J (p. 26) now also reports these latencies.

W2: novelty and why two thirds; Q3: fraction of K or gate mass? We agree that reduced-k inference is not new in itself, and we are grateful for the pointer. He et al. (2026a) give an elegant and clear demonstration, in a scaling analysis on GPT-OSS-20B and DeepSeek-Coder-V2-Lite, that lowering the activation budget degrades far more gracefully than removing experts, and that even rarely activated experts hold knowledge worth preserving. This is exactly the insight that justifies treating uniform truncation as the reference for dynamic rules, and their GPT-OSS-20B result at three of four experts agrees with ours. As mentioned above, we now cite and discuss it in the introduction and Section 2. Our study builds on that observation in a different direction: it characterises the safe budget across nine fine-grained architectures and long-form generation, and uses uniform truncation as the reference against which four dynamic rules are compared at matched measured budgets. That comparison is what shows that, at conservative budgets, the best published rules add nothing over a one-integer change, and that their value lies below about half of the native selection.

On Q3, we measured retained gate mass directly on seven models: we teacher-force each model on its own stored responses and record, at every router, the share of the native top-K weight held by the k highest-weight experts. At ceil(2K/3) the kept experts hold 0.77-0.89 of it, so the dropped third carries 11-23%. Gate mass and the fraction of K describe the safe budget about equally well: over 123 model-family-budget points, accuracy retention has Spearman correlation 0.88 with retained mass and 0.90 with k/K, and the budget at which each model and family first falls below 97% sits at k/K = 0.68 +- 0.11 or a retained mass of 0.80 +- 0.09. They are nearly interchangeable because, at a given fraction of K, retained mass varies little across models. Concentration explains some differences between models but not all: Qwen3-235B is more concentrated than Qwen3-30B (0.56 vs 0.52 of the top-K weight kept at k=3 on generated math and code tokens), in the direction of its greater tolerance, but Qwen3-30B Thinking and Instruct have virtually identical gate mass (0.526 vs 0.524), yet the thinking model keeps 5 more points on hard generative tasks at k=3. We added this analysis as a new appendix (Appendix G, pp. 22-24; Table 15, p. 25).

AuthorsOpenReview ↗

Response to Reviewer tgzA (3/3)

W2: each Section 6 claim rests on one pair; the modality pair is confounded. Q4. We agree, and we added evidence for each pair. For reasoning mode we added a second pair on a different architecture, Qwen3-Next-80B-A3B Thinking vs Instruct (hybrid attention, 512 experts, K=10), with the same four hard generative datasets and settings. The pattern replicates: the two retain similar fractions of their unpruned score from nine to four experts (100.9% vs 98.9% at k=5, 92.7% vs 93.2% at k=4), and at three experts the thinking checkpoint keeps 79.2% against 73.0%, a 6.2-point lead (5.1 points on Qwen3-30B-A3B). Section 6 and a new table in Appendix E (Table 14, p. 24) report the second pair. Section 6 now also lists what else differs within each pair, states the conclusions as associations for these checkpoints, and restricts the scale statement to the within-family comparison (across our nine models, parameter count does not predict tolerance; Spearman -0.45 to -0.15). For the modality pair we ran the reviewer's suggested control: Qwen3-VL-30B-A3B on the text model's nine likelihood-scored QA datasets, same harness, k = 1..8. On this suite it degrades almost exactly like the text model (89.7%, 80.4% and 59.8% of its unpruned score at k = 4, 3, 2, against 91.8%, 83.2% and 62.0% for the text model re-run in the same environment, which reproduces its original curve to within half a point), whereas on the multimodal suite it keeps 72.4% and 19.8% at k = 3 and 2. Its routing on text prompts is also close to the text model's (the leading three of the selected eight hold 59.8% vs 59.4% of their weight; the selected eight hold 40.0% vs 44.9% of the full softmax), far from the 49.8% and 15.5% it shows on multimodal prompts. So the added sensitivity comes from the multimodal inputs and tasks rather than from the checkpoint. We added this control to Appendix K (formerly Appendix G; pp. 27-28, dashed curve in Figure 4) and state it in Section 6.

W3: related work. Thank you for these references. Section 2 now discusses He et al. (2026a) as above. He et al. (2026b) is especially close to our findings: their insight that the capability behind mathematical reasoning concentrates in a small set of FFN dimensions, so that compression which looks harmless on commonsense multiple choice can collapse GSM8K, mirrors at the parameter level the asymmetry we observe at the level of routing, where knowledge survives budgets at which reasoning does not (Section 4.2). We now draw this connection explicitly. We also discuss MC-MoE's online pruning of less important experts, and AdaMoE, XMoE and ReMoE as training-time ways to vary the number of experts per token, which differ from the post-hoc rules we study.

Minor: DiEP. Agreed. Statements about "published rules" now refer to NAEE, DynRoute and Ban&Pick; we state that our top-K form of DiEP is our own generalisation. DiEP never wins a tier, so no conclusion changes.

Q5: larger batches and expert parallelism. We measured uniform truncation at batch sizes 1, 4, 32, 128 and 256 (vLLM, 1024-token prompts, 256 generated tokens). The decode gains are largest at small and medium batches, where decoding is bound by loading expert weights, and they persist at the aggressive tier. At 128-256 sequences they become smaller, because nearly every expert is selected by some token in the batch anyway; Qwen3-Next-80B, with 512 experts, keeps more of its gain to larger batches than the models with 64 or 128 experts. Prefill is compute-bound at large batches and keeps its speedup there. With four-way expert parallelism on Qwen3-Next-80B, where every expert is resident on some GPU, uniform truncation still speeds up decoding, relative to the unpruned model under the same parallelism: 1.09x / 1.14x at batch 4 / 32 for k=7, 1.12x / 1.20x for k=6 and 1.21x / 1.37x for k=4. The fused dynamic rules track uniform truncation at the same budget at every batch size we measured (table in our answer to W1).

Dear Reviewer tgzA,

As the discussion period draws within 4 hours, we wanted to kindly follow up on our rebuttal posted a few days ago above along with the revised paper, in which we addressed your questions in detail.

We would greatly appreciate it if you could review our responses and revisit your assessment. If any concerns remain, we would be more than happy to provide further analysis and explanations.

Thank you again for your time and review service!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their careful and constructive reviews. We have uploaded a revised paper, with all changes marked in blue, and an updated supplementary material, and we reply to each reviewer separately. The main additions are:

  1. Cost of the dynamic rules. We implemented fused kernels for Ban&Pick and DynRoute that execute only the kept experts. They run at 98-103% of the throughput of uniform truncation at the same average expert count, so the accuracy gains of the dynamic rules at matched budgets hold at matched throughput (Appendix I, Table 16, p. 26). The kernel implementation, with a throughput benchmark and correctness tests, is in the updated supplementary material (fused_kernels/).
  2. Stricter accounting. With uniform truncation interpolated to each rule's measured budget, the aggressive-tier gain is +1.32 points on average and positive on all four models; at conservative budgets there is still no gain (Table 3; Appendix F, pp. 20-22).
  3. Per-request cost. Near two thirds of the native selection, output length changes little and requests still finish faster than unpruned; under aggressive pruning, requests take 2.2-2.6x longer (Appendix J, Table 17, p. 26).
  4. Broader evaluation. Agentic tasks (seven benchmarks, five models; Appendix L, Table 18, p. 29) and long-context tasks (RULER and LongBench; Appendix M, Tables 19-20, p. 30) support the two-thirds budget.
  5. New analyses. Retained gate mass (Appendix G, pp. 22-25), benchmark noise (Appendix H, p. 24), a second reasoning-mode pair (Appendix E, Table 14, p. 24) and a text-only control for the modality pair (Appendix K, Figure 4, pp. 27-28).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.