acceptodds
Under review as a conference paper at ICLR 2027

Don't Skip the Baseline: A Controlled Comparison of Dynamic Expert Skipping in MoEs

Abstract

A growing body of work proposes dynamic expert-skipping methods for Mixture-of-Experts (MoE) models that adjust the number of experts activated per token during inference, aiming to preserve model quality while further reducing the number of active parameters. These methods have added calibrated importance scores, inter-expert similarity metrics, or learned per-token allocators in order to identify experts to skip. However, different calibration regimes, different reductions in the effective number of experts, and evaluation suites make comparisons difficult. We classify four representative prior methods into a unified taxonomy and benchmark them against a natural, simple baseline: statically reducing the number of experts (i.e., same reduced across all layers and tokens). We provide a fair comparison of these methods at matched expert-compute budget, on two modern MoEs (Gemma4-26B-A4B and Qwen3-30B-A3B) in a text-only setting and a contemporary vision-language model (Qwen3-VL) in a multimodal setting. Surprisingly, we find that the simple baseline is hard to outperform consistently across models, modalities, benchmarks, and calibration settings. The methods we benchmark often underperform the baseline and achieved gains, when present, are minor. We further find that the effective achieved (or optimized for) in calibration can diverge from the effective in testing, resulting in over- or under-estimating the compute at inference time. Based on these findings, we recommend a calibration-free per-modality static reduction and validate this on a held-out model and modality. We further argue that a static control at matched measured compute is the baseline any dynamic skipping method must beat.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
mixture-of-experts, dynamic expert skipping, iso-compute comparison, multimodal, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_akEkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

The paper proposes and investigates the questions on whether dynamic expert-skipping strategy outperform simply reducing the number of activated experts in pretrained MoE models. It compares various methods with static top-k baselines through KeffK_{\mathrm{eff}} metric, a number measuring the averaged number of activated experts. Experiments cover language, vision language with a small extension in video models. The main conclusion from the experiments is that dynamic methods rarely outperform a matched static baseline consistently. Another interesting observation from multimodal setting is that different modality leads to very different performance when doing expert skipping.

Critical strengths
  1. The study addresses an important comparison problem. The problem of systematically comparing dymanic expert skipping with static top-k is timely proposed. Measuring expert usage during evaluation with proposed metric KeffK_{\mathrm{eff}} provides a fair comparison. The finding that modality-aware static routing closes much of the performance gap could meaningfully improve evaluation protocol in this area.

  2. The empirical results are comprehensive The paper includes multiple architectures, classification and generative tasks, calibration sensitivity, and held-out transfer checks. Appendix A.7 also validates the reimplementations against published results. These efforts strengthen the central hypothesis and make the paper more convincing. However, the code has not been submitted.

Critical weaknesses
  1. The experiments do not clearly shows where the compute budget should be allocated. A dynamic method can allocate different numbers of experts across layers, modalities, and individual tokens. The proposed baseline mainly distinguishes the number of experts between vision and text. Therefore, the comparison cannot fully explain which components where the dynamic weighting helps and which adds unnecessary complexity. It would be interesting to see a more in-depth analysis in this direction, so the pratical recommendation would be more helpful.

  2. The paper does not directly show the runtime/accuracy trade-off. The main comparisons indirectly show this trade-off through the proposed metric KeffK_{\mathrm{eff}}. This is useful for comparing expert computation, but the runtimes of methods still may differ. Figure 1 and Appendix A.5 measures only the static baseline, so it does not fully show the trade-off in different settinfs. A direct comparison of runtime and accuracy would make the practical recommendation more convincing.

Questions for authors
  1. Comparing noninteger expert budgets. How should we choose between dynamic and static routing? For example, if a dynamic method averages k=2.5k=2.5 experts per token, achieves higher accuracy than static k=2k=2, and runs faster than static k=3k=3, would you recommend the dynamic method?

  2. Direct runtime/accuracy comparison. Could you provide a table comparing the accuracy and measured runtime of static and dynamic skipping directly without the KeffK_{\mathrm{eff}} metric?

  3. Identifying where to allocate computation. Could you compare other different allocation strategies, such as fixed number of experts in each layer, at matched measured expert budgets? This could help the recommendation more diverse than only changing the number of activated experts in different modalities.

3 comments

We thank Reviewer akEk for their insightful review. We have addressed each point below and have included further insights into how to allocate expert compute based on which signal.

Question 3 / Weakness 1 Allocation of computation: We consider the allocation of compute to experts on three granularity levels: Modality, Layers and Tokens.

  • Modality: Per-modality static allocation is the only signal that consistently adds value over uniform allocation.
  • (new) Layers: We added two per-layer static allocation methods to the comparison: LExI/FastMMoE do not improve on uniform top-kk on Qwen3. Especially LExI falls behind the static baseline at low KeffK_\text{eff} (table below / Appendix A.9).
  • Tokens: Per-token dynamic allocation (MoDES, DiEP, Allocator, NAEE) does not consistently outperform the static or per-modality baselines at matched KeffK_\text{eff} Recommendation: Allocate compute by modality, keeping k uniform across layers and tokens.

The tables show the new results for the layer granularities:

K_effSoftmax top-kLExI
1.556.9*47.0
278.156.4
2.582.9*79.8
387.683.3
492.389.6
697.896.7
K_effSoftmax top-k (interp.)FastMMoE
1.4454.340.2
2.3881.780.1
3.3189.188.6
4.2593.092.6
5.1995.695.8
*interpolated; Retained quality over GSM8K, ARC-C, HellaSwag and HumanEval.

Question 1 Comparing noninteger expert budget: We do not recommend the dynamic methods in this case: the static methods can be used to on average achieve the desired noninteger operating point by, for instance, alternating between the higher and lower static k per request to on average approach the desired quality level and performance characteristics. The expectation of this approach matches the targeted KeffK_\text{eff} and accuracy. This means that any point on the frontier can be achieved using this simple scheme. We elaborate on this in the revised Section 4.2.1.

Question 2 / Weakness 2 Direct runtime/accuracy comparison: We have added a clear runtime / accuracy table for our recommended static operating points on our per-modality trade-off curve. For the dynamic skipping methods we deliberately didn't provide realistic runtime figures. Any fair comparison would require highly optimized kernels, which are not provided by any of the authors, and any difference in performance would be more reflective of the amount of optimization effort rather than actual performance. Static top-k\text{top-}k serves as a bound. At any KeffK_\text{eff} it has the lowest runtime as it has no routing decision, uniform per-token compute and fixed tensor shapes that keep fused MoE kernels and CUDA graphs applicable without further kernel support. As such, a method that doesn’t beat the static top-k\text{top-}k baseline at matched KeffK_\text{eff} cannot beat it at matched runtime. This means that the comparison is favorable to the dynamic methods. Nonetheless, we have run a evaluation on the unoptimized pytorch implementations on Qwen3 and have included them to have some estimate.

Runtime for recommended Multimodal Operating Points:

Operating Point (k_vision/k_text)Avg. AccuracyTTFT, bs=1ITL, bs=1Throughput, 32 concurrent
1, 882.9% (−3.5)494 ms (−7.8%)9.6 ms (−1.3%)261 tok/s (+10.1%)
2, 884.9% (−1.4)509 ms (−5.1%)9.7 ms (−0.4%)258 tok/s (+8.8%)
1, 6 (recommended)82.8% (−3.6)492 ms (−8.3%)9.1 ms (−6.9%)260 tok/s (+9.9%)
2, 684.7% (−1.6)501 ms (−6.7%)9.1 ms (−6.8%)258 tok/s (+8.9%)
8, 8 (native)86.3%536 ms9.7 ms237 tok/s
Runtime measured on one A100 on Qwen3-VL using vLLM, each point was repeated twice.

Methods comparison (unoptimized torch) for k≈3k\approx3 on Qwen3:

MethodAvg. AccuracyPrefill tok/sDecode ms/token bs=1Decode ms/token bs=64Routing overhead (ms/step)Active experts/token
Top-8 (full model)64.412,204117.4 ± 1.8457.3 ± 0.8–8.00
Softmax top-358.715,793122.1 ± 0.3256.1 ± 0.40 (ref.)3.00
NAEE58.6*13,848191.3 ± 0.9330.0 ± 1.8+11.93.03
DiEP52.2*13,835187.7 ± 1.0334.7 ± 1.7+9.22.98
MoDES52.0*14,444174.1 ± 1.0290.9 ± 0.8≈02.98
Allocator56.812,709216.5 ± 1.7341.5 ± 2.3+38.33.10

*interpolated. Avg. Accuracy over GSM8K, ARC-C, HumanEval, HellaSwag.

At bs=1 in this implementation, kernel launch overhead negates any benefit of the skipping methods (including static).

Code Release We have attached our code as a supplementary material on open review

We hope this addresses the reviewer’s concerns and would be grateful if the reviewer were to reconsider their rating in light of these additions.

Please read the authors' rebuttal and post your comments on your raised concerns.

AC

Reviewer_akEkOpenReview ↗

Thanks for rebuttal

Thanks the author for your rebuttal.

After reading the rebuttal to other reviewers as well as the updated manuscript, most of my concerns have been resolved. I have no more questions and decided to raise the score to weak accept.

Reviewer_4ZYjOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies dynamic expert-skipping methods for Mixture-of-Experts (MoE) inference, i.e., methods that vary the number of experts activated per token/layer at inference time in order to reduce effective compute while preserving quality. The authors observe that existing methods (NAEE, DiEP, MoDES, Ada-K) report results under mutually incompatible calibration regimes, effective expert counts, and evaluation suites, and are often bundled with expert pruning, making their claimed benefits hard to attribute. The paper offers a unified taxonomy of dynamic expert-skipping methods and a controlled comparison at matched test-time compute on two text-only MoEs and one VLM, tracing full quality–compute frontiers against static top-k baselines, including a per-modality variant. The core finding is negative: no dynamic method systematically outperforms the simple static baseline; the occasional small gains arise only in heavily degraded regimes, and the strongest multimodal method beats the modality-unaware baseline but not the calibration-free per-modality one. They recommend a matched-compute static control that exploits the same structural signal (e.g., modality) as the mandatory baseline for future skipping methods.

Critical strengths
  1. A timely "critical analysis" type contribution. Challenging the evaluation conventions of recently published methods — in particular reaching conclusions that conflict with MoDES on the same model (Qwen3-VL-30B-A3B) at overlapping operating points — is exactly the kind of evaluation-standards work the community needs.
  2. Rigorous experimental design: compute is matched at measured (not calibrated) K_eff; full frontiers are traced rather than single operating points; both realistic (mixed corpus) and best-case (benchmark training split) calibration regimes are evaluated; multimodal K_eff is decomposed by modality. These choices directly address the stated incomparability of prior work.
  3. The recommendation is validated on genuinely held-out checks: MVBench (video modality, never involved in calibration or selection) and GLM-4.6V (different family, shared experts, k=8/128). This goes beyond the common practice of tuning and reporting on in-distribution benchmarks.
Critical weaknesses
  1. Narrow static control family. The deployed controls are only the global uniform k and the per-modality k. Two natural, calibration-free static controls are missing: (i) per-layer static k (input-independent, hence still static): LExI (Chitty-Venkata et al., 2025) — cited in the Background as "data-free per-layer expert budgets" — is precisely such a method, yet it is neither evaluated nor is its exclusion explained. Per-layer static k would likely achieve higher quality than uniform k at the same mean K_eff, and would naturally provide non-integer K_eff operating points; (ii) the onset-layer variant of Xia et al. (2025). Their absence leaves the claim "the static baseline is strong enough" incomplete (a stronger static baseline would only reinforce the conclusion, but could also swallow the Allocator's 2.6 pp advantage), and leaves a gap in the claim that dynamic methods do not beat the best static control.
  2. Scope claims are somewhat optimistic. Only models with k=8 are evaluated; among the "three models", the multimodal model shares the Qwen3 lineage (model card, tokenizer, calibration protocols). The conclusion "across three models as well as two held-out checks" could be misread as saying that the negative result was also validated on held-out data — in fact, MVBench and GLM-4.6V validate only the static recommendation, not the extrapolation of the negative result. Generalization to k≠8 (e.g., k=2 or k=16) is entirely unknown.
Questions for authors

Please see weaknesses above.

2 comments

We thank Reviewer 4ZYj for their review and recognizing the timeliness of the critical analysis and our rigorous experimental design. We also thank them for their critical feedback and have addressed it, integrating both an additional model and a comparison with LExI, an additional method.

Scope Claims: We agree on the limitation to k=8k=8 models and introduce GPT-OSS 20B in order to address this and cover three distinct model families. GPT-OSS natively activates k=4k=4 experts per token, this is included in the new Appendix A.10. Results are consistent at k=4 and k=8 across three model families, which covers current open-weight mid-sized MoE configurations: no advanced method consistently outperforms the static baseline. For GPT-OSS, skipping even just a single expert results in quality degradation of more than 9%, outside of most practical applications. As such we do not recommend an operating point there. The table shows the quality retained at comparable operating points. We further adjusted the wording to reflect the now 3 model families for text evaluation and reworded the conclusion so the held-out checks are no longer attached to the negative result.

TargetMethodMeasured K_effRetained QualityStatic top-k at same K_effΔ
k≈3NAEE3.1292.891.6+1.2
k≈3Allocator3.1193.091.5+1.5
k≈3MoDES3.2483.892.8−9.0
k≈3DiEP3.0280.990.7−9.8
k≈2NAEE2.0575.076.2−1.2
k≈2Allocator2.3277.180.2−3.1
k≈2MoDES2.0565.976.2−10.3
k≈2DiEP2.1065.076.9−11.9

Static top-k is linearly interpolated between integer kk. The table shows retained quality over GSM8K, HumanEval, ARC-C and HellaSwag.

We did not evaluate k=2 models such as Mixtral, which no longer reflect current fine-grained designs, and we are not aware of a mid-sized open-weight model with k>8.

Expanded static control: Based on your feedback we have added a evaluation of LExI and FastMMoE (Onset layer variant) on Qwen3 as well as an evaluation of the Onset layer variant on vision tokens for Qwen3-VL (new Appendix A.9). We reimplement their logic since no official implementation exist, and include it in our code release. In our experiment, the additional flexibility of per-layer static-kk does not provide additional benefit over the static allocation, static-kk (per-modality if applicable) remains the strongest static control tested. As shown below LExI tracks our baseline quite closely for Keff≥4K_\text{eff} \ge 4, while falling behind in low KeffK_\text{eff} regimes. We suspect this is due to the fact that it assigns more extreme expert counts per layer. For instance at KeffK_\text{eff} = 4 it assigns 18 Layers k=2, 16 Layers k=3, 1 Layer k=4 and 13 Layers k=8. This means that few layers receive a medium amount of experts. The FastMMoE approach has a large number of potential configurations, instead of sweeping them we base our selection on their results. The table highlights this as average quality retained for Qwen3 on our four text benchmarks (ARC-C, HumanEval, GSM8K, HellaSwag).

K_effSoftmax top-kLExI
1.556.9*47.0
278.156.4
2.582.9*79.8
387.683.3
492.389.6
697.896.7
K_effSoftmax top-k (interp.)FastMMoE
1.4454.340.2
2.3881.780.1
3.3189.188.6
4.2593.092.6
5.1995.695.8

*interpolated

We hope these additions address Reviewer 4ZYj’s concerns and strengthen the paper considerably.

Please read the authors' rebuttal and post your comments on your raised concerns.

AC

Reviewer_RzeQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The author presents four dynamic expert skipping methods (NAEE, DiEP, MoDES, and Ada-K) under a unified taxonomy and compare them against the static top-k baseline at matched expert-compute budget, Keff which is average no. of experts active per token per layer. The results are evaluated with Gemma4-26B-A4B and Qwen3-30B-A3B for text and Qwen3-VL-30B-A3B for vision-language tasks. The main findings are:

  • None of dynamic skipping methods consistently beats the static baseline at matched Keff.
  • Modality-specific budget is important as vision tokens can take less experts than text.
  • Calibrated Keff and measured Keff can diverge substantially.
Critical strengths

A clean negative study comparing relevant dynamic expert-skipping methods with the static top-k baseline.

An important distinction between calibrated Keff and measured Keff and shows both can diverge substantially (Table 2).

A useful insight that compared four dynamic expert skipping methods don't consistently outperform the simple static baseline at matched Keff.

Critical weaknesses

The taxonomy is useful but descriptive. A unified taxonomy for the four dynamic expert skipping methods is constructed based on the signal, but it doesn't help in predicting which method would succeed in which regime or explain their performance differences.

The differences between methods appear in a low Keff regime for text (Figure 2). However, it is of not much practical interest as the model is already broken in that region. Under a practical acceptable 5% quality drop, the differences between methods diminish.

The main focus of this study is to compare the dynamic expert skipping methods, but no results are shown on throughput analysis for dynamic expert skipping.

The practical relevance of reducing Keff is not sufficient. In MoE inference, autoregressive decoding can go memory-bandwidth-bound, reducing expert FLOPs doesn't necessarily improve latency.

The reason for collapse of generative benchmarks GSM8K and HumanEval at low Keff is not fully answered. Wonder if this indicates different Keff for prefill and decode.

Questions for authors
  • Could you provide the results (Keff) separately for prefill and decode?
  • Could you provide the throughput analysis with dynamic expert skipping?
  • Are the router scores renormalized after skipping?
  • Could you provide the guidance on how operating point should be selected and compare methods at that point?
5 comments
AuthorsOpenReview ↗

Author Response to Reviewer RzeQ (Part 1 of 2)

We thank Reviewer RzeQ for their careful review and recognizing the value of our negative study and the distinction between calibrated and measured KeffK_\text{eff}. We address each point below and have introduced further information in the revised version.

Throughput Analysis for Dynamic Expert Skipping: We deliberately did not report runtimes for dynamic methods: The core problem of throughput evaluation in a realistic scenario is that it heavily depends on implementation. A fair comparison would require highly optimized kernels for each method, which were not published by the original authors. Any difference in throughput/latency would be more reflective of implementation effort than actual performance differences. Also, static top-k\text{top-}k serves as an upper bound: at any KeffK_\text{eff} it has the lowest runtime as it has no skipping decision, uniform per-token compute and fixed tensor shapes that keep fused MoE kernels and CUDA graphs applicable without further kernel support. As such, we see no mechanism in which a method can beat static top-k at matched KeffK_\text{eff} because they all have at least some overhead. This means that the comparison is actually favorable to the dynamic methods. Nonetheless, we have run an evaluation on the unoptimized pytorch implementations on Qwen3 to have some estimate of each methods overhead:

MethodPrefill tok/sDecode ms/token bs=1Decode ms/token bs=64Routing overhead (ms/step)Active experts/token
Top-8 (full model)12,204117.4 ± 1.8457.3 ± 0.8–8.00
Softmax top-315,793122.1 ± 0.3256.1 ± 0.40 (ref.)3.00
NAEE13,848191.3 ± 0.9330.0 ± 1.8+11.93.03
DiEP13,835187.7 ± 1.0334.7 ± 1.7+9.22.98
MoDES14,444174.1 ± 1.0290.9 ± 0.8≈02.98
Allocator12,709216.5 ± 1.7341.5 ± 2.3+38.33.10

At bs=1 in this implementation, kernel launch overhead negates any benefit of the skipping methods (including static).

Prefill and Decode difference; generative benchmarks: We added a more detailed analysis of prefill and decode results to the appendix (Section A.11). Overall difference between the phases is small, different expert allocation between the phases is not the cause of generative collapse as the static baseline enforces identical allocation for both prefill and decode. The drop is likely due to compounding errors over long generations and strict formatting requirements during generation, but we believe explaining this collapse fully is out of scope for this comparison.

Operating point selection and quality loss: We agree that practical usage typically only accepts minor quality loss. Usually we would propose selecting a fixed quality tolerance and taking the smallest KeffK_\text{eff} at that point. We added a table in Section 4.2.1. which shows that at pre-defined, acceptable level of quality degradation at 5% none of the methods consistently outperform the baseline, with methods needing in some cases 2 experts more to reach the target. In the abstract of the paper we also highlight that significant differences happen mostly in a broken region without practical interest, as also pointed out by you. The table highlights the necessary KeffK_\text{eff} for reaching a 5% quality drop (lower KeffK_\text{eff} is better).

ModelMethodARC-CHellaSwagGSM8K
Gemma4-26B-A4Bsoftmax top-k4.13.05.1
Gemma4-26B-A4BNAEE3.93.15.2
Gemma4-26B-A4BMoDES5.85.37.3
Gemma4-26B-A4BDiEP–4.0–
Gemma4-26B-A4BLearned allocator5.94.76.6
-: quality level wasn't reached in sweep

Continued in the next comment

AuthorsOpenReview ↗

Author Response to Reviewer RzeQ (Part 2 of 2)

Memory-bound decoding: We fully agree that lower KeffK_\text{eff} does not translate one-to-one to lower latency in purely memory-bound decoding. As shown in our performance metrics, benefit heavily depends on the scenario used, with prefill heavy or compute bound scenarios benefiting more. This criticism however applies to the research field as a whole, not in particular to our work or the methods we benchmark. Applying expert skipping methods in memory-bound large batch-size scenarios is generally not meaningful. Nonetheless, to provide some insights, we expand our throughput analysis by evaluating real multimodal workloads (DocVQA) using vLLM and our proposed multimodal baseline highlighting concrete performance gains in prefill and decode due to continuous batching.

Operating Point (k_vision/k_text)Avg. AccuracyTTFT, bs=1ITL, bs=1Throughput, 32 concurrent
1, 882.9% (−3.5)494 ms (−7.8%)9.6 ms (−1.3%)261 tok/s (+10.1%)
2, 884.9% (−1.4)509 ms (−5.1%)9.7 ms (−0.4%)258 tok/s (+8.8%)
1, 6 (recommended)82.8% (−3.6)492 ms (−8.3%)9.1 ms (−6.9%)260 tok/s (+9.9%)
2, 684.7% (−1.6)501 ms (−6.7%)9.1 ms (−6.8%)258 tok/s (+8.9%)
8, 8 (native)86.3%536 ms9.7 ms237 tok/s

Runtime measured on one A100 on Qwen3-VL using vLLM, each point was repeated twice.

Renormalization: Routing scores are not renormalized after skipping, this is now documented in Appendix A.12 and is applied consitently across methods. Before the comparison we conducted experiments on Qwen3 and Gemma4 and determined that calculating the scores at the native k=8k=8 and then truncating them without normalization produces the best results for both models. Renormalization leads to dramatic losses in quality especially for Qwen3 as expert outputs are scaled out of distribution. We thus apply this convention for all static and dynamic methods consistently. The table below highlights this degradation for Qwen3 averaged over GSM8K, ARC-C and HellaSwag by giving the average accuracy with and without normalization. For Gemma no significant difference was observed.

k₀Avg Accuracy over (GSM8K, ARC-C, HellaSwag) without RenormalizationAvg Accuracy over(GSM8K, ARC-C, HellaSwag) with Renormalization
257.123.3
466.562.1
669.168.3
870.770.7

Taxonomy We agree that the taxonomy is descriptive. The taxonomy’s purpose is to organize the design space of the methods to provide insight into which signals they use to to decide how many experts a token requires. In general the expectation would be that increasing granularity would lead to improved performance compared to the baseline. This is refuted by our results, which shows that the additional signal does not aid performance and even hurts in some cases such as DiEP.

We hope these additions address the reviewer’s concerns, and would be grateful if Reviewer RzeQ would reconsider their rating in light of them.

Please read the authors' rebuttal and post your comments on your raised concerns.

AC

Comments on the paper

2 comments

Thank you for the constructive feedback. Based on this, we have made several changes to the manuscript:

  • Addition per-layer allocation methods LExI and FastMMoE with the goal of identifying whether the per-layer signal provides additional benefit over the static baseline.
  • Addition of GPT-OSS to verify transfer of results to models that natively select a different number of active experts (4).
  • Expansion of our runtime evaluation to cover multimodal use cases and showing the clear runtime benefit of our recommended operating point in a realistic serving scenario
  • Evaluation of the impact of decode and prefill phase on expert skipping
  • Clarification of the impact of renormalization in expert skipping and the sensitivity of Qwen3 to the application to renormalization.

The additional experiments are inline with existing experiments and thus further strengthen the central claim of our manuscript. We have highlighted improved sections in blue for convenience.

Code Release We have attached the code as supplementary material and will publish it on Github after the review process.

While preparing the rebuttal (prompted by Reviewer RzeQ’s question on the prefill/decode split), we identified a measurement artifact in our text-only evaluations: LM-Eval’s batched generation can shift Effective K. Multimodal evaluations used batch size 1 and are unaffected. We re-measured Effective K per sequence in control runs on all four text benchmarks. The effect is negligible on ARC-C, HellaSwag and GSM8K (at most 0.4%) and at most 10% on HumanEval, in both directions; averaged over the four benchmarks, the largest possible shift is 3%. The artifact does not affect accuracies, and none of our conclusions change. We document this in the appendix.

Dear Area Chair,

Thank you for handling our submission and for prompting the reviewers to respond to our rebuttal. As the discussion period closes, Reviewers akEk and RzeQ have respondend.

All three reviewers list the controlled comparison and its central finding among the strengths: at matched, measured KeffK_\text{eff}, dynamic expert-skipping methods do not consistently outperform a static top-kk baseline. The weaknesses raised ask for additional controls, runtime evidence and broader scope rather than disputing this core result. We addressed each with new experiments (marked in blue in the revision) and provide a detailed list at the end.

All new results are in line with our original conclusion and broaden the evidence for it. Reviewer akEk raised their rating after our response. Reviewer RzeQ maintained theirs. We believe their assessment undervalues the paper’s main contributions: 1) A fair, matched-compute comparison against a simple baseline (“Don’t Skip the Baseline”) is necessary and has been missing from the literature. 2) When applying this fair comparison in our paper we identify that existing methods do not systematically outperform even a simple baseline. We expand on this in our response to their comment.

We believe that this critical comparison, which shows that existing methods do not systematically outperform a simple baseline, is a valuable research contribution that helps the field progress.

Based on the comments of the reviewers we added the following:

  • Stronger static controls (4ZYj W1; akEk W1, Q3): We added per-layer static allocation, a natural extension of our baseline, by evaluating LExI and FastMMoE (App. A.9). Neither provides a consistent gain over uniform static top-kk.
  • Model coverage (4ZYj W2): To cover a model with native k≠8k \neq 8, we added GPT-OSS 20B (k=4k=4; App. A.10), a third model family. Again, no method systematically outperforms our baseline.
  • Runtime (RzeQ W3, W4; akEk W2, Q2): In a vLLM serving setup on Qwen3-VL, our per-modality static operating points give 9–10% higher throughput at 32 concurrent requests and 5–8% lower TTFT, at a cost of 1.4–3.6 accuracy points. We also report runtimes of our unoptimized PyTorch implementations of the dynamic methods on Qwen3, where all four are slower than static top-3 at matched KeffK_\text{eff}.
  • Practical operating regime (RzeQ W2, Q4): We added a comparison at a fixed 5% quality tolerance (Sec. 4.2.1). No dynamic method consistently needs fewer experts than static top-kk to reach it.
  • Further questions (RzeQ Q1, Q3; akEk Q1): We added detail on the prefill/decode split (App. A.11), on renormalization (App. A.12) and on reaching non-integer budgets with static top-kk (Sec. 4.2.1).
  • Code (akEk): The code is now attached as supplementary material.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.