- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 2
Read review
Paper summary
The paper proposes and investigates the questions on whether dynamic expert-skipping strategy outperform simply reducing the number of activated experts in pretrained MoE models. It compares various methods with static top-k baselines through metric, a number measuring the averaged number of activated experts. Experiments cover language, vision language with a small extension in video models. The main conclusion from the experiments is that dynamic methods rarely outperform a matched static baseline consistently. Another interesting observation from multimodal setting is that different modality leads to very different performance when doing expert skipping.
Critical strengths
-
The study addresses an important comparison problem. The problem of systematically comparing dymanic expert skipping with static top-k is timely proposed. Measuring expert usage during evaluation with proposed metric provides a fair comparison. The finding that modality-aware static routing closes much of the performance gap could meaningfully improve evaluation protocol in this area.
-
The empirical results are comprehensive The paper includes multiple architectures, classification and generative tasks, calibration sensitivity, and held-out transfer checks. Appendix A.7 also validates the reimplementations against published results. These efforts strengthen the central hypothesis and make the paper more convincing. However, the code has not been submitted.
Critical weaknesses
-
The experiments do not clearly shows where the compute budget should be allocated. A dynamic method can allocate different numbers of experts across layers, modalities, and individual tokens. The proposed baseline mainly distinguishes the number of experts between vision and text. Therefore, the comparison cannot fully explain which components where the dynamic weighting helps and which adds unnecessary complexity. It would be interesting to see a more in-depth analysis in this direction, so the pratical recommendation would be more helpful.
-
The paper does not directly show the runtime/accuracy trade-off. The main comparisons indirectly show this trade-off through the proposed metric . This is useful for comparing expert computation, but the runtimes of methods still may differ. Figure 1 and Appendix A.5 measures only the static baseline, so it does not fully show the trade-off in different settinfs. A direct comparison of runtime and accuracy would make the practical recommendation more convincing.
Questions for authors
-
Comparing noninteger expert budgets. How should we choose between dynamic and static routing? For example, if a dynamic method averages experts per token, achieves higher accuracy than static , and runs faster than static , would you recommend the dynamic method?
-
Direct runtime/accuracy comparison. Could you provide a table comparing the accuracy and measured runtime of static and dynamic skipping directly without the metric?
-
Identifying where to allocate computation. Could you compare other different allocation strategies, such as fixed number of experts in each layer, at matched measured expert budgets? This could help the recommendation more diverse than only changing the number of activated experts in different modalities.
3 comments
Please read the authors' rebuttal and post your comments on your raised concerns.
AC
Thanks for rebuttal
Thanks the author for your rebuttal.
After reading the rebuttal to other reviewers as well as the updated manuscript, most of my concerns have been resolved. I have no more questions and decided to raise the score to weak accept.
We thank Reviewer akEk for their insightful review. We have addressed each point below and have included further insights into how to allocate expert compute based on which signal.
Question 3 / Weakness 1 Allocation of computation: We consider the allocation of compute to experts on three granularity levels: Modality, Layers and Tokens.
The tables show the new results for the layer granularities:
Question 1 Comparing noninteger expert budget: We do not recommend the dynamic methods in this case: the static methods can be used to on average achieve the desired noninteger operating point by, for instance, alternating between the higher and lower static k per request to on average approach the desired quality level and performance characteristics. The expectation of this approach matches the targeted Keff and accuracy. This means that any point on the frontier can be achieved using this simple scheme. We elaborate on this in the revised Section 4.2.1.
Question 2 / Weakness 2 Direct runtime/accuracy comparison: We have added a clear runtime / accuracy table for our recommended static operating points on our per-modality trade-off curve. For the dynamic skipping methods we deliberately didn't provide realistic runtime figures. Any fair comparison would require highly optimized kernels, which are not provided by any of the authors, and any difference in performance would be more reflective of the amount of optimization effort rather than actual performance. Static top-k serves as a bound. At any Keff it has the lowest runtime as it has no routing decision, uniform per-token compute and fixed tensor shapes that keep fused MoE kernels and CUDA graphs applicable without further kernel support. As such, a method that doesn’t beat the static top-k baseline at matched Keff cannot beat it at matched runtime. This means that the comparison is favorable to the dynamic methods. Nonetheless, we have run a evaluation on the unoptimized pytorch implementations on Qwen3 and have included them to have some estimate.
Runtime for recommended Multimodal Operating Points:
Methods comparison (unoptimized torch) for k≈3 on Qwen3:
*interpolated. Avg. Accuracy over GSM8K, ARC-C, HumanEval, HellaSwag.
At bs=1 in this implementation, kernel launch overhead negates any benefit of the skipping methods (including static).
Code Release We have attached our code as a supplementary material on open review
We hope this addresses the reviewer’s concerns and would be grateful if the reviewer were to reconsider their rating in light of these additions.