- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose a plug-and-play method, called MAPLE, to allocate different numbers of active experts to different layers in MoE models. MAPLE first measures how sensitive each layer is to the number of active experts. It then uses a closed-form method to allocate a fixed expert budget across layers. The authors evaluate MAPLE on four MoE models, including DeepSeek-MoE-16B, DeepSeek-V2-Lite, Moonlight-16B-A3B, and OLMoE-1B-7B. Under a 75% expert budget, MAPLE consistently outperforms existing pruning methods. Experiments on GSM8K and BBH also show that MAPLE can reduce computation time while maintaining, or even improving, model accuracy.
Critical strengths
-
Comprehensive experiments. I appreciate the authors’ careful experimental design. They apply the expert allocation found on one task directly to four other tasks, without performing a new search for each task. The results show that the learned allocation can transfer across tasks, though with some performance loss. This provides useful evidence for the generality of MAPLE and makes the practical value of the proposed allocation strategy more convincing.
-
Good algorithm–system integration. Another major strength is that the paper goes beyond the algorithm and also addresses the system-level cost of heterogeneous expert allocation. The authors develop a dedicated Top-1 CUDA kernel for MAPLE and integrate it into both vLLM and SGLang. By optimizing both the expert allocation and the inference system, the paper achieves clear improvements in latency and throughput. This makes MAPLE more practical for real-world MoE deployment than methods that only reduce theoretical FLOPs.
Critical weaknesses
-
Limited evaluation of knowledge capabilities. I have some concerns about the benchmarks used in the paper. The current evaluation mostly uses simple classification tasks and basic reasoning datasets. However, MoE experts often store factual knowledge. Reducing active experts might hit a model's factual knowledge harder than its reasoning ability. To make the evaluation more convincing, the authors should test knowledge-heavy benchmarks like MMLU, GPQA, or MMMLU. It would also help to test different shot counts (e.g., zero-shot vs. few-shot), as prompting may change how expert allocation works. Without these tests, it is hard to tell if MAPLE works well on knowledge-intensive tasks.
-
Need evaluation on stronger and newer models. I also have concerns about model selection. The tested models are relatively small or older. The surprising finding that MAPLE beats the full model with fewer experts might just happen because these models are under-trained or have high redundancy. The authors should test MAPLE on stronger and newer models, such as Qwen3.5-Next-Flash or other state-of-the-art MoE models. This will show if MAPLE provides a general benefit or just takes advantage of redundancy in smaller models.
Questions for authors
-
Why can reducing the average number of active experts improve overall performance? In the experiments, MAPLE achieves higher average accuracy while using a smaller expert activation budget than the full-budget baseline. This result is somewhat counter-intuitive, since reducing the number of active experts would normally be expected to reduce model capacity and hurt performance. How do the authors explain this phenomenon?
-
How should a reasonable search set be determined? Table 8 shows that the expert allocation obtained by searching on different evaluation sets can lead to nearly a 4-point difference in average performance. This suggests that the final allocation may be sensitive to the choice of the search set. How should users choose a suitable search set in practice? More importantly, if the search set is not known in advance, how can we determine an allocation that is robust across different tasks and datasets?