MAPLE: MoE Adaptive Plug-and-play Layer-wise Expert allocation

Sparsely-activated Mixture-of-Experts (MoE) Transformers universally fix the same number of routed experts across all layers, a convention that ignores the well-documented heterogeneity in layer-wise redundancy. We demonstrate that this uniformity is suboptimal and propose MAPLE, a plug-and-play framework that reallocates the routed-expert budget heterogeneously across layers of any pretrained MoE LLM, pruning computation rather than parameters: no expert weight is modified, removed, or retrained. Our core contribution is a closed-form sensitivity-guided allocation: we probe each layer's response to variation in expert count, quantify sensitivity, and derive a budget assignment that directs capacity towards sensitive layers and absorbs reductions in redundant layers. This closed-form solution is further refined by a sensitivity-constrained genetic search that uses layer-wise sensitivity as a prior guide for exploration, yielding faster convergence and superior allocation quality. The allocation also creates an execution regime that uniform routing never encounters, which we exploit with a dedicated top-1 routing GPU kernel integrated into SGLang and vLLM. On four MoE models spanning different scales and architectures, MAPLE outperforms uniform and pruning-based baselines under a 75% total expert-activation budget. Notably, on DeepSeek-MoE-16B, MAPLE uses only 75% of the experts yet surpasses the original 100% expert-uniform baseline on ARC-E, ARC-C, and BoolQ, improving accuracy from 65.09 to 71.40, 48.49 to 51.50, and 80.03 to 82.38, respectively. These accuracy gains translate into measured deployment efficiency: implementing MAPLE in SGLang reduces single-GPU end-to-end serving latency by 32.2% and improves throughput by 47.4%. These results show that well-designed heterogeneous allocation can be more effective than simply activating more experts, making it a practical axis for improving MoE efficiency.