Less Can Be More: Learning Pruned Top-$K$ in SMoE-based LLMs

Sparse Mixture of Experts (SMoEs) typically retain the fixed Top-$K$ expert budget chosen during pretraining when deployed on downstream tasks. But is this computation budget always necessary or even optimal? Through an oracle analysis across nine reasoning and non-reasoning SMoE models, we find substantial redundancy in expert activation: smaller budgets can preserve correct predictions, while some inputs that fail under the native Top-$K$ become solvable when fewer experts are activated. Overall, the oracle reduces the average expert budget by **40.8%** while yielding a **26.8% relative accuracy improvement**. Motivated by these observations, we introduce $\text{LEPO (Learning Expert-Pruning Policies)}$, which adapts both which experts to favor and how many experts to execute while keeping pretrained model parameters frozen. LEPO combines lightweight expert-preference adaptation with grouped expert-budget policies and learns from the available context through guidance from the original model, without requiring task labels or oracle budget annotations. Across test-time adaptation, offline budget learning, and permanent expert pruning, LEPO consistently improves the accuracy and efficiency trade-off over fixed and adaptive expert-allocation baselines. In the settings, LEPO uses only **75-87%** of the native routed-expert budget while largely preserving or improving accuracy. Our results suggest that downstream Top-$K$ should be treated as an adaptive computation-allocation problem rather than a fixed consequence of pretraining.