acceptodds
Under review as a conference paper at ICLR 2027

Less Can Be More: Learning Pruned Top- in SMoE-based LLMs

Abstract

Sparse Mixture of Experts (SMoEs) typically retain the fixed Top- expert budget chosen during pretraining when deployed on downstream tasks. But is this computation budget always necessary or even optimal? Through an oracle analysis across nine reasoning and non-reasoning SMoE models, we find substantial redundancy in expert activation: smaller budgets can preserve correct predictions, while some inputs that fail under the native Top- become solvable when fewer experts are activated. Overall, the oracle reduces the average expert budget by **40.8%** while yielding a **26.8% relative accuracy improvement**. Motivated by these observations, we introduce , which adapts both which experts to favor and how many experts to execute while keeping pretrained model parameters frozen. LEPO combines lightweight expert-preference adaptation with grouped expert-budget policies and learns from the available context through guidance from the original model, without requiring task labels or oracle budget annotations. Across test-time adaptation, offline budget learning, and permanent expert pruning, LEPO consistently improves the accuracy and efficiency trade-off over fixed and adaptive expert-allocation baselines. In the settings, LEPO uses only **75-87%** of the native routed-expert budget while largely preserving or improving accuracy. Our results suggest that downstream Top- should be treated as an adaptive computation-allocation problem rather than a fixed consequence of pretraining.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Sparse Mixture of Experts, Pruned TopK, LLMs, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2LgyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper questioned the long standing problem of topk. it answers an interesting and important question that whether we can have better accuracy with less experts. it proposed to use KL from the teacher and hold the accuracy of the student and in the meantime reduce the number of experts.

Critical strengths
  1. It makes sense reducing the number of experts will not impact the scores in certain tasks which I believe.
  2. And it's clever to have a teacher model to help the student model
Critical weaknesses
  1. it's using teacher model and did test-time training so it's unclear to me where the gain coming from pruning or test time training (this is my main concern, if the authors can persuade me on this I can consider increase my score)
  2. the tables are missing the confidence interval, e.g. Human eval only has 164 problems 7.3% is just 12 problems.
Questions for authors
  1. is budget policy really useful? in table 4 with and without it doesn't seem very different.
  2. λ_o as a parameter do you need to tune it? how sensitive is it?
1 comment

no responses?

Reviewer_ij6VOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether sparse MoE language models need their pretrained Top-K expert budget downstream. SearchTopK, a label-using oracle over nine checkpoints, selects a successful reduced budget when one exists. This establishes an opportunity under oracle selection rather than the performance of an executable budget selector.

LEPO freezes pretrained weights and learns small expert-preference vectors plus token-dependent budget policies shared across groups of layers. It alternates teacher-guided preference adaptation with GRPO budget learning, using context prediction, output distillation and hidden-state matching. Evaluation covers request-local adaptation, offline learning followed by frozen evaluation, and preference calibration followed by STEP pruning.

Table 1 reports approximately 13–25% fewer routed experts during student execution with broadly preserved average accuracy. Table 2 gives promising offline results, particularly on Mixtral. Table 3 reports a 0.93-point improvement over STEP at 48 retained experts. The appendix explicitly reports substantial adaptation overhead; student expert savings do not establish lower total request cost.

Critical strengths
  1. Figure 1 and Appendix A.4 examine multiple checkpoints and distinguish label-using oracle selection, checkpoint diagnostics and learned-policy behavior. The appendix acknowledges generation-length effects, selection effects and the limited predictive value of local entropy/output geometry. The output-distillation reward ablation is also informative: removing it reduces accuracy from 40.5% to 30.5% on the 200-problem OLMoE/GSM8K study (Table 4), with the reported paired interval excluding zero.

  2. Table 2's offline Mixtral results give gains over the interpolated static baselines of 1.66 and 2.44 percentage points; these calculations reproduce from the table. Offline reuse avoids repeated request-specific optimization. Separately, LEPO preference calibration followed by STEP reaches 59.35% average accuracy versus STEP's reported 58.42% (Table 3). These results motivate further evaluation even though the attribution and cost comparisons need stronger controls.

Critical weaknesses

The paper specifies one greedy completion per item and exact-match accuracy, with 500 evaluated MATH500 problems. Such a score must be a multiple of 0.2 percentage points. LEPO's 27.86% for DeepSeek-V2-Lite and 11.18% for OLMoE cannot result from that protocol, including ordinary two-decimal rounding: they imply 139.3 and 55.9 correct answers. Similarly, Qwen's 53.25% HumanEval score does not correspond to an integer number of successes among 164 single-completion evaluations. Averaging multiple runs, using different denominators, or a table error could explain this, but the manuscript does not specify such an explanation. This observation establishes a reporting inconsistency; it does not establish its cause or imply misconduct.

Provide actual evaluated IDs and denominators, correct counts per run, numbers of runs/completions, the aggregation rule, evaluator versions, failed-generation handling and generation caps. Supply prediction/scoring records or an equivalent auditable summary, distinguish imported from rerun baselines, and reconcile the table and protocol descriptions. Correct averages and conclusions if necessary.

Questions for authors
  1. Please reconcile the MATH500 and HumanEval examples above with the stated item counts and one-completion protocol, and resolve the 512-versus-1,024-token MATH500 cap. Per-run success counts and scoring metadata could resolve the most immediate concern.

  2. A controlled comparison against that baseline and budget-only learning, including uncertainty on a generative task, would materially affect the recommendation. Please also clarify the offline training duration, budget-selection procedure, and any directly measured deployment cost after policy reuse.

Reviewer_XDXsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The proposed LEPO method freezes the pretrained backbone and learns additive expert-preference vectors together with budget policies shared across layer groups. Expert preferences use context prediction and teacher-matching losses, while budget policies are optimized through GRPO using expert cost and teacher fidelity. Experiments cover request-local adaptation, offline budget learning, and permanent expert pruning. Results indicate that student-generation expert counts can be reduced while largely preserving accuracy. However, request-local adaptation increases total computation and latency once teacher passes, rollouts, and optimization are included.

Critical strengths

S1: Adapting the downstream expert budget addresses a practical deployment problem. S2: The method introduces few trainable parameters and handles routing changes and KV-cache consistency explicitly. S3: Multiple deployment settings show promising results, and the appendix candidly reports limitations and negative findings.

Critical weaknesses

W1: Table 5 shows 2.70× native FLOPs and 1.78× latency, but these are scaled estimates and not direct measurements. W2: Table 2 reports a DeepSeek mean budget of 2.65, which is inconsistent with the documented minimum budget of 3. W3: LEPO spends more experts than Ada-K in every main comparison. Ablations do not clearly establish the benefit of input-dependent budgets over simpler fixed-budget adaptation. W4: Oracle gains depend on correctness-based selection and can shrink substantially after accounting for generation length.

Questions for authors

Q1: How is the reported budget of 2.65 possible under the stated action space? Q2: Can you demonstrate directly measured accuracy–latency improvements at matched budgets, including adaptation costs and amortization? Q3: Does adaptive budgeting outperform fixed-budget preference adaptation with identical teacher losses across models and seeds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.