acceptodds
Under review as a conference paper at ICLR 2027

Rethinking MoE Routing: Unlocking the Sparsity Potential of Pretrained Models

Abstract

Mixture-of-experts (MoE) models expand capacity through sparse activation and are widely used in large language models. However, fixed Top- routing assigns every token the same expert budget, creating redundant computation. Training-free skipping based on routing scores or importance estimates can compromise quality under aggressive pruning, while jointly adapting routers and experts requires optimizing a large pretrained parameter set. Our analysis reveals that pretrained MoEs can retain their capabilities under smaller expert budgets by relearning expert selection, while adaptive budgets across tokens and layers further reduce computation without sacrificing quality. Guided by these findings, we develop BEAM (Binary Expert Activation Masking), which combines hidden states with a routing prior to select which candidate experts execute. Under a shared sparsity objective, mask routers distribute a global expert budget across tokens and layers, while layer-wise self-distillation preserves the original model's representations and predictions with pretrained weights frozen. The learned policy exhibits dynamic sparsity, retaining computation for sensitive tokens and layers while pruning redundant activations elsewhere. On Qwen3-30B-A3B, BEAM preserves over 99% of the original model's performance while reducing expert activations by 50%. To deploy dynamic MoEs efficiently, we optimize BEAM kernels for high-performance inference engines. With our kernels integrated into vLLM, BEAM achieves up to throughput and TPOT speedups over the original MoE.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture of Expert, Inference Acceleration, Sparse Model, Self Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UvWFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores the sparsity potential by discovering the inefficiency of Top-k routing, which assigns a fixed k for every layer and every token. Instead, they propose an expert masking to dynamically allocate budget, and use self-distillation to train the representations. They also provide a hardware-efficient implementation of this method.

Critical strengths

Table 2 does seem to suggest that BEAM is promising to retain performance, and Section 5.4 suggests that it is also possible to implement that efficiently.

Critical weaknesses

Many parts are not articulated well: For section 3.1, it is unclear what does direct truncation mean, does that reducing Top-8 to Top-4, also there is no mentioning of what does router-only adaptation mean.

For Figure 1, what does frequency mean, do you mean the k-th token frequency, or probability of the routing score. It is currently hard to understand, also the authors claim that "a lower-ranked expert can be critical and a highly ranked expert locally redundant", could you point to the evidence written in the paper? It would be much clearer if more information could be provided in captions of Figure 1, 2.

There's not enough information provided in caption of Figure 4 to understand the framework, why do you use sg (stop gradient) in eq.(3)? This does not seem to be defined and justified in text.

Questions for authors
  1. Line 86, "K0" is not defined.
  2. Did you use layerwise self-distillation because changing top-k affect the intermediate representation. I would be curious about the performance if no self-distillation is done, and a comparison of self-distillation and training from scratch. Because this seems to change the representation, if it is chanegd a lot, then it is unclear for self-distillation to have a huge advantage over training.
  3. Is there any possible explanation on why BEAM+OPSD could be better than original model in GPQA-D?
  4. It is unclear to me about the connection of sensitivity to BEAM. Why do you list both prefill and decode in Fig 7? Fig 7(b) looks a bit noisy and it is unclear to me how "layers that are more sensitive to truncation retain more computation" is jusitfied.
  5. Related work should be discussed more thoroughly and explain the intuition on why BEAM is better than prior methods. (If it is too long, it can be put into Appendix).
6 comments
AuthorsOpenReview ↗

Response to Reviewer UvWF [Part 1/2]

We thank the reviewer UvWF for highlighting where BEAM's definitions, figures, and analysis-to-method connection should be made clearer. The new evidence shows that BEAM learns a structured candidate-wise sparse policy rather than merely reducing the number of active experts. We will revise the presentation so that this mechanism and its empirical support are immediately visible. Here are our responses:


U1. Direct truncation and router-only adaptation

Direct truncation changes native Top-8 execution to Top-4 by retaining the first four experts from the original router ranking, without any training. Router-only adaptation also uses a four-expert budget, but updates the router so that it can relearn expert selection and mixture scores while every expert and backbone parameter remains frozen. We will define both terms at first use and revise the caption of Table 1 accordingly.

U2. Meaning of the frequency in Figure 1

**Frequency is the fraction of evaluated contexts in which removing an expert at a given native rank causes the largest or smallest increase in next-token KL divergence. Rank 1 is the largest-impact removal in only 28.0% of contexts, while one of ranks 5–8 has the largest impact in 29.3%. Native rank is therefore informative on average, but it does not define a universally optimal deletion order. BEAM's learned masks support the same conclusion: 66.4% of generation-time token–layer decisions retain a non-prefix subset.

Removal eventFrequency
Rank-1 expert is the most important28.0%
One of ranks 5--8 is the most important29.3%
Learned generation masks that are non-prefix66.4%

We will state the definition directly in the text and expand the captions of Figures 1 and 2 so the statistic can be understood without referring to the appendix.

U3/U4. Stop-gradient and K0K_0

K0K_0 denotes the number of candidates selected by the native router, and stop-gradient keeps this native routing prior fixed while only the mask gates are optimized. This prevents mask training from silently changing the teacher routing distribution. We will define both symbols locally, justify the stop-gradient operation, and make the training path explicit in Figure 4.

U5. Role of layer-wise self-distillation

Layer-wise self-distillation protects reasoning and code capabilities after the expert subset changes. At the same expert budget, removing the hidden-state constraint produces clear drops on AIME24 and HumanEval+. This supports BEAM's lightweight post-training design: the expert knowledge is retained, while intermediate representations require alignment after routing changes.

VariantAvgKAIME24HumanEval+Nine-task Avg.
BEAM4.0376.2590.2478.45
w/o layer-wise KD4.0073.3387.2078.21
Difference+0.03+2.92+3.04+0.24

Training a 30B MoE from scratch is not a matched alternative to BEAM: our setting assumes an already trained model and asks whether its existing expert knowledge can be used with less computation through lightweight routing adaptation. From-scratch training would change both the problem definition and the resource cost by orders of magnitude. We will make this scope explicit and add the no-layer-wise-KD comparison to Section 5.3.

AuthorsOpenReview ↗

Response to Reviewer UvWF [Part 2/2]

U6. Why BEAM+OPSD can outperform Native on GPQA-D

The GPQA-D improvement follows a systematic paired correction pattern rather than longer responses or benchmark overlap. BEAM+OPSD corrects substantially more errors than it introduces relative to both Native and base BEAM. The improvement over base BEAM is statistically significant, and BEAM+OPSD also produces shorter responses than ZEDA on this task.

GPQA-D comparisonNewly correctNewly incorrectNet correctionsExact paired-test pp
BEAM+OPSD vs Native199+100.087
BEAM+OPSD vs base BEAM2711+160.014
Comparison with ZEDAAccuracy improvementMean response-length change
BEAM+OPSD vs ZEDA+8.59 points769 fewer tokens

No AIME24 or GPQA-D item overlaps the OPSD prompt set under our overlap checks. We will add the paired statistics and representative corrected cases to the analysis appendix.

U7. Connection between sensitivity and BEAM's learned allocation

The expanded analysis confirms that BEAM learns to preserve computation in layers that are difficult to prune. We repeated the static sensitivity analysis on 64 independently selected sequences containing more than 240K tokens, and compared it with BEAM's learned decode allocation.

Static pruning settingSensitivity vs learned pruningInterpretation
Mild pruning, K=6−0.63Highly sensitive layers are pruned substantially less
Target budget, K=4−0.47The correspondence remains clear at BEAM's main operating point
Aggressive pruning, K=2−0.39The same relationship persists under stronger pruning
96 generations from 8 domains32.8%–53.4% pruningBEAM also adapts its budget to the input

The reported values are layer-wise rank correlations. A negative value means that a layer with larger output deviation under static expert removal receives a lower pruning rate from BEAM. At the target budget, the correlation remains between −0.53 and −0.42 across resampled sequence sets, showing that the relationship is not driven by a few examples.

We show prefill and decode separately because BEAM learns different allocations for the two phases. The quantitative sensitivity comparison is measured on decode, where dynamic expert execution dominates generation. We will replace the original noisy case-level interpretation with this aggregate analysis and clarify the correspondence directly in Figure 7 and its caption.

U8. Positioning relative to prior dynamic MoE methods

BEAM jointly learns candidate identity and per-token budget while freezing the entire original model, then converts the resulting irregular masks into measured serving gains with a deployable kernel. This differs from training-free truncation, prefix/count adaptation, and methods that retrain the full MoE.

We will reorganize Related Work around what each method learns and which parameters it updates, making BEAM's distinction from Ada-K, ZEDA, and training-free expert-skipping methods explicit.


We sincerely thank the reviewer again for the detailed questions, which helped us identify several places where the original presentation was too compressed. In response, we clarified the routing definitions and notation, added the self-distillation ablation, analyzed the GPQA-D improvement with paired examples, expanded the sensitivity study to substantially more sequences and domains, and sharpened BEAM's positioning relative to prior work. We believe these revisions and new analyses directly address the reviewer's concerns about clarity and empirical support. If you find these concerns adequately resolved, we would be very grateful if you could reconsider your score.

In U2, I am still confused by "Frequency is the fraction of evaluated contexts in which removing an expert at a given native rank causes the largest or smallest increase in next-token KL divergence", what does evaluated contexts mean? And why frequency is defined like that, I thought it is referring to how frequent a token appears.

In terms of U8, can you provide a more thorough discussions on how prior works do, for example you could cite some related works, especially on training-free methods. And why your work is better.

Also, you could edit the manuscript too based on your stated changes, would be great if you could mark your changed parts in red.

Reviewer_e4b2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper questions the use of a fixed top-k expert selection budget for every token and layer in MoE models and shows that MoE's tolerate dynamic dropping of experts conditioned of the input. They first present an insightful analysis of expert removal impact on model performance, layer sensitivity, and token sensitivity, demonstrating that various tokens and layers can have different levels of token pruning tolerance. Inspired by that, they propose a linear mask gate that combines the token hidden state with the original router's probabilities, to decide which of the original top-k selected experts can be pruned. They additionally develop an inference kernel to translate irregular masks into operator savings. The authors applied BEAM to two MoE models, and while the method itself is not particularly novel, the thorough empirical evaluation and the insightful ablations make it a meaningful contribution.

Critical strengths
  • The paper reads and motivates the method well and via analysis shows the original ranking of experts may not be the only determining factor for retaining or pruning the experts.
  • The approach is simple and easy to use. Training linear mask gates via straight through estimation and a sparsity objective is straightforward.
  • The authors develop an MoE kernel to realize practical gains.
  • The ablations for the components of the method are thorough and the analyses on token category and layer allocation, and the pruned ranks are insightful.
Critical weaknesses

My major concern relates to the novelty of the proposed method and its significance compared to prior works.

  • The idea of allocating nonuniform token/layer budgets post-training is already proposed in prior work such as Ada-K Routing [Tongtian Yue et al. ICLR'25], in which a similar frozen-backbone premise was used. The core difference is in discrete optimization for learning the masks: Ada-k uses PPO, while Beam uses the simpler straight-through estimator. Apart from the discrete optimization angle, Ada-K predicts a categorical distribution over expert counts and can potentially use more experts for difficult tasks. Beam in contrast learns a binary mask for the pruning of the already selected experts without necessarily following the rank order.

  • Given the results in Table 2, it would be valuable to additionally report where the gain comes from compared to Ada-K. Is it the flexibility of pruning experts at any rank or is it the simpler straight-through optimization which provides the gains.

  • While pruning reduces the average cost of producing tokens, the efficiency evaluation does not capture possible changes in generation length caused by expert pruning. If as a result of expert pruning, the sparse model generates longer or more repetitive outputs, the efficiency gains will be impacted. The authors should compare response-length distributions for various methods in Table 2.

Questions for authors
  • The authors should provide more details on how they produced the numbers in Table 2 for the competing algorithms, including their training details, datasets, etc.
3 comments
AuthorsOpenReview ↗

Response to Reviewer e4b2

We thank the reviewer e4b2 for recognizing BEAM's routing analysis, thorough ablations, and deployable kernel implementation. BEAM goes beyond predicting how many experts to activate: it learns which candidates should be retained while freezing the entire original model. Our new matched control directly isolates this distinction from differences in training recipe or compute budget.


E1/E2. Source of the improvement over Ada-K

The gain comes from candidate-wise expert selection rather than additional training resources. Prefix-BEAM uses the same frozen model, data, losses, optimizer, training length, seed, and expert budget as BEAM; the only difference is that it must retain a prefix of the native ranking. At nearly identical AvgK, removing this prefix constraint improves aggregate retention and produces clear gains on reasoning, code, and instruction-following tasks. Moreover, 66.4% of BEAM's generation-time masks are non-prefix subsets, showing that the model actively uses this additional flexibility.

TaskPrefix-BEAMCandidate-wise BEAMImprovement
MATH50094.2093.60-0.60
AIME24@875.0076.25+1.25
GPQA-D57.5860.10+2.52
MMLU-R89.4489.29-0.15
GSM8K94.6294.31-0.31
Eval+88.4190.24+1.83
MBPP+79.1078.57-0.53
IFEval82.2683.36+1.10
IFBench39.0040.33+1.33
Native-normalized retention96.78%97.93%+1.15 points
Final training AvgK4.024.03+0.01

We will add this matched control to Section 5.2 and clarify that BEAM's central difference from Ada-K is learning candidate identity together with the expert budget, rather than only predicting a count along the native ranking.

E3. Response length, repetition, and total computation

Longer responses do not erase BEAM's computational savings. We audited all nine Qwen3 evaluation tasks, covering 9,159 matched requests per method. Although BEAM generates moderately longer responses than Native on average, counting every generated token still shows a substantial reduction in routed expert computation. BEAM is also shorter than Ada-K on six of nine tasks and less repetitive on five tasks.

MethodMean tokensAvgKExpert–token work vs NativeReduction
Native2130.68.001.000x0.0%
BEAM2414.34.030.571x42.9%
BEAM+OPSD2416.63.950.560x44.0%
Ada-K2475.64.020.584x41.6%
ZEDA2336.93.970.544x45.6%

The quality improvements are not explained by globally longer generation. OPSD improves aggregate retention with almost no average length change, while on GPQA-D it improves accuracy over ZEDA and produces substantially shorter responses.

ComparisonQuality changeMean response-length change
BEAM+OPSD vs ZEDA on GPQA-D+8.59 points769 fewer tokens
BEAM vs ZEDA on MMLU-R+0.14 points40 fewer tokens
BEAM+OPSD vs BEAM, aggregate+1.36 retention points+2.4 tokens

We will add the task-wise response-length distributions, repetition statistics, and completion-aware expert-token computation to Section 5.4 and the appendix.

E4. Baseline evaluation protocol

The released-checkpoint comparison and Prefix-BEAM answer complementary fairness questions. Dynamic-MoE is training-free and is applied directly to the native router scores using its published Top-p rule. Ada-K and ZEDA are evaluated from their released checkpoints with their prescribed inference settings. All methods use the same benchmark prompts, decoding configuration, evaluation harness, and scoring code.

We will explicitly identify the released baseline artifacts and separate the official-checkpoint comparison from the matched causal control in the revised experimental setup.


We sincerely thank the reviewer again for identifying the central novelty question. The matched Prefix-BEAM control shows that candidate-wise selection contributes a measurable gain beyond count/prefix routing, while the complete output audit confirms that this flexibility preserves substantial computation savings after response length is included. We will incorporate these results and clarify the baseline protocols in the revision.

Dear Reviewer e4b2,

Thank you again for your thoughtful review and for recognizing the value of BEAM's routing analysis, ablations, and deployable kernel. As the discussion period is approaching its end (November 18 at 11:59 p.m. AoE), we wanted to kindly follow up on our response.

We believe that we have carefully addressed your questions with additional evidence and corresponding revisions. For Novelty. We added a fully matched Prefix-BEAM control that keeps the model, data, objectives, training length, and expert budget fixed while changing only the allowed routing subset. For Optimization. We separately evaluated the no-STE control to distinguish expert-selection flexibility from discrete optimization. For Generation behavior. We audited response length and repetition across the complete nine-task evaluation and verified that completion-aware expert–token work remains substantially lower than Native. For Baseline reproducibility. We clarified the provenance and training settings of Ada-K and ZEDA and expanded the corresponding manuscript discussion. Relevant manuscript changes are highlighted in red for ease of review.

If you have any remaining questions about BEAM, our experiments, or the revisions, please do not hesitate to let us know. We would greatly appreciate any further feedback you may have before the discussion period closes. Thank you once again for your time and efforts.

Best regards,
The Authors

I would like to thank the authors for their response which has clarified the source of improvements compared to Ada-K, and for the additional response length study. I will keep my positive score for the paper.

Reviewer_aVftOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that fixed top-K routing in pretrained MoEs wastes computation and that the bottleneck to deeper sparsity is routing, not expert capacity. BEAM adds a linear mask gate per layer that reads the hidden state plus a centered prior from the frozen router's scores, and thresholds each candidate independently. Training uses offline trajectories from the original model, with hidden-state and output-distribution distillation at every layer. An optional on policy self distillation stage trains on the sparse model's own rollouts. Custom dispatch kernels in vLLM remove masked token–expert pairs before the expert GEMMs for efficiency.

On Qwen3-30B-A3B at about 4 of 8 experts, BEAM retains 97.93% of the native score across nine benchmarks, above Dynamic-MoE (89.06) and Ada-K (96.73) and close to ZEDA+OPSD (98.17). With OPSD, retention reaches 99.29%. On GLM-4.7-Flash, BEAM at K = 3 retains 99.33% benchmark performance. The method achieves efficiency gains of up to 1.29× throughput and 1.91× TPOT over native top-8.

Critical strengths

The method is simple and easily deployable. BEAM trains only a linear gate per layer on a frozen model in a few GPU-hours, and the authors integrate custom kernels into vLLM so that dynamic sparsity yields real throughput and latency gains.

The analysis shows that prefix truncation is the wrong pruning primitive. The single-removal analysis and the learned pruning show that useful pruning departs from the router's ranking. This motivates candidate-wise masks over count-based methods such as Ada-K and Dynamic-MoE, and BEAM outperforms both in Table 2.

Critical weaknesses

The abstract's ">99% retention" requires the optional OPSD stage (base BEAM is 97.93%, below ZEDA+OPSD at 98.17%). The OPSD checkpoint (step 4 of 16) was chosen using an AIME24@8 screen (Appendix A.3, Table 7), and AIME24 is one of the nine reported test benchmarks. The serving results use separately calibrated operating points from the quality results, so the 1.29× throughput is not demonstrated at the quality reported in Table 2. All results are single runs with no variance, on benchmarks as small as AIME24 (which has only 30 problems).

The motivating analyses are underexplored and partly inconsistent with the main results. Sec. 3.1 claims router-only adaptation is "comparable to full-parameter adaptation" (Table 1: 82.59 versus 81.95). However, in Table 2 the same two settings give 90.63% versus 93.50% retention, and even the native MATH score differs between the tables (94.80 versus 95.20). The layer and token sensitivity analyses (Figure 2) use one reasoning sequence and one prompt. Several ablation deltas in Tables 3–4 (−1.2% to −1.9%) are within plausible run-to-run variance.

Questions for authors

What are the quality results at the exact operating points used for the serving measurements, and what are the serving results at the operating point in Table 2?

What is BEAM+OPSD's score if the OPSD checkpoint is selected on a held-out validation set rather than AIME24? Alternatively, can you report all OPSD checkpoints on the full suite?

How many evaluation samples per problem are used for each benchmark (other than AIME24@8), and what is the seed-to-seed variance? Are the ablation differences in Tables 3–4 beyond that variance?

3 comments
AuthorsOpenReview ↗

Response to Reviewer aVft [Part 1/2]

We sincerely thank the reviewer aVft for recognizing BEAM's deployable design and routing insight. BEAM shows that a pretrained MoE already contains the knowledge required for sparse execution, the missing capability is learning which experts each token should use. At half of the native expert budget, BEAM achieves the strongest aggregate retention among the compared dynamic routers while leaving every original model weight unchanged.

Our new results clarify three central issues: the role of OPSD, the alignment between quality and serving measurements, and the reliability of the evaluation.

A1. Does BEAM depend on OPSD

No. Base BEAM is already a competitive standalone method: at approximately half of the native expert budget, it outperforms Dynamic-MoE on every reported task and Ada-K on average. On GLM-4.7-Flash, Base BEAM reaches 99.33% retention without OPSD.

OPSD serves a different purpose. Qwen3-30B-A3B has undergone extensive RL post-training, whereas Base BEAM learns from offline teacher trajectories. OPSD exposes the sparse policy to its own rollouts and helps recover reasoning behavior inherited from the RL-trained teacher. It raises Qwen retention from 97.93% to 99.29%, but is not required for BEAM to support dynamic sparse execution.

ModelMethodAvgKRetention
Qwen3-30B-A3BDynamic-MoE4.1989.06%
Ada-K4.0296.73%
ZEDA+OPSD3.9798.17%
Base BEAM4.0397.93%
BEAM+OPSD3.9599.29%
GLM-4.7-FlashBase BEAM, K=33.0099.33%
BEAM+OPSD, K=33.00100.25%

We will distinguish base BEAM from optional OPSD in the Abstract, Introduction, and Section 5.2.

A2. OPSD selection and leakage

Checkpoint selection uses training loss only, no benchmark score enters the rule. The selected checkpoint minimizes training loss. A conservative scan flags one transformed HumanEval+ seed, but that task is unchanged by OPSD and excluding it leaves the selected checkpoint strongest. The complete trajectory is reported below.

StepTraining OPSD lossTrain AvgKMATH500AIME24@8GPQA-DMMLU-RGSM8KEval+MBPP+IFEvalIFBenchRetention
20.11103.95194.4073.3359.0988.9395.0088.4179.1082.8142.6797.85%
4 (selected)0.10863.86594.4079.1768.1888.8794.0190.2476.9884.1039.0099.29%
80.13204.25094.4074.1758.5989.1894.6286.5978.0483.7340.3396.99%
160.12223.99094.4076.2560.1089.1694.7789.0278.3182.6242.0098.21%

The training-only rule selects the strongest full-suite checkpoint. Step 4 has both the lowest training loss and the highest full-suite retention, but only the former was used for checkpoint selection. We will correct the appendix, state the selection rule explicitly, and include the complete trajectory.

A3. Are the Speedups Measured on Quality-Preserving Models?

Yes. We separately benchmarked Base BEAM and BEAM+OPSD using the exact gate weights evaluated in Table 2. Both models achieve essentially identical serving speedups at nearly identical expert budgets, while OPSD improves quality retention from 97.93% to 99.29%. Thus, the acceleration is not obtained from a lower-quality sparse model, and the OPSD quality recovery introduces no measurable serving penalty.

Model usedRetentionWorkloadReal-token AvgKThroughputTPOT
BEAM97.93%Decode4.1101.19x1.18x
BEAM+OPSD99.29%Decode4.1181.19x1.18x
BEAM97.93%Mixed long context3.4351.16x1.15x
BEAM+OPSD99.29%Mixed long context3.4381.16x1.14x

All measurements use the same H800, saturated CUDA Graph execution, identical workloads and warmup, and three timing repetitions. The separately calibrated operating point reaches up to 1.29x throughput and will be reported only as an additional deployment result.

We will add both quality--speed rows to Section 5.4 and make clear that OPSD improves retained quality without reducing serving efficiency.

AuthorsOpenReview ↗

Response to Reviewer aVft [Part 2/2]

A4. Evaluation stability

Our AIME24 result is based on 240 scored generations, rather than a single 30-question pass. We evaluate each of the 30 problems using eight fixed decoding seeds and report the mean and standard deviation across the resulting eight complete runs.

AIME24@8 methodCorrect / 240MeanSeed-block SDSeed range
Native Top-819179.58%2.14 pp76.67--83.33%
BEAM18376.25%4.86 pp66.67--80.00%
BEAM+OPSD19079.17%4.63 pp73.33--86.67%
Ada-K17472.50%6.61 pp60.00--80.00%
ZEDA18577.08%3.75 pp73.33--83.33%

BEAM+OPSD therefore nearly matches Native across the eight runs and remains stronger than Ada-K and ZEDA. Paired prompt-level analysis over the five exact-answer benchmarks gives the following uncertainty ranges:

Paired comparisonAdvantage95% confidence range
BEAM+OPSD vs Ada-K+3.02 pp[+1.44, +4.65]
BEAM+OPSD vs ZEDA+1.80 pp[+0.27, +3.32]
BEAM vs soft/no-STE on AIME24@8+5.83 pp[+0.42, +12.08]

The other benchmarks use one fixed decoding seed per prompt in the submitted protocol. We will report the exact number of evaluated generations for every benchmark and clearly distinguish decoding-seed variation from training-seed variation.

A5. Tables 1 and 2

The two tables were evaluated in separate runs with different decoding seeds. We agree that reporting one run per table makes this confusing. We will reevaluate both comparisons with the same set of multiple decoding seeds and replace the single-run values with their averages in revised version.

A6. Sensitivity generality

The sensitivity result is not limited to the examples shown in the paper. We repeated the analysis on 64 independently selected sequences containing more than 240K tokens and compared it with BEAM's learned allocation on 96 generations from eight domains.

Analysis settingObserved relationshipInterpretation
Mild pruning (K=6K=6)-0.63Sensitive layers are pruned less
Target budget (K=4K=4)-0.47Stable across different sampled sequences
Aggressive pruning (K=2K=2)-0.39The same behavior remains under stronger pruning
Eight input domains[32.8% ~ 53.4%]BEAM adapts its budget to the input domain

A negative correlation means that layers suffering larger changes under static expert removal receive more computation from BEAM. The sign remains negative across all resampled sequence sets. We will replace the original case-level interpretation with this aggregate result.

A7. Ablation stability

We agree that not every one-point aggregate difference should be interpreted as statistically significant. Our component conclusions instead rely on the larger task-level changes and on whether the intended compute budget is actually achieved.

AblationAvgKMain observed effect
Complete BEAM4.03Reference
w/o layer-wise KD4.00−2.92 on AIME24@8; −3.04 on HumanEval+
w/o STE4.00−5.83 on AIME24@8; −4.54 on GPQA-D
w/o sparsity objective7.83Returns close to the native expert budget

For STE, the paired AIME24@8 improvement has a 95% confidence interval of +0.42 to +12.08 points. The sparsity-objective ablation answers a different question: its higher quality is obtained by executing almost all eight experts and is therefore not a matched-compute improvement. We will soften claims based on small average differences and emphasize these mechanism-specific effects.


We sincerely thank the reviewer again for the careful reading and constructive feedback. The complete OPSD trajectory, overlap audit, AIME24@8 stability analysis, matched quality–speed measurements, and expanded sensitivity study directly address the concerns about checkpoint selection, evaluation reliability, and deployment validity. We hope these results and corresponding revisions resolve the main soundness concerns, and we would be grateful if the reviewer could reconsider the score.

AuthorsOpenReview ↗

We are looking forward to your reply

Dear Reviewer aVft,

Thank you again for your thorough review and valuable comments. As the discussion period is approaching its end (November 18 at 11:59 p.m. AoE), we wanted to kindly follow up on our response.

We believe that we have carefully addressed each of your concerns with additional evidence and corresponding manuscript revisions. In particular, we clarified that base BEAM does not depend on OPSD and that the OPSD checkpoint is selected using training loss rather than benchmark performance. We also verified that the quality-preserving policies reported in Table 2 achieve the stated serving acceleration, reported AIME24@8 stability and paired uncertainty analyses, explained the score differences between Tables 1 and 2, and expanded the sensitivity analysis to substantially more sequences, tokens, and domains. All corresponding changes have now been incorporated into the revised manuscript and are highlighted in red for ease of review.

If you have any remaining questions about BEAM, our experiments, or the revisions, please do not hesitate to let us know. If the new evidence and clarifications have resolved your concerns, we would be very grateful if you could reconsider your rating. Thank you once again for your time and efforts.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary: Response to All Reviewers

We would like to thank all reviewers for the time and effort they have invested in reviewing our paper. We are grateful for the constructive feedback, which has helped us improve the overall quality of our manuscript.


Collectively, the reviewers recognized the central strengths of BEAM: Routing insight (aVft, e4b2). The token-, layer-, and rank-level analyses challenge fixed Top-K routing and directly motivate the method. Simplicity and deployability (aVft, e4b2). BEAM freezes the original model and trains only lightweight mask gates. Strong empirical support (aVft, e4b2, UvWF). The reviewers highlighted its promising quality preservation, broad evaluation, and thorough ablations. Practical acceleration (aVft, e4b2, UvWF). The dynamic MoE kernels convert learned sparsity into measurable serving gains. Overall, BEAM reduces expert activation by 50% while preserving over 99% of the original model quality.

Below, we summarize the main points raised by each reviewer, as well as the revisions and clarifications we have made.


For Reviewer aVft

Main concerns and our actions:

  1. OPSD and checkpoint selection. We clarified that base BEAM is standalone, while optional OPSD further restores long-reasoning performance. Its checkpoint is selected solely by training loss, and we report the complete trajectory and overlap audit. 2. Quality--speed alignment. We benchmarked the exact quality checkpoints used in Table 2 and confirmed that these policies themselves deliver serving acceleration. 3. Evaluation reliability. We added AIME24@8 stability and paired uncertainty analyses, explained the differences between Tables 1 and 2, and expanded the sensitivity study to 64 sequences containing more than 240K tokens.

For Reviewer e4b2

Main concerns and our actions:

  1. Source of improvement. A fully matched Prefix-BEAM control isolates the benefit of learning which experts to retain rather than only predicting an expert count. 2. Role of optimization. A separate no-STE ablation distinguishes selection flexibility from the optimization method. 3. Generation behavior. A nine-task audit of response length and repetition confirms that expert--token computation remains substantially lower after counting every generated token. 4. Baseline reproducibility. We clarified the provenance and evaluation settings of Dynamic-MoE, Ada-K, and ZEDA.

For Reviewer UvWF

Main concerns and our actions:

  1. Rank analysis. We replaced the ambiguous term Frequency with precisely defined impact rates and rewrote the corresponding analysis and captions. 2. Method clarity. We clarified the notation, figures, stop-gradient operation, and training procedure. 3. Mechanism and generality. We added the requested self-distillation ablation and GPQA-D analysis, and directly connected the expanded sensitivity study to BEAM's learned layer-wise budgets. 4. Positioning. We expanded Related Work to clearly distinguish BEAM from training-free skipping, prefix/count routing, and full-model adaptation.

Final Remarks:

We have incorporated all reviewers' comments and suggestions into the revised manuscript, with the corresponding changes highlighted in red for ease of review. Reviewers e4b2 and UvWF both indicated that our responses and revisions resolved their main questions and provided positive feedback on the rebuttal.

Once again, our deepest thanks to all reviewers for your generous contributions to this collaborative peer-review process, and we wish you all the best in your future endeavors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.