- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper explores the sparsity potential by discovering the inefficiency of Top-k routing, which assigns a fixed k for every layer and every token. Instead, they propose an expert masking to dynamically allocate budget, and use self-distillation to train the representations. They also provide a hardware-efficient implementation of this method.
Critical strengths
Table 2 does seem to suggest that BEAM is promising to retain performance, and Section 5.4 suggests that it is also possible to implement that efficiently.
Critical weaknesses
Many parts are not articulated well: For section 3.1, it is unclear what does direct truncation mean, does that reducing Top-8 to Top-4, also there is no mentioning of what does router-only adaptation mean.
For Figure 1, what does frequency mean, do you mean the k-th token frequency, or probability of the routing score. It is currently hard to understand, also the authors claim that "a lower-ranked expert can be critical and a highly ranked expert locally redundant", could you point to the evidence written in the paper? It would be much clearer if more information could be provided in captions of Figure 1, 2.
There's not enough information provided in caption of Figure 4 to understand the framework, why do you use sg (stop gradient) in eq.(3)? This does not seem to be defined and justified in text.
Questions for authors
- Line 86, "K0" is not defined.
- Did you use layerwise self-distillation because changing top-k affect the intermediate representation. I would be curious about the performance if no self-distillation is done, and a comparison of self-distillation and training from scratch. Because this seems to change the representation, if it is chanegd a lot, then it is unclear for self-distillation to have a huge advantage over training.
- Is there any possible explanation on why BEAM+OPSD could be better than original model in GPQA-D?
- It is unclear to me about the connection of sensitivity to BEAM. Why do you list both prefill and decode in Fig 7? Fig 7(b) looks a bit noisy and it is unclear to me how "layers that are more sensitive to truncation retain more computation" is jusitfied.
- Related work should be discussed more thoroughly and explain the intuition on why BEAM is better than prior methods. (If it is too long, it can be put into Appendix).
6 comments
Response to Reviewer UvWF [Part 2/2]
U6. Why BEAM+OPSD can outperform Native on GPQA-D
The GPQA-D improvement follows a systematic paired correction pattern rather than longer responses or benchmark overlap. BEAM+OPSD corrects substantially more errors than it introduces relative to both Native and base BEAM. The improvement over base BEAM is statistically significant, and BEAM+OPSD also produces shorter responses than ZEDA on this task.
| GPQA-D comparison | Newly correct | Newly incorrect | Net corrections | Exact paired-test |
|---|---|---|---|---|
| BEAM+OPSD vs Native | 19 | 9 | +10 | 0.087 |
| BEAM+OPSD vs base BEAM | 27 | 11 | +16 | 0.014 |
| Comparison with ZEDA | Accuracy improvement | Mean response-length change |
|---|---|---|
| BEAM+OPSD vs ZEDA | +8.59 points | 769 fewer tokens |
No AIME24 or GPQA-D item overlaps the OPSD prompt set under our overlap checks. We will add the paired statistics and representative corrected cases to the analysis appendix.
U7. Connection between sensitivity and BEAM's learned allocation
The expanded analysis confirms that BEAM learns to preserve computation in layers that are difficult to prune. We repeated the static sensitivity analysis on 64 independently selected sequences containing more than 240K tokens, and compared it with BEAM's learned decode allocation.
| Static pruning setting | Sensitivity vs learned pruning | Interpretation |
|---|---|---|
| Mild pruning, K=6 | −0.63 | Highly sensitive layers are pruned substantially less |
| Target budget, K=4 | −0.47 | The correspondence remains clear at BEAM's main operating point |
| Aggressive pruning, K=2 | −0.39 | The same relationship persists under stronger pruning |
| 96 generations from 8 domains | 32.8%–53.4% pruning | BEAM also adapts its budget to the input |
The reported values are layer-wise rank correlations. A negative value means that a layer with larger output deviation under static expert removal receives a lower pruning rate from BEAM. At the target budget, the correlation remains between −0.53 and −0.42 across resampled sequence sets, showing that the relationship is not driven by a few examples.
We show prefill and decode separately because BEAM learns different allocations for the two phases. The quantitative sensitivity comparison is measured on decode, where dynamic expert execution dominates generation. We will replace the original noisy case-level interpretation with this aggregate analysis and clarify the correspondence directly in Figure 7 and its caption.
U8. Positioning relative to prior dynamic MoE methods
BEAM jointly learns candidate identity and per-token budget while freezing the entire original model, then converts the resulting irregular masks into measured serving gains with a deployable kernel. This differs from training-free truncation, prefix/count adaptation, and methods that retrain the full MoE.
We will reorganize Related Work around what each method learns and which parameters it updates, making BEAM's distinction from Ada-K, ZEDA, and training-free expert-skipping methods explicit.
We sincerely thank the reviewer again for the detailed questions, which helped us identify several places where the original presentation was too compressed. In response, we clarified the routing definitions and notation, added the self-distillation ablation, analyzed the GPQA-D improvement with paired examples, expanded the sensitivity study to substantially more sequences and domains, and sharpened BEAM's positioning relative to prior work. We believe these revisions and new analyses directly address the reviewer's concerns about clarity and empirical support. If you find these concerns adequately resolved, we would be very grateful if you could reconsider your score.
In U2, I am still confused by "Frequency is the fraction of evaluated contexts in which removing an expert at a given native rank causes the largest or smallest increase in next-token KL divergence", what does evaluated contexts mean? And why frequency is defined like that, I thought it is referring to how frequent a token appears.
In terms of U8, can you provide a more thorough discussions on how prior works do, for example you could cite some related works, especially on training-free methods. And why your work is better.
Also, you could edit the manuscript too based on your stated changes, would be great if you could mark your changed parts in red.
Response to Reviewer UvWF [Part 1/2]
We thank the reviewer UvWF for highlighting where BEAM's definitions, figures, and analysis-to-method connection should be made clearer. The new evidence shows that BEAM learns a structured candidate-wise sparse policy rather than merely reducing the number of active experts. We will revise the presentation so that this mechanism and its empirical support are immediately visible. Here are our responses:
U1. Direct truncation and router-only adaptation
Direct truncation changes native Top-8 execution to Top-4 by retaining the first four experts from the original router ranking, without any training. Router-only adaptation also uses a four-expert budget, but updates the router so that it can relearn expert selection and mixture scores while every expert and backbone parameter remains frozen. We will define both terms at first use and revise the caption of Table 1 accordingly.
U2. Meaning of the frequency in Figure 1
**Frequency is the fraction of evaluated contexts in which removing an expert at a given native rank causes the largest or smallest increase in next-token KL divergence. Rank 1 is the largest-impact removal in only 28.0% of contexts, while one of ranks 5–8 has the largest impact in 29.3%. Native rank is therefore informative on average, but it does not define a universally optimal deletion order. BEAM's learned masks support the same conclusion: 66.4% of generation-time token–layer decisions retain a non-prefix subset.
We will state the definition directly in the text and expand the captions of Figures 1 and 2 so the statistic can be understood without referring to the appendix.
U3/U4. Stop-gradient and K0
K0 denotes the number of candidates selected by the native router, and stop-gradient keeps this native routing prior fixed while only the mask gates are optimized. This prevents mask training from silently changing the teacher routing distribution. We will define both symbols locally, justify the stop-gradient operation, and make the training path explicit in Figure 4.
U5. Role of layer-wise self-distillation
Layer-wise self-distillation protects reasoning and code capabilities after the expert subset changes. At the same expert budget, removing the hidden-state constraint produces clear drops on AIME24 and HumanEval+. This supports BEAM's lightweight post-training design: the expert knowledge is retained, while intermediate representations require alignment after routing changes.
Training a 30B MoE from scratch is not a matched alternative to BEAM: our setting assumes an already trained model and asks whether its existing expert knowledge can be used with less computation through lightweight routing adaptation. From-scratch training would change both the problem definition and the resource cost by orders of magnitude. We will make this scope explicit and add the no-layer-wise-KD comparison to Section 5.3.