- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the expansion of the expert union during tree-based speculative decoding for Mixture-of-Experts (MoE) models, which increases expert-weight memory traffic during verification. It proposes MoE-Spec, which aggregates routing probabilities across the draft tree, selects a budgeted subset of experts at each layer, and adjusts token-to-expert assignments accordingly. The expert budget controls the trade-off between task quality and throughput. Experiments cover three MoE architectures and five benchmarks, reporting quality-matched speedups at conservative budgets and quality–throughput trade-offs at more aggressive budgets.
Critical strengths
- The problem is clearly identified and supported by empirical analysis. The paper highlights the overhead caused by the growing union of activated experts during tree-based verification. Analyses of expert activation counts, draft-tree size, decoding speed, and runtime breakdown support this motivation.
- The method is simple and requires no additional training. Expert selection uses existing routing probabilities, with an explicit budget controlling the quality–throughput trade-off. The method integrates into the existing EAGLE verification pipeline. At conservative budgets, several configurations achieve higher throughput without reducing the reported task metrics relative to EAGLE.
Critical weaknesses
- Quality-matched efficiency comparisons against simple approximate inference baselines are missing. The authors report quality-matched speedups over EAGLE and compare the task quality of Static and Router selection in the ablation study. However, the experiments do not provide quality–throughput curves for simple alternatives such as EAGLE with a statically pruned expert set. Consequently, the additional benefit of dynamic, tree-level expert selection over these alternatives remains insufficiently established. End-to-end throughput should be compared at matched task quality under the same hardware and evaluation settings. Autoregressive decoding with expert pruning could serve as an additional baseline.
- The routing concentration analysis contains some discrepancies between the figure and the text. The caption of Figure 7 states that MBPP achieves 95% routing probability coverage at budget B=8, and the accompanying text claims that the top eight experts capture over 90% of routing probability mass for both MBPP and HumanEval. However, the corresponding bars indicate approximately 65%, respectively. Similarly, the CNN/DM bar at B=24 indicates approximately, below the stated 90% (also, the bar chart is not suitable for this situation where we need to judge the numbers). These discrepancies weaken the routing concentration analysis and its explanation of why small expert budgets are viable. The authors should clarify the coverage definition and aggregation procedure, and reconcile the plotted results with the caption and text.
Questions for authors
- Could the authors provide quality–throughput curves comparing the proposed method with EAGLE? Including autoregressive decoding with expert pruning would further help isolate the efficiency gains attributable to dynamic, tree-level expert selection.
- The author should make it clear about routing concentration analysis.
2 comments
Thanks for your clarification, and I will maintain my positive assessment.
Thank you for the detailed review.
MoE-Spec is faster than both simple baselines at similar or better accuracy (Table C1) because neither chooses experts for the tree we verify. We ran EAGLE-3 with reduced top-k and with a statically pruned expert set and applied both only during verification on the same tree (Appendix F of the revision). Reduced top-k decides for each token alone and its drops cost accuracy but save no memory traffic when other tokens in the tree still load the expert. Static pruning fixes one set in advance and fails on MATH500 and the code tasks.
Table C1 Speedup over EAGLE-3 at batch 1 on the same tree. Parentheses give the accuracy change in points.
Autoregressive decoding with half of the experts of each token stays slower than EAGLE-3 on the reasoning and code tasks and loses up to 8.8 points (Appendix F).
Appendix F also plots quality against speed for every setting we measured. Neither baseline exceeds 1.11x there on any task while MoE-Spec reaches 1.20x at its aggressive budget for an accuracy loss of about 1 point.
Thank you for spotting this issue. The bars of Figure 7 are correct but our caption and text mischaracterize them. We define coverage as the share of routing probability that the B experts with the most probability capture when we sum probability over the tokens of a draft tree. On code and math the top 8 of 64 experts capture nearly two thirds of the routing probability and the top 24 capture over 90%. On GSM8K and CNN/DM the top 8 capture about half and the top 24 capture over 85%. We replace the bars with a table of these values and correct the text in the revision (Appendix D).