- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes FairSpec, an expert selection method for speculative decoding for MoE models. Speculative decoding verifys multiple token at once, improving decoding throughput but also increasing memory pressure due to activating a larger number of experts. FairSpec selects experts according to token-wise experts ranking, providing theoretical guarantees to worst-token routing coverage. Experiments demonstrated improved success rate on several MoE models.
Critical strengths
- The paper addresses an important bottleneck when applying speculative decoding to MoE models. The observation of same sequence may have correlated experts preference is a reasonable motivation.
- The paper provides a simple but effective way for experts selection.
Critical weaknesses
- The most important concern would be experts masking shifts the original target distribution. Higher accepted length or throughput does not imply the generated distribution remains close to the original ones.
- The motivation emphasizes early-token rejection lead to a failure, but the proposed objective ignores token position.
- Experts overlapping isn't sufficiently validated, one single-batch requests is tested. Overlapping can also happens in larger batch serving scene, but the paper only discusses a specific situation that each requests corresponds to different domains.
Questions for authors
Q1: Can the authors quantify the distribution shift introduced by expert masking compared with the original MoE model?
Q2: Why does FairSpec not consider token positions in the allocation objective? How does FairSpec compare with acceptance-aware expert selection methods such as AcceptMoE?
Q3: How does expert overlap and FairSpec performance change under realistic continuous batching workloads?
2 comments
Thank you for adding the AcceptMoE comparison. My concern regarding Q1 and Q3 remain unresolved.
Q1. Distribution shift. The results do not answer my question about how much expert masking changes the output distribution. Adding experiments that compare FairSpec with full-expert speculative decoding using the same prefixes, draft candidates, and verification positions and report token-level KL would better explain the shift. As for the greedy decoding, you can report how often masking changes the verifier’s top-1 token.
Q3. Batched serving. I'm afraid I disagree with the claim that "Batch size one is standard in academic speculative-decoding evaluations". State-of-the-art speculative decoding approaches, including native MTP, DFlash, and EAGLE-3, all report serving performance across multiple batch sizes or request concurrency levels [1–3]. This is especially important for fairspec because cross-request expert overlap directly affects how many distinct experts the verifier must load.
For now, I will maintain my score.
[1] Gloeckle et al. Better & Faster Large Language Models via Multi-token Prediction. ICML, 2024.
[2] Chen et al. DFlash: Block Diffusion for Flash Speculative Decoding. ICML, 2026.
[3] Li et al. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. NeurIPS, 2025.
Rebuttal
We thank the reviewer for their thoughtful feedback. We address their comments below.
Benchmark-based quality evaluation is standard in the literature on approximate speculative decoding [1, 2, 5]. The survey [1] treats approximate speculative decoding as a distinct branch; relevant related work includes Blockwise (Stern et al., 2018) [3], SpecDec (Xia et al., 2023) [4], and BiLD (Kim et al., 2023) [5]. We will include this context in the literature review. Our paired benchmark tests detect no significant accuracy changes for FairSpec relative to full-expert SD. These results assess task quality; they do not establish preservation of the original output distribution.
We do compare FairSpec with our proposed alternatives, Lexicographic and Diagonal, that asymmetrically prioritize earlier tokens. Both perform worse in our experiments, supporting the simple worst-token coverage objective.
We have added direct comparisons with adaptive AcceptMoE and fixed-budget AcceptMoE-B on Qwen3-30B. Adaptive AcceptMoE shows no significant paired accuracy changes, but little latency benefit. It retains a large expert set, limiting verification savings, while selection overhead absorbs much of the remaining benefit. At the reference budgets, AcceptMoE-B achieves similar latency reductions to FairSpec: 15.4% versus 15.1% on Qwen. However, it loses on accuracy points. Thus, FairSpec combines substantial latency savings with no significant accuracy losses.
We provide the detailed table for Qwen3-30B-A3B-Instruct below. All runs use k = 7. Accuracy values are percentages. Latency changes are relative to each method’s SD baseline on the same machine. We will add AcceptMoE as a baseline for all models in the revision.
Batch size one is standard in academic speculative-decoding evaluations, includeing MoESpec and AcceptMoE. Our experiments use this setting on a single A100 with TP = 1. The expert-selection algorithm extends naturally to larger batches without changing the underlying selection objective, although gains depend on similarity and expert overlap across requests. We leave a detailed study of these trade-offs and TP/EP configurations to future work.
[1] Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding. Findings of ACL, 2024. Paper.
[2] Liang et al. AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding. 2026. Paper.
[3] Stern, Shazeer, and Uszkoreit. Blockwise Parallel Decoding for Deep Autoregressive Models. NeurIPS, 2018. Paper.
[4] Xia et al. Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation. Findings of EMNLP, 2023. Paper.
[5] Kim et al. Speculative Decoding with Big Little Decoder. NeurIPS, 2023. Paper.
[6] Lin et al. Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization. 2026. Paper.
[7] MoE-Spec: Expert Budgeting for Efficient Speculative Decoding. 2026. Paper.