acceptodds
Under review as a conference paper at ICLR 2027

FairSpec: Robust Expert Selection for Efficient MoE Speculative Decoding

Abstract

Speculative decoding accelerates large language-model (LLM) inference by using a lightweight drafter model to propose several tokens and verifying them simultaneously in one target-model pass. For mixture-of-experts (MoE) models, however, the speculative tokens may route to different experts, forcing each MoE layer to read more distinct expert parameters from GPU high-bandwidth memory and potentially making verification slower than autoregressive decoding. Existing methods cap this cost by retaining the experts with the greatest aggregate routing demand from speculative tokens, but aggregate selection can neglect individual tokens, shortening the accepted draft prefix and/or degrading model inference quality. We therefore propose FairSpec, a verifier-side expert routing method that allocates an expert budget that protects each speculated token's highest-ranked experts by exploiting expert overlap across tokens. FairSpec requires no model retraining or changes to the downstream MoE operator and is implemented as a fused, CUDA-graph-compatible routing kernel. We evaluate FairSpec against state-of-the-art speculative decoding methods that perform expert-subset selection. Across 4 benchmarks and 3 sparse MoE architectures, at comparable throughput, FairSpec achieves 10–50pp higher success rates under low and moderate expert budgets, with larger gains at lower budgets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, mixture of experts, llm, inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dUKLOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FairSpec, an expert selection method for speculative decoding for MoE models. Speculative decoding verifys multiple token at once, improving decoding throughput but also increasing memory pressure due to activating a larger number of experts. FairSpec selects experts according to token-wise experts ranking, providing theoretical guarantees to worst-token routing coverage. Experiments demonstrated improved success rate on several MoE models.

Critical strengths
  1. The paper addresses an important bottleneck when applying speculative decoding to MoE models. The observation of same sequence may have correlated experts preference is a reasonable motivation.
  2. The paper provides a simple but effective way for experts selection.
Critical weaknesses
  1. The most important concern would be experts masking shifts the original target distribution. Higher accepted length or throughput does not imply the generated distribution remains close to the original ones.
  2. The motivation emphasizes early-token rejection lead to a failure, but the proposed objective ignores token position.
  3. Experts overlapping isn't sufficiently validated, one single-batch requests is tested. Overlapping can also happens in larger batch serving scene, but the paper only discusses a specific situation that each requests corresponds to different domains.
Questions for authors

Q1: Can the authors quantify the distribution shift introduced by expert masking compared with the original MoE model?

Q2: Why does FairSpec not consider token positions in the allocation objective? How does FairSpec compare with acceptance-aware expert selection methods such as AcceptMoE?

Q3: How does expert overlap and FairSpec performance change under realistic continuous batching workloads?

2 comments

We thank the reviewer for their thoughtful feedback. We address their comments below.

Can the authors quantify the distribution shift introduced by expert masking compared with the original MoE model?

Benchmark-based quality evaluation is standard in the literature on approximate speculative decoding [1, 2, 5]. The survey [1] treats approximate speculative decoding as a distinct branch; relevant related work includes Blockwise (Stern et al., 2018) [3], SpecDec (Xia et al., 2023) [4], and BiLD (Kim et al., 2023) [5]. We will include this context in the literature review. Our paired benchmark tests detect no significant accuracy changes for FairSpec relative to full-expert SD. These results assess task quality; they do not establish preservation of the original output distribution.

Why does FairSpec not consider token positions in the allocation objective? How does FairSpec compare with acceptance-aware expert selection methods such as AcceptMoE?

We do compare FairSpec with our proposed alternatives, Lexicographic and Diagonal, that asymmetrically prioritize earlier tokens. Both perform worse in our experiments, supporting the simple worst-token coverage objective.

We have added direct comparisons with adaptive AcceptMoE and fixed-budget AcceptMoE-B on Qwen3-30B. Adaptive AcceptMoE shows no significant paired accuracy changes, but little latency benefit. It retains a large expert set, limiting verification savings, while selection overhead absorbs much of the remaining benefit. At the reference budgets, AcceptMoE-B achieves similar latency reductions to FairSpec: 15.4% versus 15.1% on Qwen. However, it loses on accuracy points. Thus, FairSpec combines substantial latency savings with no significant accuracy losses.

We provide the detailed table for Qwen3-30B-A3B-Instruct below. All runs use k = 7. Accuracy values are percentages. Latency changes are relative to each method’s SD baseline on the same machine. We will add AcceptMoE as a baseline for all models in the revision.

MethodExpertsGSM8KHumanEvalIFEvalMATH-500MBPPτms/tokenLatency vs. SD
FairSpec1696.493.382.886.687.63.414.26−15.1%
AcceptMoE-B1692.689.082.681.084.13.404.31−15.4%
AcceptMoEAdaptive96.392.782.486.887.43.465.12+0.4%

How does expert overlap and FairSpec performance change under realistic continuous batching workloads?

Batch size one is standard in academic speculative-decoding evaluations, includeing MoESpec and AcceptMoE. Our experiments use this setting on a single A100 with TP = 1. The expert-selection algorithm extends naturally to larger batches without changing the underlying selection objective, although gains depend on similarity and expert overlap across requests. We leave a detailed study of these trade-offs and TP/EP configurations to future work.


[1] Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding. Findings of ACL, 2024. Paper.

[2] Liang et al. AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding. 2026. Paper.

[3] Stern, Shazeer, and Uszkoreit. Blockwise Parallel Decoding for Deep Autoregressive Models. NeurIPS, 2018. Paper.

[4] Xia et al. Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation. Findings of EMNLP, 2023. Paper.

[5] Kim et al. Speculative Decoding with Big Little Decoder. NeurIPS, 2023. Paper.

[6] Lin et al. Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization. 2026. Paper.

[7] MoE-Spec: Expert Budgeting for Efficient Speculative Decoding. 2026. Paper.

Thank you for adding the AcceptMoE comparison. My concern regarding Q1 and Q3 remain unresolved.

Q1. Distribution shift. The results do not answer my question about how much expert masking changes the output distribution. Adding experiments that compare FairSpec with full-expert speculative decoding using the same prefixes, draft candidates, and verification positions and report token-level KL would better explain the shift. As for the greedy decoding, you can report how often masking changes the verifier’s top-1 token.

Q3. Batched serving. I'm afraid I disagree with the claim that "Batch size one is standard in academic speculative-decoding evaluations". State-of-the-art speculative decoding approaches, including native MTP, DFlash, and EAGLE-3, all report serving performance across multiple batch sizes or request concurrency levels [1–3]. This is especially important for fairspec because cross-request expert overlap directly affects how many distinct experts the verifier must load.

For now, I will maintain my score.

[1] Gloeckle et al. Better & Faster Large Language Models via Multi-token Prediction. ICML, 2024.

[2] Chen et al. DFlash: Block Diffusion for Flash Speculative Decoding. ICML, 2026.

[3] Li et al. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. NeurIPS, 2025.

Reviewer_ddosOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper studies verifier-side expert selection for MoE speculative decoding. Existing methods that select experts by aggregate routing probability can neglect experts important to individual tokens, potentially affecting generation quality and accepted length.
  • FairSpec allocates the expert budget in rounds according to each token’s expert preference ranking, using aggregate routing probability to fill the remaining capacity. The paper proves optimality for worst-token ranking-prefix coverage, provides a lower bound on retained routing mass, presents a fused kernel implementation, and evaluates quality and speed across three models and four tasks.
Critical strengths
  • The method is simple, with a clear connection between its theoretical formulation and implementation. The round-based selection rule provides an explicit worst-token coverage guarantee and can be implemented through a single top-MM selection. Its training-free design and integration into the verification stage give it practical value.
Critical weaknesses
  1. The methodological contribution primarily concerns the expert-selection rule within an existing framework, and the connection between the theoretical guarantees and practical gains remains insufficiently established.
    Compared with MoE-Spec, the paper retains the shared expert budget, subset masking, and subsequent verification procedure. The central change is to prioritize within-token expert rankings over aggregate routing probability. While the paper proves ranking-prefix coverage optimality and provides a lower bound on retained routing mass, these guarantees do not establish a direct quantitative relationship with final generation quality or accepted length.

  2. The experimental evaluation lacks a direct comparison with the closely related AcceptMoE method.
    The paper repeatedly discusses AcceptMoE as closely related work, but the Lexicographic baseline is only a heuristic inspired by its prioritization of earlier tokens. It does not implement commitment-probability weighting or adaptive budget sizing. Although the paper clearly describes the heuristic as such, it cannot substitute for a direct comparison with AcceptMoE. The current experiments therefore do not establish FairSpec’s quality–speed advantage over this closely related method. (Section4)

  3. Some overall claims extend beyond what the current experiments clearly support.
    Page 8 states that FairSpec consistently achieves better or on-par success rates compared with the other expert-subsetting methods. However, Table 2 reports 65.6% for FairSpec versus 67.6% for MoE-Spec on gpt-oss/MATH-500, and 91.1% versus 92.0% on Gemma/MBPP. These differences may not be statistically significant, but the paper doesn't defines a criterion for “on-par” performance. Under the reference budget in Table 2, the larger gains mainly occur on Qwen3, and the budget sweep in Figure 2 also covers only Qwen3. The claim of “10–50 percentage-point improvements” therefore needs to specify the models, budgets, and baselines to which it applies. Although Appendix Figure 6 provides five-seed results for Qwen3 at temperature 1, these do not resolve the uncertainty in comparisons among the main baselines or across models.

Questions for authors
  1. Could the authors provide further theoretical analysis or controlled experiments showing how ranking-prefix coverage and retained routing mass affect final generation quality and accepted length? Under what conditions does the proposed expert-selection rule yield practical gains over MoE-Spec?

  2. Could the authors include a direct comparison with the AcceptMoE method?

2 comments

We thank the reviewer for their thoughtful feedback. We address their comments below.

Could the authors provide further theoretical analysis or controlled experiments showing how ranking-prefix coverage and retained routing mass affect final generation quality and accepted length? Under what conditions does the proposed expert-selection rule yield practical gains over MoE-Spec?

Our motivation is that some experts or expert combinations are pivotal: removing them can substantially degrade model performance. Prior expert-selection work likewise emphasizes preserving indispensable expert combinations [6]. This motivates protecting each token’s highly ranked experts. The purpose of our theoretical model is to provide a succinct, useful representation of this selection problem. Our contribution is to show that the simple worst-token coverage objective yields robust empirical gains. We also compare with alternative objectives, including Lexicographic and Diagonal, which perform worse.

Could the authors include a direct comparison with the AcceptMoE method?

We have added direct comparisons with adaptive AcceptMoE and fixed-budget AcceptMoE-B on Qwen3-30B. Adaptive AcceptMoE shows no significant paired accuracy changes, but little latency benefit. It retains a large expert set, limiting verification savings, while selection overhead absorbs much of the remaining benefit. At the reference budgets, AcceptMoE-B achieves similar latency reductions to FairSpec: 15.4% versus 15.1% on Qwen. However, it loses on accuracy points. Thus, FairSpec combines substantial latency savings with no significant accuracy losses.

We provide the detailed table for Qwen3-30B-A3B-Instruct below. All runs use k = 7. Accuracy values are percentages. Latency changes are relative to each method’s SD baseline on the same machine. We will add AcceptMoE as a baseline for all models in the revision.

MethodExpertsGSM8KHumanEvalIFEvalMATH-500MBPPτms/tokenLatency vs. SD
FairSpec1696.493.382.886.687.63.414.26−15.1%
AcceptMoE-B1692.689.082.681.084.13.404.31−15.4%
AcceptMoEAdaptive96.392.782.486.887.43.465.12+0.4%

Page 8 states that FairSpec consistently achieves better or on-par success rates compared with the other expert-subsetting methods. However, Table 2 reports 65.6% for FairSpec versus 67.6% for MoE-Spec on gpt-oss/MATH-500, and 91.1% versus 92.0% on Gemma/MBPP. These differences may not be statistically significant, but the paper doesn't defines a criterion for “on-par” performance. Under the reference budget in Table 2, the larger gains mainly occur on Qwen3, and the budget sweep in Figure 2 also covers only Qwen3. The claim of “10–50 percentage-point improvements” therefore needs to specify the models, budgets, and baselines to which it applies. Although Appendix Figure 6 provides five-seed results for Qwen3 at temperature 1, these do not resolve the uncertainty in comparisons among the main baselines or across models.

Agreed; we should make the claim more explicit. The intended claim is that FairSpec is not materially worse across evaluated model–benchmark pairs. We will clairfy this in the revision.


[1] Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding. Findings of ACL, 2024. Paper.

[2] Liang et al. AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding. 2026. Paper.

[3] Stern, Shazeer, and Uszkoreit. Blockwise Parallel Decoding for Deep Autoregressive Models. NeurIPS, 2018. Paper.

[4] Xia et al. Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation. Findings of EMNLP, 2023. Paper.

[5] Kim et al. Speculative Decoding with Big Little Decoder. NeurIPS, 2023. Paper.

[6] Lin et al. Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization. 2026. Paper.

[7] MoE-Spec: Expert Budgeting for Efficient Speculative Decoding. 2026. Paper.

FairSpec introduces a meaningful alternative objective for budgeted expert selection, but the paper has not sufficiently established that its theoretical advantage translates into a consistent practical advantage over MoE-Spec. The empirical gains are concentrated on Qwen3, while results on other model families are largely on par or occasionally worse. Therefore, I will maintain my assessment.

Reviewer_RjhsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses the overhead of speculative decoding for mixture-of-experts (MoE) language models. During speculative verification, multiple tokens are processed in parallel, potentially activating a larger union of experts than autoregressive decoding. Consequently, expert-weight loading can become a bottleneck and offset the throughput benefits of speculation.

The paper proposes FAIRSPEC, a training-free expert-selection algorithm that prioritizes per-token expert rankings under a shared expert budget M. It first selects the union of all tokens’ top-ranked experts, then considers their second-ranked experts, and continues in successive rounds until the budget cannot accommodate another complete round. Any remaining budget is allocated according to aggregate routing mass among experts in the next round. This design aims to maximize the worst-token routing-prefix coverage under a fixed expert budget.

Critical strengths
  1. Well-motivated problem and intuitive solution. The paper identifies how aggregate expert selection can favor popular experts at the expense of tokens with less common routing preferences. FAIRSPEC addresses this by prioritizing each token’s highest-ranked experts first.
  2. Theoretical characterization. The paper formalizes worst-token routing-prefix coverage and proves that FAIRSPEC maximizes this objective under a fixed expert budget.
  3. Training-free design. FAIRSPEC requires no model retraining and operates on the verification side, making it relatively easy to integrate into existing inference systems.
Critical weaknesses
  1. Limited evaluation scope. All experiments use batch size one, tensor parallelism one, and a single NVIDIA A100 GPU, leaving FAIRSPEC’s effectiveness under realistic batched and distributed serving unclear.
  2. Insufficient analysis of output-distribution changes. Masking experts changes the target model’s computation, so preserving high-ranked experts does not guarantee preservation of the original output distribution or speculative-decoding behavior.
  3. The theoretical objective may not be sufficiently aligned with end-to-end speculative-decoding performance. The central theoretical contribution is the maximization of worst-token routing-prefix coverage. However, it remains unclear whether this objective is an effective surrogate for maximizing the acceptance rate or end-to-end throughput. In particular, the objective prioritizes expert ranks without directly accounting for the magnitude of routing weights or the probability that individual speculative tokens will be accepted. A higher worst-token coverage therefore does not necessarily translate into better speculative-decoding performance.
Questions for authors
  1. Does FAIRSPEC retain its advantages under other parallel configurations (e.g. larger-scaled distributed inference, like TP and EP?)
  2. Can the effectiveness of the method be more clearly attributed to the theoretical objective, in addition to the empirical results?
1 comment

We thank the reviewer for their thoughtful feedback. We address their comments below.

Does FAIRSPEC retain its advantages under other parallel configurations (e.g. larger-scaled distributed inference, like TP and EP?)

Batch size one is standard in academic speculative-decoding evaluations, includeing MoESpec and AcceptMoE. Our experiments use this setting on a single A100 with TP = 1. The expert-selection algorithm extends naturally to larger batches without changing the underlying selection objective, although gains depend on similarity and expert overlap across requests. We leave a detailed study of these trade-offs and TP/EP configurations to future work.

Insufficient analysis of output-distribution changes. Masking experts changes the target model’s computation, so preserving high-ranked experts does not guarantee preservation of the original output distribution or speculative-decoding behavior.

Benchmark-based quality evaluation is standard in the literature on approximate speculative decoding [1, 2, 5]. The survey [1] treats approximate speculative decoding as a distinct branch; relevant related work includes Blockwise (Stern et al., 2018) [3], SpecDec (Xia et al., 2023) [4], and BiLD (Kim et al., 2023) [5]. We will include this context in the literature review. Our paired benchmark tests detect no significant accuracy changes for FairSpec relative to full-expert SD. These results assess task quality; they do not establish preservation of the original output distribution.

Can the effectiveness of the method be more clearly attributed to the theoretical objective, in addition to the empirical results?

The purpose of theoretical modeling is to capture the relevant behavior in a succinct, useful model, even if it is imperfect. Our contribution is to show that a simple objective—maximizing worst-token routing-prefix coverage—is sufficient to produce robust gains in expert selection in our experiments. We also compare with alternative objectives, including Lexicographic and Diagonal, which perform worse.


[1] Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding. Findings of ACL, 2024. Paper.

[2] Liang et al. AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding. 2026. Paper.

[3] Stern, Shazeer, and Uszkoreit. Blockwise Parallel Decoding for Deep Autoregressive Models. NeurIPS, 2018. Paper.

[4] Xia et al. Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation. Findings of EMNLP, 2023. Paper.

[5] Kim et al. Speculative Decoding with Big Little Decoder. NeurIPS, 2023. Paper.

[6] Lin et al. Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization. 2026. Paper.

[7] MoE-Spec: Expert Budgeting for Efficient Speculative Decoding. 2026. Paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.