acceptodds
Under review as a conference paper at ICLR 2027

SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference

Abstract

Mixture-of-experts (MoE) models activate only a few experts per token, but when their expert weights are offloaded to host memory, host-to-device (H2D) expert transfers enter the critical path of decoding. Speculative decoding verifies several draft tokens in one forward pass, yet in an MoE model each verified token activates its own experts, so a larger draft tree transfers more experts. Because of this H2D cost, speculative decoding commits several tokens per verification cycle yet is sometimes even slower than autoregressive decoding. Based on this observation, we propose SpecFit, which chooses the verified draft subtree to maximize predicted throughput rather than acceptance length. SpecFit estimates the expected number of committed tokens from draft path probabilities and predicts cycle time from the node count and the per-layer union of the experts the subtree activates; this union is a monotone submodular coverage function, so each node pays only for the experts it adds. An exact search over ancestor-closed subtrees returns the subtree with the highest predicted throughput; run after each draft level, the same search also decides when to stop drafting, and the unmodified target verifies the result. On CNN/DailyMail summarization and a GSM8K subset with Qwen3-30B-A3B, SpecFit decodes 21.6% and 7.0% faster than EAGLE-3 with the same expert cache and 1.97 and 1.72 as fast as vanilla EAGLE-3.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large Language Models, Mixture-of-Experts, Speculative Decoding, Inference Acceleration, Offloading

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vEuROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies speculative decoding for MoE models under expert offloading. The key observation is that in tree-based speculative decoding, the experts activated by a draft tree are the union of its tokens' experts; when H2D transfers dominate decoding latency, a larger draft tree can be even slower than autoregressive decoding. The authors propose SpecFit, which selects the draft tree by maximizing predicted throughput rather than acceptance length. Experiments demonstrate decoding speedups.

Critical strengths
  1. This paper is the first to observe that, in offloaded MoE inference, verifying multiple speculative tokens enlarges the union of activated experts and makes decoding more transfer-bound, and it proposes a solution validated across multiple GPUs and expert-cache capacities.

  2. The theoretical modeling is rigorous.

Critical weaknesses
  1. The performance model requires knowing which experts each token activates, i.e., it relies on ground-truth routing information that is unavailable at deployment time. The paper obtains this from a route atlas collected by running inference on the test set ahead of time, which is impossible in real deployment. The authors suggest that training a lightweight route predictor could mitigate this, but that introduces additional training overhead and inference cost; moreover, the predictor works the paper cites deploy one MLP predictor per MoE layer, rather than predicting expert activations of all layers before execution begins, so their reported accuracy figures are not a meaningful reference here.

  2. The cost model does not account for factors such as sequence length; it is calibrated on GSM8K and then applied to other datasets, so its accuracy on unseen workloads is questionable.

  3. There is no ablation that separates the contribution of "fewer draft tokens" from that of "expert-union-aware selection."

Questions for authors
  1. Since a route atlas cannot be obtained in advance in real deployments, can you provide end-to-end throughput with a trained route predictor replacing the atlas?

  2. Can you report the accuracy of the cost model on other datasets (e.g., CNN/DailyMail)?

  3. Please add the following ablations: (a) EAGLE with smaller drafting depth and fewer draft tokens; (b) the cost model with β = 0. These would make it clear whether the speedup comes from "fewer draft tokens" or from "expert-union-aware selection."

Reviewer_TjZLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an interesting optimization approach by exploiting the trade-off between costly token generation in MoE vs acceptance rates. It presents a valuable idea to the community and demonstrates via experiments involving strong baselines. However, tree-based speculative decoding are known to only work well in low batch inference since memory overhead can be high. It would be valuable for the community if the experiments here can be extended for higher concurrency settings to fully evaluate the proposed technique.

Edit: Raising score from 2 to 3 after concerns have been adressed,

Critical strengths
  1. Paper explores an interesting trade-off between costly token generation in MoE vs acceptance rates.
  2. Shows commendable improvement over strong baselines.
Critical weaknesses
  1. Experiments only done with single/low concurrency.
Questions for authors
  1. All experiments are conducted at batch size 1. Can the authors provide multi-request batched decoding experiments to demonstrate whether the reported speedups persist at larger batch sizes?

  2. Tree-based speculative decoding can introduce substantial memory overhead. Can the authors evaluate GPU memory consumption and demonstrate whether the proposed method remains effective under realistic memory pressure, particularly with multiple concurrent requests?

Reviewer_c2pZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses EAGLE-3-like speculative decoding for MoE models. Verifying each node in the draft tree of this drafter requires loading a large fraction of experts, which can lead to slowdowns even compared to non-speculative decoding. The paper introduces a novel method, SpecFit, which removes unpromising nodes from the draft tree (and all subtrees). The method hinges on the insight that "pruning nodes raises throughput if it saves a larger fraction of cycle time (dominated by expert loading, but including attention and other factors) than of committed tokens". In other words, if a node incurs large additional load time but adds few additional tokens, it is pruned from the draft tree. The authors show consistent gains with their method of EAGLE-3 and autoregressive (non-speculative-decoding) baselines.

While the basic idea is solid, the paper suffers from several weaknesses (mainly how the number of experts added by a node in the draft tree is computed, and several points about evaluation), leading to the "weak reject" rating. If this omission is addressed in the rebuttal I'd be willing to amend my rating.

Critical strengths

The paper is well written and easy to follow. The method sections are described clearly and mostly easy to follow. Sometimes the authors resort to jargon, but with enough context to make sense of the text.

The method itself is well motivated and useful. While MoEs are deployed increasingly often, speculative decode for MoEs is a gap in literature, which the authors address with an elegant method.

The authors provide sensible experiments and ablations showing the benefits of SpecFit over EAGLE-3.

Critical weaknesses
  1. Counting how many experts are added by each node can only be done by forwarding tokens through the MoE, since it requires router predictions at each MoE layer. The authors circumvent this issue through the use of a "route atlas". How this atlas is created or used, and what effect it has on drafting tokens during inference, is described in very little detail, leaving much to the imagination.

  2. The authors admit the route atlas is not deployable, and briefly describe several realistic approaches to predicting which experts a token activates in section 4.6. However, the authors do not provide any inference token rates with realistic/deployable expert routing predictions. The authors mention this is "left to future work", however, since this is a crucial aspect of the, and, by the references in the paper, a solved problem, the authors should provide end-to-end latency timings that include a realistic token routing predictor whose overhead is included in the reported token/second numbers.

  3. The authors only evaluate their method on Qwen3-30B-A3B. Fully understanding whether the method has general benefits requires evaluation on a broader set of models.

  4. In the ablations in section 4.4 "Sensitivity to Offload Pressure", the authors only compare to EAGLE-3, but not to AR.

[Minor] In some instances AR yields lower H2D/token than the method proposed by the authors (although admittedly, the cycle model includes more than just expert load time). The authors neither acknowledge this fact nor provide an analysis or hypothesis explaining this phenomenon.

Questions for authors
  1. Can the authors provide more details on the route atlas?
  2. What is the variance in token generation rates? E.g. when running with different seeds, which would generate different drafts for the same prompt, potentially yielding different token generation rates downstream. While the margins of improvement or non-trivial, but it's unclear whether they fall within uncertainty bounds.
  3. Evaluation only occurs on fairly old hardware (A100-40GB; RTX 4090 24GB). Would the picture change with larger cache, faster bandwidth to on-chip SRAM, etc? Do the concerns that motivate the paper still hold and will SpecFit still address them sufficiently?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.