- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies speculative decoding for MoE models under expert offloading. The key observation is that in tree-based speculative decoding, the experts activated by a draft tree are the union of its tokens' experts; when H2D transfers dominate decoding latency, a larger draft tree can be even slower than autoregressive decoding. The authors propose SpecFit, which selects the draft tree by maximizing predicted throughput rather than acceptance length. Experiments demonstrate decoding speedups.
Critical strengths
-
This paper is the first to observe that, in offloaded MoE inference, verifying multiple speculative tokens enlarges the union of activated experts and makes decoding more transfer-bound, and it proposes a solution validated across multiple GPUs and expert-cache capacities.
-
The theoretical modeling is rigorous.
Critical weaknesses
-
The performance model requires knowing which experts each token activates, i.e., it relies on ground-truth routing information that is unavailable at deployment time. The paper obtains this from a route atlas collected by running inference on the test set ahead of time, which is impossible in real deployment. The authors suggest that training a lightweight route predictor could mitigate this, but that introduces additional training overhead and inference cost; moreover, the predictor works the paper cites deploy one MLP predictor per MoE layer, rather than predicting expert activations of all layers before execution begins, so their reported accuracy figures are not a meaningful reference here.
-
The cost model does not account for factors such as sequence length; it is calibrated on GSM8K and then applied to other datasets, so its accuracy on unseen workloads is questionable.
-
There is no ablation that separates the contribution of "fewer draft tokens" from that of "expert-union-aware selection."
Questions for authors
-
Since a route atlas cannot be obtained in advance in real deployments, can you provide end-to-end throughput with a trained route predictor replacing the atlas?
-
Can you report the accuracy of the cost model on other datasets (e.g., CNN/DailyMail)?
-
Please add the following ablations: (a) EAGLE with smaller drafting depth and fewer draft tokens; (b) the cost model with β = 0. These would make it clear whether the speedup comes from "fewer draft tokens" or from "expert-union-aware selection."